Rebuild Corrupt ASM disk

I want to share my experience when I was adding new node3 existing two node RAC node1 and node2 environment. Due to wrong disk mapping on node3, one ASM disk group has got corrupted. We was using external redundancy.

Our existing RAC nodes were node1 and node2
during root.sh, adding node3 entry into OCR file, but the OCR disk (/dev/rhdisk5) on node 3 not pointing to the actual OCR disk (/dev/rhdisk5) on node 1&2.
It was pointing to one asm disk (/dev/rhdisk3) which was part of ASMDB2 disk group on node1&2.

ANALYSIS
--------------
Check dd if=/dev/rhdisk5 bs=8192 count=1 |od –x   ------- on node3

it was showing different contents from node 1 and node 2 disk contents.

So OCR on node 3 is not pointing to correct OCR disk on node 1&2.

Solution:

For ASM disk part: I identified all disk path of ASMDB2 disk group this were (/dev/rhdisk3, /dev/rhisk2 and /dev/rhdisk1). (By view v$asm_disk)
Note: We should know all disk are using ASMDB2. Care should be taken at this step as the correct disk being picked up, otherwise it could be other data loss.

1. Shutdown ASM instance

2. dd zero out the disk header for disks of ASMDB2 disk group:

$ dd if=/dev/zero of=/dev/rhdisk3 bs=8192 count=1
$ dd  if=/dev/zero of=/dev/rhdisk2 bs=8192 count=1
$ dd if=/dev/zero of=/dev/rhdisk1 bs=8192 count=1

3. restart ASM instance
4. recreate ASMDB2 disk by sqlplus
SQl> create diskgroup ASMDB2 external redundancy disk '/dev/rhdisk3' size 102400M, '/dev/rhdisk2' size 102400M, '/dev/rdisk1' size 102400M;
5. alter diskgroup ASMDB2 mount
6. Restore all datafile of this ASMDB2 diskgroup from good RMAN backup;

Comments

Post a Comment

Popular posts from this blog

CRS-0215: Could not start resource 'ora..vip' during vipca in RAC 10g

How to perform out-of place manual upgrade from 11.2.0.1 to 11.2.0.2

CRS daemon not up after reboot node1 on two nodes RAC