Ashutosh's Oracle DBA Blog:

Please check log crsctl.* in /tmp dir, Will see like below error. $ cat crsctl.2717 Failure -2 opening file handle for (c6t0d1)  --------->> Voting disk not access from node1 Failure 1 checking the CSS voting device. --------->> Voting disk not access from node1 ocssd1.log on node1- more  /u01/crs/oracle/product/10.1.0/crs/css/log/ocssd1.log 2008-09-09 12:06:43.683 [5] >TRACE:   clssgmClientConnectMsg: Connect from con(600000000080fca0), proc(6000000000b497b0)  pid() 2008-09-09 12:08:32.836 [5] >TRACE:   clssgmClientConnectMsg: Connect from con(6000000000810120), proc(6000000000b168e0)  pid() 2008-09-09 12:08:32.848 [5] >TRACE:   clsc_receive: (6000000000b3f590) Connection failed, transport error (507, 0, 0) 2008-09-09 12:08:32.848 [5] >TRACE:   clscreceive: (6000000000810120) Physical connection (6000000000b3f590) not active, rc 11 2008-09-09 12:08:32.848 [5] >TRACE:   clscreceive: (60000000008105a0) P...

VIPCA is fail during RAC 10g.  crs_stat -t output shows VIP is offline and starting it explicitly gives error : CRS-0215: Could not start resource 'ora.dbtest2.vip'. Example: crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....st2.gsd application ONLINE ONLINE dbtest2 ora....st2.ons application ONLINE ONLINE dbtest2 ora....st2.vip application ONLINE OFFLINE # ./srvctl start nodeapps -n dbtest2 dbtest2:ora.dbtest2.vip:Interface eri0 checked failed (host=dbtest2) dbtest2:ora.dbtest2.vip:Failed to start VIP 10.11.11.198 (host=dbtest2) dbtest2:ora.dbtest2.vip:Interface eri0 checked failed (host=dbtest2) dbtest2:ora.dbtest2.vip:Failed to start VIP 10.11.11.198 (host=dbtest2) CRS-1006: No more members to consider CRS-0215: Could not start resource 'ora.dbtest2.vip'. CRS-0210: Could not find resource ora.dbtest2.LISTEN...

I want to share my experience when I was adding new node3 existing two node RAC node1 and node2 environment. Due to wrong disk mapping on node3, one ASM disk group has got corrupted. We was using external redundancy. Our existing RAC nodes were node1 and node2 during root.sh, adding node3 entry into OCR file, but the OCR disk (/dev/rhdisk5) on node 3 not pointing to the actual OCR disk (/dev/rhdisk5) on node 1&2. It was pointing to one asm disk (/dev/rhdisk3) which was part of ASMDB2 disk group on node1&2. ANALYSIS -------------- Check dd if=/dev/rhdisk5 bs=8192 count=1 |od –x   ------- on node3 it was showing different contents from node 1 and node 2 disk contents. So OCR on node 3 is not pointing to correct OCR disk on node 1&2. Solution: For ASM disk part: I identified all disk path of ASMDB2 disk group this were (/dev/rhdisk3, /dev/rhisk2 and /dev/rhdisk1). (By view v$asm_disk) Note: We should know all disk are using ASMDB2. Care should be taken at...