CRS daemon not up after reboot node1 on two nodes RAC

-
Please check log crsctl.* in /tmp dir, Will see like below error.

$ cat crsctl.2717
Failure -2 opening file handle for (c6t0d1)  --------->> Voting disk not access from node1
Failure 1 checking the CSS voting device. --------->> Voting disk not access from node1

ocssd1.log on node1-
more  /u01/crs/oracle/product/10.1.0/crs/css/log/ocssd1.log

2008-09-09 12:06:43.683 [5] >TRACE:   clssgmClientConnectMsg: Connect from con(600000000080fca0), proc(6000000000b497b0)  pid()
2008-09-09 12:08:32.836 [5] >TRACE:   clssgmClientConnectMsg: Connect from con(6000000000810120), proc(6000000000b168e0)  pid()
2008-09-09 12:08:32.848 [5] >TRACE:   clsc_receive: (6000000000b3f590) Connection failed, transport error (507, 0, 0)
2008-09-09 12:08:32.848 [5] >TRACE:   clscreceive: (6000000000810120) Physical connection (6000000000b3f590) not active, rc 11
2008-09-09 12:08:32.848 [5] >TRACE:   clscreceive: (60000000008105a0) Physical connection (6000000000b3f590) not active, rc 11

2008-09-09 14:47:25.784: Start of `ora.nsdl.nsdl1.nsdl2.sa` on member `appdb02` failed.
2008-09-09 17:36:13.629: [RUNCONTEXT::RUNSCRIPT:75412] StartResource error for ora.nsdl.nsdl1.nsdl2.srv error code = 1
2008-09-09 17:44:20.842: [RUNCONTEXT::RUNSCRIPT:75445] StartResource error for ora.nsdl.nsdl1.nsdl2.sa error code = 1
2008-09-09 17:45:14.447: [RUNCONTEXT::RUNSCRIPT:75457] StartResource error for ora.nsdl.nsdl1.nsdl2.srv error code = 1
2008-09-09 18:23:06.903: [RUNCONTEXT::RUNSCRIPT:75576] StartResource error for ora.nsdl.nsdl1.nsdl2.sa error code = 1
2008-09-09 18:24:19.130: [RUNCONTEXT::RUNSCRIPT:60] CheckResource error for ora.appdb01.ASM1.asm error code = 1
`ora.appdb01.ASM1.asm` on `appdb01` went OFFLINE unexpectedly
2008-09-09 18:24:19.162: Attempting to stop `ora.appdb01.ASM1.asm` on member `appdb01`
2008-09-09 18:24:40.582: Stop of `ora.appdb01.ons` on member `appdb01` succeeded
2008-09-09 18:24:40.635: Attempting to stop `ora.nsdl.nsdl1.inst` on member `appdb01`
2008-09-09 18:24:42.525: Stop of `ora.nsdl.nsdl1.inst` on member `appdb01` succeeded.
2008-09-09 18:24:42.635: Attempting to stop `ora.appdb01.LISTENER_APPDB01.lsnr`on member `appdb01`
2008-09-09 18:24:43.364: Stop of `ora.appdb01.LISTENER_APPDB01.lsnr` on member `appdb01` succeeded.
2008-09-09 18:24:43.381: CRSD-2: Complete Shutdown Application Request
2008-09-09 18:24:53   : Oracle CRSD got SIGTERM, exiting
2008-09-09 20:13:04.008: CRSD-2: Sending Shutdown Application Request
2008-09-09 20:13:27.171: CRSD-2: ALERT [MAIN:1] Error sending request to running CRSD
2008-09-09 20:13:27.174: CRSD-2: ALERT [MAIN:1] SHUTDOWNAPPS Could not send application shutdown request to CRSD
2008-09-09 20:15:06.004: CRSD-2: Sending Shutdown Application Request
2008-09-09 20:15:29.141: CRSD-2: ALERT [MAIN:1] Error sending request to running  CRSD
2008-09-09 20:15:29.141: CRSD-2: ALERT [MAIN:1] SHUTDOWNAPPS Could not send application shutdown request to CRSD

Check ocr location

$ cat /var/opt/oracle/ocr.loc
ocrconfig_loc=/dev/rdsk/c6t0d2
local_only=FALSE

Check permission on node1 and node2


on node1
-----------
crw-r-----   1 bin     bin 188 0x060200 Sep 10 11:45 c6t0d2
crw-r--r--   1 bin     bin       188 0x060300 Sep 10 11:45 c6t0d3
crw-rw----   1 bin     bin         188 0x060100 Sep 10 11:45 c6t0d1
crw-r--r--   1 bin     bin 188 0x060400 Sep 10 11:45 c6t0d4

Problem--- >>> Wrong onwership of OCR,VOTING,ASM disk on node1 after reboot node1

on node2
------------
crw-r-----   1 root       oinstall   188 0x060200 Sep 10 11:45 c6t0d2  
crw-r--r--   1 oracle     dba        188 0x060300 Sep 10 11:45 c6t0d3
crw-rw----   1 oracle     dba        188 0x060100 Sep 10 11:45 c6t0d1
crw-r--r--   1 oracle     dba        188 0x060400 Sep 10 11:45 c6t0d4

Solution
---------
Change the ownership and permission of OCR,VOTING,ASM disk on node1 as node2.

OCR has root ownership and other oracle.

/dev/rdsk/c6t0d2   -->>   ocr disk
/dev/rdsk/c6t0d1   ---->> voting disk
/dev/rdsk/c6t0d3    ---->> ASM disk
/dev/rdsk/c6t0d4    ---->> ASM disk

Comments

Post a Comment

Popular posts from this blog

CRS-0215: Could not start resource 'ora..vip' during vipca in RAC 10g

-
VIPCA is fail during RAC 10g.  crs_stat -t output shows VIP is offline and starting it explicitly gives error : CRS-0215: Could not start resource 'ora.dbtest2.vip'. Example: crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....st2.gsd application ONLINE ONLINE dbtest2 ora....st2.ons application ONLINE ONLINE dbtest2 ora....st2.vip application ONLINE OFFLINE # ./srvctl start nodeapps -n dbtest2 dbtest2:ora.dbtest2.vip:Interface eri0 checked failed (host=dbtest2) dbtest2:ora.dbtest2.vip:Failed to start VIP 10.11.11.198 (host=dbtest2) dbtest2:ora.dbtest2.vip:Interface eri0 checked failed (host=dbtest2) dbtest2:ora.dbtest2.vip:Failed to start VIP 10.11.11.198 (host=dbtest2) CRS-1006: No more members to consider CRS-0215: Could not start resource 'ora.dbtest2.vip'. CRS-0210: Could not find resource ora.dbtest2.LISTEN...
Read more

Rebuild Corrupt ASM disk

-
I want to share my experience when I was adding new node3 existing two node RAC node1 and node2 environment. Due to wrong disk mapping on node3, one ASM disk group has got corrupted. We was using external redundancy. Our existing RAC nodes were node1 and node2 during root.sh, adding node3 entry into OCR file, but the OCR disk (/dev/rhdisk5) on node 3 not pointing to the actual OCR disk (/dev/rhdisk5) on node 1&2. It was pointing to one asm disk (/dev/rhdisk3) which was part of ASMDB2 disk group on node1&2. ANALYSIS -------------- Check dd if=/dev/rhdisk5 bs=8192 count=1 |od –x   ------- on node3 it was showing different contents from node 1 and node 2 disk contents. So OCR on node 3 is not pointing to correct OCR disk on node 1&2. Solution: For ASM disk part: I identified all disk path of ASMDB2 disk group this were (/dev/rhdisk3, /dev/rhisk2 and /dev/rhdisk1). (By view v$asm_disk) Note: We should know all disk are using ASMDB2. Care should be taken at...
Read more

How to perform out-of place manual upgrade from 11.2.0.1 to 11.2.0.2

-
Solution : 11.2.0.2 patch set software is a full release. 11.2.0.2 OUI does not update existing 11.2.0.1 ORACLE_HOME files and performs a new installation. There are two method Out-of-Place (Oracle's recommended approach) In-Place But here, we need to understand what exactly this method is: In prior releases, how it worked   DB Upgrade 8i to 10g (Out-Of-Place) We install Oracle binaries for 10g at new location, then use DBUA or catupgrd.sql script to upgrade DB. This is Out-of-Place migration (i.e. created new oracle home, from inventory perspective). DB Upgrade 10.2.0.1 to 10.2.0.4 (In-Place) Here we won’t install 10.2.0.4 patchset in new home ( Oracle won’t let us do that, as it is a patch ), but select the existing home to upgrade. We also keep a backup of original 10.2.0.1, as we may have to fail back. As 11.2.0.2 is not a patch and full install .  It does not allow us to install binaries in existing 11.2.0.1 home (i.e. In-Place). But as I mentioned above, we can per...
Read more