CRS daemon not up after reboot node1 on two nodes RAC

Please check log crsctl.* in /tmp dir, Will see like below error.

$ cat crsctl.2717
Failure -2 opening file handle for (c6t0d1)  --------->> Voting disk not access from node1
Failure 1 checking the CSS voting device. --------->> Voting disk not access from node1

ocssd1.log on node1-
more  /u01/crs/oracle/product/10.1.0/crs/css/log/ocssd1.log

2008-09-09 12:06:43.683 [5] >TRACE:   clssgmClientConnectMsg: Connect from con(600000000080fca0), proc(6000000000b497b0)  pid()
2008-09-09 12:08:32.836 [5] >TRACE:   clssgmClientConnectMsg: Connect from con(6000000000810120), proc(6000000000b168e0)  pid()
2008-09-09 12:08:32.848 [5] >TRACE:   clsc_receive: (6000000000b3f590) Connection failed, transport error (507, 0, 0)
2008-09-09 12:08:32.848 [5] >TRACE:   clscreceive: (6000000000810120) Physical connection (6000000000b3f590) not active, rc 11
2008-09-09 12:08:32.848 [5] >TRACE:   clscreceive: (60000000008105a0) Physical connection (6000000000b3f590) not active, rc 11

2008-09-09 14:47:25.784: Start of `ora.nsdl.nsdl1.nsdl2.sa` on member `appdb02` failed.
2008-09-09 17:36:13.629: [RUNCONTEXT::RUNSCRIPT:75412] StartResource error for ora.nsdl.nsdl1.nsdl2.srv error code = 1
2008-09-09 17:44:20.842: [RUNCONTEXT::RUNSCRIPT:75445] StartResource error for ora.nsdl.nsdl1.nsdl2.sa error code = 1
2008-09-09 17:45:14.447: [RUNCONTEXT::RUNSCRIPT:75457] StartResource error for ora.nsdl.nsdl1.nsdl2.srv error code = 1
2008-09-09 18:23:06.903: [RUNCONTEXT::RUNSCRIPT:75576] StartResource error for ora.nsdl.nsdl1.nsdl2.sa error code = 1
2008-09-09 18:24:19.130: [RUNCONTEXT::RUNSCRIPT:60] CheckResource error for ora.appdb01.ASM1.asm error code = 1
`ora.appdb01.ASM1.asm` on `appdb01` went OFFLINE unexpectedly
2008-09-09 18:24:19.162: Attempting to stop `ora.appdb01.ASM1.asm` on member `appdb01`
2008-09-09 18:24:40.582: Stop of `ora.appdb01.ons` on member `appdb01` succeeded
2008-09-09 18:24:40.635: Attempting to stop `ora.nsdl.nsdl1.inst` on member `appdb01`
2008-09-09 18:24:42.525: Stop of `ora.nsdl.nsdl1.inst` on member `appdb01` succeeded.
2008-09-09 18:24:42.635: Attempting to stop `ora.appdb01.LISTENER_APPDB01.lsnr`on member `appdb01`
2008-09-09 18:24:43.364: Stop of `ora.appdb01.LISTENER_APPDB01.lsnr` on member `appdb01` succeeded.
2008-09-09 18:24:43.381: CRSD-2: Complete Shutdown Application Request
2008-09-09 18:24:53   : Oracle CRSD got SIGTERM, exiting
2008-09-09 20:13:04.008: CRSD-2: Sending Shutdown Application Request
2008-09-09 20:13:27.171: CRSD-2: ALERT [MAIN:1] Error sending request to running CRSD
2008-09-09 20:13:27.174: CRSD-2: ALERT [MAIN:1] SHUTDOWNAPPS Could not send application shutdown request to CRSD
2008-09-09 20:15:06.004: CRSD-2: Sending Shutdown Application Request
2008-09-09 20:15:29.141: CRSD-2: ALERT [MAIN:1] Error sending request to running  CRSD
2008-09-09 20:15:29.141: CRSD-2: ALERT [MAIN:1] SHUTDOWNAPPS Could not send application shutdown request to CRSD

Check ocr location

$ cat /var/opt/oracle/ocr.loc
ocrconfig_loc=/dev/rdsk/c6t0d2
local_only=FALSE

Check permission on node1 and node2


on node1
-----------
crw-r-----   1 bin     bin 188 0x060200 Sep 10 11:45 c6t0d2
crw-r--r--   1 bin     bin       188 0x060300 Sep 10 11:45 c6t0d3
crw-rw----   1 bin     bin         188 0x060100 Sep 10 11:45 c6t0d1
crw-r--r--   1 bin     bin 188 0x060400 Sep 10 11:45 c6t0d4

Problem--- >>> Wrong onwership of OCR,VOTING,ASM disk on node1 after reboot node1

on node2
------------
crw-r-----   1 root       oinstall   188 0x060200 Sep 10 11:45 c6t0d2  
crw-r--r--   1 oracle     dba        188 0x060300 Sep 10 11:45 c6t0d3
crw-rw----   1 oracle     dba        188 0x060100 Sep 10 11:45 c6t0d1
crw-r--r--   1 oracle     dba        188 0x060400 Sep 10 11:45 c6t0d4

Solution
---------
Change the ownership and permission of OCR,VOTING,ASM disk on node1 as node2.

OCR has root ownership and other oracle.

/dev/rdsk/c6t0d2   -->>   ocr disk
/dev/rdsk/c6t0d1   ---->> voting disk
/dev/rdsk/c6t0d3    ---->> ASM disk
/dev/rdsk/c6t0d4    ---->> ASM disk

Comments

Popular posts from this blog

Rebuild Corrupt ASM disk

How to perform out-of place manual upgrade from 11.2.0.1 to 11.2.0.2