怎么修kingsql中RAC节点故障

63次阅读

共计 9472 个字符，预计需要花费 24 分钟才能阅读完成。

这篇文章主要讲解了“怎么修 kingsql 中 RAC 节点故障”，文中的讲解内容简单清晰，易于学习与理解，下面请大家跟着丸趣 TV 小编的思路慢慢深入，一起来研究和学习“怎么修 kingsql 中 RAC 节点故障”吧！

很久之前安装的虚拟机，今天开机之后，rac2 和 rac3 集群服务无法启动

[grid@rac2 cssd]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
[grid@rac3 cssd]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
[grid@rac1 ~]$ srvctl status database -d kingsql
Instance kingsql1 is running on node rac1
Instance kingsql2 is not running on node rac2
Instance kingsql3 is not running on node rac3

rac2 的 cssd 日志发现问题

2016-04-19 17:00:47.205: [ CSSD][3666859776]clssnmvDHBValidateNCopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 356447081, wrtcnt, 294277, LATS 4294697710, lastSeqNo 294274, uniqueness 1461034467, timestamp 1461056437/21174804
2016-04-19 17:00:47.205: [ CSSD][3666859776]clssnmvDHBValidateNCopy: node 3, rac3, has a disk HB, but no network HB, DHB has rcfg 356447080, wrtcnt, 41563, LATS 4294697710, lastSeqNo 41560, uniqueness 1463643843, timestamp 1463643914/18290394
[grid@rac1 ~]$ crs_stop -all
CRS-2500: Cannot stop resource ora.gsd as it is not running
Attempting to stop `ora.oc4j` on member `rac1`
Attempting to stop `ora.CRS.dg` on member `rac1`
Attempting to stop `ora.DATA.dg` on member `rac1`
Attempting to stop `ora.kingsql.db` on member `rac1`
Attempting to stop `ora.ons` on member `rac1`
CRS-2789: Cannot stop resource ora.gsd as it is not running on server rac1
Stop of `ora.ons` on member `rac1` succeeded.
Attempting to stop `ora.cvu` on member `rac1`
Attempting to stop `ora.rac2.vip` on member `rac1`
Attempting to stop `ora.rac3.vip` on member `rac1`
Attempting to stop `ora.LISTENER_SCAN1.lsnr` on member `rac1`
Stop of `ora.rac3.vip` on member `rac1` succeeded.
Stop of `ora.rac2.vip` on member `rac1` succeeded.
Stop of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
Attempting to stop `ora.scan1.vip` on member `rac1`
Stop of `ora.scan1.vip` on member `rac1` succeeded.
Stop of `ora.oc4j` on member `rac1` succeeded.
Stop of `ora.cvu` on member `rac1` succeeded.
Attempting to stop `ora.net1.network` on member `rac1`
Stop of `ora.net1.network` on member `rac1` succeeded.
Stop of `ora.kingsql.db` on member `rac1` succeeded.
Stop of `ora.DATA.dg` on member `rac1` succeeded.
Stop of `ora.CRS.dg` on member `rac1` succeeded.
Attempting to stop `ora.asm` on member `rac1`
Stop of `ora.asm` on member `rac1` succeeded.
Attempting to stop `ora.asm` on member `rac1`
– 卡在这不动了，于是执行 crsctl stop crs
CRS-0184: Cannot communicate with the CRS daemon.
[root@rac1 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on rac1
CRS-2673: Attempting to stop ora.mdnsd on rac1
CRS-2673: Attempting to stop ora.ctssd on rac1
CRS-2673: Attempting to stop ora.evmd on rac1
CRS-2673: Attempting to stop ora.cluster_interconnect.haip on rac1
CRS-2677: Stop of ora.cluster_interconnect.haip on rac1 succeeded
CRS-2677: Stop of ora.mdnsd on rac1 succeeded
CRS-2677: Stop of ora.evmd on rac1 succeeded
CRS-2677: Stop of ora.ctssd on rac1 succeeded
CRS-2673: Attempting to stop ora.cssd on rac1
CRS-2677: Stop of ora.cssd on rac1 succeeded
CRS-2673: Attempting to stop ora.crf on rac1
CRS-2677: Stop of ora.crf on rac1 succeeded
CRS-2673: Attempting to stop ora.gipcd on rac1
CRS-2677: Stop of ora.gipcd on rac1 succeeded
CRS-2673: Attempting to stop ora.gpnpd on rac1
CRS-2677: Stop of ora.gpnpd on rac1 succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on rac1 has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
[root@rac3 ~]# /u01/app/11.2.3/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on rac3
CRS-2673: Attempting to stop ora.crsd on rac3
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on rac3
CRS-2673: Attempting to stop ora.CRS.dg on rac3
CRS-2673: Attempting to stop ora.DATA.dg on rac3
CRS-2677: Stop of ora.DATA.dg on rac3 succeeded
CRS-2677: Stop of ora.CRS.dg on rac3 succeeded
CRS-2673: Attempting to stop ora.asm on rac3
CRS-2677: Stop of ora.asm on rac3 succeeded
CRS-2673: Attempting to stop ora.ons on rac3
CRS-2677: Stop of ora.ons on rac3 succeeded
CRS-2673: Attempting to stop ora.net1.network on rac3
CRS-2677: Stop of ora.net1.network on rac3 succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on rac3 has completed
CRS-2677: Stop of ora.crsd on rac3 succeeded
CRS-2673: Attempting to stop ora.ctssd on rac3
CRS-2673: Attempting to stop ora.evmd on rac3
CRS-2673: Attempting to stop ora.asm on rac3
CRS-2673: Attempting to stop ora.mdnsd on rac3
CRS-2677: Stop of ora.evmd on rac3 succeeded
CRS-2677: Stop of ora.mdnsd on rac3 succeeded
CRS-2677: Stop of ora.ctssd on rac3 succeeded
CRS-2677: Stop of ora.asm on rac3 succeeded
CRS-2673: Attempting to stop ora.cluster_interconnect.haip on rac3
CRS-2677: Stop of ora.cluster_interconnect.haip on rac3 succeeded
CRS-2673: Attempting to stop ora.cssd on rac3
CRS-2677: Stop of ora.cssd on rac3 succeeded
CRS-2673: Attempting to stop ora.crf on rac3
CRS-2677: Stop of ora.crf on rac3 succeeded
CRS-2673: Attempting to stop ora.gipcd on rac3
CRS-2677: Stop of ora.gipcd on rac3 succeeded
CRS-2673: Attempting to stop ora.gpnpd on rac3
CRS-2677: Stop of ora.gpnpd on rac3 succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on rac3 has completed
CRS-4133: Oracle High Availability Services has been stopped.

启动

[root@rac1 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac2 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
– 还是有问题
[root@rac3 ~]# /u01/app/11.2.3/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[grid@rac1 ~]$ crs_start -all
[grid@rac1 ~]$ crs_stat -t
Name Type Target State Host
————————————————————
ora.CRS.dg ora….up.type ONLINE ONLINE rac1
ora.DATA.dg ora….up.type ONLINE ONLINE rac1
ora….ER.lsnr ora….er.type ONLINE ONLINE rac1
ora….N1.lsnr ora….er.type ONLINE ONLINE rac3
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type ONLINE ONLINE rac3
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora.kingsql.db ora….se.type ONLINE ONLINE rac3
ora….network ora….rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type ONLINE ONLINE rac3
ora.ons ora.ons.type ONLINE ONLINE rac1
ora….SM1.asm application ONLINE ONLINE rac1
ora….C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora….t1.type ONLINE ONLINE rac1
ora.rac2.vip ora….t1.type ONLINE ONLINE rac1
ora….SM3.asm application ONLINE ONLINE rac3
ora….C3.lsnr application ONLINE ONLINE rac3
ora.rac3.gsd application OFFLINE OFFLINE
ora.rac3.ons application ONLINE ONLINE rac3
ora.rac3.vip ora….t1.type ONLINE ONLINE rac3
ora.scan1.vip ora….ip.type ONLINE ONLINE rac3

重启 rac2 操作系统

[root@rac2 ~]# reboot
[root@rac2 ~]#
Broadcast message from root@rac2
(/dev/pts/0) at 17:40 …

The system is going down for reboot NOW!
[grid@rac1 ~]$ crs_stat -t
Name Type Target State Host
————————————————————
ora.CRS.dg ora….up.type ONLINE ONLINE rac1
ora.DATA.dg ora….up.type ONLINE ONLINE rac1
ora….ER.lsnr ora….er.type ONLINE ONLINE rac1
ora….N1.lsnr ora….er.type ONLINE ONLINE rac3
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type ONLINE ONLINE rac3
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora.kingsql.db ora….se.type ONLINE ONLINE rac3
ora….network ora….rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type ONLINE ONLINE rac3
ora.ons ora.ons.type ONLINE ONLINE rac1
ora….SM1.asm application ONLINE ONLINE rac1
ora….C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora….t1.type ONLINE ONLINE rac1
ora….SM2.asm application ONLINE ONLINE rac2
ora….C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora….t1.type ONLINE ONLINE rac2
ora….SM3.asm application ONLINE ONLINE rac3
ora….C3.lsnr application ONLINE ONLINE rac3
ora.rac3.gsd application OFFLINE OFFLINE
ora.rac3.ons application ONLINE ONLINE rac3
ora.rac3.vip ora….t1.type ONLINE ONLINE rac3
ora.scan1.vip ora….ip.type ONLINE ONLINE rac3
[grid@rac1 ~]$ crsctl status res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.CRS.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ONLINE ONLINE rac3
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ONLINE ONLINE rac3
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ONLINE ONLINE rac3
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ONLINE ONLINE rac3 Started
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
OFFLINE OFFLINE rac3
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ONLINE ONLINE rac3
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ONLINE ONLINE rac3
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac3
ora.cvu
1 ONLINE ONLINE rac3
ora.kingsql.db
1 ONLINE ONLINE rac3 Open
2 ONLINE ONLINE rac2 Open
3 ONLINE ONLINE rac1 Open
ora.oc4j
1 ONLINE ONLINE rac3
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.rac3.vip
1 ONLINE ONLINE rac3
ora.scan1.vip
1 ONLINE ONLINE rac3