共计 10386 个字符，预计需要花费 26 分钟才能阅读完成。
这篇文章主要介绍了如何解决修改 /dev/shm 大小造成 Oracle 12c 集群启动故障，具有一定借鉴价值，感兴趣的朋友可以参考下，希望大家阅读完这篇文章之后大有收获，下面让丸趣 TV 小编带着大家一起了解一下。
由于维护人员修改 Oracle Linux 7 中的 /dev/shm 大小造成其大小小于 Oracle 实例的 MEMORY_TARGET 或者 SGA_TARGET 而导致集群不能启动 (CRS-4535,CRS-4000)
[grid@jtp1 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.检查 asm 磁盘的权限是否问题，发现磁盘权限正常
[root@jtp3 ~]# ls -lrt /dev/asm*
brw-rw----. 1 grid oinstall 8, 128 Apr 3 2018 /dev/asmdisk07
brw-rw----. 1 grid oinstall 8, 48 Apr 3 2018 /dev/asmdisk02
brw-rw----. 1 grid oinstall 8, 96 Apr 3 2018 /dev/asmdisk05
brw-rw----. 1 grid oinstall 8, 112 Apr 3 2018 /dev/asmdisk06
brw-rw----. 1 grid oinstall 8, 64 Apr 3 2018 /dev/asmdisk03
brw-rw----. 1 grid oinstall 8, 80 Apr 3 2018 /dev/asmdisk04
brw-rw----. 1 grid oinstall 8, 32 Apr 3 2018 /dev/asmdisk01重启 crs
[root@jtp1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on jtp1
CRS-2673: Attempting to stop ora.mdnsd on jtp1
CRS-2673: Attempting to stop ora.gpnpd on jtp1
CRS-2677: Stop of ora.mdnsd on jtp1 succeeded
CRS-2677: Stop of ora.gpnpd on jtp1 succeeded
CRS-2673: Attempting to stop ora.cluster_interconnect.haip on jtp1
CRS-2673: Attempting to stop ora.drivers.acfs on jtp1
CRS-2677: Stop of ora.drivers.acfs on jtp1 succeeded
CRS-2677: Stop of ora.cluster_interconnect.haip on jtp1 succeeded
CRS-2673: Attempting to stop ora.ctssd on jtp1
CRS-2673: Attempting to stop ora.evmd on jtp1
CRS-2677: Stop of ora.ctssd on jtp1 succeeded
CRS-2677: Stop of ora.evmd on jtp1 succeeded
CRS-2673: Attempting to stop ora.cssd on jtp1
CRS-2677: Stop of ora.cssd on jtp1 succeeded
CRS-2673: Attempting to stop ora.gipcd on jtp1
CRS-2673: Attempting to stop ora.driver.afd on jtp1
CRS-2677: Stop of ora.driver.afd on jtp1 succeeded
CRS-2677: Stop of ora.gipcd on jtp1 succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on jtp1 has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@jtp1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.查看 crs 的 alert.log 发现磁盘组不能加载
[root@jtp1 ~]# tail -f /u01/app/grid/diag/crs/jtp1/crs/trace/alert.log
2018-04-02 18:30:21.227 [OHASD(8143)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 8143
2018-04-02 18:30:21.230 [OHASD(8143)]CRS-0714: Oracle Clusterware Release 12.2.0.1.0.
2018-04-02 18:30:21.245 [OHASD(8143)]CRS-2112: The OLR service started on node jtp1.
2018-04-02 18:30:21.262 [OHASD(8143)]CRS-8017: location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2018-04-02 18:30:21.262 [OHASD(8143)]CRS-1301: Oracle High Availability Service started on node jtp1.
2018-04-02 18:30:21.567 [ORAROOTAGENT(8214)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 8214
2018-04-02 18:30:21.600 [CSSDAGENT(8231)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 8231
2018-04-02 18:30:21.607 [CSSDMONITOR(8241)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 8241
2018-04-02 18:30:21.620 [ORAAGENT(8225)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 8225
2018-04-02 18:30:22.146 [ORAAGENT(8316)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 8316
2018-04-02 18:30:22.211 [MDNSD(8335)]CRS-8500: Oracle Clusterware MDNSD process is starting with operating system process ID 8335
2018-04-02 18:30:22.215 [EVMD(8337)]CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 8337
2018-04-02 18:30:23.259 [GPNPD(8369)]CRS-8500: Oracle Clusterware GPNPD process is starting with operating system process ID 8369
2018-04-02 18:30:24.275 [GPNPD(8369)]CRS-2328: GPNPD started on node jtp1.
2018-04-02 18:30:24.283 [GIPCD(8433)]CRS-8500: Oracle Clusterware GIPCD process is starting with operating system process ID 8433
2018-04-02 18:30:26.296 [CSSDMONITOR(8464)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 8464
2018-04-02 18:30:28.299 [CSSDAGENT(8482)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 8482
2018-04-02 18:30:28.496 [OCSSD(8497)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 8497
2018-04-02 18:30:29.538 [OCSSD(8497)]CRS-1713: CSSD daemon is started in hub mode
2018-04-02 18:30:36.015 [OCSSD(8497)]CRS-1707: Lease acquisition for node jtp1 number 1 completed
2018-04-02 18:30:37.087 [OCSSD(8497)]CRS-1605: CSSD voting file is online: AFD:CRS1; details in /u01/app/grid/diag/crs/jtp1/crs/trace/ocssd.trc.
2018-04-02 18:30:37.103 [OCSSD(8497)]CRS-1672: The number of voting files currently available 1 has fallen to the minimum number of voting files required 1.
2018-04-02 18:30:46.237 [OCSSD(8497)]CRS-1601: CSSD Reconfiguration complete. Active nodes are jtp1 .
2018-04-02 18:30:48.514 [OCTSSD(9302)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 9302
2018-04-02 18:30:48.535 [OCSSD(8497)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
2018-04-02 18:30:50.626 [OCTSSD(9302)]CRS-2407: The new Cluster Time Synchronization Service reference node is host jtp1.
2018-04-02 18:30:50.627 [OCTSSD(9302)]CRS-2401: The Cluster Time Synchronization Service started on host jtp1.
2018-04-02 18:31:04.202 [ORAROOTAGENT(8214)]CRS-5019: All OCR locations are on ASM disk groups [CRS], and none of these disk groups are mounted. Details are at (:CLSN00140:) in /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc .
2018-04-02 18:41:00.225 [ORAROOTAGENT(8214)]CRS-5818: Aborted command start for resource ora.storage . Details at (:CRSAGF00113:) {0:9:3} in /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc.
2018-04-02 18:41:03.757 [ORAROOTAGENT(8214)]CRS-5017: The resource action ora.storage start encountered the following error:
2018-04-02 18:41:03.757+Storage agent start action aborted. For details refer to (:CLSN00107:) in /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc .
2018-04-02 18:41:03.760 [OHASD(8143)]CRS-2757: Command Start timed out waiting for response from the resource ora.storage . Details at (:CRSPE00221:) {0:9:3} in /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd.trc.
2018-04-02 18:42:09.921 [ORAROOTAGENT(8214)]CRS-5019: All OCR locations are on ASM disk groups [CRS], and none of these disk groups are mounted. Details are at (:CLSN00140:) in /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc .检查跟踪文件，发现查询 ASM_DISCOVERY_ADDRESS 与 ASM_DISCOVERY_ADDRESS 属性时出现
[root@jtp1 ~]# more /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc
Trace file /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc
Oracle Database 12c Clusterware Release 12.2.0.1.0 - Production Copyright 1996, 2016 Oracle. All rights reserved.
*** TRACE CONTINUED FROM FILE /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root_93.trc ***
2018-04-02 18:42:09.165 : CSSCLNT:3554666240: clsssterm: terminating context (0x7f03c0229390)
2018-04-02 18:42:09.165 : default:3554666240: clsCredDomClose: Credctx deleted 0x7f03c0459470
2018-04-02 18:42:09.166 : GPNP:3554666240: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode= remote
2018-04-02 18:42:09.253 : CSSCLNT:3554666240: clsssinit: initialized context: (0x7f03c045c2c0) flags 0x115
2018-04-02 18:42:09.253 : CSSCLNT:3554666240: clsssterm: terminating context (0x7f03c045c2c0)
2018-04-02 18:42:09.254 : CLSNS:3554666240: clsns_SetTraceLevel:trace level set to 1.
2018-04-02 18:42:09.254 : GPNP:3554666240: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode= remote
2018-04-02 18:42:09.257 : default:3554666240: Inited LSF context: 0x7f03c04f0420
2018-04-02 18:42:09.260 : CLSCRED:3554666240: clsCredCommonInit: Inited singleton credctx.
2018-04-02 18:42:09.260 : CLSCRED:3554666240: (:CLSCRED0101:)clsCredDomInitRootDom: Using user given storage context for repository access.
2018-04-02 18:42:09.294 : USRTHRD:3554666240: {0:9:3} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2018-04-02 18:42:09.300 : USRTHRD:3554666240: {0:9:3} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2018-04-02 18:42:09.356 : CLSCRED:3554666240: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.5c82286a084bcf37ffa014144074e5dd.root not found
2018-04-02 18:42:09.356 : USRTHRD:3554666240: {0:9:3} 7755 Error 4 opening dom root in 0x7f03c064c980检查 ASM 的 alert.log 发现 /dev/shm 大小小于 MEMORY_TARGET 大小，并且给出了 /dev/shm 应该被设置的最小值
[root@jtp1 ~]# tail -f /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
WARNING: ASM does not support ipclw. Switching to skgxp
WARNING: ASM does not support ipclw. Switching to skgxp
WARNING: ASM does not support ipclw. Switching to skgxp
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal) (OS id: 9343)
2018-04-02T18:31:00.187055+08:00
CLI notifier numLatches:7 maxDescs:2301
2018-04-02T18:31:00.193961+08:00
WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to be mounted for at least 1140850688 bytes. /dev/shm is either not mounted or is mounted with available space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is 1073573888 and used is 167936 bytes. Ensure that the mount point is /dev/shm for this directory.修改 /dev/shm 的大小可以通过修改 /etc/fstab 来实现，将 /dev/shm 的大小修改为 12G
[root@jtp1 bin]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ol-root 49G 42G 7.9G 85% /
devtmpfs 12G 28K 12G 1% /dev
tmpfs 1.0G 164K 1.0G 1% /dev/shm
tmpfs 1.0G 9.3M 1015M 1% /run
tmpfs 1.0G 0 1.0G 0% /sys/fs/cgroup
/dev/sda1 1014M 141M 874M 14% /boot
[root@jtp1 bin]# vi /etc/fstab
# /etc/fstab
# Created by anaconda on Sat Mar 18 15:27:13 2017
# Accessible filesystems, by reference, are maintained under /dev/disk
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
/dev/mapper/ol-root / xfs defaults 0 0
UUID=ca5854cd-0125-4954-a5c4-1ac42c9a0f70 /boot xfs defaults 0 0
/dev/mapper/ol-swap swap swap defaults 0 0
正文完
