环境描述: Redhat Enterprise 7.9 Oracle 11.2.0.4 RAC 3节点
问题描述:数据库节点3出现hang,等待事件为ges cgs registration。
问题原因:触发Bug 30562200 - RAC instance crash due to ORA-600 [kjbrcrcvt:lms] ,RAC实例的LMS由于ORA-600 [kjbrcrcvt:lms]而出现故障
解决方案:应用补丁Bug 31321944 - Database Hang with 'gc current request'<='gc buffer busy acquire'
分析过程:
1、数据库节点3日志:
alertnode3 log: Errors in file /u01/app/oracle/diag/rdbms/prod/prod3/trace/prod3_lmse_48530_i169417.trc: ORA-00600: internal error code, arguments: [kjbrcrcvt:lms], [], [], [], [],[], [], [], [], [], [], [] ... *** 2022-10-10 14:07:06.629 *** SESSION ID:(649.1) 2022-10-10 14:07:06.629 *** CLIENT ID:() 2022-10-10 14:07:06.629 *** SERVICE NAME:(SYS$BACKGROUND) 2022-10-10 14:07:06.629 *** MODULE NAME:() 2022-10-10 14:07:06.629 *** ACTION NAME:() 2022-10-10 14:07:06.629 Dump continued from file: /u01/app/oracle/diag/rdbms/prodbprodb3/trace/prodb3_lmse_48530.trc ORA-00600: internal error code, arguments: [kjbrcrcvt:lms], [], [], [], [], [], [], [], [], [], [], [] ========= Dump for incident 169417 (ORA 600 [kjbrcrcvt:lms]) ======== ----- Beginning of Customized Incident Dump(s) ----- GCS RESOURCE 0xd889af92f0 hashq [0xd9f14cf3b0,0xd88f4287c0] name[0x115fe8.3f1] pkey 224175.0 grant 0xd590794580 cvt 0xd4c9dbc160 send 0xd590794580@0,148 write (nil),0@65536 *** 2022-10-10 14:07:06.645 dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0) ----- SQL Statement (None) ----- Current SQL information unavailable - no cursor. ----- Call Stack Trace ----- calling call entry argument values in hex location type point (? means dubious value) -------------------- -------- -------------------- ---------------------------- skdstdst()+41 call kgdsdst() 000000000 ? 000000000 ? 7FFCDB61CCA0 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? ksedst1()+103 call skdstdst() 000000000 ? 000000000 ? 7FFCDB61CCA0 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? ksedst()+39 call ksedst1() 000000000 ? 000000001 ? 7FFCDB61CCA0 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? dbkedDefDump()+2746 call ksedst() 000000000 ? 000000001 ? 7FFCDB61CCA0 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? ksedmp()+41 call dbkedDefDump() 000000003 ? 000000002 ? 7FFCDB61CCA0 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? ksfdmp()+69 call ksedmp() 000000003 ? 000000002 ? 7FFCDB61CCA0 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? dbgexPhaseII()+1764 call ksfdmp() 000000003 ? 000000002 ? 7FFCDB61CCA0 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? dbgexExplicitEndInc call dbgexPhaseII() 7F854B219748 ? 7F8547E37A98 ? ()+755 7FFCDB625AB8 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? dbgeEndDDEInvocatio call dbgexExplicitEndInc 7F854B219748 ? 7F8547E37A98 ? nImpl()+769 () 7FFCDB625AB8 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? dbgeEndDDEInvocatio call dbgeEndDDEInvocatio 7F854B219748 ? 7F8547E37A98 ? n()+52 nImpl() 7FFCDB625AB8 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? kjbrrefping()+4118 call dbgeEndDDEInvocatio 7F854B219748 ? 7F8547E37A98 ? n() 7FFCDB625AB8 ? 7FFCDB61CD78 ? 7FFCDB621820 ? 000000002 ? kjbmprefuse()+2177 call kjbrrefping() D889AF92F0 ? D4D7FF9948 ? 7FFCDB625AB8 ? FFFFFFFF89AF9340 ? E5E37600010301 ? 1000800000000 ? kjmxmpm()+1014 call kjbmprefuse() 7F8543C141F8 ? D4D7FF9948 ? 7FFCDB625AB8 ? FFFFFFFF89AF9340 ? E5E37600010301 ? 1000800000000 ? kjmpbmsg()+4706 call kjmxmpm() 7F8543C141F8 ? D4D7FF9948 ? 7FFCDB625AB8 ? FFFFFFFF89AF9340 ? E5E37600010301 ? 1000800000000 ? kjmsm()+8021 call kjmpbmsg() 7F8543C141F8 ? D4D7FF9948 ? 7FFCDB625AB8 ? FFFFFFFF89AF9340 ? E5E37600010301 ? 1000800000000 ? ksbrdp()+1045 call kjmsm() 7F8543C141F8 ? D4D7FF9948 ? 7FFCDB625AB8 ? FFFFFFFF89AF9340 ? E5E37600010301 ? 1000800000000 ? opirip()+623 call ksbrdp() 7F8543C141F8 ? D4D7FF9948 ? 7FFCDB625AB8 ? FFFFFFFF89AF9340 ? E5E37600010301 ? 1000800000000 ? opidrv()+603 call opirip() 000000032 ? 000000004 ? 7FFCDB628448 ? FFFFFFFF89AF9340 ? E5E37600010301 ? 1000800000000 ? sou2o()+103 call opidrv() 000000032 ? 000000004 ? 7FFCDB628448 ? FFFFFFFF89AF9340 ? E5E37600010301 ? 1000800000000 ? opimai_real()+250 call sou2o() 7FFCDB628420 ? 000000032 ? 000000004 ? 7FFCDB628448 ? E5E37600010301 ? 1000800000000 ? ssthrdmain()+265 call opimai_real() 000000000 ? 7FFCDB628610 ? 000000004 ? 7FFCDB628448 ? E5E37600010301 ? 1000800000000 ? main()+201 call ssthrdmain() 000000003 ? 7FFCDB628610 ? 000000001 ? 000000000 ? E5E37600010301 ? 1000800000000 ? __libc_start_main() call main() 000000003 ? 7FFCDB6287B0 ? +245 000000001 ? 000000000 ? E5E37600010301 ? 1000800000000 ? _start()+41 call __libc_start_main() 000A177F0 ? 000000001 ? 7FFCDB6287A8 ? 000000000 ? E5E37600010301 ? 1000800000000 ? --------------------- Binary Stack Dump --------------------- 参考文档:Bug 31321944 - Database Hang with 'gc current request'<='gc buffer busy acquire' (Doc ID 31321944.8) Bug 30562200 - RAC instance crash due to ORA-600 [kjbrcrcvt:lms] -- Fix has regressed and replaced with fix in 31321944.8 (Doc ID 30562200.8)