Oracle 11g ORA-600 [kjbrcrcvt:lms] 问题处理

时间:2022-10-17 17:07:50

环境描述: Redhat Enterprise 7.9 Oracle 11.2.0.4 RAC 3节点

问题描述:数据库节点3出现hang,等待事件为ges cgs registration。

问题原因:触发Bug 30562200 - RAC instance crash due to ORA-600 [kjbrcrcvt:lms] ,RAC实例的LMS由于ORA-600 [kjbrcrcvt:lms]而出现故障

Oracle 11g ORA-600 [kjbrcrcvt:lms] 问题处理

解决方案:应用补丁Bug 31321944 - Database Hang with 'gc current request'<='gc buffer busy acquire' 

分析过程:

1、数据库节点3日志:

 alertnode3 log: 
Errors in file /u01/app/oracle/diag/rdbms/prod/prod3/trace/prod3_lmse_48530_i169417.trc:
ORA-00600: internal error code, arguments: [kjbrcrcvt:lms], [], [], [], [],[], [], [], [], [], [], []
...
*** 2022-10-10 14:07:06.629
*** SESSION ID:(649.1) 2022-10-10 14:07:06.629
*** CLIENT ID:() 2022-10-10 14:07:06.629
*** SERVICE NAME:(SYS$BACKGROUND) 2022-10-10 14:07:06.629
*** MODULE NAME:() 2022-10-10 14:07:06.629
*** ACTION NAME:() 2022-10-10 14:07:06.629
 
Dump continued from file: /u01/app/oracle/diag/rdbms/prodbprodb3/trace/prodb3_lmse_48530.trc
ORA-00600: internal error code, arguments: [kjbrcrcvt:lms], [], [], [], [], [], [], [], [], [], [], []
========= Dump for incident 169417 (ORA 600 [kjbrcrcvt:lms]) ========
----- Beginning of Customized Incident Dump(s) -----
 GCS RESOURCE 0xd889af92f0 hashq [0xd9f14cf3b0,0xd88f4287c0] name[0x115fe8.3f1] pkey 224175.0
   grant 0xd590794580 cvt 0xd4c9dbc160 send 0xd590794580@0,148 write (nil),0@65536
   *** 2022-10-10 14:07:06.645
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- SQL Statement (None) -----
Current SQL information unavailable - no cursor.
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+41        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFCDB61CCA0 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
ksedst1()+103        call     skdstdst()           000000000 ? 000000000 ?
                                                   7FFCDB61CCA0 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
ksedst()+39          call     ksedst1()            000000000 ? 000000001 ?
                                                   7FFCDB61CCA0 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
dbkedDefDump()+2746  call     ksedst()             000000000 ? 000000001 ?
                                                   7FFCDB61CCA0 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
ksedmp()+41          call     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFCDB61CCA0 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
ksfdmp()+69          call     ksedmp()             000000003 ? 000000002 ?
                                                   7FFCDB61CCA0 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
dbgexPhaseII()+1764  call     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFCDB61CCA0 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
dbgexExplicitEndInc  call     dbgexPhaseII()       7F854B219748 ? 7F8547E37A98 ?
()+755                                             7FFCDB625AB8 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
dbgeEndDDEInvocatio  call     dbgexExplicitEndInc  7F854B219748 ? 7F8547E37A98 ?
nImpl()+769                   ()                   7FFCDB625AB8 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
dbgeEndDDEInvocatio  call     dbgeEndDDEInvocatio  7F854B219748 ? 7F8547E37A98 ?
n()+52                        nImpl()              7FFCDB625AB8 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
kjbrrefping()+4118   call     dbgeEndDDEInvocatio  7F854B219748 ? 7F8547E37A98 ?
                              n()                  7FFCDB625AB8 ? 7FFCDB61CD78 ?
                                                   7FFCDB621820 ? 000000002 ?
kjbmprefuse()+2177   call     kjbrrefping()        D889AF92F0 ? D4D7FF9948 ?
                                                   7FFCDB625AB8 ?
                                                   FFFFFFFF89AF9340 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
kjmxmpm()+1014       call     kjbmprefuse()        7F8543C141F8 ? D4D7FF9948 ?
                                                   7FFCDB625AB8 ?
                                                   FFFFFFFF89AF9340 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
kjmpbmsg()+4706      call     kjmxmpm()            7F8543C141F8 ? D4D7FF9948 ?
                                                   7FFCDB625AB8 ?
                                                   FFFFFFFF89AF9340 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
kjmsm()+8021         call     kjmpbmsg()           7F8543C141F8 ? D4D7FF9948 ?
                                                   7FFCDB625AB8 ?
                                                   FFFFFFFF89AF9340 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
ksbrdp()+1045        call     kjmsm()              7F8543C141F8 ? D4D7FF9948 ?
                                                   7FFCDB625AB8 ?
                                                   FFFFFFFF89AF9340 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
opirip()+623         call     ksbrdp()             7F8543C141F8 ? D4D7FF9948 ?
                                                   7FFCDB625AB8 ?
                                                   FFFFFFFF89AF9340 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
opidrv()+603         call     opirip()             000000032 ? 000000004 ?
                                                   7FFCDB628448 ?
                                                   FFFFFFFF89AF9340 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
sou2o()+103          call     opidrv()             000000032 ? 000000004 ?
                                                   7FFCDB628448 ?
                                                   FFFFFFFF89AF9340 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
opimai_real()+250    call     sou2o()              7FFCDB628420 ? 000000032 ?
                                                   000000004 ? 7FFCDB628448 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
ssthrdmain()+265     call     opimai_real()        000000000 ? 7FFCDB628610 ?
                                                   000000004 ? 7FFCDB628448 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
main()+201           call     ssthrdmain()         000000003 ? 7FFCDB628610 ?
                                                   000000001 ? 000000000 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
__libc_start_main()  call     main()               000000003 ? 7FFCDB6287B0 ?
+245                                               000000001 ? 000000000 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
_start()+41          call     __libc_start_main()  000A177F0 ? 000000001 ?
                                                   7FFCDB6287A8 ? 000000000 ?
                                                   E5E37600010301 ?
                                                   1000800000000 ?
 
--------------------- Binary Stack Dump ---------------------
参考文档:Bug 31321944 - Database Hang with 'gc current request'<='gc buffer busy acquire' (Doc ID 31321944.8)
          Bug 30562200 - RAC instance crash due to ORA-600 [kjbrcrcvt:lms] -- Fix has regressed and replaced with fix in 31321944.8 (Doc ID 30562200.8)