rac数据库的服务无法启动，报错ORA-03113: end-of-file on communication channel

启动数据库服务出现如下错误：

[grid@rac2 ~]$ srvctl start database -d orcl
PRCR-1079 : 无法启动资源 ora.orcl.db
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 21123
Session ID: 63 Serial number: 3
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.orcl.db' on 'rac1' failed
CRS-2632: There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-03113: end-of-file on communication channel
Process ID: 19773
Session ID: 63 Serial number: 3
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.orcl.db' on 'rac2' failed

查看日志文件如下

2015-05-21 10:11:22.245: [ USRTHRD][1073740096]{1:59151:1636} Thread:DedicatedThreadstop {
2015-05-21 10:11:22.245: [ USRTHRD][1073740096]{1:59151:1636} Thread:DedicatedThreadstop }
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] DbAgent:getOracleSid getResAttribute GEN_USR_ORA_INST_NAME = orcl1
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] DbAgent:getOracleSid 23 oracle_sid = orcl1
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] DbAgent:getOracleSid oracle_sid = orcl1
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] CssData Constructor groupName:DBORCL cssRegistrationType:0 memno:-1
2015-05-21 10:11:22.246: [ USRTHRD][1073740096]{1:59151:1636} CssData::initMutex &m_cssDataLock:38036618
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] CssGroup::regis registration of groupName:DBORCL mbrid:-1 m_grpNum:-1 regType:0 succeeded
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] CssData::Constructor regis succeeded groupName:DBORCL memno:-1
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] CssData::Constructor m_pCssGroup:3800c6c0 group DBORCL
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::init m_groupName:DBORCL m_pCssData 38035550
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] getOracleHomeAttrib: oracle_home = /u01/app/oracle/product/11.2.0/db_1
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::ConnectionPool 2 m_oracleHome:/u01/app/oracle/product/11.2.0/db_1, m_oracleSid:orcl1, m_usrOraEnv:
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] clean {
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop_option stop mode immediate option 1
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop {
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] getResAttrib: attrib name USR_ORA_OPI value true len 4
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] Agent::flagUsrOraOpiIsSet(true) reason not dependency
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop original reason system do shutdown abort
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] Gimh::check OH /u01/app/oracle/product/11.2.0/db_1 SID orcl1
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] (:CLSN00007:)DbAgent::check failed gimh state 0
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] clsnDbAgent:checkCbk clsagfw_res_status ret 5
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::check db/asm 2clsagfw_res_status 5 poolState 0
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] (:CLSN00007:)InstAgent::check: return unplanned offline
2015-05-21 10:11:22.246: [ USRTHRD][1073740096]{1:59151:1636} Gimh::destructor gimh_dest_query_ctx rc=0
2015-05-21 10:11:22.246: [ USRTHRD][1073740096]{1:59151:1636} Gimh::destructor gimh_dest_inst_ctx rc=0
2015-05-21 10:11:22.246: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::stopConnection
2015-05-21 10:11:22.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection connection count 0
2015-05-21 10:11:22.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection freed 0
2015-05-21 10:11:22.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::stopConnection sid orcl1 status 1
2015-05-21 10:11:22.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::check 1 prev clsagfw_res_status 32513 current clsagfw_res_status 1
2015-05-21 10:11:27.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::check prev clsagfw_res_status 1 current clsagfw_res_status 1
2015-05-21 10:11:27.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop shutdown mode: 4
2015-05-21 10:11:27.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] DbAgent::preStopCbk {
2015-05-21 10:11:27.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] DbAgentEventModule::stop Forcing stop because of clean
2015-05-21 10:11:27.247: [ USRTHRD][1073740096]{1:59151:1636} Thread:DedicatedThreadstop {
2015-05-21 10:11:27.247: [ USRTHRD][1073740096]{1:59151:1636} Thread:DedicatedThreadstop }
2015-05-21 10:11:27.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] DbAgent::preStopCbk }
2015-05-21 10:11:27.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::getConnection 160 ERROR sid orcl1 pool status 1
2015-05-21 10:11:27.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop pool pConnxn 00000000
2015-05-21 10:11:27.247: [ USRTHRD][1073740096]{1:59151:1636} InstConnection::initMutex AttachLock 00ae3210 DetachLock 00ae3228
2015-05-21 10:11:27.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] clsnInstConnection::makeConnectStr UsrOraEnv m_oracleHome /u01/app/oracle/product/11.2.0/db_1 Crshome /u01/app/11.2.0/grid
2015-05-21 10:11:27.247: [ora.orcl.db][1073740096]{1:59151:1636} [clean] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/oracle/product/11.2.0/db_1/bin/oracle)(ARGV0=oracleorcl1)(ENVS='ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1,ORACLE_SID=orcl1,LD_LIBRARY_PATH=')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=orcl1)))
2015-05-21 10:11:27.249: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop non pool pConnxn 1 38015a70
2015-05-21 10:11:27.249: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstConnection::connectInt: server not attached
2015-05-21 10:11:27.259: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0

2015-05-21 10:11:27.259: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstConnection::connectInt (2) Exception OCIException
2015-05-21 10:11:27.259: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstConnection:connect:excp OCIException OCI error 1034
2015-05-21 10:11:27.259: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop: connect1 errcode 1034
2015-05-21 10:11:27.259: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop: connect2 oracleHome /u01/app/oracle/product/11.2.0/db_1 oracleSid orcl1
2015-05-21 10:11:27.259: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstConnection::connectInt: server not attached
2015-05-21 10:11:27.269: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstConnection:connectInt connected
2015-05-21 10:11:27.269: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstConnection::shutdown mode 4
2015-05-21 10:11:27.269: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection connection count 1
2015-05-21 10:11:27.269: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection sid orcl1, InstConnection 38015a70
2015-05-21 10:11:27.269: [ USRTHRD][1073740096]{1:59151:1636} InstConnection::breakCall pConnxn:38015a70 DetachLock:00ae3228 m_pSvcH:380b2570
2015-05-21 10:11:27.269: [ USRTHRD][1073740096]{1:59151:1636} InstConnection:~InstConnection: this 38015a70
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection delete InstConnection 38015a70
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection freed 1
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::stopConnection
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection connection count 0
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection freed 0
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::stopConnection sid orcl1 status 1
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop db/asm
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::stopConnection
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection connection count 0
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::removeConnection freed 0
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] ConnectionPool::stopConnection sid orcl1 status 1
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] getResAttrib: attrib name USR_ORA_OPI value true len 4
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] Agent::flagUsrOraOpiIsSet(true) reason not dependency
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] InstAgent::stop: }
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] clean }
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [clean] (:CLSN00106:) clsn_agent::clean }
2015-05-21 10:11:27.270: [    AGFW][1073740096]{1:59151:1636} Command: clean for resource: ora.orcl.db 1 1 completed with status: SUCCESS
2015-05-21 10:11:27.270: [CLSFRAME][1161484016] TM [MultiThread] is changing desired thread # to 3. Current # is 2
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] ConnectionPool::resetConnection s_statusOfConnectionMap 00ae9760
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] ConnectionPool::resetConnection sid orcl1 status 2
2015-05-21 10:11:27.270: [    AGFW][1071638848]{1:59151:1636} Agent sending reply for: RESOURCE_CLEAN[ora.orcl.db 1 1] ID 4100:27163
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] Gimh::check OH /u01/app/oracle/product/11.2.0/db_1 SID orcl1
2015-05-21 10:11:27.270: [CLSFRAME][1161484016] TM [MultiThread] is changing desired thread # to 4. Current # is 3
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 0 exists
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] (:CLSN00007:)DbAgent::check failed gimh state 0
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] clsnDbAgent:checkCbk clsagfw_res_status ret 5
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] (:CLSN00007:)InstAgent::check: return unplanned offline
2015-05-21 10:11:27.270: [ USRTHRD][1073740096]{1:59151:1636} Gimh::destructor gimh_dest_query_ctx rc=0
2015-05-21 10:11:27.270: [ USRTHRD][1073740096]{1:59151:1636} Gimh::destructor gimh_dest_inst_ctx rc=0
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] ConnectionPool::stopConnection
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] ConnectionPool::removeConnection connection count 0
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] ConnectionPool::removeConnection freed 0
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] ConnectionPool::stopConnection sid orcl1 status 1
2015-05-21 10:11:27.270: [ora.orcl.db][1073740096]{1:59151:1636} [check] InstAgent::check 1 prev clsagfw_res_status 2 current clsagfw_res_status 1
2015-05-21 10:11:32.271: [ora.orcl.db][1073740096]{1:59151:1636} [check] InstAgent::check prev clsagfw_res_status 1 current clsagfw_res_status 1
2015-05-21 10:11:32.271: [    AGFW][1071638848]{1:59151:1636} ora.orcl.db 1 1 state changed from: CLEANING to: OFFLINE
2015-05-21 10:11:32.271: [    AGFW][1071638848]{1:59151:1636} Agent sending last reply for: RESOURCE_CLEAN[ora.orcl.db 1 1] ID 4100:27163
2015-05-21 10:11:32.271: [    AGFW][1071638848]{1:59151:1636} Agent has no resources to be monitored, Shutting down ..
2015-05-21 10:11:32.271: [    AGFW][1071638848]{1:59151:1636} Agent sending message to PE: AGENT_SHUTDOWN_REQUEST[Proxy] ID 20486:28
2015-05-21 10:11:32.273: [    AGFW][1071638848]{1:59151:1636} Agent is shutting down.
2015-05-21 10:11:32.273: [    AGFW][1071638848]{1:59151:1636} Agent is exiting with exit code: 1

没看懂日志是什么意思，有看懂的大神可以帮忙分析一下

后来从网上查找这个错误前辈们的解决方式，试了一个竟然成功了，下面是解决方法：

[grid@rac1 ~]$ su - oracle
口令：
[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Thu May 21 10:17:29 2015

Connected to an idle instance.

SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup mount
ORACLE instance started.

Total System Global Area 1636814848 bytes
Fixed Size      2253744 bytes
Variable Size   1023413328 bytes
Database Buffers   603979776 bytes
Redo Buffers      7168000 bytes
Database mounted.
SQL> select * from v$log;

    GROUP#    THREAD# SEQUENCE#      BYTES BLOCKSIZE   MEMBERS ARC
---------- ---------- ---------- ---------- ---------- ---------- ---
STATUS   FIRST_CHANGE# FIRST_TIME    NEXT_CHANGE#
---------------- ------------- ------------------- ------------
NEXT_TIME
-------------------
1     1       11   52428800    512  2 NO
CURRENT         1659086 2015-05-18 16:00:24   2.8147E+14

2 1 10 52428800 512 2 NO
INACTIVE 1623813 2015-05-18 12:08:34 1659086
2015-05-18 16:00:24

3 2 5 52428800 512 2 NO
CURRENT 1726288 2015-05-19 05:43:12 2.8147E+14

4 2 4 52428800 512 2 NO
INACTIVE 1610346 2015-05-18 11:00:44 1726288

GROUP# THREAD# SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC
---------- ---------- ---------- ---------- ---------- ---------- ---
STATUS FIRST_CHANGE# FIRST_TIME NEXT_CHANGE#
---------------- ------------- ------------------- ------------
NEXT_TIME
-------------------
2015-05-19 05:43:12

SQL*Plus: Release 11.2.0.4.0 Production on Thu May 21 10:20:43 2015

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
[oracle@rac1 ~]$ cd /u01/app/11.2.0/grid/bin/
[oracle@rac1 bin]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1
ora.DBFILE1.dg ora....up.type ONLINE    ONLINE    rac1
ora.FRA.dg     ora....up.type ONLINE    ONLINE    rac1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1
ora.cvu        ora.cvu.type   ONLINE    ONLINE    rac1
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    rac1
ora.oc4j       ora.oc4j.type ONLINE    ONLINE    rac1
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1
ora.orcl.db    ora....se.type ONLINE    ONLINE    rac1
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....C1.lsnr application    ONLINE    ONLINE    rac1
ora.rac1.gsd   application    OFFLINE   OFFLINE
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1
ora....SM2.asm application    ONLINE    ONLINE    rac2
ora....C2.lsnr application    ONLINE    ONLINE    rac2
ora.rac2.gsd   application    OFFLINE   OFFLINE
ora.rac2.ons   application    ONLINE    ONLINE    rac2
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2
ora....ry.acfs ora....fs.type ONLINE    ONLINE    rac1
ora.scan1.vip ora....ip.type ONLINE    ONLINE    rac1

到此服务已经启动成功了，虽然服务成功启动了但是原理却不是很理解，有能明白的大神希望能帮忙分析下

下面是我从网上看到的解决方法，我自己运行的时候看到数据库已经mount状态了就没再往下执行

报错ORA-03113: end-of-file on communication channel （通信通道的文件结尾）

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455 SQL> startup ORACLE instance started. Total System GlobalArea 1252663296 bytes Fixed Size2227944 bytes Variable Size704643352 bytes DatabaseBuffers 536870912 bytes Redo Buffers 8921088 bytesDatabasemounted. ORA-03113: end-of-fileoncommunication channel Process ID: 8117 Session ID: 191 Serial number: 3网上分析，可能由于昨晚数据库强制关闭，导致文件状态可能不一致，因为正常关闭数据库会同步校验各文件，使得重新启动的时候文件时间点一致。解决方案如下启动DB SQL> conn / assysdba Connected toan idle instance. SQL> startup mountORACLE instance started.直接reset不成功 SQL> select* fromv$log; GROUP# THREAD#SEQUENCE# BYTES BLOCKSIZE MEMBERS ARC---------- ---------- ---------- ---------- ---------- ---------- ---STATUS FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME---------------- ------------- --------- ------------ --------- 1 1 112 52428800 512 1NOINACTIVE 3826382 16-JUL-13 3857158 17-JUL-13 3 1 114 52428800 512 1NOCURRENT3886899 17-JUL-13 2.8147E+14 2 1 113 52428800 512 1NOINACTIVE 3857158 17-JUL-13 3886899 17-JUL-13 SQL> alterdatabaseopenresetlogs 2; alterdatabaseopenresetlogs 2 *ERROR atline 1: ORA-02288: invalidOPENmode 恢复数据库 SQL> alterdatabaseopenresetlogs; alterdatabaseopenresetlogs * ERROR atline 1: ORA-01139: RESETLOGSoptiononlyvalid afteran incomplete databaserecovery SQL> recover databaseuntil time'2013-07-05'Media recovery complete.SQL> alterdatabaseopenresetlogs; Databasealtered. 成功。

秒客网

rac数据库的服务无法启动，报错ORA-03113: end-of-file on communication channel

相关文章