今天启动RAC 做实验，发现RAC 实例怎么启动不了。
[oracle@rac2 bin]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE OFFLINE
ora....oltp.cs application ONLINE OFFLINE
ora....cl1.srv application ONLINE OFFLINE
ora....cl2.srv application ONLINE OFFLINE
ora....l1.inst application ONLINE OFFLINE
ora....l2.inst application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
看了一下ASM 的磁盘组，都挂载成功。
[oracle@rac2 bin]$ export ORACLE_SID=+ASM2
[oracle@rac2 bin]$ sqlplus / as sysdba;
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 28 04:33:24 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
SQL> select name,state from v$asm_diskgroup;
NAME STATE
------------------------------ -----------
DATA MOUNTED
FLASH_RECOVERY_AREA MOUNTED
SQL>
之前遇到过这种情况情况，所以第一时间就查看了这个ASM 磁盘组的信息。
ASM 没有问题，只能查看数据库的alert log了。
[oracle@rac1 bdump]$ pwd
/u01/app/oracle/admin/orcl/bdump
[oracle@rac1 bdump]$ ls
alert_orcl1.log orcl1_diag_22153.trc orcl1_lms0_22169.trc
cdmp_20100917173112 orcl1_lgwr_22186.trc orcl1_smon_22190.trc
orcl1_arc0_24180.trc orcl1_lmd0_22167.trc
orcl1_arc1_24182.trc orcl1_lmon_22165.trc
[oracle@rac1 bdump]$ tail -100 alert_orcl1.log
Tue Sep 28 04:05:20 2010
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMON started with pid=15, OS id=5650
Tue Sep 28 04:05:20 2010
starting up 1 shared server(s) ...
Tue Sep 28 04:05:20 2010
lmon registered with NM - instance id 1 (internal mem no 0)
Tue Sep 28 04:05:21 2010
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Tue Sep 28 04:05:21 2010
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Tue Sep 28 04:05:21 2010
LMS 0: 0 GCS shadows traversed, 0 replayed
Tue Sep 28 04:05:21 2010
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=19, OS id=5683
Tue Sep 28 04:05:22 2010
ALTER DATABASE MOUNT
Tue Sep 28 04:05:22 2010
This instance was first to mount
Tue Sep 28 04:05:22 2010
Starting background process ASMB
ASMB started with pid=21, OS id=5706
Starting background process RBAL
RBAL started with pid=22, OS id=5710
Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
Tue Sep 28 04:05:26 2010
SUCCESS: diskgroup DATA was mounted
SUCCESS: diskgroup FLASH_RECOVERY_AREA was mounted
Tue Sep 28 04:05:34 2010
Setting recovery target incarnation to 3
Tue Sep 28 04:05:35 2010
Successful mount of redo thread 1, with mount id 1258660482
Tue Sep 28 04:05:35 2010
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Tue Sep 28 04:05:35 2010
ALTER DATABASE OPEN
This instance was first to open
ORA-1589 signalled during: ALTER DATABASE OPEN...
Tue Sep 28 04:05:39 2010
Shutting down instance (abort)
License high water mark = 1
Instance terminated by USER, pid = 6080
从日志里我们发现了一个很重要的信息：
ALTER DATABASE OPEN
This instance was first to open
ORA-1589 signalled during: ALTER DATABASE OPEN...
Tue Sep 28 04:05:39 2010
Shutting down instance (abort)
查看了一下错误代码：
ORA-01589: |
must use RESETLOGS or NORESETLOGS option for database open |
Cause: |
Either incomplete or backup control file recovery has been performed. After these types of recovery you must specify either the RESETLOGS option or the NORESETLOGS option to open your database. |
Action: |
Specify the appropriate option. |
提示我们必须使用resetlogs 模式打开数据。 开到这才明白，昨天做RMAN 恢复后直接把数据库关了，没有用resetlogs打开。 杯具中..
手工启动数据库至mount 状态，在用resetlogs 打开：
[oracle@rac1 bdump]$ export ORACLE_SID=orcl1
[oracle@rac1 bdump]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 28 04:17:47 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> conn / as sysdba;
Connected to an idle instance.
SQL> startup mount;
ORACLE instance started.
Total System Global Area 180355072 bytes
Fixed Size 1218388 bytes
Variable Size 109054124 bytes
Database Buffers 67108864 bytes
Redo Buffers 2973696 bytes
Database mounted.
SQL> alter database open resetlogs;
Database altered.
SQL>
在另一个节点，直接正常启动就可以了。
[oracle@rac2 bin]$ export ORACLE_SID=orcl2
[oracle@rac2 bin]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 28 04:22:50 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup mount;
ORACLE instance started.
Total System Global Area 180355072 bytes
Fixed Size 1218388 bytes
Variable Size 88082604 bytes
Database Buffers 88080384 bytes
Redo Buffers 2973696 bytes
Database mounted.
SQL> alter database open;
Database altered.
SQL>
查看一下RAC 进程的状态：
[oracle@rac2 bin]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac2
ora....oltp.cs application ONLINE ONLINE rac2
ora....cl1.srv application ONLINE ONLINE rac1
ora....cl2.srv application ONLINE ONLINE rac2
ora....l1.inst application ONLINE ONLINE rac1
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
一切正常，从这个小错误中明白，遇到错误，要养成看alert log 的习惯。 不管是单实例还是RAC 实例，看log 永远是第一条。 还有就是要心细，昨天偷了懒，为了补这个懒，花的时间就比昨天多了很多。 杯具啊.
------------------------------------------------------------------------------
Blog： http://blog.csdn.net/tianlesoftware
网上资源： http://tianlesoftware.download.csdn.net
相关视频：http://blog.csdn.net/tianlesoftware/archive/2009/11/27/4886500.aspx
DBA1 群：62697716(满); DBA2 群：62697977(满)
DBA3 群：63306533; 聊天 群：40132017
--加群需要在备注说明SGA的组成部分，否则拒绝申请