Wed Mar 9 19:03:38 2011以上可以看到lgwr关键进程在数据库open后几秒后遭遇了ORA-00600:[2667]内部错误后终止了实例。该数据库在之前因为丢失当前日志文件进行了已经实施了一系列的非常规恢复操作，包括设置一系列的underscore参数:
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 117699122479
Resetting resetlogs activation ID 2197911857 (0x83017931)
Wed Mar 9 19:03:47 2011
LGWR: Primary database is in CLUSTER CONSISTENT mode
Assigning activation ID 2284878888 (0x88307c28)
Thread 1 opened at log sequence 1
Current log# 3 seq# 1 mem# 0: /s01/maclean/oradata/PROD/redo03.log
Successful open of redo thread 1.
Wed Mar 9 19:03:47 2011
SMON: enabling cache recovery
Wed Mar 9 19:03:47 2011
Errors in file /s01/maclean/admin/PROD/udump/PROD_ora_585914.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
Wed Mar 9 19:03:48 2011
Errors in file /s01/maclean/admin/PROD/bdump/PROD_pmon_602286.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
Wed Mar 9 19:03:50 2011
Errors in file /s01/maclean/admin/PROD/bdump/PROD_lgwr_548884.trc:
ORA-00600: internal error code, arguments: [2667], [1], [1], [3], [1739273391], [1739273391], [1739273391], [7267]
LGWR: terminating instance due to error 600
相关的初始化参数
fast_start_mttr_target = 300
_allow_resetlogs_corruption= TRUE
undo_management = AUTO
undo_tablespace = UNDOTBS1
Before I provide the steps to fix the ora-00600 error, I want to tell you that this database is opened with the unsupported parameter但在完成以上设置后仍不能避免ora-00600[2667]的发生，下位errpt日志中磁盘阵列损坏信息:
"allow_resetlogs_corruption".
*************************************************************************
* By forcing open the database using this parameter, there is a strong *
* likelihood of logical corruption, possibly affecting the data *
* dictionary. Oracle does not guarantee that all of the data will be *
* accessible nor will it support a database that has been opened by *
* this method and that the database users will be allowed to continue *
* work. All this does is provide a way to get at the contents of the *
* database for extraction, usually by export. It is up to you to *
* determine the amount of lost data and to correct any logical *
* corruption issues. *
* *
*************************************************************************
2) The steps to get rid of the ora-00600 are as follows:
+ Change UNDO_MANAGEMENT=AUTO to
UNDO_MANAGEMENT=MANUAL
+ Remove or comment out UNDO_TABLESPACE and UNDO_RETENTION.
+ Add
_CORRUPTED_ROLLBACK_SEGMENTS =(comma separated list of Automatic Undo segments)
Example:
_CORRUPTED_ROLLBACK_SEGMENTS = (_SYSSMU1$, _SYSSMU2$, _SYSSMU3$, _SYSSMU4$,
_SYSSMU5$, _SYSSMU6$, _SYSSMU7$, _SYSSMU8$, _SYSSMU9$, _SYSSMU10$)
Note, sometimes the alert log will tell you what Automatic Undo segments are in use.
Search the alert log for SYSS. If the alert log does not contain that information then use _SYSSMU1$
through _SYSSMU10$ as shown in the example above.
In UNIX you can issue this command to get the undo segment names:
$ strings system01.dbf | grep _SYSSMU | cut -d $ -f 1 | sort -u
From the output of the strings command above, add a $ to end of each _SYSSMU undo segment name.
++ Startup mount the database as follows:
SQL > startup mount
SQL > recover database;
SQl > alter database open;
*._corrupted_rollback_segments= (_SYSSMU730$, _SYSSMU731$, _SYSSMU732$, _SYSSMU733$, _SYSSMU734$,
_SYSSMU735$, _SYSSMU736$, _SYSSMU737$, _SYSSMU738$, _SYSSMU739$, _SYSSMU744$, _SYSSMU740$, _SYSSMU741$,
_SYSSMU742$, _SYSSMU743$, _SYSSMU744$, _SYSSMU745$, _SYSSMU746$, _SYSSMU747$, _SYSSMU748$, _SYSSMU749$, _SYSSMU74t$,
_SYSSMU75$, _SYSSMU750$, _SYSSMU751$, _SYSSMU752$, _SYSSMU753$, _SYSSMU754$, _SYSSMU755$, _SYSSMU756$, _SYSSMU757$,
_SYSSMU758$, _SYSSMU759$, _SYSSMU76$, _SYSSMU760$, _SYSSMU761$, _SYSSMU762$, _SYSSMU763$, _SYSSMU764$, _SYSSMU765$,
_SYSSMU766$, _SYSSMU767$, _SYSSMU768$)
errpt -a因为lgwr的crash是发生在database open后，所以实际上我们是可以在这段时间内操作数据库的，这个case最终通过新建undo tablespace代替老的问题回滚表空间解决了。
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR6
IDENTIFIER: B9735AF4
Date/Time: Thu Mar 10 19:01:51 THAIST 2011
Sequence Number: 106800
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
SUBSYSTEM COMPONENT FAILURE
Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D544 0000 0000 0000 0000
0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 32C1 3033 3130 3131 2F30 3630 3533 3000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR6
IDENTIFIER: B9735AF4
Date/Time: Thu Mar 10 19:01:51 THAIST 2011
Sequence Number: 106799
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
SUBSYSTEM COMPONENT FAILURE
Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D524 0000 0000 0000 0000
0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 32C0 3033 3130 3131 2F30 3630 3533 3000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR10
IDENTIFIER: C86ACB7E
Date/Time: Thu Mar 10 19:01:51 THAIST 2011
Sequence Number: 106798
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: INFO
Resource Name: dac0
Resource Class: array
Resource Type: ibm-dac-V4
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
Part Number.................24288-00
ROS Level and ID............0916
Description
ARRAY CONFIGURATION CHANGED
Probable Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS
Failure Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS
Recommended Actions
NO ACTION NECESSARY
Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 9502 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0008 1800 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 32BF 3033 3130 3131 2F30 3630 3533 3000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR6
IDENTIFIER: B9735AF4
Date/Time: Thu Mar 10 19:01:47 THAIST 2011
Sequence Number: 106797
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: hdisk4
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L0
Description
SUBSYSTEM COMPONENT FAILURE
Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D544 0000 0000 0000 0000
0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 3215 3033 3130 3131 2F30 3630 3532 3600 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR6
IDENTIFIER: B9735AF4
Date/Time: Thu Mar 10 19:01:47 THAIST 2011
Sequence Number: 106796
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: hdisk4
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L0
Description
SUBSYSTEM COMPONENT FAILURE
Probable Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
Failure Causes
ARRAY DASD MEDIA
POWER OR FAN COMPONENT
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 3FC6 0600 0000 0000 0000 0000 0000 D524 0000 0000 0000 0000
0008 5000 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 3214 3033 3130 3131 2F30 3630 3532 3600 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR10
IDENTIFIER: C86ACB7E
Date/Time: Thu Mar 10 19:01:47 THAIST 2011
Sequence Number: 106795
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: INFO
Resource Name: dac0
Resource Class: array
Resource Type: ibm-dac-V4
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
Part Number.................24288-00
ROS Level and ID............0916
Description
ARRAY CONFIGURATION CHANGED
Probable Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS
Failure Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS
Recommended Actions
NO ACTION NECESSARY
Detail Data
SENSE DATA
0600 0308 0000 FF00 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 7000 0600
0000 0098 0000 0000 9502 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0008 1800 0000 0000 0000 0000 0000 0000 0000 5347 3830 3730 3033 3339 2020 2020
2020 0660 2200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0005 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5B8C 3213 3033 3130 3131 2F30 3630 3532 3600 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106794
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2800 4644 2F00 0003 0004 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106793
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2A00 132E C948 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106792
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2A00 1335 55F0 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106791
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2A00 136C 0F98 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106790
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2A00 13D7 29B0 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106789
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2A00 1337 FD88 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106788
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2A00 136C 1F08 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106787
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2A00 2578 E838 0000 0804 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106786
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2800 4644 3300 0001 0004 0000 0000 0000 0000 0007 7497 0200 0400 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR4
IDENTIFIER: D5385D18
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106785
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: hdisk5
Resource Class: disk
Resource Type: array
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21-L1000000000000
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD MEDIA
ARRAY DASD DEVICE
Failure Causes
DASD MEDIA
DISK DRIVE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2800 4643 BA00 0004 0004 0000 0000 0000 0000 0007 7497 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 5262 E000 F205 3703 0000 0200 0000 0000 0000 0000 0000 0002 0000 0010
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR8
IDENTIFIER: 483C9D10
Date/Time: Thu Mar 10 18:49:23 THAIST 2011
Sequence Number: 106784
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: INFO
Resource Name: dac0
Resource Class: array
Resource Type: ibm-dac-V4
Location: U787B.001.DNWGK9Y-P1-C4-T1-W200500A0B8484C21
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
Part Number.................24288-00
ROS Level and ID............0916
Description
ARRAY ACTIVE CONTROLLER SWITCH
Probable Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS
Failure Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS
Recommended Actions
NO ACTION NECESSARY
Detail Data
SENSE DATA
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0400 00EE 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0000 0000 0000 0000 0002 0000 0001
0000 0000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR3
IDENTIFIER: D9770360
Date/Time: Thu Mar 10 18:48:47 THAIST 2011
Sequence Number: 106783
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: dac0utm
Resource Class: NONE
Resource Type: NONE
Location:
Description
ARRAY OPERATION ERROR
Probable Causes
ARRAY DASD DEVICE
STORAGE DEVICE CABLE
EB4F
Failure Causes
DISK DRIVE
DISK DRIVE ELECTRONICS
STORAGE DEVICE CABLE
ARRAY CONTROLLER
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 2800 0000 0200 0000 0104 0000 0000 0000 0000 0000 0013 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 1103 8000 F205 3703 0000 0200 0000 0000 0000 0000 0040 0002 0000 0000
0000 0000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7
Date/Time: Thu Mar 10 18:48:47 THAIST 2011
Sequence Number: 106782
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi0
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C4-T1
Description
ADAPTER ERROR
Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB
Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION
Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0500 0000 0000
0001 0400 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 0812 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2005 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
1105 C000
---------------------------------------------------------------------------
LABEL: FCP_ARRAY_ERR9
IDENTIFIER: 8B79A4BD
Date/Time: Thu Mar 10 18:48:45 THAIST 2011
Sequence Number: 106781
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: PERM
Resource Name: dac1
Resource Class: array
Resource Type: ibm-dac-V4
Location: U787B.001.DNWGK9Y-P1-C1-T1-W200400A0B8484C21
VPD:
Manufacturer................IBM
Machine Type and Model......1814 FAStT
Part Number.................24288-00
ROS Level and ID............0916
Description
ARRAY CONTROLLER SWITCH FAILURE
Probable Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS
Failure Causes
ARRAY CONTROLLER
CABLES AND CONNECTIONS
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
SENSE DATA
0A00 5A00 2C01 0000 0002 0004 0000 0000 0000 0000 0000 0000 0200 0B00 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 526D 6000 F205 3703 0000 0000 0000 0010 0000 0000 0040 0002 0000 0000
0000 0000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7
Date/Time: Thu Mar 10 18:48:45 THAIST 2011
Sequence Number: 106780
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi1
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C1-T1
Description
ADAPTER ERROR
Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB
Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION
Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 1212 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0FFF A000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7
Date/Time: Thu Mar 10 18:48:43 THAIST 2011
Sequence Number: 106779
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi1
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C1-T1
Description
ADAPTER ERROR
Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB
Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION
Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 1112 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0FFF A000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7
Date/Time: Thu Mar 10 18:48:42 THAIST 2011
Sequence Number: 106778
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi0
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C4-T1
Description
ADAPTER ERROR
Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB
Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION
Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0500 0000 0000
0001 0400 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 0712 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2005 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
1105 C000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7
Date/Time: Thu Mar 10 18:48:41 THAIST 2011
Sequence Number: 106777
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi1
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C1-T1
Description
ADAPTER ERROR
Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB
Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION
Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 1012 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0FFF A000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7
Date/Time: Thu Mar 10 18:48:39 THAIST 2011
Sequence Number: 106776
Machine Id: 000786F9D600
Node Id: p550su
Class: H
Type: TEMP
Resource Name: fscsi1
Resource Class: driver
Resource Type: efscsi
Location: U787B.001.DNWGK9Y-P1-C1-T1
Description
ADAPTER ERROR
Probable Causes
ADAPTER HARDWARE OR CABLE
ADAPTER MICROCODE
FIBRE CHANNEL SWITCH OR FC-AL HUB
Failure Causes
ADAPTER
CABLES AND CONNECTIONS
DEVICE
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
CHECK CABLES AND THEIR CONNECTIONS
VERIFY DEVICE CONFIGURATION
Detail Data
SENSE DATA
0000 0000 0000 00B1 0000 0045 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0100 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 422F 0000 0F12 0002 0000 0100 0000 0000 0001 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0100 2004 00A0
B848 4C21 2004 00A0 B848 4C20 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0FFF A000
---------------------------------------------------------------------------
LABEL: FSCSI_ERR4
IDENTIFIER: 3074FEB7
We have been repair the disk controller, but ora-600[2667] still occur when we try to open database.
After recreating the undo tablespace we were able to open the database.
本文出自 “Ask Maclean Liu Oracle” 博客，请务必保留此出处http://maclean.blog.51cto.com/2923249/1277756