DMESG查看日志，发现设备问题。
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)device-mapper: table: 253:27: multipath: error getting devicedevice-mapper: ioctl: error adding target to tabledevice-mapper: table: 253:27: multipath: error getting devicedevice-mapper: ioctl: error adding target to table
查看db日志，07 15:50:36 2013 出现了IO问题
Checkpoint not complete
Current log# 3 seq# 847 mem# 0: +CORE_REDO01/coredb/onlinelog/group_3.261.812494827
Current log# 3 seq# 847 mem# 1: +CORE_REDO02/coredb/onlinelog/group_3.261.812494831
Thread 2 advanced to log sequence 848 (LGWR switch)
Current log# 4 seq# 848 mem# 0: +CORE_REDO01/coredb/onlinelog/group_4.262.812494835
Current log# 4 seq# 848 mem# 1: +CORE_REDO02/coredb/onlinelog/group_4.262.812494837
Fri Jun 07 15:50:36 2013
Errors in file /oracle/app/oracle/diag/rdbms/coredb/coredb2/trace/coredb2_ora_46624.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 262144
WARNING: Write Failed. group:7 disk:0 AU:126728 offset:262144 size:262144
Errors in file /oracle/app/oracle/diag/rdbms/coredb/coredb2/trace/coredb2_ora_46624.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 7490 logical extent 0 of file 296 in group 7 on disk 0 allocation unit 126728
Fri Jun 07 15:50:40 2013
Thread 2 cannot allocate new log, sequence 849
Checkpoint not complete
Current log# 4 seq# 848 mem# 0: +CORE_REDO01/coredb/onlinelog/group_4.262.812494835
Current log# 4 seq# 848 mem# 1: +CORE_REDO02/coredb/onlinelog/group_4.262.812494837
Fri Jun 07 16:00:37 2013
WARNING:io_getevents timed out 600 sec
DD测试存储
dd if=/dev/asm_core_data01 of=/dev/null bs=1024k conut=1hang。。。。。。。。。。。。
联系存储厂商解决：
原因：存储快照分给的*空间太小。写满走后导致LUN OFFLINE
解决：扩展存储LUN 。