背景介绍
由于修改了备库监听为动态监听，主备通过静态监听无法通讯，导致主备出现归档日志GAP
之后，DG运行中，发现主库正常，备库挂了。启动备库
alter database open
*
ERROR at line 1:
ORA-10458: standby database requires recovery
ORA-01196: file 1 is inconsistent due to a failed media recovery session
ORA-01110: data file 1: '/home/ora11g/11g/oradata/xcky/system01.dbf'
告警日志
alter database open
AUDIT_TRAIL initialization parameter is changed to OS, as DB is NOT compatible for database opened with read-only access
Beginning standby crash recovery.
Serial Media Recovery started
Managed Standby Recovery starting Real Time Apply
Media Recovery of Online Log [Thread=1, Seq=255]
Recovery of Online Redo Log: Thread 1 Group 4 Seq 255 Reading mem 0
Mem# 0: /home/ora11g/11g/oradata/xcky/stdredo01.log
Thu Sep 28 01:15:51 2017
Standby crash recovery failed to bring standby database to a consistent
point because needed redo hasn't arrived yet.
MRP: Wait timeout: thread 1 sequence# 0
Standby crash recovery aborted due to error 16016.
Errors in file /home/ora11g/11g/diag/rdbms/prixcky/xcky/trace/xcky_ora_13704.trc:
ORA-16016: archived log for thread 1 sequence# 255 unavailable
Recovery interrupted!
Some recovered datafiles maybe left media fuzzy
Media recovery may continue but open resetlogs may fail
Completed standby crash recovery.
Errors in file /home/ora11g/11g/diag/rdbms/prixcky/xcky/trace/xcky_ora_13704.trc:
ORA-10458: standby database requires recovery
ORA-01196: file 1 is inconsistent due to a failed media recovery session
ORA-01110: data file 1: '/home/ora11g/11g/oradata/xcky/system01.dbf'
ORA-10458 signalled during: alter database open...
Thu Sep 28 01:18:15 2017
Errors in file /home/ora11g/11g/diag/rdbms/prixcky/xcky/trace/xcky_psp0_13419.trc:
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
经过查看v$archived_log视图，发现备库归档序列到255就断了，而主库归档序列号最新为281；依据告警日志
'Recovery of Online Redo Log: Thread 1 Group 4 Seq 255 Reading mem 0
Mem# 0: /home/ora11g/11g/oradata/xcky/stdredo01.log
Thu Sep 28 01:15:51 2017
Standby crash recovery failed to bring standby database to a consistent
point because needed redo hasn't arrived yet.
可以确定是由于归档未传输到备库，导致主备不一致。
解决方案
主备启动到mount状态
1、从255号归档开始，从主库将备库未接收的主库的所有归档拷贝到备库；
2、将拷贝来的归档日志注册到备库控制文件；
3、开启日志恢复。
4、重启备库到mount状态，开启备库；开启主库。
5、开启RTA'
计算归档日志文件绝对路径中，归档文件名称前字符串长度
select length('/home/ora11g/archlog/')+1 from dual;
select 'scp /home/ora11g/archlog/'||substr(name,22)||' '||sys_context('userenv','host')||':/home/ora11g/archlog/' from v$archived_log
where SEQUENCE#>254 and SEQUENCE#<=281 and thread#=1 and NAME<>'standby service name';
主库查找所有备库未接收的归档日志，拼归档日志的注册SQL
sys@stdxcky>select 'ALTER DATABASE REGISTER PHYSICAL LOGFILE ''/home/ora11g/archlog/'||substr(name,22)||''';' from v$archived_log where SEQUENCE#>254 and SEQUENCE#<=281 and thread#=1 and NAME<>'standby service name';
'ALTERDATABASEREGISTERPHYSICALLOGFILE''/HOME/ORA11G/ARCHLOG/'||SUBSTR(NAME,22)||''';'
--------------------------------------------------------------------------------------
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_255_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_256_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_257_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_258_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_259_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_260_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_261_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_262_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_263_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_264_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_265_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_266_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_267_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_268_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_269_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_270_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_271_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_272_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_273_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_274_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_275_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_276_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_277_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_278_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_279_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_280_951172571.dbf';
ALTER DATABASE REGISTER PHYSICAL LOGFILE '/home/ora11g/archlog/1_281_951172571.dbf';
执行以上SQL脚本注册归档日志
alter database recover managed standby database using current logfile disconnect from session;
查看备库应用归档的情况，直到281号归档applied即可视为归档应用成功。
sys@prixcky>select NAME,THREAD#,FIRST_TIME,sequence#,applied from v$archived_log
order by FIRST_TIME,THREAD#,sequence#,DEST_ID;
NAME THREAD# FIRST_TIME SEQUENCE# APPLIED
------------------------------ ---------- ------------------- ---------- ---------
/home/ora11g/archlog/1_255_951 1 2017-09-22 22:27:58 255 YES
172571.dbf
/home/ora11g/archlog/1_256_951 1 2017-09-23 01:47:35 256 YES
172571.dbf
/home/ora11g/archlog/1_257_951 1 2017-09-23 06:00:14 257 YES
172571.dbf
/home/ora11g/archlog/1_258_951 1 2017-09-23 10:08:12 258 YES
172571.dbf
/home/ora11g/archlog/1_259_951 1 2017-09-23 18:08:15 259 YES
172571.dbf
/home/ora11g/archlog/1_260_951 1 2017-09-23 22:28:15 260 YES
172571.dbf
/home/ora11g/archlog/1_261_951 1 2017-09-23 22:28:18 261 YES
172571.dbf
/home/ora11g/archlog/1_262_951 1 2017-09-24 06:00:09 262 YES
172571.dbf
/home/ora11g/archlog/1_263_951 1 2017-09-24 07:00:54 263 YES
172571.dbf
/home/ora11g/archlog/1_264_951 1 2017-09-24 14:08:33 264 YES
172571.dbf
/home/ora11g/archlog/1_265_951 1 2017-09-24 22:00:39 265 YES
172571.dbf
/home/ora11g/archlog/1_266_951 1 2017-09-24 22:28:33 266 YES
172571.dbf
/home/ora11g/archlog/1_267_951 1 2017-09-25 03:13:33 267 YES
172571.dbf
/home/ora11g/archlog/1_268_951 1 2017-09-25 12:03:42 268 YES
172571.dbf
/home/ora11g/archlog/1_269_951 1 2017-09-25 21:00:09 269 YES
172571.dbf
/home/ora11g/archlog/1_270_951 1 2017-09-25 22:00:24 270 YES
172571.dbf
/home/ora11g/archlog/1_271_951 1 2017-09-25 22:28:50 271 YES
172571.dbf
/home/ora11g/archlog/1_272_951 1 2017-09-26 03:37:43 272 YES
172571.dbf
/home/ora11g/archlog/1_273_951 1 2017-09-26 12:21:40 273 YES
172571.dbf
/home/ora11g/archlog/1_274_951 1 2017-09-26 21:03:43 274 YES
172571.dbf
/home/ora11g/archlog/1_275_951 1 2017-09-26 22:00:19 275 YES
172571.dbf
/home/ora11g/archlog/1_276_951 1 2017-09-26 22:29:07 276 YES
172571.dbf
/home/ora11g/archlog/1_277_951 1 2017-09-27 03:00:16 277 YES
172571.dbf
/home/ora11g/archlog/1_278_951 1 2017-09-27 12:00:07 278 YES
172571.dbf
/home/ora11g/archlog/1_279_951 1 2017-09-27 22:37:49 279 YES
172571.dbf
/home/ora11g/archlog/1_280_951 1 2017-09-27 22:37:52 280 YES
172571.dbf
/home/ora11g/archlog/1_281_951 1 2017-09-28 00:40:10 281 YES
172571.dbf
重启备库到mount,开备库，开主库。
查看当前日志序列
主库
sys@stdxcky>select max(sequence#) from v$archived_log;
MAX(SEQUENCE#)
--------------
286
备库
sys@prixcky>select max(sequence#) from v$archived_log;
MAX(SEQUENCE#)
--------------
286
查看备库的恢复模式（为IDLE，未开启RTA）
sys@prixcky>select dest_id,recovery_mode from v$archive_dest_status where dest_id=1;
DEST_ID RECOVERY_MODE
---------- -----------------------
1 IDLE
开启RTA
sys@prixcky>alter database recover managed standby database using current logfile disconnect from session;
Database altered.
sys@prixcky>select dest_id,recovery_mode from v$archive_dest_status where dest_id=1;
DEST_ID RECOVERY_MODE
---------- -----------------------
1 MANAGED REAL TIME APPLY