晚上,接到信息科值班电话,说一个生产库连不进去了。
于是v*n上去,用工具连也连不进去,报错:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925
ORA-02002: error while writing to audit trail
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925
09925. 00000 - "Unable to create audit trail file"
*Cause: ORACLE was not able to create the file being used to hold
audit trail records.
*Action: Check the UNIX error number for a possible operating system error.
If there is no error, contact ORACLE customer support.
供应商代码 9925
看到No space于是df -h发现磁盘未满呀。
sqlplus 也登不进去:
ERROR:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925
ORA-01075: you are currently logged on
打开日志文件查看:
刚发生的情况,可能和新上系统有关系。
Audit file写不进去,Audit file在哪里呢?找到路径发现目录下文件很多,显示不完。于是直接 rm -rf,报错:
-bash: /bin/rm: Argument list too long
网上搜索答案:https://www.cnblogs.com/tzhyy/p/10712339.html
Oracle11g <wbr>adump目录下面.aud增长导致空间撑满无法删除导致CRS无法启动的解决方法
输入下面的命令,删除成功:
ls | xargs -n 10 rm -fr ls
命令解释为:输出所有的文件名(用空格分割) xargs就是将ls的输出,每10个为一组(以空格为分隔符),
作为rm -rf的参数也就是说将所有文件名10个为一组,由rm -rf删除
分析原因:
发生上面错误的原因 有以下三种可能:
① 磁盘空间已满,导致无法创建审核文件,使用df -h查询磁盘空间;
② 磁盘inodes已经占用完了,导致无法打开新的文件,使用df -i查看inode使用情况;
③ 路径权限不足,无法创建审核文件,查看路径权限情况ls -l。
现在确定是 磁盘inodes已经占用完引起的。
# df -i 可查看inodes使用情况
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/rootvg-root 2441216 2943 2438273 1% /
......
/dev/mapper/dbvg-oraclelv 51432 50950 482 100% /oracle
查看SUD文件:
[[email protected] adump]# more orcl2_ora_17679_20200306210542452850143795.aud
Audit file /u01/app/oracle/admin/ORCL/adump/xxxxxx2_ora_17679_20200306210542452850143795.aud
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1
System name: Linux
Node name: ORCL
Release: 2.6.32-431.el6.x86_64
Version: #1 SMP Sun Nov 10 22:19:54 EST 2013
Machine: x86_64
Instance name: ORCL2
Redo thread mounted by this instance: 1
Oracle process number: 72
Unix process pid: 17679, image: [email protected]
Fri Mar 6 21:05:42 2020 +08:00
LENGTH: "357"
SESSIONID:[10] "4294967295" ENTRYID:[1] "1" STATEMENT:[1] "1" USERID:[5] "USER" USERHOST:[10] "server2897" TERMINAL:[7] "
unknown" ACTION:[3] "100" RETURNCODE:[1] "0" COMMENT$TEXT:[98] "Authenticated by: DATABASE; Client address: (ADDRESS=(PROT
OCOL=tcp)(HOST=10.10.28.97)(PORT=62616))" OS$USERID:[13] "Administrator" DBID:[9] "509355099" PRIV$USED:[1] "5"
Fri Mar 6 21:05:42 2020 +08:00
LENGTH: "229"
SESSIONID:[10] "4294967295" ENTRYID:[1] "1" USERID:[5] "U1228" ACTION:[3] "101" RETURNCODE:[1] "0" LOGOFF$PREAD:[1] "0" LO
GOFF$LREAD:[5] "10302" LOGOFF$LWRITE:[1] "0" LOGOFF$DEAD:[1] "0" DBID:[9] "509355099" SESSIONCPU:[2] "10"
结论:由上面服务器某个用户引起,联系程序提供商解决。
暂时解决办法:
(1)删除:ls | xargs -n 10 rm -fr ls
(2)配置crontab -e 定时删除:15 * * * * find /u01/app/oracle/admin/ORCL/adump/ -name "*aud" -exec rm -rf {} \;
长期解决办法:
(1)联系应用服务商查找原因;
(2)配置sqlnet.ora文件增加(未确定是否起作用) :SQLNET.AUTHENTICATION_SERVICES = (NONE)