线上问题总结-获取不到连接池(logback 配置+代码问题)

时间：2024-07-19 18:52:04

原文链接

出错问题复盘

1、问题回顾

早上 6:00 多开始报警，数据库连接池获取不到连接，日志的报错如下：

org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
### The error may exist in class path resource [com/ xxx/xxxMapper.xml]
### The error may involve xxx.getListByOrgIdOrRealName
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms

导致的结果就是：

40 多笔提现失败，因为那个时候刚好有客户要提现

2、定位问题

2.1 首先看应用日志

看应用日志，最主要的信息就是 Failed to obtain JDBC Connection、HikariPool-1 - Connection is not available，request timed out after 30000ms

1、看这个信息的第一反应就是数据库连接是不是被打满了，先去看了数据库监控，发现连接数很平稳，没有太大的变化(zabbix 的监控)。

2、然后数据库连接没有很多的话，第 2 步去看看 Hikari 连接池的配置，配置如下：

spring.datasource.hikari.connectionTimeout = 100000
spring.datasource.hikari.validationTimeout = 300000
spring.datasource.hikari.maximum-pool-size = 200
spring.datasource.hikari.minimum-idle = 50

3、当时是想着把 maximum-pool-size=500(就是 200 调整为 500)，当然此时还没有改，也就没有发布、重启

2.2 看 jstack 日志
2.1 jsatck 日志如下
“xxxx_Worker-1” #206 prio=5 os_prio=0 tid=0x00007f508f6ab000 nid=0x1705 waiting on condition [0x00007f4f61818000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x0000000720ab5208> (a java.util.concurrent.locks.AbstractQueuedSynchronizerC o n d i t i o n O b j e c t ) a t j a v a . u t i l . c o n c u r r e n t . l o c k s . L o c k S u p p o r t . p a r k ( L o c k S u p p o r t . j a v a : 175 ) a t j a v a . u t i l . c o n c u r r e n t . l o c k s . A b s t r a c t Q u e u e d S y n c h r o n i z e r ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizerConditionObject)atjava.util.concurrent.locks.LockSupport.park(LockSupport.java:175)atjava.util.concurrent.locks.AbstractQueuedSynchronizerConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353)
at ch.qos.logback.core.AsyncAppenderBase.putUninterruptibly(AsyncAppenderBase.java:181)
at ch.qos.logback.core.AsyncAppenderBase.puto r i g i n a l originaloriginalXaWJEOe0(AsyncAppenderBase.java:172)
at ch.qos.logback.core.AsyncAppenderBase.puto r i g i n a l originaloriginalXaWJEOe0$accessor67 X f D E R 7 ( A s y n c A p p e n d e r B a s e . j a v a ) a t c h . q o s . l o g b a c k . c o r e . A s y n c A p p e n d e r B a s e 67XfDER7(AsyncAppenderBase.java) at ch.qos.logback.core.AsyncAppenderBase67XfDER7(AsyncAppenderBase.java)atch.qos.logback.core.AsyncAppenderBaseauxiliary$fPj3kqdc.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)

相关文章