使用Zipkin TracingStatementInterceptor死锁启动c3po连接池

时间:2019-05-10 04:26:34

标签: mysql c3p0 zipkin

通过在Spring Boot应用程序中包含以下库来尝试添加Zipkin mysql跟踪工具

compile group: 'io.zipkin.brave', name: 'brave-instrumentation-mysql', version: '5.6.3'

并将以下内容附加到我的jdbc连接字符串中:

?statementInterceptors=brave.mysql.TracingStatementInterceptor

现有的数据库连接使用的c3po连接池配置如下:

<bean id="dataSourceBean" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close" primary="true">
    <property name="driverClass" value="${db.driver}" />
    <property name="jdbcUrl" value="${db.url}" />
    <property name="user" value="${db.username}" />
    <property name="password" value="${db.password}" />
    <property name="initialPoolSize" value="${db.initial.pool.size}" />
    <property name="minPoolSize" value="${db.min.pool.size}" />
    <property name="maxPoolSize" value="${db.max.pool.size}" />
    <property name="idleConnectionTestPeriod" value="30" />
    <property name="maxIdleTimeExcessConnections" value="120" />
</bean>

但是在尝试从池中获取连接时遇到启动服务问题:

2019-05-09 22:54:40.752  WARN [api,,,] 31975 --- [-AdminTaskTimer] c.m.v.a.ThreadPoolAsynchronousRunner     : com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@222a008d -- APPARENT DEADLOCK!!! Complete Status: 
    Managed Threads: 3
    Active Threads: 3
    Active Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5cf72aaf
            on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#0
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@67c13f4f
            on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#1
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@dff1fcc
            on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#2
    Pending Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@775717e4
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1290e9f3
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7cc597a9
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@9b1e682
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@46963867
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@53dc8e37
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3f4233a
Pool thread stack traces:
    Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#0,5,main]
        org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:388)
        org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
        org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
        org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
        org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
        brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
        brave.Tracer.nextContext(Tracer.java:252)
        brave.Tracer.newRootContext(Tracer.java:212)
        brave.Tracer.newTrace(Tracer.java:142)
        brave.Tracer.nextSpan(Tracer.java:436)
        brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
        brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
        com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
        com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
        com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
        com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
        com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
        com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
        com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
        sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
        com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
    Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#1,5,main]
        org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:388)
        org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
        org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
        org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
        org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
        brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
        brave.Tracer.nextContext(Tracer.java:252)
        brave.Tracer.newRootContext(Tracer.java:212)
        brave.Tracer.newTrace(Tracer.java:142)
        brave.Tracer.nextSpan(Tracer.java:436)
        brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
        brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
        com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
        com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
        com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
        com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
        com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
        com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
        com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
        sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
        com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
    Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#2,5,main]
        org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:204)
        org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:392)
        org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305)
        org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144)
        org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
        org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:356)
        org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$570/35986993.getObject(Unknown Source)
        org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:389)
        org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
        org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
        org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
        org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
        brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
        brave.Tracer.nextContext(Tracer.java:252)
        brave.Tracer.newRootContext(Tracer.java:212)
        brave.Tracer.newTrace(Tracer.java:142)
        brave.Tracer.nextSpan(Tracer.java:436)
        brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
        brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
        com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
        com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
        com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
        com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
        com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
        com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
        com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
        sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
        com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

1 个答案:

答案 0 :(得分:0)

该跟踪似乎触发了一个老式的反向锁定顺序死锁冻结线程,这些线程正试图获取新的连接。

三个死锁线程中的最后一个是trying to get a lock on some singleton or bean。它已经通过并且可能已获得对GenericScope的锁定。

另外两个线程是trying to acquire a lock on a GenericScope,大概第一个线程具有。

zipkin代码意外进入Spring会产生死锁。 c3p0有一个固定大小的线程池,该线程池会通知其所有线程(这里只有3个,c3p0的默认线程)都被永久冻结,然后(在这种情况下非常正确)声明死锁并替换阻塞线程以希望恢复。

c3p0是否恢复?这是罕见的还是频繁的僵局?要防止这种死锁,您可以轻松地做很多事情,我想您要么必须容忍它,要么在没有仪器的情况下做。

相关问题