Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-17481

Ignite shutdown sequence throws a ClassCastException from inside GridManagerAdapter on latest Java 11.0.16 and 17.0.4 point releases

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0, 2.1, 2.2, (9)
      2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 2.10, 2.12, 2.13
    • 2.14
    • general
    • None
    • Fixed ClassCastException that appears during deployment manager stop in Java versions 11.0.16+ and 17.0.4+
    • Docs Required, Release Notes Required

    Description

       

      When ClassLoaders are undeployed, the `GridDeploymentStoreAdapter.clearSerializationCache()` method attempts to clear serialization caches to avoid PermGen memory leaks.  The implementation of this optimization seems to no longer work as the underlying JVM implementaiton of `java.io.ObjectInputStream$Caches` and java.io.ObjectOutputStream$Caches` no longer maintain a private cache of subclass security audit results as a java.util.Map, which Ignite expects inside `GridDeploymentStoreAdapter.clearSerializationCache()`.

       

      Stacktrace

       

      [INFO ] 2022-08-06T20:28:04,778+0000 T=[vert.x-eventloop-thread-4] L=[GridDeploymentLocalStore] - Removed undeployed class: GridDeployment [ts=1659817673460, depMode=SHARED, clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@277050dc, clsLdrId=b2497d47281-7ff6d972-ec5d-4d9c-bc60-95463b5e10b6, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap, pendingUndeploy=false, undeployed=true, usage=0]
      [ERROR] 2022-08-06T20:28:04,778+0000 T=[vert.x-eventloop-thread-4] L=[local] - Failed to stop component (ignoring): GridManagerAdapter [enabled=true, name=o.a.i.i.managers.deployment.GridDeploymentManager]
      java.lang.ClassCastException: class java.io.ObjectInputStream$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectInputStream$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')
              at org.apache.ignite.internal.managers.deployment.GridDeploymentStoreAdapter.clearSerializationCache(GridDeploymentStoreAdapter.java:151) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.managers.deployment.GridDeploymentStoreAdapter.clearSerializationCaches(GridDeploymentStoreAdapter.java:120) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.undeploy(GridDeploymentLocalStore.java:565) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.stop(GridDeploymentLocalStore.java:101) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.managers.deployment.GridDeploymentManager.storesStop(GridDeploymentManager.java:630) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.managers.deployment.GridDeploymentManager.stop(GridDeploymentManager.java:137) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1928) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1806) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2382) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2205) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:350) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at org.apache.ignite.Ignition.stop(Ignition.java:230) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at io.appliedtheory.disco.services.IgniteClusterBootstrap.stop(IgniteClusterBootstrap.java:1148) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at io.appliedtheory.disco.services.IgniteClusterService.doStop(IgniteClusterService.java:255) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at io.appliedtheory.core.component.AbstractComponent.stop(AbstractComponent.java:43) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at io.appliedtheory.framework.container.ServiceContainerEngine.lambda$doClose$1(ServiceContainerEngine.java:220) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at io.reactivex.rxjava3.internal.observers.AbstractDisposableAutoRelease.onComplete(AbstractDisposableAutoRelease.java:89) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at io.vertx.rxjava3.impl.AsyncResultCompletable.lambda$subscribeActual$0(AsyncResultCompletable.java:55) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at io.vertx.core.impl.VertxImpl$1$1.lambda$operationComplete$0(VertxImpl.java:860) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
              at java.lang.Thread.run(Thread.java:829) ~[?:?]
       
      
      

      As detailed by the commit references below, recent versions of Java 11 and 17 have changed the underlying implementation of `java.io.ObjectInputStream$Caches` from using a  `java.util.concurrent.ConcurrentMap` to a `java.lang.ClassValue`.

      JDK 17 change

      https://github.com/openjdk/jdk17/commit/8fed8ab29cae4f189f44609c23f116967eef6bdf

      JDK 11 change

      https://github.com/openjdk/jdk11u/commit/01c4d3fec62d198b2575e1b4480eb8e659776c56

       

      Current Workaround

      Pin Java 17 to 17.0.3 and earlier.

      Pin Java 11 to 11.0.15 and earlier

       

      Issue Links

        Activity

          ignitetcbot Ignite TC Bot added a comment -
          Branch: [pull/10216/head] Base: [master] : No blockers found!
          Branch: [pull/10216/head] Base: [master] : No new tests found!

          TeamCity --> Run :: All Results

          ptupitsyn Pavel Tupitsyn added a comment -

          Cherry-picked to ignite-2.14: d4e7dc62b42fdecd17bd16b013ae8a4a6b1bef5d

          People

            ibessonov Ivan Bessonov
            paolodedios Paolo de Dios
            Aleksandr Polovtcev Aleksandr Polovtcev
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m