Fixed ClassCastException that appears during deployment manager stop in Java versions 11.0.16+ and 17.0.4+
Docs Required, Release Notes Required
Description
When ClassLoaders are undeployed, the `GridDeploymentStoreAdapter.clearSerializationCache()` method attempts to clear serialization caches to avoid PermGen memory leaks. The implementation of this optimization seems to no longer work as the underlying JVM implementaiton of `java.io.ObjectInputStream$Caches` and java.io.ObjectOutputStream$Caches` no longer maintain a private cache of subclass security audit results as a java.util.Map, which Ignite expects inside `GridDeploymentStoreAdapter.clearSerializationCache()`.
Stacktrace
[INFO ] 2022-08-06T20:28:04,778+0000 T=[vert.x-eventloop-thread-4] L=[GridDeploymentLocalStore] - Removed undeployed class: GridDeployment [ts=1659817673460, depMode=SHARED, clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@277050dc, clsLdrId=b2497d47281-7ff6d972-ec5d-4d9c-bc60-95463b5e10b6, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap, pendingUndeploy=false, undeployed=true, usage=0]
[ERROR] 2022-08-06T20:28:04,778+0000 T=[vert.x-eventloop-thread-4] L=[local] - Failed to stop component (ignoring): GridManagerAdapter [enabled=true, name=o.a.i.i.managers.deployment.GridDeploymentManager]
java.lang.ClassCastException: class java.io.ObjectInputStream$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectInputStream$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')
at org.apache.ignite.internal.managers.deployment.GridDeploymentStoreAdapter.clearSerializationCache(GridDeploymentStoreAdapter.java:151) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.managers.deployment.GridDeploymentStoreAdapter.clearSerializationCaches(GridDeploymentStoreAdapter.java:120) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.undeploy(GridDeploymentLocalStore.java:565) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.stop(GridDeploymentLocalStore.java:101) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.managers.deployment.GridDeploymentManager.storesStop(GridDeploymentManager.java:630) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.managers.deployment.GridDeploymentManager.stop(GridDeploymentManager.java:137) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1928) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1806) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2382) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2205) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:350) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at org.apache.ignite.Ignition.stop(Ignition.java:230) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at io.appliedtheory.disco.services.IgniteClusterBootstrap.stop(IgniteClusterBootstrap.java:1148) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at io.appliedtheory.disco.services.IgniteClusterService.doStop(IgniteClusterService.java:255) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at io.appliedtheory.core.component.AbstractComponent.stop(AbstractComponent.java:43) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at io.appliedtheory.framework.container.ServiceContainerEngine.lambda$doClose$1(ServiceContainerEngine.java:220) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at io.reactivex.rxjava3.internal.observers.AbstractDisposableAutoRelease.onComplete(AbstractDisposableAutoRelease.java:89) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at io.vertx.rxjava3.impl.AsyncResultCompletable.lambda$subscribeActual$0(AsyncResultCompletable.java:55) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at io.vertx.core.impl.VertxImpl$1$1.lambda$operationComplete$0(VertxImpl.java:860) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
As detailed by the commit references below, recent versions of Java 11 and 17 have changed the underlying implementation of `java.io.ObjectInputStream$Caches` from using a `java.util.concurrent.ConcurrentMap` to a `java.lang.ClassValue`.
Cherry-picked to ignite-2.14: d4e7dc62b42fdecd17bd16b013ae8a4a6b1bef5d
People
Ivan Bessonov
Paolo de Dios
Aleksandr Polovtcev
Votes:
0Vote for this issue
Watchers:
4Start watching this issue
Dates
Created:
Updated:
Resolved:
Time Tracking
Estimated:
Not Specified
Remaining:
0h
Logged:
20m
{"report":{"fcp":1853.599998474121,"ttfb":64.39999961853027,"pageVisibility":"visible","entityId":13475575,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{"app-header":1853.599998474121},"userDeviceMemory":2,"userDeviceProcessors":1,"apdex":0.5,"journeyId":"d07411f7-b452-41ab-93f8-e80aa6671ef9","navigationType":0,"readyForUser":2216.5,"redirectCount":0,"resourceLoadedEnd":2381.5,"resourceLoadedStart":98.79999923706055,"resourceTiming":[{"duration":60.20000076293945,"initiatorType":"link","name":"https://issues.apache.org/jira/s/b62489a2eaac59d9b8a093c1a51d034f-CDN/40u070/820010/13pdxe5/49fa3aa3d35a2cc689cbf274e66cc41a/_/download/contextbatch/css/_super/batch.css","startTime":98.79999923706055,"connectEnd":106.89999961853027,"connectStart":99.69999885559082,"domainLookupEnd":99.69999885559082,"domainLookupStart":99.69999885559082,"fetchStart":98.79999923706055,"redirectEnd":0,"redirectStart":0,"requestStart":107,"responseEnd":159,"responseStart":124.69999885559082,"secureConnectionStart":102.5},{"duration":96.39999961853027,"initiatorType":"link","name":"https://issues.apache.org/jira/s/b23aeed077b86f32eca6cdc2838ba51e-CDN/40u070/820010/13pdxe5/98c3d6e8e62666d13309aa902e8622ce/_/download/contextbatch/css/project.issue.navigator,jira.view.issue,atl.general,atl.global,jira.global,jira.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&richediton=true&slack-enabled=true","startTime":102.19999885559082,"connectEnd":110,"connectStart":103.5,"domainLookupEnd":103.5,"domainLookupStart":103.5,"fetchStart":102.19999885559082,"redirectEnd":0,"redirectStart":0,"requestStart":110.19999885559082,"responseEnd":198.5999984741211,"responseStart":127,"secureConnectionStart":106.5},{"duration":124.60000038146973,"initiatorType":"script","name":"https://issues.apache.org/jira/s/0ea06fcee28d3e8214682cd28932a715-CDN/40u070/820010/13pdxe5/49fa3aa3d35a2cc689cbf274e66cc41a/_/download/contextbatch/js/_super/batch.js?locale=en-UK","startTime":104.29999923706055,"connectEnd":116.69999885559082,"connectStart":106.39999961853027,"domainLookupEnd":106.39999961853027,"domainLookupStart":106.39999961853027,"fetchStart":104.29999923706055,"redirectEnd":0,"redirectStart":0,"requestStart":136.5,"responseEnd":228.89999961853027,"responseStart":156.5,"secureConnectionStart":109.19999885559082},{"duration":47.20000076293945,"initiatorType":"script","name":"https://issues.apache.org/jira/s/1ca480f688281999ab265f15776f257d-CDN/40u070/820010/13pdxe5/f6337caf25773f7abc6dddfd9178ef37/_/download/contextbatch/js/jira.browse.project,-_super/batch.js?locale=en-UK","startTime":105.69999885559082,"connectEnd":118.39999961853027,"connectStart":106.0999984741211,"domainLookupEnd":106.0999984741211,"domainLookupStart":106.0999984741211,"fetchStart":105.69999885559082,"redirectEnd":0,"redirectStart":0,"requestStart":140.69999885559082,"responseEnd":152.89999961853027,"responseStart":148.29999923706055,"secureConnectionStart":108.69999885559082},{"duration":587.6000003814697,"initiatorType":"script","name":"https://issues.apache.org/jira/s/2882d4ac34059c86d866a0187d8f65db-CDN/40u070/820010/13pdxe5/98c3d6e8e62666d13309aa902e8622ce/_/download/contextbatch/js/project.issue.navigator,jira.view.issue,atl.general,atl.global,jira.global,jira.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en-UK&richediton=true&slack-enabled=true","startTime":111.19999885559082,"connectEnd":111.19999885559082,"connectStart":111.19999885559082,"domainLookupEnd":111.19999885559082,"domainLookupStart":111.19999885559082,"fetchStart":111.19999885559082,"redirectEnd":0,"redirectStart":0,"requestStart":143.19999885559082,"responseEnd":698.7999992370605,"responseStart":164.5,"secureConnectionStart":111.19999885559082},{"duration":46.30000114440918,"initiatorType":"script","name":"https://issues.apache.org/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/40u070/820010/13pdxe5/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":112.0999984741211,"connectEnd":112.0999984741211,"connectStart":112.0999984741211,"domainLookupEnd":112.0999984741211,"domainLookupStart":112.0999984741211,"fetchStart":112.0999984741211,"redirectEnd":0,"redirectStart":0,"requestStart":141.5999984741211,"responseEnd":158.39999961853027,"responseStart":150.19999885559082,"secureConnectionStart":112.0999984741211},{"duration":74.19999885559082,"initiatorType":"script","name":"https://issues.apache.org/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/40u070/820010/13pdxe5/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":114.39999961853027,"connectEnd":114.39999961853027,"connectStart":114.39999961853027,"domainLookupEnd":114.39999961853027,"domainLookupStart":114.39999961853027,"fetchStart":114.39999961853027,"redirectEnd":0,"redirectStart":0,"requestStart":168.5999984741211,"responseEnd":188.5999984741211,"responseStart":178.39999961853027,"secureConnectionStart":114.39999961853027},{"duration":72.80000114440918,"initiatorType":"link","name":"https://issues.apache.org/jira/s/981f587853769311cda7c3b845131a06-CDN/40u070/820010/13pdxe5/cb5a5495a038c0744457f25821ba9ee8/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":115.19999885559082,"connectEnd":115.19999885559082,"connectStart":115.19999885559082,"domainLookupEnd":115.19999885559082,"domainLookupStart":115.19999885559082,"fetchStart":115.19999885559082,"redirectEnd":0,"redirectStart":0,"requestStart":167,"responseEnd":188,"responseStart":176.39999961853027,"secureConnectionStart":115.19999885559082},{"duration":78.19999885559082,"initiatorType":"script","name":"https://issues.apache.org/jira/rest/api/1.0/shortcuts/820010/1acd5111678149bedc8f9039ea1d8aff/shortcuts.js?context=issuenavigation&context=issueaction","startTime":115.89999961853027,"connectEnd":115.89999961853027,"connectStart":115.89999961853027,"domainLookupEnd":115.89999961853027,"domainLookupStart":115.89999961853027,"fetchStart":115.89999961853027,"redirectEnd":0,"redirectStart":0,"requestStart":170,"responseEnd":194.0999984741211,"responseStart":184.5,"secureConnectionStart":115.89999961853027},{"duration":11.69999885559082,"initiatorType":"link","name":"https://issues.apache.org/jira/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/40u070/820010/13pdxe5/efa42a25652b26dfd802540c024826b3/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-jira.view.issue,-project.issue.navigator/batch.css?jira.create.linked.issue=true&richediton=true","startTime":305.8999996185303,"connectEnd":305.8999996185303,"connectStart":305.8999996185303,"domainLookupEnd":305.8999996185303,"domainLookupStart":305.8999996185303,"fetchStart":305.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":306.6999988555908,"responseEnd":317.5999984741211,"responseStart":314.1999988555908,"secureConnectionStart":305.8999996185303},{"duration":10.200000762939453,"initiatorType":"script","name":"https://issues.apache.org/jira/s/05fcc126e08c7d630f681b488e4e862d-CDN/40u070/820010/13pdxe5/efa42a25652b26dfd802540c024826b3/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-jira.view.issue,-project.issue.navigator/batch.js?jira.create.linked.issue=true&locale=en-UK&richediton=true","startTime":308.29999923706055,"connectEnd":308.29999923706055,"connectStart":308.29999923706055,"domainLookupEnd":308.29999923706055,"domainLookupStart":308.29999923706055,"fetchStart":308.29999923706055,"redirectEnd":0,"redirectStart":0,"requestStart":309,"responseEnd":318.5,"responseStart":315,"secureConnectionStart":308.29999923706055},{"duration":150.39999961853027,"initiatorType":"script","name":"https://issues.apache.org/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/40u070/820010/13pdxe5/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":311.8999996185303,"connectEnd":311.8999996185303,"connectStart":311.8999996185303,"domainLookupEnd":311.8999996185303,"domainLookupStart":311.8999996185303,"fetchStart":311.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":457,"responseEnd":462.29999923706055,"responseStart":461.5,"secureConnectionStart":311.8999996185303},{"duration":561.1999988555908,"initiatorType":"script","name":"https://issues.apache.org/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/40u070/820010/13pdxe5/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":311.8999996185303,"connectEnd":311.8999996185303,"connectStart":311.8999996185303,"domainLookupEnd":311.8999996185303,"domainLookupStart":311.8999996185303,"fetchStart":311.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":864.7999992370605,"responseEnd":873.0999984741211,"responseStart":870,"secureConnectionStart":311.8999996185303},{"duration":24.399999618530273,"initiatorType":"xmlhttprequest","name":"https://issues.apache.org/jira/rest/webResources/1.0/resources","startTime":860.2999992370605,"connectEnd":860.2999992370605,"connectStart":860.2999992370605,"domainLookupEnd":860.2999992370605,"domainLookupStart":860.2999992370605,"fetchStart":860.2999992370605,"redirectEnd":0,"redirectStart":0,"requestStart":861.0999984741211,"responseEnd":884.6999988555908,"responseStart":879,"secureConnectionStart":860.2999992370605},{"duration":21.69999885559082,"initiatorType":"xmlhttprequest","name":"https://issues.apache.org/jira/rest/webResources/1.0/resources","startTime":1314.5,"connectEnd":1314.5,"connectStart":1314.5,"domainLookupEnd":1314.5,"domainLookupStart":1314.5,"fetchStart":1314.5,"redirectEnd":0,"redirectStart":0,"requestStart":1315.099998474121,"responseEnd":1336.1999988555908,"responseStart":1334.3999996185303,"secureConnectionStart":1314.5},{"duration":25.30000114440918,"initiatorType":"script","name":"https://issues.apache.org/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/40u070/820010/13pdxe5/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&richediton=true&slack-enabled=true","startTime":1727.599998474121,"connectEnd":1727.599998474121,"connectStart":1727.599998474121,"domainLookupEnd":1727.599998474121,"domainLookupStart":1727.599998474121,"fetchStart":1727.599998474121,"redirectEnd":0,"redirectStart":0,"requestStart":1743.599998474121,"responseEnd":1752.8999996185303,"responseStart":1751.2999992370605,"secureConnectionStart":1727.599998474121},{"duration":103.39999961853027,"initiatorType":"css","name":"https://issues.apache.org/jira/s/40u070/820010/13pdxe5/9.2.3-4dc984d9f/_/download/resources/com.atlassian.auiplugin:split_aui.splitchunk.56dfb54d0c/assets/adgs-icons.woff","startTime":1812.5,"connectEnd":1812.5,"connectStart":1812.5,"domainLookupEnd":1812.5,"domainLookupStart":1812.5,"fetchStart":1812.5,"redirectEnd":0,"redirectStart":0,"requestStart":1908.1999988555908,"responseEnd":1915.8999996185303,"responseStart":1913.099998474121,"secureConnectionStart":1812.5},{"duration":9.899999618530273,"initiatorType":"css","name":"https://issues.apache.org/jira/s/40u070/820010/13pdxe5/1.0/_/images/icons/sprites/icons_module.png","startTime":1918.6999988555908,"connectEnd":1918.6999988555908,"connectStart":1918.6999988555908,"domainLookupEnd":1918.6999988555908,"domainLookupStart":1918.6999988555908,"fetchStart":1918.6999988555908,"redirectEnd":0,"redirectStart":0,"requestStart":1919.2999992370605,"responseEnd":1928.599998474121,"responseStart":1924.099998474121,"secureConnectionStart":1918.6999988555908},{"duration":19.600000381469727,"initiatorType":"xmlhttprequest","name":"https://issues.apache.org/jira/rest/webResources/1.0/resources","startTime":2131.099998474121,"connectEnd":2131.099998474121,"connectStart":2131.099998474121,"domainLookupEnd":2131.099998474121,"domainLookupStart":2131.099998474121,"fetchStart":2131.099998474121,"redirectEnd":0,"redirectStart":0,"requestStart":2132.2999992370605,"responseEnd":2150.699998855591,"responseStart":2148.199998855591,"secureConnectionStart":2131.099998474121},{"duration":42.89999961853027,"initiatorType":"link","name":"https://issues.apache.org/jira/s/e5e4b108c1f64d5e2a48fa234fe64053-CDN/40u070/820010/13pdxe5/d6fbf88a416ebe0c033ea3b0e4aa277d/_/download/contextbatch/css/jira.rich.editor.api,jira.project.sidebar,jira.rich.editor,-_super,-project.issue.navigator,-atl.global,-jira.view.issue,-atl.general,-jira.global,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&richediton=true&slack-enabled=true","startTime":2362.699998855591,"connectEnd":2362.699998855591,"connectStart":2362.699998855591,"domainLookupEnd":2362.699998855591,"domainLookupStart":2362.699998855591,"fetchStart":2362.699998855591,"redirectEnd":0,"redirectStart":0,"requestStart":2363.5,"responseEnd":2405.599998474121,"responseStart":2377.8999996185303,"secureConnectionStart":2362.699998855591},{"duration":60,"initiatorType":"script","name":"https://issues.apache.org/jira/s/66ce4e6e38abf678d3759a7bba50361b-CDN/40u070/820010/13pdxe5/d6fbf88a416ebe0c033ea3b0e4aa277d/_/download/contextbatch/js/jira.rich.editor.api,jira.project.sidebar,jira.rich.editor,-_super,-project.issue.navigator,-atl.global,-jira.view.issue,-atl.general,-jira.global,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en-UK&richediton=true&slack-enabled=true","startTime":2365.3999996185303,"connectEnd":2365.3999996185303,"connectStart":2365.3999996185303,"domainLookupEnd":2365.3999996185303,"domainLookupStart":2365.3999996185303,"fetchStart":2365.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":2366.199998855591,"responseEnd":2425.3999996185303,"responseStart":2385.2999992370605,"secureConnectionStart":2365.3999996185303}],"fetchStart":0,"domainLookupStart":17,"domainLookupEnd":19,"connectStart":19,"connectEnd":37,"secureConnectionStart":22,"requestStart":37,"responseStart":64,"responseEnd":133,"domLoading":85,"domInteractive":2396,"domContentLoadedEventStart":2396,"domContentLoadedEventEnd":2539,"domComplete":2968,"loadEventStart":2968,"loadEventEnd":2976,"userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/105.0.5195.102 Safari/537.36","marks":[{"name":"bigPipe.sidebar-id.start","time":2279.099998474121},{"name":"bigPipe.sidebar-id.end","time":2280.199998855591},{"name":"bigPipe.activity-panel-pipe-id.start","time":2280.3999996185303},{"name":"bigPipe.activity-panel-pipe-id.end","time":2281.8999996185303},{"name":"activityTabFullyLoaded","time":2649.699998855591}],"measures":[],"correlationId":"fb8db578a3017f","effectiveType":"4g","downlink":9.7,"rtt":0,"serverDuration":87,"dbReadsTimeInMs":3,"dbConnsTimeInMs":10,"applicationHash":"ace47f9899e9ee25d7157d59aa17ab06aee30d3d","experiments":[]}}
Cached: [IGNITE-17481] Ignite shutdown sequence throws a ClassCastException from inside GridManagerAdapter on latest Java 11.0.16 and 17.0.4 point releases - ASF JIRA
TeamCity --> Run :: All Results