ClassCastException if loaded by different class loaders in OSGi runtime

[mmeldo]

Description

Our code (Eclipse plugin) using Jersey client is running in OSGi runtime (Eclipse IDE). There are some Eclipse platforms that contain version of Jersey libraries different from that required by our plugin. Therefore our plugin puts the required version to the classpath. However, each Jersey version is then loaded by a different class loader. There are two places where this is causing problems:

• lookup of the InjectionManagerFactory in org.glassfish.jersey.internal.inject.Injections
• configuring of auto discoverable providers in org.glassfish.jersey.model.internal.CommonConfig

The problem is that in those places and in this scenario, cast is done using classes from different class loaders resulting in ClassCastException stopping the request completely. We think these casts could be avoided and the request should be able to proceed.

Lookup of the InjectionManagerFactory

Note: I have truncated the exception stack trace to show only the Jersey stuff because it is quite long with all the Eclipse stuff.

Exception:

java.lang.ClassCastException: class org.glassfish.jersey.inject.hk2.Hk2InjectionManagerFactory cannot be cast to class org.glassfish.jersey.internal.inject.InjectionManagerFactory (org.glassfish.jersey.inject.hk2.Hk2InjectionManagerFactory is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @381dbeba; org.glassfish.jersey.internal.inject.InjectionManagerFactory is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @1d49806)
	at org.glassfish.jersey.internal.inject.Injections.lookupInjectionManagerFactory(Injections.java:74) ~[org.glassfish.jersey.core.jersey-common_3.0.4.jar:?]
	at org.glassfish.jersey.internal.inject.Injections.createInjectionManager(Injections.java:44) ~[org.glassfish.jersey.core.jersey-common_3.0.4.jar:?]
	at org.glassfish.jersey.client.ClientConfig$State.initRuntime(ClientConfig.java:413) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:317) ~[org.glassfish.jersey.core.jersey-common_3.0.4.jar:?]
	at org.glassfish.jersey.client.ClientConfig.getRuntime(ClientConfig.java:820) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.ClientRequest.getClientRuntime(ClientRequest.java:176) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.ClientRequest.getInjectionManager(ClientRequest.java:567) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.onBuilder(JerseyWebTarget.java:371) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:206) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at com.ca.endevor.service.EndevorServiceRest$DatasourceAuth.getAsJson(EndevorServiceRest.java:525) ~[com.ca.endevor.uiProject/:?]

Code snippet:

jersey/core-common/src/main/java/org/glassfish/jersey/internal/inject/Injections.java

Lines 72 to 93 in 8f7fa99

	private static InjectionManagerFactory lookupInjectionManagerFactory() {
	return lookupService(InjectionManagerFactory.class)
	.orElseThrow(() -> new IllegalStateException(LocalizationMessages.INJECTION_MANAGER_FACTORY_NOT_FOUND()));
	}
	/**
	* Look for a service of given type. If more then one service is found the method sorts them are returns the one with highest
	* priority.
	*
	* @param clazz type of service to look for.
	* @param <T> type of service to look for.
	* @return instance of service with highest priority or {@code null} if service of given type cannot be found.
	* @see javax.annotation.Priority
	*/
	private static <T> Optional<T> lookupService(final Class<T> clazz) {
	List<RankedProvider<T>> providers = new LinkedList<>();
	for (T provider : ServiceFinder.find(clazz)) {
	providers.add(new RankedProvider<>(provider));
	}
	providers.sort(new RankedComparator<>(RankedComparator.Order.DESCENDING));
	return providers.isEmpty() ? Optional.empty() : Optional.ofNullable(providers.get(0).getProvider());
	}

Configuring of AutoDiscoverable

Note: I have truncated the exception stack trace to show only the Jersey stuff because it is quite long with all the Eclipse stuff.

Exception:

java.lang.ClassCastException: class org.glassfish.jersey.server.wadl.internal.WadlAutoDiscoverable cannot be cast to class org.glassfish.jersey.internal.spi.AutoDiscoverable (org.glassfish.jersey.server.wadl.internal.WadlAutoDiscoverable is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @654b2ed4; org.glassfish.jersey.internal.spi.AutoDiscoverable is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @55bc4134)
	at java.util.TreeMap.put(TreeMap.java:787) ~[?:?]
	at java.util.TreeMap.put(TreeMap.java:534) ~[?:?]
	at java.util.TreeSet.add(TreeSet.java:255) ~[?:?]
	at java.util.AbstractCollection.addAll(AbstractCollection.java:336) ~[?:?]
	at java.util.TreeSet.addAll(TreeSet.java:310) ~[?:?]
	at org.glassfish.jersey.model.internal.CommonConfig.configureAutoDiscoverableProviders(CommonConfig.java:609) ~[org.glassfish.jersey.core.jersey-common_3.0.4.jar:?]
	at org.glassfish.jersey.client.ClientConfig$State.configureAutoDiscoverableProviders(ClientConfig.java:384) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.ClientConfig$State.initRuntime(ClientConfig.java:439) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:317) ~[org.glassfish.jersey.core.jersey-common_3.0.4.jar:?]
	at org.glassfish.jersey.client.ClientConfig.getRuntime(ClientConfig.java:820) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.ClientRequest.getClientRuntime(ClientRequest.java:176) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.ClientRequest.getInjectionManager(ClientRequest.java:567) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.onBuilder(JerseyWebTarget.java:371) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:206) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:38) ~[org.glassfish.jersey.core.jersey-client_3.0.4.jar:?]
	at com.ca.endevor.service.EndevorServiceRest$DatasourceAuth.getAsJson(EndevorServiceRest.java:524) ~[com.ca.endevor.uiProject/:?]

Code snippet:

jersey/core-common/src/main/java/org/glassfish/jersey/model/internal/CommonConfig.java

Lines 582 to 629 in 8f7fa99

	/**
	* Configure {@link AutoDiscoverable auto-discoverables} in the injection manager.
	*
	* @param injectionManager injection manager in which the auto-discoverables should be configured.
	* @param autoDiscoverables list of registered auto discoverable components.
	* @param forcedOnly defines whether all or only forced auto-discoverables should be configured.
	*/
	public void configureAutoDiscoverableProviders(final InjectionManager injectionManager,
	final Collection<AutoDiscoverable> autoDiscoverables,
	final boolean forcedOnly) {
	// Check whether meta providers have been initialized for a config this config has been loaded from.
	if (!disableMetaProviderConfiguration) {
	final Set<AutoDiscoverable> providers = new TreeSet<>((o1, o2) -> {
	final int p1 = o1.getClass().isAnnotationPresent(Priority.class)
	? o1.getClass().getAnnotation(Priority.class).value() : Priorities.USER;
	final int p2 = o2.getClass().isAnnotationPresent(Priority.class)
	? o2.getClass().getAnnotation(Priority.class).value() : Priorities.USER;
	return (p1 < p2 || p1 == p2) ? -1 : 1;
	});
	// Forced (always invoked).
	final List<ForcedAutoDiscoverable> forcedAutoDiscroverables = new LinkedList<>();
	for (Class<ForcedAutoDiscoverable> forcedADType : ServiceFinder.find(ForcedAutoDiscoverable.class, true)
	.toClassArray()) {
	forcedAutoDiscroverables.add(injectionManager.createAndInitialize(forcedADType));
	}
	providers.addAll(forcedAutoDiscroverables);
	// Regular.
	if (!forcedOnly) {
	providers.addAll(autoDiscoverables);
	}
	for (final AutoDiscoverable autoDiscoverable : providers) {
	final ConstrainedTo constrainedTo = autoDiscoverable.getClass().getAnnotation(ConstrainedTo.class);
	if (constrainedTo == null || type.equals(constrainedTo.value())) {
	try {
	autoDiscoverable.configure(this);
	} catch (final Exception e) {
	LOGGER.log(Level.FINE,
	LocalizationMessages.AUTODISCOVERABLE_CONFIGURATION_FAILED(autoDiscoverable.getClass()), e);
	}
	}
	}
	}
	}

Suggested solution

I have tested a solution with our code that will simply check in both places if the cast can be done. Do you think the suggested solution is OK? Or is there some other way to solve this situation?
I can open a PR if you think the solution is OK.

[jansupol]

I do not think this is a proper solution:

• Having a single Jersey version with 2 classloaders won't work
• Having two versions of Jersey with a single classloader won't work

The issue seems to be that a single classloader sees 2 versions of Jersey, but the classloaders should be restricted to a single version of Jersey (as well as all dependencies).

To me, when you add a check on these two places, you will get a similar ClassCastException only a little longer, there would be many places where you would get the same issue. HK2 proxy is one big example where two classloaders make issues, the injections likely will stop working.

Perhaps you could make your OSGi headers of your app to require the EXACT version of Jersey instead of range of versions?

[mmeldo]

We are running under the Eclipse runtime, so the dependency versions are defined in the manifest. In our manifest, there already is an EXACT version (3.0.4) for Jersey:

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: CA Endevor SCM Plug-in for Eclipse
Bundle-SymbolicName: com.ca.endevor.uiProject; singleton:=true
Bundle-Version: 18.1.8.qualifier
Bundle-Activator: com.ca.endevor.ui.UIPlugin
Bundle-Vendor: %providerName
Bundle-Localization: plugin
Bundle-ClassPath: uiEndevorProject.jar,
 lib/json-20180813.jar,
 lib/httpclient-4.5.6.jar,
 lib/httpcore-4.4.10.jar,
 lib/commons-codec-1.15.jar,
 lib/commons-configuration2-2.7.jar,
 lib/commons-io-2.11.0.jar,
 lib/commons-lang3-3.12.0.jar,
 lib/log4j-api-2.17.1.jar,
 lib/log4j-core-2.17.1.jar
Export-Package: 
 com.ca.endevor.core,
 com.ca.endevor.core.log,
 com.ca.endevor.core.utils,
 com.ca.endevor.element,
 com.ca.endevor.model,
 com.ca.endevor.service,
 com.ca.endevor.ui,
 com.ca.endevor.ui.actions,
 com.ca.endevor.ui.logview,
 com.ca.endevor.ui.views,
 com.ca.endevor.ui.wizards
Require-Bundle: org.eclipse.ui,
 org.eclipse.core.resources,
 org.eclipse.team.ui,
 org.eclipse.ui.views,
 org.eclipse.ui.ide,
 org.eclipse.team.core,
 org.eclipse.core.runtime;bundle-version="[3.2.0,4.0.0)",
 com.ibm.icu,
 org.eclipse.ui.editors;bundle-version="3.6.1",
 org.eclipse.jface.text;bundle-version="3.6.1",
 org.eclipse.search;bundle-version="3.6.0",
 org.eclipse.ui.forms;bundle-version="3.5.2",
 org.eclipse.jface,
 org.eclipse.core.expressions,
 org.eclipse.compare;bundle-version="3.5.501",
 org.eclipse.equinox.security
Import-Package: com.fasterxml.jackson.annotation;version="2.13.2",
 com.fasterxml.jackson.core;version="2.13.2",
 com.fasterxml.jackson.core.type;version="2.13.2",
 com.fasterxml.jackson.databind;version="2.13.2.2",
 com.fasterxml.jackson.dataformat.xml;version="2.13.2",
 jakarta.ws.rs;version="3.0.0",
 jakarta.ws.rs.client;version="3.0.0",
 jakarta.ws.rs.core;version="3.0.0",
 jakarta.ws.rs.ext;version="3.0.0",
 org.glassfish.jersey.client;version="[3.0.4,3.0.4]",
 org.glassfish.jersey.client.authentication;version="[3.0.4,3.0.4]",
 org.glassfish.jersey.client.filter;version="[3.0.4,3.0.4]",
 org.glassfish.jersey.inject.hk2;version="[3.0.4,3.0.4]",
 org.glassfish.jersey.internal;version="[3.0.4,3.0.4]",
 org.glassfish.jersey.media.multipart;version="[3.0.4,3.0.4]",
 org.glassfish.jersey.media.multipart.file;version="[3.0.4,3.0.4]",
 org.glassfish.jersey.message;version="[3.0.4,3.0.4]",
 org.glassfish.jersey.uri;version="[3.0.4,3.0.4]",
 org.glassfish.jersey.internal.inject;version="[3.0.4,3.0.4]"
Bundle-ActivationPolicy: lazy
Bundle-RequiredExecutionEnvironment: JavaSE-1.8
Eclipse-BundleShape: dir

The other version in the Eclipse platform is 2.30.1

I do not think that issue is that single classloader sees 2 versions of Jersey. As it says in the exception, the problem is that there are 2 classloaders:
org.eclipse.osgi.internal.loader.EquinoxClassLoader @381dbeba vs org.eclipse.osgi.internal.loader.EquinoxClassLoader @1d49806
each with a different version of Jersey.

I traced all the Jersey bundles registered by the OsgiRegistry, and it registers Jersey bundles for both versions. Later, when the ServiceFinder is looking for particular services/classes, it find them for both versions and then it throws ClassCastException when it is trying to cast from class in one classloader to the other.

In our case, we are creating only the Jersey client and I have tested my suggested changes, and did not hit any other issue.
Of course, I do not know the code as you do, so you may know a better solution and not just adding a check to those 2 places.

[mmeldo]

First, one correction. The ClassCastException is not because those classes are loaded by different class loaders. It seems Eclipse runtime uses different bundle class loaders for every bundle.

But it does not change a fact that Jersey's OsgiRegistry class registers bundles for both versions, therefore it finds both when looking for a service provider. Last few days, I tried to persuade the Eclipse runtime (Equinox OSGi) and Jersey to register only required versions without any success.

You mentioned that it is not a proper solution to do the class cast check only on those 2 places. I have tested a solution where I put the check into the OsgiRegistry.locateAllProviders which constructs the list of all possible providers for particular service.
It is a simple check if the service class is assignable from the provider class. Would that be OK?

[jansupol]

Updating OsgiRegistry sounds good.

Ideally, JerseyVersion#{getVersion, getBuildId} could be used to compare incoming Jersey bundles and their version and not to register them with OsgiRegistry when the version differs. I have not tested that, not sure if it is feasible.

[mmeldo]

Updating OsgiRegistry sounds good.

Ideally, JerseyVersion#{getVersion, getBuildId} could be used to compare incoming Jersey bundles and their version and not to register them with OsgiRegistry when the version differs. I have not tested that, not sure if it is feasible.

Thanks for the suggestion with the JerseyVersion#{getVersion, getBuildId}, I will check that when I have a chance.

[mmeldo]

I was trying to implement the suggestion but to me it does not seem feasible.
Not sure, how you meant to use the org.glassfish.jersey.internal.Version#{getVersion, getBuildId}, but I ended up using the org.osgi.framework.Bundle#{getVersion, getSymbolicName}.

Anyway, In my opinion the check in the OsgiRegistry.locateAllProviders is better solution as it will just prevent a situation when the provider class cannot be cast to service class, which would subsequently cause an exception preventing the request to be completed. Even though it could have been completed with different services.

Maybe I can open a PR so we can discuss the solution further.