Inconsistent classloaders between war and warPlusRuntime

Technical Discussion
1

I’ve been using Moqui to call some remote ejb methods and upgrading the remote server to Wildfly 32 and hence the jboss-client libraries I’ve started to have some errors in Moqui. After debugging the errors for a while I discovered that the problem was because of the classloaders, because locally the calls were working, but when running Moqui in Docker they were not working anymore. The only difference is that locally I was using moqui.war and docker uses moqui-plus-runtime.war.

The classloaders when running locally:

ServiceLocator(components/moqui-linic-legacy): MClassLoader
RemoteService(components/moqui-linic-legacy/lib/colibri_stat_ejb.jar): MClassLoader
WildFlyInitialContextFactory(components/moqui-linic-legacy/lib/jboss-client.jar): MClassLoader

The classloaders when running inside docker:

ServiceLocator(components/moqui-linic-legacy): oejew.WebAppClassLoader
RemoteService(components/moqui-linic-legacy/lib/colibri_stat_ejb.jar): oejew.WebAppClassLoader
WildFlyInitialContextFactory(components/moqui-linic-legacy/lib/jboss-client.jar): MoquiStart$StartClassLoader

From my understanding classes that use different classloaders do not see each other, hence why ServiceLocator was failling, because it was not seeing WildFlyInitialContextFactory.

Shouldn’t classloading be consistent in order to avoid these types of errors between different environments? Because I see that moquiPlusRuntime puts all jars in the WEB-INF directory, whereas moqui.war doesn’t put the components jars there.

2

on which version of moqui are you facing this?

3

4.0.0 the latest version. I do have some customizations compared to upstream, but nothing related to classloading, mostly just localization.

4

What doesn’t make much sense is that MClassLoader is not there which should always be there regardless of whether starting from MoquiStart or servlet container. How did you determine that MClassLoader is not there? how did you investigate the classloader stack. and from where? This could help shed light on things. The logic that changed in moqui 4 was actually to dig for more, instead of just parent classloader, it goes 2 levels up to be able to reach all the way to MoquiStart in case of war file embedded jetty. So a little info on your debugging might help.

5

The remote ejb calls were made in a service, so I just put some logging before those calls, like this:

ec.logger.info("RemoteService class loader = " + ro.colibri.beans.RemoteService.class.getClassLoader());
ec.logger.info("RemoteService code source = " + ro.colibri.beans.RemoteService.class.getProtectionDomain().getCodeSource());
ec.logger.info("WildFlyInitialContextFactory class loader = " + org.wildfly.naming.client.WildFlyInitialContextFactory.class.getClassLoader());
ec.logger.info("WildFlyInitialContextFactory code source = " + org.wildfly.naming.client.WildFlyInitialContextFactory.class.getProtectionDomain().getCodeSource());
ec.logger.info("ServiceLocator class loader = " + ServiceLocator.class.getClassLoader());
ec.logger.info("ServiceLocator code source = " + ServiceLocator.class.getProtectionDomain().getCodeSource());

What I understand by using ChatGPT is that the class is assigned to the first classloader that loads it, which would make sense given the output that I get.

Here is my point of view why this happens(untested, just a hypothesis):
StartClassLoader loads all jars from moqui.war, but since the war doesn’t contain jars from the components, neither of the 3 classes are loaded, they are loaded later by MClassLoader directly from the runtime directory; but when moqui-plus-runtime.war is exploded all jars are put into WEB-INF, including component jars, thus StartClassLoader loads all jars, but I think classes are lazy loaded only on demand. So I guess WildFlyInitialContextFactory auto registers itself during startup, and the other 2 classes(ServiceLocator and RemoteService) are only loaded when needed, that means when the moqui service is called, hence why they are loaded by WebAppClassLoader. And I think MClassLoader in this case doesn’t load them since they are already loaded by parent classloaders.

6

OK, I think this is not the right way to identify the classloader chain. The right way to do it for both embedded and servlet-container, is to get the parent in a chain. So you need to climb up and print the class starting from somewhere in your moqui code and just climb up from current thread all the way up and print. This will compare apple to apple the classloaders from both launches.

Now, the way the class loading works in moqui, is that it FIRST checks if I can load from parent, up to 2 parents up (to reach MoquiStart.java in case of embedded) and in case of servlet container it’s whatever that container provides. Regardless, the proper way is to consult parent first, and if not found go to MClassLoader. So if the load happened by the servlet container, it should work fine, and the same if coming from MClassLoader.

My suggestion to you is to investigate both stacks, compare clearly the two. Then, we check what is the custom class loader of your servlet container and how high up the hierarchy is it, if it is larger than 2, the possibly this could be the reason. If it’s within MclassLoader + parent + parent then it should work. There is no problem of repitition in class loaders, because the logic already tries to find the right thing first before consulting the other class loaders. I hope that makes sense.

Let’s start with that investigation, then we can pin down where this is coming from.