I’ve talked with a few people about this idea, and seems enough have already thought about it or like it that I did a little research on what it would look like. I also included a few notes on things to do differently from the beginning, since this is starting over with the implementation while trying to keep to the design sufficiently that current XML Screens, Services, Entities, and even configurations will mostly just be the same.
The biggest issues are the scripting language and template engine. I haven’t settled on what we might do about templates, looks like there are options, but for scripting this seems like a good opportunity to migrate to JavaScript so we use a browser-compatible language on the server side.
In addition to moving to JavaScript instead of Groovy, the ElasticFacade is begging to be redone and I’d like to put something in place that uses either the embedded Postgres or OpenSearch for full-text and parametric JSON doc search. Oh, and embedded PGLite would replace H2 so we have a more complete database for dev and small production setups. Small production setups would no longer require anything external, just PGLite embedded.
Here is a first pass on the architecture proposal:
This is VERY early concept stuff, and I figure we should iterate on this and let it soak for a while before anyone starts pointing an LLM at it. Once we do settle on a plan and have it laid out more comprehensively in implementation plans, I’m all for others spending their tokens instead of mine and their initial review eyeballs instead of mine getting the thing implemented.
I guess before hashing out the architecture I want to understand the purpose of all of this. To me the programming language itself is just a means to an end.
The slowest part of systems based on moqui is network and storage. Compute and memeory are sort of secondary in that respect. So languages like rust, being more efficient won’t make a big difference no?
So what is the end goal here? Why not for example replace groovy with javascript while staying on the JVM, same with freemarker. Why replace everything, what do we get in return? In fact if we stay on the JVM, doesn’t that mean we can support both languages as two different engines in the service facade for example? What would stop us from improving elastic facade as another example?
Maybe clarifying the purpose helps shape the direction?
For Rust vs Java one motivation is speed performance, another is memory usage performance, and one that turns out to be sort of interesting is external dependency reduction.
It’s looking like we’ll be able to have Moqui with Postgres embedded, and with support for search & analytics functionality on both OpenSearch and Postgres an embedded copy of Postgres also removes the OpenSearch external dependency (unless it is desired, which for big systems I’m pretty sure it still will be… but I suppose clusters of pgsql servers that do search and analytics might compete well with a cluster of OpenSearch servers dedicated to the same task…).
So, another motivation ends up being a Moqui executable with no external dependencies, not even a JDK or JRE, that still has a full app server and a relational database and data store for search & analytics.
Realistically, none of this would be a consideration without AI-assisted code migration. It just wouldn’t be worth it. With AI-assistance the game changes.
If we look at the biggest pain points that remain with Moqui they are largely things that are really difficult to change at this point. For example we’ve had a poorly functioning prototype for server-side HTML-to-PDF for many years in the hopes of dropping XSL-FO, and a re-architecture gives us options for that and helps us convert the XSL-FO files. That isn’t the best example of what AI assistance enables because we don’t have very many XSL-FO files, but we can also get serious about replacing Groovy with JavaScript for the expressions and scripts in XML Screens so that they can finally turn into a single file that can be rendered client or server… which will require a bunch of changes to inline Groovy in existing screens so AI assistance makes that reasonable to do.
We also end up dropping a lot of the dead weight that comes with Java infrastructure. We lose a lot of the tools, but as I’ve been researching this more it turns out that most of those tools are the ones we’ve been wanting to replace anyway. So yeah, no more Groovy… but we’ve been talking about moving to JavaScript for a long time, and with a native Rust JS engine we can do that quite efficiently for little expressions and expanded strings all the way to “compiled” XML Action script for large services.
What this really does is open up more options for Moqui. It makes Moqui itself easier to embed in things like desktop apps. We can get into running modes with very low startup times for specific tasks, and startup times for production servers configs like current Moqui ones should improve significantly as well. Deployment options improve because it doesn’t have to be a memory monster in every configuration (though of course it will be in some… the goal is even more to be able to stack on thousands of concurrent users and take advantage of app servers with 512GB RAM).
I’ve been vetting this idea and working on plans, so the repo has enough docs now to show in some detail what it would look like, and the README has a section that summarizes some of the more significant changes:
All that said, this is a wild undertaking… probably 100k lines of code, most ported and some needing to be written. Just like the world of Java, options for TX management and DB connection pooling are limited in the world of Rust (even more limited), so my plan so far is to actually port relevant portions of Bitronix… and do so selectively… which scratches another itch I’ve had to pull out some of the parts of Bitronix that are annoying and not useful (like the transaction logs that we have no way to feasibly use in a recovery anyway, can’t restore the app memory state so the TX being processed is lost either way).
Anyway, now I think I’m getting long winded… but hopefully you can see where this is going.
The reason of why at all and why now come down to the cost/benefit formula changing dramatically with current AI. This is recent too, probably earlier this year might have been the earliest time to even consider doing something like this.
And yes, many of the “significant redesigns” could absolutely be done in Moqui Java, even a transition from Groovy to JavaScript for XML Screens with a decent-enough JS engine (or a better one for those wanting to do GraalVM sorts of things). Now that we have AI, doing so for all screens in Moqui and recommending that everyone switch all of theirs over time might actually happen… without AI we have discussed it over and over but the price has always been too high.
So, what is MR-Moqui? In a way it is all the stuff we’ve wanted but the price has been too high for us to get it in the past. The “base” of that iceberg is Rust, the tip being something more like Groovy-to-JavaScript. Yeah, going all the way to Rust is extreme, but there are some significant benefits to it that we just don’t have in the Java world… and it brings a few of these things together with a clear line, people knowing that migration is necessary and much backward compatibility is just gone (well, without hopefully minor migrations anyway… components using Moqui XMl artifacts like the OOTB ones do will be the priority to make easy to port, being the first targets to port).
So, in addition to things like ElasticFacade to SearchFacade (with support for OpenSearch and Postgres), Client-rendered XML Screens, using the same language for templates and scripts (GString templates concept to replace FTL, here implemented with JavaScript), and print documents via server-side HTML-to-PDF… what other sorts of improvements to Moqui have you been long wishing for?
There are potentially many things we could change in Moqui with an AI-assisted partial rewrite like this, but I’m trying to focus on the ones where there is a long history of it coming up over and over. The ones I’ve focused on so far in the MR Moqui plans are mostly based on filling the gaps for things lost by leaving the Java ecosystem. That includes JDBC & JTA, Groovy & FreeMarker, and while certain things like XSL-FO have some support it seems weaker.
Those are some big gaps to fill, but they are also tools that some of us have been talking about and looking into replacing for many years… like back in 2015 (IIRC) when I was working on that CSSBox stuff in the moqui-fop component for server-side rendering of HTML to PDF for print.
Going beyond that, what else might be worth considering? For the most part, I don’t want to change anything and for a big code port like this to work out well we’ll want to stick to what exists as much as possible. I’m even planning to keep most of the performance optimizations that exist in Moqui, like in MNode and MCache, and the low-level array index based referencing of entity field info so that tight loop stays tight and we don’t go back to looking up field info by name with a hash table.
So, what would be cool and/or has been a real pain? Let’s talk about it, especially if it is relevant to core framework functionality. There is a lot of other possibly cool stuff related to this, but this initial focus is the framework itself that would enable whatever else.
One way I’m looking at this is as an offering for AI era of information automation, so potential mass adoption is an architecture consideration. That means making it as lightweight and easy to run in various places as possible, as well as moving to more “universal” standards like JavaScript as a scripting language versus Groovy, and even HTML/CSS for print layout instead of XSL-FO. Plus, it’ll have the “Rust” buzzword so people will think it’s faster even if it’s not (we’ll make sure it is, of course).
One concern is licensing of dependencies, but from what I’m seeing so far the world of Rust is quite liberal-license-friendly, lots of MIT and Apache licensing. The planning is far enough along to do an initial review, and it checks out clean with a few things to note when attempting to bundle the world into one big binary:
