LLM Client, AI Assist Screen, Moqui Sim Mode, etc (llm-client branches)

Technical Discussion
1

There are two new branches called ‘llm-client’, in moqui-framework and moqui-runtime:

These contain a proposed architecture and implementation of an LlmClient class in the framework that handles talking to an external LLM, and it handles tool calling, agent loop, logging, etc. The initial tools include artifact info browsing, skill lookup, skill leaning in a simulation overlay, and calling screen transitions, services, etc using the current user’s authz.

The tooling architecture and skill learning are designed in a particular way to help mitigate issues with AI automation in business systems. The primary mode is “skill-first” where the agent looks for a relevant skill, and if it does not find one it spawns a sub-agent in a simulated environment to experiment and find a solution, then document it in a skill.

The simulation environment uses a number of existing context and production isolation mechanisms in the framework, and adds a new one that I’ve been thinking of doing for a long time: TransactionCacheDb. This uses H2 as an in-memory database to overlay the external database and act as a multi-transaction read-through cache where writes stay in the cache DB until flushed to the external DB, or as in the case of the sim application, thrown away.

Built on top of these is the Assist screen. This is a new screen in the tools component available to all users. It has a LLM chat on the left, and a screen canvas on the right. It supports two render modes for the LLM: a simple JSON form spec output, and a Vue Single-File Component output. The overall workflow is ask the LLM to do something, it figures out how to and creates a form for it, even if the form is just descriptive text and a submit button. The user always maintains control, changes only made when buttons are clicked.

With this in place, and once it is all cleaned up and production-worthy, the game changes for what is most important to build and maintain. Creating screens is more about the structure and data+logic encapsulation that they offer than the output they produce. The XML REST API becomes a far more viable approach for internal as well as external data access and logic. Ideally, only very few screens are ever made again, focusing on those with more complex interactions and requirements, and that will be used frequently or for a very particular important purpose.

For general notes see the ReleaseNotes.md file (in the branch):

For an overview of the skill learning and TransactionCacheDb stuff, see the new LlmSkillLearning.md plan:

3 Likes
2

A note on status for llm-client: this was all slapped together quickly with AI assistance. It has automated tests, and has been tested end-to-end including skill learning and use, but it has NOT been evaluated for performance, reasonable resource usage in production settings, or ensuring that the sim environment is sufficiently isolated.

In other words, this should definitely still be considered experimental and NOT used in production. It’s great for local use with an LLM running in LM Studio or vLLM or whatever, and that has been the only real test case so far.

If anyone would like to help with testing, using it on a staging server with a clone of a production DB is a test that needs to be done a bunch sooner or later, but any testing and review would be helpful. I made sure a few things were done in very particular ways, but there could still be a bunch of stuff in there that I haven’t caught… because I have not yet read/scanned all of the code, not tested from various angles to catch things that I’d probably miss in a code read anyway.

You’ve been warned. That said, have fun! I am. :slight_smile:

3

This is an excellent feature — looking forward to the official release soon!

Another project in the community (schue/moqui-mcp: MCP component for Moqui) provides MCP and built-in Agent capabilities. With this feature upgrade, the Agent functionality in that project can be deprecated. But what about its MCP functionality? Could it be promoted to a standard feature of the framework?

4

The ‘tools’ functionality is basically MCP, but Moqui-specific and not a general plugin sort of thing. More tools could be added, but most things would run through services, screen transitions, etc and external tools would plugin better through those interfaces so that security (authz) and other protections are in place. That is an important part of the design.

This is an alternative to what @schue put together with his Moqui MCP work, and is complementary to the more recent SystemMessage based things (ie those could be adapted to use this).

5

Hi David,

I wonder if it is convenient to extend the llm-client branch to also support the A2A standard as an interoperability layer, so a Moqui instance could act as a remote agent for other agents (and reach out to them in turn). Here is the link to the core concepts: Core Concepts - A2A Protocol

Perhaps a mapping could be as follows:

  • Message → LlmMessage, but holding one or more Parts (below) rather than today’s flat content String.
  • Part → the fundamental content container, shared by both Message and Artifact. Carried as typed JSON at first (matching the *Json field pattern), but since it’s referenced from two parents it could probably have its own child entity.
  • Task → perhaps the only piece currently missing, and I don’t think it wants to be grafted onto the LlmConversation lifecycle — a conversation is the session, not a unit of work. It feels more like a new first-class entity living in the skill-execution layer, closer to LlmSkillUse (which already records a run of a skill) than to a conversation. TaskStatus would then be its own lifecycle (submitted/working/input-required/completed/…), separate from conversation status.
  • contextId → this is where LlmConversation fits naturally: the session that groups related Tasks, rather than being a Task itself.
  • Artifact → a new entity (with a name that doesn’t collide with the framework’s own Artifact* concepts — ArtifactHit/ArtifactAuthz/…), likewise composed of Parts.
  • AgentCard → a discovery document assembled from profile config plus a curated projection over LlmSkill; AgentSkill would add the advertise-side metadata (tags/examples/io modes) that LlmSkill doesn’t carry today, since its role is internal procedural memory rather than a public capability list.

What I find most compelling is that a remote agent reaching Moqui over A2A would inherit the skill-first, sim-protected, authz-gated execution for free — which arguably makes Moqui a safer A2A remote agent than a bare LLM wrapper.

Thanks,
Igor

1 Like
6

Thanks @igor.gallingani, that’s a good idea. With the surge of desktop agents something like this might end up being a key enabling feature. I probably won’t get to the implementation until next week, but here is a first pass on an impl plan:

7

Hi David, I’m wondering if it makes sense to upgrade the code in pull request Replace synchronized blocks with ReentrantLock; adopt virtual threads by moqui-industrial · Pull Request #718 · moqui/moqui-framework · GitHub , which introduces virtual threads, so it can be used in the llm-client branch. I think it could be useful in distributed or disconnected LLM/A2A scenarios.

1 Like
8

I played around with this for a while and it is very interesting. It does put you in the position of phrasing your tasks in the form of a screen definition. A weird corollary of this would be prompt based screen definitions that use prose (:`-D)

Qwen 3.8 27B did fine with simple prompts but more complicated requests would generate a lot of traffic with the model and eventually stop without a response.

That said, all really cool stuff! I’ll port the CommunicationEvent based flow I was playing with to these services.

Something else I wonder about would be AI “process maps” where plain language generates WorkEffort based task flow templates and visualizes them with Mermaid or some other charting tool. ie. Business process mapping - Wikipedia

9

Fuel for discussion on Friday, here is an adaption of the ECA/SECA triggers I was working on to wire AI into existing features on Moqui. You can communicate with agents by just messaging them in the Moqui console. I think we are going to need a lot more skills though and maybe some managerial layer that figures out which agent it should route user requests to.

10

These are task-specific skills so far, but I’ve started a library for the Marble ERP app in the ‘llm-client’ branch (to match the name in framework and runtime) of the MarbleERP component:

I’ve gone through a couple rounds of testing and refining these, along with testing and refining other prompts (which have had errors, misleading examples, missing information, etc, etc)… and chances are they will go through more rounds of refinement.

I’ve tested a bit with Qwen 3.8 27B but am mostly testing with Ornith 1.5 or Qwen 3.6 35B because they are so much faster, and are sufficiently compliant that most of the issues I’ve run into were because of a bug in the harness (Assist screen and related infra) rather than in the openui-lang output from the LLM. Using the new-ish “Adjust” button makes it easier to request corrections and respond to errors, and that has worked fairly well even with Ornith or Qwen 35B.

That said, Qwen 3.8 27B is on another tier entirely when it comes to its intelligence and ability to get things done, quite an impressive model for its size, and from what I can tell was released about 6 months after roughly equivalent flagship models from OpenAI and Anthropic. We’re probably less than 6 months from a ~30B parameter model doing what Astra and Fable and Argon and all those fancy new names are doing now…

Anyway, from my testing so far I’ve had far more trouble getting all the fiddly bits right in the harness than with the local LLM responding adequately. I don’t know if that is an indicator of future things for developing harnesses like the Assist screen, but likely to become more true over time rather than less. In other words, when running into issues with the Assist screen the first assumption when seeing an error is that there is a bug in Moqui, unless the LLM is on the “older” and/or “smaller” side of things… but still worth making sure the LLM output was correct of course, and that is an issue, just has not been so far during development (which of course is very different from when the Assist harness is mature, ie hopefully it won’t always be that way).

All that said, I haven’t tried asking it to do very complex things… that’s another interesting topic. For complex analysis sorts of things I think they could do pretty well but we may need more flexibility with certain data endpoints. For complex workflows, where the agent has to figure out which tools to use… I could imagine even an fancy frontier model needing a bunch of time in the sim to piece things together.

I don’t know, would be interesting to try, especially if any have enough corpus knowledge of the business side of Moqui to find things quickly. Chances are it’s going to have to use the ‘browse’ tool a whole bunch to find info about artifacts/endpoints/etc available, as well as any skills we might put together over time…

11

Part of how I’m approaching it is the “everybody is an agent” state of mind. We can’t really tell at this point whether a logged in user is an agent or not so, by corollary, we should make sure that an agent in our environment can do things the same way an agent with BrowserUse would use things. Then we are at least at parity. We should be able to provide a lot of useful services to the agents though.

Another thing I’m thinking through is how much of this skill data could also be used as user facing training if it is in Wikis. I’d like to try initializing agent runs with Wiki entries and then we already have an editing interface.

12

Hi David, you can find the branch for the open responses API here: GitHub - moqui-industrial/moqui-framework at open-responses · GitHub.

The extension implements the Specification standard.

13

The “Assist Screen” is now an “Assist Panel”, based on our conversation on the Moqui call on Friday. These changes include a new tool called ‘screen_use’ that is sort of like a browser-use tool, with restrictions for hopefully safer use like no clicking/selecting buttons that change data which is the only way it works right now but from some early feedback I think we’ll want a Settings toggle for that to allow it to go ham when people want it to. This one is a bit more fun than Assist to test, and in general to see working… navigating around, filling in forms, searching for stuff… kind of cool.