How to Handle Client Interactivity in Moqui (a discussion)

Technical Discussion
1

When the /vapps and /qapps screen render modes were made, the main objective of them was to allow client reactivity in Moqui. Actually creating these render modes was a large challenge, and contribute to a large fraction of Moqui’s complexity. As I’m migrating from vue 2 to vue 3, I’m contemplating that there might be a better way to handle client reactivity without requiring a fancy javascript framework.

Htmx

One potential is a library that has had a recent surge of interest from the web community in back end apis. Tech twitter seems to be completely enamored with htmx which reached #2 in the 2023 JavaScript Rising Stars “Front-end Frameworks” category.

Htmx allows you to add attributes to any html element to query a server for html. This way your templates can be rendered on the server without having to replicate state and logic on both the client and the server. It has a simple api that allows for reactivity in web applications.

The example that made me realize why this is so helpful is the hx-swap attribute. If you have a menu bar to the side, and an screen rendered to the right. The menu bar is a list of elements like

  <div id="menu">
    <button hx-get="/order" hx-target="#response-div" hx-swap="innerHTML">Go to Orders</div>
    <button hx-get="/party" hx-target="#response-div" hx-swap="innerHTML">Go to Parties</div>
  </div>
  <div id="response-div">Choose a Screen</div>

Then when you click on “Go to Orders” the /order endpoint returns html and the #response-div gets replaced.

There are many more examples for this here.

Ruby on Rails

As I’ve thought about the web framework part of Moqui, I thought that looking at how other web frameworks handle javascript, client side interactivity, and mobile would be helpful.

Ruby on rails is similar to Moqui in that it uses templates to inject html with data. A good example of this is the rails Action View Form Helpers. Where it differs from Moqui is generating templates within templates. In rails, it is fairly trivial to have a template within a template that can be reused in multiple templates. (While this is possible in freemarker, the default in Moqui is to not edit the freemarker files for screens.)

Javascript Bundling

Ruby on rails has been shipping different ways to handle javascript for web apps. In rails 7, there are 3 ways to handle javascript.

One is import mapping (most similar to Moqui), which is a cli application that allows you to add es6 native esm modules to your application javascript through a cdn. This requires no js runtime at all and uses HTTP/2 multiplexing on queries to have minimal overhead when serving many files. This also allows for a typical import { createApp, ref } from 'vue' type syntax. Import mapping does not have any transpilation and uses browser native technology (with polyfills) to have a great experience on all actively supported browsers (after Microsoft finally dropped support for IE 11).

jsbundling-rails supports typical js tools by using a js runtime like node / bun to transpile and bundle using common build tools like rollup, esbuild, webpack, and bun. Then, the output gets directly added to the rails asset pipeline that will transform images, javascript, and other typically static files to the final product suitable for production use in rails. This way the latest node ecosystem features can be used like typescript while still having one server to deploy the app onto. This kind of functionality seems like a promising short term feature in Moqui to get javascript ecosystem developers developing a UI in Moqui.

The last option is to just use Rails as an API, and have a completely seperate server running all the client logic in any front end framework like Next.js or flutter. This is what most custom UIs in Moqui are currently doing.

Hotwire

Right now in Moqui, if you want to develop a mobile application using a similar UI, you would need to port all the components, ftl template logic, and screens to a Quasar CLI application. This is quite unfeasable, and is probably not worth using any existing Moqui screens for a custom or existing app.

There is another way that mobile could be done using the existing Moqui logic (with some hacking ofc). This would be implementing something like hotwire. Hotwire is a set of libraries that extend rails to stream HTML to the client instead of JSON. Hotwire consists of 3 libraries: turbo, stimulus, and strada.

Turbo has 4 parts: Turbo Drive, Turbo Frames, Turbo Streams, and Turbo Native. Turbo Drive and Streams together are similar to htmx, they allows for specific dom updates on javascript events for full or partial screen updates over http or websocket using html element parameters and minimal javascript. What Turbo does differently is Turbo Native enables for hybrid applications for iOS and Android out of the box. Turbo Frames allows for a certain screens to have an independent context which can be used similarly to Vue components. That isn’t nearly enough to go over everything in Turbo, but that should give you a good enough idea.

Turbo only handles 80% of the uses cases. Stimulus is a simple javascript framework for treating html output like ruby controllers for when server side javascript doesn’t quite cut it.

Strada the bridge between web components and native components in hotwire.

Overall hotwire is an interesting approach and could be used to stream html down to mobile applications.

2 Likes
2

You hit a raw nerve for us. Indeed the biggest challenge to us is customizing the “dynamic” behavior of /vapps and /qapps. I like the research you’ve done on the topic and htmx sounds like one of the best contenders to introduce as it simplifies things.

I want to attack this topic from a different angle though. Let’s take a step back and look at the architecture of moqui UI. What we have right now is a) xml-widgets → b) parsed node → c) XML-based freemarker → d) Final output of each render mode.

One issue that results from this design is that it becomes difficult to reuse macro templates in runtime/template/screen-macro. They are specifically designed to transform a complex MNode coming from parsed widgets to the render-mode that is desired. It’s just doing too much and it becomes difficult to modify or extend this stuff because it is mixing the tree traversal with the final UI output.

What if instead, we have sort of two layers of these macros, one layer is just pure UI logic, it’s a function that takes some input, does stuff, produces output. We can have a library of these things that are as granular as possible. The second layer is where the “composing” and “tree-traversal” happens against the parsed node coming from the XML widgets.

I would imagine if we attack these templates directly and break down the big structure into small reusable pieces, then it would be much easier to reuse and customize (I can just call the macros directly). This in turn makes it easier and more managable to introduce dynamic behavior mixed with XML widgets and can operate on different render-modes.

Wouldn’t fixing anything then in any render-mode be easier? Because the source macros that generated them are simpler? WDYT?

1 Like
3

Do you have a some context for this? I’m not following how it’s difficult to reuse the screen-macros.

I may not understand what you’re saying enough to really comment on it. Can you give an example of what it currently looks like and how this would improve things?

This might be naive, but thinking about this from first principals. What needs to happen is data needs to be fetched from the database, and displayed to the user. This is currently done in html and css with some js for interactivity and dynamic functionality based on the dom eventing model. The data from the database needs to be injected into the html. Ideally this uses a templating language rendered in the server and/or client depending on the model.

Right now we have high level xml widgets (which are very convenient) that render down into freemarker to inject data into vue components (typically passed through a custom wrapped Vue component’s parameters. Then they get rendered by the vue compiler in the browser to html.

4

Hi @michael , yeah I think it’s my bad for not clarifying. I will lay down a few comments and assumptions to make sure that I’m following your points as I answer your questions:

  1. We like XML widgets, they save a lot of effort and time
  2. We like dynamic interactive systems with continuous request / response cycles in the page.
  3. Dynamic behavior is DOM manipulation with JavaScript (like /apps) or using JavaScript component frameworks (like /vapps and /qapps)
  4. The current render-modes employing dynamic behavior are challenging to build and maintain as you mentioned:
  1. Customizing screens inside these render modes is also a challenge, especially if you want to mix widgets with your own custom interactive code.

So maybe an important question now is: is the complexity arising from being a JavaScript framework or is it the interactions between freemarker and these frameworks? In other words, if you’re just developing a website on quasar directly, would it be equally challenging to implement?

If the answer is no, then maybe part of the problem is that the interaction is split between javascript (WebrootVue.qvt.js for example) And freemarker templates all over the place including the root WebrootVue.qvt.ftl and in each screen AND also in the macros.

What the macros are doing, is receiving a big .node that holds the structure of the screen, and then translating it into freemarker XML macros. This can be recursive whenever you have sub-screens, form-fields, and so on. The components that we reach eventually like those in WebrootVue.qvt.js are big, and they take many arguments that fulfill the behavior we’re trying to achieve. In addition these components have additional complexities from receiving dynamically some template snippets (slots).

The above design poses a few challenges. Let’s say I have a 1,000 line XML screen, and in the middle of that screen I want to create a custom drop-down, such that whenever its value changes, I want to fire an event, or I want to popup a dialog. This stuff is already implemented in moqui, it’s just not implemented for the drop-down, or not on value-change for that drop-down. My options at this point become limited because I cannot call anything in the screen macros, I have to either do the data preparation myself and then call WebrootVue myself, OR I make the whole screen dynamic and reimplement the 1000 lines just because I cannot add the dynamic behavior I want. Now if the macros are callable directly, the first problem of reusing freemarker from custom render-modes becomes easier, I just call the available macro, it’s not expecting a .node, it’s just expecting some arguments.

Another problem to tackle is the maintenance of the theme system and the framework UI as a whole. The puzzle of knowing where things are coming from and how they interact becomes overwhelming. You saw a bug, where is this bug coming from? is it WebrootVue, is it the dynamic template it’s calling with v-slot coming from the freemarker template? Is it from the root template WebrootVue.qvt.ftl. Is it a mistake in the way I’m calling these big components from freemarker.

This leads me to think why not make Vue components smaller, the freemarker macros that call it are smaller, and the composability happens in higher level freemarker and/or components that eventually touches the .node coming from the screen. And allow for injecting code or applying something like component wrapper patterns to help compose these small freemarker macros and components into bigger ones, such that each one can be more isolated in testing and designing it.

I mean I could be wrong about all of this or I might be missing the point. I’m just sharing things out loud to try and think through a way to make it easier to deal with the intricacies of the UI part of the framework.

1 Like
5

I meant to respond to this earlier.

I would like to see an example of this, and figure out if it is solvable. A couple ideas are using the <render-mode> element or creating a custom element by extending a WebrootVue.x.ftl in a component (as done here).

I believe we’ve made some attempts to help with this. In dev mode, you can see where the screen was rendered in the comments like <!-- BEGIN screen[@location=component://coarchy/screen/settings/settings.xml].widgets -->. We could enable a mode that has more granular comments. Another way to solve this would be creating dev tools that like Vue Devtools that understands Moqui components. I’m not a huge fan of this because it is using a complicated tool to mitigate complicated code instead of fixing the complicated code.

To get a better solution to that problem, I’d need to have it. If you can share an example that would be great. @jonesde Any ideas?

Would making the Vue components smaller fix the above problem with maintenance of the theme system?

I think I’m understanding more of what you mean by smaller components. Smaller components are a design to fix the problem of composability and .node assumptions that are inherently built in to the architecture of the freemarker templates.

6

Hello @michael and thank you for following up on all of this! OK let me try to take it apart.

Let’s do a simple example that I actually experienced:

  • I have a form to create something (maybe OrderItem or RequestItem or something)
  • I have fields inside that form, one of these fields is the product drop-down (productId)
  • When I type in that field, I might not find matching results (no product with this ID or identifier, etc …)
  • If no results are matched, I would like to show a button inside or next to the drop-down that says “create-new”.
  • When I click on that button, I get a popup dialog with some form to fill product information
  • When I submit this information, I should be able to create a product and fill the drop-down automatically with this new product’s ID
  • If I fail in creating the product for whatever reasons, I should show proper errors and whatever else is common behavior in moqui.

Mind you in this simple example I might have a massive screen, and I just want to modify this drop-down, I don’t want to touch anything else or redo the hard work again of rebuilding this screen. So what are my options, Can I reuse any of the following:

  • Freemarker macros? No, they are designed to process a large .node tree structure and traverse it and then eventually calling WebrootVue
  • WebrootVue? No, these components are designed again for specific XML widgets. They’re not general purpose fields that I can reuse in different ways.
  • Render-mode? No, I cannot use it without first creating my own custom vue components and loading them into the theme. And even then, it means everything that I want to develop (the popup, the form and fields inside that form, the event listening) I have to write from scratch. There is nothing to reuse here and I have to add my own component definitions into the theme (either override root screens or add javascript files defining my components into moqui.screen.ScreenThemeResource)
  • Extend the widget-system? Well … maybe but it would be quite difficult, I have to create a new widget and map it to this very custom behavior that I described in my above example. So I will create a new widget-type maybe call it <drop-down-dialog .../> or <drop-down-popup .../> and implement inside the full behavior including traversing the rest of the node. But then this won’t be generic, it would only work for products and only products that are created in a certain way, so maybe it should be <drop-down-quick-add-product... />.

Well it might help if not only do they get smaller, but also not married to the widget system. We should differentiate for example between a general-purpose drop-down field with event listening and other dynamic behavior and moqui-specific drop-downs that come from the XML widgets. They are two entirely different things. Now the moqui-specific one can be implemented using the more generic one.

Why am I asking for a generic one? Well, not only can you call it directly with different use cases. BUT you can also expose a freemarker macro that communicates with the DB and whatnot and feed into that component in a way that I can call from any render-mode. This freemarker is immediately callable, not expecting a .node structure, something like the below

<#macro renderDropdown name, options>
<your-simple-component-here name="${name}" options="options">
    <maybe-even-compose-stuff .../>
</your-simple-component-here>
</#macro>

With this approach it’s possible that we strike two birds with one stone: simplify the implementation (your original concern which started this topic) AND allow for custom development without pain. The main change that has to happen is maybe to remove the 1-1 relationship between components and XML widgets, and instead break it down to layers the lower ones being more generic and reusable, and the higher ones being more tied deeply with “moqui” and the big nested .node structure.

There will always be a shiny new UI framework, today it’s react or vue, tomomrrow htmx, the day after who knows. We also as a community have to maintain all this stuff even if we introduce solutions like htmx (which is quite appealing in its selling points). So I’m thinking maybe we need an architecture that is friendly to all these render-modes by maybe having this layered approach.

Oh and one final point to make, this approach could actually also allow for code sharing between the different render modes where it makes sense.

Again I could be wrong about this but is it clearer? Good / bad idea? WDYT?

1 Like
7

Let’s talk about this on the call this week. I think the hard part isn’t the ftl. It’s the javascript, and the assumptions that are made about custom components.

After I made some changes, this is making more sense. Although you would probably want to do something like this instead.

<#macro renderDropdown name, options>
<your-simple-component-here name="${name}" options="options">
    <#recurese/>
</your-simple-component-here>
</#macro>

I’m glad you agree. It could be totally not worth it, but I find it appealing. Especially using htmx with a back-end oriented framework like Moqui.

That could make things a lot better.

This is very much clearer. I’m starting to understand the problem more now, but not enough to understand how your solution fixes the problem.

During this weeks call I’d like to go over what I did, and the problems that I ran into. Hopefully that conversation will clarify things.

8

Hi @michael great I look forward to having our chat tonight. I’ll make some minor comments here:

First regarding your video, the product that you created is not automatically populating the drop-down, which is a critical step. you’ve essentially created two separate fields and you have to go to the other field and manually search again and populate. The whole idea of create new is to make it quick to enter if missing. This is the exact requirement that we got from our client and had to go through a lot of trouble to implement. You got part of the needed behavior though in that you show the button only when product is missing.

Well … that’s what I meant by higher level calls. You won’t recurse in these lower level simple macros. These are things that can be rendererd immediately, they do not represent a tree structure and a .node, they’re just drawable things that also might interact with the database or context or whatever. Or at least that’s how I’m imagining it for now.

I find this discussion enjoyable overall, I think we might find some root solutions to some of the UI issues we’re facing.

1 Like
9

For what it’s worth, here’s the patch of my changes to simple screens as an example: Copied vue and ftl components to allow overriding default behavior · GitHub.

1 Like
10

Great thank you for sharing @michael!

I’m not sure why the macro-template element exists? I don’t see any CustomScreenMacros.qvt.ftl. I am assuming that maybe it’s also copied from default macros with your override but it’s just not showing in this patch?

1 Like
11

@michael and @taher I assume this is related to the conversation we had on the Moqui Community call on Feb 16.
This may be oversimplifying but as @jonesde suggested on our call, the easiest thing to do under the current Moqui UI architecture is to just drop into a .qvue screen. This gives you all the control you need (I think?) while also still working within and leveraging the Moqui framework for component structure, common headers, footers, menus, etc.
I shared the examples we have bulilt here:

More broadly it sounds like worth considering some refactoring / modernizing / simplification of the overall UI framework. I agree with @taher it can be tricky to figure out where something is happening and how to get control over certain things.
Ultimately the service and entity engine do a lot of the heavy lifting so it was pretty straight forward for us to build the Sales and Work Management components entirely using Quasar/Vue.
Maybe the best approach to refactoring would be to make it as easy as possible for front end developers to use their tool of choice. That can be done today if you define everything you need as a REST service and treat Moqui entirely as a headless backend. This requires more work for the front end developer but also gives them complete control.
The way we used .qvue screens was a nice balance of getting control while not having to build the entire UI from scratch but it requires that you use Quasar/Vue. Given the widespread adoption of React it would be nice to open things up to that community. My personal opinion is that the best way to do this is to focus more on defining APIs along with some documentation / tutuorials on “how to build apps on Moqui”.

3 Likes
12

The macro template was copied from the example component. CustomScreenMacros.qvt.ftl overrides the DefaultScreenMacros.qvt.ftl and includes the DefaultScreenMacros.qvt.ftl for that screen.

I missed the CustomScreenMacros.qvt.ftl file in the diff. It should make more sense with the recent change.

1 Like
13

Thank you @michael this example now makes much more sense.

The good thing about this example is that it works and allows mixing XML and custom vue which is great. The downside of this approach is that you copied pretty much all of the drop-down vue component to only add a few new lines and the same for the freemarker macro. So we have duplicate code that does not benefit from bug fixes and future improvements.

As was already discussed in the community call, I suppose one way to resolve this copy-paste pattern is to try and contribute useful work back to the community.

As for @vince.clark custom code, first of all, this is quite useful thank you for sharing. The problem that I was originally trying to tackle though is what if you have substantial large XML screens that you already built? Rebuilding this from scratch with hand-rolled qvue templates is a lot of work and you lose all the power and productivity gains from being able to quickly develop UI. XML widgets are the fastest way to get UI in the hands of clients. Take for example your qvue file, that’s 1,176 lines of what seems to be hard tested work with with not only lots of UI code but significant communication code (ajax calls) and other business logic all of which is hand rolled. That’s what I’m trying to avoid unless I “Have To”. In other words, this should preferably be the exception rather than the norm.

So what I learned from all of this is that the best way to resolve UI is to contribute more to the macro templates and their corresponding vue components, followed by refactoring and breaking down where it makes sense.

2 Likes
14

@taher for the specific use case you are trying to solve would a combination of <dynamic-dialog> and <drop-down> with <dynamic-options> address this? You can use the dialog to add a new value and then the drop down will populate dynamically. I haven’t tried it. Would require that the transition used to add the new value in the dynamic dialog NOT reload the page. I think there is a way to do this. Maybe <default-response type="none">

1 Like
15

Hi @vince.clark

So yes I’ve tried the dynamic dialog, which essentially renders a separate screen in your screen and refreshes it dynamically. It has limited use cases and definitely does not cover the example that I have indicated here. The only two solutions that seem to be viable are what @michael showed in his nicely done example, or going fully client-rendered like the heavily dynamic components you shared (thank you again)

1 Like
16

Good. Sorry about that.

Yeah this wouldn’t be too hard with the slot pattern in vue, and in FTL we can do some extensions using the visit pattern.

I personally find the text element to be quite nice for ftl template injection and vue as done here (vue injection) and here (ftl conditionals). I just realized this was possible today thanks to @marwand. With those three options, for different use cases I think we’re doing pretty solid as far as customizing Moqui UIs.

1 Like