Hi, as you might know, currently moqui is not tracking artifact hits for REST paths, this is explicitly disabled in the code (RestApi class) with a comment: “for now don’t track/count artifact hits for REST path”.
We have some cases where we expose some API endpoints and would like to measure the response times, and with the latest changes in the moqui-metrics component this would be trivial to do using artifact hits (it is actually using a Entity ECA to count hits and durations).
Is there any reason not to enable it, and would it be preferable to disable some specific API calls (like /rest/s1/moiterp/session/userOrgInfo and /rest/s1/mantle/my/noticeCounts)?
1 Like
This sounds like a great idea thanks for bringing this up.
I would say that the main reason why Moqui doesn’t automatically track API endpoints by default is because if you look at almost any production Moqui install, the most used used tables are ArtifactHit and ArtifactHitBin and depending on your system you may not want to keep track.
There are a couple options:
a. keep the functionality as is, but allow for configuration to allow rest api endpoints / files to track rest endpoints
b. set an environment variable / configuration to allow keeping track rest api calls in the artifact hits
c. a combination of the two
I think I’d prefer a for upstream changes.
One pattern that might work well is to enable it but with configuration like others that are similar, ie a global setting for all REST paths in an attribute in the Moqui Conf XML file + an attribute for each service rest endpoint to disable individual ones (and yes, some should have stats tracking disabled OOTB like mantle/my/noticeCounts
1 Like
That sounds like a good plan to me.
I have been working a bit to get it going, and it works. Pull requests for Framework (allow tracking of artifact hits for rest paths (defaults to false) by jenshp · Pull Request #637 · moqui/moqui-framework · GitHub) and Runtime (allow tracking of artifact hits for rest paths (defaults to false) by jenshp · Pull Request #231 · moqui/moqui-runtime · GitHub) are created.
However, the idea of using the ArtifactHit as the basis for the metrics component is that it requires no intervention to the moqui-framework as it works based on an EECA on the ArtifactHit entity. But in order to have metrics on any artifact execution, a prerequisite is to activate persistence of ArtifactHits, which generates way more overhead than directly registering the artifact execution usage.
So, maybe we should add the metrics hit recording directly instead of relying on the EECA on the ArtifactHit entity, and allow to record metrics without the need to also record the Artifact Hits and so having a lot less of overhead. The moqui-metrics project already has a ToolFactory implemented, so we can add it dynamically, so when such a Tool is initialized, the method abstract void registerArtifactHit(String artifactType, String artifactName, BigDecimal runningTimeMillis, Character slowHit, String wasError) is called for each hit. This change would need to be done in the Framework, specifically where the ExecutionContextFactoryImpl.countArtifactHit method is called.
1 Like
@jenshp I looked at your code in moqui-metrics, moqui-framework, and moqui-runtime, and it would be great to have some documentation / seed data on how to get metrics in grafana for a specific rest api or screen tree.
After looking at your code not requiring the ArtifactHit entity to track metrics makes a lot of sense. Especially because ArtifactHit entities are the most used entities in any production install.
One potential fallback is if you want to record the Artifact Hits over time, then storing it in a persistent storage would be required. I’m not sure if Prometheus would be considered a persistent storage.
Thanks @michael for your feedback, I added a demo data file and referenced it in the README: GitHub - Moitcl/moqui-metrics: Metrics for moqui to use by external monitoring
As for the need to persist metrics information on the moqui side, that will not be necessary. The way the prometheus metrics are specified is using counters so there is no need to keep track of what data has already been handed to the prometheus server, just keep the total counts from the instance startup, and only for the local moqui instance. The prometheus server monitors each instance, making any necessary aggregations. It also identifies and handles resets correctly, so any persistence is handled at the prometheus side.
1 Like
That’s very helpful thank you
Okay it’s good to know that prometheus will handle persistence.
Also AWS by default will track some metrics on ec2 / ecs etc.
I’m just thinking it may be necessary for those who don’t have prometheus or AWS, but allowing the metrics to be handled by an external service should be allowed
As a side note, I’d like to try out grafana / prometheus in a dev environment. Do you have any docker commands / a docker compose that sets everything up?