We are working on feed generation capabilities with Moqui, for this the feed files can have large data set, at-least or more than 10k records per file.
Is there a recommended way to prepare and store such large files in Moqui for this use case?
On analysis of some existing references in Moqui, I could find that we prepare the file in-memory and then can store it using Resource Facade capabilities.
Has anyone worked on a similar scenario?
We have worked with relatively big files in some cases (a little bit under 100 MB), and the main problem of using the Database to store the files is that you need to able to handle the full file in-memory, at least when using the standard Resource Facade methods (sometimes several times the size of the file, depending on how you handle and process it, although this is more when you generate the file based on e.g. a screen). This sometimes leads to very high usage of memory, so when this becomes an issue, we have switched to use files stored in directories. These can be handled using the Resource Facade, we have a directory mounted inside the runtime directory so we use a “runtime://…” contentLocation string and this does not have the limitation of having to load the file in-memory to send it through the web facade and such.
Then issue with using files is that you have to make sure that all moqui instances have access to the files, so you might need to set up some file sharing or use the cloud-provider-specific file storage solution.
Otherwise, storing big files in the DB has an impact on the backup process of the DB, and while having to backup DB + files means you need to setup two separate processes, you have the advantage that each process handles its particular task more efficiently than having one-size-fits-all solution.
Hi Jens,
Thank you for sharing these details.
For now, we also proceeded with the approach to use file system to store such large files.
We have the mounted directory as “runtime://datamanager” in which all the feed files are generated.
But we are facing an issue when Moqui needs to be re-deployed or new war is build and deployed using Tomcat. During re-deployment the symlink is causing issue with unlinking and linking the directory as the ROOT folder gets deleted during deployment.
Can you share some insights if you faced a similar issue or how can this handled?
We are using containers to manage all of our solutions, so in our case, the environment makes the setup before starting the container (with the OOTB jetty server). We have used this with docker, docker-compose, docker-swarm and mainly kubernetes.
And we are not going back to using an application server like tomcat anytime soon, it is so much more manageable, clean and flexible with containers.