I’ve been playing around with embeddings in open search, and I think it’s a feature that could be added to Moqui. This will make search smarter, and it’ll be the start of Moqui being integrated with some AI stuff.
If people are interested enough in using this, I’ll put in the effort to make it a feature in Moqui. Otherwise I’ll just do my project.
If you’re interested, comment on here how you would use it.
Here’s how it would work:
Details
Open search has a ML pipeline for a way to setup certain fields on certain indexes that when they are uploaded, you use a sentence transformer model with the text to generate an embedding vector:
There would be a DataDocument would fields to store a index.knn / a default_pipeline and DataDocumentField that would have a special type that is not associated with a database field for the knn_vector type.
Then for search, the search string gets sent through a sentence transformer to an embedding vector, then a search between the existing embeddings and the search string embedding.
Then, we may need additional changes to the ElasticFacade.java line 77’s search method. To determine which search method to use.
If there’s enough need for it, additional entities can be added for models and pipelines.
We may also need screens for setting up an model / pipeline.