Sort with search#DataDocuments service orderByFields field

I’ve got an error in Open Search using the search#DataDocuments’s orderByFields field. It says:

  "error": {
    "root_cause": [
        "type": "illegal_argument_exception",
        "reason": "Text fields are not optimised for operations that require 
per-document field data like aggregations and sorting, so these operations 
are disabled by default. Please use a keyword field instead. Alternatively, 
set fielddata=true on [categoryName] in order to load field data by 
uninverting the inverted index. Note that this can use significant memory."
    "status": 400

I’m wondering if anyone has encountered this before, and if they have any ideas on how to fix it. I believe that this error has something to do with aggregations needing a specific format to untokenize the fields to their raw text values.

One interesting thing about this is that sorting works with what I can tell primary key fields, but not others. Is there a way to change this functionality? I’m not seeing anything in the Data Search Documentation or the data model.

I was able to replicate this problem in Elastic Search also. To replicate, go to an elastic or open search instance with a tool like Elastic View. Navigate to the mantle_product index and add a string productId to the sort query parameter in Custom Search:

Then if you replace productId with name and search, then it should look like this:

This can also be done with code.

The data types for text in ElasticSearch are funny, optimized for search (tokenized) or for lookup and sort. For more info see the sortable field on the DataDocumentField entity:

This is used by the code that sends JSON document schema info in what ElasticSearch calls a ‘mapping’. This code is in the method ElasticFacadeImpl.makeElasticSearchMapping()

What this does is add a sub-field called ‘keyword’ of type keyword and then in search code sort by the categoryName.keyword field instead of the non-sortable categoryName field (tokenized for search).

For one example of code that handles this see the orderByFields field in the search#Party service:

1 Like

That makes a ton of sense. I guess I was missing the sortable field. I could’ve sworn I’ve looked for that before.