Don't create a search/pulldata database when pulldata is used over an Entity List

[lognaturel]

Following #6451, pulldata over an Entity List uses the local Entity List database. However, the corresponding search/pulldata database is still being created. This increases form open time for users and leads to side effects like #6461

Currently search() does not use the Entity List database so knowing that a source CSV is for an Entity List is not a sufficient signal to skip search/pulldata database creation.

We should fix this by skipping the creation of the dynamic preload data DB if a form meets the follow two conditions:

• Does not include search
• If the form uses pulldata, it's only on entity lists

Notes

The big change we'll need to make for that is to persist the "flag" that an instance (media file) is an entity list so that can be determined when parsing a form. There are a lot of other reasons that storing details about media files could be useful though, so I think that's a good change to make here.

[seadowg]

I'm thinking we should fix this by skipping the creation of the dynamic preload data DB if a form meets the follow two conditions:

• Does not include search
• If the uses pulldata, it's only on entity lists

The big change we'll need to make for that is to persist the "flag" that an instance (media file) is an entity list so that can be determined when parsing a form. There are a lot of other reasons that storing details about media files could be useful though, so I think that's a good change to make here.

Very related to this is how we then deal with search which, as @lognaturel points out, does not currently support local entities. I'm thinking we can approach it like this:

• Introduce a new XLSForm sugar to eventually replace search that pyxform translates to a normal instance('blah)...` expression while continuing to optimize different underlying expressions for entities (Optimize more entity filter expressions #6498)
• Deprecate search
• Rework downloads to build confidence in processing entities (and eventually secondary instances)
• Optimize secondary instances predicates (Optimise external secondary instance memory use #6454)
• Remove search support from Collect

@lognaturel what are your thoughts on this plan? I think it's good to have a long term plan so we know the context surrounding fixing this issue.

[lognaturel]

The short-term plan sounds good and I agree that we will need to know whether a form attachment is an Entity List at a time other than download for other reasons, most notably for #6425. I think that issue is higher priority than this one so maybe it can drive out storing that state.

Introduce a new XLSForm sugar to eventually replace search that pyxform translates to a normal instance('blah)...` expression

I think we already have this: select_one_from_file with a choice filter. And I think it has largely replaced search already because it's so much more comfortable to use. search is much less common than pulldata.

[seadowg]

The short-term plan sounds good and I agree that we will need to know whether a form attachment is an Entity List at a time other than download for other reasons, most notably for #6425. I think that issue is higher priority than this one so maybe it can drive out storing that state.

Agreed!

I think we already have this: select_one_from_file with a choice filter. And I think it has largely replaced search already because it's so much more comfortable to use. search is much less common than pulldata.

Great point! I was forgetting that choice_filter already removes the need to deal with instance('blah')/root/item. There are other places where you do need to resort to that syntax, but search wouldn't help you there anyway.

What are your thoughts on deprecating search?

[seadowg]

What are your thoughts on deprecating search?

We discussed this, and there are few things that we'd need to support more generally before getting rid of search:

• search supports multiple languages
• search supports showing media in select options
• search allows specifying some static choices (like Other, Not Applicable) at the bottom of dynamic choices. We could do something like use the | operator to support specifying a join of multiple nodesets as the choice list

[seadowg]

@lognaturel just checking for my own sanity here: we don't actually need to do this if we do #6552 in the same release right?

[lognaturel]

That's right, we could potentially go straight to #6552.