Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend parameter annotatedFor on GET /mappings #176

Closed
stefandesu opened this issue Jun 27, 2022 · 2 comments
Closed

Extend parameter annotatedFor on GET /mappings #176

stefandesu opened this issue Jun 27, 2022 · 2 comments

Comments

@stefandesu
Copy link
Member

We need some additional values for parameter annotatedFor on GET /mappings:

  • any: has any kind of annotation
  • none: has no annotations at all
  • negative operator with !, e.g. !assessing: does not have annotations with that motivation

Required for gbv/cocoda#605.

stefandesu added a commit that referenced this issue Jun 27, 2022
Including tests and documentation.
@stefandesu
Copy link
Member Author

Unfortunately, there are big performance issues with the new negative assertions (none and !xyz) because it always has to go through the whole result set. There are actually two issues:

  1. Sorting the results takes very long. I was able to mitigate this by moving the sorting process to a different place in the query pipeline, so it's not a problem.
  2. Counting the total results of the query is actually the bigger issue for the above additions. With a big result set, it takes 30+ seconds to count the results because it has to load everything from the drive.

My current workaround for 2 is to skip counting the values and return an unreasonably big number for X-Total-Count (like 9 999 999). Not returning any total count breaks our client applications (and I couldn't find a convention to say "there are more values, but we don't know how many"), and as far as I can see, there is no disadvantage to this except that it's a fake value. I think if this is documented properly, it should be okay.

@nichtich Do you have an opinion on this?

stefandesu added a commit that referenced this issue Jun 28, 2022
Found a more efficient way of counting the total number of results for negative annotation assertion queries.
@stefandesu
Copy link
Member Author

I was able to find a different solution for counting the total results for negative assertions for annotatedFor: Count the results for the same query without annotatedFor, then subtract the results from the same query but with opposite annotatedFor. The code changes for this were a little difficult, but it seems to work very well. Closing this issue now!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant