Implement filtering of some endpoints

[mkeeler]

Fixes: #4222

Data Filtering

This PR will implement filtering for the following endpoints:

Supported HTTP Endpoints

• /agent/checks
• /agent/services
• /catalog/nodes
• /catalog/service/:service
• /catalog/connect/:service
• /catalog/node/:node
• /health/node/:node
• /health/checks/:service
• /health/service/:service
• /health/connect/:service
• /health/state/:state
• /internal/ui/nodes
• /internal/ui/services

More can be added going forward and any endpoint which is used to list some data is a good candidate.

Usage

When using the HTTP API a filter query parameter can be used to pass a filter expression to Consul. Filter Expressions take the general form of:

<selector> == <value>
<selector> != <value>
<value> in <selector>
<value> not in <selector>
<selector> contains <value>
<selector> not contains <value>
<selector> is empty
<selector> is not empty
not <other expression>
<expression 1> and <expression 2>
<expression 1> or <expression 2>

Normal boolean logic and precedence is supported. All of the actual filtering and evaluation logic is coming from the go-bexpr library

Other changes

Adding the Internal.ServiceDump RPC endpoint. This will allow the UI to filter services better.

Remaining Tasks:

• Tests to verify that the fields for each type are what we want. These are autogenerated currently and it would be a good idea to capture what they currently are and ensure they remain the same. When they do get updated we will need to update documentation.
• Tests to verify that the filtering works (specifically for Consul). The go-bexpr library has lots of unit tests to verify that it does the correct thing already.
  • /agent/checks
  • /agent/services
  • /catalog/nodes
  • /catalog/service/:service
  • /catalog/connect/:service
  • /catalog/node/:node
  • /health/node/:node
  • /health/checks/:service
  • /health/service/:service
  • /health/connect/:service
  • /health/state/:state
  • /internal/ui/nodes
  • /internal/ui/services
  • Catalog.ListNodes RPC
  • Catalog.ServiceNodes RPC
  • Catalog.NodeServices RPC
  • Health.NodeChecks RPC
  • Health.ServiceChecks RPC
  • Health.ServiceNodes RPC
  • Health.ChecksInState RPC
  • Internal.NodeDump RPC
  • Internal.ServiceDump RPC
• Consider using a pre-parsed filter AST to send with the RPC instead of the raw string. This would have an advantage when caching RPCs of normalizing the filter to hopefully help improve cache hit frequency. Also it may reduce some cpu usage on the servers.
• Update and test API package.
  • Update Agent API & Tests
  • Update Catalog API & Tests
  • Update Health API & Tests
• Update and test consul catalog cli.
• Update the API docs.
• Cut a release of go-bexpr and update the mod vendoring
• Any other documentation changes?
• Make HTTP Options request dump a JSON payload of the allowed fields and types (Optional) Not going to happen for the initial release.
• Determine what to do with the agent caching.

[varnson]

It looks cool, I like it.

[pierresouchay]

That's a great and powerful feature, thank you!

I only read from my phone, so I probably missed it, but I did not see how you will handle cached results (since I don't think the filter is used by the key used to compute the hash of cached requests)

[mkeeler]

@pierresouchay You didn't miss it. I just haven't handled that yet. There are a couple of ways I am thinking about handling the cache.

Option 1, is to just make the filter part of the cache key. This will just do what is expected in all cases.

Option 2, would be to make unfiltered RPC requests to the servers and then execute the filter on the results returned from the cache. Multiple different queries that use the same RPC could then use the same raw results and the local agent could filter for each request independently. This would also have a benefit of moving some of the CPU load for the filtering from the servers to the edges. The downside is that if you have a ton of data coming back through that RPC the filtering will not reduce the bandwidth necessary to transmit the request.

I think for now Option 1 is where I am going to take this. We already take the tag/meta filter into account for the cache key so it would make sense to do the same for this.

[pierresouchay]

Yes, even with crazy scenarii like I was trying to think of (several clients using cache with different filtering criteria), I think approach 1 is pragmatic for now and make sense regarding the existing codebase

[banks]

At a high level this is looking awesome @mkeeler! 🎉

I agree that adding filter to to the Cache request type makes total sense for now.

The TODO list also looks perfect - I'll hold off approving until more of that is done but this is looking great to me so far!

[rboyer]

Maybe it would be clearer to settle on either -G or -X GET but not a random mix of both in the examples provided here.

[mkeeler]

-G and -X GET are not functionally equivalent.

From the man pages:

For -G

              When used, this option will make all data specified with -d, --data, --data-binary or
              --data-urlencode  to  be used in an HTTP GET request instead of the POST request that
              otherwise would be used. The data will be appended to the URL with a '?' separator.

For -X

              This  option only changes the actual word used in the HTTP request, it does not alter
              the way curl behaves. So for example if you want to make a proper HEAD request, using
              -X HEAD will not suffice. You need to use the -I, --head option.

In general we could omit the -X GET as it will do a GET by default. However after talking with @kaitlincarter-hc the consensus was that to be as consistent with the rest of our docs we should have it in there. For all the queries where
--data-urlencode is used, curl would by default make it a POST request and put the data in the body of the request. Using -X GET makes it a GET request but the filter is then still transmitted in the body. Using -G forces curl to shove the filter in as a query param while still performing the urlencoding.

This seems like a deficiency with curl not being able to urlencode query params easier but its all we have to work with right now.

Also of note is that in all the cases where we are using -X GET we could use -G so we could standardize on that here. @kaitlincarter-hc what do you think?

[rboyer]

LGTM

[kaitlincart]

The docs turned out really well, nice work!