Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement filtering of some endpoints #5579

Merged
merged 62 commits into from
Apr 16, 2019
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
11e1411
Initial filtering implementation
mkeeler Mar 28, 2019
8a5652c
Merge branch 'master' of github.com:hashicorp/consul into filtering
mkeeler Mar 28, 2019
7db84d1
Update vendoring
mkeeler Mar 28, 2019
f17beca
Fixup go.sum
mkeeler Mar 28, 2019
43aa3de
Make the agent cache key take the filter into account
mkeeler Mar 29, 2019
f94fd7b
Merge remote-tracking branch 'origin/master' into filtering
mkeeler Mar 29, 2019
008e3ce
Pull in go-bexpr updates
mkeeler Mar 29, 2019
e312540
Implement tests to ensure the expected filter fields are correct.
mkeeler Mar 29, 2019
cec6e1f
Fix a ui endpoint test and add an /agent/checks filtering test
mkeeler Mar 29, 2019
43eec2c
Add /agent/services filtering test
mkeeler Mar 29, 2019
a2d172b
Add /catalog/nodes filter test
mkeeler Apr 1, 2019
e08274a
Implement a test for /catalog/service/:service
mkeeler Apr 1, 2019
edb77fe
Add test for /catalog/connect/:service filtering
mkeeler Apr 1, 2019
3f683f5
Add test for /catalog/node/:node filtering
mkeeler Apr 1, 2019
2d95577
Add /health/node/:node filtering test
mkeeler Apr 1, 2019
b1375da
tidy the go.sum file
mkeeler Apr 1, 2019
788dde0
Fixup comment
mkeeler Apr 1, 2019
5b6d418
Add test for /health/checks/:service filtering
mkeeler Apr 1, 2019
4665b6b
Add test for /health/service/:service filtering
mkeeler Apr 1, 2019
5a07080
Add test for /health/connect/:service filtering
mkeeler Apr 1, 2019
2c33792
Add test for /health/state/:state filtering
mkeeler Apr 1, 2019
e8ec84b
Add test for /ui/internal/nodes filtering
mkeeler Apr 1, 2019
6037b90
Add test for /ui/internal/services filtering
mkeeler Apr 1, 2019
d3af587
Add Agent.(Checks|Services)WithFilter functions
mkeeler Apr 1, 2019
1cf25c0
Add filtering to catalog API
mkeeler Apr 1, 2019
3565d9d
Get rid of the retry.Run in the new catalog API tests
mkeeler Apr 2, 2019
fd9573c
Add filtering to the health API
mkeeler Apr 2, 2019
f288c4d
Add filter to consul catalog nodes command
mkeeler Apr 2, 2019
6317e92
Implement filtering test for the Catalog RPC endpoints.
mkeeler Apr 2, 2019
88163c3
Merge remote-tracking branch 'origin/master' into filtering
mkeeler Apr 2, 2019
aef945b
Update API vendoring
mkeeler Apr 2, 2019
131d49b
Add Health RPC filtering tests
mkeeler Apr 3, 2019
2ee4145
Implement filtering tests for internal RPC endpoints
mkeeler Apr 3, 2019
46d0cc8
Update go-bexpr dependency to v0.1.0
mkeeler Apr 3, 2019
609568a
Fix accidental modification
mkeeler Apr 3, 2019
ef51dce
API endpoint filtering documentation.
mkeeler Apr 3, 2019
a3e1719
First pass at top level filtering docs
mkeeler Apr 4, 2019
c8bdc82
Add some information about selectors and values.
mkeeler Apr 5, 2019
58c222b
Add test for the returned data of the /agent/services endpoint for fi…
mkeeler Apr 5, 2019
31755e0
Update docs to be more explicit about which part of the data is getti…
mkeeler Apr 5, 2019
716807b
Fix a couple of typos
mkeeler Apr 5, 2019
bcff054
Revendor the api :(
mkeeler Apr 5, 2019
a8f0aea
Support passing the filter from stdin or a file as well.
mkeeler Apr 5, 2019
8dcd499
Update catalog nodes command docs
mkeeler Apr 5, 2019
fe7ccde
Fix typo
mkeeler Apr 5, 2019
c981ff0
Updated the structure of the API site. Still working on the filtering…
kaitlincart Apr 5, 2019
9a3cf01
Merge branch 'filtering' of github.com:hashicorp/consul into filtering
kaitlincart Apr 5, 2019
2ca5876
Remove filter parsing from endpoint we are not enabling it for yet.
mkeeler Apr 9, 2019
fafe6e7
Merge branch 'filtering' of github.com:hashicorp/consul into filtering
mkeeler Apr 9, 2019
17eb199
updating the language
kaitlincart Apr 9, 2019
30eabd6
Merge branch 'filtering' of github.com:hashicorp/consul into filtering
kaitlincart Apr 9, 2019
da76f9c
adding example sections
kaitlincart Apr 9, 2019
6933a9b
Some filtering doc updates
mkeeler Apr 11, 2019
8d93781
Some formatting updates
mkeeler Apr 11, 2019
00c298f
Minor doc update
mkeeler Apr 11, 2019
24ed818
Wording update
mkeeler Apr 11, 2019
d2e6851
Update website/source/api/catalog.html.md
rboyer Apr 15, 2019
051e3ec
Update website/source/api/features/filtering.html.md
rboyer Apr 15, 2019
e0ddb55
Ride the Filter within QueryOptions both for RPCs and in the API pakage
mkeeler Apr 15, 2019
1136dae
Merge branch 'filtering' of github.com:hashicorp/consul into filtering
mkeeler Apr 15, 2019
4ad9824
Move filter parsing into generic parse function.
mkeeler Apr 15, 2019
ae04743
Fix some tests
mkeeler Apr 16, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions agent/agent_endpoint.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import (
"github.com/hashicorp/consul/lib/file"
"github.com/hashicorp/consul/logger"
"github.com/hashicorp/consul/types"
bexpr "github.com/hashicorp/go-bexpr"
"github.com/hashicorp/logutils"
"github.com/hashicorp/serf/coordinate"
"github.com/hashicorp/serf/serf"
Expand Down Expand Up @@ -219,6 +220,9 @@ func (s *HTTPServer) AgentServices(resp http.ResponseWriter, req *http.Request)
var token string
s.parseToken(req, &token)

var filterExpression string
s.parseFilter(req, &filterExpression)

services := s.agent.State.Services()
if err := s.agent.filterServices(token, &services); err != nil {
return nil, err
Expand All @@ -238,7 +242,12 @@ func (s *HTTPServer) AgentServices(resp http.ResponseWriter, req *http.Request)
agentSvcs[id] = &agentService
}

return agentSvcs, nil
filter, err := bexpr.CreateFilter(filterExpression, nil, agentSvcs)
if err != nil {
return nil, err
}

return filter.Execute(agentSvcs)
}

// GET /v1/agent/service/:service_id
Expand Down Expand Up @@ -403,6 +412,13 @@ func (s *HTTPServer) AgentChecks(resp http.ResponseWriter, req *http.Request) (i
var token string
s.parseToken(req, &token)

var filterExpression string
s.parseFilter(req, &filterExpression)
filter, err := bexpr.CreateFilter(filterExpression, nil, nil)
if err != nil {
return nil, err
}

checks := s.agent.State.Checks()
if err := s.agent.filterChecks(token, &checks); err != nil {
return nil, err
Expand All @@ -417,7 +433,7 @@ func (s *HTTPServer) AgentChecks(resp http.ResponseWriter, req *http.Request) (i
}
}

return checks, nil
return filter.Execute(checks)
}

func (s *HTTPServer) AgentMembers(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
Expand Down
76 changes: 75 additions & 1 deletion agent/agent_endpoint_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"io/ioutil"
"net/http"
"net/http/httptest"
"net/url"
"os"
"reflect"
"strings"
Expand All @@ -27,8 +28,8 @@ import (
"github.com/hashicorp/consul/api"
"github.com/hashicorp/consul/lib"
"github.com/hashicorp/consul/logger"
"github.com/hashicorp/consul/testrpc"
"github.com/hashicorp/consul/sdk/testutil/retry"
"github.com/hashicorp/consul/testrpc"
"github.com/hashicorp/consul/types"
"github.com/hashicorp/go-uuid"
"github.com/hashicorp/serf/serf"
Expand Down Expand Up @@ -101,6 +102,48 @@ func TestAgent_Services(t *testing.T) {
assert.Equal(t, prxy1.Upstreams.ToAPI(), val["mysql"].Connect.Proxy.Upstreams)
}

func TestAgent_ServicesFiltered(t *testing.T) {
t.Parallel()
a := NewTestAgent(t, t.Name(), "")
defer a.Shutdown()

testrpc.WaitForTestAgent(t, a.RPC, "dc1")
srv1 := &structs.NodeService{
ID: "mysql",
Service: "mysql",
Tags: []string{"master"},
Meta: map[string]string{
"foo": "bar",
},
Port: 5000,
}
require.NoError(t, a.State.AddService(srv1, ""))

// Add another service
srv2 := &structs.NodeService{
ID: "redis",
Service: "redis",
Tags: []string{"kv"},
Meta: map[string]string{
"foo": "bar",
},
Port: 1234,
}
require.NoError(t, a.State.AddService(srv2, ""))

req, _ := http.NewRequest("GET", "/v1/agent/services?filter="+url.QueryEscape("foo in Meta"), nil)
obj, err := a.srv.AgentServices(nil, req)
require.NoError(t, err)
val := obj.(map[string]*api.AgentService)
require.Len(t, val, 2)

req, _ = http.NewRequest("GET", "/v1/agent/services?filter="+url.QueryEscape("kv in Tags"), nil)
obj, err = a.srv.AgentServices(nil, req)
require.NoError(t, err)
val = obj.(map[string]*api.AgentService)
require.Len(t, val, 1)
}

// This tests that the agent services endpoint (/v1/agent/services) returns
// Connect proxies.
func TestAgent_Services_ExternalConnectProxy(t *testing.T) {
Expand Down Expand Up @@ -629,6 +672,37 @@ func TestAgent_Checks(t *testing.T) {
}
}

func TestAgent_ChecksWithFilter(t *testing.T) {
t.Parallel()
a := NewTestAgent(t, t.Name(), "")
defer a.Shutdown()

testrpc.WaitForTestAgent(t, a.RPC, "dc1")
chk1 := &structs.HealthCheck{
Node: a.Config.NodeName,
CheckID: "mysql",
Name: "mysql",
Status: api.HealthPassing,
}
a.State.AddCheck(chk1, "")

chk2 := &structs.HealthCheck{
Node: a.Config.NodeName,
CheckID: "redis",
Name: "redis",
Status: api.HealthPassing,
}
a.State.AddCheck(chk2, "")

req, _ := http.NewRequest("GET", "/v1/agent/checks?filter="+url.QueryEscape("Name == `redis`"), nil)
obj, err := a.srv.AgentChecks(nil, req)
require.NoError(t, err)
val := obj.(map[types.CheckID]*structs.HealthCheck)
require.Len(t, val, 1)
_, ok := val["redis"]
require.True(t, ok)
}

func TestAgent_HealthServiceByID(t *testing.T) {
t.Parallel()
a := NewTestAgent(t, t.Name(), "")
Expand Down
4 changes: 4 additions & 0 deletions agent/catalog_endpoint.go
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ func (s *HTTPServer) CatalogNodes(resp http.ResponseWriter, req *http.Request) (
// Setup the request
args := structs.DCSpecificRequest{}
s.parseSource(req, &args.Source)
s.parseFilter(req, &args.Filter)
args.NodeMetaFilters = s.parseMetaFilter(req)
if done := s.parse(resp, req, &args.Datacenter, &args.QueryOptions); done {
metrics.IncrCounterWithLabels([]string{"client", "rpc", "error", "catalog_nodes"}, 1,
Expand Down Expand Up @@ -180,6 +181,7 @@ func (s *HTTPServer) catalogServiceNodes(resp http.ResponseWriter, req *http.Req
// Set default DC
args := structs.ServiceSpecificRequest{Connect: connect}
s.parseSource(req, &args.Source)
s.parseFilter(req, &args.Filter)
args.NodeMetaFilters = s.parseMetaFilter(req)
if done := s.parse(resp, req, &args.Datacenter, &args.QueryOptions); done {
return nil, nil
Expand Down Expand Up @@ -261,6 +263,8 @@ func (s *HTTPServer) CatalogNodeServices(resp http.ResponseWriter, req *http.Req
return nil, nil
}

s.parseFilter(req, &args.Filter)

// Pull out the node name
args.Node = strings.TrimPrefix(req.URL.Path, "/v1/catalog/node/")
if args.Node == "" {
Expand Down
Loading