New config parser, HCL support, multiple bind addrs #3480

magiconair · 2017-09-18T21:51:24Z

This patch implements a new config parser for the consul agent which
makes the following changes to the previous implementation:

add HCL support
all configuration fragments in tests and for default config are
expressed as HCL fragments
HCL fragments can be provided on the command line so that they
can eventually replace the command line flags.
HCL/JSON fragments are parsed into a temporary Config structure
which can be merged using reflection (all values are pointers).
The existing merge logic of overwrite for values and append
for slices has been preserved.
A single builder process generates a typed runtime configuration
for the agent.

The new implementation is more strict and fails in the builder process if no
valid runtime configuration can be generated. Therefore, additional
validations in other parts of the code should be removed.

The builder also pre-computes all required network addresses so that no
address/port magic should be required where the configuration is used and
should therefore be removed.

As a side effect the code also supports multiple bind addresses for HTTP,
HTTPS and DNS via go-sockaddr and static ip addresses.

Fixes #895
Fixes #2217
Fixes #2825

magiconair · 2017-09-18T21:51:50Z

There are still a handful of failing tests which I'll address tomorrow.

magiconair · 2017-09-20T07:20:15Z

Still need to update all the docs ...

shilov · 2017-09-20T15:41:41Z

👍 big thanks for adding HCL support + enabling multiple bind addresses

slackpad · 2017-09-21T18:55:45Z

agent/acl_test.go

-	cfg.ACLEnforceVersion8 = Bool(false)
-	a := NewTestAgent(t.Name(), cfg)
+	a := NewTestAgent(t.Name(), `
+ 		acl_enforce_version_8 = true


This actually set it to true when it was supposed to be false. I'll fix this and make sure this test positively ensures that things are working.

The default is set to a wrong key acl_enforce_version8 (not the lacking final underscore) which may contribute to the test failing.

I am working on a patch which detects invalid field names again and restores the previous functionality. However, since encoding/json and hashicorp/hcl decode embedded structs differently (map[string]interface{} vs []map[string]interface{}). Therefore, just using mapstructure as before to detect unused fields won't work. This is a bug in HCL itself for which there is no easy fix AFAICT. Also, using the embedded JSON parser in hcl will also return []map[string]interface{} which will trip up mapstructure. hashicorp/hcl#209

Unless there is an option in mapstructure to do the mapping I think there are several viable approaches:

Patching all []map to map and then use mapstructure for the mapping and invalid field detection. However, watches has []map[string]interface{} so there would need to be an exception for that field and others in the future.

Build a custom invalid field detector which given a map[string]interface{} and struct type+tag name can detect which fields will not be mappable.

Enhance the hashicorp/hcl decoder with an invalid field detector and do the same thing with encoding/json by first forking it and then adding that to the custom code until proposal: encoding/json: reject unknown fields in Decoder golang/go#15314 is added to the stdlib.

Add that capability to mapstructure.

For the 1.0 milestone, the first option seems to be the one with the least amount of work so I'll try that first.

slackpad · 2017-09-22T00:58:58Z

agent/agent_test.go

@@ -167,13 +101,20 @@ func TestAgent_TokenStore(t *testing.T) {
 }

 func TestAgent_CheckPerformanceSettings(t *testing.T) {
+	t.Skip("this test fails because of the change in agent.consulConfig(). We need to decide what to do")


TODO - I'll take a look at this.

I've followed your suggestion and dropped the PerformanceRaftMultiplier parameter and instead compute the right values during the build step.

slackpad · 2017-09-22T21:58:26Z

agent/agent.go

 			defer a.wgServers.Done()
-
-			err := s.ListenAndServe(p.Net, p.Addr, func() { notif <- p })
+			// todo(fs): does this also work for unix sockets?


We should add unix socket unit tests before we finish the beta.

The underlying DNS lib does not support Unix sockets. We can add this at a later stage.

Got it - yeah if DNS never could have supported it previously it's not important to get that in now (also seems like a fairly rare / low priority use case).

unless you want to test DNS on the socket without exposing it on a network interface. But lets add that later.

slackpad · 2017-09-22T23:15:58Z

agent/config/runtime.go

+	return tc.IncomingTLSConfig()
+}
+
+func (c *RuntimeConfig) Sanitized() RuntimeConfig {


We should add a test for this - It would be good to fuzz the structure, sanitize, and make sure anything with token in there is redacted. Maybe we should just use reflect here and a small list of other places to redact and make the logic generic.

Added a test and implemented it reflectively. The expected test result is static so the test should fail once a new field with token, key or secret in the name is added. gofuzz failed on me with some error but I don't think it is necessary with this approach.

I've used different values for the WAN retry join case in a later commit.

slackpad · 2017-09-22T23:32:47Z

agent/config/builder.go

+		Head:  []Source{DefaultSource()},
+	}
+
+	if b.stringVal(b.Flags.DeprecatedDatacenter) != "" && b.stringVal(b.Flags.Config.Datacenter) == "" {


Why does this need a special case here? It's also handled in Build().

This looks like a left over from moving code around.

It is a bit more complex than that but why do we want to carry on deprecated flags and fields to 1.0 in the first place? This looks like a good time to finally remove them.

slackpad · 2017-09-23T00:43:49Z

agent/config/config.go

+	CleanupDeadServers      *bool   `json:"cleanup_dead_servers,omitempty" hcl:"cleanup_dead_servers"`
+	DisableUpgradeMigration *bool   `json:"disable_upgrade_migration,omitempty" hcl:"disable_upgrade_migration"`
+	LastContactThreshold    *string `json:"last_contact_threshold,omitempty" hcl:"last_contact_threshold"`
+	// todo(fs): do we need uint64 here? If yes, then I need to write a special parser b/c of JSON limit of 2^53-1 for ints


This will always be much lower than the max. We could even accept an int here and cast it later.

thought so. will fix

slackpad · 2017-09-23T00:59:53Z

agent/config/runtime_test.go

+				rt.DataDir = dataDir
+			},
+		},
+		{ // todo(fs): shouldn't this be '-encrypt-key'?


No this looks correct.

slackpad · 2017-09-23T01:00:57Z

agent/config/runtime_test.go

+				rt.DataDir = dataDir
+			},
+		},
+		{ // todo(fs): shouldn't this be '-node-name'?


This is correct for the CLI flags (it's node_name in JSON).

yeah, the fact that they are different is one of the issues.

slackpad · 2017-09-23T01:05:13Z

agent/config/runtime_test.go

+				rt.DataDir = dataDir
+			},
+		},
+		{


This is a duplicate of the previous test case - did you have something else in mind here?

no, copy/paste left over when creating additional test cases.

I've removed it.

slackpad · 2017-09-24T02:54:23Z

agent/config/runtime_test.go

+			},
+		},
+		{
+			desc:  "client addr, addresses and ports < 0",


Paused here.

Got to this spot - I'll continue from here...

slackpad · 2017-09-24T07:11:16Z

Agree I was thinking the same and some of these are quite old - let's remove them as long as we are clear in the upgrade guide and list them all.

magiconair · 2017-09-24T11:14:24Z

I've pushed a change which removes all deprecated fields and flags and keeps a list in agent/config/doc.go The last remaining thing is recursor since we have both recursor and recursors and usually merge the single recursor into the list.

slackpad

Made it through the whole thing and I think we are super close. If we can get a check in there to catch unknown fields (the simple plan with an exception for watches sounded reasonable) before the beta I think that would a good cross-check for the all the unit tests and to help users.

slackpad · 2017-09-25T03:06:56Z

agent/config/runtime_test.go

+			patch: func(rt *RuntimeConfig) {
+				rt.Bootstrap = true
+				rt.ServerMode = true
+				rt.LeaveOnTerm = false


In client mode the LeaveOnTerm and SkipLeaveOnInt values should be the opposite of server mode, so true and false, respectively. We should add a case that covers that behavior.

slackpad · 2017-09-25T03:07:44Z

agent/config/runtime_test.go

+			flags: []string{
+				`-data-dir=runtime_test.go`,
+			},
+			patch: func(rt *RuntimeConfig) {


Do we need a patch section when it's checking for an error?

slackpad · 2017-09-25T03:18:39Z

agent/config/runtime_test.go

+			hcl:  []string{`dns_config = { udp_answer_limit = 0 }`},
+			err:  "dns_config.udp_answer_limit cannot be 0. Must be positive",
+		},
+		{


Duplicate of previous test.

slackpad · 2017-09-25T03:29:19Z

agent/config/runtime_test.go

+				"dogstatsd_addr": "0wSndumK",
+				"dogstatsd_tags": [ "3N81zSUB","Xtj8AnXZ" ],
+				"filter_default": true,
+				"prefix_fi


Seems like we can eventually divorce the runtime config from the structure with both check and checks.

slackpad · 2017-09-25T03:34:39Z

agent/dns_test.go

@@ -2913,7 +2858,7 @@ func TestDNS_ServiceLookup_AnswerLimits(t *testing.T) {
 		expectedANYQuery    int
 		expectedANYQueryID  int
 	}{
-		{"0", 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
+		//{"0", 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},


Curious what happened here? Does this fail to validate then we can just delete it.

The new code did not allow dns_config.udp_answer_limit to be zero which made this test fail. I guess we commented this out while getting things to work. I've restored the previous behavior.

slackpad · 2017-09-25T03:40:15Z

agent/node-id

@@ -0,0 +1 @@
+9e8edccb-1f67-0ef6-a36f-e5121d91d8fb


This file should be deleted (or does a test use this)?

slackpad · 2017-09-25T03:42:24Z

agent/segment_stub.go

-
-	return nil
-}
+// done(fs): func ValidateSegments(conf *Config) error {


Did this get moved elsewhere? There's an enterprise version of this file that's in the enterprise fork, so we had this out here so we can swap it with build tags.

I had incorporated this into the normal validation. However, to support different types of validation for enterprise I've split up the tests so that we can overload them. I'll send you the ent fragments you can add.

slackpad · 2017-09-25T03:45:04Z

agent/testagent_test.go

+
+func TestDefaultConfig(t *testing.T) {
+	for i := 0; i < 500; i++ {
+		t.Run("bla", func(t *testing.T) {


bla might as well be an empty string

slackpad · 2017-09-25T03:51:53Z

ipaddr/detect.go

+
+// GetPrivateIPv4 returns the list of private network IPv4 addresses on
+// all active interfaces.
+func GetPrivateIPv4() ([]*net.IPAddr, error) {


We could probably remove these helpers and change the default config to go-sockaddr templates that accomplishes this as well.

That makes mocking the address resolution more difficult. Let me think about a solution.

slackpad · 2017-09-25T03:54:02Z

vendor/github.com/hashicorp/go-sockaddr/ipv4addr.go

@@ -58,7 +58,8 @@ func NewIPv4Addr(ipv4Str string) (IPv4Addr, error) {
 	// Strip off any bogus hex-encoded netmasks that will be mis-parsed by Go.  In
 	// particular, clients with the Barracuda VPN client will see something like:
 	// `192.168.3.51/00ffffff` as their IP address.
-	trailingHexNetmaskRe := trailingHexNetmaskRE.Copy()
+	// trailingHexNetmaskRe := trailingHexNetmaskRE.Copy()


Is this a monkey patch? Why was this needed?

Yes, for some reason this creates a data race which made tests fail randomly at high concurrency. I still need to understand why that is.

slackpad · 2017-09-25T03:57:12Z

I'll try to sort the Raft multiplier stuff.

slackpad · 2017-09-25T04:53:06Z

@magiconair I'm thinking we can get rid of PerformanceRaftMultiplier from the runtime struct and do this scaling at the end of builder.go:Build(), like we do for other derived parameters:

base.RaftConfig.HeartbeatTimeout = a.config.ConsulRaftHeartbeatTimeout * time.Duration(a.config.PerformanceRaftMultiplier)
base.RaftConfig.LeaderLeaseTimeout = a.config.ConsulRaftLeaderLeaseTimeout * time.Duration(a.config.PerformanceRaftMultiplier)
base.RaftConfig.ElectionTimeout = a.config.ConsulRaftElectionTimeout * time.Duration(a.config.PerformanceRaftMultiplier)

slackpad · 2017-09-25T05:12:21Z

@magiconair Heads up I pushed my small stuff from the -slackpad branch up onto yours, so please pull that.

magiconair · 2017-09-25T07:02:10Z

Will do and will go through the comments. I have something working for the invalid fields. Not pretty but it should do for the beta. I'll go through the rest of the comments and fix/respond

This patch implements a new config parser for the consul agent which makes the following changes to the previous implementation: * add HCL support * all configuration fragments in tests and for default config are expressed as HCL fragments * HCL fragments can be provided on the command line so that they can eventually replace the command line flags. * HCL/JSON fragments are parsed into a temporary Config structure which can be merged using reflection (all values are pointers). The existing merge logic of overwrite for values and append for slices has been preserved. * A single builder process generates a typed runtime configuration for the agent. The new implementation is more strict and fails in the builder process if no valid runtime configuration can be generated. Therefore, additional validations in other parts of the code should be removed. The builder also pre-computes all required network addresses so that no address/port magic should be required where the configuration is used and should therefore be removed.

magiconair · 2017-09-25T10:35:43Z

I've rebased the branch onto master and added the code for detecting invalid fields. I've immediately found two tests which were broken so this is very useful. The approach is a bit hacky but I've added a long description of why we're doing this. If I can come up with a better approach then I'll try over the next days.

The open issues are:

~~drop recursor in favor of recursors~~
~~look at the commented out section int TestDNS_ServiceLookup_AnswerLimits~~
document/follow up the monkey patches for hashicorp/go-sockaddr and hashicorp/hcl
require .json or .hcl extension for all config files. (need input/decision)

@slackpad you need to incorporate the enterprise test fragments for the segment code I've sent you.

slackpad · 2017-09-25T13:03:21Z

That's fair and the mock is very nice for testing. We can keep this as is going in to the beta.

magiconair · 2017-09-25T14:37:58Z

We also need to decide where to put the documentation that was part of agent/config.Config. We can either restore it to the Config struct, add it to the RuntimeConfig or omit it from code altogether and put it in the public documentation. Opinions?

… the test

…for builder.

magiconair · 2017-09-25T18:45:06Z

F I N A L L Y !

magiconair requested a review from slackpad September 18, 2017 21:51

magiconair self-assigned this Sep 20, 2017

magiconair mentioned this pull request Sep 20, 2017

Allow binding multiple client addresses #2217

Closed

magiconair mentioned this pull request Sep 20, 2017

Add IPv6 private address ranges #2825

Closed

magiconair force-pushed the refactor-config branch from d8b8905 to a923011 Compare September 20, 2017 08:12

magiconair changed the title ~~New config parser~~ New config parser, HCL support, multiple bind addrs Sep 20, 2017

This was referenced Sep 22, 2017

Unable to disable http & https #3425

Closed

Does consul support HCL for config files? #895

Closed

Replace Statsite prefix with Service prefix #3293

Closed

slackpad reviewed Sep 24, 2017

View reviewed changes

magiconair added this to the 1.0 milestone Sep 25, 2017

slackpad reviewed Sep 25, 2017

View reviewed changes

magiconair added 9 commits September 25, 2017 09:59

Upgrade github.com/hashicorp/hcl to support int64

5561263

improve error messages

63414c6

fix directory permission test

ba536fb

Fix rtt test

60fb0eb

Fix ForceLeave test

699e525

Skip performance test for now until we know what to do

fd21e5b

Update github.com/hashicorp/memberlist to update log prefix

925f0a7

Make memberlist use the default logger

075254a

magiconair added 6 commits September 25, 2017 10:00

remove stray node-id file

5009ea6

drop unnecessary patch section in test

997382b

drop duplicate test

2549572

add test for LeaveOnTerm and SkipLeaveOnInt in client mode

0d44e1f

drop "bla" and add clarifying comment for the test

9f9a251

split up tests to support enterprise/non-enterprise tests

dddc6af

magiconair force-pushed the refactor-config branch from 29f1ee4 to dddc6af Compare September 25, 2017 08:21

magiconair added 4 commits September 25, 2017 11:27

drop raft multiplier and derive values during build phase

019bb3d

sanitize runtime config reflectively and add test

3dd3d72

detect invalid config fields

b6b6269

fix tests with invalid config fields

62ccedb

use different values for wan sanitiziation test

57c3155

magiconair added 2 commits September 25, 2017 16:24

drop recursor in favor of recursors

7b43964

allow dns_config.udp_answer_limit to be zero

be3f6b8

magiconair and others added 5 commits September 25, 2017 17:43

make sure tests run on machines with multiple ips

a8d55f3

Fix failing tests in a few more places by providing a bind address in…

3832b58

… the test

Gets rid of skipped TestAgent_CheckPerformanceSettings and adds case …

69882c2

…for builder.

Add porter to server_test.go to make tests there less flaky

c6a587f

go fmt

b787e70

slackpad merged commit 1221658 into master Sep 25, 2017

slackpad deleted the refactor-config branch September 25, 2017 18:40

slackpad mentioned this pull request Sep 25, 2017

Config refactor burn down #3492

Closed

10 tasks

123BLiN mentioned this pull request Oct 18, 2017

Consul validate fails on the part of the configs #3591

Closed

jenshadlich mentioned this pull request Oct 26, 2017

consul validate only accepts .json or .hcl files => impossible to roll out + validate configs e.g. with puppet #3620

Closed

sean- mentioned this pull request Jan 22, 2018

client network_interface config doesn't parse sockaddr templates hashicorp/nomad#3675

Closed

		@@ -0,0 +1 @@
		9e8edccb-1f67-0ef6-a36f-e5121d91d8fb

New config parser, HCL support, multiple bind addrs #3480

New config parser, HCL support, multiple bind addrs #3480

Conversation

magiconair commented Sep 18, 2017 • edited Loading

magiconair commented Sep 18, 2017

magiconair commented Sep 20, 2017

shilov commented Sep 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slackpad commented Sep 24, 2017 via email • edited by magiconair Loading

magiconair commented Sep 24, 2017

slackpad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slackpad commented Sep 25, 2017

slackpad commented Sep 25, 2017 • edited Loading

slackpad commented Sep 25, 2017

magiconair commented Sep 25, 2017

magiconair commented Sep 25, 2017 • edited Loading

slackpad commented Sep 25, 2017 via email • edited by magiconair Loading

magiconair commented Sep 25, 2017

magiconair commented Sep 25, 2017

magiconair commented Sep 18, 2017 •

edited

Loading

slackpad commented Sep 24, 2017 via email •

edited by magiconair

Loading

slackpad commented Sep 25, 2017 •

edited

Loading

magiconair commented Sep 25, 2017 •

edited

Loading

slackpad commented Sep 25, 2017 via email •

edited by magiconair

Loading