Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Certificate Rotatation in APIServer #1154

Merged

Conversation

MatthewHinton56
Copy link
Contributor

@MatthewHinton56 MatthewHinton56 commented Aug 26, 2020

fixes: #1152

Self signed certificates are now stored in files rather than memory. This allows for the use of dynamic certificate objects. That will automatically process the update

The caCertController is updated immediately through RunOnce, whereas the DynamicServingContent for the serving certificate can take up to one minute

Attached is a log for an antrea controller with a fixed certificate rotation every 3 minutes. It shows that it can take up to 1 minute for the certificate to be detected, but in production, the rotation period will be closer to months.
antrea-contoller-info.txt

@antrea-bot
Copy link
Collaborator

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e, conformance and network policy tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

  • /test-e2e: to trigger e2e tests.
  • /skip-e2e: to skip e2e tests.
  • /test-conformance: to trigger conformance tests.
  • /skip-conformance: to skip conformance tests.
  • /test-whole-conformance: to trigger all conformance tests on linux.
  • /skip-whole-conformance: to skip all conformance tests on linux.
  • /test-networkpolicy: to trigger networkpolicy tests.
  • /skip-networkpolicy: to skip networkpolicy tests.
  • /test-windows-conformance: to trigger windows conformance tests.
  • /skip-windows-conformance: to skip windows conformance tests.
  • /test-windows-networkpolicy: to trigger windows networkpolicy tests.
  • /skip-windows-networkpolicy: to skip windows networkpolicy tests.
  • /test-hw-offload: to trigger ovs hardware offload test.
  • /skip-hw-offload: to skip ovs hardware offload test.
  • /test-all: to trigger all tests (except whole conformance).
  • /skip-all: to skip all tests (except whole conformance).

@MatthewHinton56 MatthewHinton56 force-pushed the Update-Serving-Certificate branch from 6b92482 to c69be3d Compare August 26, 2020 21:00
@antoninbas antoninbas requested a review from tnqn August 26, 2020 22:07
Copy link
Contributor

@antoninbas antoninbas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I'll defer to @tnqn for the review. I wonder if we should improve testing for rotation, since the original tests were not enough to detect this issue (the test should verify that the apiserver switches to the new certificate). But I don't want to delay this PR either...

Thanks @MatthewHinton56 for realizing there was an issue in the first place.

pkg/apiserver/certificate/certificate.go Show resolved Hide resolved
@@ -124,15 +127,14 @@ func generateSelfSignedCertificate(secureServing *options.SecureServingOptionsWi
var caContentProvider dynamiccertificates.CAContentProvider

// Set the PairName but leave certificate directory blank to generate in-memory by default.
secureServing.ServerCert.CertDirectory = ""
secureServing.ServerCert.CertDirectory = selfSignedCertDir
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The above comment should be updated too.

// UpdateCertificate updates the certificate to a new one. Used to rotate statically signed certificates before they
// expire.
func (c *CACertController) UpdateCertificate(caContentProvider dynamiccertificates.CAContentProvider) {
c.mutex.Lock()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mutex is no longer needed.

if err != nil {
klog.Errorf("error generating new cert: %v", err)
return
}
c.UpdateCertificate(caContentProvider)
c.RunOnce()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c.RunOnce calls caContentProvider.RunOnce() and c.syncCACert(), while the former will trigger the latter too, then it will be executed twice.
I think it could just call caContentProvider.RunOnce(), the first argument of this function could be *DynamicFileCAContent.

@MatthewHinton56 MatthewHinton56 force-pushed the Update-Serving-Certificate branch from c69be3d to 3c23287 Compare August 27, 2020 17:10
Copy link
Member

@tnqn tnqn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining comment about unit test message, otherwise LGTM.
@antoninbas please go ahead merging this once the comment is addressed.

BTW, the attached PDF is not the log. But I have verified this PR too.

} else {
assert.Nil(t, secureServing.ServerCert.GeneratedCert)
assert.NotEqual(t, genericoptions.CertKey{CertFile: selfSignedCertDir + "/antrea-controller.crt", KeyFile: selfSignedCertDir + "/antrea-controller.key"}, secureServing.ServerCert.CertKey, "CertKey doesn't match")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message "CertKey doesn't match" is wrong if it asserts they are not equal? Perhaps you could assert it equals to the expected CertKey of user provided case.

@MatthewHinton56
Copy link
Contributor Author

I updated the log, sorry about that.

@MatthewHinton56 MatthewHinton56 force-pushed the Update-Serving-Certificate branch from 3c23287 to 588dcdd Compare August 27, 2020 18:02
@antoninbas
Copy link
Contributor

/test-all

Copy link
Contributor

@antoninbas antoninbas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I believe Quan's comments have all been addressed. Will merge after CI jobs pass and include in 0.9.2.

@antoninbas
Copy link
Contributor

/test-windows-networkpolicy

@antoninbas
Copy link
Contributor

Seems that there is an unrelated issue with the jenkins-windows-networkpolicy CI job

@antoninbas antoninbas merged commit 6bab6d6 into antrea-io:master Aug 27, 2020
antoninbas pushed a commit to antoninbas/antrea that referenced this pull request Aug 27, 2020
Self signed certificates are now stored in files rather than memory. This allows for the use of dynamic certificate objects. That will automatically process the update

The caCertController is updated immediately through RunOnce, whereas the DynamicServingContent for the serving certificate can take up to one minute

Fixes antrea-io#1152
antoninbas pushed a commit that referenced this pull request Aug 28, 2020
Self signed certificates are now stored in files rather than memory. This allows for the use of dynamic certificate objects. That will automatically process the update

The caCertController is updated immediately through RunOnce, whereas the DynamicServingContent for the serving certificate can take up to one minute

Fixes #1152
GraysonWu pushed a commit to GraysonWu/antrea that referenced this pull request Sep 22, 2020
Self signed certificates are now stored in files rather than memory. This allows for the use of dynamic certificate objects. That will automatically process the update

The caCertController is updated immediately through RunOnce, whereas the DynamicServingContent for the serving certificate can take up to one minute

Fixes antrea-io#1152
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Certificate rotation doesn't take effect in apiserver of antrea-controller
4 participants