-
Notifications
You must be signed in to change notification settings - Fork 321
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[EKS] [request]: A reliable EKS AMI release process #319
Comments
EKS AMI releases appear to have simply ceased since 29 March. I could find not AMIs that are newer, and so important fixes to self-inflicted wounds like The project home page and releases page lists the latest AMI's as 27 March. And the AWS Marketplace says the latest AMI version is 20 February. |
Additionally, if there is a supported method for subscribing to EKS AMI updates, I haven't found it in the documentation. |
@wolverian you could watch https://github.com/awslabs/amazon-eks-ami in releases-only mode, that corresponded well with the releases so far. |
This is exactly what I do. But this only works when the release process is reliable, which is what this issue is about 🙏 |
@max-rocket-internet I meant by release-time, not necessarily by content ;-) Although I have to agree that the tagging in this case is slightly misleading. |
I get you but number 7 on my list shows that there have been releases in AWS console but not on github. That's why I mentioned it 😃 |
Hi, sorry for the issues and trouble caused. We are working on an more standard AMI release process which can be used to release AMIs more frequently. BTW, the github release doesn't have correlation with our AMI release, e.g. we may release new AMIs in case of security patches. |
Thank you, that sounds great! Sorry for nagging, but would it be possible to synchronize GitHub releases with AMI releases? If not, maybe disabling GitHub releases altogether would make sense, to reduce potential confusion. |
Synchronized releases would be ❤️. We pull information about recent updates for all our services over RSS and GH fits nicely into this workflow. |
Exactly 💯 An SNS topic is not very user friendly when much of our current software and work flow already revolves around Github. |
@max-rocket-internet @szymonpk There is actually three component that can be involved in release process here:
For Github release, I think it can be synced with 1 and 2 above. |
@M00nF1sh that all sounds like a big improvement on the current situation. There is also an immediate need, as the latest available EKS AMIs are more than 3 months old and contain some pretty problematic bugs, like this |
@whereisaaron We'll release new amis on monday 😄 |
@wolverian more context on what @M00nF1sh mentioned, we are working on launching an AMI SSM parameter that you can use along with an SNS topic that you can subscribe to. See #231 |
Sure: We don't build AMIs. We want to avoid this administrative overhead. We just want reliable, bug free AMIs provided from AWS with:
(the numbers are references to my points in my first post).
This problem is non-existent for Terraform users 😃 |
@M00nF1sh The best solution for us, would be synchronized 1, 2 and 3. I see 3 may not happen, so it may be a good idea to have something similar to ALAS2 bulletin but for EKS. However, I see it is rather unlikely to have separate security feeds for AWS services. |
Well, it appears I spoke too soon 🙁 In the Hashicorp EKS Terraform module, we filter AMI IDs by name. This means we can easily specify a release by its name and not worry about what region we are in. That's item number 3 is important. But in AWS region Even if you aren't using Terraform, how would you know what AMI is the correct one? Noted in awslabs/amazon-eks-ami#291 This is what I mean by "release process". I don't know what the process behind the scenes is, or how it works, but if it was automated in a proper this should be impossible. |
@tabern I just want to highlight one more point related SSM, I got an issue in which the SSM got updated to amazon/amazon-eks-node-1.15-v20200312 around around 3 weeks back, however CF didn't support the new value for Just checked and find out that newest release for AMI https://github.com/awslabs/amazon-eks-ami/releases v20200406, not sure if it's supported by CF or not. |
Not sure if it is entirely related to this issue, but I find the Kubernetes version matching AMIs quite uncomfortable. Preferably, I'd like to see a list of Kubernetes versions (including the patch version) and the AMI in a single list. As far as I can tell, there is no way to tell which exact Kubernetes version is used in an AMI. |
Bumping up against the |
is it planned to support AWS AMI builder awslabs/amazon-eks-ami#548 ? |
Been a while since any activity on this, and I feel our release process has become much more reliable. One outstanding item is #734, but otherwise I'll close this issue soon if there aren't any other outstanding requests |
@max-rocket-internet Is it an acceptable approach to tackle this specific problem by making the AMI names unique by adding |
I'm still seeing inconsistencies in release process that feel very related : awslabs/amazon-eks-ami#662 |
More issues with the current release I understand bugs and mistakes happen, but at this point really feels despite what was said in #319 (comment) by @mikestef9 |
@anish I'm in agreement with you. The CHANGELOG is still showing the wrong K8s versions on the last two releases despite it being reported and an AWS engineer responding. I think the CHANGELOG is manually generated after a release, |
It looks like AMIs are pushed before the release notes have been updated. |
The current documentation at https://docs.aws.amazon.com/eks/latest/userguide/eks-linux-ami-versions.html is missing the last release which happened 10 days ago and included version bumps to K8s (amongst other things). The quoted version of add-on images such as kube-proxy are incorrect in the docs too. It's got to the point where the documentation needs to be considered incorrect and everything needs looking up via the CLI. Come on AWS this isn't good enough, the docs should be created and published as part of the release process. |
Apologies for the confusion @stevehipwell. We're looking into how we can improve the consistency between our AMI releases and the documentation. In the meantime, the documentation changes with the latest EKS AMI info will be published shortly. |
@akestner I can see you got the release notes out and for |
@akestner The changelog also seems to have been published wrong. The v20211117 release notes claim an update to |
@anish I suspect that the commits in this repo aren't actually tied to the release process and are mirrored here from a private internal repo. |
@stevehipwell quite likely, since the changelogs seem to match commits, just not the release tags. |
Wanted to report it here in addition to nagging AWS directly, but we are seeing Node failures in We did not author this AWS forum post, but it describes our issue precisely. Once we remove the IMO, this points to an issue with AMI |
I hadn't seen this issue reappear for a year but back again with https://github.com/awslabs/amazon-eks-ami/releases/tag/v20221104 |
@anish the most recent issue with v20221104 can take many days or weeks to appear in our experience, fairly random and some nodes go a week or more without the kernel bug triggering. So I am less surprised AWS did not spot this one ahead of time. On Nov 9 AWS sent Operational Notifications to anyone running the bad AMI with references. So I thought that was relatively well handled. What could have been better:
The use of dated versions is non-obvious unless you dig into AMI ids, I made a small script to make these checks easier, that lists the dated versions for an AMIs present in a cluster.
|
@whereisaaron I was more specifically talking about this awslabs/amazon-eks-ami#1093 (comment) (changelog, release and makefile being constantly out of sync with each other) which is an on going issue that had not been seen for a while |
The last release was messed up again which is a few commits behind the actual release awslabs/amazon-eks-ami#1357 |
fixed. thanks for the quick turnaround @cartermckinnon |
My request: The AMI release process needs to be reliable
We have been EKS users since the first preview and one significant pain point is issues with the AMI. From the outside it looks like the release process is inconsistent and/or unreliable.
Here's some notable examples:
ulimit
settings: Latest AMI version lowers ulimit and breaks Elasticsearch awslabs/amazon-eks-ami#193v20190329
released in the AWS console but not on Github: Latest AMIs (v20190220) missing commits in /etc/eks/bootstrap.sh awslabs/amazon-eks-ami#233 (comment)Now, I know and expect some bugs, I also recognise it is (or was) a new service so of course there's a few kinks to work out, but the last 2 items are recent. This is not the quality that I have come to expect from my favourite cloud provider 💔
The text was updated successfully, but these errors were encountered: