-
Notifications
You must be signed in to change notification settings - Fork 687
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Local automated upgrade testing #3075
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #3075 +/- ##
========================================
Coverage 85.79% 85.79%
========================================
Files 34 34
Lines 2154 2154
Branches 238 238
========================================
Hits 1848 1848
Misses 250 250
Partials 56 56 Continue to review full report at Codecov.
|
In the test instructions there should also be |
@msheiny at this stage I'm not doing a proper review, just discovering the great work you did 💯. And dumping a few inconsequential remarks on the way ;-) Both |
That is a very good point. I managed to get some of that upstreamed to molecule... I need to do the rest of the tweaks. |
Nice ! URL ? |
molecule/upgrade_test/playbook.yml
Outdated
state: running | ||
tags: always | ||
|
||
- name: Ensure tor is running |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/tor is running/supervisor is running/
|
The
Retrying with 0.5.2 debs, rebuilding now... |
Even with 0.5.2 debs, still seeing:
|
ca952d6
to
f250640
Compare
Rebased on |
Trying to reproduce now... Just to clarify, this is on the |
@conorsch can you get me more data on that error? I'm not able to recreate locally... |
@kushaldas can you take a look at |
subprocess.check_call(sysprep_cmd.split()) | ||
|
||
def vagrant_metadata(self, img_location): | ||
# type: (str) -> dict |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is actually wrong. We will have to install mypy_extensions
and use TypedDict for the return type of the function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good to know, @kushaldas. Currently the typelint
check isn't running against this file. Even adding it locally to the Makefile target, I don't see any errors reported.
Is this what I should do to get packages for both versions under test?
|
@dachary can you clarify your question ? I don't quite understand. I'm assuming you are referring to the upgrade_test scenario .. in that case you don't really need |
I'm confused about which version I should run |
37edb45
to
472aa4c
Compare
Our unique changes that were added to support suppressing NFS were recently merged upstream in molecule 2.13: * ansible/molecule#1235 * ansible/molecule#1233
0ab9386
to
8a298dd
Compare
heyyyyy @conorsch this PR is getting really huge and unwieldy ... everytime I rebase I'm hitting more weird conflicts that I then have to go in and fix ... discovering some really weird issues that are out of scope for this PR (see this last commit Sorry to nudge ya here but can you give another go through when you have some downtime? Keep in mind this PR really only affects CI and a local dev environment edge-case for specific Linux users so I kindly ask that you keep review pegged to the the The sooner I can get this in I can also start working on the server-side story to try and get this running in CI ❤️ |
Taking another look, @msheiny. Might ping you for some real-time collab if I run into any snags during re-review. |
ping @conorsch i slightly updated the testing steps here. There is another scenario to test proxy thru to apt-test |
Ran through the entire workflow again today. The upgrade flow is quite sound. Built local debs from the Still could not run the
After doing so, encountered the same error, reported above during prior reviews:
@msheiny, let's sync on the error above tomorrow in real time to debug. The effort should be timeboxed, though: if we can't solve it to our mutual satisfaction within an hour, I propose we separate out the package logic and work on it separately. The upgrade logic is highly valuable, and should be merged soon to unblock others on the QA workflow. The package logic is less critical, given that it need only be run once per release. |
I really don't want to do this. The package logic only affects you and me at the moment. No one else is going to be running it (not even CI). This is very similar to the condition when we initially merged the docker logic when it was undocumented and had failing tests. Sometimes its easier to keep merging undocumented and isolated components early as possible and continue to iterate and improve. I believe this is one of those scenarios. It's been really painful to continue to rebase this PR. I keep finding unrelated bugs and having to fix them and it has really snow-balled into a huge ordeal. I'd strongly prefer to merge as is, assuming you do not find any issues with the upgrade testing component. |
Seems that with the latest molecule bump, the relative pathing calculation has changed in respect to the ansible host/group vars directories. Previously it was checking from the ephemeral dir, now its from the scenario dir.
Not sure why this wasnt needed before :| This will soon be obsolete as soon as we are testing against post 0.7 (which includes the ssh over local net features).
Especially want to make sure we avoid vagrant images and additional build folders
Also added `.python3` temp folder to the ignore list
These are out of scope on this PR for me to address and for the sake of brevity I'm going to exclude them for now since they are not recent changes.
This applies strictly to the upgrade test scenario, if you have an environment variable `QA_APTTEST` set to yes/true then your upgrade test will be getting packages from apt-test instead of from local apt packages.
In the bump of molecule we also brought up the version of testinfra which introduced a bug in detecting open udp ports. They switched to using `ss` instead of `netstat` when its available. I took the path of least resistance, opened a bug report, and made a hacky work-around using `lsof`. Bug report -> pytest-dev/pytest-testinfra#311
3f7eb02
to
aa396fb
Compare
Rebased and added a fix for that issue you saw ;) |
With the bump to molecule, the ephemeral directory spot has changed. For now, lets utilize the old directory space since its already git/docker ignored. We just need to ensure that we intentionally create that directory. There was also a change in how the vagrant boxes are named. The `.molecule` prefix is gone and replaced with the scenario name.
Without this SSHd will not be listening on all interfaces upon reboot
fd03c60
to
4800743
Compare
Spent some time with the one-and-only @msheiny debugging the vagrant-package workflow today. The failure related to chowning to the ossec user, reported several times above, turned out to be caused by the entire Once we clarified that issue, I was able to confirm working end-to-end scenario flows for both Thanks for your patient assistance on this, @msheiny. Let's get it in! |
Status
Ready for review
Description of Changes
Changes proposed in this pull request:
Fixes first check-box in #3018
Add two scenarios:
vagrant_packager
-- builds two vagrant boxes ready for redistributionupgrade
--- fires up boxes from the first scenario (pulled from s3), and chucks all locally built deps to a local apt server.Testing
How should the reviewer test this PR?
Caveat
- this PR testing will only work under a system with libvirt/kvm. There is another ticket to also add support for virtualbox.Packager testing:
make vagrant-package
- be prepared to enter a sudo password about 20 min in .molecule/vagrant_packager/push.yml
dont run that now... but glance it over :)Upgrade testing (local deb packages):
make build-debs
- make you sure have a version of debian packages that are higher than0.6
if thebuild/
directory. If you don't, running this command should give you0.7.0~rc1
molecule converge -s upgrade
to get.0.6
SD servers up. You'll get passed an onion address at the end. Navigate to that in a web-browser and note the version at the bottom.molecule side-effect -s upgrade
- Navigate back to the onion address and confirm version has bumped.Upgrade testing (apt-test proxy):
molecule converge -s upgrade
to get.0.6
SD servers up.molecule login -s upgrade --host app-staging
- To get a shell onto the app serversudo apt-get update
(confirm no ERRORS , you'll see a warning about duplicate repos but thats not an error and is a diff issue)apt-cache policy securedrop-config
(same terminal ^^) and you should see all the packages pulled in from aboveQA_APTTEST=yes molecule converge -s upgrade -- --diff -t apt
2 -> 4
again. This time you should see versions of securedrop-config pulled in from apt-test (this should have a few versions back and look like below):Deployment
Any special considerations for deployment? Consider both:
None - This only affects developer upgrade testing environment.
Checklist
If you made changes to the app code:
None
If you made changes to the system configuration:
None
If you made changes to documentation:
None