-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test suite discussion (Friday June 23) #2333
Comments
See also #2293. |
Some initial thoughts:
|
I was able to spin up a Travis-like VM where these tests can be debugged and tested... |
I know I'm on vacay but I couldn't help but add some thoughts :)
|
@mdbenjam when is this conversation actually happening? Is it the time that would be right after standup today if there were standup? |
Action items:
|
This is the |
Just an update on the shared CI pipeline with vets.gov. @CyberKoz has a prototype Jenkins k8s cluster ready and @mogrenAtWork will start developing against it. There are the steps.
There are three key advantages for this setup.
For timeline reference, @CyberKoz expects the k8s setup to go to production in 2-3 weeks. For Appeals, it depends on how fast we can get the container and scripts ready. |
Why?
Regardless, this does increase the devops cost on DSVA overall. I am still not sold that we shouldn't use a SaaS solution to decrease our devops burden. @askldjd what are the benefits of Jenkins over CircleCI? I agree that Jenkins costs less money, but the budget may not be the biggest concern here. |
Arbitrary. We can discuss this.
Maybe, maybe not. I think once the infrastructure is up, adding Appeals to the CI pipeline will fit well within the economy of scale. Personally, I would much prefer to have our infrastructure stack lining up with vets.gov team. Having two CI solutions for DSVA seems to make no sense when our software stack is near identical.
Performance. I haven't seen performance number from Circle CI. However, they are unlikely to be able to match the performance of a custom Jenkin swarm. If Circle CI isn't fast enough, we will have to speed up our test. With Jenkins, we can just throw more hardware at it. Vets.gov team has the Jenkins CI Swarm for over 6 months now and they are happy customer. Like us, they outgrew Travis, and evaluated all the SaaS solutions. Jenkins CI Swarm was their solution, and they have been happy since. Jenkins isn't going away no matter what. Our CD pipeline can only be done by Jenkins. I think it is more advantageous to invest time in Jenkins competency instead of investing in Circle CI competency. |
I agree with this. Unless CircleCI competency is trivial compared to the work of maintaining more Jenkins environments.
I agree, but I wonder why DSVA doesn't use a SaaS solution instead of Jenkins for CI as well! 😄
This is a good data point. I would love to chat with them more about why they chose Jenkins.
I agree that we still need to investigate further, but I wouldn't be surprised if CircleCI could have comparable performance to Jenkins. Can we not also "throw more hardware" at CircleCI by giving them more money? |
As a follow-up to my earlier comments: I chatted with @CyberKoz today. Vets.gov went to Jenkins instead of a SaaS CI because they were frustrated with Travis being too slow even at the max subscription level and were afraid that other SaaS products would have the same problem. He thinks that the additional overhead of Appeals running PR builds on Vets.gov Jenkins would be negligible. Extra work to generally improve their CI setup is minimal as well. Strangely, the pricing pages for both Travis and CircleCI only talk in terms of horizontal scaling, not vertical. I'm going to reach out to CircleCI and ask about their ability to do vertical scaling. Until then, I'm fine moving forward with Jenkins. I am also curious to try CircleCI when we get our dockerfile ready and see how the two compare. |
Adding norms based on the discussion from #2333.
For what it's worth, a few other SaaS CI providers: |
Update from talking to @CyberKoz: the problem that Vets.gov had with Travis was horizontal scaling, not vertical. CircleCI offers up to 62 concurrent builders. I think we'd have a hard time hitting that. For me, the biggest unknown will be how fast a single build runs. Let's try both CircleCI and custom Jenkins side-by-side and see how they compare. |
It seems every couple of months we have a discussion about our test suite. There are two main complaints:
Proposal
Capybara tests have proved to be difficult to maintain. I suggest that we move away from Capybara feature tests by doing the following.
Feel free to add new ideas, or push back against what I've posted. Let's have a conversation about this on Monday June 19.
The text was updated successfully, but these errors were encountered: