[CI] Consider replacing Jenkins EC2 plugin

[hcho3]

Jenkins was down last night, as it could not launch any EC2 workers. Some CI jobs timed out after 4 hours of waiting for workers.

Currently, Jenkins CI is using the Amazon EC2 plugin to provision build workers. Five days ago, a new version of the plugin is released. When it was installed on to our Jenkins server, it led to this error message:

com.amazonaws.services.ec2.model.AmazonEC2Exception: Network interfaces and an instance-level security groups may not be specified on the same request (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterCombination; Request ID: 612b9c14-759c-4cba-8f69-0c7c4367e9e4)

As a result, no worker could be launched.

I fixed this problem for now by downgrading the plugin to a previous release. There are two options I can take:

1. Try hard and make the latest EC2 plugin work with our use case.
2. Roll our own script to provision EC2 workers.

I'll need to see which option is easier.

[trivialfis]

I find digging into the haystack of AWS's document very daunting ... So if there's another project dedicated to maintain the plugin I would like to avoid reinventing the wheel.

[hcho3]

I don't like re-inventing the wheel either. The issue is that the latest version of the Amazon EC2 plugin is broken.

[hcho3]

I posted a question to their Gitter chat: https://gitter.im/ec2-plugin/Lobby?at=5ea943c9adb0ec5c2bd2b621.

[hcho3]

Tracking issue for the EC2 plugin: https://issues.jenkins-ci.org/browse/JENKINS-62090

[VoVAllen]

I met the same problem. Uncheck the associate public ip option works for me.

[hcho3]

We ended up keeping the EC2 plugin. The mentioned issue has been addressed. There are many failure modes involved in launching and terminating EC2 instances, and my experience has been that the plugin handles these modes fairly well. Also, I do not want to maintain a large script to manage EC2 workers.

For the purpose of cost control, we arrived at a lightweight solution, where we provide a small function to monitor EC2 cost (#5884).