[SPARK-8338] Use AMI preinstalled ganglia

[smartkiwi]

The root cause was old ganglia/init.sh code that left here since the time when ganglia was not preinstalled.

[shivaram]

Couple of points

1. Can we just comment out the code instead of deleting it ?
2. Even better can we check the version that is currently installed in the AMI and then have code to install the right version if the version doesn't match ? I don't think its much of a hack to uninstall ganglia and reinstall a specific version that matches the config file we have ?

[smartkiwi]

@shivaram

1. Can we just comment out the code instead of deleting it ?

Sure. BTW Git will keep previous version for us.

1. Even better can we check the version that is currently installed in the AMI and then have code to install the right version if the version doesn't match ? I don't think its much of a hack to uninstall ganglia and reinstall a specific version that matches the config file we have ?

Well, I don't like the latter - this would add more complexity - like the problem we have with Ganglia and httpd versions configs. I noticed when installing and uninstalling ganglia and httpd packages that rpm moves configuration files for it. It would be too difficult to maintain in the future.

[smartkiwi]

@shivaram
What do you think on how likely it can happen that AMI won't have ganglia preinstalled in the future? How this could happen?

[shivaram]

Well it could happen - I just like the scripts to be a bit more robust and install the the versions of software required by the config files we maintain (at some level spark-ec2 is about configuring things to work). This is what we do with Spark and Hadoop and I don't see why ganglia should be any different. This is partly why I favor the uninstall anything on the AMI and install things we want approach.

BTW there are other ways to solving this problem -- Instead of having a hard coded config file that we copy over we could have a diff file that we patch on to the config file. As far as I know the only things we need to patch from the default version is the port number [1] and the permissions [2]

[1] https://github.com/mesos/spark-ec2/blob/branch-1.4/templates/etc/httpd/conf/httpd.conf#L136
[2] https://github.com/mesos/spark-ec2/blob/branch-1.4/templates/etc/httpd/conf.d/ganglia.conf#L10

[smartkiwi]

I created python script to check and install packages using yum API (somewhat similar to chef). https://github.com/smartkiwi/spark-ec2/blob/branch-1.4-no_ganglia_install_check/ganglia/check_packages.py

Some package versions available in AMI (i.e. ganglia 3.3) are not available in Amazon rpm repository - there is only ganglia 3.6.
The same could happen with httpd and php - old versions could be removed from repository.

This is another point why we should always use ganglia packages preinstalled with AMI.

I suggest to merge my changes as they are now and build new AMI with latest version of ganglia and its dependencies in near future.
(unless there is repo with older versions)

PS. With Spark and Hadoop - we install them from other repositories than Amazon rpm repository, and we do not support old versions of Spark if they would be removed from locations.

[smartkiwi]

Ok. I've finished python script that uses yum API to check for installed httpd and ganglia version and installing the correct version is the wrong one is installed.

[shivaram]

Thanks @smartkiwi -- I'll try this out today.

cc @nchammas who might be a better reviewer for the Python code.

[smartkiwi]

BTW
here is command line I've used to test it:

/spark-ec2 -k {key_name} -i {key_file} --region=us-east-1 -z us-east-1b -s 1 --spark-ec2-git-repo=https://github.com/smartkiwi/spark-ec2 --spark-ec2-git-branch=branch-1.4-no_ganglia_install_check launch "testcluster" 

[nchammas]

Haven't had time to test this yet, but a few questions for @smartkiwi:

1. Where are we tracking the yum dependency for the Python script? I assume it's not in the standard library.

2. Instead of this Python script to pin Ganglia and httpd versions, can't we just have a simple shell script that always uninstalls Ganglia and httpd and installs the pinned versions we want?

   For example, something like:

   sudo yum uninstall ganglia httpd
   sudo yum install httpd-2.4.6-6 ganglia-3.3
   

[shivaram]

@smartkiwi I tried this out today and my first run ran into an error, but it seemed to work when I re-ran it. But thinking more about it, it is a hassle to keep relying on these old versions. So I tried an alternate method where we get rid of the httpd.conf and only use sed to set a listen port on it in amplab/spark-ec2#4.

@nchammas Would be interested in hearing which strategy sounds better to you as well.