Java Buildpack 2.1

[nebhale]

Starting with the 2.1 release of the Java Buildpack, packaged versions of the buildpack should not be linked via submodule. Instead, a packaged version of the buildpack, (complete with version baked into the code) should be used.

This change removes the existing Java Buildpack submodule and replaces it with an exploded packaged version. In addition, the buildpack_java package was updated to cope with the lack of a git directory (the git hash of the package is baked into the version). Finally, the package spec was updated so that it includes all files since the packaged buildpack is already as minimal as it can be.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: http://www.pivotaltracker.com/story/show/69522964. This repo is managed by the 'Runtime' team.

[nebhale]

/cc @rmorgan

[youngm]

@nebhale can I get some context for this change? Is the main benefit removing all the tests and such from the buildpack package? Or are there other reasons to copy all the files into cf-release instead of just linking the submodule?

[nebhale]

@youngm with the advent of packaged builds, I'd like the official sanctioned release package to be used. That package has an artifact in it (config/version.yml) that contains information such as a user-specified version, which git remote the package is based on, which git hash the package is based on, and whether the buildpack was packaged as an offline buildpack.

This is an enhancement of the previous version information displayed at staging:

-----> Java Buildpack Version: v2.1 | https://github.com/cloudfoundry/java-buildpack.git#ded1e56

This information cannot be determined when the package is used since it's not based on a git repository which is why we package it internally. We also can't check the same information in (chicken and egg problem determining the hash).

[nebhale]

@youngm For any Cloud Foundry installation with a custom buildpack, I'd expect that they'd be packaging up their own buildpacks and using create-buildpack/update-buildpack anyway.

[youngm]

@nebhale I see. If you produced a distribution package as part of a bosh pre_package script could you get the same result? Sure it wouldn't be a copy of the exact bits but it should be the same given the same git revision right?

As a forker of the buildpack and cf-release I think we actually prefer to link our buildpack updates with cf-release updates. Only using update-buildpack for an out of cycle update. It would be much easier for us to just bump the submodule just like we do everything else in cf-release. With this change we will need to introduce a special process to update the java buildpack.

[nebhale]

@youngm I'd be very skeptical of building that package with the same identifiers from source. That puts us in the position where something is labeled as 2.1, but there's no way to know if it differs in any way from what we packaged (there's no documentation on the environment that those scripts run in, so I have no idea if it differs from where we run the packaging). Reproducibility for troubleshooting is then a crapshoot.

If you've already forked the buildpack, then I'm guessing you've replaced the submodule reference in a fork ofcf-release already. When you attempt to merge the upstream changes (from this PR) into your fork of cf-release there would be conflict and I'd expect you to take your changes not mine, preserving the forked java-buildpack submodule linkage. At that point, your fork continues as it always has, and the root cf-release uses the checked in files.

[chou]

@MarkKropf, what do you think?

CF Community Pair (@chou & @jfoley)

[youngm]

Thanks @nebhale it is good to know that continuing as a submodule will still be a supported option.

I guess we'll have to see for ourselves if the benefits of copying a package in will be worth the efforts. It would be nice to have the best of both options and be in line with how origin works.

Thanks,
Mike

[mkocher]

I have some concerns about vendoring the built asset - we've experimented with that kind of thing with both UAA and loggregator and eventually moved away from it in both cases. It seems like prepackaging could easily give you the version tag or current git hash it what you're deploying isn't tagged with a version.

[hiremaga]

A variant of the approach in this PR might be to publish the packaged build to a tag and have the submodule in cf-release point at this tag.

We think this approach solves the chicken and egg problem of determining the hash without vendoring the packaged build.

We also think this preserves the reproducibility of a vendored packaged build and might even improve this by being able to trace the submodule/packaged-build sha back to its source sha.

– Eric (@ematpl) and Abhi

[rmorgan]

@ematpl @MarkKropf Today and Monday are public holidays in the UK so I'm not sure if @nebhale is going to have a chance to respond, I've sent him a note to see if he can look at this today.

[nebhale]

@mkocher Perhaps I'm approaching this incorrectly by specifying a solution with the PR. To phrase this more properly as a requirement, what should we be doing to ensure that the version of the buildpack in the release is identical to the one that we package?

[hiremaga]

@nebhale The suggestion @ematpl and I made earlier might be one way to do this that avoids pre-packaging. Do you think this might work?

@mkocher Thoughts?

[nebhale]

@hiremaga I'm not a fan of checking binaries into source control. It's a sure way to balloon a repository's size, especially in Git where every clone downloads the entire repository. You wouldn't check in your dependency Gems (or Maven JARs) and I wouldn't expect to check in our binaries either.