GRR v1.4.0: The One With The Minor Networking Changes

• Verified with grootfs-release v0.15.0

Hi Gardeneers!

GRR v1.4.0 mostly introduces some changes to the network plugin API and introduces a first-pass experimental ability to limit container block IO. Please, contain your enthusiasm.

Network Plugin / Networking Changes

• We now allow the network plugin to return a set of DNS servers which we will ensure are set on the container's resolv.conf. This allows DNS policy to be set by the network plugin if enabled.
• When the network plugin is not enabled, there's now an additional_dns_servers property that allows specifying extra DNS servers which should be appended to any inherited from resolv.conf or set to the dns_servers property.
• /etc/hosts and /etc/resolv.conf are now bind-mounted in to the container rather than being written in a chroot. This interacts better with user namespaces and rootless containers and is generally more simple and secure.
• We no longer set up the built-in networkers iptable chains when an external network plugin is enabled, to avoid confusion caused by two things both creating iptable state on the host
• Log messages from NetOut(log=true) rules now properly truncate handles so that space separation is preserved when the handle is longer than 29 characters

BlockIO Limiting (Experimental)

• We've added an experimental default_container_blockio_weight property to allow assigning a blockio weight to all containers created by garden. This requires the CFQ scheduler to be enabled to be useful and applies to all garden-created containers on the host. The feature is being released experimentally via the bosh property, feedback welcome!

Rootless Mode

• You can now follow the updated experimental rootless mode doc to add networking support to your rootless containers.

GRR v1.3.0: People say I am rootless

• Verified with grootfs-release v0.15.0

Lots of nice little changes and two big ones: CPU maximums support and alpha/experimental support for running containers without root.

Experimental! Support for running containers without root

This is very much a v1 which we would not suggest running in production (yet!). It is now possible to run garden without needing root (other than for one-time setup). This increases the security posture of garden and the number of places where it can potentially be used. Manual instructions and known gaps in the attached doc: https://github.com/cloudfoundry/garden-runc-release/blob/develop/docs/rootless-containers.md.

You can also opt-in using the experimental_rootless_mode bosh property which will do all the work for you!

NOTE: Not ready - or secure - for production use yet, but feedback very welcome!

CPU Maximums

It is now possible to request a maximum CPU limit proportional to the existing fair-share CPU limit. The previous cpu limit just ensured that if two containers competed for cpu they would receive it in proportion to their requested cpu share (in the Cloud Foundry use case, this meant in proportion to their memory limit). This meant, for example, if only one container was active on a host it could potentially consume all of the host's resources -- far more than were paid for.

The new cpu_quota_per_share_in_us property, which - if set - applies to all containers created on a host, enforces an additional maximum amount of cpu usage in each quota period (which unless changed on the system is 100ms) in proportion to the existing shares limit.

Example: Using Cloud Foundry as an example, since CF sets a container's cpu limit (ContainerSpec.Limits.Cpu.LimitInShares) based on the memory limit, if cpu_quota_per_share_in_us is set to 100 on a single-core system then a 64MB container will be able to use a maximum of 6.4ms (64 * 100us = 6400us = 6.4ms) of cpu every 100ms.

A Few More Things..

• Container MTU now defaults to the MTU of the external network interface on the host rather than 1500. This is pretty much always what you want and avoids issues on environments with lower MTUs than the default.
• We now strip all entries starting 127.* from resolv.conf on the host when setting up networking for the container. Since 127.* on the host can never be accessed in the container it's never the right thing for it to survive into the container's resolv.conf (and causes slow DNS lookups when it does).
• Our CI now tracks container creation performance using bucket bench
• Golang bumped to 1.8.0
• Squished some misleading log messages. Feel good knowing you will no longer be misled (as much).

Enjoy!

GRR v1.2.0

• Verified with grootfs-release v0.11.0

Hi garden fans! Today we have a bumper release with lots of nice little fixes and improvements:

API Changes and Deprecations:

• Docker Auth Support: The client.Create call now supports an Image.URI field which deprecates the existing RootfsPath field. It acts in exactly the same way as the existing field, however it is now possible to specify Image.Username and Image.Password which will be passed to the image_plugin if configured. This allows an image_plugin (such as grootfs) to support authenticated Docker images.
• Create-Time NetIn/Out: NetIn and NetOut are now able to be specified on the client.Create call, and we recommend all clients switch to this as the dynamic methods are now deprecated. If specified, they are passed to the network_plugin on create, which allows better integration with CNI up/down hooks. The existing methods will continue to be supported in the built-in kawasaki networker until the next major version bump.
• Image Plugin API: Experimental image_plugin API continues to evolve, now does not needlessly swap uid before running the plugin in unprivileged mode (this is up to the plugin to do if it wishes).
• Grace Time: grace_time now defaults to 0 (i.e. infinity). Most clients were explicitly overriding our default here anyway, and without this default it is impossible to later ask for a container not to have a grace time (since 0 in client.Create means "use the default"). Clients must now explicitly set the grace_time bosh property if they wish containers to have a grace time by default.

General Improvements

• RunC was bumped to the latest version
• Inspector-garden is no longer needed! You can now interact with containers without any extra steps. (Yay!)
• Fixed handling of the bosh release shutdown script to avoid a case where a non-zero exit was returned to bosh even though the server had been killed
• The bosh release now increases pid limits to avoid running out of pids in large deployments (to avoid pid exhaustion from containers we recommend configuring ContainerSpec.Pids.Max in the client.Create call).
• It is possible to opt-out of apparmor (for environments which cannot support it), by specifying an empty string for the apparmor_profile property. It is also possible to request a different apparmor profile than the garden-default be used, so long as this is installed on the host (for example as a bosh pre-start job in an add-on).
• iptables rules now add a descriptive comment containing the container guid for easier debuggability
• Work continues on the experimental rootless mode, there is now a separate setup command which can be run as root, allowing the main guardian server to start up without needing root.

GRR v1.1.1

• Verified with grootfs-release v0.7.0

Patches runC to address a security vulnerability (CVE-2016-9962). Garden never runs user processes as pid 1 (which the mentioned exploit relies on) and enables apparmor (which prevents ptrace), but the patch also works around a kernel mis-ordering of operations that could very briefly expose an fd in a container.

GRR v1.1.0

• Verified with grootfs-release v0.7.0

Happy 2017 Garden Fans!

New year, new garden release. Get it while it's hot. Here are some highlights:

• v1.1.0 marks the official start of garden's 'semantic version policy'. From this release you can expect that we will maintain backwards compatibility for existing clients with new garden versions in the v1+ series (in other words if a client says it has been tested with garden 1.N, we expect any version from 1.N to 2 to work with that client. You are still advised to test well in your own CI).
• We are now automatically creating a single-binary version of garden, called 'gdn' (and pronounced "gdn") - linked from these very release notes! - so that you can easily take it for a spin on your own server. Just download and run.
• More performance work: we now test performance over a large number of creates in our CI and will be tracking any performance regressions. More to come on this front.
• Fixed a nasty deadlock in the case where runc blows up and prints large amounts to stderr before we're ready to read it.
• No longer listening on debug_address unless explicitly requested
• Removed spurious and very annoying auplink error message on destroys

GRR v1.0.4

• Verified with grootfs-release v0.6.0

Hi garden consumers! We're pleased to present a fresh new garden release with the following great new features:

• Fixed significant performance degradation for long-running deployments (due to a memory leak in a dependency).
• Support for a new ContainerSpec.Limits.Pid.Max field to limit max processes in a container (note: uses new PID cgroup, requires 4.4+ kernels)
• No default for debug_listen_address property any more - default is not to listen. If you wish to enable debugging, you should explicitly opt in.
• We now use the iptables-restore packaged in the bosh release rather than relying on whatever's in the base OS
• Improvements to network_plugin API for BulkNetOut
• Improvements to image_plugin API. Now testing against grootfs in pipeline and listing compatible versions in these very release notes.

You will notice that these very release notes now list compatible versions of grootfs. These are the releases we have tested this version of garden-runc against. GrootFS is the new Garden Root Filesystem management component. We intend to eventually deprecate the built-in rootfs management in Garden in favour of GrootFS. At this point GrootFS is (very) experimental, but those who wish to test with it can find instructions in the grootfs-release repo. We will officially deprecate the built-in rootfs manager and then give a timeline for its removal in later releases.

Happy Gardening!

GRR v1.0.3

implements new BulkNetOut API for external networking plugins

GRR v1.0.2

• Swap to using iptables-restore due to regular iptables becoming extremely slow with lots of rules on recent kernels
• Minor changes to experimental image_plugin api

GRR v1.0.1

Minor fix and some work on the image plugin API

• Hostnames are now truncated where the handle is super-long (https://www.pivotaltracker.com/story/show/132858029)
• Various improvements to image_plugin API for grootfs

GRR v1.0.0

no big deal