Diego v0.1437.0

Changes from v0.1436.0 to v0.1437.0

• Depends on garden-linux-release v0.308.0.
• Depends on etcd-release 16.

Other significant changes

• As a Diego developer, I would like a benchmark suite for the nsync-bulker's retrieval of DesiredLRP data (in flight)
• As a Diego team member, I would like to run the bbs benchmark suite in CI (in flight)
• Include bosh-lite versions of manifests for the DATs and Diego Smoke Test errands in diego-release
• Bump cflinuxfs2 stack to 1.11.0+
• Auction state requests should be resilient to network errors
• Flakey diego-ssh/scp/scp_test
• cloudfoundry-incubator/rep #4: fixes the tests on windows

BOSH job changes

None.

BOSH property changes

• Add benchmark-bbs.log_level: Control log level of test suite in BBS benchmark errand.

Diego v0.1436.0

Changes from v0.1435.0 to v0.1436.0

• Depends on garden-linux-release v0.308.0.
• Depends on etcd-release 15.

Significant changes

• As a Diego operator, I would like the README to use customary methods of generating CF manifests for bosh-lite
• As a Diego developer, I would like a benchmark suite for the nsync-bulker's retrieval of DesiredLRP data
• BBS migration 1441411196 panics if deserializing encrypted data
• flake: encryption format.Encoding on unencrypted data
• Tune nf_conntrack_max value on Diego cells

BOSH job changes

• Add benchmark-bbs job for certain Diego benchmarks against the BBS API. As with the acceptance-tests and smoke-tests jobs, this is an errand that runs a test suite. It is not presently suitable to run against a production deployment, as the benchmark suite will alter the BBS database.

BOSH property changes

• Add properties under benchmark-bbs.* for the BBS-benchmarking errand.

Diego v0.1435.0

Changes from v0.1434.0 to v0.1435.0

• Depends on garden-linux-release v0.307.0.
• Depends on etcd-release 14.

Significant changes

• Pair with MEGA to incorporate etcd-release into diego-release
• As a Diego operator, I would like to allow developers to stage images from insecure private registries specified in my deployment manifest (in flight)
• As a Diego operator, I would like the README to use customary methods of generating CF manifests for bosh-lite
• Diego manifest generation should extract the director_uuid from the CF manifest
• Include bosh-lite versions of manifests for the DATs and Diego Smoke Test errands in diego-release (in flight)
• Re-run 10-cell performance experiment
• As a Diego developer, I would like a benchmark suite for the BBS's retrieval of DesiredLRP and ActualLRP data during LRP convergence
• A latency metric should not be emitted for requests to the BBS's events endpoints
• cloudfoundry-incubator/route-emitter #2: Include Route Service URL in Route Registration
• cloudfoundry-incubator/executor #14: fixes a typo, err -> info.Err
• Explore a minimal Diego deployment on AWS with continual stress to reproduce auction communication failures
• Update golang.org/x/crypto
• bump gomega in diego-release to include VerifyProtoRepresenting and RespondWithProto
• replace db/consul with locket
• remove garden-linux support blobs from diego-release
• remove dead code from runtime-schema

BOSH job changes

• etcd job removed from diego-release in favor of the same job from etcd-release.

BOSH property changes

• Add diego.stager.insecure_docker_registry_list: List of insecure Docker registries to allow access to when staging.
• Remove diego.stager.insecure_docker_registry.

Diego 0.1434.0

Changes from 0.1432.0 to 0.1434.0

• Depends on garden-linux-release v0.307.0.

Breaking changes from 0.1432.0

Buildpack Staging Response

• The staging response from Diego to CC has changed format to be better suited for the process-types work coming in the v3 CC API. Once the CF deployment has been updated to a version past commit ea88d1e, which will be in CF v219, CC and Diego will be able to communicate correctly during staging. Associated to CAPI story "the buildpack lifecycle should return all process types in the staging response".

Other significant changes

• Garden's Ping method should return an error if the btrfs graph path is not writable
• Diego should not set CF_STACK env var for running CF instances
• The BBS's etcd clients should avoid doing SSL handshakes with etcd unnecessarily
• BBS clients should avoid doing SSL handshakes with the BBS server unnecessarily
• cloudfoundry-incubator/windows_app_lifecycle #3: OutputMetadata conforms to Buildpack Staging Response protocol
• Update diego-release to golang 1.4.3
• Bump cloudfoundry-incubator/garden in diego-release
• The Diego BOSH manifest should configure garden to enable graph cleanup by default

BOSH job changes

None.

BOSH property changes

• Add parameters for tuning SSL connections between BBS clients and servers:
  • Properties:
    • diego.*.bbs.client_session_cache_size: Size of session ID cache for component's BBS client to maintain.
    • diego.*.bbs.max_idle_conns_per_host: Number of idle connections for each BBS client to maintain to BBS servers
  • Jobs:
    • auctioneer
    • converger
    • nsync
    • rep
    • route_emitter
    • ssh_proxy
    • stager
    • tps
• Add diego.bbs.etcd.client_session_cache_size: Size of session ID cache for BBS's etcd client to maintain.
• Add diego.bbs.etcd.max_idle_conns_per_host: Number of idle connections for BBS's etcd client to maintain to etcd.

Overrides of garden-linux-release property defaults

The spiff-based manifest-generation templates in diego-release provide values for the following properties in garden-linux-release:

• garden.enable_graph_cleanup: Default to true.
• garden.persistent_image_list: Default to ["/var/vcap/packages/rootfs_cflinuxfs2/rootfs"]

Values in the property-overrides.yml stub can override both of these defaults.

Diego 0.1433.0

Create final release 0.1433.0

Diego 0.1432.0

Changes from 0.1431.0 to 0.1432.0

• Depends on garden-linux-release v0.306.0.

Breaking changes from 0.1431.0

SSH Authentication to CF Instances

Associated to Diego story "The Diego SSH Proxy no longer accepts a user's access token as an SSH password for CF app instances".

Diego's SSH proxy no longer accepts a CF user's access token as a password for access to a CF app instance. It will instead accept only a one-time authorization code issued by UAA for its client. This client must also be registered with the UAA: for example, this client is registered for BOSH-lite deployments. As long as the name of the client is ssh-proxy, CC will advertise the correct client name in its /v2/info endpoint, and the Diego manifest-generation templates will flow the client secret to the SSH Proxy job.

For SSH access to CF app instances running on this release, we recommend you upgrade to version 0.2.0 or later of the Diego SSH plugin, or consult the diego-ssh repo for the current curl-based instructions to request a code from UAA.

Other significant changes

SSH

• The Diego SSH Proxy can receive an authorization code as the SSH password to access a CF app instance
• The SSH plugin provides a command to print a one-time authorization code issued for the SSH proxy client
• The SSH plugin establishes SSH connections to CF app instances by sending an authorization code as the SSH password
• As a CF user, when I establish a port-forwarding session with the SSH plugin, I expect it not to drop when going through a load balancer with an idle timeout
• Audit records for SSH access to CF instances should include which index was accessed

Performance

• BBS clients should avoid doing SSL handshakes with the BBS server unnecessarily (still in flight)
• The BBS's etcd clients should avoid doing SSL handshakes with etcd unnecessarily (still in flight)
• Change default route-emitter communication timeout to 30s

Misc

• Remove the Receptor
• Merge PRs for Routing info change
• operator should be able to verify that when mapping apps to a route already bound to a service instance for which the broker returned a route_service_url, CC sends updateDesiredLRP calls to Diego
• Merge PRs for CAPI backwards-incompatible change to Diego staging response
• the buildpack lifecycle should return all process types in the staging response
• The Diego BOSH manifest should include the cflinuxfs2 rootfs path in the garden.persistent_image_list property
• As a Diego operator, if a cell rep fails to start because it cannot ping Garden successfully, I can see a metric reporting how long the rep has been stalled
• Bump cflinuxfs2 stack to 1.9.0+
• Document how to secure BBS with mutual SSL auth in the diego-release README
• cloudfoundry-incubator/rep #3: Update client test to mach go 1.5 http errors
• cloudfoundry-incubator/executor #13: Improve leak detection in keyed lock test
• cloudfoundry/dropsonde #10: Added HasValue to FakeMetricSender

BOSH job changes

• Remove receptor job. The Lattice team has taken ownership of the receptor component and will be maintaining it from now on.

BOSH property changes

• Add diego.ssh_proxy.uaa_token_url: URL for the SSH proxy to use to request an access token from the UAA in exchange for its one-time auth code.
• Add diego.ssh_proxy.uaa_secret: Client secret for the SSH proxy to supply to UAA.
• Remove all properties under diego.receptor.

Diego 0.1431.0

Changes from 0.1430.0 to 0.1431.0

• Depends on garden-linux-release v0.305.0.

Breaking changes

Auction request payloads

Associated to Diego story "The auction should only send resources and identifiers back and forth". This may cause instance downtime during an upgrade from 0.1430.0 and earlier.

Mutual SSL Auth to BBS

Associated to Diego story "All communication with the BBS should be secured via mutually-authenticated SSL". By default, the BBS will now require mutual SSL authentication for access to its API endpoints. If this is enabled, components on an older release will be unable to communicate with the BBS when deploying an update, so cells may be unable to evacuate.

To configure the BBS with SSL correctly, it is easiest to use the scripts/generate-bbs-certs script to generate a CA certificate and key and certificates and keys for the BBS server and its clients. The contents of these certificates and client and server keys must then be included in the deployment manifest. If using the spiff-based manifest-generation tooling, these values can be included in the property-overrides.yml stub once and will flow to the BBS server and its clients.

Significant changes

• DesiredLRP data should be split across separate records
• As a BBS client, I can efficiently get frequently accessed data for all DesiredLRPs in a domain
• NSYNC's bulker should fetch the minimal set of DesiredLRP data
• Route-Emitter's bulk loop should fetch the minimal set of DesiredLRP data
• If a migration fails, BOSH aborts the deploy and I should be able to BOSH deploy the previously deployed release and recover.
• If no /version key is present in etcd, the BBS should not run any migrations
• As a Diego developer, I would like to run vizzini as an errand
• As a Diego operator, I can specify a set of decryption keys to use to decrypt data at rest, with the BBS migrating data to the designated active key in the set
• Diego etcd on bosh-lite should default to requiring ssl
• As a Diego operator, I can opt out of the SSH DATs that do not use the plugin
• vizzini test errand runs against BBS with mutual SSL auth enabled
• Provide vizzini job with BBS URL and local consul agent URL

BOSH job changes

• Added vizzini job to run the vizzini test suite as an errand.

BOSH property changes

• Added acceptance_tests.skip_ssh_without_plugin_tests: When true, skip SSH DATs that do not use the SSH plugin.
• Added properties for vizzini job:
  • vizzini.bbs.api_location: Address for vizzini to reach the BBS.
  • vizzini.routable_domain_suffix: Domain to use for vizzini to register routes during the test.
  • vizzini.nodes: Number of tests to run in parallel.
  • vizzini.verbose: Whether to log verbosely during the test run.
• Added BBS encryption properties:
  • diego.bbs.encryption_keys: List of label/passphrase pairs available to the BBS for encryption.
  • diego.bbs.active_key_label: Label of the encryption key to be used to encrypt the database.
• Added BBS mutual SSL auth properties:
  • Properties for BBS server job:
    • diego.bbs.require_ssl: whether the BBS requires SSL for communication.
    • diego.bbs.ca_cert: CA certificate used to sign BBS client and server SSL certificates.
    • diego.bbs.server_cert: SSL certificate that the BBS presents.
    • diego.bbs.server_key: Private key paired with the BBS's SSL certificate.
  • New BBS properties for client jobs:
    • Properties:
      • diego.*.bbs.ca_cert
      • diego.*.bbs.client_cert
      • diego.*.bbs.client_key
      • diego.*.bbs.require_ssl
    • Jobs:
      • auctioneer
      • converger
      • nsync
      • receptor
      • rep
      • route_emitter
      • ssh_proxy
      • stager
      • tps
      • vizzini
• Changed diego.*.bbs.api_url to diego.*.bbs.api_location for all jobs using the old property.
• Removed etcd communication properties from Diego core jobs:
  • Properties:
    • diego.*.etcd.machines
    • diego.*.etcd.ca_cert
    • diego.*.etcd.client_cert
    • diego.*.etcd.client_key
    • diego.*.etcd.require_ssl
  • Jobs:
    • auctioneer
    • converger
    • receptor
    • rep

Diego 0.1430.0

Version 0.1430.0 of Diego is recommended for use with CF v218.

Changes from 0.1428.0 to 0.1430.0

• Depends on garden-linux-release v0.305.0.

Configuration notes

• If upgrading from 0.1428.0 to 0.1430.0, we recommend you to deploy 0.1428.0 with the diego.bbs.serialization_format BOSH property set to proto. 0.1430.0 contains a BBS migration that encodes all the data in etcd as protobufs, which the first BBS server that receives the update will run. Setting this property to proto in advance guarantees that the other BBS servers will not accidentally write JSON-encoded records back into etcd before they also update to 0.1430.0. Note that unfortunately this property is not configurable via the manifest-generation templates in 0.1428.0, but it can be added directly to the properties section of the BOSH manifest.

Significant changes

• cloudfoundry-incubator/diego-release #72: ./scripts/update should fail fast when permission denied
• All CC-Bridge communication should happen directly with the BBS
• All Route-Emitter communication should happen directly with the BBS
• All SSH-Proxy communication should happen directly with the BBS
• All access to the BBS should go through one, master-elected, BBS server
• BBS server should emit metrics, remove the metrics server
• After a BOSH deploy, all data in the BBS should be stored in base64 encoded protobuf format
• If the Rep repeatedly fails to mark its ActualLRPs as EVACUATING it should fail to drain and the BOSH deploy should abort.
• Bump up the converger http timeout to one minute
• Never log environment variables and commands/arguments
• BBS Client should retry requests that fail because the BBS is migrating/lost the lock
• update cflinuxfs2 rootfs to 1.8.0+
• cloudfoundry-incubator/diego-ssh #5: Add -skipCertVerify to ssh-proxy
• The windows app lifecycle bundle should include a dummy diego-sshd executable
• provide user with helpful error message when they push a non-valid app

BOSH job changes

• Remove runtime_metrics_server job: the BBS server now emits Task and LRP metrics during convergence, and periodically emits etcd metrics.

BOSH property changes

• Add diego.nsync.bbs.api_url: Address for the Nsync processes to contact the BBS server.
• Add diego.route_emitter.bbs.api_url: Address for the Route-Emitter to contact the BBS server.
• Add diego.ssh_proxy.bbs.api_url: Address for the SSH-Proxy to contact the BBS server.
• Add diego.ssh_proxy.diego_credentials: Credentials to be used with the Diego authentication method.
• Add diego.tps.bbs.api_url: Address for the TPS processes to contact the BBS server.
• Remove diego.bbs.serialization_format.
• Remove diego.nsync.diego_api_url.
• Remove diego.route_emitter.diego_api_url.
• Remove diego.ssh_proxy.diego_api_url.
• Remove diego.tps.diego_api_url.

Diego 0.1429.0

Create final release 0.1429.0

Diego 0.1428.0

Version 0.1428.0 of Diego is recommended for use with CF v217.

Changes from 0.1412.0 to 0.1428.0

• garden-linux-release
  is no longer a submodule of diego-release, therefore is not bundled with
  Diego and needs to be uploaded to BOSH separately.
• Depends on garden-linux-release v0.303.0.

Breaking changes

• As a Diego operator, I can colocate all the diego-release jobs onto a single
  VM: This changes the
  default ports of some of the Diego components, meaning that they will no
  longer be reachable on the previous ports. Specifically, the default port for
  the receptor component changed from 8888 to 8887.
• Refactor error handling in the
  BBS: This changes how
  errors are serialized internally and may cause inter-component communication to
  fail unexpectedly during a deploy. No major outage is predicted
  for this change.
• As a Diego operator, I can run the static file-server and the cc-uploader
  server on separate VMs:
  This introduces a new component, cc-uploader, split out of the file-server.
  This change may cause staging failures during the deploy until the at least one of
  the cc-uploaders is up and running.
• Make BBS API consistently
  RPC-ful: This is a
  re-organization of the internal BBS API, changing every endpoint from a
  REST-like interface to a RPC-like interface. During the deploy, it will cause
  requests to fail, and may cause LRPs to lose routability temporarily.

Other significant changes

• Consul agents and servers communicate securely to the consul
  cluster
• DATs Java buildpack test should not specify the buildpack
  explicitly
• Update cflinuxfs2 stack to
  1.4.0+
• The BBS server handles calling the task completion callback
  URL
• Remove extra disk allocation for staging
  tasks
• Diego should provide the appropriate instance environment variables to its
  containers
• As a CF app developer, when staging an app on Diego fails, I would like to
  see a specific error type depending on the type of
  failure
• Fix instance env var
  names
• As a Diego operator, I can configure garden's allow_host_access flag
  through the manifest-generation
  templates
• Update cflinuxfs2 rootfs to
  1.5.0+
• clean up and integration test the Task
  BBS
• Clean up Task BBS code from
  runtime-schema
• Rename 'internal' packages to be compatible with Go 1.5 package
  conventions
• As a Diego operator, I want always to allow 'root' actions on the
  cells
• DiskMB limit should apply to total disk size for an instance with a docker
  rootfs
• Add test coverage for DesiredLRP event
  stream
• auctioneer should time out requests to cells
  correctly
• Executor GardenStore RemainingResources and TotalResources can now
  fail
• All Stager communication should happen directly with the
  BBS
• Only detect buildpack on java
  DAT
• Diego core components evacuate ActualLRPs through the BBS API
  server
• Diego core components create and update DesiredLRPs through the BBS API
  server
• Diego core components converge LRPs through the BBS API
  server
• Diego core components desire a task through the BBS API
  server
• Diego core components take a task through its lifecycle through the BBS API
  server
• Diego core components trigger task convergence through the BBS API
  server
• As a CF developer, I expect my app to have CC's global running environment
  variables in its environment when running on
  Diego
• The BBS server serializes different versions of Task data based on BOSH-deployed config
• As a CF user, I would like my cached docker images to start regardless of
  the IP addresses of the docker registry
  nodes
• All BBS serialization can be encoded based on BOSH-deployed config
• As a Diego operator or developer, I would like Diego to consume
  garden-linux-release

BOSH job changes

• Add cc_uploader job: contains cc-upload-brokering handlers formerly present in the file-server.
• Add rootfses job: unpackages the cflinuxfs2 rootfs.
• Remove garden-linux job.

BOSH property changes

• Move diego.file_server.cc.* to diego.cc_uploader.cc.*
  • This includes: diego.cc_uploader.cc.base_url,
    diego.cc_uploader.cc.basic_auth_password,
    diego.cc_uploader.cc.job_polling_interval_in_seconds,
    diego.cc_uploader.cc.staging_upload_user and
    diego.cc_uploader.cc.staging_upload_password.
  • Also keeps diego.file_server.log_level and diego.cc_uploader.log_level
    available.
• Add diego.cc_uploader.address: Address on which cc-uploader handles requests.
• Add diego.cc_uploader.debug_addr: Address for cc-uploader debug server.
• Add diego.cc_uploader.cc.external_port: CC Port for cc-uploader.
• Add diego.rep.evacuation_timeout_in_seconds: The time to wait for evacuation to complete in seconds.
• Add diego.bbs.serialization_format: Default format for BBS records.
• Add diego.converger.bbs.api_url: Address for the converger to contact the BBS server.
• Add diego.stager.bbs.api_url: Address for the stager to contact the BBS server.
• Add diego.stager.cc_uploader_url: Address for the stager to contact the cc-uploader.
• Add diego.stager.docker_registry_address: Address for stager to contact the caching docker registry.
• Remove diego.auctioneer.receptor_task_handler_url.
• Remove diego.converger.receptor_task_handler_url.
• Remove diego.rep.receptor_task_handler_url.
• Remove diego.stager.diego_api_url.
• Remove diego.executor.allow_privileged: Executor now always allows privileged actions (those running as 'root').
• Remove diego.garden-linux in favor of garden:
  • diego.garden-linux.listen_network => garden.listen_network
  • diego.garden-linux.listen_address => garden.listen_address
  • diego.garden-linux.allow_networks => garden.allow_networks
  • diego.garden-linux.insecure_docker_registry_list => garden.insecure_docker_registry_list
  • diego.garden-linux.mtu => garden.network_mtu
  • Add garden.deny_networks: List of CIDR blocks to which containers will be denied access.
  • A full list of the garden-linux-release properties can be found here