Merge branch 'feature/vic-machine-service' [full ci] #6665

Use an OpenAPI (Swagger) document to define a preliminary API for vic-machine based on the vic-machine API design document (#6116). This document is expected to evolve as implementation proceeds.

Introduce a pair of handlers for listing information about VCHs, optionally scoped to a compute resource or datacenter. Include basic authentication and error handling, which should be improved on as development continues.

Introduce a pair of handlers for accessing information about the VCH with a given identifier within a vSphere target or named datacenter. There are ways in which this work is incomplete, marked by TODO: - Malformed managed object references are ignored - Malformed or unexpected PKI material is ignored

Introduce a pair of handlers for deploying new VCHs, as described by a JSON object, within a vSphere target or datacenter.

This was a mechanical change using the following commands: gvt delete github.com/tylerb/graceful gvt fetch -revision 4654dfbb6ad53cb5e27f37d99b02e16c1872fbbb \ https://github.com/tylerb/graceful This is needed to pick up the fix for tylerstillwater/graceful#85

Notes: * To disable HTTPS for testing, invoke the server with --scheme http * To configure https, use --tls-port, --tls-certificate, and --tls-key

Implement VCH certificate endpoint

* clean up after rebase * integration test added to 6-04 * integration test md file modified * integration test md file change

This could be squashed into "Define an OpenAPI document for vic-machine" before merging to master.

Organize the the imports in bufferedPipe.go in alphabetical order. Without this change, `make goimports` (run as a part of CI) fails.

Rename bufferedPipe.go and bufferedPipe_test.go (camel case) to buffered_pipe.go and buffered_pipe_test.go (snake case), which better follow the naming conventions of the repository.

Switch from pointer receiver to value receiver and add missing methods for wrappedError to ensure error handling code works as designed. Introduce unit tests to verify behavior.

To include the correct set of CORS headers on each response, we need to add an appropriate middleware. This change vendors the middleware recommended by go-swagger and its transitive dependencies. This was a mechanical change using the following command: gvt fetch -tag v1.1 github.com/rs/cors See also: go-swagger/go-swagger#481

Because the vSphere H5 client plugin and the vic-machine-server will be served from different hosts, the H5 client plugin will be making cross- origin requests. As these would normally be prevented by the browser's same-origin policy, the service must support responding to all requests (including preflight requests) with the appropriate CORS headers.

Introduce a single-test robot suite for the version endpoint as well as a utility resource for use by this suite and others within the group.

Introduce a robot suite for the VCH listing endpoints leveraging the group-scoped keywords introduced for other API tests. Test the successful operation of the endpoints as well as common error cases.

* vch log API handler stub added * vch log handler stub added * VCH log handler implemented * whitespaces * whitespace * whitespace * cleaned up integration test temp files * log API handler implemented, integration test added * remove log file download function from logger * whitespace * make the log files automatically sort by timestamp * .log filename suffix added to log files; sort files based on timestamp * sort log files based on timestamp * modified error message for finding VCH from ID * remove unused package declaration * fixed error code when provided VCH ID does not exist * VCH-log integration test md modified * 404 test for log API handler added after log file's removed * log handler integration test refactored * use status code constants from http package; minor code improvements in log API handler * minor bug fixes * const declaration * minor bug fix in common.go

…der (#6545) * create API updated to include log streaming to datastore feature * comment added for const in create API handler * Const name change in create API handler

* removed container store from cli and vch data * goimports

…6563) * trace.NewOperation pattern used for vic-machine service API * whitespace * whitespace * clean up trace.op pattern in API handler functions * bug fix: validateTarget takes in context since used by buildData

Fix a minor copy & paste error in the vic-machine-server Dockerfile.

* syslog support added to VCH create API * syslog addr added to inspect output * syslog_addr format changed in swagger * use status code in http package * empty syslog_addr string checking * swagger format changed for syslog_addr * roll back in swagger: only tcp and udp are accepted

Because JSON and Go have different models for inheritance, properly interpreting the structure of the API parameters can be confusing. In this case, we need to reference the nested units variable from the "subclass" instead of the units variable in the parent, which will always have its zero value (the empty string).

This change moves the logic which processes operations credentials up a level so that they will always be processed, even if no endpoint settings are provided.

As a part of handling a VCH creation API request, we pass the desired VCH configuration to the validator code. When the validator returns an error, we should report it to the user.

This change ensures that omitted (i.e., nil) managed objects are correctly handled in calls to the VCH creation API. This allows for a more minimal set of fields to be supplied.

To allow for profiling and debugging of the VCH Management API, run pprof on localhost:6060 by default. Different settings may be passed via command-line arguments or environment variables.

This change allows the vic-machine-server to accept a vSphere session clone ticket in the X-VMWARE-TICKET header of incoming requests. This ticket is used by govmomi to clone the user's session to allow the vic-machine-server govmomi client to make vSphere requests on the user's behalf.

Previously, the API handler for VCH creation was dropping the processed proxy settings. With this change, they are processed as they would be during CLI-based creation.

A previous change replaced most http status codes with the appropriate constants. This replaces a lingering instance.

For consistency with the rest of the product (and other products which deal with storage size), represent disk size using base 10 (KB, MB, GB, etc.) instead of base 2 (KiB, MiB, GiB, etc.).

Volume store locations are returned as a URL with a scheme of "ds", a host as the name of the datastore, and a path as a vSphere datastore path (which is prefixed by the name of the datastore in brackets). To return a value in the format that the user expects (i.e., in the format that was provided), we extract the actual path portion from the vSphere path so that the datastore name is not repeated in both the host and the path.

Implement a suite of positive and negative tests for API-based VCH creation. These tests are not intended to be exhaustive; we can use unit tests for that.

Refactor and clean up the robot test group for the VCH Management API.

In "Implement tests for the VCH Creation API", a change was made to the way that the PUBLIC_NETWORK environment variable was quoted. This worked for robot test group 23, but broke several others. This commit reverts the global change and handles the quoting issue in a localized way.

error output is formatted to string as well.

Make minor adjustments to formatting suggested during review of the merge commit for the VCH creation API feature branch.

Include issue numbers with all TODOs added as a part of the initial VCH creation API project, as suggested during review of the merge commit for the VCH creation API feature branch.

Clarify an unclear comment identified during review of the merge commit for the VCH creation API feature branch.

Remove the serviceapi target as a dependency for the appliance.iso and ensure all generated code is cleaned up.

Introduce a pair of handlers for deleting VCHs within a vSphere target or datacenter. By default, deletion includes the VCH and powered off containers. Deletion may optionally include powered on containers and/or volume stores as well. If any containers remain, the VCH is not deleted. If the VCH is not deleted, the response includes a non-2xx status code. Define a suite of end-to-end tests which verify the intended deletion behavior. End-to-end tests do not attempt to verify the behavior of concurrent operations.

Registry blacklist functionality was designed, but has not been fully implemented in the engine. Remove references to it from the API.

When creating a VCH via the API, we should not write key/certificate files to the API server's disk. This change introduces the behavior as a flag so that it could be exposed as an option for the CLI at some point in the future, but does not expose it at this time.

The vic-machine CLI has differing requirements for gateway routing information, depending on the type of network. According to the CLI help: - a client gateway must specify one or more routing destinations - a public gateway must not specify any routing destinations - a management gateway must specify one or more routing destinations - a container gateway must specify exactly one routing destination This does not seem to be enforced in code, and may simply be more of a suggestion about how these gateways should be used than a requirement. Update the parsing for client, public, and management to support all zero or more routing destinations in all cases; defer to the existing ProcessNetwork code to ensure consistent validation behavior now and in the future. Additionally update the parsing for container to provide a clear error message if the expected routing destination is not supplied.

The vic-machine CLI requires that static addresses are specified as a CIDR, which allows the static address and subnet mask to be supplied in a compact way on the command line. This pattern does not allow for static addresses to be expressed in terms of a hostname. Update the API to match this convention. We may wish to allow for more flexibility in the API in the future, but there's value in at least starting with consistent behavior.

The vic-machine CLI allows IP ranges to be expressed in CIDR notation or as simple ranges in some places, but requires that CIDR notation be used in others. Initially there was a hope of making the API behave more consistently, but that requires changes to underlying logic (some of which is not well covered by existing tests). For now, be consistent with the CLI.

The vic-machine CLI supports specifying a "trust level" for each container network using the --container-network-firewall option. Support the same functionality in the API. Additionally, fix a small bug with the way ip ranges are returned by the inspect API.

Use the client network address, not the public network address, when displaying the admin portal URI and the docker host information as a part of API responses. Additionally, eliminate code duplication.

Additionally, move the binary to a subdirectory of opt to allow the ISO files to be packaged with it. See also: * http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/opt.html

Log each request and response to a configurable destination. Use trace.Operation to associate the request and response information with the logs for the handler to allow for easy debugging, even when there are several requests being handled concurrently. Adjust logging in server's main.go to avoid printing to stdout. Relies on external configuration for log rotation.

For each conceptual vic-machine operation, create a trace.Operation with an operation-local logger. Emit log messages using the log methods on the operation instead of logrus global methods. This will write to the vic-machine.log using the global trace.Logger and to the user using the operation-local logger. This ensures that both the CLI and API can print log messages to both a local file and the user (via a TTY, if the CLI, and the datastore). * In the CLI, use the operation-local logger to print to the console (using the default Logrus TTY formatting) and write to both the datastore and vic-machine.log file with the global trace.Logger (using VIC's custom log formatter). * In the API, use the the operation-local logger to write to the datastore and write to the server's console with the global trace.Logger. In both cases, use VIC's custom log formatter. Update code in packages used by vic-machine to accept a context, coerce/convert it into a trace.Operation, and use its log methods. Improve readability of logs for concurrent operations by including a operation/context in trace.Begin calls when one is available. Retain configuration for the global logrus logger in vic-machine so that we "fail ugly" (i.e., have lines with mismatched log formats) on the CLI if something uses the old pattern, but don't lose information. Update tests which parse log messages to understand this new format instead of or, where necessary, in addition to old format(s).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge branch 'feature/vic-machine-service' [full ci] #6665

Merge branch 'feature/vic-machine-service' [full ci] #6665

Commits on Nov 20, 2017