pillar

EVE microservices for device and app instance management

For information about EVE see https://www.lfedge.org/projects/eve/

The onboarding of devices is done by scripts/device-steps.sh, and after onboarding that script proceeds to start the agents implementing the various microservices. That script runs on each boot. Once the agents are running they are operated from the controller using the API specified in https://github.com/lf-edge/eve-api

All agents are located in ./cmd. A potentially incomplete list of the agents is:

• ledmanager - make LEDs light/blink for feedback to the person installing the hardware
• nim - network interface manager (ensures that there is connectivity to the controller); manages radio-silence
• waitforaddr - merely waiting for a few minutes max for IP address(es) for a more orderly boot in the normal case
• vaultmgr - responsible of creation and operations over, encrypted vault(s) for mutable application images and device configuration data
• nodeagent - montior the device health, while node is in baseos upgrade or normal operation mode. Also orchestrates baseos installation and upgrade validation by interacting with baseosmgr and zedagent.
• zedagent - communicate using the device API to the controller to retrieve configuration and send status and metrics
• loguploader - send gzip logs to the controller for debugging of these agents
• baseosmgr - handle updates of the base OS (hypervisors plus all of the services which make up EVE) using dual partitions for fallback
• volumemgr - create volumes based on downloads or from scratch
• downloader - download objects like images and certificates
• verifier - verify cryptographic checksums and signatures on downloaded objects
• zedmanager - drive the application instance lifecycle
• zedrouter - drive the lifecycle for the connectivity for the instances. Includes services like DHCP, DNS, and Access Control Lists. Provides different connectivity like local, switch, cloud, and mesh networks
• domainmgr - interface with the hypervisor to start and stop application images. Includes performing device assignment
• identitymgr - used when mesh networks desire locally created key pairs for the cryptographic application instance identities
• zfsmanager - handle zfs devices managed by mdev
• tpmmgr - manages the Trusted Platform Module

In addition there are debugging tools like:

• diag - prints the state of the connectivity on the console each time there is a change
• ipcmonitor - subscribes to the agents/collections passed between the different microservices

In order to conserve filesystem space, as well as memory usage in Linux, where multiple copies of an executable binary are loaded only once into read-only memory, all of the agents above are built into a single executable (zedbox) and are differentiated based on the symbolic link (very similar to how BusyBox does it with traditional UNIX utilities).

The design pattern for each agent is documented in ./cmd/README.md. To add a new agent:

1. Create a directory for the agent in ./cmd
2. Design the agent according to the pattern in ./cmd/README.md
3. Add the agent to the list above
4. Add the agent's exposed implementation of types.AgentRunner to the map of entrypoints in zedbox.go.

Building

Generally, the build is done inside a Docker container to ensure environment consistency. This container is FROM golang:${GOVER}-alpine. When running via docker-for-mac or docker-for-windows, you can run in a container that is derived from the library golang image. However, when building on Linux, the output artifacts will have the wrong ownership. Thus, we build a simple special image that contains and writes to the correct user.

All output binaries and links are in dist/.

Make targets of note:

• make build: build pillar containerized
• make build BUILD=local: build pillar via your local golang installation
• make builder-image: build the builder image for your user
• make shell: launch a shell inside the builder image, whence you can run all commands. It also sets BUILD=local, so that you can run just make build inside.

States and their transitions

Object inside agents (more precisely inside pubsub) may have two descriptions:

1. ObjectConfig - description of object comes directly from config or generated during parsing of config
2. ObjectStatus - description of object generated by agent as a current state of processing of ObjectConfig

Those structures are described in types. Agents have publishers and subscribers via pubsub to types from other agents. ObjectStatus may have several states inside, generally, they have SwState, but may have additional booleans or SubStates described current state of object. Agents obtain ObjectConfig and do their work by creating ObjectStatus in Initial state and moving it to another values. ObjectStatus is sending through pubsub to another agents, which have subscribes, and actions are fired in them depending on the State. State transitions of the particular ObjectStatus should be written in one place (in one function, you can find several with names starts with doUpdate).

Actions may be sync (they must be executable in limited time and not depends on another, for example, check file or directory existence, touch file, make directory and so on) and async. The second variant is done by worker package. Worker instance has limited job queue and parallel workers in pool. Package worker provides utilities for creating background workers to perform tasks, that need to be queued up or will take a long time to run. In both cases, upon completion, actions call function with state transitions to toggle states if required.

Local Dependencies

In general, pillar depends on its own directory and subdirectories, and external packages, which can be found in go.mod. It also vendors its dependencies, which can be updated with go mod vendor.

In addition, pillar depends upon eve libraries from this repository. These libraries are their own go module. They must be treated like any other external module, imported via a fully pathed import, e.g. go get github.com/lf-edge/eve-libs@master.

It is imperative not to use replace directives in go.mod to point to local paths, e.g.

replace github.com/lf-edge/eve-libs => ../../libs

These will break various dependency chains and make it difficult to track what was included and licensing in final compiled binaries.

During local development only, you can use the above replace directives to test local changes to libs/ and their downstream impact. However, when testing is complete, do not commit the replace directive. Instead, commit the libs changes, merge them in, and then update the dependency here in pillar's go.mod like any other external dependency with explicit versioning.