Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go/runtime: Support hot-loading of runtime bundles #5961

Merged
merged 27 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
827f6bc
go/oasis-test-runner/scenario/e2e/runtime: Add additional logging
peternose Nov 21, 2024
c98c21c
go/worker/common/committee/group: Use runtime ID instead of runtime
peternose Nov 17, 2024
549d54f
go/runtime/history: Shorten name for has local storage worker flag
peternose Nov 17, 2024
e52eed0
go/runtime/registry/host: Simplify runtime host node
peternose Nov 17, 2024
d0bb785
go/runtime/registry/host: Remove function WaitHostedRuntime
peternose Nov 17, 2024
12a2b5c
go/runtime/registry/host: Provision every version only once
peternose Nov 19, 2024
312f123
go/runtime/registry: Add runtime constructor
peternose Nov 17, 2024
5231491
go/runtime/registry/config: Refactor function newRuntimeConfig
peternose Nov 17, 2024
aac8259
go/runtime/registry: Add support for adding new runtime versions
peternose Nov 19, 2024
6080a4c
go/worker/common/committee: Provision newly discovered runtime versions
peternose Nov 19, 2024
2bda379
go/runtime/bundle/manifest: Add version to component
peternose Dec 1, 2024
15f22f5
go/runtime/host: Replace runtime bundle with exploded components
peternose Nov 19, 2024
cd092b7
go/runtime/host/sandbox: Clear cmds before notifying of runtime start
peternose Dec 2, 2024
7aeb422
go/runtime/bundle: Add recommended filename to bundle
peternose Dec 2, 2024
9276539
go/runtime: Support hot-loading of runtime bundles
peternose Nov 23, 2024
28f70e2
go/control/api: Add bundle status to GetStatus output
peternose Nov 21, 2024
146aa95
go/runtime/bundle/discovery: Add timeout to http requests
peternose Dec 8, 2024
1802d4f
go/runtime/bundle/discovery: Add bundle size limit
peternose Dec 8, 2024
a7bab02
go/runtime/bundle/discovery: Lock mutex in Init as late as possible
peternose Dec 8, 2024
3a2a455
go/runtime/bundle/registry: Sort versions
peternose Dec 11, 2024
e6e72cf
go/runtime/bundle: Fix comments and errors
peternose Dec 11, 2024
efeadb0
go/runtime/bundle: Verify manifest hash when opening a bundle
peternose Dec 11, 2024
3b50086
go/runtime/bundle: Move component to a new file
peternose Dec 12, 2024
a9c7820
go/runtime/bundle: Make max bundle size configurable
peternose Dec 12, 2024
8af27bd
go/runtime/bundle: Close zip file if Open fails
peternose Dec 12, 2024
3bd356f
go/runtime/bundle: Use indirection when downloading bundles
peternose Dec 16, 2024
3716fee
go/runtime/bundle/discovery: Increase request timeout
peternose Dec 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .changelog/5962.cfg.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
go/runtime: Support hot-loading of runtime bundles

The node can now fetch and verify runtime bundles from remote repositories
and automatically update to new versions.

The following configuration option has been removed:

- `runtime.components`

The following configuration option has been deprecated:

- `runtime.config`

The following configuration options have been added:

- `runtime.runtimes.id` is the runtime identifier,

- `runtime.runtimes.components` is the list of components to configure,

- `runtime.runtimes.config` is the runtime local configuration,

- `runtime.runtimes.repositories` is the list of runtime specific URLs
used to fetch runtime bundle metadata,

- `runtime.repositories` is the list of global URLs used to fetch
runtime bundle metadata,

- `runtime.max_bundle_size` is the maximum allowed bundle size.
12 changes: 12 additions & 0 deletions go/common/version/version.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,18 @@ func (v Version) ValidateBasic() error {
return nil
}

// Cmp returns a negative number when the given version is larger, a positive
// number when smaller, and zero when both versions are equal.
func (v Version) Cmp(other Version) int {
if v.Major != other.Major {
return int(v.Major) - int(other.Major)
}
if v.Minor != other.Minor {
return int(v.Minor) - int(other.Minor)
}
return int(v.Patch) - int(other.Patch)
}

// ToU64 returns the version as platform-dependent uint64.
func (v Version) ToU64() uint64 {
return (uint64(v.Major) << 32) | (uint64(v.Minor) << 16) | (uint64(v.Patch))
Expand Down
36 changes: 36 additions & 0 deletions go/control/api/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,13 @@ import (
"github.com/oasisprotocol/oasis-core/go/common/crypto/signature"
"github.com/oasisprotocol/oasis-core/go/common/errors"
"github.com/oasisprotocol/oasis-core/go/common/node"
"github.com/oasisprotocol/oasis-core/go/common/version"
"github.com/oasisprotocol/oasis-core/go/config"
consensus "github.com/oasisprotocol/oasis-core/go/consensus/api"
p2p "github.com/oasisprotocol/oasis-core/go/p2p/api"
registry "github.com/oasisprotocol/oasis-core/go/registry/api"
block "github.com/oasisprotocol/oasis-core/go/roothash/api/block"
"github.com/oasisprotocol/oasis-core/go/runtime/bundle/component"
storage "github.com/oasisprotocol/oasis-core/go/storage/api"
upgrade "github.com/oasisprotocol/oasis-core/go/upgrade/api"
commonWorker "github.com/oasisprotocol/oasis-core/go/worker/common/api"
Expand Down Expand Up @@ -85,6 +87,9 @@ type Status struct {
// Runtimes is the status overview for each runtime supported by the node.
Runtimes map[common.Namespace]RuntimeStatus `json:"runtimes,omitempty"`

// Bundles is the status overview of known runtime bundles.
Bundles []BundleStatus `json:"bundles,omitempty"`

// Registration is the node's registration status.
Registration *RegistrationStatus `json:"registration,omitempty"`

Expand Down Expand Up @@ -188,6 +193,37 @@ type RuntimeStatus struct {
Provisioner string `json:"provisioner,omitempty"`
}

// BundleStatus is the per-runtime bundle status overview.
type BundleStatus struct {
// Name is the optional human readable runtime name.
Name string `json:"name,omitempty"`

// ID is the runtime identifier.
ID common.Namespace `json:"id"`

// Components contains statuses of the runtime components.
Components []ComponentStatus `json:"components,omitempty"`
}

// ComponentStatus is the component status overview.
type ComponentStatus struct {
// Kind is the component kind.
Kind component.Kind `json:"kind"`

// Name is the name of the component.
Name string `json:"name,omitempty"`

// Version is the component version.
Version version.Version `json:"version,omitempty"`

// Detached specifies whether the component was in a detached bundle.
Detached bool `json:"detached,omitempty"`

// Disabled specifies whether the component is disabled by default
// and needs to be explicitly enabled via node configuration to be used.
Disabled bool `json:"disabled,omitempty"`
kostko marked this conversation as resolved.
Show resolved Hide resolved
}

// SeedStatus is the status of the seed node.
type SeedStatus struct {
// ChainContext is the chain domain separation context.
Expand Down
4 changes: 2 additions & 2 deletions go/oasis-net-runner/fixtures/default.go
Original file line number Diff line number Diff line change
Expand Up @@ -241,11 +241,11 @@ func newDefaultFixture() (*oasis.NetworkFixture, error) {
GovernanceModel: registry.GovernanceEntity,
Deployments: []oasis.DeploymentCfg{
{
Version: rtVersion,
ValidFrom: 0,
Components: []oasis.ComponentCfg{
{
Kind: component.RONL,
Kind: component.RONL,
Version: rtVersion,
Binaries: map[node.TEEHardware]string{
tee: rt,
},
Expand Down
4 changes: 2 additions & 2 deletions go/oasis-node/cmd/node/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
"github.com/oasisprotocol/oasis-core/go/oasis-node/cmd/common/flags"
cmdGrpc "github.com/oasisprotocol/oasis-core/go/oasis-node/cmd/common/grpc"
cmdSigner "github.com/oasisprotocol/oasis-core/go/oasis-node/cmd/common/signer"
runtimeRegistry "github.com/oasisprotocol/oasis-core/go/runtime/registry"
"github.com/oasisprotocol/oasis-core/go/runtime/bundle"
workerStorage "github.com/oasisprotocol/oasis-core/go/worker/storage"
)

Expand Down Expand Up @@ -38,7 +38,7 @@ func init() {
for _, v := range []*flag.FlagSet{
cmdGrpc.ServerLocalFlags,
cmdSigner.Flags,
runtimeRegistry.Flags,
bundle.Flags,
workerStorage.Flags,
crash.InitFlags(),
} {
Expand Down
7 changes: 6 additions & 1 deletion go/oasis-node/cmd/node/node.go
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ func (n *Node) initRuntimeWorkers() error {
if err != nil {
return err
}
n.svcMgr.RegisterCleanupOnly(n.RuntimeRegistry, "runtime registry")
n.svcMgr.Register(n.RuntimeRegistry)

// Initialize the common worker.
n.CommonWorker, err = workerCommon.New(
Expand Down Expand Up @@ -353,6 +353,11 @@ func (n *Node) initRuntimeWorkers() error {
}

func (n *Node) startRuntimeWorkers() error {
// Start the runtime registry.
if err := n.RuntimeRegistry.Start(); err != nil {
return err
}

peternose marked this conversation as resolved.
Show resolved Hide resolved
// Start the common worker.
if err := n.CommonWorker.Start(); err != nil {
return err
Expand Down
45 changes: 45 additions & 0 deletions go/oasis-node/cmd/node/node_control.go
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,11 @@ func (n *Node) GetStatus(ctx context.Context) (*control.Status, error) {
return nil, fmt.Errorf("failed to get runtime status: %w", err)
}

bundles, err := n.getBundleStatus()
if err != nil {
return nil, fmt.Errorf("failed to get bundle status: %w", err)
}

kms, err := n.getKeymanagerStatus()
if err != nil {
return nil, fmt.Errorf("failed to get key manager worker status: %w", err)
Expand Down Expand Up @@ -165,6 +170,7 @@ func (n *Node) GetStatus(ctx context.Context) (*control.Status, error) {
Consensus: cs,
LightClient: lcs,
Runtimes: runtimes,
Bundles: bundles,
Keymanager: kms,
Registration: rs,
PendingUpgrades: pendingUpgrades,
Expand Down Expand Up @@ -199,6 +205,10 @@ func (n *Node) getRuntimeStatus(ctx context.Context) (map[common.Namespace]contr
runtimes := make(map[common.Namespace]control.RuntimeStatus)

for _, rt := range n.RuntimeRegistry.Runtimes() {
if !rt.IsManaged() {
continue
}

var status control.RuntimeStatus

// Fetch runtime registry descriptor. Do not wait too long for the descriptor to become
Expand Down Expand Up @@ -343,6 +353,41 @@ func (n *Node) getRuntimeStatus(ctx context.Context) (map[common.Namespace]contr
return runtimes, nil
}

func (n *Node) getBundleStatus() ([]control.BundleStatus, error) {
bundleRegistry := n.RuntimeRegistry.GetBundleRegistry()
manifests := bundleRegistry.GetManifests()
bundles := make([]control.BundleStatus, 0, len(manifests))

for _, manifest := range manifests {
explodedComponents, err := bundleRegistry.GetComponents(manifest.ID, manifest.GetVersion())
if err != nil {
return nil, err
}

components := make([]control.ComponentStatus, 0, len(explodedComponents))
for _, comp := range explodedComponents {
component := control.ComponentStatus{
Kind: comp.Kind,
Name: comp.Name,
Version: comp.Version,
Detached: comp.Detached,
Disabled: comp.Disabled,
}
components = append(components, component)
}

bundle := control.BundleStatus{
Name: manifest.Name,
ID: manifest.ID,
Components: components,
}

bundles = append(bundles, bundle)
}

return bundles, nil
}

func (n *Node) getKeymanagerStatus() (*keymanagerWorker.Status, error) {
if n.KeymanagerWorker == nil || !n.KeymanagerWorker.Enabled() {
return nil, nil
Expand Down
5 changes: 3 additions & 2 deletions go/oasis-node/cmd/storage/storage.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
"github.com/oasisprotocol/oasis-core/go/config"
cmdCommon "github.com/oasisprotocol/oasis-core/go/oasis-node/cmd/common"
roothash "github.com/oasisprotocol/oasis-core/go/roothash/api"
"github.com/oasisprotocol/oasis-core/go/runtime/bundle"
"github.com/oasisprotocol/oasis-core/go/runtime/history"
"github.com/oasisprotocol/oasis-core/go/runtime/registry"
db "github.com/oasisprotocol/oasis-core/go/storage/mkvs/db/api"
Expand Down Expand Up @@ -284,8 +285,8 @@ func doRenameNs(_ *cobra.Command, args []string) error {

// Register registers the client sub-command and all of its children.
func Register(parentCmd *cobra.Command) {
storageMigrateCmd.Flags().AddFlagSet(registry.Flags)
storageCheckCmd.Flags().AddFlagSet(registry.Flags)
storageMigrateCmd.Flags().AddFlagSet(bundle.Flags)
storageCheckCmd.Flags().AddFlagSet(bundle.Flags)
storageCmd.AddCommand(storageMigrateCmd)
storageCmd.AddCommand(storageCheckCmd)
storageCmd.AddCommand(storageRenameNsCmd)
Expand Down
4 changes: 2 additions & 2 deletions go/oasis-node/node_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,10 @@ import (
registryTests "github.com/oasisprotocol/oasis-core/go/registry/tests"
roothash "github.com/oasisprotocol/oasis-core/go/roothash/api"
roothashTests "github.com/oasisprotocol/oasis-core/go/roothash/tests"
"github.com/oasisprotocol/oasis-core/go/runtime/bundle"
runtimeClient "github.com/oasisprotocol/oasis-core/go/runtime/client/api"
clientTests "github.com/oasisprotocol/oasis-core/go/runtime/client/tests"
runtimeConfig "github.com/oasisprotocol/oasis-core/go/runtime/config"
runtimeRegistry "github.com/oasisprotocol/oasis-core/go/runtime/registry"
scheduler "github.com/oasisprotocol/oasis-core/go/scheduler/api"
schedulerTests "github.com/oasisprotocol/oasis-core/go/scheduler/tests"
staking "github.com/oasisprotocol/oasis-core/go/staking/api"
Expand Down Expand Up @@ -164,7 +164,7 @@ func newTestNode(t *testing.T) *testNode {

config.GlobalConfig.Common.DataDir = dataDir
config.GlobalConfig.Common.Log.File = filepath.Join(dataDir, "test-node.log")
viper.Set(runtimeRegistry.CfgDebugMockIDs, []string{
viper.Set(bundle.CfgDebugMockIDs, []string{
testRuntimeID.String(),
})
for _, kv := range testNodeStaticConfig {
Expand Down
8 changes: 8 additions & 0 deletions go/oasis-test-runner/oasis/keymanager.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
"github.com/oasisprotocol/oasis-core/go/oasis-test-runner/env"
registry "github.com/oasisprotocol/oasis-core/go/registry/api"
"github.com/oasisprotocol/oasis-core/go/runtime/bundle"
runtimeCfg "github.com/oasisprotocol/oasis-core/go/runtime/config"
runtimeConfig "github.com/oasisprotocol/oasis-core/go/runtime/config"
keymanagerConfig "github.com/oasisprotocol/oasis-core/go/worker/keymanager/config"
)
Expand Down Expand Up @@ -296,7 +297,14 @@ func (km *Keymanager) ModifyConfig() error {
km.Config.Runtime.Provisioner = km.runtimeProvisioner
km.Config.Runtime.SGXLoader = km.net.cfg.RuntimeSGXLoaderBinary
km.Config.Runtime.AttestInterval = km.net.cfg.RuntimeAttestInterval

rtCfg := runtimeCfg.RuntimeConfig{
ID: km.runtime.cfgSave.id,
}

km.Config.Runtime.Runtimes = append(km.Config.Runtime.Runtimes, rtCfg)
km.Config.Runtime.Paths = append(km.Config.Runtime.Paths, km.runtime.BundlePaths()...)
km.Config.Runtime.Repositories = []string{fmt.Sprintf("http://127.0.0.1:%d", km.net.getProvisionedPort(netPortRepository))}

km.Config.Keymanager.RuntimeID = km.runtime.ID().String()
km.Config.Keymanager.PrivatePeerPubKeys = km.privatePeerPubKeys
Expand Down
30 changes: 21 additions & 9 deletions go/oasis-test-runner/oasis/network.go
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,9 @@ type Network struct { // nolint: maligned

iasProxy *iasProxy

cfg *NetworkCfg
nextNodePort uint16
cfg *NetworkCfg
nextPort uint16
ports map[string]uint16

logWatchers []*log.Watcher

Expand Down Expand Up @@ -296,7 +297,7 @@ func (net *Network) GetNamedNode(defaultName string, cfg *NodeCfg) (*Node, error
Name: name,
net: net,
dir: nodeDir,
assignedPorts: map[string]uint16{},
ports: map[string]uint16{},
hostedRuntimes: map[common.Namespace]*hostedRuntime{},
}

Expand Down Expand Up @@ -986,6 +987,16 @@ func (net *Network) provisionNodeIdentity(dataDir *env.Dir, seed string) (signat
return nodeIdentity.NodeSigner.Public(), nodeIdentity.P2PSigner.Public(), sentryCert, nil
}

func (net *Network) getProvisionedPort(portName string) uint16 {
port, ok := net.ports[portName]
if !ok {
port = net.nextPort
net.nextPort++
net.ports[portName] = port
}
return port
}

// New creates a new test Oasis network.
func New(env *env.Env, cfg *NetworkCfg) (*Network, error) {
baseDir, err := env.NewSubDir("network")
Expand Down Expand Up @@ -1034,12 +1045,13 @@ func New(env *env.Env, cfg *NetworkCfg) (*Network, error) {
}

net := &Network{
logger: logging.GetLogger("oasis/" + env.Name()),
env: env,
baseDir: baseDir,
cfg: &cfgCopy,
nextNodePort: baseNodePort,
errCh: make(chan error, maxNodes),
logger: logging.GetLogger("oasis/" + env.Name()),
env: env,
baseDir: baseDir,
cfg: &cfgCopy,
nextPort: basePort,
ports: make(map[string]uint16),
errCh: make(chan error, maxNodes),
}

// Pre-provision node objects if they were listed in the top-level network fixture.
Expand Down
Loading
Loading