Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Corrected the confusion of steps in the document due to formatting #220

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 68 additions & 68 deletions content/en/docs/logging/pluggable-types.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ weight: 1830

Rekor supports pluggable types (aka different schemas) for entries stored in the transparency log. This will allow you to develop your own manifest type in your preferred formatting style (json|yaml|xml).

### Currently supported types
## Currently supported types

The list of currently supported types and their schema is [maintained in the repository](https://github.com/sigstore/rekor/tree/main/pkg/types#currently-supported-types).

## Base schema

The base schema for all types is modeled off of the schema used by Kubernetes and can be found in `openapi.yaml` as `#/definitions/ProposedEntry`:

```
```bash
definitions:
ProposedEntry:
type: object
Expand All @@ -34,83 +34,83 @@ To add a new type (called `newType` in this example):

1. Add a new definition in `openapi.yaml` that is a derived type of ProposedEntry (expressed in the `allOf` list seen below); for example:

```yaml
newType:
type: object
description: newType object
allOf:
- $ref: "#/definitions/ProposedEntry"
- properties:
version:
type: string
metadata:
type: object
additionalProperties: true
data:
type: object
$ref: "pkg/types/newType/newType_schema.json"
required:
- version
- data
additionalProperties: false
```

> Note: the `$ref` feature can be used to refer to an externally defined JSON schema document; however it is also permitted to describe the entirety of the type in valid Swagger (aka OpenAPI) v2 format within `openapi.yaml`.
```yaml
newType:
type: object
description: newType object
allOf:
- $ref: "#/definitions/ProposedEntry"
- properties:
version:
type: string
metadata:
type: object
additionalProperties: true
data:
type: object
$ref: "pkg/types/newType/newType_schema.json"
required:
- version
- data
additionalProperties: false
```

> Note: the `$ref` feature can be used to refer to an externally defined JSON schema document; however it is also permitted to describe the entirety of the type in valid Swagger (aka OpenAPI) v2 format within `openapi.yaml`.

2. Create a subdirectory under `pkg/types/` with your type name (e.g. `newType`) as a new Go package

3. In this new Go package, define a struct that implements the `TypeImpl` interface as defined in `pkg/types/types.go`:

```go
type TypeImpl interface {
CreateProposedEntry(context.Context, version string, ArtifactProperties) (models.ProposedEntry, error)
DefaultVersion() string
SupportedVersions() []string
IsSupportedVersion(version string) bool
UnmarshalEntry(pe models.ProposedEntry) (EntryImpl, error)
}
```

- `CreateProposedEntry` creates an instance of a proposed entry based on the specified API version and the provided artifact properties
- `DefaultVersion` returns the default API version string across all types (to be used if a caller does not specify an explicit version)
- `SupportedVersions` returns the list of all API version strings that can currently be inserted into the log.
- `IsSupportedVersion` returns a boolean denoting whether the specified version could be inserted into the log
- `UnmarshalEntry` will be called with a pointer to a struct that was automatically generated for the type defined in `openapi.yaml` by the [go-swagger](http://github.com/go-swagger/go-swagger) tool used by Rekor
- This struct will be defined in the generated file at `pkg/generated/models/newType.go` (where `newType` is replaced with the name of the type you are adding)
- This method should return a pointer to an instance of a struct that implements the `EntryImpl` interface as defined in `pkg/types/types.go`, or a `nil` pointer with an error specified

Also, the `Kind` constant must return the _exact_ same string as you named your new type in `openapi.yaml` (e.g. "`newType`")
```go
type TypeImpl interface {
CreateProposedEntry(context.Context, version string, ArtifactProperties) (models.ProposedEntry, error)
DefaultVersion() string
SupportedVersions() []string
IsSupportedVersion(version string) bool
UnmarshalEntry(pe models.ProposedEntry) (EntryImpl, error)
}
```

- `CreateProposedEntry` creates an instance of a proposed entry based on the specified API version and the provided artifact properties
- `DefaultVersion` returns the default API version string across all types (to be used if a caller does not specify an explicit version)
- `SupportedVersions` returns the list of all API version strings that can currently be inserted into the log.
- `IsSupportedVersion` returns a boolean denoting whether the specified version could be inserted into the log
- `UnmarshalEntry` will be called with a pointer to a struct that was automatically generated for the type defined in `openapi.yaml` by the [go-swagger](http://github.com/go-swagger/go-swagger) tool used by Rekor
- This struct will be defined in the generated file at `pkg/generated/models/newType.go` (where `newType` is replaced with the name of the type you are adding)
- This method should return a pointer to an instance of a struct that implements the `EntryImpl` interface as defined in `pkg/types/types.go`, or a `nil` pointer with an error specified

Also, the `Kind` constant must return the _exact_ same string as you named your new type in `openapi.yaml` (e.g. "`newType`")

4. Also in this Go package, provide an implementation of the `EntryImpl` interface as defined in `pkg/types/entries.go`:

```go
type EntryImpl interface {
APIVersion() string // the supported versions for this implementation
IndexKeys() ([]string, error) // the keys that should be added to the external index for this entry
Canonicalize(ctx context.Context) ([]byte, error) // marshal the canonical entry to be put into the tlog
Unmarshal(e models.ProposedEntry) error // unmarshal the abstract entry into the specific struct for this versioned type
CreateFromArtifactProperties(context.Context, ArtifactProperties) (models.ProposedEntry, error)
Verifier() (pki.PublicKey, error)
Insertable() (bool, error) // denotes whether the entry that was unmarshalled has the writeOnly fields required to validate and insert into the log
}
```

- `APIVersion` should return a version string that identifies the version of the type supported by the Rekor server
- `IndexKeys` should return a `[]string` containing the keys that are stored in the search index that should map to this log entry's ID.
- `Canonicalize` should return a `[]byte` containing the canonicalized contents representing the entry. The canonicalization of contents is important as we should have one record per unique signed object in the transparency log.
- `Unmarshal` will be called with a pointer to a struct that was automatically generated for the type defined in `openapi.yaml` by the [go-swagger](http://github.com/go-swagger/go-swagger) tool used by Rekor
- This method should validate the contents of the struct to ensure any string or cross-field dependencies are met to successfully insert an entry of this type into the transparency log
- `CreateFromArtifactProperties` returns a proposed entry of this specific entry implementation given the provided artifact properties
- `Verifier` returns the verification material that was used to verify the digital signature
- `Insertable` introspects the entry and determines if the object is sufficiently hydrated to make a new entry into the log. Entry instances that are created by reading the contents stored in the log may not be sufficiently hydrated.
```go
type EntryImpl interface {
APIVersion() string // the supported versions for this implementation
IndexKeys() ([]string, error) // the keys that should be added to the external index for this entry
Canonicalize(ctx context.Context) ([]byte, error) // marshal the canonical entry to be put into the tlog
Unmarshal(e models.ProposedEntry) error // unmarshal the abstract entry into the specific struct for this versioned type
CreateFromArtifactProperties(context.Context, ArtifactProperties) (models.ProposedEntry, error)
Verifier() (pki.PublicKey, error)
Insertable() (bool, error) // denotes whether the entry that was unmarshalled has the writeOnly fields required to validate and insert into the log
}
```

- `APIVersion` should return a version string that identifies the version of the type supported by the Rekor server
- `IndexKeys` should return a `[]string` containing the keys that are stored in the search index that should map to this log entry's ID.
- `Canonicalize` should return a `[]byte` containing the canonicalized contents representing the entry. The canonicalization of contents is important as we should have one record per unique signed object in the transparency log.
- `Unmarshal` will be called with a pointer to a struct that was automatically generated for the type defined in `openapi.yaml` by the [go-swagger](http://github.com/go-swagger/go-swagger) tool used by Rekor
- This method should validate the contents of the struct to ensure any string or cross-field dependencies are met to successfully insert an entry of this type into the transparency log
- `CreateFromArtifactProperties` returns a proposed entry of this specific entry implementation given the provided artifact properties
- `Verifier` returns the verification material that was used to verify the digital signature
- `Insertable` introspects the entry and determines if the object is sufficiently hydrated to make a new entry into the log. Entry instances that are created by reading the contents stored in the log may not be sufficiently hydrated.

5. In the Go package you have created for the new type, be sure to add an entry in the `TypeMap` in `github.com/sigstore/rekor/pkg/types` for your new type in the `init` method for your package. The key for the map is the unique string used to define your type in `openapi.yaml` (e.g. `newType`), and the value for the map is the name of a factory function for an instance of `TypeImpl`.

```go
func init() {
types.TypeMap.Set("newType", NewEntry)
}
```
```go
func init() {
types.TypeMap.Set("newType", NewEntry)
}
```

6. Add an entry to `pluggableTypeMap` in `cmd/server/app/serve.go` that provides a reference to your package. This ensures that the `init` function of your type (and optionally, your version implementation) will be called before the server starts to process incoming requests and therefore will be added to the map that is used to route request processing for different types.

Expand Down
48 changes: 24 additions & 24 deletions content/en/docs/logging/sharding.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ Sharding the log allows for:
* Easier and faster querying for entries from the tree
* Easier scaling and platform migrations


## How does this impact user experience?

It shouldn't!
Expand All @@ -33,49 +32,50 @@ Querying by log index works as well, since log indices are distinct and increase

For more details around sharding, see the original [design doc](https://docs.google.com/document/d/1QBTyK-wquplNdeUB5_aqztQHigJOepCvd-4FL4H-zl8/edit?resourcekey=0-grdVbSltkTvpNvhj03laCQ#heading=h.al4txfo7pxwl)!

_Note: You'll need to join the sigstore-dev@googlegroups.com Google group for access to the doc._

_Note: You'll need to join the [sigstore-dev@googlegroups.com](https://groups.google.com/g/sigstore-dev/about) Google group for access to the doc._

## How do I shard the Rekor log?

**Sharding the Rekor log will require some downtime in your Rekor service.**
This is necessary because you'll need the length of the current shard later on, so new entries can't be added while sharding is in progress.

Follow these steps to shard the log:

1. Stop all traffic to Rekor so new entries can't be added to the log
2. Store the tree ID and length of the current active shard:
```
CURRENT_TREE_ID=$(rekor-cli loginfo --format json | jq -r .TreeID)
CURRENT_SHARD_LENGTH=$(rekor-cli loginfo --format json | jq -r .TreeSize)
```

```bash
CURRENT_TREE_ID=$(rekor-cli loginfo --format json | jq -r .TreeID)
CURRENT_SHARD_LENGTH=$(rekor-cli loginfo --format json | jq -r .TreeSize)
```

3. Connect to the production cluster. Port-forward the running `trillian_logserver` container and run the [createtree](https://github.com/google/trillian/blob/master/cmd/createtree/main.go) script.
This will create a new Merkle Tree which will become the new active shard.

```
kubectl port-forward -n trillian-system deploy/trillian-log-server 8090:8090
# This is the Tree ID of the new active shard
NEW_TREE_ID=$(createtree --admin_server localhost:8090)
```
```bash
kubectl port-forward -n trillian-system deploy/trillian-log-server 8090:8090
# This is the Tree ID of the new active shard
NEW_TREE_ID=$(createtree --admin_server localhost:8090)
```

4. Update the Rekor `sharding-config` ConfigMap with details of the inactive shard:

```
kubectl edit configmap sharding-config -n rekor-system
```
```bash
kubectl edit configmap sharding-config -n rekor-system
```

Append the following onto the `sharding-config.yaml` key (it will be empty if this is the first shard):
Append the following onto the `sharding-config.yaml` key (it will be empty if this is the first shard):

```yaml
- treeID: $CURRENT_TREE_ID
treeLength: $CURRENT_SHARD_LENGTH
```
```yaml
- treeID: $CURRENT_TREE_ID
treeLength: $CURRENT_SHARD_LENGTH
```

5. In your rekor-server [Deployment](https://github.com/sigstore/rekor/blob/main/config/rekor.yaml), update the `--trillian_log_server.tlog_id` flag to point to the new Tree ID.

```
"--trillian_log_server.tlog_id=$NEW_TREE_ID",
```
```bash
"--trillian_log_server.tlog_id=$NEW_TREE_ID",
```

6. Redeploy Rekor to the cluster with these changes.

Expand All @@ -85,7 +85,7 @@ Append the following onto the `sharding-config.yaml` key (it will be empty if th

## Identifier Definitions: EntryID, UUID, LogID, Log Index

An **EntryID** is the unique identifier for an artifact in Rekor. It is made up of two parts, the TreeID and UUID:
An **EntryID** is the unique identifier for an artifact in Rekor. It is made up of two parts, the TreeID and UUID:

`EntryID = TreeID (8 byte hex) + UUID (32 byte hex)`

Expand Down