Flipt Filesystem Backends (for GitOps)

[GeorgeMac]

For anyone who is landing on this discussion late, this feature was added in v1.23.0 of Flipt
Checkout the documentation for enabling it here: https://www.flipt.io/docs/experimental/filesystem-backends

The team have been discussing and exploring ways Flipt could be better positioned to support GitOps-style workflows. For those unfamiliar with GitOps practices, we recommend the WeaveWorks Guide To GitOps.

This proposal adds two ways to configure Flipt to serve flag state content encoded as yaml stored in files. The two ways are local and git.

We’re keen to understand if this idea resonates with you and your use case. This will help us shape the implementation, which is currently underway.

Note that all the changes described by this proposal are designed to be opt-in alternative ways to run Flipt. We will continue to maintain backwards compatibility with our relational database backend support as we introduce these new concepts into Flipt. You do not have to change how you are using Flipt to keep up to date with new releases.

Proposal

Here is a summary of what we’re proposing:

• We’re keen to have Flipt use git directly as its source of truth. This is in order to:
  • Minimize required external dependencies.
  • Avoid error-prone and brittle reconciliation processes.
  • Leverage git's functionality for versioning, rollback and branching.
  • Potentially leverage git’s content addressability for caching.
• Support for configuring Flipt over a directory for a quick local development experience.
• Build a collection of CLI tools for static validation of your configuration during CI.

Proof(s)-of-concept

Over the past month, the team has been developing some proof-of-concept prototypes in and around the Flipt project.
These prototypes can be found in the following locations:

Flipt’s git and local filesystem backends

This can be found in the gm/filesystem-backends branch of the Flipt repository.
The Flipt git and local backend prototype support serving Flag state directly from the yaml format you can obtain by calling flipt export against your existing Flipt instance.

Here is the existing export documentation: https://www.flipt.io/docs/configuration/storage#export

Example:

namespace: default
flags:
- key: new-contact-page
  name: New Contact Page
  description: Show users our Beta contact page
  enabled: true
  variants:
  - key: blue
    name: Blue
  - key: green
    name: Green

By storing a yaml like this in a file named features.yaml in a directory, Flipt can discover and serve its contents. The directory can either be locally reachable or in a remote git repository.

In the prototype, every file named features.yaml recursively from the target directory is loaded into the serving instance. Allowing you to define multiple namespaces via multiple file definitions. However, this isn’t necessarily the exact or only way we ultimately intend to support sourcing feature state. We will get more into this in the Decisions: Outstanding section below.

If you’re familiar with building Flipt from source, you’re welcome to give this a try.

## For local development using native binary
mage dev

## or

## For a development docker build
cd build
mage build:flipt # see output for resulting docker image tag

For serving a local directory you can use a Flipt configuration file similar to the following:

The proof of concept still needs access to a database. This is why there is still a db key mentioned in the configuration files. This will not be the case in the final product. It will only be required if you need to enable authentication in Flipt.

storage:
  type: local
  local:
    path: "."

db:
  url: file:flipt.db

For serving a git repository, you can configure Flipt like so:

storage:
  type: git
  git:
    repository: <some-remote-url>
    ref: main

db:
  url: file:flipt.db

flipt validate subcommand

In order to avoid runtime errors due to invalid Flipt configuration files, flipt will ship with a new subcommand validate. This change is already merged into master and likely available in the next nightly release. This subcommand is intended for both local use and for invoking during CI workflows.

We’re actively working on a prebuilt GitHub action to reduce the effort required to leverage this in GitHub workflows.

Decisions 🤔

The section documents some of the preliminary and outstanding decisions we have made around the implementation details required to deliver on these new backends.

Please feel free to contribute your thoughts on these decision in this discussion.

Preliminary 🥇

1. Support both local and git backends

Flipt will support both these “filesystem” types.

local will allow for quick debugging and a local development experience.
From the root of a directory or even a git repo you can quickly experiment with un-committed changes.
Local changes on disk will propagate into updated state in Flipt.

git will track a remote reference (branch, tag or even a static SHA) in a target git repository.
As references update in the target upstream, Flipt will update its internal representation of Flipt state accordingly. The will allow you to do things such as use trunk branches for production, alternate branches for environments (e.g. staging) or even cut arbitrary branches for preview environments or sharing with colleagues.

2. Asynchronous updates of flag state

Feature flag state is a read heavy workload. Configuration changes are made infrequently compared to how often they’re evaluated or read at runtime by your systems. As such, we want to minimise impact on read performance when flag state changes in the target source of truth. In order to support this, we intend to represent the state of Flipt internally as a snapshot of state in time. Snapshots will be built asynchronously from any read or evaluation API calls being made.

local could poll for new state periodically or it could leverage file system watching mechanisms (our POC uses fsnotify).
git can periodically fetch a refs current target and update to new commit state atomically.

3. Use existing import/export Flipt yaml format

It seems prudent to start out with the import and export format we already have, and see if and how much pain is experience by users. If necessary then we can design a v2 format which can be delivered as its own project. The design goals for this project will be to learn from your experiences writing these configuration files by hand.

4. Flags can be defined across multiple files

We believe that being able to separate flag definitions across multiple files will be an added super power for these backends. This will be most valuable for the git backend, where users will be able to leverage mechanisms such as GH Codeowners to provide an extra level of authorization control over the top of Flipt.

5. UI will run in read-only mode

Once git is the source of truth for flag state, it becomes much more complicated to support our existing write API actions. Git and contribution workflows (such as Pull Requests) provide a very different experience for making changes. The individual resource level APIs we have today are incongruent with the experience we want to create around updating state in Flipt. As such, choosing this new storage type for Flipt will cause the UI to enter into a read-only state.

Outstanding 🥈

1. Flipt state file(s) discovery index

Flipt needs a way to locate which files in a repository should be considered when serving content. Our POC currently treats any file named features.yaml as source material. We need to understand if this works for Flipt going forward.

2. Private repository support

We expect this to be a desirable feature from the get-go. So we intend to implement it as part of a first official release. However, we have yet to decide the specific kinds of authentication we intend to support.
Some likely first candidates are:

• SSH private key authentication
• GitHub API key support

3. Timestamps and pagination

When sourcing flipt resources from a filesystem, we don’t have accurate created or updated at timestamps to hand. This impacts the resource fields returned via the API and in particular list operation order.

We’re keen to hear your thoughts on our plan. Drop them here or head over to https://flipt.io/discord to chat with us live.

4. Handling duplicate key definitions across flag state files

We dont have a good answer yet for what to do when a resource is declared in multiple files.
Do we:

• fail fast?
• attempt to merge?

5. Support for audit events

We currently won't be able to support the audit feature of Flipt. Since that is populated via the write APIs.
We need to explore ways to support reproducing this state from the git history or git events on the server side (server reflog).