Create API compatibility and Supported Versions Policy for Feast

[mrzzy]

Background

As Feast evolves and matures over time, and new users integrate Feast into their systems, we are faced with two conflicting needs:

• Developers need a way make breaking changes to add new features/rectify design decisions that no longer make sense:
  • Removing versions in FeatureSets PR Add support for feature set updates and remove versions #676
  • Adding Field Status metadata to Online Serving PR Add Field Status metadata to Online Serving #658
  • Generalising Source model from being tied to Kafka PR Refactor Source & Job data model and Stop Duplicate Ingestion Jobs #685
  • Evolving Feast's Data model Feast API: Feature references, concept hierarchy, and data model #479
• Users need time to transition their systems that are built on top of Feast during which Feast should offer compatibility guarantees . Upgrades should be as painless as possible. They also need to aware about what changes they have to make to transition from one version to another.
  • Introduce changes in Evolutionally Friendly way by @ches
  • Versioning APIs by @zhilingc

For guidance on how to resolve this we can look at Kubernetes does it:

• Handling Version Skew in Components/Backward Compatiblity
• What constitutes a compatible change
• Versioning APIs to introducing backward incompatible changes
• Deprecating APIs

User Facing APIs in Question for context:

• Ingestion (FeatureRow) API
• Core API
• Serving API
• Feast Python SDK/CLI
• Feast Java SDK
• Feast Golang SDK

Problem

How do we resolve the two conflicting needs of Developers and Users while satisfy the following requirements:

• Providing a clear way for Developers to introduce breaking changes as Feast evolves.
• Creating a clear transition path for users integrate these changes into their systems.
• Prevent code bloat resulting in Developers having to maintain too many API versions.
• Introduce backward compatibility to buy users time to transition their systems to the new API.

Proposed Solution

Create and API compatibility and Support Versions Policy to document how:

• What steps should developers take introduce breaking changes to the API? (ie Bump API Version, Deprecation etc.)
• What & How much API compatibility guarantees can we provide to users? (ie Deprecation happens 1 minor version before removal, Backwards compatibility with older SDKs up to 1 minor release. etc.)
• How do we provide a clear way for developers to announce breaking changes to users? (ie Slack? Deprecation notices? Issues? etc.)
• How can we provide a clear transition path for existing users to upgrade to new versions with potentially breaking changes? (ie Highlight with action required in Release note what needs to be changed etc.)

[woop]

Thanks for taking the time to write this out @mrzzy. A pleasant surprise.

Developers having to maintain too many API versions.

You touched on a core issue here. If we had unlimited manpower and bandwidth we could easily maintain multiple code paths and gradually deprecate old APIs. We don't, unfortunately.

I like the idea behind the policy you are proposing. I think we should start with a pebble (to steal @ches's saying). We can keep the policy as simple as possible for the time being, and over time it can grow to contain more and more information.

We are already documenting breaking api changes with the compat/breaking label. This means that all PRs should have a release note that describes this change. We could then compile these release notes (perhaps even automatically) into a release notes document.

Here is an example of what I mean: https://cloud.google.com/kubernetes-engine/docs/release-notes

This would allow users to at least identify upcoming breakages and plan for their upgrade.

What steps should developers take introduce breaking changes to the API? (ie Bump API Version, Deprecation etc.)
What & How much API compatibility guarantees can we provide to users? (ie Deprecation happens 1 minor version before removal, Backwards compatibility with older SDKs up to 1 minor release. etc.)

It would be great if you could provide "reasonable defaults" here that we can attack as a straw man. We probably want to only have breaking changes on minor version bumps (patch versions are for.. patches). And minimally we want to have 1 minor version of backward compatibility, but I can see ourselves breaking that rule as we are with projects right now.

How do we provide a clear way for developers to announce breaking changes to users? (ie Slack? Deprecation notices? Issues? etc.)

Perhaps every release should come out with an email on the mailing list. This should also communicate breaking changes and possibly a link to find out more information on upgrades.

How can we provide a clear transition path for existing users to upgrade to new versions with potentially breaking changes? (ie Highlight with action required in Release note what needs to be changed etc.)

I had something like this in mind: https://docs.getdbt.com/docs/guides/migration-guide/upgrading-from-0-10-to-0-11

[mrzzy]

Tying everything together into a policy:

Defining API:

• Core Database Schema
• SDK user facing methods/Objects.
• gRPC/Protobuf API for Feast Serving/Core.
• Configuration files.

Defining Breaking Change:

• Adding something that users need to use for existing functionality. (ie adding a required parameter).
• Changing how a given API should be interpreted.
• Changing a default value in parameter in the API.
• Removing something from the API.

What steps should developers take introduce breaking changes to the API?

• Make sure proposed breaking/removal changes have been deprecated for one minor release.
• Flag deprecated APIs to with deprecated in code and in release notes.
• Flag Breaking changes in release notes with breaking and action required.

What & How much API compatibility guarantees can we provide to users?

• Removal/Breaking user facing APIs should be flagged with a deprecation for one minor release (ie v0.5 -> v0.6) before making the change.
• Backward compatibility should be maintained for one minor release of the SDK. (ie v0.4 SDK should be interact with v0.5 Feast Core and Serving).

How do we provide a clear way for developers to announce breaking changes to users? (ie Slack? Deprecation notices? Issues? etc.)

• Deprecation/Breaking changes for each release to announced to the Feast mailing list.
• Release notes for each version should highlight deprecation/breaking changes clearly and provide link to migration guide.

How can we provide a clear transition path for existing users to upgrade to new versions with potentially breaking changes?

• Compile a migration guide for users to migrate their system to upgrade to a newer Feast version API.
• Migration guide should contain migrations for both breaking changes and deprecated APIs.

[mrzzy]

^ Once we are happy with the policy, I will update the documentation to reflect the policy.
Feel free to edit the comment if you have anything add/change etc.
@ches @woop

Open Questions:

• Should the storage representation in Stores be considered an API with compatibility guarantees?

[woop]

I read through the policy. It looks good to me.

Should the storage representation in Stores be considered an API with compatibility guarantees?

Also thought about this. I think for now we can exclude it since the assumption is that code would be contributed to Feast, and that the code owners should keep it up to date with the current interface. Once we have a plugin interface then it should definitely be included in this policy though.

Flag Breaking changes in release notes with breaking and action required.

We should also use the compat/breaking label on those PRs.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.