Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for tracking changes to Django and Wagtail #208

Merged
merged 1 commit into from
Jul 27, 2020
Merged

Add documentation for tracking changes to Django and Wagtail #208

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 140 additions & 0 deletions guides/tracking-django-wagtail.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
# Tracking Django and Wagtail

- [Philosophy](#philosophy)
- [Considerations when supporting multiple versions](#considerations-when-supporting-multiple-versions)
- [Consult the release notes](#consult-the-release-notes)
- [Use feature detection](#use-feature-detection)
- [Imports](#imports)
- [Version detection](#version-detection)
- [Matrix testing](#matrix-testing)
- [Upgrade process](#upgrade-process)
- [Ensure libraries and satellite apps support required versions](#ensure-libraries-and-satellite-apps-support-required-versions)
- [Ensure consumerfinance.gov supports required versions](#ensure-consumerfinancegov-supports-required-versions)
- [Upgrading consumerfinance.gov](#upgrading-consumerfinancegov)

## Philosophy

We have two general types of Python projects:

- The running [consumerfinance.gov](https://github.com/cfpb/cfgov-refresh) website
- Satellite applications of consumerfinance.gov that live in their own repository, e.g., our satellite apps of consumerfinance.gov, like [ccdb5-ui](https://github.com/cfpb/ccdb5-ui)
- Libraries that provide some functionality independent of consumerfinance.gov, e.g., [wagtail-sharing](https://github.com/cfpb/wagtail-sharing)

We use a rubric like the one below to determine which versions of Python and major dependencies should be supported.

|| Library | consumerfinance.gov | Satellite application |
|--|--|--|--|
|Oldest Python 3 supported by Django LTS | ✔️| | |
|Current production Python | | ✔️ | ✔️|
|Latest Python | ✔️ | ✔️ | ✔️|
|Latest Django LTS version | ✔️ | ✔️ | ✔️|
|Latest Django version | ✔️ | ✔️ | ✔️|
|Latest Wagtail LTS version | ✔️ | ✔️ | ✔️|
|Latest Wagtail version | ✔️ | ✔️ | ✔️|

consumerfinance.gov and its satellite apps should support the version of Python 3 we currently run in production and the latest version. At the time of writing that is version 3.6 and 3.8.

Our libraries should support the oldest version of Python 3 that the library's major dependencies supports, and the latest version of Python. For example, [Django 2.2 supports Python 3.5 and up](https://docs.djangoproject.com/en/2.2/releases/2.2/#python-compatibility), so our Django and Wagtail libraries that support Django 2.2 should do the same.

Both consumerfinance.gov and its satellite apps, and our libraries should support the LTS (long-term support) version of Django and Wagtail that we currently run in production, and the latest version (LTS or not) of Django and Wagtail.

## Considerations when supporting multiple versions

### Consult the release notes

Django and Wagtail are generally very good about listing upgrading considerations, deprecations, and breaking changes in their release notes. The release notes should drive the changes that we make.

### Use feature detection

When dealing with support for multiple versions of Python or major dependencies, we prefer to [use feature detection instead of version detection](https://docs.python.org/3/howto/pyporting.html#use-feature-detection-instead-of-version-detection) for incompatibilities/breaking changes that may exist between versions. Feature detection `try`/`except` blocks should always attempt to use **new** code first and then fall back on old code.

### Imports

To detect features on import, or import paths have changed between versions, we prefer to use `try`/`except` to import the **new** version and fall back on the old version:

```python
try:
# Django >= 2.0
from django.urls import include
except ImportError:
# Django < 2.0
from django.conf.urls import include
```

### Version detection

Only when [feature-detection](https://docs.python.org/3/howto/pyporting.html#use-feature-detection-instead-of-version-detection) is not sufficient to support multiple versions of a dependency do we fall back on version-checking. This can happen when APIs change but the naming conventions remain the same, and exception-handling is insufficient or too greedy to detect the difference between an API change and another reason an execption may be raised.

When using version detection, we prefer `if`/`else` conditionals with the **new** API used in the `if`, falling back on the old API in the `else`. For example:

```python
import django

if django.VERSION >= (2, 0, 0):
is_anonymous = bool(request.user.is_anonymous)
else:
is_anonymous = bool(request.user.is_anonymous())
```

When performing a version check, the hard-coded version should always be **inclusive** of the first version to support the new API and should specify the entire semantic version.

Both Django and Wagtail provide versions as tuples of (major version, minor version, patch version, sub-version, sub-version release). For example:

```python
>>> import django
>>> django.VERSION
(2, 2, 13, 'final', 0)
```

Version-detection of Django and Wagtail should specify `(major, minor, patch)`.

### Matrix testing

We use [tox](https://tox.readthedocs.io/en/latest/) for matrix testing against all the versions listed above, and we provide a [sample tox configuration](../tox.ini) that tests against both our current production the latest versions Python and Django suitable as a basis for satellite apps and libraries.

## Upgrade process

The upgrade process for consumerfinance.gov is fairly straight-forward, and reasonably similar for all three categories of projects we maintain.

### Ensure libraries and satellite apps support required versions

1. Add support for the new version(s) to the `tox` matrix
2. Consult release notes of the version(s) being added and make necessary code changes
3. Fix any tests that break as a result
4. Ensure that running functionality works as expected
5. Clean up support for any previous versions that can be dropped
6. Create a new release

New releases of libraries should be automatically uploaded to PyPI by our GitHub Actions.

New releases of satellite apps should be automatically added to their GitHub release page by our GitHub Actions.

### Ensure consumerfinance.gov supports required versions

consumerfinance.gov's [tox.ini](https://github.com/cfpb/cfgov-refresh/blob/master/tox.ini) matrixes on "current" vs "future" versions rather than on specific version numbers, and unlike our satellite apps and libraries pins its dependencies to specific versions. This can create incompatibilities that we need to deal with.

1. Add support for the new version(s) to the `tox` `future-config` matrix configuration.

Include new pins of any required libraries. For example, if a future Wagtail requires an update to BeautifulSoup that is incompatible with our current Wagtail's dependency on BeautifulSoup, pin it in the tox `future-config` `deps`:

```ini
[future-config]
…
deps=
Django>=2.2,<3.0
wagtail>=2.10,<2.11
beautifulsoup4>=4.8.2
```

2. Pin any new releases of satellite apps or libraries in the [requirements](https://github.com/cfpb/cfgov-refresh/blob/master/requirements/libraries.txt) file.
3. Consult release notes of the version(s) being added and make necessary code changes
4. Fix any tests that break as a result
5. Ensure that running functionality works as expected
6. Clean up support for any previous versions that can be dropped

### Upgrading consumerfinance.gov

When we have a new LTS version of Django, or when there's a new release of Wagtail to deploy, the process of upgrading consumerfinance.gov should be simple, because we already have been tracking support for those versions.

1. Pin [Django](https://github.com/cfpb/cfgov-refresh/blob/master/requirements/django.txt) and [Wagtail](https://github.com/cfpb/cfgov-refresh/blob/master/requirements/wagtail.txt) in their respective requirements files.
2. Update the pins of any libraries that we pinned in the tox `future-config` `deps` in the [libraries](https://github.com/cfpb/cfgov-refresh/blob/master/requirements/libraries.txt) requirements file.
33 changes: 6 additions & 27 deletions standards/python.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
- [Style](#style)
- [Linting](#linting)
- [Imports](#imports)
- [Dependency support/testing matrix](#dependency-supporttesting-matrix)
- [Dependency support and matrix testing](#dependency-support-and-matrix-testing)
- [Considerations](#considerations)
- [Specifying dependencies in `requirements.txt` or `setup.py`](#specifying-dependencies-in-requirementstxt-or-setuppy)

Expand Down Expand Up @@ -42,37 +42,16 @@ We use [isort](https://github.com/timothycrosley/isort) to lint imports to compl

The isort configuration can go into a `tox.ini` file under an `[isort]` header.

## Dependency support/testing matrix
## Dependency support and matrix testing

We have two general types of Python projects:
### Matrix testing

- Applications that have a running instance, e.g., [cfgov-refresh](https://github.com/cfpb/cfgov-refresh)
- Libraries that provide some functionality independent of applications

We use a rubric like the one below to determine which versions of Python and major dependencies should be supported. Assuming the project has Django and Wagtail as its major dependencies, our recommendation would be this:

|| Library | Application |
|--|--|--|
|Oldest Python 3 supported by Django LTS | ✔️| |
|Current production Python | | ✔️ |
|Latest Python | ✔️ | ✔️ |
|Latest Django LTS version | ✔️ | ✔️ |
|Latest Django version | ✔️ | ✔️ |
|Latest Wagtail LTS version | ✔️ | ✔️ |
|Latest Wagtail version | ✔️ | ✔️ |

Our applications should support the version of Python 3 we currently run in production and the latest version. For example, [cfgov-refresh](https://github.com/cfpb/cfgov-refresh) should support version 3.6 and 3.8 at time of writing.

Our libraries should support the oldest version of Python 3 that the library's major dependencies supports, and the latest version of Python. For example, [Django 2.2 supports Python 3.5 and up](https://docs.djangoproject.com/en/2.2/releases/2.2/#python-compatibility), so our Django and Wagtail libraries that support Django 2.2 should do the same.

"Major dependencies" are the frameworks upon which our applications and libraries are built. Examples are Django and Wagtail. Both our applications and our libraries should support the LTS (long-term support) version of that dependency we currently run in production, and the latest version (LTS or not) of that dependency.

### Considerations

We recommend using [tox](https://tox.readthedocs.io/en/latest/) for matrix testing against all the versions listed above, and we provide a [sample tox configuration](../tox.ini) that tests against both our current production the latest versions Python and Django.
We recommend using [tox](https://tox.readthedocs.io/en/latest/) for matrix testing against our supported versions of our core dependencies, and we provide a [sample tox configuration](../tox.ini) that tests against both our current production the latest versions Python and Django.

When dealing with support for multiple versions of Python or major dependencies, we prefer to [use feature detection instead of version detection](https://docs.python.org/3/howto/pyporting.html#use-feature-detection-instead-of-version-detection) for incompatibilities/breaking changes that may exist between versions.

See [Tracking Django and Wagtail](../guides/tracking-django-wagtail.md) for more detailed guidance.

### Specifying dependencies in `requirements.txt` or `setup.py`

The difference between pinned versions of dependencies in `requirements.txt` and dependencies specified in `setup.py` is frequently confusing and often misunderstood. The difference is between *concrete* and *abstract* dependencies, which is most easily understood in terms of applications (intended to be deployed and run), which specify *concrete* dependencies, and libraries (reusable code that may be included in applications), which specify *abstract* dependencies.
Expand Down