This site is made with Pelican, a Python-powered static site generator. There are a lot of SSGs out there, but I went with this one as it's written in Python, so I can easily extend it if I need to.
I wrote the theme myself. It's designed to be minimalist, have semantic markup, clean, modern CSS and use only system fonts. All these help keep the page size to a minimum. There are currently no cookies, and no plans to add any. This is not for any ideoligcal reason — I just don't need them.
The design has incorporated accessibility from the start. It should pass WCAG 2.1 AA. Getting it AAA compliant would be a lot more work for almost no benefit, so it'll only happen if I have some free time and nothing else to do. However, that doesn't mean I won't improve the accessibility generally.
The site is hosted on GitHub Pages for the sake of simplicity and not having to needlessly throw money away.
This site is made with Pelican, a Python-powered static site generator. +There are a lot of SSGs out there, but I went with this one as it's written in Python, +so I can easily extend it if I need to.
+I wrote the theme myself. It's designed to be minimalist, have semantic markup, +clean, modern CSS and use only system fonts. All these help keep the page size to a +minimum. There are currently no cookies, and no plans to add any. +This is not for any ideoligcal reason — I just don't need them.
+The design has incorporated accessibility from the start. It should pass WCAG 2.1 AA. +Getting it AAA compliant would be a lot more work for almost no benefit, so it'll only +happen if I have some free time and nothing else to do. However, that doesn't mean I +won't improve the accessibility generally.
+The site is hosted on GitHub Pages for the sake of simplicity and not having +to needlessly throw money away.
+ +Back in 2022, I wrote about automatically updating dependencies with dependabot. However, this can be quite tedious. If you have a lot of dependencies, or even if you don't, you'll likely get spammed with many pull requests on your scheduled update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, newer versions will often have performance improvements, bug fixes and generally fewer problems. Keeping them up to date over multiple codebases can be quite painful, but it's possible to bring this effort down to close to zero with a …
You probably already know the value of testing your code. Your Django migrations are code, therefore you should test them. However, testing data migrations in particular can be tricky, and there's no documentation on how to do it.
Typically your schema migrations don't need any testing as your migrations are …
Back in 2022, I wrote about automatically updating dependencies with dependabot. +However, this can be quite tedious. If you have a lot of dependencies, or even +if you don't, you'll likely get spammed with many pull requests on your scheduled +update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, +newer versions will often have performance improvements, bug fixes and generally +fewer problems. Keeping them up to date over multiple codebases can be quite +painful, but it's possible to bring this effort down to close to zero with a …
You probably already know the value of testing your code. +Your Django migrations are code, therefore you should test them. +However, testing data migrations in particular can be tricky, +and there's no documentation on how to do it.
+Typically your schema migrations don't need any testing as your +migrations are …
Outdated packages can open your application up to security issues. Moreover, newer versions will often have performance improvements, bug fixes and generally fewer problems. Keeping them up to date over multiple codebases can be quite painful, but it's possible to bring this effort down to close to zero with a bit of up-front effort.
This post applies only if you're hosting your code on GitHub. If not, there are managed options for dependency upgrades, such as Snyk. It's very expensive, but does a lot of other things too. If you are on GitHub, this is a free or cheap alternative.
First, a few pre-requisites. You'll need to ensure "Allow auto-merge" is enabled in your repository settings. You should also configure dependabot security alerts in the "Security" tab. These alerts happen outside of the dependabot schedule we'll set up later, and will ensure you get critical security updates as quickly as possible. You'll also need to make sure GitHub Actions is enabled for your repository.
Next, you'll need to create a new file for the dependabot configuration:
.github/dependabot.yml
version: 2
-updates:
- - package-ecosystem: "github-actions"
- assignees:
- - "<your_github_username>"
- directory: "/"
- reviewers:
- - "<your_github_username>"
- schedule:
- day: "monday"
- interval: "weekly"
- time: "09:00"
- timezone: "Europe/Amsterdam"
- - package-ecosystem: "pip"
- allow:
- - dependency-type: all
- assignees:
- - "<your_github_username>"
- directory: "/"
- reviewers:
- - "<your_github_username>"
- schedule:
- interval: "daily"
- time: "09:00"
- timezone: "Europe/Amsterdam"
-The configuration is quite straightforward. Note there are two package-ecosystem sections defined. One of these is for pip, which will also work for pipenv and poetry. The other will keep any GitHub actions you use up to date. We'll be using one of these soon, so it seemed good to add it. You can set the schedule however makes sense for your workflow. You don't need to set assignees and reviewers, but I like to add them so they show up in my GitHub notifications and I can make sure I don't miss anything. Not every update will be able to be merged automatically, so it's good to know about them.
One other important thing here is the allow section. Here I've set dependency-type to all, which means that sub-dependencies will also be updated. Without this, only explicitly defined dependencies will be updated.
Other configuration options are available.
Adding this file is enough for dependabot to submit pull requests on your repo with package updates. We need to do a bit more to get them merging automatically. For this, we need GitHub Actions.
Warning
You should only have your dependencies merge automatically if you have an exhaustive CI set up. Otherwise, you may be vulnerable to obnoxious developers and supply chain attacks. Supply chain attacks may still get through your CI however, so you should judge the level of risk you're willing to tolerate.
We'll create a new file, .github/workflows/dependabot-automerge.yml.
name: Dependabot auto-merge
-on: pull_request_target
-
-permissions:
- pull-requests: write
- contents: write
-
-jobs:
- dependabot-auto-merge:
- runs-on: ubuntu-latest
- if: ${{ github.actor == 'dependabot[bot]' }}
- env:
- PR_URL: ${{ github.event.pull_request.html_url }}
- GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- steps:
- - name: Dependabot metadata
- id: metadata
- uses: dependabot/fetch-metadata@v1.1.1
- with:
- github-token: "${{ secrets.GITHUB_TOKEN }}"
- - name: Approve PR
- run: gh pr review --approve "$PR_URL"
- - name: Merge PR
- if: ${{ steps.metadata.outputs.update-type != 'version-update:semver-major' }}
- run: gh pr merge --auto --squash "$PR_URL"
-Exactly what you put in this file will depend on your workflow. For example, if your pull requests do not require reviews, you can omit that section. Are you feeling risk-averse? Remove the auto-merge, or use it only for patch versions. The opposite? Remove the conditional and auto-merge everything.
This example will approve every dependabot pull request, and merge any non-major semver versions automatically so long as all required checks pass. For major versions, I like to review them to make sure nothing will break. If the project isn't using semantic versioning, it's a little difficult to know what happens here, however. My assumption is that it'll do its best to figure it out and assume it's a major version. But it may be that if it doesn't look like it's using semver, it will not count as a major update and will be merged automatically. If anyone does know the answer to this, I would love to hear it. Luckily, most packages use something similar to semantic versioning these days so I haven't yet had any problems. I'll update this post if I notice any.
I hope this saves you some time you could be using for something more fun.
Outdated packages can open your application up to security issues. Moreover, +newer versions will often have performance improvements, bug fixes and generally +fewer problems. Keeping them up to date over multiple codebases can be quite +painful, but it's possible to bring this effort down to close to zero with a bit +of up-front effort.
+This post applies only if you're hosting your code on GitHub. If not, there are +managed options for dependency upgrades, such as Snyk. It's very expensive, +but does a lot of other things too. If you are on GitHub, this is a free or +cheap alternative.
+First, a few pre-requisites. You'll need to ensure "Allow auto-merge" is +enabled in your repository settings. You should also configure dependabot +security alerts in the "Security" tab. These alerts happen outside of the +dependabot schedule we'll set up later, and will ensure you get critical security +updates as quickly as possible. You'll also need to make sure GitHub Actions +is enabled for your repository.
+Next, you'll need to create a new file for the dependabot configuration:
+.github/dependabot.yml
+version: 2
+updates:
+ - package-ecosystem: "github-actions"
+ assignees:
+ - "<your_github_username>"
+ directory: "/"
+ reviewers:
+ - "<your_github_username>"
+ schedule:
+ day: "monday"
+ interval: "weekly"
+ time: "09:00"
+ timezone: "Europe/Amsterdam"
+ - package-ecosystem: "pip"
+ allow:
+ - dependency-type: all
+ assignees:
+ - "<your_github_username>"
+ directory: "/"
+ reviewers:
+ - "<your_github_username>"
+ schedule:
+ interval: "daily"
+ time: "09:00"
+ timezone: "Europe/Amsterdam"
+The configuration is quite straightforward. Note there are two package-ecosystem +sections defined. One of these is for pip, which will also work for pipenv and poetry. +The other will keep any GitHub actions you use up to date. We'll be using one of +these soon, so it seemed good to add it. You can set the schedule however makes sense +for your workflow. You don't need to set assignees and reviewers, but I like to add +them so they show up in my GitHub notifications and I can make sure I don't miss +anything. Not every update will be able to be merged automatically, so it's good +to know about them.
+One other important thing here is the allow section. Here I've set +dependency-type to all, which means that sub-dependencies will also be updated. +Without this, only explicitly defined dependencies will be updated.
+Other configuration options are available.
+Adding this file is enough for dependabot to submit pull requests on your repo with +package updates. We need to do a bit more to get them merging automatically. For this, +we need GitHub Actions.
+Warning
+You should only have your dependencies merge automatically if you have an +exhaustive CI set up. Otherwise, you may be vulnerable to obnoxious developers +and supply chain attacks. Supply chain attacks may still get through your CI +however, so you should judge the level of risk you're willing to tolerate.
+We'll create a new file, .github/workflows/dependabot-automerge.yml.
+name: Dependabot auto-merge
+on: pull_request_target
+
+permissions:
+ pull-requests: write
+ contents: write
+
+jobs:
+ dependabot-auto-merge:
+ runs-on: ubuntu-latest
+ if: ${{ github.actor == 'dependabot[bot]' }}
+ env:
+ PR_URL: ${{ github.event.pull_request.html_url }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ steps:
+ - name: Dependabot metadata
+ id: metadata
+ uses: dependabot/fetch-metadata@v1.1.1
+ with:
+ github-token: "${{ secrets.GITHUB_TOKEN }}"
+ - name: Approve PR
+ run: gh pr review --approve "$PR_URL"
+ - name: Merge PR
+ if: ${{ steps.metadata.outputs.update-type != 'version-update:semver-major' }}
+ run: gh pr merge --auto --squash "$PR_URL"
+Exactly what you put in this file will depend on your workflow. For example, if +your pull requests do not require reviews, you can omit that section. Are you +feeling risk-averse? Remove the auto-merge, or use it only for patch versions. +The opposite? Remove the conditional and auto-merge everything.
+This example will approve every dependabot pull request, and merge any non-major +semver versions automatically so long as all required checks pass. +For major versions, I like to review them to make sure nothing will break. +If the project isn't using semantic versioning, it's a little difficult to know +what happens here, however. My assumption is that it'll do its best to figure it out +and assume it's a major version. But it may be that if it doesn't look like it's using +semver, it will not count as a major update and will be merged automatically. +If anyone does know the answer to this, I would love to hear it. Luckily, most +packages use something similar to semantic versioning these days so I haven't yet had any +problems. I'll update this post if I notice any.
+I hope this saves you some time you could be using for something more fun.
+ +Back in 2022, I wrote about automatically updating dependencies with dependabot. However, this can be quite tedious. If you have a lot of dependencies, or even if you don't, you'll likely get spammed with many pull requests on your scheduled update time. This can be quite annoying, especially if you have a lot of repositories.
Every time you merge one of these pull requests, if you're strict about only merging branches when they're up to date with the base branch, you'll have to update the next pull request, wait for all your CI checks to pass, burning through your GitHub actions minutes (or other CI credits), and warming the planet a little. And then, the next PR. And the next. And the next.
Now, you can just YOLO merge without updating the branch if you're happy to take the risk. And it doesn't take much time to update the branch, you can spread this out over a day in those few minutes between tasks, or when pretending to pay attention in a meeting, but you don't need to do any of this anymore.
Last year, GitHub released a grouping feature for dependabot. You can now group your dependencies in the dependabot configuration like so:
.github/dependabot.yml
version: 2
-- package-ecosystem: "pip"
- directory: "/"
- groups:
- all:
- patterns:
- - "*"
- schedule:
- day: "monday"
- interval: "weekly"
- time: "09:00"
- timezone: "Europe/Amsterdam"
-I simplified the configuration a bit from last time, but the important thing is the groups key. This allows you to group your dependencies by a pattern. In this case, I'm grouping all dependencies with the pattern *. This will group all dependencies together, so you'll only get one pull request for all your dependencies, rather than one for each. You can have multiple groups, so you can group your dependencies however you like, although I don't personally find much utility in this.
A huge time saver, and a huge annoyance saver.
Back in 2022, I wrote about automatically updating dependencies with dependabot. +However, this can be quite tedious. If you have a lot of dependencies, or even +if you don't, you'll likely get spammed with many pull requests on your scheduled +update time. This can be quite annoying, especially if you have a lot of repositories.
+Every time you merge one of these pull requests, if you're strict about only merging +branches when they're up to date with the base branch, you'll have to update the next +pull request, wait for all your CI checks to pass, burning through your GitHub actions +minutes (or other CI credits), and warming the planet a little. And then, the next PR. +And the next. And the next.
+Now, you can just YOLO merge without updating the branch if you're happy to take the risk. +And it doesn't take much time to update the branch, you can spread this out over a day in +those few minutes between tasks, or when pretending to pay attention in a meeting, +but you don't need to do any of this anymore.
+Last year, GitHub released a grouping feature for dependabot. +You can now group your dependencies in the dependabot configuration like so:
+.github/dependabot.yml
+version: 2
+- package-ecosystem: "pip"
+ directory: "/"
+ groups:
+ all:
+ patterns:
+ - "*"
+ schedule:
+ day: "monday"
+ interval: "weekly"
+ time: "09:00"
+ timezone: "Europe/Amsterdam"
+I simplified the configuration a bit from last time, but the important thing is the +groups key. This allows you to group your dependencies by a pattern. In this case, +I'm grouping all dependencies with the pattern *. This will group all dependencies +together, so you'll only get one pull request for all your dependencies, rather than +one for each. You can have multiple groups, so you can group your dependencies however +you like, although I don't personally find much utility in this.
+A huge time saver, and a huge annoyance saver.
+ +Back in 2022, I wrote about automatically updating dependencies with dependabot. However, this can be quite tedious. If you have a lot of dependencies, or even if you don't, you'll likely get spammed with many pull requests on your scheduled update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, newer versions will often have performance improvements, bug fixes and generally fewer problems. Keeping them up to date over multiple codebases can be quite painful, but it's possible to bring this effort down to close to zero with a …
You probably already know the value of testing your code. Your Django migrations are code, therefore you should test them. However, testing data migrations in particular can be tricky, and there's no documentation on how to do it.
Typically your schema migrations don't need any testing as your migrations are …
Back in 2022, I wrote about automatically updating dependencies with dependabot. +However, this can be quite tedious. If you have a lot of dependencies, or even +if you don't, you'll likely get spammed with many pull requests on your scheduled +update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, +newer versions will often have performance improvements, bug fixes and generally +fewer problems. Keeping them up to date over multiple codebases can be quite +painful, but it's possible to bring this effort down to close to zero with a …
You probably already know the value of testing your code. +Your Django migrations are code, therefore you should test them. +However, testing data migrations in particular can be tricky, +and there's no documentation on how to do it.
+Typically your schema migrations don't need any testing as your +migrations are …
Back in 2022, I wrote about automatically updating dependencies with dependabot. However, this can be quite tedious. If you have a lot of dependencies, or even if you don't, you'll likely get spammed with many pull requests on your scheduled update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, newer versions will often have performance improvements, bug fixes and generally fewer problems. Keeping them up to date over multiple codebases can be quite painful, but it's possible to bring this effort down to close to zero with a …
Back in 2022, I wrote about automatically updating dependencies with dependabot. +However, this can be quite tedious. If you have a lot of dependencies, or even +if you don't, you'll likely get spammed with many pull requests on your scheduled +update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, +newer versions will often have performance improvements, bug fixes and generally +fewer problems. Keeping them up to date over multiple codebases can be quite +painful, but it's possible to bring this effort down to close to zero with a …
You probably already know the value of testing your code. Your Django migrations are code, therefore you should test them. However, testing data migrations in particular can be tricky, and there's no documentation on how to do it.
Typically your schema migrations don't need any testing as your migrations are …
You probably already know the value of testing your code. +Your Django migrations are code, therefore you should test them. +However, testing data migrations in particular can be tricky, +and there's no documentation on how to do it.
+Typically your schema migrations don't need any testing as your +migrations are …
Back in 2022, I wrote about automatically updating dependencies with dependabot. However, this can be quite tedious. If you have a lot of dependencies, or even if you don't, you'll likely get spammed with many pull requests on your scheduled update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, newer versions will often have performance improvements, bug fixes and generally fewer problems. Keeping them up to date over multiple codebases can be quite painful, but it's possible to bring this effort down to close to zero with a …
Back in 2022, I wrote about automatically updating dependencies with dependabot. +However, this can be quite tedious. If you have a lot of dependencies, or even +if you don't, you'll likely get spammed with many pull requests on your scheduled +update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, +newer versions will often have performance improvements, bug fixes and generally +fewer problems. Keeping them up to date over multiple codebases can be quite +painful, but it's possible to bring this effort down to close to zero with a …
You probably already know the value of testing your code. Your Django migrations are code, therefore you should test them. However, testing data migrations in particular can be tricky, and there's no documentation on how to do it.
Typically your schema migrations don't need any testing as your migrations are …
You probably already know the value of testing your code. +Your Django migrations are code, therefore you should test them. +However, testing data migrations in particular can be tricky, +and there's no documentation on how to do it.
+Typically your schema migrations don't need any testing as your +migrations are …
You probably already know the value of testing your code. Your Django migrations are code, therefore you should test them. However, testing data migrations in particular can be tricky, and there's no documentation on how to do it.
Typically your schema migrations don't need any testing as your migrations are …
You probably already know the value of testing your code. +Your Django migrations are code, therefore you should test them. +However, testing data migrations in particular can be tricky, +and there's no documentation on how to do it.
+Typically your schema migrations don't need any testing as your +migrations are …
You probably already know the value of testing your code. Your Django migrations are code, therefore you should test them. However, testing data migrations in particular can be tricky, and there's no documentation on how to do it.
Typically your schema migrations don't need any testing as your migrations are run during tests, unless you're skipping migrations to speed up your tests. I recommend against doing this, but that's another blog post.
However, only your data migrations forwards are tested. Even then, they are only tested if they don't have any branching, and if there is something to do. Most data migrations during tests have nothing to do because there is no data during migrations to run against. But why test data migrations in the first place?
I often use data migrations for a few purposes. The first is to add initial data. For example, you might want to store a list of countries in the database. You don't want to add them directly to the database, but you want them to always be there, and if some new country is formed in the future, you can add it in another data migration, say. This could also be done with a management command or in a multitude of other ways, so it's not the main reason for using a data migration.
The other reason is probably more useful. You'll often have a new requirement that needs you to restructure your database, and as a part of that, you have to make a change to some of your data. For example, you might decide to add a required username to your user model, and to populate the initial values, you want to use the first part of the user's email address. Please don't do this in real life. Generating public information from private information is not very good for a user's privacy.
Testing these with real data is important. You may have users in your database with valid emails where the username part wouldn't validate as a username. As migrations are typically only ran once in production, you'll certainly find out when you deploy and run your migrations that something is wrong. However, is the data in your staging environment as good as that in your production environment? Even if it is, it would be nice to find these problems earlier, and without the stress of dealing with potential problems that aborting a deployment can entail.
If you find yourself doing this a lot, there is a package called django-test-migrations. I haven't used it myself but it will probably be more robust than the simple approach below. However, if you're averse to installing yet another package, let's see what we can put together without too much work.
Let's use the example above and say we have the following data migration:
from django.db import migrations
-
-def populate_usernames(apps, schema_editor):
- User = apps.get_model("accounts", "User")
- for user in User.objects.all():
- user.username = user.email.rpartition("@")[0]
- user.save()
-
-def depopulate_usernames(apps, schema_editor):
- User = apps.get_model("accounts", "User")
- User.objects.update(username=None)
-
-class Migration(migrations.Migration):
- dependencies = [("accounts", "0002_add_username")]
- operations = [migrations.RunPython(populate_usernames, depopulate_usernames)]
-To test migrations like this, I decided to write a small class to run the migrations. You may notice that this is a class with a single method and __init__, so it could just be a function, but setting the apps attribute felt better than returning apps directly.
from django.db import connection
-from django.db.migrations.executor import MigrationExecutor
-
-
-class Migrator:
- def __init__(self, connection=connection):
- self.executor = MigrationExecutor(connection)
-
- def migrate(self, app_label: str, migration: str):
- target = [(app_label, migration)]
- self.executor.loader.build_graph()
- self.executor.migrate(target)
- self.apps = self.executor.loader.project_state(target).apps
-There's not too much going on here. We initialise the class with a migration executor, using a passed connection, or the default connection if none is passed. Then you can run the migrate method, with your app label and migration name. This will run the migration and set the apps attribute that you'll use in your test to make sure you have the right version of the models, similar to how it's used in the data migration itself.
Using the migrator in pytest is pretty simple. We can write a single-line fixture:
@pytest.fixture
-def migrator():
- return Migrator
-Then we can write our tests, migrating to where we need to be:
@pytest.mark.django_db
-def test_populate_emails(migrator):
- migrator = migrator()
- migrator.migrate("accounts", "0002_add_username")
- User = migrator.apps.get_model("accounts", "User")
- user = User.objects.create_user(email="test123@example.com")
- assert user.username is None
-
- migrator.migrate("accounts", "0003_populate_usernames")
- assert User.objects.filter(email="test123@example.com", username="test123").exists()
-Of course, we can also migrate backwards. Typically, migrating backwards is only used when developing, but in case you want that 100% coverage or really want to be sure:
@pytest.mark.django_db
-def test_depopulate_emails(migrator):
- migrator = migrator()
- migrator.migrate("accounts", "0002_add_username")
- User = migrator.apps.get_model("accounts", "User")
- user = User.objects.create_user(email="test123@example.com")
- migrator.migrate("accounts", "0003_populate_usernames")
- migrator.migrate("accounts", "0002_add_username")
- User = migrator.apps.get_model("accounts", "user")
- assert User.objects.get(email="test123@example.com").username is None
-If you're using Django's default unittest framework, you can use it in much the same way:
from django.test import TestCase
-
-
-class MigrationTest(TestCase):
- def setUp(self):
- self.migrator = Migrator()
- self.migrator.migrate("accounts", "0002_add_username")
-
- def test_populate_usernames(self):
- User = self.migrator.apps.get_model("accounts", "User")
- user = User.objects.create_user(email="test123@example.com")
- assert user.username is None
-
- migrator.migrate("accounts", "0003_populate_usernames")
- assert User.objects.filter(
- email="test123@example.com", username="test123"
- ).exists()
-
- def test_depopulate_emails(migrator):
- User = migrator.apps.get_model("accounts", "User")
- user = User.objects.create_user(email="test123@example.com")
- migrator.migrate("accounts", "0003_populate_usernames")
- migrator.migrate("accounts", "0002_add_username")
- User = migrator.apps.get_model("accounts", "user")
- assert User.objects.get(email="test123@example.com").username is None
-So, get out there and test those data migrations.
You probably already know the value of testing your code. +Your Django migrations are code, therefore you should test them. +However, testing data migrations in particular can be tricky, +and there's no documentation on how to do it.
+Typically your schema migrations don't need any testing as your +migrations are run during tests, unless you're skipping migrations +to speed up your tests. I recommend against doing this, but that's another +blog post.
+However, only your data migrations forwards are tested. Even then, they are +only tested if they don't have any branching, and if there is something to do. +Most data migrations during tests have nothing to do because there is no data +during migrations to run against. But why test data migrations in the first +place?
+I often use data migrations for a few purposes. The first is to add initial data. +For example, you might want to store a list of countries in the database. You don't +want to add them directly to the database, but you want them to always be there, +and if some new country is formed in the future, you can add it in another data +migration, say. This could also be done with a management command or in a multitude +of other ways, so it's not the main reason for using a data migration.
+The other reason is probably more useful. You'll often have a new requirement that +needs you to restructure your database, and as a part of that, you have to make a +change to some of your data. For example, you might decide to add a required username +to your user model, and to populate the initial values, you want to use the first part +of the user's email address. Please don't do this in real life. Generating public +information from private information is not very good for a user's privacy.
+Testing these with real data is important. You may have users in your database with +valid emails where the username part wouldn't validate as a username. As migrations +are typically only ran once in production, you'll certainly find out when you deploy +and run your migrations that something is wrong. However, is the data in your staging +environment as good as that in your production environment? Even if it is, +it would be nice to find these problems earlier, and without the stress of dealing +with potential problems that aborting a deployment can entail.
+If you find yourself doing this a lot, there is a package called +django-test-migrations. I haven't used it myself but it will probably be +more robust than the simple approach below. +However, if you're averse to installing yet another package, let's see what +we can put together without too much work.
+Let's use the example above and say we have the following data migration:
+from django.db import migrations
+
+def populate_usernames(apps, schema_editor):
+ User = apps.get_model("accounts", "User")
+ for user in User.objects.all():
+ user.username = user.email.rpartition("@")[0]
+ user.save()
+
+def depopulate_usernames(apps, schema_editor):
+ User = apps.get_model("accounts", "User")
+ User.objects.update(username=None)
+
+class Migration(migrations.Migration):
+ dependencies = [("accounts", "0002_add_username")]
+ operations = [migrations.RunPython(populate_usernames, depopulate_usernames)]
+To test migrations like this, I decided to write a small class to +run the migrations. You may notice that this is a class with a single +method and __init__, so it could just be a function, but setting +the apps attribute felt better than returning apps directly.
+from django.db import connection
+from django.db.migrations.executor import MigrationExecutor
+
+
+class Migrator:
+ def __init__(self, connection=connection):
+ self.executor = MigrationExecutor(connection)
+
+ def migrate(self, app_label: str, migration: str):
+ target = [(app_label, migration)]
+ self.executor.loader.build_graph()
+ self.executor.migrate(target)
+ self.apps = self.executor.loader.project_state(target).apps
+There's not too much going on here. We initialise the class with a +migration executor, using a passed connection, or the default connection +if none is passed. Then you can run the migrate method, with your app +label and migration name. This will run the migration and set the apps +attribute that you'll use in your test to make sure you have the right +version of the models, similar to how it's used in the data migration itself.
+Using the migrator in pytest is pretty simple. +We can write a single-line fixture:
+@pytest.fixture
+def migrator():
+ return Migrator
+Then we can write our tests, migrating to where we need to be:
+@pytest.mark.django_db
+def test_populate_emails(migrator):
+ migrator = migrator()
+ migrator.migrate("accounts", "0002_add_username")
+ User = migrator.apps.get_model("accounts", "User")
+ user = User.objects.create_user(email="test123@example.com")
+ assert user.username is None
+
+ migrator.migrate("accounts", "0003_populate_usernames")
+ assert User.objects.filter(email="test123@example.com", username="test123").exists()
+Of course, we can also migrate backwards. Typically, migrating backwards +is only used when developing, but in case you want that 100% coverage or +really want to be sure:
+@pytest.mark.django_db
+def test_depopulate_emails(migrator):
+ migrator = migrator()
+ migrator.migrate("accounts", "0002_add_username")
+ User = migrator.apps.get_model("accounts", "User")
+ user = User.objects.create_user(email="test123@example.com")
+ migrator.migrate("accounts", "0003_populate_usernames")
+ migrator.migrate("accounts", "0002_add_username")
+ User = migrator.apps.get_model("accounts", "user")
+ assert User.objects.get(email="test123@example.com").username is None
+If you're using Django's default unittest framework, +you can use it in much the same way:
+from django.test import TestCase
+
+
+class MigrationTest(TestCase):
+ def setUp(self):
+ self.migrator = Migrator()
+ self.migrator.migrate("accounts", "0002_add_username")
+
+ def test_populate_usernames(self):
+ User = self.migrator.apps.get_model("accounts", "User")
+ user = User.objects.create_user(email="test123@example.com")
+ assert user.username is None
+
+ migrator.migrate("accounts", "0003_populate_usernames")
+ assert User.objects.filter(
+ email="test123@example.com", username="test123"
+ ).exists()
+
+ def test_depopulate_emails(migrator):
+ User = migrator.apps.get_model("accounts", "User")
+ user = User.objects.create_user(email="test123@example.com")
+ migrator.migrate("accounts", "0003_populate_usernames")
+ migrator.migrate("accounts", "0002_add_username")
+ User = migrator.apps.get_model("accounts", "user")
+ assert User.objects.get(email="test123@example.com").username is None
+So, get out there and test those data migrations.
+Back in 2022, I wrote about automatically updating dependencies with dependabot. However, this can be quite tedious. If you have a lot of dependencies, or even if you don't, you'll likely get spammed with many pull requests on your scheduled update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, newer versions will often have performance improvements, bug fixes and generally fewer problems. Keeping them up to date over multiple codebases can be quite painful, but it's possible to bring this effort down to close to zero with a …
You probably already know the value of testing your code. Your Django migrations are code, therefore you should test them. However, testing data migrations in particular can be tricky, and there's no documentation on how to do it.
Typically your schema migrations don't need any testing as your migrations are …
Back in 2022, I wrote about automatically updating dependencies with dependabot. +However, this can be quite tedious. If you have a lot of dependencies, or even +if you don't, you'll likely get spammed with many pull requests on your scheduled +update time. This can be quite annoying, especially if you …
Outdated packages can open your application up to security issues. Moreover, +newer versions will often have performance improvements, bug fixes and generally +fewer problems. Keeping them up to date over multiple codebases can be quite +painful, but it's possible to bring this effort down to close to zero with a …
You probably already know the value of testing your code. +Your Django migrations are code, therefore you should test them. +However, testing data migrations in particular can be tricky, +and there's no documentation on how to do it.
+Typically your schema migrations don't need any testing as your +migrations are …
Hello. I'm Tom Carrick, and this is my curriculum vitae, or my resume, if you prefer. Maybe I linked this to you instead of sending a PDF, maybe I saved this as a PDF and sent it to you,or perhaps you found your own way here.
Originally from the North-East of England, but I've been living in Amsterdam, Netherlands since the beginning of 2019. I'm a software engineer working mostly with web technologies, particularly Python / Django / PostgreSQL. Usually I'm building complex web applications with REST APIs.
This CV is ordered in such a way that the most important things are at top, and the less useful things are at the bottom. So don't feel like you need to read the whole thing. It's also not 100% complete — there's a lot of freelance work I've excluded, as well as less relevant work experience. With that said, let's get on with it.
I contribute to open source projects where I can. This mostly manifests in occasional contributions to Django .
I'm a semi-regular coach at Django Girls events. I try to get to the ones at DjangoCon EU and anything reasonably close to where I happen to be at the time.
I'm a member of and helped create Django's Accessibility Team. It's still in its formative days and we're still setting things up.
I'm helping (in a small way) to organise DjangoCon Europe 2023.
↖️ You are here. There's not much to say about it, but I've written up the technical details regardless.
django-snakeoil is a project I've had for a while for keeping SEO and other meta tags updated in a Django project. It works well enough for me, but since I rewrote this site in Pelican I don't use it myself anymore.
Carbon Timeline (code) is a small project I wrote to teach myself Vue.js. It uses your Google Maps timeline data (downloaded from Google Takeout) to estimate the carbon footprint of your travel using Mike Berners-Lee's (yes, Tim's brother) estimates from his book How Bad Are Bananas?. It's a nice project but I'm more than 99% sure that I'm the only perosn to ever use it. Maybe that will change now that you've read this?
harmony is just a discord bot with some fun commands. It's not very interesting 🙂.
Masters thesis: Detecting cheating in online poker. Developed applications to analyse more than 20 million hands of online poker in order to detect collusion, suspicious play, and other patterns. Included a quite early use of Neo4j, a graph database.
This degree was partially taught, and partially research-based. In addition to thesis, the teaching had a focus on security and the web.
Your typical undergraduate CS degree 🤷.
I know, nobody cares, this shouldn't be on your CV, etc. etc., but this is a website, so I'm not wasting space on the page 🙂. Feel free to stop reading here.
I enjoy bouldering, but I'm not amazing at it. I typically boulder indoors at around font level 6b. I play a bit of tennis occasionally but am extremely bad at it.
I would code for fun if it wasn't my full time job.
I enjoy hiking, foraging, anything outdoors in nature, really.
I spend too much time thinking about climate change.
+ Hello. I'm Tom Carrick, and this is my curriculum vitae, or my resume, + if you prefer. Maybe I linked this to you instead of sending a PDF, + maybe I saved this as a PDF and sent it to you,or perhaps you found + your own way here. +
+ ++ Originally from the North-East of England, but I've been living in + Amsterdam, Netherlands since the beginning of 2019. + I'm a software engineer working mostly with web technologies, + particularly Python / Django / PostgreSQL. Usually I'm building complex + web applications with REST APIs. +
+ ++ This CV is ordered in such a way that the most important things are at + top, and the less useful things are at the bottom. So don't feel like + you need to read the whole thing. It's also not 100% complete — + there's a lot of freelance work I've excluded, as well as less relevant + work experience. With that said, let's get on with it. +
++ I contribute to open source projects where I can. + This mostly manifests in occasional + + contributions to Django + . +
++ I'm a semi-regular coach at + Django Girls events. + I try to get to the ones at DjangoCon EU + and anything reasonably close to where I happen to be at the time. +
++ I'm a member of and helped create Django's Accessibility Team. + It's still in its formative days and we're still setting things up. +
++ I'm helping (in a small way) to organise + DjangoCon Europe 2023. +
++ ↖️ You are here. There's not much to say about it, but I've written up + the technical details regardless. +
++ django-snakeoil + is a project I've had for a while for keeping SEO and other meta tags + updated in a Django project. It works well enough for me, but since I + rewrote this site in Pelican I don't use it myself anymore. +
++ Carbon Timeline + (code) is a small + project I wrote to teach myself Vue.js. It uses your Google Maps timeline + data (downloaded from Google Takeout) to estimate the carbon footprint of + your travel using Mike Berners-Lee's (yes, Tim's brother) estimates from + his book How Bad Are Bananas?. It's a nice project but I'm + more than 99% sure that I'm the only perosn to ever use it. + Maybe that will change now that you've read this? +
++ harmony is just a discord bot + with some fun commands. It's not very interesting 🙂. +
++ Masters thesis: Detecting cheating in online poker. + Developed applications to analyse more than 20 million hands of online poker + in order to detect collusion, suspicious play, and other patterns. Included + a quite early use of Neo4j, a graph database. +
++ This degree was partially taught, and partially research-based. + In addition to thesis, the teaching had a focus on security and the web. +
+Your typical undergraduate CS degree 🤷.
++ I know, nobody cares, this shouldn't be on your CV, etc. etc., but this + is a website, so I'm not wasting space on the page 🙂. + Feel free to stop reading here. +
++ I enjoy bouldering, but I'm not amazing at it. I typically boulder indoors + at around font level 6b. I play a bit of tennis occasionally but am + extremely bad at it. +
+I would code for fun if it wasn't my full time job.
+I enjoy hiking, foraging, anything outdoors in nature, really.
+I spend too much time thinking about climate change.
+Hello there 👋🏻
I'm Tom Carrick, professional web developer. I work mostly with Python, Django, PostgreSQL, and sometimes FastAPI. I'm currently rekindling my JS skills.
The most useful things on this site at the moment are the blog and my CV.
My focus is on web development. Usually I work on application backends but I also have an interest in frontend development, particularly around accessibility. As you can perhaps tell from the design of this site, I also have an interest in keeping page sizes down and designs uncluttered to help reduce the CO2e emissions of sites, and I'm also interested in using infrastructure in the most efficient and least carbon-intensive way possible.
I'm a semi-regular contributor to Django, and occasionally to other open source projects. I sometimes coach for Django Girls workshops (pandemic permitting), and I still don't feel like I'm doing enough.
Hello there 👋🏻
+I'm Tom Carrick, professional web developer. I work mostly with Python, +Django, PostgreSQL, and sometimes FastAPI. I'm currently rekindling my JS skills.
+The most useful things on this site at the moment are the blog and my CV.
+My focus is on web development. Usually I work on application backends but I also +have an interest in frontend development, particularly around accessibility. As +you can perhaps tell from the design of this site, I also have an interest in keeping +page sizes down and designs uncluttered to help reduce the CO2e emissions of sites, +and I'm also interested in using infrastructure in the most efficient and least +carbon-intensive way possible.
+I'm a semi-regular contributor to Django, and occasionally to other open source +projects. I sometimes coach for Django Girls workshops (pandemic permitting), +and I still don't feel like I'm doing enough.
+ +