Skip to content

Automated script runner aka "Migrations" for Neo4j. Inspired by Flyway.

Notifications You must be signed in to change notification settings

katya-dovgalets/neo4j-migrations

 
 

Repository files navigation

Neo4j Migrations

This is a tool for defining Java based migrations that should be applied and recorded against a Neo4j instance. The only dependencies are the Neo4j Java driver and ClassGraph, the later being used to find migrations on the classpath.

Neo4j Migrations is inspired by FlywayDB, which is an awesome tool for migration of relational databases.

badge badge

Usage

As standalone CLI

Since 0.0.5, you can grab binaries of neo4j-migrations CLI version at Maven central: neo4j-migrations-0.1.1.zip

Unzip them (unzip neo4j-migrations-cli-0.1.1.zip), go the the director (cd neo4j-migrations-0.1.1) and execute the program (neo4j-migrations on Unix-like operating systems and neo4j-migrations.cmd on Windows):

./neo4j-migrations -h
Usage: neo4j-migrations [-hvV] -p[=<password>] [-p[=<password>]]...
                        [-a=<address>] [-d=<database>]
                        [--transaction-mode=<transactionMode>] [-u=<user>]
                        [--location=<locationsToScan>]...
                        [--package=<packagesToScan>]... [COMMAND]
Migrates Neo4j databases.
  -a, --address=<address>   The address this migration should connect to. The
                              driver supports bolt, bolt+routing or neo4j as
                              schemes.
  -d, --database=<database> The database to migration (Neo4j 4.0+).
  -h, --help                Show this help message and exit.
      --location=<locationsToScan>
                            Location to scan. Repeat for multiple locations.
  -p, --password[=<password>]
                            The password of the user connecting to the database.
      --package=<packagesToScan>
                            Package to scan. Repeat for multiple packages.
      --transaction-mode=<transactionMode>
                            The transaction mode to use.
  -u, --username=<user>     The login of the user connecting to the database.
  -v                        Log the configuration and a couple of other things.
  -V, --version             Print version information and exit.
Commands:
  info     Retrieves all applied and pending informations, prints them and
             exits.
  migrate  Retrieves all pending migrations, verify and applies them.

Here’s an example that looks for migrations in a Java package and it’s subpackages and in a filesystem location for Cypher based migrations. It uses the info command to tell you which migrations have been applied and which not:

./neo4j-migrations -uneo4j -psecret --package some.migrations  --location file:$HOME/Desktop/foo info

Database: Neo4j/4.0.0@localhost:7687

+---------+-----------------------------+--------+--------------+----+----------------+---------+--------------------------------------------------------+
| Version | Description                 | Type   | Installed on | by | Execution time | State   | Source                                                 |
+---------+-----------------------------+--------+--------------+----+----------------+---------+--------------------------------------------------------+
| 001     | FirstMigration              | JAVA   |              |    |                | PENDING | some.migrations.changeset1.V001__FirstMigration        |
| 002     | AnotherMigration            | JAVA   |              |    |                | PENDING | some.migrations.changeset1.V002__AnotherMigration      |
| 023     | NichtsIstWieEsScheint       | JAVA   |              |    |                | PENDING | some.migrations.changeset2.V023__NichtsIstWieEsScheint |
| 025     | SlowMigration               | JAVA   |              |    |                | PENDING | some.migrations.changeset3.V025__SlowMigration         |
| 030     | Something based on a script | CYPHER |              |    |                | PENDING | V030__Something_based_on_a_script.cypher               |
| 042     | The truth                   | CYPHER |              |    |                | PENDING | V042__The truth.cypher                                 |
+---------+-----------------------------+--------+--------------+----+----------------+---------+--------------------------------------------------------+

You can repeat both --package and --location parameter for fine grained control. Use migrate to apply migrations:

./neo4j-migrations -uneo4j -psecret --package some.migrations.changeset1 --package some.migrations.changeset2 migrate
Applied migration 001 ("FirstMigration")
Applied migration 002 ("AnotherMigration")
Applied migration 023 ("NichtsIstWieEsScheint")
Database migrated to version 023.

If we go back to the info example above and grab all migrations again, we find the following result:

./neo4j-migrations -uneo4j -psecret --package some.migrations  --location file:$HOME/Desktop/foo info

Database: Neo4j/4.0.0@localhost:7687

+---------+-----------------------------+--------+-------------------------------+---------------+----------------+---------+--------------------------------------------------------+
| Version | Description                 | Type   | Installed on                  | by            | Execution time | State   | Source                                                 |
+---------+-----------------------------+--------+-------------------------------+---------------+----------------+---------+--------------------------------------------------------+
| 001     | FirstMigration              | JAVA   | 2020-01-17T15:34:16.388Z[UTC] | msimons/neo4j | PT0S           | APPLIED | some.migrations.changeset1.V001__FirstMigration        |
| 002     | AnotherMigration            | JAVA   | 2020-01-17T15:34:16.406Z[UTC] | msimons/neo4j | PT0S           | APPLIED | some.migrations.changeset1.V002__AnotherMigration      |
| 023     | NichtsIstWieEsScheint       | JAVA   | 2020-01-17T15:34:16.417Z[UTC] | msimons/neo4j | PT0S           | APPLIED | some.migrations.changeset2.V023__NichtsIstWieEsScheint |
| 025     | SlowMigration               | JAVA   |                               |               |                | PENDING | some.migrations.changeset3.V025__SlowMigration         |
| 030     | Something based on a script | CYPHER |                               |               |                | PENDING | V030__Something_based_on_a_script.cypher               |
| 042     | The truth                   | CYPHER |                               |               |                | PENDING | V042__The truth.cypher                                 |
+---------+-----------------------------+--------+-------------------------------+---------------+----------------+---------+--------------------------------------------------------+

Another migrate - this time with all packages - gives us the following output and result:

./neo4j-migrations -uneo4j -psecret --package some.migrations  --location file:$HOME/Desktop/foo migrate
Skipping already applied migration 001 ("FirstMigration")
Skipping already applied migration 002 ("AnotherMigration")
Skipping already applied migration 023 ("NichtsIstWieEsScheint")
Applied migration 025 ("SlowMigration")
Applied migration 030 ("Something based on a script")
Applied migration 042 ("The truth")
Database migrated to version 042.

./neo4j-migrations -uneo4j -psecret --package some.migrations  --location file:$HOME/Desktop/foo info

Database: Neo4j/4.0.0@localhost:7687

+---------+-----------------------------+--------+-------------------------------+---------------+----------------+---------+--------------------------------------------------------+
| Version | Description                 | Type   | Installed on                  | by            | Execution time | State   | Source                                                 |
+---------+-----------------------------+--------+-------------------------------+---------------+----------------+---------+--------------------------------------------------------+
| 001     | FirstMigration              | JAVA   | 2020-01-17T15:34:16.388Z[UTC] | msimons/neo4j | PT0S           | APPLIED | some.migrations.changeset1.V001__FirstMigration        |
| 002     | AnotherMigration            | JAVA   | 2020-01-17T15:34:16.406Z[UTC] | msimons/neo4j | PT0S           | APPLIED | some.migrations.changeset1.V002__AnotherMigration      |
| 023     | NichtsIstWieEsScheint       | JAVA   | 2020-01-17T15:34:16.417Z[UTC] | msimons/neo4j | PT0S           | APPLIED | some.migrations.changeset2.V023__NichtsIstWieEsScheint |
| 025     | SlowMigration               | JAVA   | 2020-01-17T15:36:06.899Z[UTC] | msimons/neo4j | PT0.503S       | APPLIED | some.migrations.changeset3.V025__SlowMigration         |
| 030     | Something based on a script | CYPHER | 2020-01-17T15:36:07.001Z[UTC] | msimons/neo4j | PT0.004S       | APPLIED | V030__Something_based_on_a_script.cypher               |
| 042     | The truth                   | CYPHER | 2020-01-17T15:36:07.016Z[UTC] | msimons/neo4j | PT0.003S       | APPLIED | V042__The truth.cypher                                 |
+---------+-----------------------------+--------+-------------------------------+---------------+----------------+---------+--------------------------------------------------------+

From your build tool

Maven Plugin

You can trigger Neo4j Migrations from your build via plugin:

<plugin>
    <groupId>eu.michael-simons.neo4j</groupId>
    <artifactId>neo4j-migrations-maven-plugin</artifactId>
    <version>0.1.1</version>
    <executions>
        <execution>
            <id>migrate</id>
            <goals>
                <goal>migrate</goal>
            </goals>
            <configuration>
                <user>neo4j</user>
                <password>secret</password>
                <address>bolt://localhost:${it-database-port}</address>
                <verbose>true</verbose>
            </configuration>
        </execution>
    </executions>
</plugin>

By default, the plugin will look in neo4j/migrations resource. You can change that via locationsToScan:

<locationsToScan>
    <locationToScan>file://${project.build.outputDirectory}/custom/path/to/migrate</locationToScan>
</locationsToScan>

Add multiple elements for multiple locations. The plugin has the same parameters as the standalone or CLI version.

Inside your application

In a Spring Boot application

We provide a starter with automatic configuration for Spring Boot. Declare the following dependency in your Spring Boot application:

<dependency>
    <groupId>eu.michael-simons.neo4j</groupId>
    <artifactId>neo4j-migrations-spring-boot-starter</artifactId>
    <version>0.1.1</version>
</dependency>

That starter itself depends on the Neo4j Java Driver. The driver is managed by Spring Boot since 2.4 and you can enjoy configuration support directly through Spring Boot. For Boot versions prior to Spring Boot 2.4, please have a look at version 0.0.13 of this library.

Neo4j Migrations will automatically look for migrations in classpath:neo4j/migrations and will fail if this location does not exists. It does not scan by default for Java based migrations.

Here’s an example on how to configure the driver and the migrations:

spring.neo4j.authentication.username=neo4j
spring.neo4j.authentication.password=secret
spring.neo4j.uri=bolt://localhost:7687

# Add configuration for your migrations, for example, additional packages to scan
org.neo4j.migrations.packages-to-scan=your.changesets, another.changeset

# Or disable the check if the location exists
org.neo4j.migrations.check-location=false

The following configuration properties are supported:

Name Type Default Description

org.neo4j.migrations.check-location

java.lang.Boolean

true

Whether to check that migration scripts location exists.

org.neo4j.migrations.database

java.lang.String

null

The database that should be migrated (Neo4j 4.0+ only). Leave {@literal null} for using the default database.

org.neo4j.migrations.enabled

java.lang.Boolean

true

Whether to enable Neo4j migrations or not.

org.neo4j.migrations.encoding

java.nio.charset.Charset

UTF-8

Encoding of Cypher migrations.

org.neo4j.migrations.installed-by

java.lang.String

System user

Username recorded as property {@literal by} on the MIGRATED_TO relationship.

org.neo4j.migrations.locations-to-scan

java.lang.String[]

classpath:neo4j/migrations

Locations of migrations scripts.

org.neo4j.migrations.packages-to-scan

java.lang.String[]

An empty array

List of packages to scan for Java migrations.

org.neo4j.migrations.transaction-mode

TransactionMode

PER_MIGRATION

The transaction mode in use (Defaults to "per migration", meaning one script is run in one transaction).

Note
Migrations can be disabled by setting org.neo4j.migrations.enabled to false.

Other applications

Declare the extension as Maven dependency:

<dependency>
    <groupId>eu.michael-simons.neo4j</groupId>
    <artifactId>neo4j-migrations</artifactId>
    <version>0.1.1</version>
</dependency>

Put your migrations as Java classes into your project:

import ac.simons.neo4j.migrations.core.JavaBasedMigration;
import ac.simons.neo4j.migrations.core.MigrationContext;

import org.neo4j.driver.Driver;
import org.neo4j.driver.Session;

public class V001__MyFirstMigration implements JavaBasedMigration {

    @Override
    public void apply(MigrationContext context) {
        try (Session session = context.getSession()) { // (1)
            // Steps necessary for a migration
        }
    }
}
  1. It is important that you use the supplied session (or the SessionConfig if you want to use another type of Session) for your session retrieval, otherwise you may run that migration in a different database than in which the tool itself is run. However, you are free in which database you run this. Your mileage may vary.

The class names must start with a V followed by digits followed by __ and than some valid Java class name.

To use them create a Migrations instance to scan your project and apply all found migrations:

Migrations migrations = new Migrations(
    MigrationsConfig.builder().withPackagesToScan("org.company.changeset1").build(),
    GraphDatabase.driver("bolt://localhost:7687", AuthTokens.basic("neo4j", "secret"))
);

migrations.apply();

You’re migrations will be recorded as a chain of applied migrations (as nodes with the label __Neo4jMigration). They can use the driver any way they like.

There’s no rollback yet. If any migration fails, the chain will stop, but will not rollback previous migrations.

Cypher script based migrations

You can put Cypher scripts ending with .cypher inside your classpath resources under neo4j/migrations. From there on they’ll be picked up automatically.

Here’s an example:

Listing 1. neo4j/migrations/V007__BondTheNameIsBond.cypher
CREATE (agent:`007`) RETURN agent;
UNWIND RANGE(1,6) AS i
WITH i CREATE (n:OtherAgents {idx: '00' + i})
RETURN n
;

Scripts can contain multiple statements, separated by a ; followed by a newline. Statements will be executed in one transaction by default. That behaviour can be changed as follows:

Migrations migrations= new Migrations(
    MigrationsConfig.builder()
        .withTransactionMode(MigrationsConfig.TransactionMode.PER_STATEMENT)
        .build(),
    driver
);
migrations.apply();

If you want Migrations to look at other places, configure it as follows:

Migrations migrations = new Migrations(
    MigrationsConfig.builder()
        .withLocationsToScan(
            "classpath:my/awesome/migrations",
            "file:/path/to/migration"
        ).build(),
    GraphDatabase.driver("bolt://localhost:7687", AuthTokens.basic("neo4j", "secret"))
);

migrations.apply();

For the adventurous

The Neo4j Java driver and this application supports native compilation with GraalVM so that you can create a native executable. Read more about it here.

After installing at least GraalVM 20.1.0, prepare your environment as follows:

export GRAALVM_HOME=/Library/Java/JavaVirtualMachines/graalvm-ce-java11-20.1.0/Contents/Home
export JAVA_HOME=$GRAALVM_HOME

The paths are probably different on your system.

The neo4j-migrations-cli module has a build profile create-native-image that you use to create a binary for your OS. Run it with:

./mvnw -Pcreate-native-image clean package

The resulting migration tool can only be used to load Cypher script based migrations in file locations. It won’t find classes, as those are instantiated dynamically via reflection, which is not supported in full in a native image.

About

Automated script runner aka "Migrations" for Neo4j. Inspired by Flyway.

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 98.2%
  • Other 1.8%