Go Migrations don't work

[paulrozhkin]

Goose v3.14.0

I have one .go migration file that I register with via AddMigrationContext. After registering and trying to migrate, I get an ErrNoMigrationFiles error.
I think the error is in collectMigrationsFS on line 297. Maybe there should be a !ok check here?

[mfridman]

There's a check to make sure registered Go migrations exist on the filesystem (os or embedded).

I realize the example doesn't highlight the important part, which is:

1. You need to init() that AddMigrationContext function
2. Blank import that package as part of your build

If you point me at a repo or an example, I could help you debug it. But, I think it's probably best I sat down and provided a blog/sample repo for how to do this step-by-step.

[edulop91]

hi @mfridman ! We're also running into this issue when we upgraded to 3.15. Things worked fine before.
Confirming we do both steps you suggest:

You need to init() that AddMigrationContext function
Blank import that package as part of your build

Since we have no fsGoMigrations, it seems like this block will never add any registeredGoMigrations

	// Go migrations registered via goose.AddMigration().
	for _, migration := range registeredGoMigrations {
		v, err := NumericComponent(migration.Source)
		if err != nil {
			return nil, fmt.Errorf("could not parse go migration file %q: %w", migration.Source, err)
		}
		if !versionFilter(v, current, target) {
			continue
		}
		if _, ok := fsGoMigrations[v]; ok {
			migrations = append(migrations, migration)
		}
	}

[tom5760]

Ran into the same issue. For us, we are including all of our SQL files in an embed.FS. A work-around we found was to also add the *.go to our embedded FS.

// Needed to add *.go below
//go:embed *.sql *.go
var schemaFS embed.FS

// ...

goose.SetBaseFS(schemaFS)

[mfridman]

Indeed, there was an assumption the .go files would live alongside the .sql files and could be read from either the os filesystem or an embedded filesystem.

But, that appears to be a wrong assumption. Sorry about that, I need to think about how to resolve this so it doesn't break folks from #553.

Two immediate solutions come to mind

• relax the filesystem check for go migrations
• add ability to provide a custom "golang migration registry", iirc there was an existing PR
  • add ability to replace golang migration registry with custom one #351

[mfridman]

@edulop91 @paulrozhkin is it safe to assume you're also embedding migrations similar to @tom5760 ?

[paulrozhkin]

@mfridman Now I made downgrade to previous version 3.13.4.
I expected that AddMigration in init() to not require embed resources.

[mfridman]

Alright, I think there's a path forward here.

1. If Go migrations have been registered, but there are no corresponding .go files in the filesystem, add them to the migrations slice (old behaviour). This should resolve the current issue

• Go Migrations don't work #582

2. If Go migrations have been registered, and there are .go files in the filesystem, only include those in the migrations slices. This should maintain the desired behaviour from

• Go migrations run even if they are outside of the target migrations directory #552
• Filter registered migrations by fs dirpath #553

Now, there's one property I'd like to maintain, if .go files (that have a version number) exist in the filesystem that have not been registered raise an error.

Return an error if the given sources contain a versioned Go migration that has not been
registered. This is a sanity check to ensure users didn't accidentally create a valid
looking Go migration file and forget to register it.

This is almost always a user error.

[mfridman]

Put up a PR to resolve this issue. #588

If anyone would like to provide early feedback (or even better, take this branch for a spin) that'd be great.

TL;DR #588 should bring back the original behaviour, while also providing a way to filter migrations if they exist in the filesystem.

EDIT: also added a bunch of tests to capture all 3 scenarios mentioned in #582 (comment)