runtime error: index out of range when loading data

[agramsci]

Description

Cayley 0.7.4 fails with a panic when trying to load a Freebase dump.

Steps to reproduce the issue:

1. cayley --db leveldb --dbpath graphdb load --init freebase.pq.gz

Received results:

$ ../cayley_0.7.4_linux_amd64/cayley --db leveldb --dbpath graphdb load --init freebase.pq.gz --verbose 2 --batch 100000
I0911 22:30:35.186468   15954 cayley.go:63] Cayley version: 0.7.4 (431ee2850fa29f8f7dfa0de6a03c07237871d504)
I0911 22:30:35.186583   15954 database.go:187] using backend "leveldb" (graphdb)
I0911 22:30:38.687174   15954 load.go:155] Wrote 100000 quads.
[...snip...]
I0912 03:02:06.054281   15954 load.go:155] Wrote 84600000 quads.
I0912 03:02:16.663510   15954 load.go:155] Wrote 84700000 quads.
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/cayleygraph/cayley/vendor/github.com/tylertreat/BoomFilters.(*Buckets).setBits(0xc420280090, 0x100000078, 0xc400000001)
	/go/src/github.com/cayleygraph/cayley/vendor/github.com/tylertreat/BoomFilters/buckets.go:101 +0x128
github.com/cayleygraph/cayley/vendor/github.com/tylertreat/BoomFilters.(*Buckets).Set(0xc420280090, 0x78, 0x1, 0x11106c0)
	/go/src/github.com/cayleygraph/cayley/vendor/github.com/tylertreat/BoomFilters/buckets.go:62 +0x53
github.com/cayleygraph/cayley/vendor/github.com/tylertreat/BoomFilters.(*DeletableBloomFilter).Add(0xc420286000, 0xc4200de5a0, 0x18, 0x18, 0x8, 0x0)
	/go/src/github.com/cayleygraph/cayley/vendor/github.com/tylertreat/BoomFilters/deletable.go:96 +0x106
github.com/cayleygraph/cayley/graph/kv.(*QuadStore).bloomAdd(0xc4200fc6e0, 0xc425f64f88)
	/go/src/github.com/cayleygraph/cayley/graph/kv/indexing.go:951 +0xca
github.com/cayleygraph/cayley/graph/kv.(*QuadStore).indexLink(0xc4200fc6e0, 0x110cc00, 0xc42ccdde00, 0xc425f64f88, 0x0, 0x0)
	/go/src/github.com/cayleygraph/cayley/graph/kv/indexing.go:535 +0x22d
github.com/cayleygraph/cayley/graph/kv.(*QuadStore).indexLinks(0xc4200fc6e0, 0x110c900, 0xc4200c2018, 0x110cc00, 0xc42ccdde00, 0xc43d188000, 0x1864f, 0x1864f, 0x0, 0x0)
	/go/src/github.com/cayleygraph/cayley/graph/kv/indexing.go:518 +0x116
github.com/cayleygraph/cayley/graph/kv.(*QuadStore).ApplyDeltas(0xc4200fc6e0, 0xc43588c000, 0x1864f, 0x186a0, 0xc42f6b0001, 0x0, 0x0)
	/go/src/github.com/cayleygraph/cayley/graph/kv/indexing.go:408 +0xbe6
github.com/cayleygraph/cayley/writer.(*Single).AddQuadSet(0xc4202080a0, 0xc42071c000, 0x186a0, 0x186a0, 0xc42f6b3190, 0x1104b20)
	/go/src/github.com/cayleygraph/cayley/writer/single.go:69 +0x1cd
github.com/cayleygraph/cayley/graph.(*batchWriter).WriteQuads(0xc4201b4000, 0xc42071c000, 0x186a0, 0x186a0, 0xc4200d2140, 0xc42071c000, 0x186a0)
	/go/src/github.com/cayleygraph/cayley/graph/quadwriter.go:226 +0x55
github.com/cayleygraph/cayley/internal.(*batchLogger).WriteQuads(0xc4200d2160, 0xc42071c000, 0x186a0, 0x186a0, 0x186a0, 0x0, 0x0)
	/go/src/github.com/cayleygraph/cayley/internal/load.go:152 +0x59
github.com/cayleygraph/cayley/quad.CopyBatch(0x10fd1c0, 0xc4200d2160, 0x7f3e366d8040, 0xc4200d2140, 0x186a0, 0xc4200d2140, 0x0, 0x0)
	/go/src/github.com/cayleygraph/cayley/quad/rw.go:142 +0x1d3
github.com/cayleygraph/cayley/internal.DecompressAndLoad(0x1110660, 0xc4202080a0, 0x186a0, 0x7ffc63d45565, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cayleygraph/cayley/internal/load.go:139 +0x1c7
github.com/cayleygraph/cayley/internal.Load(0x1110660, 0xc4202080a0, 0x186a0, 0x7ffc63d45565, 0xe, 0x0, 0x0, 0x102d3d7, 0x4)
	/go/src/github.com/cayleygraph/cayley/internal/load.go:22 +0x7d
github.com/cayleygraph/cayley/cmd/cayley/command.NewLoadDatabaseCmd.func1(0xc42029e000, 0xc4202a10e0, 0x1, 0xa, 0x0, 0x0)
	/go/src/github.com/cayleygraph/cayley/cmd/cayley/command/database.go:116 +0x272
github.com/cayleygraph/cayley/vendor/github.com/spf13/cobra.(*Command).execute(0xc42029e000, 0xc4202a0fa0, 0xa, 0xa, 0xc42029e000, 0xc4202a0fa0)
	/go/src/github.com/cayleygraph/cayley/vendor/github.com/spf13/cobra/command.go:620 +0x3e4
github.com/cayleygraph/cayley/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1848de0, 0x1849020, 0xc42016df78, 0x405c3c)
	/go/src/github.com/cayleygraph/cayley/vendor/github.com/spf13/cobra/command.go:699 +0x2d4
github.com/cayleygraph/cayley/vendor/github.com/spf13/cobra.(*Command).Execute(0x1848de0, 0xc42029e900, 0x0)
	/go/src/github.com/cayleygraph/cayley/vendor/github.com/spf13/cobra/command.go:658 +0x2b
main.main()
	/go/src/github.com/cayleygraph/cayley/cmd/cayley/cayley.go:187 +0x31

The same error happens with the bolt KV backend. The error occurs sooner when using a smaller batch size.

Expected results:

The data is loaded successfully.

Output of cayley version or commit hash:

Cayley version: 0.7.4
Git commit hash: 431ee2850fa29f8f7dfa0de6a03c07237871d504
Build date: 2018-06-03T15:44:12Z

Environment details:

Backend database: leveldb, bolt

[dennwc]

This must have been solved with a new BoomFilter version. Will check it, thanks!

[dennwc]

Fixed in #744