Go-Hit

This is a KV-store tool implemented in Go.

Simple objective: To make a tool which is super scalable, fast, can handle tons of data for scalability.

To start the server:

go run cmd/go-hit-server/main.go

Simple objective of this is to create the best of all worlds like solution that can support multiple backend storage. Also, it should be concurrent in nature.

Why do we need this?

• Easy to digest, fast to read and fault tolerant to restart.
• Can support 100k writes a second and data reload (on restart) should be atleast as fast as 100k read per sec. Note that we are just talking about a simple benchmark of write a {"key":"value"} no matter how we store in rdbms, nosql, file storage, etc.

Benchmark Report:

Benchmark Name	Operations	Time per Operation	Memory Allocs per Operation	Notes
Postgres
BenchmarkSaveToDisk	8886	138285 ns/op	462 B/op
BenchmarkLoad	9631	2622185 ns/op	2046178 B/op
Postgres with IO concurrency = 10
BenchmarkSaveToDisk	29551	37869 ns/op	493 B/op
BenchmarkLoad	1450	7427697 ns/op	3173033 B/op
SFW with IO concurrency = 10
BenchmarkSaveToDisk	16446	104123 ns/op	782 B/op
BenchmarkSaveToDisk(without cleanup)	30753	35351 ns/op	4862 B/op
BenchmarkLoad	1704	9426406 ns/op	3909961 B/op
Cassandra with IO concurrency = 500
BenchmarkSaveToDisk	103744	9803 ns/op	2899 B/op
BenchmarkLoad	638	2193735 ns/op	132143 B/op

One interesting thing to notice about sfw and cassandra load operation is that when we do load operation without benchmarking tool, it works pretty fast