Skip to content

Commit

Permalink
update README with a couple of diagrams
Browse files Browse the repository at this point in the history
  • Loading branch information
shortishly committed Oct 10, 2024
1 parent 4bd8189 commit 94a80bc
Showing 1 changed file with 72 additions and 20 deletions.
92 changes: 72 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,56 @@
# Tansu

[Tansu][github-com-tansu-io] is an Apache Kafka API compatible broker
with PostgreSQL and S3 storage engines. Acting as a drop in replacement,
existing clients connect to Tansu, producing and fetching messages.
Written in 100% safe async 🦀 [Rust][rust-lang-org] 🚀.
[Tansu][github-com-tansu-io] is a modern drop-in replacement for
Apache Kafka. Without the cost of broker replicated storage for
durability. Tansu is in **early** development, licensed under
the [GNU AGPL][agpl-license]. Written in async 🚀
[Rust][rust-lang-org] 🦀

Tansu brokers are:

- Kafka API compatible (exceptions: transactions and idempotent
producer)
- Stateless with instant scaling up or down. No more planning and
reassigning partitions to a broker
- Available with PostgreSQL or S3 storage engines

For data durability:

- S3 is designed to exceed [99.999999999% (11
nines)][aws-s3-storage-classes]
- PostgreSQL with [continuous archiving][continuous-archiving]
streaming transaction logs files to an archive.

Stateless brokers are cost effective, with no network replication and
duplicate data storage charges.

Stateless brokers do not have the ceremony of [Raft][raft-consensus]
or [ZooKeeper][apache-zookeeper].

## S3

Tansu with the S3 storage engine:
You can have 3 brokers running in separate [Availability
Zones][aws-regions-zones] for resilience:

![Tansu S3](https://shortishly.com/assets/images/tansu-s3-2024-10-09.svg)

Each broker is stateless. Brokers can come and go. Without affecting
leadership of consumer groups. The leader and In-Sync-Replica is the
broker serving your request. No more client broker ping pong.

With Tansu there is no replication between brokers. The data transfer
cost between the [Availability Zones][aws-regions-zones] is a $0 line
item. There are $0 in duplicated storage charges.

- Brokers do not replicate messages, instead relying on the underlying
storage for resilience.
- Brokers do not require a consensus protocol (e.g., Raft or
Zookeeper), instead conditional PUTs are used to coordinate state.
- Brokers are stateless.
- All brokers are leaders.
With stateless brokers, you can run Tansu in a [server-less
architecture][wikipedia-serverless]:

Note that, Tansu requires that the underlying S3 service support conditional
![Tansu Lambda S3](https://shortishly.com/assets/images/tansu-lambda-s3-2024-10-09.svg)

Spin up a broker for the duration of a Kafka API request. Then spin
down. No more idle brokers.

Tansu requires that the underlying S3 service support conditional
PUT requests. While
[AWS S3 does now support conditional writes][aws-s3-conditional-writes],
the support is
Expand All @@ -25,11 +59,16 @@ To have stateless brokers we need to
[use a compare and set operation][tigris-conditional-writes],
which is not currently available in AWS S3.

Much like the Kafka protocol, the open nature of the S3 protocol allows vendors
to differentiate with different levels of service while retaining compatibility
with the underlying API. We have tried [minio][min-io]
and [tigis][tigris-conditional-writes], among a number of other vendors now supporting
conditional put.
Much like the Kafka protocol, the S3 protocol allows vendors to
differentiate. Different levels of service while retaining
compatibility with the underlying API. You can use [minio][min-io] or
[tigis][tigris-conditional-writes], among a number of other vendors
supporting conditional put.

Tansu uses [object store][crates-io-object-store], providing a
multi-cloud API for storage. There is an alternative option to use a
[DynamoDB-based commit protocol, to provide conditional write support
for AWS S3][object-store-dynamo-conditional-put] instead.

### configuration

Expand All @@ -51,7 +90,8 @@ Just bring minio up, without tansu:
docker compose up -d minio
```

The minio console should now be running on http://localhost:9001, login using
The minio console should now be running on
[http://localhost:9001](http://localhost:9001), login using
the default user credentials of "minioadmin", with password "minioadmin". Follow
the [bucket creation instructions][minio-create-bucket]
to create a bucket called "tansu", and then
Expand Down Expand Up @@ -201,18 +241,30 @@ Consumer:
-b localhost:9092
```

## Feedback

Please [raise an issue][tansu-issues] if you encounter a problem.

## License

Tansu is licensed under the [GNU AGPL][agpl-license].

[agpl-license]: https://www.gnu.org/licenses/agpl-3.0.en.html
[apache-zookeeper]: https://en.wikipedia.org/wiki/Apache_ZooKeeper
[aws-regions-zones]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
[aws-s3-conditional-requests]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html
[aws-s3-conditional-writes]: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/
[aws-s3-storage-classes]: https://aws.amazon.com/s3/storage-classes/
[continuous-archiving]: https://www.postgresql.org/docs/current/continuous-archiving.html
[crates-io-object-store]: https://crates.io/crates/object_store
[github-com-tansu-io]: https://github.com/tansu-io/tansu
[librdkafka]: https://github.com/confluentinc/librdkafka
[min-io]: https://min.io
[minio-create-access-key]: https://min.io/docs/minio/container/administration/console/security-and-access.html#id1
[minio-create-bucket]: https://min.io/docs/minio/container/administration/console/managing-objects.html#creating-buckets
[object-store-dynamo-conditional-put]: https://docs.rs/object_store/0.11.0/object_store/aws/struct.DynamoCommit.html
[raft-consensus]: https://raft.github.io
[rust-lang-org]: https://www.rust-lang.org
[tansu-issues]: https://github.com/tansu-io/tansu/issues
[tigris-conditional-writes]: https://www.tigrisdata.com/blog/s3-conditional-writes/
[minio-create-bucket]: https://min.io/docs/minio/container/administration/console/managing-objects.html#creating-buckets
[minio-create-access-key]: https://min.io/docs/minio/container/administration/console/security-and-access.html#id1
[wikipedia-serverless]: https://en.wikipedia.org/wiki/Serverless_computing

0 comments on commit 94a80bc

Please sign in to comment.