From 94a80bc3aafbb50706f12d553adedddd2e9f58ed Mon Sep 17 00:00:00 2001 From: Peter Morgan Date: Thu, 10 Oct 2024 16:14:40 +0100 Subject: [PATCH] update README with a couple of diagrams --- README.md | 92 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 72 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index a559c8e..ca244c4 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,56 @@ # Tansu -[Tansu][github-com-tansu-io] is an Apache Kafka API compatible broker -with PostgreSQL and S3 storage engines. Acting as a drop in replacement, -existing clients connect to Tansu, producing and fetching messages. -Written in 100% safe async 🦀 [Rust][rust-lang-org] 🚀. +[Tansu][github-com-tansu-io] is a modern drop-in replacement for +Apache Kafka. Without the cost of broker replicated storage for +durability. Tansu is in **early** development, licensed under +the [GNU AGPL][agpl-license]. Written in async 🚀 +[Rust][rust-lang-org] 🦀 + +Tansu brokers are: + +- Kafka API compatible (exceptions: transactions and idempotent + producer) +- Stateless with instant scaling up or down. No more planning and + reassigning partitions to a broker +- Available with PostgreSQL or S3 storage engines + +For data durability: + +- S3 is designed to exceed [99.999999999% (11 + nines)][aws-s3-storage-classes] +- PostgreSQL with [continuous archiving][continuous-archiving] + streaming transaction logs files to an archive. + +Stateless brokers are cost effective, with no network replication and +duplicate data storage charges. + +Stateless brokers do not have the ceremony of [Raft][raft-consensus] +or [ZooKeeper][apache-zookeeper]. ## S3 -Tansu with the S3 storage engine: +You can have 3 brokers running in separate [Availability +Zones][aws-regions-zones] for resilience: + +![Tansu S3](https://shortishly.com/assets/images/tansu-s3-2024-10-09.svg) + +Each broker is stateless. Brokers can come and go. Without affecting +leadership of consumer groups. The leader and In-Sync-Replica is the +broker serving your request. No more client broker ping pong. + +With Tansu there is no replication between brokers. The data transfer +cost between the [Availability Zones][aws-regions-zones] is a $0 line +item. There are $0 in duplicated storage charges. -- Brokers do not replicate messages, instead relying on the underlying - storage for resilience. -- Brokers do not require a consensus protocol (e.g., Raft or - Zookeeper), instead conditional PUTs are used to coordinate state. -- Brokers are stateless. -- All brokers are leaders. +With stateless brokers, you can run Tansu in a [server-less +architecture][wikipedia-serverless]: -Note that, Tansu requires that the underlying S3 service support conditional +![Tansu Lambda S3](https://shortishly.com/assets/images/tansu-lambda-s3-2024-10-09.svg) + +Spin up a broker for the duration of a Kafka API request. Then spin +down. No more idle brokers. + +Tansu requires that the underlying S3 service support conditional PUT requests. While [AWS S3 does now support conditional writes][aws-s3-conditional-writes], the support is @@ -25,11 +59,16 @@ To have stateless brokers we need to [use a compare and set operation][tigris-conditional-writes], which is not currently available in AWS S3. -Much like the Kafka protocol, the open nature of the S3 protocol allows vendors -to differentiate with different levels of service while retaining compatibility -with the underlying API. We have tried [minio][min-io] -and [tigis][tigris-conditional-writes], among a number of other vendors now supporting -conditional put. +Much like the Kafka protocol, the S3 protocol allows vendors to +differentiate. Different levels of service while retaining +compatibility with the underlying API. You can use [minio][min-io] or +[tigis][tigris-conditional-writes], among a number of other vendors +supporting conditional put. + +Tansu uses [object store][crates-io-object-store], providing a +multi-cloud API for storage. There is an alternative option to use a +[DynamoDB-based commit protocol, to provide conditional write support +for AWS S3][object-store-dynamo-conditional-put] instead. ### configuration @@ -51,7 +90,8 @@ Just bring minio up, without tansu: docker compose up -d minio ``` -The minio console should now be running on http://localhost:9001, login using +The minio console should now be running on +[http://localhost:9001](http://localhost:9001), login using the default user credentials of "minioadmin", with password "minioadmin". Follow the [bucket creation instructions][minio-create-bucket] to create a bucket called "tansu", and then @@ -201,18 +241,30 @@ Consumer: -b localhost:9092 ``` +## Feedback + +Please [raise an issue][tansu-issues] if you encounter a problem. + ## License Tansu is licensed under the [GNU AGPL][agpl-license]. [agpl-license]: https://www.gnu.org/licenses/agpl-3.0.en.html +[apache-zookeeper]: https://en.wikipedia.org/wiki/Apache_ZooKeeper +[aws-regions-zones]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html [aws-s3-conditional-requests]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html [aws-s3-conditional-writes]: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/ +[aws-s3-storage-classes]: https://aws.amazon.com/s3/storage-classes/ [continuous-archiving]: https://www.postgresql.org/docs/current/continuous-archiving.html +[crates-io-object-store]: https://crates.io/crates/object_store [github-com-tansu-io]: https://github.com/tansu-io/tansu [librdkafka]: https://github.com/confluentinc/librdkafka [min-io]: https://min.io +[minio-create-access-key]: https://min.io/docs/minio/container/administration/console/security-and-access.html#id1 +[minio-create-bucket]: https://min.io/docs/minio/container/administration/console/managing-objects.html#creating-buckets +[object-store-dynamo-conditional-put]: https://docs.rs/object_store/0.11.0/object_store/aws/struct.DynamoCommit.html +[raft-consensus]: https://raft.github.io [rust-lang-org]: https://www.rust-lang.org +[tansu-issues]: https://github.com/tansu-io/tansu/issues [tigris-conditional-writes]: https://www.tigrisdata.com/blog/s3-conditional-writes/ -[minio-create-bucket]: https://min.io/docs/minio/container/administration/console/managing-objects.html#creating-buckets -[minio-create-access-key]: https://min.io/docs/minio/container/administration/console/security-and-access.html#id1 +[wikipedia-serverless]: https://en.wikipedia.org/wiki/Serverless_computing