🐳 kafka-docker-playground

Fully automated Kafka and Confluent Docker based examples // Easily build reproduction models

🔥 Check out kafka-docker-playground.io to learn how to use the playground, how it works and how to build your own reproduction models !

👾 Content

🔗 Connectors

• ActiveMQ Sink (also with 🔑 mTLS)  
• ActiveMQ Source (also with 🔑 mTLS)  
• Amazon CloudWatch Logs Source  
• Amazon CloudWatch Metrics Sink  
• Amazon DynamoDB Sink  
• Amazon Kinesis Source (also with 🌐 proxy)  
• Amazon Redshift Sink      
• Amazon Redshift Source (using JDBC)      
• Amazon S3 Sink (also with 🌐 proxy)  
• Amazon S3 Source  
• Amazon SQS Source  
• AMPS Source  
• Apache Kudu Source  
• Apache Kudu Sink  
• AWS Lambda Sink  
• Azure Blob Storage Sink  
• Azure Blob Storage Source  
• Azure Cosmos DB Sink  
• Azure Cosmos DB Source  
• Azure Data Lake Storage Gen1 Sink  
• Azure Data Lake Storage Gen2 Sink  
• Azure Event Hubs Source  
• Azure Functions Sink  
• Azure Search Sink  
• Azure Service Bus Source  
• Azure SQL Data Warehouse Sink  
• Cassandra Sink  
• Couchbase Sink  
• Couchbase Source  
• Debezium CDC Microsoft SQL Server Source  
• Debezium CDC MySQL Source  
• Debezium CDC PostgreSQL Source (also with 🔑 SSL and mTLS)  
• Debezium CDC MongoDB Source  
• Data Diode Sink  
• Data Diode Source  
• Datadog Metrics Sink  
• ElasticSearch Sink  
• FilePulse Source  
• FileStream Source
• FileStream Sink
• FTPS Sink  
• FTPS Source  
• Gemfire Sink  
• Github Source  
• Google BigQuery Sink  
• Google Cloud BigTable Sink  
• Google Cloud Functions Sink  
• Google Cloud Pub/Sub Source (also with 🌐 proxy)  
• Google Cloud Spanner Sink  
• Google Cloud Storage Sink  
• Google Cloud Storage Source  
• Google Firebase Realtime Database Sink  
• Google Firebase Realtime Database Source  
• HBase Sink  
• HDFS 2 Source  
• HDFS 3 Source  
• HDFS 2 Sink  
• HDFS 3 Sink  
• HTTP Sink  
• IBM MQ Sink (also with 🔑 SSL and mTLS)  
• IBM MQ Source (also with 🔑 SSL and mTLS)  
• InfluxDB Sink  
• InfluxDB Source  
• JDBC CockroachDB Source  
• JDBC IBM DB2 Sink  
• JDBC IBM DB2 Source  
• JDBC MySQL Sink (also with 🔑 SSL and mTLS)  
• JDBC Oracle 11 Sink  
• JDBC Oracle 12 Sink (also with 🔑 SSL and mTLS)  
• JDBC Oracle 19c Sink (also with 🔑 SSL and mTLS)  
• JDBC PostGreSQL Sink (also with 🔑 SSL and mTLS)  
• JDBC Microsoft SQL Server Sink (also with 🔑 SSL)  
• JDBC MySQL Source (also with 🔑 SSL and mTLS)  
• JDBC Oracle 11 Source  
• JDBC Oracle 12 Source (also with 🔑 SSL and mTLS)  
• JDBC Oracle 19c Source (also with 🔑 SSL and mTLS)  
• JDBC PostGreSQL Source (also with 🔑 SSL and mTLS)  
• JDBC Microsoft SQL Server Source (also with 🔑 SSL)  
• JDBC Snowflake Sink  
• JDBC Snowflake Source  
• JDBC Vertica Sink  
• JIRA Source  
• JMS ActiveMQ Sink  
• JMS Solace Sink  
• JMS TIBCO EMS Sink  
• JMS TIBCO EMS Source  
• JMS Oracle Weblogic Sink  
• JMS Oracle Weblogic Source  
• Mapr Sink  
• Marketo Source  
• Minio Sink  
• MongoDB Sink  
• MongoDB Source  
• MQTT Sink  
• MQTT Source  
• Neo4j Sink  
• OmniSci Sink  
• Oracle 11 CDC Source  
• Oracle 12 CDC Source (also with 🔑 SSL and mTLS)  
• Oracle 19c CDC Source (also with 🔑 SSL and mTLS)  
• Oracle Weblogic Source  
• PagerDuty Sink  
• Prometheus Sink  
• RabbitMQ Sink  
• RabbitMQ Source  
• Redis Sink  
• SalesForce Bulk API Sink  
• SalesForce Bulk API Source  
• SalesForce CDC Source  
• SalesForce Platform Events Sink  
• SalesForce Platform Events Source  
• SalesForce PushTopics Source  
• SalesForce SObject Sink  
• ServiceNow Sink  
• ServiceNow Source (also with 🌐 proxy)  
• SFTP Sink (also with 🔑 Kerberos)  
• SFTP Source (also with 🔑 Kerberos)  
• SNMP Trap Source  
• Snowflake Sink  
• Solace Sink  
• Solace Source  
• Splunk Sink  
• Splunk Source  
• Splunk S2S Source  
• Spool Dir Source  
• Syslog Source  
• TIBCO EMS Sink  
• TIBCO EMS Source  
• Vertica Sink  
• Zendesk Source  

* You can change default connector version by setting CONNECTOR_TAG environment variable before starting a test, get more details here

☁️ Confluent Cloud

💻️ Demo

• 🔌 How to connect your components to Confluent Cloud
• 📈 How to monitor your Confluent Cloud cluster using Metrics API and Grafana, Splunk, New Relic, etc...

🔗 Connectors connected to Confluent Cloud

• AWS Kinesis source
• ServiceNow source
• ServiceNow sink
• MongoDB source
• MQTT Source
• Firebase

🚀 And much more...

• ⛈️ Using cp-ansible with Confluent Cloud
• 🌤 Using cp-helm-charts with Confluent Cloud
• 🌦️ Using Confluent Operator with Confluent Cloud
• 🌨 Using Confluent for Kubenetes with Confluent Cloud
• 📈 Demo using dabz/ccloudexporter
• 💠 .NET client (producer/consumer)
• 🗯 Go client (producer/consumer)
• 🐚 KafkaJS client (producer/consumer)
• 🐍 Python client (producer/consumer)
• 🧑‍🏭 Managing topics and ACLs using matt-mangia/kafka-admin
• ♻️ Confluent Replicator OnPrem to cloud and Cloud to Cloud examples
• 🗺️ Multi-Cluster Schema Registry with hybrid configuration (onprem/confluent cloud)
• 🔑 Confluent REST Proxy Security Plugin with Principal Propagation
• 🗝️ Confluent Schema Registry Security Plugin
• 📦 Migrate Schemas to Confluent Cloud using Confluent Replicator
• 🔰 Confluent Cloud Networking using HAProxy
• 🌎️ Apache Mirror Maker 2

♻️ Confluent Replicator and Mirror Maker 2

Using Multi-Data-Center setup with 🇺🇸 and 🇪🇺 clusters

• 🔗 Confluent Replicator as connector
  • With PLAINTEXT
  • With SASL_PLAIN
  • With Kerberos
• 🖇️ Confluent Replicator as executable
  • With PLAINTEXT
  • With SASL_PLAIN
  • With Kerberos
• 🌎️ Apache Mirror Maker 2
  • With PLAINTEXT

🔐 Environments

Using single cluster:

• PLAINTEXT: no security
• SASL/PLAIN: no SSL encryption, SASL/PLAIN authentication
• SASL/SCRAM no SSL encryption, SASL/SCRAM-SHA-256 authentication
• SASL/SSL: SSL encryption, SASL/PLAIN authentication
• 2WAY/SSL: SSL encryption, SSL authentication
• KERBEROS: no SSL encryption, Kerberos GSSAPI authentication
• SSL/KERBEROS SSL encryption, Kerberos GSSAPI authentication
• LDAP Authentication with SASL/PLAIN no SSL encryption, SASL/PLAIN authentication using LDAP
• LDAP Authorizer with SASL/PLAIN no SSL encryption, SASL/PLAIN authentication, LDAP Authorizer for ACL authorization
• RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication

Using Multi-Data-Center setup with 🇺🇸 and 🇪🇺 clusters

• PLAINTEXT: no security
• SASL/PLAIN: no SSL encryption, SASL/PLAIN authentication
• KERBEROS: no SSL encryption, Kerberos GSSAPI authentication

💸 Confluent Commercial

• 💻 Control Center
  • Control Center in "Read-Only" mode
  • Configuring Control Center with LDAP authentication
• 💾 Tiered Storage
  • Tiered storage with AWS S3
  • Tiered storage with Minio (unsupported)
• ⚖ Confluent Rebalancer
• 👴 JMS Client
• 〽️ RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication
• 🗃️ Audit Logs
• 🗝️ Confluent Schema Registry Security Plugin
• 🔒️ Confluent REST Proxy Security Plugin with SASL/SSL and 2WAY/SSL Principal Propagation
• ♻️ Cluster Linking
• 📒 Testing RBAC with Azure AD
• 🛂 Schema Validation on Confluent Server Schema Validation on Confluent Server

🪄 CP-Ansible Playground

Easily play with Confluent Platform Ansible playbooks by using Ubuntu based Docker images generated daily from this cp-ansible-playground repository

There is also a Confluent Cloud version available here

👾 Other Playgrounds

• 👬 Separate principals when using connectors: example with SFTP source
• 📃 How to write logs to files when using docker-compose
• 📢 Publish logs to kafka with Elastic Filebeat
• 💠 .NET basic producer
• 🐚 KafkaJS client (producer/consumer)
• 🐶 Monitor Confluent Platform with Datadog
• 👻 Testing KIP-108 Create Topic Policy
• 📉 Monitoring Demo
• 🕵️‍♀️ Kafka Connect Sink Monitoring Demo Showcase different Kafka Connect Sink troubleshooting scenarios
• 🎫 Multiple Event Types in the Same Topic