Skip to content

it consumes message from multiple kafka topic, parses the required attribute and serialize it into AVRO format and stores it into local filesystem.

Notifications You must be signed in to change notification settings

amarkum/twitter-spark-processor

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 

Repository files navigation

twitter-spark-processor

Twitter Spark Processor Application cosnumes messages from multiple kafka topic and thier partition
It serailizes and stores only the specific attributes from tweet JSON, serialzes and writes it to AVRO.

When Shutdown Hook is called, It gracefully shuts downs the SparkSession and flushes the Avro to a file in local system.

About

it consumes message from multiple kafka topic, parses the required attribute and serialize it into AVRO format and stores it into local filesystem.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages