Skip to content

Utility project for converting JSON to exploded union Avro JSON

License

Notifications You must be signed in to change notification settings

carlislefox/json-to-avro-json

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

JSON to Avro JSON

Utility project for converting JSON to... for lack of a better way of describing it... exploded union Avro JSON. This project is a mutation of https://github.com/allegro/json-avro-converter, specifically with the ability to take raw JSON and convert it to the Union JSON expected by Avro.

Optional fields in Avro are by definition Union types, as they can be either null or whatever their primitive type is, say a long or a string or an int or whatever. In order for the Avro JSON encoder to work it needs to be told explicitly what type these values are, for example your JSON might look like this:

{
   "name": "Liam"
}

However attempting to publish this to Kafka using the Avro encoder with say... Confluent's kafka-avro-console-producer... will throw an error like this:

org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING

What you actually need to publish looks more like this:

{
   "name": {
      "string": "Liam"
   }
}

This presents a problem when you are just trying to stuff some JSON through Avro for testing or whatever. Anyway, this project has everything you need to stuff plain JSON in one end and get the magic Avro JSON out of the other - I could find a lot of people with this issue searching online but I couldn't find any actual solutions... so when i got something working it seemed the right thing to do to share it back.

Usage

Usage is simple, you need an instance of the converter, your avro schema, and the json you are trying to convert.

final String json = "Your JSON file";
final String schema = "Your avro schema"

final JsonToAvroJson jsonToAvroJson = new JsonToAvroJson();
final String convertedJson = jsonToAvroJson.convert(json, schema);

About

Utility project for converting JSON to exploded union Avro JSON

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Languages