Add Avro support

[dannyd4315]

This PR implements the ability to encode messages in the Avro binary format prior to publishing to Kafka, this gives us the benefit reducing the size of our messages, and being able to use schema's for the messages we are producing.

By using a schema registry such as Confluent's Schema Registry producers can post the versioned schema to said registry and consumers are able to retrieve the correct version of the schema for the message its consuming.

I have utilised the great open-source avro-turf gem and also included the avro_patches gem to allow for logical types within schemas.

[balvig]

Found it a little difficult to do a proper suggestions branch, as what I wanted to show relied on the refactoring in #63, hope it's ok that I pushed the commits directly here instead and that they make sense 🙇

Mainly I didn't feel like we needed to repeat all those "method not implemented" tests for both classes, but let me know if that seems wrong.

Otherwise looks 👍 to me, this avro thing looks really useful, nice!

[ChrisBr]

Nice one 👍

[dannyd4315]

♻️ Please - Updated rspec helper that is used by global-web

[balvig]

👌 but I think there is one remaining issue we need to sort out:

Basically we include two types of test helpers:

assert_published_event # for minitest, test-unit
expect_event # for RSpec

...and I guess it's sort of expected that they behave the same way 🤔

Currently, only the RSpec one "hashifys" the deliveries hash?

Indeed, although it is currently "dormant" 😴, global-reports makes use of assert_published_event:

    assert_published_event(
      topic: "pageviews",
      type: "counted_recipe_views",
      body: hash_including(
        from: Time.zone.parse("2017/01/01"),
        to: Time.zone.parse("2017/01/02"),
        counts: counts
      )
    )

Source

...and I suppose it would be subject to the same problem as global-web.

Meanwhile, within this lib, in the tests we've written I guess we're finding it useful to test the "raw" attributes that get sent in some situations, to check that things are actually working.

I think it would be fine to merge and move on to make progress, but moving forward maybe you could try to find a way to handle both of these use cases, and make sure the helpers stay consistent?

[dannyd4315]

I think it would be fine to merge and move on to make progress, but moving forward maybe you could try to find a way to handle both of these use cases, and make sure the helpers stay consistent?

👍 Thanks, will do. Will take a look at that next. Version bumping to 0.3.0 to incorporate these changes.