Handle raw readings published on device/sck/device_token:/readings/raw

[vicobarberan]

Add handling for raw readings published by the kit in device/sck/device_token:/readings/raw topic.
With this change the size of the payload is reduced to around 35% of original size.
This is the firmware commit that enables raw readings publishing on the kit side.

[viktorsmari]

Nice job @vicobarberan ! 👏

I am adding a test for the handle_raw function

Todo:

• Should recorded at be recorded_at (with underscore _) ?
• After parsing, we have an extra colon : sign in the date: :2017-03-24T13:35:14Z should it be there?
• Create a test with 'fake' raw data to test if everything is parsed and processed correctly.

Example output after parsing this test data I copied from the Firmware example:

smartcitizen-api/spec/lib/mqtt_messages_handler_spec.rb

Line 137 in 88c1b97

the_data = "{ t:2017-03-24T13:35:14Z, 29:48.45, 13:66, 12:28, 10:4.45 }"

JSON.parse(reading)
{"recorded at"=>":2017-03-24T13:35:14Z", "sensors"=>[{"id"=>29, "value"=>48.45}, {"id"=>13, "value"=>66}, {"id"=>12, "value"=>28}, {"id"=>10, "value"=>4.45}]}

[vicobarberan]

Thanks for your fast response!!
Fixed first two points.
About the test, I'm not sure how to proceed (this are my first steps on ruby). Can you point me in the right direction please?

[viktorsmari]

@vicobarberan I will add the tests.

But do you know why are we manually creating a JSON object here:

smartcitizen-api/app/lib/mqtt_messages_handler.rb

Lines 47 to 53 in 88c1b97

	reading = "{\"recorded at\": \"#{clean_tm}\", \"sensors\": ["
	raw_readings.each do |raw_read|
	raw_id = raw_read.split(":")[0]
	raw_value = raw_read.split(":")[1]
	reading = "#{reading} { \"id\": #{raw_id}, \"value\": #{raw_value} },"
	end

I think if we already have the data on the server, we just need to send it to the Storer.new (no need to create a JSON object)

This is a normal readings received to the Storer.new:
{"recorded_at"=>"2016-06-08 10:30:00", "sensors"=>[{"id"=>1, "value"=>21}]}

This is the new raw received by the Storer.new:
"{\"recorded_at\": \"2017-03-24T13:35:14Z\", \"sensors\": [{ \"id\": 29, \"value\": 48.45 }, { \"id\": 13, \"value\": 66 }, { \"id\": 12, \"value\": 28 }, { \"id\": 10, \"value\": 4.45 }]}"

I think we could simply add the values as a Ruby Hash
https://ruby-doc.org/core-2.7.2/Hash.html

Then Line 47 could start like this:
reading = {"recorded_at" => clean_tm }
But then we need to add the rest to the sensors key in the hash

[viktorsmari]

I updated the code to use the Hash and we are progressing.

Now when using a normal reading example:

reading
{"recorded_at"=>"2016-06-08 10:30:00", "sensors"=>[{"id"=>1, "value"=>21}]}

Storer.parse_reading(device, reading)

{
:_data=>[{ 
  :name=>nil, 
  :timestamp=>1465374600000,
  :value=>21.0,
  :tags=> {:device_id=>10095, :method=>"REST" }
 }],
 :sql_data=>{""=>2016-06-08 10:30:00 +0200, "1_raw"=>21.0, 1=>21.0}, 
 :readings=>  { nil=>[1, 21.0, 21.0] },
 :parsed_ts=>2016-06-08 10:30:00 +0200,
 :ts=>1465374600000
}

And then the raw example:

reading
{"recorded_at"=>"2017-03-24T13:35:14Z", "sensors"=>[{"id"=>"29", "value"=>"48.45"}, {"id"=>"13", "value"=>"66"}, {"id"=>"12", "value"=>"28"}, {"id"=>"10", "value"=>"4.45"}]}

Storer.parse_reading(device, reading)

{
  :_data=>[],
  :sql_data=>{""=>2017-03-24 13:35:14 UTC},
  :readings=>{},
  :parsed_ts=>2017-03-24 13:35:14 UTC,
  :ts=>1490362514000
}

UPDATE If we change the first test id 1:48.45, (from id 29 to 1) we are able to get results:

{
 :_data=>[{
  :name=>nil,
  :timestamp=>1490362514000,
  :value=>48.45,
  :tags=>{:device_id=>10229, :method=>"REST"}
}], 
 :sql_data=>{""=>2017-03-24 13:35:14 UTC, "1_raw"=>48.45, 1=>48.45},
 :readings=>{nil=>[1, 48.45, 48.45]},
 :parsed_ts=>2017-03-24 13:35:14 UTC,
 :ts=>1490362514000
}

But now I get the error:

*** RuntimeError Exception: No timestamp given

TODO:

• The parsed_ts does not look the same in the raw and readings example