Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decode msgpack and json consistently #566

Merged
merged 4 commits into from
Nov 18, 2019
Merged

Decode msgpack and json consistently #566

merged 4 commits into from
Nov 18, 2019

Conversation

eddyashton
Copy link
Member

Our partially recursive failed to decode strings in lists and key-values, causing inconsistent behaviour depending on whether the format chosen was "json" or "msgpack".

>>> o
{'abc': 'def', 'ghi': {'jkl': ['a', 'b', 'c'], 'mno': {}}}
>>> json.loads(json.dumps(o))
{'abc': 'def', 'ghi': {'jkl': ['a', 'b', 'c'], 'mno': {}}}
>>> msgpack.unpackb(msgpack.packb(o))
{b'abc': b'def', b'ghi': {b'jkl': [b'a', b'b', b'c'], b'mno': {}}}

Ideally they would be identical, and they are if we call msgpack.unpackb(..., raw=False).

As an associated change, the raw quote is passed as a byte-vector rather than a string. This does mean it appears as a long int list when looking at the raw JSON/Python, so its a prime candidate to be hex-encoded (#278).

@eddyashton eddyashton requested a review from a team as a code owner November 18, 2019 16:01
@cimetrics
Copy link

images

@codecov-io
Copy link

codecov-io commented Nov 18, 2019

Codecov Report

Merging #566 into master will increase coverage by 0.22%.
The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #566      +/-   ##
==========================================
+ Coverage   78.42%   78.64%   +0.22%     
==========================================
  Files         140      140              
  Lines       10508    10508              
==========================================
+ Hits         8240     8263      +23     
+ Misses       2268     2245      -23
Flag Coverage Δ
#e2e_BFT 50.28% <0%> (-0.45%) ⬇️
#e2e_CFT 73.43% <0%> (+0.11%) ⬆️
#unit_BFT 64.39% <ø> (-0.83%) ⬇️
#unit_CFT 72.74% <ø> (ø) ⬆️
Impacted Files Coverage Δ
src/node/rpc/nodecalltypes.h 100% <ø> (ø) ⬆️
src/node/nodestate.h 83.48% <0%> (ø) ⬆️
src/tls/intel_drng.h 72% <0%> (+0.8%) ⬆️
src/host/tcp.h 73.93% <0%> (+0.85%) ⬆️
src/consensus/raft/raft.h 83.86% <0%> (+1.27%) ⬆️
src/tls/entropy.h 94.12% <0%> (+70.59%) ⬆️

@cimetrics
Copy link

images

@eddyashton eddyashton merged commit 91c7295 into master Nov 18, 2019
@eddyashton eddyashton deleted the msgpack_decode branch November 18, 2019 17:04
eddyashton added a commit to eddyashton/CCF that referenced this pull request Mar 24, 2020
* Use msgpack's decoder

* raw quote is a byte-array, not string

* Schema changed

* Remove manual re-encode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants