Support 'blob_value' / 'blob_key_value' in datastore protobufs

[tseaver]

In addition to the scalar types we already support, there are the following value types we don't yet
support:

• 'entity_value' (PR Support entity protobufs #232)
• 'list_value' (issue Support list values in entities #223)
• 'blob_value' (see https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/6073e01540264db58ee50c30a1e9f006677eb794/proto/datastore_v1.proto#L115)
• 'blob_key_value' (see https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/6073e01540264db58ee50c30a1e9f006677eb794/proto/datastore_v1.proto#L111)

[dhermes]

This is something we could check in an automated fashion:

>>> from gcloud.datastore import datastore_v1_pb2 as pb2
>>> for attr in dir(pb2.Value):
...     if attr[0] == attr[0].lower() and attr[0] != '_':
...         print attr
... 
blob_key_value
blob_value
boolean_value
double_value
entity_value
indexed
integer_value
key_value
list_value
meaning
string_value
timestamp_microseconds_value

or

>>> for key in sorted(pb2._VALUE.fields_by_name.keys()):
...     print key
... 
blob_key_value
blob_value
boolean_value
double_value
entity_value
indexed
integer_value
key_value
list_value
meaning
string_value
timestamp_microseconds_value

[tseaver]

Note that handling 'blob_value' values will require us to think harder about handling "bytes" (marshalled via 'blob_value') vs. "text" (marshalled via 'string_value'). At the moment, we rather sloppily conflate handling both Python2 'str' and 'unicode' via 'string_value'.

[silvolu]

@tseaver as you mentioned in #157, we should raise exceptions after the value types coverage is complete.

[silvolu]

I think we're missing value types in _get_protobuf_attribute_and_value(https://github.com/GoogleCloudPlatform/gcloud-python/blob/master/gcloud/datastore/_helpers.py#L17) as well.

[tseaver]

That will get tricksy:

• 'blob_value' should map to 'str' / 'bytes' in Python2, and 'bytes' in Python3
• 'string_value' should map to 'unicode' in Python2, and 'str' in Python3

Right now:

>>> from gcloud.datastore._helpers import _get_protobuf_attribute_and_value
>>> _get_protobuf_attribute_and_value(b'foo')
('string_value', 'foo')
>>> _get_protobuf_attribute_and_value(u'foo')
('string_value', u'foo')
>>> _get_protobuf_attribute_and_value('foo')
('string_value', 'foo')

which can't be right, long-term. It should be (under Python2):

>>> _get_protobuf_attribute_and_value(b'foo')
('blob_value', 'foo')
>>> _get_protobuf_attribute_and_value(u'foo')
('string_value', u'foo')
>>> _get_protobuf_attribute_and_value('foo')
('blob_value', 'foo')

and under Py3k:

>>> _get_protobuf_attribute_and_value(b'foo')
('blob_value', 'foo')
>>> _get_protobuf_attribute_and_value('foo')
('string_value', u'foo')
>>> _get_protobuf_attribute_and_value('foo')
('string_value', 'foo')

OTOH, '_get_value_from_value_pb' looks right:

>>> from gcloud.datastore.datastore_v1_pb2 import Value
>>> v = Value()
>>> v.string_value = 'abc'
>>> from gcloud.datastore._helpers import _get_value_from_value_pb
>>> _get_value_from_value_pb(v)
u'abc'

We'd like for the following to work:

>>> vb = Value()
>>> vb.blob_value = b'abc'
>>> _get_value_from_value_pb(vb)
'abc'

but it currently returns None.

[tseaver]

Changing to use 'blob_value' for bytes / native str in Python2 will be the one which confounds users expectations.

[dhermes]

Indeed. I wonder if people have done things to deal with this in other libraries. Potentially encoding/decoding to ASCII or something similar?

[silvolu]

@tseaver Saw your fix for blob_value in #233.
If we add list_value to _set_protobuf_value, then we should support all value types when writing entities.

[tseaver]

@silvolu see #265.

[tseaver]

#263 adds 'blob_value' support. @silvolu says we can skip 'blob_key_value'.