Get Unique Entity Key String

[shaunc869]

When I view an entity in the web based datastore viewer I see a big long guid-esq string that I can normally use in Python to walk back to the entity without knowing its type, for example:

ahNzfnByb21wdGFwcG9pbnRtZW50chULEghTY2hlZHVsZRiAgICD2e7xxxxx

get me back a specific entity and with Python I can get to this by saying:

my_entity = ndb.Key(urlsafe="ahNzfnByb21wdGFwcG9pbnRtZW50chULEghTY2hlZHVsZRiAgICD2xxxx")

Can this be done with the gcloud nodejs library and/or is this entirely a bad method for some reason to begin with? Thanks!

[jgeewax]

That's a good question. What you're looking at is really just a (mostly) base64 encoded version of the entity_pb.Reference protobuf. Since a key can be encoded and decoded anyway you want, and the path (ie, ['Person', 1, 'Playlist', 1234]) is unique to the entity, you might want to consider just serializing the path and using that as a unique identifier.

So for example:

var path = ['Person', 1, 'Playlist', 1234];

var encodedPath = btoa(JSON.stringify(path)); // "WyJQZXJzb24iLDEsIlBsYXlsaXN0IiwxMjM0XQ=="
var decodedPath = JSON.parse(atob(encodedPath)); // ["Person",1,"Playlist",1234]

@ryanseys @stephenplusplus : Do you think it's worthwhile to add this method to the Key class so we don't make people implement it themselves?

[jgeewax]

(For reference, here is the code that deals with the urlsafe stuff in NDB: https://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/key.py#803)

[ryanseys]

Hmm, trying to see the use case for this. I think we'd suggest that gcloud-node be used primarily driven by API calls, and I don't see that this information (base64'd reference protobuf stuff) is directly accessible through the API, rather you'd have to do extraneous work to put it in that format or manually extract it from the web view, so it's likely an edge case we won't directly support. @jgeewax does it sound like I am understanding you correctly given what you've said above?

[shaunc869]

I can understand if this is an edge case, but in general I have worked with the datastore in both Java and Python, but it looks like the return values in both of these languages when trying to convert them to json come out a lot different than gcloud is there a reason why in Java and Python the format is a bit flatter and the keys don't include paths (see to_dict() method for example in Python)? I was thinking this might be why the guid keys were being used to keep the json a bit flatter, maybe?

Also I tried running the code above on a guid key and it doesn't seem to come out right, is that more pseudo code or should it work on a full key guid string?

Thanks for all the help!

[jgeewax]

@ryanseys If we think of this use case as ... "Make it easy for people to pass around key=" and retrieve it on a request, I think it'd be useful to have a method that serializes a string in something that is URL safe... That is, in express...

var express = require('express');
var gcloud = require('gcloud')({ projectId: 'my-project' });

var app = express();
var dataset = gcloud.datastore.dataset();

// GET /?key=WyJQZXJzb24iLDEsIlBsYXlsaXN0IiwxMjM0XQ==
app.get('/', function(req, res){
  dataset.get(dataset.key({encodedValue: req.query.key}, function(err, entity) {
    res.redirect('/otherPath?key=' + entity.key.encodedValue());
  });
});

app.listen(3000);

@shaunc869 What problems are you seeing with that same code...? Can you be more specific? I'm just using Javascript to JSON stringify a list and base-64 encode it... nothing gcloud-specific here.

[shaunc869]

@jgeewax I agree, that helper method would be awesome. I am trying to copy and past the entity key guid string from the datastore entity editor on the developers console and then base64 decode it and I am not getting the full path for some reason.

[jgeewax]

@shaunc869 That code wasn't a way of decoding from the exact same format as the entity editor -- it was just an example demonstrating how we might go about this. You could certainly figure it out assuming it's using the same encoding style that NDB does (NDB's decoding logic is here: https://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/key.py#803).

[shaunc869]

Ah that makes sense, I will try my best with that, but I generally wait for much smarter developers like you to figure this stuff out. I will take a crack at the ndb stuff and see what I can come up. If this was to be implemented here would you use the same algorithm as ndb, I was assuming this was some kind of standard?

[jgeewax]

I believe (I could be wrong) but all that's happening is base64 encoding with a couple find/replace characters (-'s and /'s?) and then chopping off trailing ='s.

[shaunc869]

I copied the exact code from that ndb code and ran it against the app engine console string and it does not come as a readable path. What am I doing wrong?

[jgeewax]

To help you, I'd need to see the exact code you're talking about... @shaunc869

[shaunc869]

I grabbed the ndb code:

def _DecodeUrlSafe(urlsafe):
  """Decode a url-safe base64-encoded string.

  This returns the decoded string.
  """
  if not isinstance(urlsafe, basestring):
    raise TypeError('urlsafe must be a string; received %r' % urlsafe)
  if isinstance(urlsafe, unicode):
    urlsafe = urlsafe.encode('utf8')
  mod = len(urlsafe) % 4
  if mod:
    urlsafe += '=' * (4 - mod)
  # This is 3-4x faster than urlsafe_b64decode()
  return base64.b64decode(urlsafe.replace('-', '+').replace('_', '/'))

And if you try to run this you don't get a path like I would expect, I must be missing something?

[jgeewax]

It's not returning the path -- it's returning the binary of a Reference protobuf. What I wrote above:

What you're looking at is really just a (mostly) base64 encoded version of the entity_pb.Reference protobuf.

So you'd need to then create the protobuf object ... and then read the path property from that... :(

It doesn't look like it'd be fun code to write or use (and seems like it's totally overkill when a path suffices) so it's really really unlikely that gcloud-node would go that route when "coming up with a serializable version" of a key. (We'd likely go the route of just serializing the path, since that's unique...)

Is there some specific reason that you need to emulate the exact same encoding of the key from the Datastore UI?

[shaunc869]

My current situation is I have an app engine Python app that I want to move to node so I am trying to emulate the same ajax methods I have on my python webapp2 app, which has a getter for all the entity types and returns a nice flat JSON model that I can manipulate on the client side and then save back.

This is probably related, but just now when I went to save back the data I sent the following:

{
  key: {
    path: [ 
      "PhoneNumber",
      5313141421375488
    ]
  },
  data: {
    company_id: {
      path: [
        "Company",
        4791049928048640
      ]
    },
    sid: "PN07d490d21f4f803c82bcadad342348",
    phone_number: "+13601234333",
    call_recording_enabled: false,
    date_added: "2015-04-26T22:10:48.000Z",
    date_updated: "2015-04-26T22:19:52.818Z",
    name: "Main Phone Number",
    form_id: {
      path: [
        "Form",
        473767298046333336
      ]
    },
    twiml: [ ]
  }
}

Normally I would expect this to update the datastore and save a reference to both the Form object and Company object in the datastore like it does in Python, but instead I see:

51 bytes, SHA-1 = c25e70c16756daa25d5149437471b06647b957cd for the company_id

and

48 bytes, SHA-1 = 47b70d6b2be9960fd0ad6f164b56ae756e1a503c for the form,

the datastore viewer isn't resolving this to the appropriate objects in the datastore like it would had I converted the JSON to a Python object via:

phone_number = PhoneNumber(**number_json)
phone_number.put()

Thanks for all the help!

[jgeewax]

Hm.. I think this kind of stuff might be better answered over on Stack Overflow as it's getting pretty specific about your app and less so about gcloud-node.

I'm going to leave this issue open, with the goal that we expose some way of nicely serializing a key's path so that users don't have to come up with their own ways of doing it, but I don't think I'm going to be as much help on your specific case (but SO is probably going to be much more useful)!

[shaunc869]

Sounds good I will move it over there, thanks!

On Mon, May 25, 2015 at 11:04 AM, JJ Geewax notifications@github.com
wrote:

Hm.. I think this kind of stuff might be better answered over on Stack
Overflow as it's getting pretty specific about your app and less so about
gcloud-node.

I'm going to leave this issue open, with the goal that we expose some way
of nicely serializing a key's path so that users don't have to come up with
their own ways of doing it, but I don't think I'm going to be as much help
on your specific case (but SO is probably going to be much more useful)!

—
Reply to this email directly or view it on GitHub
#625 (comment)
.

Shaun
shaun@invoicesherpa.com
http://www.invoicesherpa.com
888-898-8302

[jgeewax]

Just a ping on the main issue here (which is "let's provide some URL-encodable option for the Key"):

Couldn't someone wanting to pass a key around simply do:

key = dataset.key({
  namespace: 'my-namespace',
  path: ['Kind', 'asdf', 'ChildKind', 1234]
});
var keyForURI = encodeURIComponent(JSON.stringify(key));

? Or does that not work...?

[stephenplusplus]

Yeah, that would produce:

encoded:

%7B%22namespace%22%3A%22my-namespace%22%2C%22path%22%3A%5B%22Kind%22%2C%22asdf%22%2C%22ChildKind%22%2C1234%5D%7D

decoded:

{"namespace":"my-namespace","path":["Kind","asdf","ChildKind",1234]}

That's what I would do!

[jgeewax]

Is it worth providing a .to<something>() method on the Key class? Or should we close this out as ... "ya'll figure it out"?

[stephenplusplus]

I think it should be figured out by the user, as we're not doing anything unique. Just stringifying and uri encoding. The encoding can be skipped in most libraries:

var postKey = dataset.key(['Post', 'post1']);

require('request').get({
  uri: 'http://localhost/',
  qs: {
    key: JSON.stringify(postKey) // sending a key object as a string. JSON.stringify is perfect
  }
});

// GET /saveKey?key=%7B%22path%22%3A%5B%22Post%22%2C%22post1%22%5D%7D

[arliber]

I might have missed something but the referenced articles do not provide any progress on the matter and the suggested solution from above - using atob doesn't work for me.

Are there any news regarding the ability to query datastore using the URL-safe key in Node?

[BransonGitomeh]

hey, for me on datastore, i found out that the results from an insertion matches something like

const key = {
    kind: 'projects',
    path: ['projects', id],
    id,
  }

so the id is a number matching something like 5664248772427776

meaning that you can pass this around in your app and then when you need to fetch it you construct the object in the way datastore.get expects it to be

ie to fetch a project

const project = async ({ id }, { datastore }) => {
  const entity = await datastore.get({
    kind: 'projects',
    path: ['projects', id],
    id,
  });

  return Object.assign(entity.pop(), {
    id,
  });
};

will return the data as it in on the data store row.

when creating the entity on data store, you can get the id like

const create = async ({ project }, { datastore }) => {
  const key = datastore.key('projects');
  await datastore.save({
    key,
    data: project,
  });
  const { id } = key;
  return Object.assign(project, {
    id,
  });
};

returns an object with an id you can use around in url's