Batch dataset.put and dataset.delete.

[silvolu]

Like dataset.get, there should be a dataset.delete and dataset.put that accepts a list of keys / entities.

[dhermes]

I need to get on #336 too and hopefully these can come together. We should actually hash out the discussion in #188 as well, since many of our core patterns are very JavaScript-like and may not be the right way forward.

[tseaver]

Given a list of entities in hand, what is the advantage of a "convenience" method on the dataset to save or delete them, against just iterating the list and calling save or delete on each one? The get_entity and
get_entities methods on a dataset are there for when you don't already have the entities.

I guess there might be some utility to a delete method taking only keys, so that the user doesn't need to either fetch them via get_entities or convert them to protobufs to hand to the connection's delete_entities method.

[dhermes]

I think dataset.delete and dataset.put are valid, somewhat like ndb.put_multi and ndb.delete_multi.

[tseaver]

The Zen of Python says:

There should be one– and preferably only one –obvious way to do it.

What use is Dataset.put()? To make it work you have to have the entity objects in hand already, so why not just:

for entity in entities_to_put:
    entity.save()

If Dataset.delete() takes entities, the same applies (entities already have a delete() method). If it takes keys (only), then it might be useful, if you can reasonably expect that there is a way to have a bunch of keys in hand without also having the corresponding entities.

[silvolu]

It's mostly for performances reason. Doing:

for entity in entities_to_put:
    entity.save()

Means one RPC for each save. Batching means one RPC for all saves. (and same for deletes).

[tseaver]

Isn't that what you get from using a transaction? All the adds / saves / deletes get sent as a single RPC.

[silvolu]

That's meant for a transactional context, where if one fails, they all fail.

[elibixby]

I think it makes sense to have this be a mode/flag/subclass of Dataset rather than extra methods. 1. it keeps the interface of the class cleaner, 2. it adds support for batch mutations which contain a mix of mutation types. Ideally syntax could be similar to transaction syntax.

with dataset.batch_mutation() as b:
    for entity in entities_to_put:
        b.put(entity)
    for key in keys_to_delete:
        b.delete(key)

[dhermes]

IMO we need to focus on feature parity with the API before we can add nice sugar like that.

As @silvolu says, we need methods for batch puts and deletes.

[tseaver]

What is the expectation for error propagation after a non-transactional batch update / delete, where some of the operations failed?

[elibixby]

@tseaver There is none. If guarantees need to be made about mutation success transactions must be used.

@dhermes It's not necessarily sugar. If instead of immediately calling connection.commit() dataset writes to a mutation_pb buffer and immediately flushes that buffer (by calling connection.commit()) then a batch_mutator class could override that behavior and only flush the buffer on context exit. I think there is a lot of user experience value in having the same method signatures in a different context for batch mutations.

This implementation also works quite nicely for transactions. I have a class hierarchy I made for the Entityless implementation of the library which could easily be adapted, if you would be interested.

[tseaver]

@elibixby are you saying there is no way to detect the failures in batch mode? Who would use an API like that, and why would we go out of our way to support them?

[elibixby]

@tseaver this is fundamentally a property of the datastore, not the API. You could for instance make a bunch of batch writes, and then do other stuff. Then when you need to use the data, in a transaction retrieve all of the writes you made, redo the writes that failed, and operate on your data as normal. This will be faster than simply making all the writes in the transaction, as fewer writes will have to propagate through the datastore, (since transactions demand snapshot isolation and strong consistency)

EDIT: to expand on this, transactions are for making mutations that are dependent upon a certain state in the datastore (e.g. updating data based on it's current value)

OTOH you may want to make a huge number of mutations that don't depend on the state of the datastore. If you want to add 10,000 entities to the datastore (during a short time, and not dependent on the current content of the datastore), it would be EXTREMELY bad to slam the datastore with 10,000 api calls. This is a frequent use case during datastore initialization or clean up (in testing for example)

[tseaver]

@dhermes we have examples of context-manager-based batching in Storage already:

@property
def batch(self):
    """Return a context manager which defers/batches updates.

    E.g., to batch multiple updates to a bucket::

        >>> with bucket.batch:
        ...     bucket.enable_versioning()
        ...     bucket.disable_website()

    or for a key::

        >>> with key.batch:
        ...     key.content_type = 'image/jpeg'
        ...     key.content_encoding = 'gzip'

    Updates will be aggregated and sent as a single call to
    :meth:`_patch_properties` IFF the ``with`` block exits without
    an exception.

    :rtype: :class:`_PropertyBatch`
    """
    return _PropertyBatch(self)

[silvolu]

@tseaver with the major refactorings, is this still under the radar?

[dhermes]

It's right on the horizon (for one of us at least).