How should we resolve the fact that we can't make HappyBase create_table() atomic

[dhermes]

See comment. The core issue is that in HBase/Thrift, there is a way to create the table along with the column families all at once while in Cloud Bigtable we have to make 1 create table request and then n create column family requests. If any of those fail then create_table() will exit in an "undefined" state.

[jgeewax]

And there's no way to wrap this in a transaction?

What about if the CFs call fails, and they retry -- what happens then ?

/cc @maxluebbe: Creating a table + CFs can't happen atomically in Cloud Bigtable?

[tseaver]

Hmm, looks like Bigtable V2 may have removed the restriction. The V1 proto for CreateTableRequest has:

message CreateTableRequest {
  // The unique name of the cluster in which to create the new table.
  string name = 1;

  // The name by which the new table should be referred to within the cluster,
  // e.g. "foobar" rather than "<cluster_name>/tables/foobar".
  string table_id = 2;

  // The Table to create. The `name` field of the Table and all of its
  // ColumnFamilies must be left blank, and will be populated in the response.
  Table table = 3;

but in V2:

message CreateTableRequest {
  // An initial split point for a newly created table.
  message Split {
    // Row key to use as an initial tablet boundary.
    bytes key = 1;
  }

  // The unique name of the instance in which to create the table.
  // Values are of the form projects/<project>/instances/<instance>
  string parent = 1;

  // The name by which the new table should be referred to within the parent
  // instance, e.g. "foobar" rather than "<parent>/tables/foobar".
  string table_id = 2;

  // The Table to create.
  Table table = 3;

@garye do I jump to conclusions, or can we populate column families into the table entry in V2?

[dhermes]

We could always check by making live requests against the API

[garye]

I haven't tested this but I'm reason sure that you can add column families
at creation time.

On Fri, Jul 1, 2016, 8:42 PM Danny Hermes notifications@github.com wrote:

We could always check by making live requests against the API

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1524 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AACTr5Ko0g8rhJ1MCeVGHyEnrNKH2YNTks5qRbPvgaJpZM4Hh7KQ
.

[tseaver]

@dhermes would you like me to tackle that?

[dhermes]

I'll handle it.

[dhermes]

@garye I confirmed this works and also confirmed that it results in an INVALID_ARGUMENT with

Table's 'name' field must be empty during creation

if the CreateTableRequest.Table.name is set, the backend probably does this to avoid having to check a conflict with CreateTableRequest.table_id.

---

Implementing it now, here is the snippet I used to confirm it works:

from gcloud import _helpers
from gcloud.bigtable._generated_v2 import bigtable_table_admin_pb2
from gcloud.bigtable._generated_v2 import table_pb2
from gcloud.bigtable.client import Client
from gcloud.environment_vars import TESTS_PROJECT


def main():
    # Set-up test env.
    _helpers.PROJECT = TESTS_PROJECT

    client = Client(admin=True)
    client.start()

    # New instance.
    instance_id = 'gcloud-test-issue-1524'
    instance = client.instance(instance_id, 'us-central1-c')
    instance.create()

    # New table object, but manually construct PB to create.
    table = instance.table('gcloud-python-test-table')

    table_pb = table_pb2.Table()
    col_fam_pb = table_pb.column_families['col-fam-id1']
    col_fam_pb.gc_rule.max_num_versions = 1
    request_pb = bigtable_table_admin_pb2.CreateTableRequest(
        parent=instance.name,
        table_id=table.table_id,
        table=table_pb,
    )
    try:
        client._table_stub.CreateTable(request_pb, client.timeout_seconds)
    finally:
        instance.delete()
        client.stop()


if __name__ == '__main__':
    main()

[garye]

Awesome! Good to hear.

On Tue, Jul 19, 2016, 6:13 PM Danny Hermes notifications@github.com wrote:

@garye https://github.com/garye I confirmed this works and also
confirmed that it results in an INVALID_ARGUMENT with

Table's 'name' field must be empty during creation

if the CreateTableRequest.Table.name is set, the backend probably does

this to avoid having to check a conflict with CreateTableRequest.table_id.

Implementing it now, here is the snippet I used to confirm it works:

from gcloud import _helpersfrom gcloud.bigtable._generated_v2 import bigtable_table_admin_pb2from gcloud.bigtable._generated_v2 import table_pb2from gcloud.bigtable.client import Clientfrom gcloud.environment_vars import TESTS_PROJECT

def main():
# Set-up test env.
_helpers.PROJECT = TESTS_PROJECT

client = Client(admin=True)
client.start()

# New instance.
instance_id = 'gcloud-test-issue-1524'
instance = client.instance(instance_id, 'us-central1-c')
instance.create()

# New table object, but manually construct PB to create.
table = instance.table('gcloud-python-test-table')

# table_pb = table_pb2.Table(name=table.name)
table_pb = table_pb2.Table()
col_fam_pb = table_pb.column_families['col-fam-id1']
col_fam_pb.gc_rule.max_num_versions = 1
request_pb = bigtable_table_admin_pb2.CreateTableRequest(
    parent=instance.name,
    table_id=table.table_id,
    table=table_pb,
)
try:
    client._table_stub.CreateTable(request_pb, client.timeout_seconds)
finally:
    instance.delete()
    client.stop()

if name == 'main':
main()

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1524 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACTr83kVAFPFnkfYxu2RYRC2S5G3zt8ks5qXUv_gaJpZM4Hh7KQ
.