Skip to content

Latest commit

 

History

History
168 lines (131 loc) · 4.09 KB

README.md

File metadata and controls

168 lines (131 loc) · 4.09 KB
Raw

Big Querier

Build Status

Big Query object-to-row mapper.

What it is

Big Querier helps to automatically map objects to big query insert rows and vice-versa. There are 3 parts of big querier, each of them can be used separately:

  • The Client: A Big Query wrapper that simplifies query creation and result retrieval.
  • The Contract: A Row-to-object mapper, can convert object to row for insertion or selected row back to object.
  • The Dispatcher Service: A backend logger helper that dispatches logs to BigQuery in background.

Client Usage

var client = new Trafi.BigQuerier.BigQueryClient(
    "trafi-app-dev",
    "../Much/Path/To/Cert.p12",
    "very-secret",
    "very-mail@developer.gserviceaccount.com"
);

Has method to get or create a table:

var table = await client
    .GetTableClient("dataset_id", "table_id", yourSchema);

And then insert a row:

await table.InsertRows(new[] {
    new BigQueryInsertRow { ... }
});

Or get the rows:

var results = await client
    .Query("SELECT * FROM dataset_id.table_id");

This library throws BigQuerierException exceptions.

Mapper

Describe your contract:

[QuerierContract]
class Item
{
    public string Name { get; set; }
    public long? Count { get; set; }
    public long[] Values { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime? ModifiedAt { get; set; }
}

[QuerierContract]
class MyItem
{
    public string MyKey { get; set; }
    public bool? MyValue { get; set; }
    public Item[] Items { get; set; }
    public string[] Strings { get; set; }
    public long[] Values { get; set; }
    public Item Subitem { get; set; }
}

Supported types are:

  • string, int, long, double, bool, DateTime;
  • arrays or optional values of all mentioned types;
  • arrays of other contracts;
  • other contracts as properties.

Start by creating a contract. This auto-generates a mapper that can be reused:

var contract = Contract<MyItem>.Create();

Contract builds schema for you, so it's easier to create tables:

var table = await client
    .GetTableClient("dataset_id", "table_id", contract.Schema);

Contract helps to convert your items to rows:

var row = contract.ToRow(
    new MyItem
    {
        MyKey = "0001",
        MyValue = false,
        Items = new Item[]
        {
            new Item
            {
                Name = "Nerijus",
                CreatedAt = DateTime.UtcNow,
                ModifiedAt = DateTime.UtcNow,
            }
        },
        Strings = new string[] {"A", "B", "C"},
        Values = new long[] {42, 11, 111},
        Subitem = new Item
        {
            Name = "Stuff",
            Count = 3,
            Values = new long[] {42, 11, 111},
            CreatedAt = DateTime.UtcNow,
        },
    }
);
await table.InsertRows(new[] {row});

Contract helps to convert rows back to items:

var rowsEnumerable = await client.Query("SELECT * FROM dataset_id.table_id");
var rows = await rowsEnumerable.Select(contract.FromRow).ToList();

It is possible to specify query options and cancellation token, both are optional. As an example, we can fall back to Legacy SQL dialect (Google's BigQuery client uses Standard SQL by default):

using Google.Cloud.BigQuery.V2;

var rowsEnumerable = await client
    .Query(
        "SELECT * FROM dataset_id.table_id",
        options: new QueryOptions { UseLegacySql = true },
        ct: ct
    );
var rows = await rowsEnumerable
    .Select(contract.FromRow)
    .ToList(ct);

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.