Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Spanner stale read sample. #475

Merged
merged 2 commits into from
Sep 14, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 7 additions & 30 deletions spanner/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<img src="https://avatars2.githubusercontent.com/u/2810941?v=3&s=96" alt="Google Cloud Platform logo" title="Google Cloud Platform" align="right" height="96" width="96"/>

# Google Cloud Spanner Node.js Samples
# Cloud Spanner: Node.js Samples

[![Build](https://storage.googleapis.com/cloud-docs-samples-badges/GoogleCloudPlatform/nodejs-docs-samples/nodejs-docs-samples-spanner.svg)]()
[![Build](https://storage.googleapis.com/.svg)]()

[Cloud Spanner](https://cloud.google.com/spanner/docs/) is a fully managed, mission-critical, relational database service that offers transactional consistency at global scale, schemas, SQL (ANSI 2011 with extensions), and automatic, synchronous replication for high availability.

Expand All @@ -18,19 +18,6 @@

## Setup

1. Read [Prerequisites][prereq] and [How to run a sample][run] first.
1. Install dependencies:

With **npm**:

npm install

With **yarn**:

yarn install

[prereq]: ../README.md#prerequisites
[run]: ../README.md#how-to-run-a-sample

## Samples

Expand Down Expand Up @@ -70,10 +57,11 @@ __Usage:__ `node crud.js --help`

```
Commands:
update <instanceName> <databaseName> Modifies existing rows of data in an example Cloud Spanner table.
query <instanceName> <databaseName> Executes a read-only SQL query against an example Cloud Spanner table.
insert <instanceName> <databaseName> Inserts new rows of data into an example Cloud Spanner table.
read <instanceName> <databaseName> Reads data in an example Cloud Spanner table.
update <instanceName> <databaseName> Modifies existing rows of data in an example Cloud Spanner table.
query <instanceName> <databaseName> Executes a read-only SQL query against an example Cloud Spanner table.
insert <instanceName> <databaseName> Inserts new rows of data into an example Cloud Spanner table.
read <instanceName> <databaseName> Reads data in an example Cloud Spanner table.
read-stale <instanceName> <databaseName> Reads data in an example Cloud Spanner table.

Options:
--help Show help [boolean]
Expand Down Expand Up @@ -151,14 +139,3 @@ For more information, see https://cloud.google.com/spanner/docs

## Running the tests

1. Set the **GCLOUD_PROJECT** and **GOOGLE_APPLICATION_CREDENTIALS** environment variables.

1. Run the tests:

With **npm**:

npm test

With **yarn**:

yarn test
51 changes: 51 additions & 0 deletions spanner/crud.js
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,50 @@ function readData (instanceId, databaseId) {
// [END read_data]
}

function readStaleData (instanceId, databaseId) {
// [START read_stale_data]
// Imports the Google Cloud client library
const Spanner = require('@google-cloud/spanner');

// Instantiates a client
const spanner = Spanner();

// Uncomment these lines to specify the instance and database to use
// const instanceId = 'my-instance';
// const databaseId = 'my-database';

// Gets a reference to a Cloud Spanner instance and database
const instance = spanner.instance(instanceId);
const database = instance.database(databaseId);

// Read rows from the Albums table
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: inconsistent comment tense.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

const albumsTable = database.table('Albums');

const query = {
columns: ['SingerId', 'AlbumId', 'AlbumTitle', 'MarketingBudget'],
keySet: {
all: true
}
};

const options = {
// Guarantees that all writes that have committed more than 10 seconds ago
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: remove that have and combine into one line

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

// are visible
exactStaleness: 10
};

albumsTable.read(query, options)
.then((results) => {
const rows = results[0];

rows.forEach((row) => {
const json = row.toJSON();
console.log(`SingerId: ${json.SingerId.value}, AlbumId: ${json.AlbumId.value}, AlbumTitle: ${json.AlbumTitle}, MarketingBudget: ${json.MarketingBudget ? json.MarketingBudget.value : ''}`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we split these into separate lines? This is a lot to read at once...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

});
});
// [END read_stale_data]
}

const cli = require(`yargs`)
.demand(1)
.command(
Expand All @@ -189,10 +233,17 @@ const cli = require(`yargs`)
{},
(opts) => readData(opts.instanceName, opts.databaseName)
)
.command(
`read-stale <instanceName> <databaseName>`,
`Reads data in an example Cloud Spanner table.`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: mention/explain stale reads in the command description?

e.g. Reads _stale_ data in an example Cloud Spanner table.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

{},
(opts) => readStaleData(opts.instanceName, opts.databaseName)
)
.example(`node $0 update "my-instance" "my-database"`)
.example(`node $0 query "my-instance" "my-database"`)
.example(`node $0 insert "my-instance" "my-database"`)
.example(`node $0 read "my-instance" "my-database"`)
.example(`node $0 read-stale "my-instance" "my-database"`)
.wrap(120)
.recommendCommands()
.epilogue(`For more information, see https://cloud.google.com/spanner/docs`);
Expand Down
4 changes: 2 additions & 2 deletions spanner/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@
},
"devDependencies": {
"@google-cloud/nodejs-repo-tools": "1.4.17",
"ava": "0.21.0",
"ava": "0.22.0",
"proxyquire": "1.8.0",
"sinon": "3.2.0"
"sinon": "3.2.1"
},
"cloud-repo-tools": {
"requiresKeyFile": true,
Expand Down
122 changes: 79 additions & 43 deletions spanner/system-test/spanner.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -53,103 +53,139 @@ test.after.always(async (t) => {

// create_database
test.serial(`should create an example database`, async (t) => {
const output = await tools.runAsync(`${schemaCmd} createDatabase "${INSTANCE_ID}" "${DATABASE_ID}"`, cwd);
t.true(output.includes(`Waiting for operation on ${DATABASE_ID} to complete...`));
t.true(output.includes(`Created database ${DATABASE_ID} on instance ${INSTANCE_ID}.`));
const results = await tools.runAsyncWithIO(`${schemaCmd} createDatabase "${INSTANCE_ID}" "${DATABASE_ID}"`, cwd);
Copy link
Contributor

@ace-n ace-n Sep 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the point of this? (Having stderr in the test printout?)

stderr should be ignored unless we expect an error to occur. (If an error occurs when it shouldn't, our test should fail - though perhaps not silently.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've discover lately that sometimes tests/commands fail silently (return a 0 exit code), but do print some useful info to stderr. This improves test debuggability.

const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`Waiting for operation on ${DATABASE_ID} to complete...`));
t.regex(output, new RegExp(`Created database ${DATABASE_ID} on instance ${INSTANCE_ID}.`));
});

// insert_data
test.serial(`should insert rows into an example table`, async (t) => {
let output = await tools.runAsync(`${crudCmd} insert ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Inserted data.`));
const results = await tools.runAsyncWithIO(`${crudCmd} insert ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`Inserted data.`));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace t.regex(output, new RegExp($STRING)) with t.regex(output, /$STRING/). Apply this comment throughout your PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

});

// query_data
test.serial(`should query an example table and return matching rows`, async (t) => {
const output = await tools.runAsync(`${crudCmd} query ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go`));
const results = await tools.runAsyncWithIO(`${crudCmd} query ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go`));
});

// read_data
test.serial(`should read an example table`, async (t) => {
const output = await tools.runAsync(`${crudCmd} read ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go`));
const results = await tools.runAsyncWithIO(`${crudCmd} read ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go`));
});

// add_column
test.serial(`should add a column to a table`, async (t) => {
const output = await tools.runAsync(`${schemaCmd} addColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Waiting for operation to complete...`));
t.true(output.includes(`Added the MarketingBudget column.`));
const results = await tools.runAsyncWithIO(`${schemaCmd} addColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`Waiting for operation to complete...`));
t.regex(output, new RegExp(`Added the MarketingBudget column.`));
});

// update_data
test.serial(`should update existing rows in an example table`, async (t) => {
let output = await tools.runAsync(`${crudCmd} update ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Updated data.`));
const results = await tools.runAsyncWithIO(`${crudCmd} update ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`Updated data.`));
});

// read_stale_data
test.serial(`should read stale data from an example table`, (t) => {
t.plan(2);
// read-stale-data reads data that is exactly 10 seconds old. So, make sure
// 10 seconds have elapsed since the update_data test.
return new Promise((resolve, reject) => {
setTimeout(async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: can you promisify the setTimeout() call, and wrap the tests in a then as was done here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

const results = await tools.runAsyncWithIO(`${crudCmd} read-stale ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
try {
t.regex(output, new RegExp(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget: 100000`));
t.regex(output, new RegExp(`SingerId: 2, AlbumId: 2, AlbumTitle: Forever Hold your Peace, MarketingBudget: 500000`));
resolve();
} catch (err) {
reject(err);
}
}, 11000);
});
});

// query_data_with_new_column
test.serial(`should query an example table with an additional column and return matching rows`, async (t) => {
const output = await tools.runAsync(`${schemaCmd} queryNewColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, MarketingBudget: 100000`));
t.true(output.includes(`SingerId: 2, AlbumId: 2, MarketingBudget: 500000`));
const results = await tools.runAsyncWithIO(`${schemaCmd} queryNewColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`SingerId: 1, AlbumId: 1, MarketingBudget: 100000`));
t.regex(output, new RegExp(`SingerId: 2, AlbumId: 2, MarketingBudget: 500000`));
});

// create_index
test.serial(`should create an index in an example table`, async (t) => {
let output = await tools.runAsync(`${indexingCmd} createIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Waiting for operation to complete...`));
t.true(output.includes(`Added the AlbumsByAlbumTitle index.`));
const results = await tools.runAsyncWithIO(`${indexingCmd} createIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`Waiting for operation to complete...`));
t.regex(output, new RegExp(`Added the AlbumsByAlbumTitle index.`));
});

// create_storing_index
test.serial(`should create a storing index in an example table`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} createStoringIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`Waiting for operation to complete...`));
t.true(output.includes(`Added the AlbumsByAlbumTitle2 index.`));
const results = await tools.runAsyncWithIO(`${indexingCmd} createStoringIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`Waiting for operation to complete...`));
t.regex(output, new RegExp(`Added the AlbumsByAlbumTitle2 index.`));
});

// query_data_with_index
test.serial(`should query an example table with an index and return matching rows`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} queryIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget:`));
const results = await tools.runAsyncWithIO(`${indexingCmd} queryIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget:`));
t.false(output.includes(`AlbumId: 2, AlbumTitle: Total Junk, MarketingBudget:`));
});

test.serial(`should respect query boundaries when querying an example table with an index`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} queryIndex ${INSTANCE_ID} ${DATABASE_ID} -s Ardvark -e Zoo`, cwd);
t.true(output.includes(`AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget:`));
t.true(output.includes(`AlbumId: 2, AlbumTitle: Total Junk, MarketingBudget:`));
const results = await tools.runAsyncWithIO(`${indexingCmd} queryIndex ${INSTANCE_ID} ${DATABASE_ID} -s Ardvark -e Zoo`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`AlbumId: 1, AlbumTitle: Go, Go, Go, MarketingBudget:`));
t.regex(output, new RegExp(`AlbumId: 2, AlbumTitle: Total Junk, MarketingBudget:`));
});

// read_data_with_index
test.serial(`should read an example table with an index`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} readIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`AlbumId: 1, AlbumTitle: Go, Go, Go`));
const results = await tools.runAsyncWithIO(`${indexingCmd} readIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`AlbumId: 1, AlbumTitle: Go, Go, Go`));
});

// read_data_with_storing_index
test.serial(`should read an example table with a storing index`, async (t) => {
const output = await tools.runAsync(`${indexingCmd} readStoringIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`AlbumId: 1, AlbumTitle: Go, Go, Go`));
const results = await tools.runAsyncWithIO(`${indexingCmd} readStoringIndex ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`AlbumId: 1, AlbumTitle: Go, Go, Go`));
});

// read_only_transaction
test.serial(`should read an example table using transactions`, async (t) => {
const output = await tools.runAsync(`${transactionCmd} readOnly ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go`));
t.true(output.includes(`Successfully executed read-only transaction.`));
const results = await tools.runAsyncWithIO(`${transactionCmd} readOnly ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
const output = results.stdout + results.stderr;
t.regex(output, new RegExp(`SingerId: 1, AlbumId: 1, AlbumTitle: Go, Go, Go`));
t.regex(output, new RegExp(`Successfully executed read-only transaction.`));
});

// read_write_transaction
test.serial(`should read from and write to an example table using transactions`, async (t) => {
let output = await tools.runAsync(`${transactionCmd} readWrite ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`The first album's marketing budget: 100000`));
t.true(output.includes(`The second album's marketing budget: 500000`));
t.true(output.includes(`Successfully executed read-write transaction to transfer 200000 from Album 2 to Album 1.`));

output = await tools.runAsync(`${schemaCmd} queryNewColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
t.true(output.includes(`SingerId: 1, AlbumId: 1, MarketingBudget: 300000`));
t.true(output.includes(`SingerId: 2, AlbumId: 2, MarketingBudget: 300000`));
let results = await tools.runAsyncWithIO(`${transactionCmd} readWrite ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
let output = results.stdout + results.stderr;
t.regex(output, new RegExp(`The first album's marketing budget: 100000`));
t.regex(output, new RegExp(`The second album's marketing budget: 500000`));
t.regex(output, new RegExp(`Successfully executed read-write transaction to transfer 200000 from Album 2 to Album 1.`));

results = await tools.runAsyncWithIO(`${schemaCmd} queryNewColumn ${INSTANCE_ID} ${DATABASE_ID}`, cwd);
output = results.stdout + results.stderr;
t.regex(output, new RegExp(`SingerId: 1, AlbumId: 1, MarketingBudget: 300000`));
t.regex(output, new RegExp(`SingerId: 2, AlbumId: 2, MarketingBudget: 300000`));
});