Add DML samples for BigQuery. #546

tswast · 2016-09-28T17:46:35Z

No description provided.

This generates a random MySQL database (tables defined in create_sample_db.sql) with random user actions. The reason for putting these into a SQL database instead of directly into BigQuery is that it will be used to show export form a SQL database into BigQuery. Hopefully the limits on query sizes for BigQuery are large enough that this works for larger databases. Change-Id: I446c3af72dab60d9ed79a2c814a68f05801ae17b

This tests the SQL code to create the tables, as well as the Python code that creates the rows. Uses SQLAlchemy to abstract away differences between database engines. Change-Id: Id9e70eef56f5e203921b6c1f21708631f3a767f7

This sample reads a SQL file (for example: one that was output from mysqldump) and executes each line as a query. At least in my configuration of mysqldump, each insert statement was on a single line, so I was able to write data from MySQL to BigQuery with this sample. Change-Id: Id14b648b0ce6bac651e436d402f480c56d80bd37

Also, adds a few explanatory comments for the docs. Change-Id: I623bf226839ab43f8da8297938223323a04e5838

Change-Id: Ie0d12ac9aedfdd83d2f6f533ad30265f75126e4f

theacodes

I may have said this in my original review, but the populate_db.py script is a really complex sample. I'm curious as to the reasoning behind choosing such a complex sample. Is there a much simpler sample that could teach the same thing?

theacodes · 2016-09-28T17:48:57Z

bigquery/dml/insert_sql.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Sample to run line-separated SQL statements in Big Query from a file.


Awkward phrasing, maybe: Sample that runs a file containing line-separated SQL statements in Big Query.

Done. (I realized this sample was a bit more complicated than it needed to be, too. I changed it to "Sample that runs a file containing INSERT SQL statements in Big Query." and modified the loop to look for lines that start with INSERT (to match the command-line sample I wrote for the docs)

theacodes · 2016-09-28T17:49:29Z

bigquery/dml/insert_sql.py

+from __future__ import print_function
+
+import argparse
+# [START insert_sql]


Either include all imports, or include none of them.

Done. (Included all)

theacodes · 2016-09-28T17:50:20Z

bigquery/dml/insert_sql.py

+            query.run()
+            return
+        except exceptions.GCloudError as err:
+


This newline isn't needed (blank newline is recommended between expressions and statements, but not between two statements)

Thanks, good to know. I deleted this function entirely based on your previous feedback. (I made some command-line samples to do the same thing and it really does complicate it a lot to add retries. It doesn't seem to be helping for most errors, anyway)

theacodes · 2016-09-28T17:52:03Z

bigquery/dml/insert_sql.py

+        parser = argparse.ArgumentParser(
+            description=__doc__,
+            formatter_class=argparse.RawDescriptionHelpFormatter)
+        parser.add_argument('--project', help='Google Cloud project name')


use positional command-line args for required items, use flags for items with sensible defaults.

theacodes · 2016-09-28T17:54:37Z

bigquery/dml/populate_db_test.py

+    session = create_session(engine)
+
+    try:
+        populate_db(session, total_users=100)


Can we do like < 10 for the sake of speed?

Removes unnecessary UserActions table from populate_db.py. Removes retry logic and changes sample to only look for INSERT lines in insert_sql.py Change-Id: If8994c420cd95babf3c4673a3b87affbfca4f32a

tswast

I simplified the populate_db.py script. You are right that it was too complicated. The UserActions table was unnecessary for the docs I have written.

tswast · 2016-09-28T18:28:10Z

bigquery/dml/insert_sql.py

+from __future__ import print_function
+
+import argparse
+# [START insert_sql]


Done. (Included all)

tswast · 2016-09-28T18:29:31Z

bigquery/dml/insert_sql.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Sample to run line-separated SQL statements in Big Query from a file.


Done. (I realized this sample was a bit more complicated than it needed to be, too. I changed it to "Sample that runs a file containing INSERT SQL statements in Big Query." and modified the loop to look for lines that start with INSERT (to match the command-line sample I wrote for the docs)

tswast · 2016-09-28T18:31:29Z

bigquery/dml/insert_sql.py

+            query.run()
+            return
+        except exceptions.GCloudError as err:
+


Thanks, good to know. I deleted this function entirely based on your previous feedback. (I made some command-line samples to do the same thing and it really does complicate it a lot to add retries. It doesn't seem to be helping for most errors, anyway)

tswast · 2016-09-28T18:31:35Z

bigquery/dml/insert_sql.py

+        parser = argparse.ArgumentParser(
+            description=__doc__,
+            formatter_class=argparse.RawDescriptionHelpFormatter)
+        parser.add_argument('--project', help='Google Cloud project name')


tswast · 2016-09-28T18:31:46Z

bigquery/dml/populate_db_test.py

+    session = create_session(engine)
+
+    try:
+        populate_db(session, total_users=100)


* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>

tswast added 6 commits September 2, 2016 12:45

Add tests to BigQuery DML samples. Convert to SQLAlchemy.

c2a86fe

This tests the SQL code to create the tables, as well as the Python code that creates the rows. Uses SQLAlchemy to abstract away differences between database engines. Change-Id: Id9e70eef56f5e203921b6c1f21708631f3a767f7

Add region tags for DML sample.

16f4835

Also, adds a few explanatory comments for the docs. Change-Id: I623bf226839ab43f8da8297938223323a04e5838

Merge remote-tracking branch 'github/master' into tswast-bq-dml-insert

e78d902

Add docs link to BigQuery DML README.

cc92c05

Change-Id: Ie0d12ac9aedfdd83d2f6f533ad30265f75126e4f

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Sep 28, 2016

tswast assigned theacodes Sep 28, 2016

theacodes suggested changes Sep 28, 2016

View reviewed changes

Simplify DML samples.

ef345a2

Removes unnecessary UserActions table from populate_db.py. Removes retry logic and changes sample to only look for INSERT lines in insert_sql.py Change-Id: If8994c420cd95babf3c4673a3b87affbfca4f32a

tswast commented Sep 28, 2016

View reviewed changes

theacodes approved these changes Sep 28, 2016

View reviewed changes

theacodes merged commit 0f85ad0 into master Sep 28, 2016

theacodes deleted the tswast-bq-dml-insert branch September 28, 2016 19:06

telpirion mentioned this pull request Jan 18, 2023

chore(dialogflow): migrate code from googleapis/python-dialogflow #9011

Closed

8 tasks

yil532 mentioned this pull request Feb 1, 2023

chore(dialogflow): migrate code from googleapis/python-dialogflow #9078

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DML samples for BigQuery. #546

Add DML samples for BigQuery. #546

tswast commented Sep 28, 2016

theacodes left a comment

theacodes Sep 28, 2016

tswast Sep 28, 2016

theacodes Sep 28, 2016

tswast Sep 28, 2016

theacodes Sep 28, 2016

tswast Sep 28, 2016

theacodes Sep 28, 2016

tswast Sep 28, 2016

theacodes Sep 28, 2016

tswast Sep 28, 2016

tswast left a comment

tswast Sep 28, 2016

tswast Sep 28, 2016

tswast Sep 28, 2016

tswast Sep 28, 2016

tswast Sep 28, 2016

Add DML samples for BigQuery. #546

Add DML samples for BigQuery. #546

Conversation

tswast commented Sep 28, 2016

theacodes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tswast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment