Add support for Stored Procedure for Apache Spark

[marblejenka]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

BigQuery Stored procedures for Apache Spark is introduced as a public preview feature.

Stored procedure is already supported by #4493 , so we can just add a new kind of stored procedure in resource_bigquery_routine.

New or Affected Resource(s)

Affected resource

• google_bigquery_routine

Potential Terraform Configuration

resource "google_bigquery_dataset" "test" {
	dataset_id = "<%= ctx[:vars]['dataset_id'] %>"
}

resource "google_bigquery_connection" "test" {
	dataset_id = "<%= ctx[:vars]['dataset_id'] %>"
	spark { }
}

resource "google_bigquery_routine" "pyspark" {
  dataset_id = google_bigquery_dataset.test.dataset_id
  routine_id     = "<%= ctx[:vars]['routine_id'] %>"
  routine_type = "PROCEDURE"
  language = "PYTHON"
  connection = google_bigquery_connection.test.id
  definition_body = <<-EOS
    from pyspark.sql import SparkSession

    spark = SparkSession.builder.appName("spark-bigquery-demo").getOrCreate()
    
    # Load data from BigQuery.
    words = spark.read.format("bigquery") \
      .option("table", "bigquery-public-data:samples.shakespeare") \
      .load()
    words.createOrReplaceTempView("words")
    
    # Perform word count.
    word_count = words.select('word', 'word_count').groupBy('word').sum('word_count').withColumnRenamed("sum(word_count)", "sum_word_count")
    word_count.show()
    word_count.printSchema()
    
    # Saving the data to BigQuery
    word_count.write.format("bigquery") \
      .option("writeMethod", "direct") \
      .save("wordcount_dataset.wordcount_output")
  EOS
}

References

• Allow creation of UDFs in a dataset #4493
• The connection resources required to create the Stored Procedure for Apache Spark are implemented in bigquery connection - spark connection type GoogleCloudPlatform/magic-modules#7498

b/319619249

[marblejenka]

I will work on this issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.