feat: Allow users to configure egress policies for all the VPC-SC per…

…imeters (#189) * use regular service perimeter module instead of terraform resource * allow configuration of vpc-sc egress rules for all the perimeters * restore usage of terraform resource and use egress rule dynamic creation * add resource to ignore chances in the service perimeter * fix typo Co-authored-by: Bharath KKB <bharathkrishnakb@gmail.com> * removed ignore changes Co-authored-by: Luiz Dutra <luizdutra@ciandt.com>
GoogleCloudPlatform · Dec 1, 2021 · 63c70c9 · 63c70c9
1 parent 8064184
commit 63c70c9
Show file tree

Hide file tree

Showing 5 changed files with 96 additions and 39 deletions.
diff --git a/README.md b/README.md
@@ -54,11 +54,14 @@ module "secured_data_warehouse" {
 | cmek\_keyring\_name | The Keyring name for the KMS Customer Managed Encryption Keys being provisioned. | `string` | n/a | yes |
 | confidential\_access\_members | List of members in the standard GCP form: user:{email}, serviceAccount:{email}, group:{email} who will have access to confidential information in BigQuery. | `list(string)` | `[]` | no |
 | confidential\_data\_dataflow\_deployer\_identities | List of members in the standard GCP form: user:{email}, serviceAccount:{email} that will deploy Dataflow jobs in the Confidential Data project. These identities will be added to the VPC-SC secure data exchange egress rules. | `list(string)` | `[]` | no |
+| confidential\_data\_egress\_policies | A list of all [egress policies](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#egress-rules-reference) for the Confidential Data perimeter, each list object has a `from` and `to` value that describes egress\_from and egress\_to. See also [secure data exchange](https://cloud.google.com/vpc-service-controls/docs/secure-data-exchange#allow_access_to_a_google_cloud_resource_outside_the_perimeter) and the [VPC-SC](https://github.com/terraform-google-modules/terraform-google-vpc-service-controls/blob/v3.1.0/modules/regular_service_perimeter/README.md) module. | <pre>list(object({<br>    from = any<br>    to   = any<br>  }))</pre> | `[]` | no |
 | confidential\_data\_project\_id | Project where the confidential datasets and tables are created. | `string` | n/a | yes |
 | confidential\_dataset\_default\_table\_expiration\_ms | TTL of tables using the dataset in MS. The default value is null. | `number` | `null` | no |
 | confidential\_dataset\_id | Unique ID for the confidential dataset being provisioned. | `string` | `"secured_dataset"` | no |
+| data\_governance\_egress\_policies | A list of all [egress policies](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#egress-rules-reference) for the Data Governance perimeter, each list object has a `from` and `to` value that describes egress\_from and egress\_to. See also [secure data exchange](https://cloud.google.com/vpc-service-controls/docs/secure-data-exchange#allow_access_to_a_google_cloud_resource_outside_the_perimeter) and the [VPC-SC](https://github.com/terraform-google-modules/terraform-google-vpc-service-controls/blob/v3.1.0/modules/regular_service_perimeter/README.md) module. | <pre>list(object({<br>    from = any<br>    to   = any<br>  }))</pre> | `[]` | no |
 | data\_governance\_project\_id | The ID of the project in which the data governance resources will be created. | `string` | n/a | yes |
 | data\_ingestion\_dataflow\_deployer\_identities | List of members in the standard GCP form: user:{email}, serviceAccount:{email} that will deploy Dataflow jobs in the Data Ingestion project. These identities will be added to the VPC-SC secure data exchange egress rules. | `list(string)` | `[]` | no |
+| data\_ingestion\_egress\_policies | A list of all [egress policies](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#egress-rules-reference) for the Data Ingestion perimeter, each list object has a `from` and `to` value that describes egress\_from and egress\_to. See also [secure data exchange](https://cloud.google.com/vpc-service-controls/docs/secure-data-exchange#allow_access_to_a_google_cloud_resource_outside_the_perimeter) and the [VPC-SC](https://github.com/terraform-google-modules/terraform-google-vpc-service-controls/blob/v3.1.0/modules/regular_service_perimeter/README.md) module. | <pre>list(object({<br>    from = any<br>    to   = any<br>  }))</pre> | `[]` | no |
 | data\_ingestion\_project\_id | The ID of the project in which the data ingestion resources will be created | `string` | n/a | yes |
 | dataset\_default\_table\_expiration\_ms | TTL of tables using the dataset in MS. The default value is null. | `number` | `null` | no |
 | dataset\_description | Dataset description. | `string` | `"Data-ingestion dataset"` | no |

diff --git a/modules/dwh-vpc-sc/main.tf b/modules/dwh-vpc-sc/main.tf
@@ -49,9 +49,8 @@ module "access_level_policy" {
   regions        = var.access_level_regions
 }
 
-# We are using the terraform resource `google_access_context_manager_service_perimeter`
-# instead of the module "terraform-google-modules/vpc-service-controls/google/modules/regular_service_perimeter"
-# until ingress and egress rules are added to the module https://github.com/terraform-google-modules/terraform-google-vpc-service-controls/issues/53
+# Cannot use the module "terraform-google-modules/vpc-service-controls/google/modules/regular_service_perimeter"
+# because we need to set the  lifecycle of the resource.
 resource "google_access_context_manager_service_perimeter" "regular_service_perimeter" {
   provider       = google
   parent         = "accessPolicies/${local.actual_policy}"
@@ -68,22 +67,30 @@ resource "google_access_context_manager_service_perimeter" "regular_service_peri
       [module.access_level_policy.name]
     )
 
-    # Configure Egress rule to allow fetch of External Dataflow flex template jobs.
-    # Flex templates in public buckets don't need an egress rule.
     dynamic "egress_policies" {
-      for_each = var.sdx_egress_rule
+      for_each = var.egress_policies
       content {
+        egress_from {
+          identity_type = lookup(egress_policies.value["from"], "identity_type", null)
+          identities    = lookup(egress_policies.value["from"], "identities", null)
+        }
         egress_to {
-          operations {
-            service_name = egress_policies.value.service_name
-            method_selectors {
-              method = egress_policies.value.method
+          resources = lookup(egress_policies.value["to"], "resources", ["*"])
+          dynamic "operations" {
+            for_each = lookup(egress_policies.value["to"], "operations", [])
+            content {
+              service_name = operations.key
+              dynamic "method_selectors" {
+                for_each = merge(
+                  { for k, v in lookup(operations.value, "methods", {}) : v => "method" },
+                { for k, v in lookup(operations.value, "permissions", {}) : v => "permission" })
+                content {
+                  method     = method_selectors.value == "method" ? method_selectors.key : ""
+                  permission = method_selectors.value == "permission" ? method_selectors.key : ""
+                }
+              }
             }
           }
-          resources = ["projects/${egress_policies.value.sdx_project_number}"]
-        }
-        egress_from {
-          identities = egress_policies.value.sdx_identities
         }
       }
     }

diff --git a/modules/dwh-vpc-sc/variables.tf b/modules/dwh-vpc-sc/variables.tf
@@ -68,13 +68,11 @@ variable "access_level_regions" {
   default     = []
 }
 
-variable "sdx_egress_rule" {
-  description = "List of Egress rule configurations to allow Dataflow to fetch the Flex template in Cloud Storage in an external project, See: https://cloud.google.com/vpc-service-controls/docs/secure-data-exchange#allow_access_to_a_google_cloud_resource_outside_the_perimeter. `sdx_project_number`, is the Project Number to configure Secure data exchange with the egress rule for the dataflow templates and `sdx_identities`, is the list of email address of the Google Identities that will need to access the external flex template. format is `user:<USER_EMAIL>` or `serviceAccount:<SERVICE_ACCOUNT_EMAIL>`. These are the identities that will deploy Dataflow jobs."
+variable "egress_policies" {
+  description = "A list of all [egress policies](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#egress-rules-reference), each list object has a `from` and `to` value that describes egress_from and egress_to. See also [secure data exchange](https://cloud.google.com/vpc-service-controls/docs/secure-data-exchange#allow_access_to_a_google_cloud_resource_outside_the_perimeter)."
   type = list(object({
-    sdx_project_number = string
-    service_name       = string
-    method             = string
-    sdx_identities     = list(string)
+    from = any
+    to   = any
   }))
   default = []
 }
diff --git a/service_control.tf b/service_control.tf
@@ -96,17 +96,27 @@ module "data_ingestion_vpc_sc" {
   perimeter_members                = local.perimeter_members_data_ingestion
   restricted_services              = local.restricted_services
 
-  sdx_egress_rule = [
+  egress_policies = distinct(concat([
     {
-      sdx_identities = distinct(concat(
-        var.data_ingestion_dataflow_deployer_identities,
-        ["serviceAccount:${var.terraform_service_account}"]
-      ))
-      sdx_project_number = var.sdx_project_number
-      service_name       = "storage.googleapis.com"
-      method             = "google.storage.objects.get"
-    }
-  ]
+      "from" = {
+        "identity_type" = ""
+        "identities" = distinct(concat(
+          var.data_ingestion_dataflow_deployer_identities,
+          ["serviceAccount:${var.terraform_service_account}"]
+        ))
+      },
+      "to" = {
+        "resources" = ["projects/${var.sdx_project_number}"]
+        "operations" = {
+          "storage.googleapis.com" = {
+            "methods" = [
+              "google.storage.objects.get"
+            ]
+          }
+        }
+      }
+    },
+  ], var.data_ingestion_egress_policies))
 
   # depends_on needed to prevent intermittent errors
   # when the VPC-SC is created but perimeter member
@@ -128,6 +138,8 @@ module "data_governance_vpc_sc" {
   perimeter_members                = local.perimeter_members_governance
   restricted_services              = local.restricted_services
 
+  egress_policies = var.data_governance_egress_policies
+
   # depends_on needed to prevent intermittent errors
   # when the VPC-SC is created but perimeter member
   # not yet propagated.
@@ -148,17 +160,27 @@ module "confidential_data_vpc_sc" {
   perimeter_members                = local.perimeter_members_confidential
   restricted_services              = local.restricted_services
 
-  sdx_egress_rule = [
+  egress_policies = distinct(concat([
     {
-      sdx_identities = distinct(concat(
-        var.confidential_data_dataflow_deployer_identities,
-        ["serviceAccount:${var.terraform_service_account}"]
-      ))
-      sdx_project_number = var.sdx_project_number
-      service_name       = "storage.googleapis.com"
-      method             = "google.storage.objects.get"
+      "from" = {
+        "identity_type" = ""
+        "identities" = distinct(concat(
+          var.confidential_data_dataflow_deployer_identities,
+          ["serviceAccount:${var.terraform_service_account}"]
+        ))
+      },
+      "to" = {
+        "resources" = ["projects/${var.sdx_project_number}"]
+        "operations" = {
+          "storage.googleapis.com" = {
+            "methods" = [
+              "google.storage.objects.get"
+            ]
+          }
+        }
+      }
     }
-  ]
+  ], var.confidential_data_egress_policies))
 
   # depends_on needed to prevent intermittent errors
   # when the VPC-SC is created but perimeter member

diff --git a/variables.tf b/variables.tf
@@ -195,6 +195,33 @@ variable "kms_key_protection_level" {
   default     = "HSM"
 }
 
+variable "data_ingestion_egress_policies" {
+  description = "A list of all [egress policies](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#egress-rules-reference) for the Data Ingestion perimeter, each list object has a `from` and `to` value that describes egress_from and egress_to. See also [secure data exchange](https://cloud.google.com/vpc-service-controls/docs/secure-data-exchange#allow_access_to_a_google_cloud_resource_outside_the_perimeter) and the [VPC-SC](https://github.com/terraform-google-modules/terraform-google-vpc-service-controls/blob/v3.1.0/modules/regular_service_perimeter/README.md) module."
+  type = list(object({
+    from = any
+    to   = any
+  }))
+  default = []
+}
+
+variable "data_governance_egress_policies" {
+  description = "A list of all [egress policies](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#egress-rules-reference) for the Data Governance perimeter, each list object has a `from` and `to` value that describes egress_from and egress_to. See also [secure data exchange](https://cloud.google.com/vpc-service-controls/docs/secure-data-exchange#allow_access_to_a_google_cloud_resource_outside_the_perimeter) and the [VPC-SC](https://github.com/terraform-google-modules/terraform-google-vpc-service-controls/blob/v3.1.0/modules/regular_service_perimeter/README.md) module."
+  type = list(object({
+    from = any
+    to   = any
+  }))
+  default = []
+}
+
+variable "confidential_data_egress_policies" {
+  description = "A list of all [egress policies](https://cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#egress-rules-reference) for the Confidential Data perimeter, each list object has a `from` and `to` value that describes egress_from and egress_to. See also [secure data exchange](https://cloud.google.com/vpc-service-controls/docs/secure-data-exchange#allow_access_to_a_google_cloud_resource_outside_the_perimeter) and the [VPC-SC](https://github.com/terraform-google-modules/terraform-google-vpc-service-controls/blob/v3.1.0/modules/regular_service_perimeter/README.md) module."
+  type = list(object({
+    from = any
+    to   = any
+  }))
+  default = []
+}
+
 variable "additional_restricted_services" {
   description = "The list of additional Google services to be protected by the VPC-SC service perimeters."
   type        = list(string)