-
Notifications
You must be signed in to change notification settings - Fork 197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Operation timed out trying to connect to GCP DB via proxy #45
Comments
EDIT: My issue ended up being due to network access control list (NACL) settings that were blocking the traffic between the subnet with my CodeBuild project and the subnet with RDS Aurora. My mistake! I am seeing similar behavior when trying to connect to an RDS Aurora cluster endpoint. I get the "Still creating..." message until my project in AWS CodeBuild eventually times out. Here is my provider configuration. Perhaps there is an issue with connecting via the GoCloud? I know this is new in version 1.11.0.
Like @wilhelmi, I am also connecting to a DB that only has a private DNS name (in a private subnet), and I am running Terraform from a container that is running in AWS CodeBuild in the same subnet. I have combed through the |
@wilhelmi Thanks for opening the issue. It will be hard for us to debug your problem as it seems to be more network configuration problems. But I'll try to execute your example as soon as I can to see if it works no my side. |
I am also getting the exact same error. I have an existing database cluster that was created successfully from Terraform:
And then when I try to create an additional postgres db by appending to the same Terraform file:
And running
And my
|
Hi @cfloody , From where are your running Terraform? |
@cfloody, I ran into this issue last week and my set-up is also in a private subnet. For me, I had to adjust my NACL to allow traffic between the private subnet running Terraform and the private subnet where I launched my RDS cluster. You should also check your security group rules and route tables. Any of those three things could potentially be blocking the traffic, in which case you'll see the behavior you're seeing. I was able to troubleshoot this was a network issue by trying to connect with For reference, here is the provider configuration that works for me. Note, I am using aurora postgres serverless. I am also reading the RDS connection information from a data provider because I launched my cluster from a different Terraform configuration.
You can get the AWS Root Cert from the RDS documentation here, https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html. I found this worked better than letting the GO CDK do it, because that was returning x509 cert errors. This seems to be a GO-AWS thing, but folks are working on it. |
I am running this from a local workstation outside of GCP, the target in this case. From my reading of the gocloud library it should be using the cloud-sql-proxy lib to connect to the DB. This should connect to the DB even though it only has a private IP. I can connect to the target DB from the same machine if I start cloud-sql-proxy from the CLI. So I think the network route works. One thing that jumped at me was the 3307 port when 5432 was specifically configured. |
Also running into this. From what I can tell the port being not 5432 is ok (although I initially reacted to it as well): https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/ec20300fbae3d11e93a9cc62946463632f3d9773/proxy/proxy/client.go#L84-L86 |
I've been battling this all day as well. However, like @carlpett, I had no luck connecting with the cloud_sql_proxy as long as the instance does not have a public v4 address. As per https://cloud.google.com/sql/docs/postgres/connect-admin-proxy#authentication-options it also seems that "IAM user database authentication is not currently integrated with the Cloud SQL Auth proxy", which might be the root of my issues. Anyway, assigning a v4 (can be ephemeral) to the instance allows the provider to access the database¹ - GoogleCloudPlatform/cloud-sql-proxy#323 While this doesn't solve the issue itself, I was able to work around it with the following (abridged) setup: # terraform-service-account is a GCP service account with a bunch of scopes, for Cloud SQL & Networking we need at least:
# - roles/cloudsql.admin
# - roles/compute.networkAdmin
$ TF_VAR_GOOGLE_CREDENTIALS="$(cat terraform-service-account-key.json)" terraform <command> The relevant bits from my provider "google-beta" {
credentials = var.GOOGLE_CREDENTIALS
# [...]
}
data "google_project" "default" {}
resource "google_compute_network" "default" {
provider = google-beta
name = "default"
# [...]
auto_create_subnetworks = true
}
resource "google_compute_subnetwork" "default" {
provider = google-beta
name = "default"
region = "europe-west3"
network = google_compute_network.default.id
ip_cidr_range = "10.156.0.0/20"
purpose = "PRIVATE"
private_ip_google_access = true
}
provider "postgresql" {
alias = "pg-main"
host = google_sql_database_instance.pg-main.connection_name
# I tried using a service account here, didn't work. Fell back to the "postgres" default user, which does work.
username = google_sql_user.pg-main-postgres.name
password = google_sql_user.pg-main-postgres.password
scheme = "gcppostgres"
superuser = false
expected_version = var.POSTGRES_VERSION
}
resource "google_sql_database_instance" "pg-main" {
provider = google-beta
name = "pg-main"
database_version = "POSTGRES_${var.POSTGRES_VERSION}"
# [...]
settings {
ip_configuration {
ipv4_enabled = true
require_ssl = true
private_network = data.google_compute_network.default.id
}
# :(
database_flags {
name = "cloudsql.iam_authentication"
value = "on"
}
# [...]
}
}
resource "google_compute_global_address" "pg-main-private" {
provider = google-beta
name = "pg-main-private"
purpose = "VPC_PEERING"
address_type = "INTERNAL"
prefix_length = 20
network = data.google_compute_network.default.id
}
resource "google_service_networking_connection" "pg-main-private-vpc" {
provider = google-beta
network = data.google_compute_network.default.id
service = "servicenetworking.googleapis.com"
reserved_peering_ranges = [
google_compute_global_address.pg-main-private.name
]
depends_on = [google_compute_global_address.pg-main-private]
}
resource "random_password" "pg-main-postgres" {
length = 32
special = true
override_special = "_-"
}
resource "google_sql_user" "pg-main-postgres" {
name = "postgres"
password = random_password.pg-main-postgres.result
instance = google_sql_database_instance.pg-main.name
}
# terraform-provider-postgres now works:
resource "postgresql_database" "pg-main-db1" {
provider = postgresql.pg-main
name = "db1"
owner = "some-role-you-can-also-define-with-the-provider"
} ¹: It does not have to be listed as an "authorized network" on the Cloud SQL instance, but somehow magically makes things work. I'm at wit's end here, really interested in @wilhelmi's setup... ²: Edit: I realized today while going through some logs that during my frantic attempts to make this work I must've run |
@itspngu Without addressing your other points, the proxy does support IAM auth, see this pull request. It does seem to require adding some parameters to the invocation, though. |
For what it’s worth, the proxy doesn’t create a network path. So if you’re trying to run operations against an instance that is private IP only and your terraform runner isn’t in the same VPC, it’s never going to work. |
Ran into this today |
The Connector there is a newer version than the one used here. The one used here will always try public and private IP in that order. |
Hi there,
Thank you for opening an issue. Please provide the following information:
Terraform Version
0.13.6
Affected Resource(s)
Terraform Configuration Files
Debug Output
Expected Behavior
The new database is created, via cloud_sql_proxy tunnel provided by the go cloud lib.
Actual Behavior
postgresql_database.my_db: Still creating... [1m10s elapsed]
Error: Error creating database "my_db": dial tcp 10.35.0.3:3307: connect: operation timed out
Steps to Reproduce
Please list the steps required to reproduce the issue, for example:
terraform apply
Important Factoids
I can connect to the DB via local cloud_sql_proxy. DB does not have a public IP.
The text was updated successfully, but these errors were encountered: