syslog configuration doesn't work with provided docker_image

[mcousens]

Describe the bug

Enabling the advertised syslog integration in the global config options https://www.consul.io/docs/nia/configuration#global-config-options while running CTS via the provided docker image results in the below error:

[ERROR] cli: error setting up logging: error="error setting up syslog logger: Unix syslog delivery error"

Of note, this error only occurs when explicitly enabling the integration (when you don't pass enabled = true the daemon will start but the syslog integration fails silently and doesn't do anything - despite "Specifying other option also enables syslog logging")

syslog {
  enabled = true
  facility = "local7"
}

Versions

Consul Terraform Sync

consul-terraform-sync v0.5.2
From image: hashicorp/consul-terraform-sync-enterprise:0.5.2-ent

Consul Version

Consul 1.8.4

Terraform Version

Terraform v0.15.0

Other Details

• docker image: hashicorp/consul-terraform-sync-enterprise:0.5.2-ent
• driver: terraform-cloud
• orchestrating daemon via Nomad
• using Terraform Enterprise

Configuration File(s)

Click to toggle contents of config file

log_level = "DEBUG"

buffer_period {
  enabled = true
  min     = "5s"
}

syslog {
  enabled = true
  facility = "local7"
}

# Consul connection setup
consul {
  address = "<redacted>"
  tls {
    enabled = true
    verify  = false
  }
}

# CTS task: monitors Consul service catalog and apply TF code
# in specified module whenever service layout changes
task {
  name           = "core_alb_core_backend-qa"
  description    = "Dynamically manage target groups in qa via Consul-Terraform-Sync (CTS)"
  enabled        = true
  providers      = ["aws"]
  module         = "<redacted>/core-alb-backend/cts"
  # if you don't pin module version, latest will be used
  # version        = "1.0.1"
  variable_files = ["/local/core_alb_core_backend_qa.tfvars"]

  condition "services" {
    names = ["<redacted>"]
    filter     = "Service.Tags contains \"<redacted>\""
  }
}


# TFE connection setup
driver "terraform-cloud" {
  hostname         = "<redacted>"
  organization     = "<redacted>"
  token            = "<redacted>"
  workspaces {
    prefix = "cts-"
    tags   = ["source:cts"]
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.1.0"
    }
  }
}

# AWS provider setup
terraform_provider "aws" {
  region = "us-east-1"
  task_env {
    "AWS_ACCESS_KEY_ID"     = "<redacted>"
    "AWS_SECRET_ACCESS_KEY" = "<redacted>"
  }
}

Terraform Configuration Files Generated by Consul-Terraform-Sync

Click to toggle contents of main.tf

# This file is generated by Consul Terraform Sync.
#
# The HCL blocks, arguments, variables, and values are derived from the
# operator configuration for Sync. Any manual changes to this file
# may not be preserved and could be overwritten by a subsequent update.
#
# Task: core_alb_backend-qa
# Description: Dynamically manage core ALB target groups in qa via Consul-Terraform-Sync (CTS)

terraform {
  required_version = ">= 0.13.0, < 1.2.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "4.1.0"
    }
  }
}

provider "aws" {
  region = var.aws.region
}

# Dynamically manage core ALB target groups in qa via Consul-Terraform-Sync (CTS)
module "core_alb_backend-qa" {
  source   = "<redacted>/core-alb-backend/cts"
  services = var.services

  environment  = var.environment
  target_group = var.target_group
}

Click to toggle contents of terraform.tfvars

environment  = "qa"
target_group = "core"

Terraform Module

If using a private or local Terraform module, share relevant parts of your module here.

Task Variable Files

If passing in task variable file(s), share relevant parts of your variable file(s) here.

Expected Behavior

Ability to pass host/port of syslog to CTS so that it can reach syslog running outside of container.

Actual Behavior

Daemon can't start, errors with

[ERROR] cli: error setting up logging: error="error setting up syslog logger: Unix syslog delivery error"

consul-terraform-sync/logging/logging.go

Lines 61 to 64 in 3030b23

	syslog, err := gsyslog.NewLogger(gsyslog.LOG_NOTICE, config.SyslogFacility, config.SyslogName)
	if err != nil {
	return fmt.Errorf("error setting up syslog logger: %s", err)
	}

It looks like the syslog golang wrapper library used defaults to localhost
https://github.com/hashicorp/go-syslog/blob/master/builtin.go#L56-L58
https://pkg.go.dev/log/syslog#Dial
and doesn't allow this to be overwritten. Thus, CTS tries to connect to syslog locally and the container is not running syslog so it throws an error.

Steps to Reproduce

1. Run CTS via docker image
2. Explicitly enable the syslog integration
3. Start CTS
4. See error

Additional Context

[lornasong]

HI @mcousens thanks for opening this issue!

If I understand you correctly, you've found two issues. Please correct me if I've misunderstood.

1. The syslog block requires users to set enabled to be true even though other fields are set. This contradicts the docs, as quoted in the description
2. The syslog configuration only works for local syslog server

Thanks for all the details and research you've included in the issue. Really appreciate it! I was able to reproduce both issues you found.

For the first issue, I can open up a PR to fix this bug.

For the second issue, it looks like we might be able to use the gsyslog.DialLogger instead of the gsyslog.NewLogger (DialLogger link). Looks like it may require adding some new configuration to the syslog block e.g. network and address. Do you have any thoughts on these configurations? Wanted to check to see if it sounds like they would help with the issue you're encountering?

Thank you for your time!

[mcousens]

Hi @lornasong, thanks so much for looking into this. Your understanding is spot on, and that’s great that it was reproducible.

For the second issue, adding network and address parameters sounds like a good approach to us. As long as we can pass the network protocol and host:port to reach the syslog server outside the docker container we are happy campers.

[lornasong]

Hi @mcousens thanks so much for the reply and confirmation! That's helpful to know.

For the first issue, I've put up a fix #815 that will go out with the next CTS release. Please let me know if you see any issues.

For the second issue, I checked in with our team and wanted to see if we could get input on a few questions:

• We were wondering if this is blocking you and your team's ability to deploy CTS?
• Wanted to know a little more about your deployment - is this for production or something else?
• Are there any workarounds that you all were able to find? Totally ok if the answer is no.

CC-ing our Product Manager @devarshishah3

Thank you

[mcousens]

Hi @lornasong

Thanks for fix #815. I'm not a Go developer, but logically it looks sane to me :) See responses below:

• We were wondering if this is blocking you and your team's ability to deploy CTS?
  Our original outlined requirements for a production deployment was to persist the CTS logs, so this was blocking us originally. However, due to the inability to find a workaround for log persistence, and the fact that the most important logging to us is naturally persisted in the Terraform Enterprise workspaces, we moved this requirement to a nice to have and are continuing with our production rollout.

• Wanted to know a little more about your deployment - is this for production or something else?
  Yes, it’s for production. We run the CTS daemon via Nomad using the enterprise docker image and it’s connecting to our Terraform Enterprise and Consul services. Let me know if you have any specific questions about our setup.

• Are there any workarounds that you all were able to find? Totally ok if the answer is no.
  Unfortunately, no. As a larger project, we’d like to explore persisting anything that writes to stderr in a Nomad allocation, but this is a bit larger of a project to take on.

[devarshishah3]

@mcousens. Thanks for raising the issue and using CTS in your production environment. We would love to understand your environment, and CTS use case in more detail and gather feedback from you to shape CTS roadmap. Would you or someone from you team be open to a 45 min chat with us?

[mcousens]

Hi @devarshishah3, we'd be happy to chat. Feel free to reach out over email - Eventbrite has a hashicorp rep that should have my contact info (not sure if I should place it in this public forum :) )