Adds support for scanning tfplan json file

[kanchwala-yusuf]

This PR adds support for scanning terraform plan json file. The terraform plan json is the output of terraform show -json. The user needs to typically perform the following steps:

$ terraform plan -out tfplan.out
$ terraform show -json tfplan.out > tfplan.json
$ terrascan scan -i tfplan -t aws -f tfplan.json

JSON created out of terraform state files are not supported.

Resolves #407

[codecov]

Codecov Report

Merging #562 (9386f86) into master (bda153e) will increase coverage by 0.10%.
The diff coverage is 81.66%.

@@            Coverage Diff             @@
##           master     #562      +/-   ##
==========================================
+ Coverage   77.25%   77.36%   +0.10%     
==========================================
  Files          98      102       +4     
  Lines        2387     2447      +60     
==========================================
+ Hits         1844     1893      +49     
- Misses        400      408       +8     
- Partials      143      146       +3     

Impacted Files	Coverage Δ
pkg/iac-providers/tfplan/v1/load-file.go	77.77% <77.77%> (ø)
pkg/utils/jqhelper.go	86.36% <86.36%> (ø)
pkg/iac-providers/tfplan.go	100.00% <100.00%> (ø)
pkg/iac-providers/tfplan/v1/load-dir.go	100.00% <100.00%> (ø)

[ismailyenigul]

I am going to build terrascan from this branch and will make some tests with terraform json files.
I will keep you updated.

[ismailyenigul]

File value is empty and Line number is 0.

$ jq < tf.json  >tf-formatted.json
$ wc -l tf.json 
       1 tf.json
$ wc -l tf-formatted.json 
     232 tf-formatted.json
$ terrascan scan -t aws -i tfplan -f tf-formatted.json 


Violation Details -
    
	Description    :	ssh port open to internet
	File           :	
	Line           :	0
	Severity       :	HIGH
	-----------------------------------------------------------------------
	

Scan Summary -

	File/Folder         :	/Users/ismail/dev/sg/sg/tf.json
	IaC Type            :	tfplan
	Scanned At          :	2021-02-22 11:04:42.305015 +0000 UTC
	Policies Validated  :	159
	Violated Policies   :	1
	Low                 :	0
	Medium              :	0
	High                :	1

it is interesting that terrascan could not detect the issue with ssh allowed from all ( I think I should create another ticket for this)

$ terrascan scan -t aws -i terraform .


Scan Summary -

	File/Folder         :	/Users/ismail/dev/sg/sg
	IaC Type            :	terraform
	Scanned At          :	2021-02-22 11:05:28.959902 +0000 UTC
	Policies Validated  :	159
	Violated Policies   :	0
	Low                 :	0
	Medium              :	0
	High                :	0

all content

$ cat main.tf variables.tf provider.tf versions.tf 
resource "aws_security_group" "mysg" {
  name        = "${var.company}-${var.environment}-${var.application}-sg"
  vpc_id      = var.vpc_id
  description = "sg"

  # SSH access
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = var.cidrs
  }


  # outbound internet access
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "${var.company}-${var.environment}-${var.application}-sg"
  }
}
variable "vpc_id" {
  type        = string
  description = "VPC ID in which to deploy RDS"
  default     = "vpc-123"
}

variable "company" {
  description = "Name of the Company"
  default     = "test"
}

variable "environment" {
  description = "The aws environment"
  default     = "test"
}

variable "application" {
  description = "Application purpose of resource"
  default     = "test"
}

variable "cidrs" {
  description = "[List] IP CIDRs to whitelist in the passwork's security group"
  type        = list(string)
  default     = ["0.0.0.0/0"]
}

provider "aws" {
  region = "eu-west-1"
}

terraform {
  required_version = ">= 0.12"
}

and terraform plan output

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_security_group.mysg will be created
  + resource "aws_security_group" "mysg" {
      + arn                    = (known after apply)
      + description            = "sg"
      + egress                 = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = ""
              + from_port        = 0
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "-1"
              + security_groups  = []
              + self             = false
              + to_port          = 0
            },
        ]
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = ""
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
        ]
      + name                   = "test-test-test-sg"
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags                   = {
          + "Name" = "test-test-test-sg"
        }
      + vpc_id                 = "vpc-123"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

[ismailyenigul]

Scanning of tfplan json file definitely reports more finding than terraform but it seems there is a false report
for cloudtrail logging.

Terraform scanning

$  terrascan scan -i terraform -t aws  .

Violation Details -
    
	Description    :	Cloud Trail Multi Region not enabled
	File           :	../../main.tf
	Line           :	1
	Severity       :	MEDIUM
	-----------------------------------------------------------------------
	

Scan Summary -

	File/Folder         :	/Users/ismail/dev/star/cloudtrail/terraform-aws-cloudtrail-0.16.0/examples/complete
	IaC Type            :	terraform
	Scanned At          :	2021-02-22 13:14:54.933818 +0000 UTC
	Policies Validated  :	159
	Violated Policies   :	1
	Low                 :	0
	Medium              :	1
	High                :	0

tfplan scanning

~/d/s/c/t/e/complete $ terraform show -json tf.out | jq  > tf.json
~/d/s/c/t/e/complete $ terrascan scan -i tfplan -t aws  -f tf.json


Violation Details -
    
	Description    :	Ensure appropriate subscribers to each SNS topic
	File           :	
	Line           :	0
	Severity       :	MEDIUM
	-----------------------------------------------------------------------
	
	Description    :	Cloud Trail Log Not Enabled
	File           :	
	Line           :	0
	Severity       :	HIGH
	-----------------------------------------------------------------------
	
	Description    :	Cloud Trail Multi Region not enabled
	File           :	
	Line           :	0
	Severity       :	MEDIUM
	-----------------------------------------------------------------------
	
	Description    :	Enabling S3 versioning will enable easy recovery from both unintended user actions, like deletes and overwrites
	File           :	
	Line           :	0
	Severity       :	HIGH
	-----------------------------------------------------------------------
	

Scan Summary -

	File/Folder         :	/Users/ismail/dev/star/cloudtrail/terraform-aws-cloudtrail-0.16.0/examples/complete/tf.json
	IaC Type            :	tfplan
	Scanned At          :	2021-02-22 13:16:40.05334 +0000 UTC
	Policies Validated  :	159
	Violated Policies   :	4
	Low                 :	0
	Medium              :	2
	

terraform plan output

# module.cloudtrail.aws_cloudtrail.default[0] will be created
  + resource "aws_cloudtrail" "default" {
      + arn                           = (known after apply)
      + enable_log_file_validation    = true
      + enable_logging                = true
      + home_region                   = (known after apply)
      + id                            = (known after apply)
      + include_global_service_events = false
      + is_multi_region_trail         = false
      + is_organization_trail         = false
      + name                          = "eg-test-cloudtrail-test"
      + s3_bucket_name                = (known after apply)
      + tags                          = {
          + "Name"      = "eg-test-cloudtrail-test"
          + "Namespace" = "eg"
          + "Stage"     = "test"
        }
    }

as you can see above enable_logging = true so tfplan scanning should not report Cloud Trail Log Not Enabled

related lines in tf.json file

resources": [
 125             {
 126               "address": "module.cloudtrail.aws_cloudtrail.default[0]",
 127               "mode": "managed",
 128               "type": "aws_cloudtrail",
 129               "name": "default",
 130               "index": 0,
 131               "provider_name": "aws",
 132               "schema_version": 0,
 133               "values": {
 134                 "cloud_watch_logs_group_arn": "",
 135                 "cloud_watch_logs_role_arn": "",
 136                 "enable_log_file_validation": true,
 137                 "enable_logging": true,
 138                 "event_selector": [],
 139                 "include_global_service_events": false,
 140                 "insight_selector": [],
 141                 "is_multi_region_trail": false,
 142                 "is_organization_trail": false,
 143                 "kms_key_id": null,
 144                 "name": "eg-test-cloudtrail-test",
 145                 "s3_key_prefix": null,
 146                 "sns_topic_name": null,
 147                 "tags": {
 148                   "Name": "eg-test-cloudtrail-test",
 149                   "Namespace": "eg",
 150                   "Stage": "test"
 151                 }

Most probably the code does not scan child modules in tf.json
Attaching to the related tf.json file
tfplan.txt

Testing with https://github.com/cloudposse/terraform-aws-cloudtrail/tree/master/examples/complete code.

[kanchwala-yusuf]

Hi @ismailyenigul,

Thank you for trying out this PR.

Just a few things:

1. Figuring out file and line info is not really possible with tf plan files because the file/line info is not present in the tf plan itself. TF plan only consists of resource configurations without file/line info. In regular terrascan scanning with -i terraform, we can populate the file and line info because we parse all the .tf files which is not not the case for tf plan file.
   But if you think there is a way with which we can figure this info out, please let us know.

2. Cloudtrail logging: This may be an issue with the policy for cloud trail logging, need to see if the same gets reported with the regular terrascan scanning with -i terraform.

3. Scanning child modules: It is expected terraform plan output to have configurations for all the resources present including that of children modules as well. Is that a fair expectation?

[ismailyenigul]

1. Figuring out file and line info is not really possible with tf plan files because the file/line info is not present in the tf plan itself. TF plan only consists of resource configurations without file/line info. In regular terrascan scanning with -i terraform, we can populate the file and line info because we parse all the .tf files which is not not the case for tf plan file.

But if you think there is a way with which we can figure this info out, please let us know.
I also use checkov iac scanner, it reports the related lines in the tfplan
exampe:

Check: CKV_AWS_67: "Ensure CloudTrail is enabled in all Regions"
	FAILED for resource: aws_cloudtrail.default
	File: /tf.json:132-151
	Guide: https://docs.bridgecrew.io/docs/logging_1

		133 |               "values": {
		134 |                 "cloud_watch_logs_group_arn": "",
		135 |                 "cloud_watch_logs_role_arn": "",
		136 |                 "enable_log_file_validation": true,
		137 |                 "enable_logging": true,
		138 |                 "event_selector": [],
		139 |                 "include_global_service_events": false,
		140 |                 "insight_selector": [],
		141 |                 "is_multi_region_trail": false,
		142 |                 "is_organization_trail": false,
		143 |                 "kms_key_id": null,
		144 |                 "name": "eg-test-cloudtrail-test",
		145 |                 "s3_key_prefix": null,
		146 |                 "sns_topic_name": null,
		147 |                 "tags": {
		148 |                   "Name": "eg-test-cloudtrail-test",
		149 |                   "Namespace": "eg",
		150 |                   "Stage": "test"
		151 |                 }

but that's fine if terrascan does not.

2. Cloudtrail logging: This may be an issue with the policy for cloud trail logging, need to see if the same gets reported
   with the regular terrascan scanning with -i terraform.

yes in my second comment you can see the result with -i terraform and it does not report any failure which is correct. adding the same output here again.

$  terrascan scan -i terraform -t aws  .

Violation Details -
    
	Description    :	Cloud Trail Multi Region not enabled
	File           :	../../main.tf
	Line           :	1
	Severity       :	MEDIUM
	-----------------------------------------------------------------------
	

Scan Summary -

	File/Folder         :	/Users/ismail/dev/star/cloudtrail/terraform-aws-cloudtrail-0.16.0/examples/complete
	IaC Type            :	terraform
	Scanned At          :	2021-02-22 13:14:54.933818 +0000 UTC
	Policies Validated  :	159
	Violated Policies   :	1
	Low                 :	0
	Medium              :	1
	High                :	0

3. Scanning child modules: It is expected terraform plan output to have configurations for all the resources present including that of children modules as well. Is that a fair expectation?
   yes all avaiable data in tf.json file but maybe your code only check certain types of resources.
   I attached sample tf.json file for above checks. As you can see below enable_logging is true in after section. So it is enabled but terrascan can't detect that. I thought maybe code is not checking all possible parts in the json.

 {
 292       "address": "module.cloudtrail.aws_cloudtrail.default[0]",
 293       "module_address": "module.cloudtrail",
 294       "mode": "managed",
 295       "type": "aws_cloudtrail",
 296       "name": "default",
 297       "index": 0,
 298       "provider_name": "aws",
 299       "change": {
 300         "actions": [
 301           "create"
 302         ],
 303         "before": null,
 304         "after": {
 305           "cloud_watch_logs_group_arn": "",
 306           "cloud_watch_logs_role_arn": "",
 307           "enable_log_file_validation": true,
 308           "enable_logging": true,

[cesar-rodriguez]

Great job @kanchwala-yusuf !

I tried scanning @ismailyenigul's repository with both:

terrascan scan -i terraform -t aws --config-only -o json

and

terrascan scan -i tfplan -t aws -f tfplan.json --config-only -o json

and the output was similar in terms of what's given to the policy engine.

[patilpankaj212]

LGTM!!

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[patilpankaj212]

LGTM!!

[kewalaka]

Unfortunately this is no longer working because the format version is v1.1, and the existing code expects v0.1 or v0.2.

This has been raised here: #1483

Are there plans to address this, or should we assume that scanning the tfplan is an edge case and to be avoided?

[bekahmark12]

Are there any updates on the issue mentioned by @kewalaka? I am having the same problem with the outdated terraform format version. Current version is 1.1 but the format version test is only allowing for 0.1 or 0.2. This is preventing all use of terrascan for my company.