Feature Request: Kinesis Data Analytics StartApplication

[arturoherrero]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

After creating a Kinesis Data Analytics application, we must call StartApplication operation to start the application. After the application starts, it begins consuming the input data, processes it, and writes the output to the configured destination.

https://docs.aws.amazon.com/kinesisanalytics/latest/dev/API_StartApplication.html

At the moment, there's no functionality to start the Kinesis Data Analytics application. It's necessary to do it from the UI:

New or Affected Resource(s)

• aws_kinesis_analytics_application

References

• r/kinesis_analytics_application: new resource #5456 (comment)

[kl4w]

Been reading the docs and trying to see how this could be done and I'm not sure if this resource is the right place. Looking for thoughts around possible implementation, or if this approach even makes sense and maybe it's better to keep start/stop actions outside of this resource.

The StartApplication action has a few required parameters, one of those being InputId which is auto-generated by AWS so I managing the versions of inputs may be difficult.

I also believe that we want to try and conform the creation schema to the resulting structure of DescribeApplication action, so I believe creating a running application would possibly have to look like below.

resource aws_kinesis_analytics_application test {
  name   = "test"
  code   = "testCode\n"
  status = "RUNNING"

  inputs = {
    name_prefix = "test_prefix"
    version = "1.1"

    kinesis_stream = {
      resource_arn = "${aws_kinesis_stream.test.arn}"
      role_arn     = "${aws_iam_role.test.arn}"
    }

    parallelism = {
      count = 2
    }

    schema = {
      record_columns = {
        mapping  = "$.test"
        name     = "test"
        sql_type = "VARCHAR(10)"
      }

      record_encoding = "UTF-8"

      record_format = {
        mapping_parameters = {
          csv = {
            record_column_delimiter = ","
            record_row_delimiter    = "\n"
          }
        }
      }
    }

    starting_position_configuration = {
      starting_position = "TRIM_HORIZON"
    }
}

[daugustus]

This is a feature we could use as well. Our workaround includes a series of aws cli commands within our Jenkins deployment pipeline:

stage('Start KAA'){
      // assumes application is not in the "STOPPING" state
      def kaa_current_status = ""
      def kaa_input_id = ""
      def kaa_starting = ""
      def app_running = false
      def stop_counter = 0
      def sleep_period = 5  // seconds
      // loop until app is started or we exceed the time limit
      while ( (! app_running) && (stop_counter < 24) ) {
        println("Checking Kinesis Analytics Application Status")
        // determine current status
        kaa_current_status = sh(
          script: "aws --region ${AWSRegion} kinesisanalytics describe-application --application-name sls-sacoreprocessor | jq '.ApplicationDetail.ApplicationStatus' -r",
          returnStdout: true
        ).trim()
        // loop until app is started
        switch (kaa_current_status.toString().toUpperCase()) {
          case 'READY':
              println("App is Ready, sending Start command")
              // get input Id
              kaa_input_id = sh(
                script: "aws --region ${AWSRegion} kinesisanalytics describe-application --application-name sls-sacoreprocessor | jq '.ApplicationDetail.InputDescriptions[0].InputId' -r",
                returnStdout: true
              ).trim()
              kaa_starting = sh(
              script: "aws --region ${AWSRegion} kinesisanalytics start-application --application-name sls-sacoreprocessor  --input-configurations Id=${kaa_input_id},InputStartingPositionConfiguration={InputStartingPosition=LAST_STOPPED_POINT}",
              returnStdout: true
              ).trim()
              break
          case ['STARTING', 'UPDATING']:
            println("App Status: ${kaa_current_status}, sleeping")
            sleep sleep_period
            break
          case 'RUNNING':
            println("App took " + (stop_counter * sleep_period) + " seconds to start")
            println("App Status: ${kaa_current_status}, finished")
            app_running = true
            break
          default:
            println("App Status: ${kaa_current_status}, aborting!")
            stop_counter = 99
            break;
        }  // end of switch statement
        stop_counter++
      } // end of the while loop
    } // end of stage

[ewbankkit]

I intend to add a start_application attribute, inspired by the aws_synthetics_canary start_canary attribute.

[ghost]

This has been released in version 3.32.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!