feat(quicksight): QuickSight resources and import functionality

[Disciple153]

Developing QuickSight resources with CDK is currently unnecessarily complicated due to the fact the there is no way to create new QuickSight Analyses, Dashboards, Templates, or Themes without manually gathering data about existing resources and loading them into the QuickSight CfnResources. This PR solves this complexity by allowing QuickSight Resources to be imported and then used as dependencies. This is easier than the current solution because when a CDK QuickSight Resource is used as a dependency, complex objects are automatically mapped from the SDK objects to their CDK/CFN counterparts. This significantly reduces the amount of code that developers need to write to create QuickSight resources.

Addresses #19212

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[gitpod-io]

[aws-cdk-automation]

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

[padaszewski]

+1!

[Disciple153]

I am currently getting an error in my integ test where the dummy value (undefined) is being returned, where an error should be thrown by my context provider instead. I suspect that my ContextProviders are not set up correctly, and that the default ContextProvider is being used instead, but I'm not sure how to correctly configure my ContextProviders. Any help solving this problem would be greatly appreciated.

Relevant files:

• Calls to ContextProviders:
  • packages/@aws-cdk/aws-quicksight/lib/*: .fromId().Import.()
• Integ test:
  • packages/@aws-cdk/aws-quicksight/test/integ.quicksight.ts
• ContextProviders:
  • packages/aws-cdk/lib/context-providers/quicksight/*

Error:

/home/eppercan/documents/opensource/aws-cdk/packages/@aws-cdk/aws-quicksight/lib/template.js:98
                        throw Error(`No Template found in account ${contextProps.account} and region ${contextProps.region} with id ${contextProps.templateId}`);
                              ^

Error: No Template found in account 732958832353 and region us-west-2 with id test-template
    at Import.get template [as template] (/home/eppercan/documents/opensource/aws-cdk/packages/@aws-cdk/aws-quicksight/lib/template.js:98:31)
    at Import.get templateArn [as templateArn] (/home/eppercan/documents/opensource/aws-cdk/packages/@aws-cdk/aws-quicksight/lib/template.js:157:29)
    at new Analysis (/home/eppercan/documents/opensource/aws-cdk/packages/@aws-cdk/aws-quicksight/lib/analysis.js:40:53)
    at main (/home/eppercan/documents/opensource/aws-cdk/packages/@aws-cdk/aws-quicksight/test/integ.quicksight.js:184:16)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

(The above error should have been thrown from the TemplateContextProviderPlugin, not from Template.fromId())

[TheRealAmazonKendra]

The bots are out of control... I did this. Putting up a fix ASAP.

[TheRealAmazonKendra]

Manually cancelled the linter for now.

[TheRealAmazonKendra]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated

[TheRealAmazonKendra]

Sorry for the massive spam on this. I just pushed a fix so I'll watch and make sure it doesn't start looping through the automations again.

[TheRealAmazonKendra]

Looks like I got it fixed.

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
• Commit ID: 79cd86d
• Result: FAILED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[moltar]

What happened @Disciple153? I was looking forward to this PR! :)

[Disciple153]

@moltar

What happened @Disciple153? I was looking forward to this PR! :)

It's still coming, but after talking to the CDK team, it was decided that this construct will start as a self-published resource.

I will let you know when it becomes available.

Thank you for your interest!

[moltar]

@Disciple153 Please share your repo, I am interested in collaborating. This repo is too noisy and too complicated to foster drive-by collaboration. :)

[aws-cdk-automation]

The pull request linter fails with the following errors:

❌ Features must contain a change to a README file.
❌ Features must contain a change to an integration test file and the resulting snapshot.
❌ CLI code has changed. A maintainer must run the code through the testing pipeline (git fetch origin pull/23980/head && git push -f origin FETCH_HEAD:test-main-pipeline), then add the 'pr-linter/cli-integ-tested' label when the pipeline succeeds.

PRs must pass status checks before we can provide a meaningful review.

If you would like to request an exemption from the status checks or clarification on feedback, please leave a comment on this PR containing Exemption Request and/or Clarification Request.

[OperationalFallacy]

We experienced a similar issue. With the frequent changes that data engineers want to make in the Quicksight UI, it's challenging to define Quicksight resources directly from the code.

We addressed this with a LTD (load, transform, deploy) method. We crafted functions such as prepareDataSet which accept a dataset definition and modify it to align with CfnDataSetProps.

For instance, the snippet below uses object-mapper library to map and perform simple transformations like updating account IDs and environments.

const relationalTablePropertyMap = {
  "RelationalTable.Name": "relationalTable.name",
  "RelationalTable.DataSourceArn": {
    key: "relationalTable.dataSourceArn",
    transform: function (value: string) {
      return replaceAccountAndEnv(value, awsAccount, environment);
    },
  },
  "RelationalTable.InputColumns[].Type": "relationalTable.inputColumns[].type",
  "RelationalTable.InputColumns[].Name": "relationalTable.inputColumns[].name",
};

This method proved effective for us in managing an extensive Quicksight setup. Data engineers develop via the Quicksight UI, then use a pipeline for deploying across all environments, including development.

It takes less than hour to add new datasets, mostly waiting for PR review and the pipeline to finish.

[moltar]

It's still coming, but after talking to the CDK team, it was decided that this construct will start as a self-published resource.

@Disciple153 What's the progress on this? Thanks!

[Disciple153]

@moltar Sorry to disappoint, but unfortunately I haven't been able to work on this in quite a while due to changing priorities from my manager. At this point, I would recommend pulling my PR and using a custom build of CDK. I would really like to finish this at some point, but it will take a lot more work to get this to a quality product with the introduction of the "definition" parameter in the Quicksight SDK, especially since the "definition" parameter is so obnoxious to work with.