feat(msk): Add Kafka version and support for tiered storage mode

[chrispidcock]

---

This patch adds support for the tiered storage mode and Kafka versions 2.8.2.tiered & 3.5.1

Notes:

• Need to update the snapshots still, but this is proving to be difficult currently.

---

---

Current testing, is as follows. I have been unable to test deploy to aws.. yet.

Current state:

npx lerna run test --scope=@aws-cdk/aws-msk-alpha
lerna notice cli v7.3.0
lerna notice filter including "@aws-cdk/aws-msk-alpha"
lerna info filter [ '@aws-cdk/aws-msk-alpha' ]

   ⠸    Nx is waiting on 8 dependent project tasks before running tasks from @aws-cdk/aws-msk-alpha...

   ✔    14/14 dependent project tasks succeeded [14 read from cache]
The following outputs are invalid:
 - !{projectRoot}/node_modules/**/*

   ⠼    Nx is waiting on 7 dependent project tasks before running tasks from @aws-cdk/aws-msk-alpha...


   ✔    22/22 dependent project tasks succeeded [22 read from cache]

   Hint: you can run the command with --verbose to see the full dependent project outputs

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————


> @aws-cdk/aws-msk-alpha:build  [existing outputs match the cache, left as is]

yarn run v1.22.19
$ cdk-build
warning JSII6: A "peerDependency" on "constructs" at "^10.0.0" means you should take a "devDependency" on "constructs" at "10.0.0" (found "^10.0.0")
lib/cluster-version.ts:128:3 - warning JSII7000: Documentation for method "isTieredStorageCompatible" refers to non-existent @param "kafkaVersion"

128   public static isTieredStorageCompatible(version: KafkaVersion | string) {
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
129     if (version instanceof KafkaVersion) {
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... 
133     return version.indexOf(suffix, version.length - suffix.length) !== -1;
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
134   };
    ~~~
Build times for @aws-cdk/aws-msk-alpha: Total time (33.0s) | /home/piddy/github/aws-cdk/tools/@aws-cdk/cdk-build-tools/node_modules/jsii/bin/jsii (25.5s) | /home/piddy/github/aws-cdk/node_modules/eslint/bin/eslint.js (6.8s) | /home/piddy/github/aws-cdk/tools/@aws-cdk/cdk-build-tools/bin/cdk-awslint (0.5s) | '/home/piddy/.nvm/versions/node/v18.18.1/bin/node' (0.1s) | pkglint (0.1s)
Done in 33.07s.

> @aws-cdk/aws-msk-alpha:test

> @aws-cdk/aws-msk-alpha@0.0.0 test
> cdk-test
 FAIL  test/cluster.test.ts
  MSK Cluster
    ✓ created with expected Kafka version {"version":"1.1.1"} (280 ms)
    ✓ created with expected Kafka version {"version":"2.2.1"} (30 ms)
    ✓ created with expected Kafka version {"version":"2.3.1"} (24 ms)
    ✓ created with expected Kafka version {"version":"2.4.1.1"} (20 ms)
    ✓ created with expected Kafka version {"version":"2.5.1"} (18 ms)
    ✓ created with expected Kafka version {"version":"2.6.0"} (16 ms)
    ✓ created with expected Kafka version {"version":"2.6.1"} (18 ms)
    ✓ created with expected Kafka version {"version":"2.6.2"} (15 ms)
    ✓ created with expected Kafka version {"version":"2.6.3"} (17 ms)
    ✓ created with expected Kafka version {"version":"2.7.0"} (15 ms)
    ✓ created with expected Kafka version {"version":"2.7.1"} (14 ms)
    ✓ created with expected Kafka version {"version":"2.7.2"} (15 ms)
    ✓ created with expected Kafka version {"version":"2.8.0"} (14 ms)
    ✓ created with expected Kafka version {"version":"2.8.1"} (15 ms)
    ✓ created with expected Kafka version {"version":"2.8.2.tiered"} (21 ms)
    ✓ created with expected Kafka version {"version":"3.1.1"} (27 ms)
    ✓ created with expected Kafka version {"version":"3.2.0"} (13 ms)
    ✓ created with expected Kafka version {"version":"3.3.1"} (13 ms)
    ✓ created with expected Kafka version {"version":"3.3.2"} (13 ms)
    ✓ created with expected Kafka version {"version":"3.4.0"} (13 ms)
    ✓ created with expected Kafka version {"version":"3.5.1"} (16 ms)
    ✓ created with default properties (16 ms)
    ✓ prefixes instance type with "kafka" (15 ms)
    ✓ create an encrypted cluster with a custom KMS key (14 ms)
    ✕ Snapshot test with all values set (19 ms)
    created with authentication enabled
      ✓ fails if more than one authentication method is enabled (2 ms)
      with tls auth
        ✓ fails if client broker encryption is set to plaintext (34 ms)
      with sasl/scram auth
        ✓ fails if tls encryption is set to plaintext (2 ms)
        ✓ fails if tls encryption is set to tls and plaintext (3 ms)
      with sasl/iam auth
        ✓ iam enabled is true (15 ms)
        ✓ fails if tls encryption is set to plaintext (2 ms)
        ✓ fails if tls encryption is set to tls and plaintext (2 ms)
      with sasl/iam auth and tls
        ✕ Snapshot test with all values set (iam/sasl) (26 ms)
      creates a customer master key
        ✓ with alias msk/${clusterName}/sasl/scram (19 ms)
        ✓ with a policy allowing the secrets manager service to use the key (18 ms)
    created with an instance type set
      ✓ prefixes instance type with "kafka" (15 ms)
    created with logging enabled
      ✓ log group is set (15 ms)
      ✓ s3 bucket is set (18 ms)
      ✓ feature flag @aws-cdk/aws-s3:defaultBucketPolicy (24 ms)
      ✓ firehose delivery stream is set (14 ms)
    ebs volume size is within bounds
      ✓ exceeds max (2 ms)
      ✓ below minimum (2 ms)
    importing an existing cluster with an ARN
      ✓ cluster name is set (1 ms)
      ✓ cluster arn is set (2 ms)
    when creating sasl/scram users
      ✓ fails if sasl/scram not enabled (2 ms)
      ✓ creates a secret with the secret name prefixed with AmazonMSK_ (114 ms)
    created with storage mode
      with tiered storage mode
        ✓ fails if incompatiable Kafka version (3 ms)
        ✓ fails if instance type of t3.small (2 ms)
        ✓ create a cluster with tiered storage mode (18 ms)
      with local storage mode
        ✓ create a cluster with local storage mode (14 ms)
  ● MSK Cluster › created with authentication enabled › with sasl/iam auth and tls › Snapshot test with all values set (iam/sasl)
    expect(received).toMatchSnapshot()
    Snapshot name: `MSK Cluster created with authentication enabled with sasl/iam auth and tls Snapshot test with all values set (iam/sasl) 1`
    - Snapshot  - 0
    + Received  + 1
    @@ -463,10 +463,11 @@
                  "NodeExporter": {
                    "EnabledInBroker": true,
                  },
                },
              },
    +         "StorageMode": "LOCAL",
            },
            "Type": "AWS::MSK::Cluster",
            "UpdateReplacePolicy": "Retain",
          },
          "sg1fromsg32181E6F4C07E": {

      283 |
      284 |         // THEN
    > 285 |         expect(Template.fromStack(stack)).toMatchSnapshot();
          |                                           ^
      286 |       });
      287 |     });
      288 |

      at Object.<anonymous> (test/cluster.test.ts:285:43)
  ● MSK Cluster › Snapshot test with all values set
    expect(received).toMatchSnapshot()
    Snapshot name: `MSK Cluster Snapshot test with all values set 1`
    - Snapshot  - 0
    + Received  + 1
    @@ -458,10 +458,11 @@
                  "NodeExporter": {
                    "EnabledInBroker": true,
                  },
                },
              },
    +         "StorageMode": "LOCAL",
            },
            "Type": "AWS::MSK::Cluster",
            "UpdateReplacePolicy": "Retain",
          },
          "sg1fromsg32181E6F4C07E": {

      735 |
      736 |     // THEN
    > 737 |     expect(Template.fromStack(stack)).toMatchSnapshot();
          |                                       ^
      738 |   });
      739 |
      740 |   describe('when creating sasl/scram users', () => {

      at Object.<anonymous> (test/cluster.test.ts:737:39)
 › 2 snapshots failed.
=============================== Coverage summary ===============================
Statements   : 88.88% ( 120/135 )
Branches     : 89.04% ( 65/73 )
Functions    : 72.41% ( 21/29 )
Lines        : 88.88% ( 120/135 )
================================================================================
Snapshot Summary
 › 2 snapshots failed from 1 test suite. Inspect your code changes or re-run jest with `-u` to update them.
Test Suites: 1 failed, 1 total
Tests:       2 failed, 48 passed, 50 total
Snapshots:   2 failed, 2 total
Time:        2.1 s
Ran all test suites.
Error: /home/piddy/github/aws-cdk/node_modules/jest/bin/jest.js exited with error code 1
Tests failed. Total time (2.5s) | /home/piddy/github/aws-cdk/node_modules/jest/bin/jest.js (2.5s)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
npm ERR! Lifecycle script `test` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: @aws-cdk/aws-msk-alpha@0.0.0 
npm ERR!   at location: /home/piddy/github/aws-cdk/packages/@aws-cdk/aws-msk-alpha 

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  Lerna (powered by Nx)   Ran target test for project @aws-cdk/aws-msk-alpha and 23 task(s) they depend on (5s)
 
    ✖    1/24 failed
    ✔    23/24 succeeded [23 read from cache]

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license