Does the AWS Clickstream solution take into account session_id stickiness? #1041
Replies: 1 comment 5 replies
-
|
Clickstream does not support session stickiness out-of-box. This does not align with the design philosophies of the ingestion module. However, it is a high-throughput and reliable ingestion server that can handle hundreds of thousands of requests per second in our benchmark. It is SDK-agnostic and supports Clickstream SDKs, GTM, and other third-party SDKs. It's important to note that the session might not always be available in the data. Clickstream SDKs compress multiple events and then base64 encodes them by default before sending the data. However, this process may consume lots of compute resources and increase latencies if parsing the attribute of each event on the ingestion server. To process events per session in real-time, you can have a consumer process the events in KDS/MSK. This involves decoding, uncompressing, and extracting the session information of events. Once this is done, the events can be put into another downstream stream (KDS/MSK topic) for consumption by your business consumers. If you have a large volume of requests, MSK might be a cost-efficient option. If you want to customize the ingestion server, there is an option to forward events with the session serving as the partition key. If you are using Clickstream SDKs, you will also need to customize them to send events per batch in the same session. The complete source code, including the container image of the ingestion server, is available in this repository, and you can modify it to meet your specific requirements. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your extremely well put answer. I think I have identified the file to modify for adding the partition key as Presumably then, when I log into the newly deployed dashboard and create a new project/pipeline through the control, it will use new stacks with the updated configuration/modifications to deploy the new pipeline (i.e, it will use the new partition key). I'm new to cloud formation stacks and CDK, so any clarity you can provide about the re-deployment steps after making changes is appreciated. |
Beta Was this translation helpful? Give feedback.
-
|
The Clickstream solution has a critical on-click user experience to simplify the deployment. There is an in-house procedure to build and publish (using scripts in I recommend a simpler procedure for deploying the customized modules from source code.
If this is your first time deploying with AWS CDK, you must do |
Beta Was this translation helpful? Give feedback.
-
|
I used the solution web console to create a new pipeline (kinesis pipeline in this scenario). Then I ran the following command:
and all seemed to run fine until it failed to deploy due to the error shown below. Presumably I've run a correct deploy command? but not with the correct stackNameKey and stackName. This seems to re-deploy the entire pipeline and not just the ingestion server, correct? I'm guessing there is a way to only update the ingestion server specific part of the pipeline stack without redeploying the entire pipeline (which would also bypass this error that occurs when the S3 sink update is attempted). What are the correct parameters to provide to the following command in order to only redeploy the ingestion server?
It's probably something like the following but I'm not sure what stackNameKey should be used deploy nested ingestion server stack correctly (given that I'm assuming that the correct stack to modify has the following name pattern: Clickstream-Ingestion-kinesis-XXXXXXXXXXXX-IngestionServerK2C0100NestedStackIngestion-XXXXXXXXXXXX )
EDIT: Does there exist an obvious place in the repository with a list of the stack name keys that can be used? Here is the error when re-deploying the entire pipeline stack for reference: `12:05:39 PM | CREATE_FAILED | AWS::Lambda::EventSourceMapping | kinesisDataStreamT...rovisioned1418BC98 12:05:39 PM | CREATE_FAILED | AWS::Lambda::EventSourceMapping | kinesisDataStreamT...rovisioned1418BC98 12:05:42 PM | UPDATE_FAILED | AWS::CloudFormation::Stack | kinesisDataStreamT...ckResourceA3CE46B9 ❌ Clickstream-Ingestion-kinesis-766d3e26c06041919a1eae8b14117961 failed: Error: The stack named Clickstream-Ingestion-kinesis-766d3e26c06041919a1eae8b14117961 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The event source arn (" arn:aws:kinesis:eu-west-1:714462417551:stream/Clickstream_clickstream_collector_ehtc_c5c361d0 ") and function (" Clickstream-Ingestion-kin-kinesisToS3LambdaB660939-405lqf9IM29V ") provided mapping already exists. Please update or delete the existing mapping with UUID fa5ca6de-b52e-4283-92bb-1dda46802a97 (Service: Lambda, Status Code: 409, Request ID: 294c2699-82e9-47db-8763-c5158a0511a6)" (RequestToken: d9309714-4fed-f75e-e3be-019a1d23cb8d, HandlerErrorCode: AlreadyExists), Embedded stack arn:aws:cloudformation:eu-west-1:714462417551:stack/Clickstream-Ingestion-kinesis-766d3e26c0604-kinesisDataStreamToS3ProvisionedNestedStac-15LYIZ5YR372P/c5c361d0-f7ed-11ee-90dc-021161952dff was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [kinesisToS3LambdaKinesisEventSourceClickstreamIngestionkinesis766d3e26c06041919a1eae8b14117961kinesisDataStreamToS3ProvisionedkinesisStreamProvisioned1418BC98CEBA1101]. The following resource(s) failed to update: [kinesisDataStreamMetricsCustomResource]. ❌ Deployment failed: Error: The stack named Clickstream-Ingestion-kinesis-766d3e26c06041919a1eae8b14117961 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The event source arn (" arn:aws:kinesis:eu-west-1:714462417551:stream/Clickstream_clickstream_collector_ehtc_c5c361d0 ") and function (" Clickstream-Ingestion-kin-kinesisToS3LambdaB660939-405lqf9IM29V ") provided mapping already exists. Please update or delete the existing mapping with UUID fa5ca6de-b52e-4283-92bb-1dda46802a97 (Service: Lambda, Status Code: 409, Request ID: 294c2699-82e9-47db-8763-c5158a0511a6)" (RequestToken: d9309714-4fed-f75e-e3be-019a1d23cb8d, HandlerErrorCode: AlreadyExists), Embedded stack arn:aws:cloudformation:eu-west-1:714462417551:stack/Clickstream-Ingestion-kinesis-766d3e26c0604-kinesisDataStreamToS3ProvisionedNestedStac-15LYIZ5YR372P/c5c361d0-f7ed-11ee-90dc-021161952dff was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [kinesisToS3LambdaKinesisEventSourceClickstreamIngestionkinesis766d3e26c06041919a1eae8b14117961kinesisDataStreamToS3ProvisionedkinesisStreamProvisioned1418BC98CEBA1101]. The following resource(s) failed to update: [kinesisDataStreamMetricsCustomResource]. The stack named Clickstream-Ingestion-kinesis-766d3e26c06041919a1eae8b14117961 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The event source arn (" arn:aws:kinesis:eu-west-1:714462417551:stream/Clickstream_clickstream_collector_ehtc_c5c361d0 ") and function (" Clickstream-Ingestion-kin-kinesisToS3LambdaB660939-405lqf9IM29V ") provided mapping already exists. Please update or delete the existing mapping with UUID fa5ca6de-b52e-4283-92bb-1dda46802a97 (Service: Lambda, Status Code: 409, Request ID: 294c2699-82e9-47db-8763-c5158a0511a6)" (RequestToken: d9309714-4fed-f75e-e3be-019a1d23cb8d, HandlerErrorCode: AlreadyExists), Embedded stack arn:aws:cloudformation:eu-west-1:714462417551:stack/Clickstream-Ingestion-kinesis-766d3e26c0604-kinesisDataStreamToS3ProvisionedNestedStac-15LYIZ5YR372P/c5c361d0-f7ed-11ee-90dc-021161952dff was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to create: [kinesisToS3LambdaKinesisEventSourceClickstreamIngestionkinesis766d3e26c06041919a1eae8b14117961kinesisDataStreamToS3ProvisionedkinesisStreamProvisioned1418BC98CEBA1101]. The following resource(s) failed to update: [kinesisDataStreamMetricsCustomResource]. ` |
Beta Was this translation helpful? Give feedback.
-
|
EDIT 2: It seems like doing the re-deploy to update the stack that exists already does not work as expected (resources already existing) Likely more errors will occur if I can even find a hack to fix this new error. It seems like I will have to do a from-scratch deployment of a brand new pipeline using the CDK, with the huge list of parameters using How do I find the correct stack name to put in this command and the parameter list? |
Beta Was this translation helpful? Give feedback.
-
|
Adam, you used the script correctly. But the failure was caused by the logical ID of some resources changing after synthesizing the stack with a different name. There is a workaround for it.
bash e2e-deploy.sh -n ingestToKinesisStackName -s <stack arn>NOTE: the script has different parameters when using the latest code |
Beta Was this translation helpful? Give feedback.
-
I have followed the implementation guide for the AWS clickstream solution, currently using Kinesis on demand, sinking to an S3 bucket (just following the steps in the setup). I'm not bothered with setting up data processing and analytics dashboards. I simply want to use the data ingestion module and consume from the stream (MSK or Kinesis) using custom built processing consumers.
Requirements: I need to spin up a fleet of consumers as I potentially will be dealing with huge volumes of data that requires parallelised consumption and processing of the data in the stream. All events that belong to a single session (example '_session_id' ) MUST end up in the same consumer.
If the events get randomly passed into different partitions or shards, then the session becomes split across multiple consumers which completely breaks my processing of the data stream. I know kinesis and Kafka support partition keys so that data arriving into the streams with the same partition key (ideally a session ID of some kind) are ensured to end up in the same partition and therefore end up in the same consumer. The documentation is not clear about how incoming events sent from the client application within a single "_session_id" (or some other identifier of a users events) are split up across partitions within within the data stream.
The clickstream solution seems designed to dump the data into an S3 bucket which only fine if I wasn't doing real-time processing of my data. I could simply sort my S3 data into sessions and do batch processing then, but this is not what I want.
Are there any experts on the AWS clickstream solution that can tell me if partition keys are taken into account in order to sort the session's events into partitions/shards? Or is the ingestion module designed to just dump all incoming data randomly into whatever partition/shard it wants? Session stickiness across partitions is an absolute requirement of my project.
EDIT: Maybe another related question is: If it does not partition the data by a session_id or other similar ID, is it possible to modify the solution to customise it? I'm guessing it would be clone the AWS clickstream analytics repo -> modify the source code for the vector server (the configuration toml files I'm assuming) to partition by session_id -> re-bootstrap the cdk, re-deploy the stack with the modified vector server configuration? Is this possible? Or are there components that the solution pulls from image repositories that are not created by the CDK and therefore non-modifiable? In other words, is the entire solution completely customisable by forking and modifying the GitHub repo?
Thanks.
Beta Was this translation helpful? Give feedback.
All reactions