Support for DAX DDB operations. Scan, query, etc

[paul-uz]

I am trying to test out DAX and it seems that the paginateScan from client-dynamodb does not support the DAXClient as the client parameter.

Is this simply due to the fact DAX doesn't need to scan recursively to get all the results? Or is it a case of it hasn;t be implemented yet?

[paul-uz]

Actually, it seems the client-dax package only deals with DAX administration tasks, and you cannot use it at all to perform a scan, query, get, etc

Do I need to use the old v2 package https://www.npmjs.com/package/amazon-dax-client? If so that sucks, because I'm using the new v3 SDK...

[AllanZhengYP]

Hi @paul-uz,

I have opened a internal feature request to the DAX service team. I will share more progress here if we know more.

[gsingh1]

Hi all, is there any update on this @AllanZhengYP? This is really needed now that v3 is becoming more and more mainstream. Thanks.

[paul-uz]

@gsingh1 with the speed at which features are implemented at AWS, I wouldn't hold your breath :(

[mdusafp]

I upset to know that AWS doesn't support that functionality in v3 😭

[shobbsd]

Hi @AllanZhengYP is it possible to get an update on the state of this feature request?

[dzou]

We need this! 🙏

[paul-uz]

@AllanZhengYP close to a year now, and no movement?

[dzou]

🥺

[tabvn]

@aws team ?

[valencar]

wow, hoping to get some attention here from the AWS team.

[li-sergekuranda]

Hi everyone ... we need also this feature. If we can't use DAX with SDK v3, what is the solution ? we don't want to rollback to sdk v2. 😭

[paul-uz]

Paging @AllanZhengYP

This is getting pretty ridiculous now. SDK v3 has been out for ages, and this functionality still hasn't been addressed.

[poloroid]

Hi @AllanZhengYP @kellertk
=> Saddened by the lack of response, what's taking so long? 😞
Is DAX being deprecated or something I'm unaware of?

[kellertk]

Hi!

For those curious, this is why there's been a delay on this. The DynamoDB DAX client is an alternative to the AWS JavaScript SDK DDB client and is published by the DynamoDB team directly. The DynamoDB folks have published this client to work with v2 of the JavaScript SDK, but not with v3 yet. They're completing this work independently of the teams that publish the AWS SDKs, which is why you haven't seen direct communication from that team here.

The SDK team doesn't presently plan to develop a DAX client for DDB in the SDK directly; we will continue to rely on our parters on the DDB team internally to provide this high level client. I've raised the question on v3 support internally to the DDB team again and I'll post here when I hear more from them.

[paul-uz]

Hi!

For those curious, this is why there's been a delay on this. The DynamoDB DAX client is an alternative to the AWS JavaScript SDK DDB client and is published by the DynamoDB team directly. The DynamoDB folks have published this client to work with v2 of the JavaScript SDK, but not with v3 yet. They're completing this work independently of the teams that publish the AWS SDKs, which is why you haven't seen direct communication from that team here.

The SDK team doesn't presently plan to develop a DAX client for DDB in the SDK directly; we will continue to rely on our parters on the DDB team internally to provide this high level client. I've raised the question on v3 support internally to the DDB team again and I'll post here when I hear more from them.

A delay? When was the v3 SDK released?

Why exactly haven't they implemented it yet? It's been a year since I reported this and I was late to the v3 SDK party so its longer than that.

Sounds like they don't have a real answer and have realised they done goofed.

[kellertk]

A delay? When was the v3 SDK released?

2020-12-15.

Why exactly haven't they implemented it yet? It's been a year since I reported this and I was late to the v3 SDK party so its longer than that.

I'm unsure right now, but as I said I've raised the question with the team that publishes the DAX client (which is also the team that develops the DynamoDB service). Right now I don't believe that team has any immediate plans to support v3 in the DAX client, but I'm waiting for confirmation on that point.

In the meantime, a workaround is to use the v2 DDB client in your application, although I appreciate that this isn't an elegant solution.

[paul-uz]

A delay? When was the v3 SDK released?

2020-12-15.

Why exactly haven't they implemented it yet? It's been a year since I reported this and I was late to the v3 SDK party so its longer than that.

I'm unsure right now, but as I said I've raised the question with the team that publishes the DAX client (which is also the team that develops the DynamoDB service). Right now I don't believe that team has any immediate plans to support v3 in the DAX client, but I'm waiting for confirmation on that point.

In the meantime, a workaround is to use the v2 DDB client in your application, although I appreciate that this isn't an elegant solution.

Using v2 is not a solution at all.

Why are there no plans to support DAX with v3? This seems like a huge oversight and is frankly ridiculous.

With the release of Node 18 for Lambda, which even bundles the v3 SDK, the DAX library should be compatible!

2 years and no movement. Typical AWS levels of service.

[kellertk]

I've confirmed with the DynamoDB team that they have no plans to support new major SDK versions or other AWS SDKs in the immediate future.

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[paul-uz]

I've confirmed with the DynamoDB team that they have no plans to support new major SDK versions or other AWS SDKs in the immediate future.

So is DAX being binned off?

Why won't they support DAX with the v3 SDK?

[valencar]

It looks like we need DAX alternatives, I am glad our systems didn't make it to production yet, so we can remove it.

[paul-uz]

@kellertk any updates on this? It's been 9 months since the DDB said no, but I'm hoping there.migjt now be something on the horizon as it seems quite ridiculous that DAX has effectively been locked out for anyone using NodeJS and the v3 SDK which should be the go to SDK now.

[jose-garrido-martinez]

+1, please support it's needed for this. It's crazy after 3 years from realising SDK v3, there are no news about DAX Support

[eerenwong]

Hi, any update on the timeline for this ?

[paul-uz]

@eerenwong take the silence as a "no"

[joe455]

Any update on this?

[rbezant-drtavel]

Not supporting DAX in the v3 SDK is ridiculous. Why would the team that develops DynamoDB (which has sufficient support in the v3 SDK), who also develops DAX, not add this functionality? I am absolutely not going mix v2 with v3 just for DAX, and would rather go without. I'm sure many potential DAX users share this sentiment.

Please reopen this issue and also consider adding native DAX support to client-dynamodb.

[khosalan]

Is there any way to use DAX with V3 SDK?

[paul-uz]

Is there any way to use DAX with V3 SDK?

if you'd have read this thread, you'd know, no there isn't. Hence the issue, and everyone kicking off about it.

[pkpp1233]

Any update on this?

[dforsber]

I was going to use DAX, but bumped into this. So, not using it.

[paul-uz]

3 years since I reported this, obviously longer since the V3 SDK was released, and still nothing. Terrific.

[dforsber]

I switched to Elasticache Memcached. But I guess the "normal" evolution path is to switch to Redis at some point. But for me, not yet :).

[ronkoaveone]

Like others, I am simply. stunned. at this.

Going back to AWS SDK v2 is nonsensical. This also is dissuading our entire use of DAX, which until now I was ramping for important business deliverables. We badly need, request, support for DynamoDB DAX via the AWS SDK v3.

[kwojcicki]

I've managed to get the DAX client library (atleast the functionality I've used DAX for) to work with sdkv3. published it to https://www.npmjs.com/package/amazon-dax-client-sdkv3. Constructor interface is slightly different but the actual usage should remain the same.

[paul-uz]

I've managed to get the DAX client library (atleast the functionality I've used DAX for) to work with sdkv3. published it to https://www.npmjs.com/package/amazon-dax-client-sdkv3. Constructor interface is slightly different but the actual usage should remain the same.

Holy moly!

AWS; pay this guy.

[daveharig]

Barvo @kwojcicki

It's crazy you have the bandwith and skill to build this client when the AWS service teams could not do it in 3+ years.

[hari-houdini]

I've managed to get the DAX client library (atleast the functionality I've used DAX for) to work with sdkv3. published it to https://www.npmjs.com/package/amazon-dax-client-sdkv3. Constructor interface is slightly different but the actual usage should remain the same.

Thanks a ton @kwojcicki really really needed this

[yuatpocketgems]

I tried to use the adapted V3 client, but run into a bunch of issues wrapping it inside a DynamoDBDocumentClient. A simpler solution would be a proxy around the official DAX client like this:

  const { DynamoDB } = require('@aws-sdk/client-dynamodb')
  const AwsDaxClient = require('amazon-dax-client')

  const dbClient = new DynamoDB({ endpoint: '', region: '' })
  const daxClient = new AwsDaxClient({
    endpoints: [process.env.DAX_ENDPOINT]
  })

  // The goal is to make daxClient V2 API compatible with dbClient V3, 
  // so DaynmoDBDocument can consume daxClient
  // The primary differences are
  // a) promise() no longer exist, so we ought to call it in proxy
  // b) some new properties are expected by the V3 clients
  const daxV3 = new Proxy(daxClient, {
    get: (target, prop) => {
      if (typeof target[prop] === 'function') {
        return (...args) => target[prop](...args).promise()
      }
      // V3 clients added config and middlewareStack properties, 
      // re-use those from DynamoDB instance
      return dbClient[prop]
    }
  })
  const v3ClientWithDax = DynamoDBDocument.from(daxV3, { marshallOptions })

I've tested this proxy with get, put, update, delete, batchGet, transactGet, transactWrite, query and scan.

If you don't want to manage this proxy yourself, get this client through our @pocketgems/dynamodb library (supports schema and transaction) like this:

const db = require('@pocketgems/dynamodb')
const { dbClient, daxClient: documentClientWithDax, documentClient: : docuementClientWithoutDax } = db.Model
// or
const { dbClient, daxClient: documentClientWithDax, documentClient: : documentClientWithoutDax } = db.Transaction

[kwojcicki]

but run into a bunch of issues

Hey @yuatpocketgems feel free to make a issue against https://github.com/kwojcicki/amazon-dax-client-v3/issues describing your issues.

but run into a bunch of issues wrapping it inside a DynamoDBDocumentClient

DAX is meant to be the wrapper, you shouldn't pass a DAX client to DynamoDBDocumentClient. You can see an example here

const documentDaxClient = new AmazonDaxClient({
    client: DynamoDBDocumentClient.from(new DynamoDBClient({
        endpoint: process.env.dax,
        region: 'us-east-2'
    }))
});

That should just work.

A simpler solution would be a proxy around the official DAX client like this:

The reason folks don't want to use the official DAX client is because it does not utilize SDKv3, that means if you are using AWS Lambda (for example) you need to bundle in the entire SDKV2 (which doesn't support tree shaking super well), meaning you are likely to exceed the 250MB zip file limit.

Which is exactly what happened to myself which is why I upgraded the library.

You can see this by checking out your own project (https://github.com/pocketgems/dynamodb).

git checkout https://github.com/pocketgems/dynamodb.git
cd dynamodb
npm install
grep \"aws-sdk\" ./package-lock.json -A 2

Which outputs

        "aws-sdk": ">=2.365.0",
        "bignumber.js": "^3.0.0",
        "jmespath": "^0.15.0",
--
        "aws-sdk": ">=2.365.0",
        "bignumber.js": "^3.0.0",
        "jmespath": "^0.15.0",
--
    "aws-sdk": {
      "version": "2.1669.0",
      "resolved": "https://registry.npmjs.org/aws-sdk/-/aws-sdk-2.1669.0.tgz",

i.e, your library still uses aws-sdk-v2.

With that being said, your proxy idea is really really cool and if you are okay with it, I'd like to include it as an example in the v3 ports readme if you're okay with that 😄

[yuatpocketgems]

OFC, feel free to use the code.

The official DAX client is supposed to be passed as service into DynamoDBDocumentClient, instead of passing DynamoDBCLient as service. The APIs on the document client are put, delete, transactGet etc, while the APIs on the adapted DAX client and DynamoDBClient are putItem, deleteItem, transactGetItems. The parameter you pass to the APIs are also different for the 2 sets of APIs. For us, we want to use the more succinct put, batchGet, delete (originally for the ease of consumption but now) mainly to minimize code changes for this upgrade. Originally, the document client worked as a translation layer on top, and since the adapted version of DAX client had APIs similar to DynamoDBClient, I tried to nest it under another DynamoDBDocumentClient, which gave me stream.pipe is not a function error. Not sure if that is an intended use case for the fork, if so, I'd happily open an issue there.

What we are trying to achieve with this Proxy is that we can use aws-sdk v3 in our application code everywhere, and not getting forced to also depend on aws-sdk v2 just for the document client. Meanwhile, a sub-dependency on aws-sdk may not always be under our control, e.g. if we used yet another package that hasn't been updated to sdk v3.

Lambda's 250mb limit is unfortunate, though size isn't a problem for us, since we are using ECS containers to host our backend. I think the limit on lambda only applied to the .zip way of provisioning functions (container way is fine I think).

... now that I think about it, since lambda function still come with aws-sdk pre-installed (to my recollection), you can override the subdependency through package.json so node_modules don't contain it when you package the function, like this

"overrides": { "aws-sdk": "npm:dry-uninstall" }

And if lambda stops shipping v2 sdk with the environment, you might be able to pull something like this off too:

// For demo only, most likely won't work off the shelf
// package.json
"overrides": { "aws-sdk": "file:src/aws-v3-to-v2-proxy" }

// aws-v3-to-v2-proxy.js
const DynamoDBClient = require("@aws-sdk/client-dynamodb")

class MyProxy {
  constructor () {
    return new Proxy(new DynamoDBClient(), {
      get: (target, prop) => {
        return (...args) => {
          const promise = target[prop](...args)
          return { promise: () => promise }
        }
      }
    }) 
  }
}

module.exports = {
  AWS: { DynamoDB: MyProxy }
}

but at this point, it's probably better to do things the right (hard) way.

[kwojcicki]

now that I think about it, since lambda function still come with aws-sdk pre-installed (to my recollection)

The default runtime (nodejs 18) and above ship with sdkv3 by default. So you are forced to package the entire sdkv2 (with some attempt at tree shaking) if you want to use the official dax client.

Originally, the document client worked as a translation layer on top, and since the adapted version of DAX client had APIs similar to DynamoDBClient, I tried to nest it under another DynamoDBDocumentClient, which gave me stream.pipe is not a function error. Not sure if that is an intended use case for the fork, if so, I'd happily open an issue there.

Not really no, I can't remember why I changed the constructor interface to what I did but that's how its gonna stay xD.

[daveharig]

@AllanZhengYP Please let us know if you can weigh in on this. Not so much on/if AWS will get around to making an official V3 DAX client, but rather the architecture/approach you would take if you did? Proxy vs. wrapper etc. Thanks.