Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[lambda] specify multiple folders as a single lambda asset #984

Open
efimk-lu opened this issue Oct 22, 2018 · 26 comments
Open

[lambda] specify multiple folders as a single lambda asset #984

efimk-lu opened this issue Oct 22, 2018 · 26 comments
Labels
@aws-cdk/aws-lambda Related to AWS Lambda effort/small Small work item – less than a day of effort feature/enhancement A new API to make things easier or more intuitive. A catch-all for general feature requests. feature-request A feature should be added or improved. p2

Comments

@efimk-lu
Copy link

Hello,
Great framework, keep up the good work.

Right now in order to define a lambda function you need to set where the code resides, e.g. code: lambda.Code.asset('lambda'). Sometimes you would like to define multiple folders, for example the actual code and node_modules, although packing can be done as a preprocess step before actually running cdk deploy, it would be cleaner to do it as part of the lambda configuration.

@rix0rrr
Copy link
Contributor

rix0rrr commented Oct 22, 2018

Wouldn't you pick the single directory that contains both your code and the node_modules directory, like this?

.
|-- my_code.ts
`-- node_modules
    `-- some_library

@efimk-lu
Copy link
Author

I've been thinking on the following structure

/
... package.json
... Various conf file
... src
...... my_code.js
... test
...... test_my_code.js
... node_modules
... cdk
...... deploy.js

@eladb
Copy link
Contributor

eladb commented Oct 23, 2018

I think that's definitely an interesting idea. One would also need to map the sources to their location in the destination tree (for example, you would probably want src/my_code.js in the root of the destination tree and node_modules next to it, right?

@efimk-lu
Copy link
Author

@eladb right

@rix0rrr rix0rrr added enhancement package/tools Related to AWS CDK Tools or CLI labels Nov 6, 2018
@debora-ito debora-ito added the @aws-cdk/aws-lambda Related to AWS Lambda label Nov 8, 2018
@justinmchase
Copy link
Contributor

I have a mono-repo and so I can't just include the root of the project. I use tsc to compile code from src/ -> dist/, from ts to js. My node_modules exists at the root of the project and I don't want to include all dependencies even.

Something like this would be great:

    const example = new Function(this, 'Example'), {
      runtime: Runtime.NodeJS810,
      handler: 'index.handler',
      code: Code.files([
        'dist/funcs/example/*',
        'node_modules/*'
        '!node_modules/aws-sdk'
    });

Similar to the way gulp would let you include src files.

Right now I'm looking at adding a package.json into my func folder and running npm i --production but its a bit of a pain, I don't have any mechanism for copying non-code files currently.

@eladb
Copy link
Contributor

eladb commented Dec 9, 2018

I can see the value in this, and I think we can support something like that, but I was wondering if this is actually the best approach for bundling node.js dependencies for Lambda. Excluding certain modules from the root of your node_modules directory might really not be the right approach. I'd argue that your npm i --production approach to generate a clean node_modules tree that includes only the deps needed for your runtime code is a much more robust approach.

I wonder if perhaps there's something we can add to the CDK such that utilizing this approach will be straightforward (i.e. run npm install from within the toolkit?)

It's definitely a topic of interest for us, which we would love to address end-to-end (for node and all other languages as well). Copy: @sam-goodwin

@justinmchase
Copy link
Contributor

@eladb I agree after digging into it more.

I wrote a little script to do this, to illustrate the issue:

#!/bin/bash
set -e

function prep-func {
  func=$1

  echo "$func:"
  cp src/funcs/$func/package.json dist/funcs/$func/package.json
  cp src/funcs/$func/package-lock.json dist/funcs/$func/package-lock.json
  pushd dist/funcs/$func      &> /dev/null
  npm i --production --silent
  popd                        &> /dev/null
}

echo "compiling..." 
npx tsc

echo "preparing functions..."
prep-func "shared"
prep-func "exportDelivery"

echo "deploying to amazon..."
npx cdk deploy --app dist/cloud/index.js

echo "done."

I think its fine to let people resolve this on their own on second thought, because I'm not sure how you could really solve this in a general way. These are steps for the pre-deploy build system of each individual project, and if I have a mono-repo I need to ensure that each of those sup-repos are building their dependencies at the correct time with the correct settings, not you.

@srchase srchase added feature-request A feature should be added or improved. and removed enhancement labels Jan 3, 2019
@eladb eladb self-assigned this Aug 12, 2019
@eladb eladb assigned nija-at and unassigned eladb and shivlaks Sep 3, 2019
@nija-at nija-at removed the package/tools Related to AWS CDK Tools or CLI label Sep 25, 2019
@nija-at
Copy link
Contributor

nija-at commented Oct 18, 2019

still relevant

@nija-at nija-at changed the title Allow multiple folders for Lambda configuration [lambda] specify multiple folders as a single lambda assets Nov 26, 2019
@nija-at nija-at changed the title [lambda] specify multiple folders as a single lambda assets [lambda] specify multiple folders as a single lambda asset Nov 26, 2019
@nija-at nija-at added the effort/large Large work item – several weeks of effort label Feb 6, 2020
@shahbour
Copy link

Any plan on this feature ?

@eladb
Copy link
Contributor

eladb commented Jun 16, 2020

Perhaps @jogold will be interested to add this capability to our new shiny staging system.

@nija-at nija-at added the @aws-cdk/assets Related to the @aws-cdk/assets package label Aug 25, 2020
@nija-at nija-at assigned eladb and unassigned nija-at Aug 25, 2020
@nija-at nija-at removed the @aws-cdk/assets Related to the @aws-cdk/assets package label Aug 25, 2020
@nija-at nija-at removed the effort/large Large work item – several weeks of effort label Aug 25, 2020
@eladb eladb added effort/small Small work item – less than a day of effort p2 labels Aug 26, 2020
@gabrielpeery
Copy link

Hey has there been any movement on this since August? I could really use this feature in my mono-repo.

@eladb eladb removed their assignment Feb 25, 2021
@ericzbeard ericzbeard added the feature/enhancement A new API to make things easier or more intuitive. A catch-all for general feature requests. label Apr 6, 2021
@zarnoevic
Copy link

+1

@bes
Copy link

bes commented Jul 28, 2021

Would love this feature, have a somewhat intricate folder structure and I really don't want to duplicate a lot of files...

@ronnathaniel
Copy link

+1

@github-actions github-actions bot added p1 and removed p2 labels Apr 10, 2022
@github-actions
Copy link

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

@awspbade
Copy link

+1

@ahzia
Copy link

ahzia commented Jul 23, 2022

I have created a draft PR for this, it is not finalized but I appreciate any review or feedback for it.
PR link

@github-anis-snoussi
Copy link

+1

2 similar comments
@coolbotic
Copy link

+1

@talhankoc
Copy link

+1

@TheRealAmazonKendra TheRealAmazonKendra added p2 and removed p1 labels Jan 24, 2023
@madeline-k
Copy link
Contributor

#4776 looks to be related to this

@mccauleyp
Copy link

+1

1 similar comment
@elena-cav
Copy link

elena-cav commented Jun 20, 2023

+1

@mfittko
Copy link

mfittko commented Oct 13, 2023

so any way around except for using docker for packaging a custom folder structure? +1

@kareldonk
Copy link

+1

1 similar comment
@isabellanicacio
Copy link

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-lambda Related to AWS Lambda effort/small Small work item – less than a day of effort feature/enhancement A new API to make things easier or more intuitive. A catch-all for general feature requests. feature-request A feature should be added or improved. p2
Projects
None yet