Does not play nice with multer

[sean-hill]

I'm trying to create a little app that processes images of celebrities. I'm utilizing the multer library to handle my image uploads. When tested with only express, the image uploads work great, but when wrapped with serverless-http the image uploads are corrupted. Here is a sample repository that demonstrates the problem. You can test with serverless-offline, but the problem also persists when released to AWS as well. Thanks for any insight you can provide!

[dougmoscrop]

Thanks for the feedback!

I've only had a peek, but it looks like serverless-offline does not support binary mode things yet -- dherault/serverless-offline#335

Did you configure binary mode support in your API Gateway when you tested it released to AWS?

I know my documentation isn't great https://github.com/dougmoscrop/serverless-http/blob/master/README.md#binary-support about what it means to use binary support. I can try to improve it, but it is just one of those weird like.. intersections of AWS configuration outside of your function and what this library does inside your function.

A colleague of mine wrote a great plugin: https://github.com/LiyueWang/serverless-plugin-custom-binary for enabling binary via serverless, which works better than the other 2 alternatives IMO. The only problem there is you will then have a discrepancy between have serverless-offline works and how the real thing works.

[dougmoscrop]

And sorry once you do enable binary support in API Gateway, you have to tell serverless-http which content-types should be interpreted as binary:

https://github.com/dougmoscrop/serverless-http/blob/master/lib/is-binary.js#L4

[sean-hill]

Hey @dougmoscrop thanks for the feedback! I haven't done anything outside of what is in that provided repo, so let me look through your docs and I'll try enabling binary mode. I'll post back my findings in here. Thanks!

[sean-hill]

Okay. I've attempted to add binary support, but I think I might be missing a piece. I've added it to the API Gateway:

I've added it to serverless-http:

I'm not sure if I have to do some other steps, but with those two, Rekogition is still complaining that the image data buffer is invalid.

[dougmoscrop]

Did you use the plug-in that I linked to set up binary mode?

…

On Sat, Dec 16, 2017, 10:58 PM Sean Hill ***@***.***> wrote: Okay. I've attempted to add binary support, but I think I might be missing a piece. I've added it to the API Gateway: [image: screen shot 2017-12-16 at 8 55 19 pm] <https://user-images.githubusercontent.com/1205866/34076402-7c98450e-e2a3-11e7-9533-e57ab04353e7.png> I've added it to serverless-http: [image: screen shot 2017-12-16 at 8 57 15 pm] <https://user-images.githubusercontent.com/1205866/34076409-bbe3d5e8-e2a3-11e7-9ffa-a62bebda9aaa.png> I'm not sure if I have to do some other steps, but with those two, Rekogition is still complaining that the image data buffer is invalid. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAjzFG0eTuQfGHWTmw8V9jtO0ReGq0Zmks5tBJFTgaJpZM4RDqW1> .

[sean-hill]

@dougmoscrop I have attempted to make the changes as seen here. My serverless.yml now looks like this:

service: serverless-rekognition

plugins:
  - serverless-offline
  - serverless-plugin-custom-binary

custom:
  apigatewayBinary:
    types:
      - 'image/png'
      - 'image/jpeg'

provider:
  name: aws
  runtime: nodejs6.10
  stage: dev
  region: us-east-1
  iamRoleStatements:
    - Effect: Allow
      Action:
        - rekognition:*
      Resource: '*'

functions:
  app:
    handler: index.handler
    events:
      - http: 
          path: /
          method: GET
      - http: 
          path: /{proxy+}
          method: ANY
          contentHandling: CONVERT_TO_BINARY

I am still facing the same issue, where the images that are uploaded are not coming through the API gateway properly, their encoding is all messed up. This is just a side project, so no worries if we can't figure it out 😄

Here's my url I'm using to POST an image to:

https://4xiwvn50fj.execute-api.us-east-1.amazonaws.com/dev/lookup-celebrities

[dougmoscrop]

OK I will have a look!

[lfreneda]

Facing the same issue, somebody solved? @dougmoscrop @sean-hill

I'm not using Rekogition, just porting a simple upload file api to lambda :(

[colin-kirkham]

I have a similar problem, or at least it looks that way. Specifically I'm trying to return PDF documents and have tried various ways to get this working but all to no avail. I've created a GitHub project to illustrate here: https://github.com/colin-kirkham/pdf-serverless/

Any help would be gratefully received - this example is probably small enough to be genuinely useful to other people struggling with a similar issue so I'd love to collaborate and share a working solution.

As I mention in the README, using no plugins or changes gives the 'best' result but the binary streams inside the PDF are being changed somewhere along the line. I'm guessing this is some kind of encoding issue.

[lfreneda]

@dougmoscrop do you have some news to share with us? thanks

[lfreneda]

:(

[dougmoscrop]

@sean-hill I deployed your repo, and did

curl {{endpoint}} and I get a bunch of obviously base-64 encoded data back.

I do

curl {{endpoint}} -H "Accept: application/pdf" and I get back a binary stream.

Accept: application/pdf, text/html works
Accept: application/* does not work
Accept: text/html, application/pdf does not work

API Gateway only performs the CONVERT_TO_BINARY when both the Content-Type header and the Accept header exactly match or the Accept: header has the exact match as the first option.

This is in the AWS documentation for binary mode in API gateway, but it's not very obvious.

[dougmoscrop]

I think this is more problematic for the image one, where you don't know if it's going to be an image/jpeg or image/png etc. -- this is all a deficiency with API Gateway, not serverless-http.

You can 'simplify' the problem/example by just doing a Lambda:

module.exports.handler = function (event, context, callback) {
  const data = fs.readFileSync('test.pdf')
  const body = Buffer.from(data).toString('base64')
  callback(null, {
    isBase64Encoded: true,
    statusCode: 200,
    headers: { content-type: 'application/pdf', content-length: body.length },
    body
  })
}

With binaryMediaTypes configured, and CONVERT_TO_BINARY content handling set, you will only get back the data correctly if Accept: matches the binaryMediaType

[ramiel]

The way to resolve this is to set, on API Gateway, the following types as binary:

• image/*
• */*

I guess by the way, this would be a problem if you want to mix binary and not binary responses

[sean-hill]

Thanks guys, I think there has been enough conversation and research done here. You're awesome 👍

[Jun711]

@ramiel when u set binary type as * / *, do u have any problem with OPTIONS call?
I got Internal server error when I set binary as * / *
CodeGenieApp/serverless-express#58

[tomyates]

Took me many attempts to get this working. Just incase it helps others this are the settings that eventually worked for me after installing serverless-apigw-binary plugin.

serverless.yml:

plugins:
  - serverless-apigw-binary

custom:
  apigwBinary:
    types:
      - '*/*' 

index.js:

module.exports.handler = serverless(app, {
  binary: ['image/png', 'image/gif']
});

[evangow]

@tomyates did you get this working with serverless-offline as well?

[YongHoonJJo]

It works well..!!

[adamwilbert]

Took me many attempts to get this working. Just incase it helps others this are the settings that eventually worked for me after installing serverless-apigw-binary plugin.

serverless.yml:

plugins:
  - serverless-apigw-binary

custom:
  apigwBinary:
    types:
      - '*/*' 

index.js:

module.exports.handler = serverless(app, {
  binary: ['image/png', 'image/gif']
});

What are you setting headers to on your request here? Accept /? and content type?

[ajouve]

Do you have a solution ?
The serverless-apigw-binary and binary is not working for me.

Reading the documentation the following code seems to be for response and not request, I do not really understand how it can fix the issue. https://github.com/dougmoscrop/serverless-http/blob/8680abb47c4f5cfcc680a974d6dc43f3e3bd29be/docs/ADVANCED.md#binary-mode

module.exports.handler = serverless(app, {
  binary: ['image/png', 'image/gif']
});

My actual solution is to convert my files to base64 before upload, I do not like but only way to make it works

[washedrepent]

I finally got this working, I was facing the same issue. I realized I am passing multipart/form-data so I needed to add that as well to the binary media types section on API-Gateway.
The last piece that got it working for me was to set the HTTP request headers on the /{proxy+} Any method request.

I did theses steps:

1. Go to the API Gateway settings tab for your API and add multipart/form-data to the binary media types section.
2. Add Content-Type and Accept to the request headers for your proxy method
3. Re-deploy the API

I found this from the second most voted answer from https://stackoverflow.com/questions/41756190/api-gateway-post-multipart-form-data/41770688#41770688

Once I did that it worked.
For reference here is my serverless.yaml config

service: my-express-service

plugins:
  -serverless-apigw-binary

custom:
  apigwBinary:
    types:
      - 'image/jpeg'
      - 'image/gif'
      - 'image/png'

provider:
  name: aws
  runtime: nodejs12.x
  stage: dev
  region: us-east-1

functions:
  app:
    handler: app.handler
    events:
      - http: ANY /
        cors: true
      - http: 'ANY {proxy+}'
        cors: true
        contentHandling: CONVERT_TO_BINARY
    environment:
       AWSKEY: ${file(../config.${self:provider.stage}.json):AWSKEY}
       AWSSECRET: ${file(../config.${self:provider.stage}.json):AWSSECRET}

My handler from app.js

module.exports.handler = serverless(app,{callbackWaitsForEmptyEventLoop: false, binary:['image/png', 'image/jpeg', 'image/gif'] });

And my routing file

var aws = require('aws-sdk');
const multer = require('multer');
var multerS3 =  require('multer-s3');

aws.config.update({
  accessKeyId: process.env.AWSKEY,
  secretAccessKey: process.env.AWSSECRET,
  region:'us-east-1',
  signatureVersion: 'v4'
});

var s3 = new aws.S3();

var upload = multer({
    storage: multerS3({
        s3: s3,
        bucket: 'my_bucket_name',
        acl: 'public-read',
        contentType:function (req, file, cb){
          cb(null, file.mimetype);
        },
        metadata: function (req, file, cb) {
          cb(null, { fieldName: file.fieldname });
        },
        key: function (req, file, cb) {
            console.log(file);
            var newFileName = new Date().toISOString()+ '-' + file.originalname
            var fullpath = 'express/' + newFileName;
            cb(null, fullpath ); 
        }
    })
});

module.exports = function(app) {
	app.post('/upload', upload.array('uploads[]',1), (req, res, next) => {
		res.json({
			'message': 'File uploaded succesfully.',
			data:req.files
		});
	});
};

Hopefully this helps.

[hjrobinson]

washedrepent could you provide a more complete example? I'm not sure how the app.js file and your routing file tie together (there is no reference to express, express router etc). I did my best to piece things together but I'm still getting an error.