S3 : Sending campain with private bucket file attachement fails because of incorrrect PresignedUrl

[julienvienne]

Version:

• listmonk: [eg: v2.5.1]
• OS: [ubuntu 22.04]

Description of the bug and steps to reproduce:

I try to use AWS S3 private bucket to save medias and send a new campain.
File upload is going well in my bucket using credentials, but when I try to send a new campain, listmonk creates a presigned URL to get the file back. This URL is not working in my case. I get a 403 error because :

<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

The URL is build in this source code, but I cannot understand why the signature mismatch.
According to this StackOverflow post, it may be related to incorrect file path in the URL.

As a consequence, the campain stays indefinitively on the Running status

[knadh]

Can you blur the sensitive details and share your S3 settings?

[julienvienne]

Thank you for your concern. Here they are :

After few investigation, it seems that the generated link is fine in the media management dialog.
After uploading an image, it appears correcly in the dialog, and the image link is a presigned URL which is working fine.

However, sending the campain with this media goes to infinie running loop :

In the logs dialog, I can read :

2023/09/04 13:47:27error processing campaign XXX : error fetching attachment 19 on campaign XXX: status code: 404 Not Found
2023/09/04 13:47:32error processing campaign XXX : error fetching attachment 19 on campaign XXX: status code: 404 Not Found
2023/09/04 13:47:37error processing campaign XXX: error fetching attachment 19 on campaign XXX: status code: 404 Not Found
...

[julienvienne]

Hello @knadh,

I may found the bug here :
https://github.com/knadh/listmonk/blob/master/internal/media/providers/s3/s3.go#L108

It seems that ObjectKey: c.makeBucketPath(filepath.Base(url)) is used to get the S3 Object key from URL.
However, the presigned URL may contain additional data like X-Amz-Algorithm, X-Amz-SignedHeaders and others headers for AWS. As a result, the ObjectKey may not be correct.

I am not sure that the makeBucketPath(name string) cleans the header informations correctly.

[ashso]

I'm encountering a problem similar to Julien's. I tried using Cloudflare R2, which is S3 compatible. I successfully uploaded a file to R2 and verified that I could fetch it from there. However, I'm seeing the same error in the log, but the link in the Media section seems to be incorrect as it's using a subdomain of s3.amazonaws.com. I've attached the configuration I used for reference.

p.s. I am also using the same version of listmonk, but run in a container

[julienvienne]

To sum up :

• the presigned URL is not the problem here (contrary to the issue title). It is working fine in the media dialog
• the bug seems to come from the GetBlob(url string) , which is used to retrieve the file to create the campain attachment.
• In this function, the s3 object key is determined from url in a way that may be incorrect.
• as a result, the file is not found on s3 and we get 404 error

[knadh]

Thanks for investigating and detailing this issue @julienvienne. Turns out, there were multiple issues here.

• There's a silly bug with the default 14d expiry value. d is not a valid timestring suffix. This has to be specified in hours. 8f2a08b addresses this.
• Discovered that S3 only supports a max of 7 days for presigned URLs. Anything above, it throws an error.
• GetBlob() indeed had a bug, which 04e571d fixes.

@ashso you've to set the appropriate Cloudflare URL in the "custom public URL" field.