Skip to content
This repository has been archived by the owner on Feb 22, 2023. It is now read-only.

[BUG] Cant Upload Files #73

Closed
Whitei520 opened this issue Jun 10, 2022 · 32 comments
Closed

[BUG] Cant Upload Files #73

Whitei520 opened this issue Jun 10, 2022 · 32 comments
Labels
bug Something isn't working

Comments

@Whitei520
Copy link

Container                             Repository                        Tag                 Image Id            Size
------------------------------------------------------------------------------------------------------------
api-gateway-standalone                standardnotes/api-gateway         1.40.1              614ea1d3f64c        851MB
auth-standalone                       standardnotes/auth                1.46.2              6648d9486d27        928MB
auth-worker-standalone                standardnotes/auth                1.46.2              6648d9486d27        928MB
cache-standalone                      redis                             6.0-alpine          09401fed2a42        24.7MB
db-standalone                         mysql                             5.6                 dd3b2a5dcb48        303MB
files-standalone                      standardnotes/files               1.9.0               baa6121a09d1        1.05GB
syncing-server-js-standalone          standardnotes/syncing-server-js   1.54.1              2455980ced7d        973MB
syncing-server-js-worker-standalone   standardnotes/syncing-server-js   1.54.1              2455980ced7d        973MB

Describe the bug
Error when uploading attachments or files.

Screenshot 2022-06-10 172230

To Reproduce
Steps to reproduce the behavior:

  1. Upload a file

Logs

Request URL: https://files-notes.REDACTED.com/v1/files/upload/create-session
Request Method: OPTIONS
Status Code: 204 
Request URL: https://files-notes.REDACTED.com/v1/files/upload/create-session
Request Method: POST
Status Code: 400 

Screenshot 2022-06-10 182226

Additional context
In its current state the docker compose file doesn't account for setting a files url at all so I changed
FILES_SERVER_URL: 'http://localhost:${EXPOSED_FILES_PORT}'
to
FILES_SERVER_URL: '${FILES_SERVER_URL}'

Then added
FILES_SERVER_URL=https://files-notes.REDACTED.com
to the .env file.

This resulted with the correct domain showing up in the network tab of developer tools:
{"filesServerUrl":"https://files-notes.REDACTED.com"}

@Whitei520 Whitei520 added the bug Something isn't working label Jun 10, 2022
@eric-pierce
Copy link
Contributor

eric-pierce commented Jun 10, 2022

@Whitei520 according to #69 (comment) file uploads aren't supported on the standalone/self hosted infrastructure yet.

@Whitei520
Copy link
Author

@Whitei520 according to #69 (comment) file uploads aren't supported on the standalone/self hosted infrastructure yet.

Support has just been added.
Screenshot 2022-06-10 191252

@techmechanic1
Copy link

I'm still unable to upload files after this update...
image

@karolsojko
Copy link
Member

@techmechanic1 did you set up the FILES_SERVER_URL value in your .env file and restarted the services?

anything you are seeing in logs for docker compose logs -f files ?

@Whitei520
Copy link
Author

This was closed but I still have the issue. My troubleshooting steps was not the solution, it was merely my steps to try and fix part of the problem. Even after getting the correct "filesServerUrl" showing up in the network logs I still get the same upload error.

@karolsojko
Copy link
Member

karolsojko commented Jun 15, 2022

@Whitei520 could you send the output of docker compose logs -f files and docker compose logs -f auth so I could see what's the potential error ?

also anything in the response body of that 400 call ?

@karolsojko karolsojko reopened this Jun 15, 2022
@techmechanic1
Copy link

techmechanic1 commented Jun 15, 2022

Here is the output of docker-compose logs -f files...

Attaching to files-standalone
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | waiting for db:3306...
files-standalone | db (172.18.0.2:3306) open
files-standalone | db:3306 is up - executing command
files-standalone | cache (172.18.0.4:6379) open
files-standalone | cache:6379 is up - executing command
files-standalone | Starting Web...
files-standalone | yarn run v1.22.15
files-standalone | $ node dist/bin/server.js
files-standalone | {"v":0,"level":30,"name":"newrelic_bootstrap","hostname":"568ddd4a2a1c","pid":88,"time":"2022-06-14T20:20:12.502Z","msg":"Unable to find configuration file. If a configuration file is desired (common for non-containerized environments), a base configuration file can be copied from /var/www/node_modules/newrelic/newrelic.js and renamed to "newrelic.js" in the directory from which you will start your application. Attempting to start agent using environment variables."}
files-standalone | {"message":"Server started on port 3000","level":"info"}


The Files storage quota is showing 0 B of 0 B used...

Screenshot 2022-06-15 074546

@karolsojko
Copy link
Member

@techmechanic1 did you create a subscription for yourself? https://docs.standardnotes.com/self-hosting/subscriptions

@techmechanic1
Copy link

Yes, all the other premium features are working...

@Whitei520
Copy link
Author

Whitei520 commented Jun 15, 2022

@Whitei520 could you send the output of docker compose logs -f files and docker compose logs -f auth so I could see what's the potential error ?

also anything in the response body of that 400 call ?

Thanks for reopening. I checked the docker logs and found that the container did not have file permissions in the /data folder, I fixed that and now I can upload files without any errors. However I can't view or download any uploaded files. I get a 400 error in the network console with a response of "File download requires range header to be set."

RESPONSE

Request URL: https://files-notes.example.com/v1/files
Request Method: GET
Status Code: 400 
Remote Address: 104.21.12.84:443
Referrer Policy: strict-origin-when-cross-origin
access-control-allow-origin: *
access-control-expose-headers: Content-Range,Accept-Ranges
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
cf-cache-status: DYNAMIC
cf-ray: 71bb5541bcf571db-LHR
content-length: 46
content-security-policy: default-src https: 'self';base-uri 'self';child-src * blob:;connect-src *;font-src * 'self';form-action 'self';frame-ancestors * *.standardnotes.org *.standardnotes.com;frame-src * blob:;img-src 'self' * data:;manifest-src 'self';media-src 'self';object-src 'self';script-src 'self';style-src 'self'
content-type: text/plain; charset=utf-8
date: Wed, 15 Jun 2022 12:33:14 GMT
etag: W/"2e-5Dp512EBepCWZARNQm+8AgbWacM"
expect-ct: max-age=0
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
referrer-policy: no-referrer
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=BH2U2bkXN5Mr%2Fm29ihibeQBDO3e9vsz7AzD"}],"group":"cf-nel","max_age":604800}
server: cloudflare
strict-transport-security: max-age=15552000; includeSubDomains
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-download-options: noopen
x-files-version: local
x-frame-options: SAMEORIGIN
x-permitted-cross-domain-policies: none
x-xss-protection: 0

REQUEST

:authority: files-notes.example.com
:method: GET
:path: /v1/files
:scheme: https
accept: */*
accept-encoding: identity
accept-language: en-GB,en-US;q=0.9,en;q=0.8,la;q=0.7
content-type: application/json
origin: https://notes.example.com
range: bytes=0-
referer: https://notes.example.com/
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="102", "Google Chrome";v="102"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-site
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36
x-application-version: Web-3.20.2
x-chunk-size: 135865
x-snjs-version: 2.114.12
x-valet-token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyVXVpZCI6IjRhY2ZkMTQxLTVhOWQtNDB...

As an extra troubleshooting step I also disabled Cloudflare proxy but the same error showed up.

@rofra
Copy link

rofra commented Jun 15, 2022

I had the similar bug, effectively coming from permissions. I had to run

chmod -R 775 data
mkdir -p data/uploads
sudo chmod -R 755 data/uploads
sudo chown -R 1001.1001 data/uploads

Maybe a permissions clarification would be needed in the documentation, but no other issue for me, everything running smoothly

@techmechanic1: Have you seen the comment in the .env.sample file ?
set the adress of the server from the outside world

For my case, as I am hosting it in another computer, default configuration with localhost was not possible

@techmechanic1
Copy link

Thanks for that! @rofra

I changed the permissions and changed the FILES_SERVER_URL and it works...I'm hosting the files on the same server but using localhost doesn't work anyway...

@rofra
Copy link

rofra commented Jun 15, 2022

@techmechanic1:Another clarification for the file upload, if you want to implement "Storage Quota", you could simply add an entry (per user) to set the quota.
Here is an example for 10 Gigas (into bytes):

INSERT INTO subscription_settings(uuid, name, value, created_at, updated_at, user_subscription_uuid) VALUES (UUID(), "FILE_UPLOAD_BYTES_LIMIT", 10*10737418240, FLOOR(UNIX_TIMESTAMP(NOW(6))*1000000), FLOOR(UNIX_TIMESTAMP(NOW(6))*1000000), (SELECT us.uuid FROM user_subscriptions us INNER JOIN users u ON us.user_uuid=u.uuid WHERE u.email="my@email.com"));

StandardNote Team, could you add the cleaner one to the official documentation ?

@Whitei520
Copy link
Author

Whitei520 commented Jun 16, 2022

Okay, well I finally found the problem I was having with "File download requires range header to be set." error.
It turns out Cloudflare for unknown reasons does not pass the "Accept-Ranges" header through its proxy. I did find a forum post though that solved the problem for me: here

After adding this line to my nginx config everything is now fixed. This doesn't explain why it still wasn't working with Cloudflare disabled but I will leave that conundrum for someone else to solve. :)

proxy_cache off; # Required by cloudflare to pass 'Accept-Ranges' header

I would strongly suggest this be added to the self hosted documentation for other people who use nginx and Cloudflare.

@karolsojko
Copy link
Member

Added everything mentioned in here to docs with this: standardnotes/docs@642df79

Thanks everyone for the input, hope it solves the issue :)

@rofra
Copy link

rofra commented Jun 17, 2022

@karolsojko: There is an error in the doumentationc, 1073741824 bytes shoud be 1Gb instead of 10G.

In my original SQL Request, I put 10*10737418240 which was correct for 10G.

@karolsojko
Copy link
Member

good catch @rofra - fixed in here

@Nicicalu
Copy link

Accept-Ranges

Where did you put this?

@BobWs
Copy link

BobWs commented Jun 27, 2022

Having the same problem, cannot upload files keep getting the same error.
Already tried everything suggested above. I'm self-hosting on Docker on a Synology NAS.
Standardnotes works just fine only the newly added files part isn't working.

This is my log from the files container.

db (172.18.0.6:3306) open

db:3306 is up - executing command

cache (172.18.0.5:6379) open

cache:6379 is up - executing command

Starting Web...

yarn run v1.22.15

$ node dist/bin/server.js

New Relic for Node.js was unable to bootstrap itself due to an error:

Error: New Relic requires that you name this application!

Set app_name in your newrelic.js file or set environment variable

NEW_RELIC_APP_NAME. Not starting!

    at createAgent (/var/www/node_modules/newrelic/index.js:145:11)

    at initialize (/var/www/node_modules/newrelic/index.js:84:15)

    at Object.<anonymous> (/var/www/node_modules/newrelic/index.js:37:3)

    at Module._compile (node:internal/modules/cjs/loader:1101:14)

    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)

    at Module.load (node:internal/modules/cjs/loader:981:32)

    at Function.Module._load (node:internal/modules/cjs/loader:822:12)

    at Module.require (node:internal/modules/cjs/loader:1005:19)

    at require (node:internal/modules/cjs/helpers:102:18)

    at Object.<anonymous> (/var/www/dist/bin/server.js:4:1)

{"v":0,"level":30,"name":"newrelic_bootstrap","hostname":"standardnotes-files","pid":31,"time":"2022-06-27T14:00:22.290Z","msg":"Unable to find configuration file. If a configuration file is desired (common for non-containerized environments), a base configuration file can be copied from /var/www/node_modules/newrelic/newrelic.js and renamed to \"newrelic.js\" in the directory from which you will start your application. Attempting to start agent using environment variables."}

{"message":"Server started on port 3000","level":"info"}

{"message":"ValetTokenAuthMiddleware authentication failure.","level":"debug"}

{"message":"ValetTokenAuthMiddleware authentication failure.","level":"debug"}

files.env:

# File upload path (relative to root directory)
FILE_UPLOAD_PATH=/volume1/docker/standardnotes/uploads

# File uploads
VALET_TOKEN_SECRET=**************************************b47779fd1e382 (generated with openssl rand -hex 32)
EXPOSED_FILES_PORT=3125
# Public-facing URL of the Files server. Endpoint on which clients will access it.
FILES_SERVER_URL=http://192.168.178.100:3125

files part of my docker-compose file:

  files:
    image: standardnotes/files:$FILES_VERSION
    container_name: standardnotes-files
    entrypoint: [
      "./wait-for.sh", "db", "3306",
      "./wait-for.sh", "cache", "6379",
      "./docker/entrypoint.sh", "start-web"
    ]
    hostname: standardnotes-files
    env_file: files.env
    mem_limit: 85m
    mem_reservation: 85m
    ports:
      - ${EXPOSED_FILES_PORT}:3000
    environment:
      FILE_UPLOAD_PATH: '${FILE_UPLOAD_PATH}'
      VALET_TOKEN_SECRET: '${VALET_TOKEN_SECRET}'
    volumes:
      - /volume1/docker/standardnotes/uploads:/var/www/${FILE_UPLOAD_PATH}
    restart: always
    networks:
      - standardnotes_network
   

Any help how to solve this!
TIA

@karolsojko
Copy link
Member

@BobWs can you make sure auth and files have the same value on VALET_TOKEN_SECRET ?

@BobWs
Copy link

BobWs commented Jun 28, 2022

@BobWs can you make sure auth and files have the same value on VALET_TOKEN_SECRET ?

@karolsojko They are the same. They are both pulling the key from the .env file

@karolsojko
Copy link
Member

@BobWs seems like you are missing the valet token in your request: https://github.com/standardnotes/files/blob/develop/src/Controller/ValetTokenAuthMiddleware.ts#L36

Do you have the latest version of the client application? Could you see what is sent in the request to files server - the valet token should be sent either in headers or in request parameters: https://github.com/standardnotes/files/blob/develop/src/Controller/ValetTokenAuthMiddleware.ts#L19

@BobWs
Copy link

BobWs commented Jun 28, 2022

Do you have the latest version of the client application? Could you see what is sent in the request to files server - the valet token should be sent either in headers or in request parameters: https://github.com/standardnotes/files/blob/develop/src/Controller/ValetTokenAuthMiddleware.ts#L19

Yes I have the latest MacOS desktop client version 3.22.17.
"Could you see what is sent in the request to files server - the valet token should be sent either in headers or in request parameters"
How do or where can I check this?

FILES_SERVER_URL=http://192.168.178.100:3125 is set to the IP of the host (my Synology NAS where Docker is running) is this good?

@BobWs
Copy link

BobWs commented Jun 28, 2022

After re-checking everything and recreated the containers I now get this in the error log

     

db (172.18.0.10:3306) open

db:3306 is up - executing command

cache (172.18.0.3:6379) open

cache:6379 is up - executing command

Starting Web...

yarn run v1.22.15

$ node dist/bin/server.js

{"v":0,"level":30,"name":"newrelic_bootstrap","hostname":"standardnotes-files","pid":30,"time":"2022-06-28T11:46:13.712Z","msg":"NEW_RELIC_NO_CONFIG_FILE set, deferring to environment variables."}

{"message":"Server started on port 3000","level":"info"}

{"message":"Could not create upload session for resource: 7447c434-ae0f-4581-b8ee-cad8c632518e - EACCES: permission denied, mkdir '/volume1'","level":"error"}

I don't know why it is showing permission denied, I have set the permission to 1001:1001 as mention above, I also tried to set it to root but still no go.

What am I doing wrong?

@BobWs
Copy link

BobWs commented Jun 28, 2022

Solved it!!
The problems was that I had set FILE_UPLOAD_PATH=/volume1/docker-data/standardnotes/uploads instead of FILE_UPLOAD_PATH=standardnotes/uploads Also the VALET_TOKEN_TTL=7200 was missing from the auth.env file (don't know if that was really necessary) but after changing all this it works and I can upload files.

Only problem I see is that I'm not able to preview the file/attachments via the mobile apps (iOS). I can upload files with the mobile apps but can't preview them I'm getting this error:
Screenshot 2022-06-28 at 14 34 38

@jonwsoto
Copy link

jonwsoto commented Jul 25, 2022

Hmm can't figure this out.

so when you go to your FILES_SERVER_URL by typing it straight into the browser you should be getting something similar to this right?

https://filesafe.standardnotes.org

when I paste my custom URL I get
Cannot GET /

and under developer tools you see

example.mydomain.com/v1/files/valet-tokens:1 Failed to load resource: the server responded with a status of 403 ()

edit***

looking more closely in network it says

{meta: {auth: {userUuid: "xxxxxxx-xxxxx-xxxxx-xxxxxx-xxxxxxxxxx,…},…},…}
data: {success: false, reason: "no-subscription"}
reason: "no-subscription"
success: false

so for some reason still says no subcription despite me updating the database and all the other features working.

any clue??

@karolsojko
Copy link
Member

so when you go to your FILES_SERVER_URL by typing it straight into the browser you should be getting something similar to this right?

Files service is a different service than filesafe. Filesafe is a deprecated service. The FILES_SERVER_URL should be pointing to the file service you have running.

so for some reason still says no subcription despite me updating the database and all the other features working.

Did you follow these steps: https://docs.standardnotes.com/self-hosting/subscriptions

Can you also verify if the record is in the database and the auth service is connected precisely to that database ?

@jonwsoto
Copy link

I reviewed everything I did and I realized in your script for adding subscriptions this

subscription_type="regular";' \

was missing from the examples given by the other users in this chat. I added “regular” value to the row and now it works!!

@BobWs
Copy link

BobWs commented Jul 28, 2022

subscription_type="regular";' \

Just curious, where did you edit this?
Can you elaborate a bit more so that other people will know what to do in case of problems.
TIA

@jonwsoto
Copy link

jonwsoto commented Jul 28, 2022

Under the server.sh file you can see the lines of code that handle the database modifications to add subscriptions.

$DOCKER_COMPOSE_COMMAND exec db sh -c "MYSQL_PWD=\$MYSQL_ROOT_PASSWORD mysql \$MYSQL_DATABASE -e \
  'INSERT INTO user_roles (role_uuid , user_uuid) VALUES ((SELECT uuid FROM roles WHERE name=\"PRO_USER\" ORDER BY version DESC limit 1) ,(SELECT uuid FROM users WHERE email=\"$EMAIL\")) ON DUPLICATE KEY UPDATE role_uuid = VALUES(role_uuid);' \
"

$DOCKER_COMPOSE_COMMAND exec db sh -c "MYSQL_PWD=\$MYSQL_ROOT_PASSWORD mysql \$MYSQL_DATABASE -e \
  'INSERT INTO user_subscriptions SET uuid=UUID(), plan_name=\"PRO_PLAN\", ends_at=8640000000000000, created_at=0, updated_at=0, user_uuid=(SELECT uuid FROM users WHERE email=\"$EMAIL\"), subscription_id=1, subscription_type=\"regular\";' \
"

Instead of running the script I just used phpmyadmin to modify the database manually. In the second command I was missing the subscription_type.

On a side note, I am not sure if you figured out your header issue but I am using cloud flare and you need to add

proxy_cache off;

to your reverse proxy. I am using Nginx

@BobWs
Copy link

BobWs commented Jul 28, 2022

Under the server.sh file you can see the lines of code that handle the database modifications to add subscriptions.

Thanks for sharing!

I followed the instructions for the pro plan from here: #64 and I see now that subscription_type="regular";' \ was missing in the instructions.

So do I need to do this all over for all my users or is there a easier way to get it done. For example how do I accomplish this with phpmyadmin?
TIA

@karolsojko
Copy link
Member

So do I need to do this all over for all my users or is there a easier way to get it done. For example how do I accomplish this with phpmyadmin? TIA

UPDATE user_subscriptions SET subscription_type="regular";

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

8 participants