[Bug]: api can't download remote file #1394

toebbel · 2023-07-24T18:12:12Z

Verified issue does not already exist?

I have searched and found no existing issue

What happened?

I try to write a script that interacts with a remote server via @actual/api like described here.

Server version: v23.7.2
@actual/api version: 6.1.1

The script I use:

const api = require('@actual-app/api');

(async () => {
  await api.init({
    //dataDir: './',
    serverURL: '<https url to a fly VM>',
    password: '<server password>',
  });
  console.log(api);

  await api.downloadBudget('< my sync id >', { password: '< file password>' } );

  let accounts = await api.getAccounts();
	for (account in accounts) {
		console.log(account.name);
	}
  await api.shutdown();
})();

I execute the script via node v20.2.0.

I tried the following:

use the budget name instead of the id --> I see an error that the sync ID can't be found
use another password --> I see an error that the encryption password is wrong
create a new budget file on the server, without password protection --> same error as before

What error did you receive?

Error: Something went wrong trying to download that file, sorry! Visit https://actualbudget.org/contact/ for support. (fileId: a96b84c9-c873-41a2-9314-da454c5bc386)
at handlers.api/download-budget (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32814:31)

Where are you hosting Actual?

Fly.io

What browsers are you seeing the problem on?

Other

Operating System

Mac OSX

The text was updated successfully, but these errors were encountered:

j-f1 · 2023-07-24T19:33:23Z

Can you try downloading the beta API package from this build (the actual-api artifact)? Once you’ve downloaded that zip, uncompress it to get a tgz file. Then pass that to npm install /path/to/actual-api.tgz to install it into your project. Can you try again and let me know what error you get (please send the full error log)? That build has some improved error messaging for this situation that will be included in the next version, but if we can also fix your issue directly that would be great to do too.

toebbel · 2023-07-24T20:19:45Z

Thanks @j-f1 for getting back on this matter so quickly.

I've unzipped the actual-api build artefact and pointed the package.json file to the unzipped path.

This is the error log I see now:

Full error details {
  type: 'FileDownloadError',
  reason: 'internal',
  meta: { fileId: '4a0719ca-fbba-<retracted>' }
}
/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32223
                        throw new Error((0, _shared_errors__WEBPACK_IMPORTED_MODULE_2__.getDownloadError)(result.error));
                              ^

Error: Something went wrong trying to download that file, sorry! Visit https://actualbudget.org/contact/ for support. (reason: internal, fileId: 4a0719ca-fbba-<retracted>)
    at handlers.api/download-budget (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32223:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

j-f1 · 2023-07-24T20:30:50Z

Thanks for checking that! I now see where the error happened. Can you try out this build? That adds a bit more logging. My guess is that somehow the file ID is incorrect, since that’s the main way this could fail (assuming your connection to the server is correct)

toebbel · 2023-07-24T20:48:40Z

Ahh. The error shows something odd

Error fetching file info FetchError: request to https://<redacted>/sync/get-user-file-info failed, reason: socket hang up
    at ClientRequest.<anonymous> (/Users/tobi/projects/actual-to-google-sheets/node_modules/node-fetch/lib/index.js:1505:11)
    at ClientRequest.emit (node:events:511:28)
    at TLSSocket.socketOnEnd (node:_http_client:519:9)
    at TLSSocket.emit (node:events:523:35)
    at endReadableNT (node:internal/streams/readable:1367:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNRESET',
  code: 'ECONNRESET'
}
Full error details {
  type: 'FileDownloadError',
  reason: 'internal',
  meta: { fileId: '4a0719ca-fbba-<redacted>' }
}
/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32223
                        throw new Error((0, _shared_errors__WEBPACK_IMPORTED_MODULE_2__.getDownloadError)(result.error));
                              ^

Error: Something went wrong trying to download that file, sorry! Visit https://actualbudget.org/contact/ for support. (reason: internal, fileId: 4a0719ca-fbba-<redacted>)
    at handlers.api/download-budget (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:32223:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v20.2.0

When I open the path https://<redacted>/sync/get-user-file-info in Safari I see this response

{"status":"error","reason":"unauthorized","details":"token-not-found"}

Same, when I use curl

% curl -v https://<redacted>/sync/get-user-file-info
*   Trying <redacted>:443...
* Connected to <redacted> (<redacted>) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.fly.dev
*  start date: Jun  9 23:43:51 2023 GMT
*  expire date: Sep  7 23:43:50 2023 GMT
*  subjectAltName: host "<redacted>" matched cert's "*.fly.dev"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /sync/get-user-file-info]
* h2h3 [:scheme: https]
* h2h3 [:authority: <redacted>]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x12000a800)
> GET /sync/get-user-file-info HTTP/2
> Host: <redacted>
> user-agent: curl/7.88.1
> accept: */*
> 
< HTTP/2 401 
< x-powered-by: Express
< access-control-allow-origin: *
< ratelimit-limit: 500
< ratelimit-remaining: 499
< ratelimit-reset: 42
< content-type: application/json; charset=utf-8
< content-length: 70
< etag: W/"46-arBLVEwoDaCK5mPJtWkb6+x8MW0"
< date: Mon, 24 Jul 2023 20:38:26 GMT
< server: Fly/a0b91024 (2023-06-13)
< via: 2 fly.io
< fly-request-id: 01H64SZTEQ4PMME0SNCJKWFP0H-arn
< 
* Connection #0 to host <redacted> left intact
{"status":"error","reason":"unauthorized","details":"token-not-found"}%

First thing I tested: change the password that is sent to the server to some gibberish. I do get a different error response:

Error: Could not get remote files

I double-checked the password in my original script: it is the one I use to authenticate to the remote-server.

Second thing: I looked at the request headers that are passed with the /sync request from the web app to the server. I see that a header X-ACTUAL-TOKEN is passed. I attempted to call /sync/get-user-file-info with this header via
curl -H "X-ACTUAL-TOKEN: <redacted>" https://<redacted>/sync/get-user-file-info.

I am also puzzled, because the web app makes requests to sync/list-user-files instead of /sync/get-user-file-info; but I don't have much experience with the code-base or endpoints.

I'll be AFK until tomorrow morning CEST. Thanks for the quick support :)

j-f1 · 2023-07-25T12:18:56Z

Yep… you’re actually not the first person with that particular issue. (See https://discord.com/channels/937901803608096828/1123193332697419776/1123193332697419776). Something weird must be happening on Fly’s end. The only thing I can think of trying is changing what headers we send to see what the difference is between the browser/curl and Node?

Did some research, and it looks like it may be this bug which was just fixed & released: node-fetch/node-fetch#1735

Can you try again with this build? It has the updated version of node-fetch.

toebbel · 2023-07-25T19:29:08Z

Oh wow - great find!

I attempted to test the build, but I get this error:

/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/index.js:33
const node_fetch_1 = __importDefault(require("node-fetch"));
                                     ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/tobi/projects/actual-to-google-sheets/node_modules/node-fetch/src/index.js from /Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/index.js not supported.
Instead change the require of /Users/tobi/projects/actual-to-google-sheets/node_modules/node-fetch/src/index.js in /Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/index.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/index.js:33:38)
    at Object.<anonymous> (/Users/tobi/projects/actual-to-google-sheets/index.js:1:13) {
  code: 'ERR_REQUIRE_ESM'
}

Node.js v18.17.0

I'm not sure, but it seems like the dist folder contains javascript that has some invalid require statements?

I have first tried with node version 20.2.0, then 20.5.0 and now I switched to the LTS version of node. My package.json only contains this:

{
  "dependencies": {
    "@actual-app/api": "file:/Users/tobi/Downloads/actual-api"
  }
}

j-f1 · 2023-07-25T20:49:03Z

Yep, that’s my fault. Can you give the build from here a shot? I’ve updated the code so that it uses import() to import the node-fetch module (I was misled because our source code uses ESM syntax, but the built code is in CJS for compatibility). I also removed the reference to node-fetch from the bundle.api.js file.

toebbel · 2023-07-27T15:31:08Z

I'm always puzzled by web pack options. Thanks for looking into this!
Unfortunately, I get a different error now:

/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:28559
            const fetch = self.fetch;
                          ^

ReferenceError: self is not defined
    at ./packages/loot-core/src/platform/server/fetch/index.api.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:28559:27)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)
    at ./packages/loot-core/src/server/post.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:38398:92)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)
    at ./packages/loot-core/src/server/sync/index.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:40342:76)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)
    at ./packages/loot-core/src/server/db/index.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:36879:75)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)
    at ./packages/loot-core/src/server/accounts/sync.ts (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:30106:73)
    at __webpack_require__ (/Users/tobi/projects/actual-to-google-sheets/node_modules/@actual-app/api/dist/app/bundle.api.js:45954:48)

Node.js v18.17.0

j-f1 · 2023-07-27T18:46:11Z

Sorry about that! Gave the code in this build a more through test and you should not be seeing that issue now.

toebbel · 2023-07-28T16:44:09Z

awesome - this works like a charm!
Thank you very much @j-f1 for the great support :)

andrewzolotukhin · 2023-12-07T03:18:29Z

Hi! I'm still getting the same problem using the latest (6.3.0) version of API. Just wondering if the was released?

toebbel added the bug Something isn't working label Jul 24, 2023

j-f1 mentioned this issue Jul 24, 2023

Improve messaging when the API fails to download a file #1396

Merged

j-f1 added the API Issues with the @actual-app/api package label Jul 24, 2023

j-f1 mentioned this issue Jul 24, 2023

Update node-fetch in the API to fix connection drop issue #1397

Merged

j-f1 closed this as completed in #1397 Jul 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: api can't download remote file #1394

[Bug]: api can't download remote file #1394

toebbel commented Jul 24, 2023

j-f1 commented Jul 24, 2023

toebbel commented Jul 24, 2023

j-f1 commented Jul 24, 2023

toebbel commented Jul 24, 2023

j-f1 commented Jul 25, 2023

toebbel commented Jul 25, 2023

j-f1 commented Jul 25, 2023

toebbel commented Jul 27, 2023

j-f1 commented Jul 27, 2023

toebbel commented Jul 28, 2023

andrewzolotukhin commented Dec 7, 2023

[Bug]: api can't download remote file #1394

[Bug]: api can't download remote file #1394

Comments

toebbel commented Jul 24, 2023

Verified issue does not already exist?

What happened?

What error did you receive?

Where are you hosting Actual?

What browsers are you seeing the problem on?

Operating System

j-f1 commented Jul 24, 2023

toebbel commented Jul 24, 2023

j-f1 commented Jul 24, 2023

toebbel commented Jul 24, 2023

j-f1 commented Jul 25, 2023

toebbel commented Jul 25, 2023

j-f1 commented Jul 25, 2023

toebbel commented Jul 27, 2023

j-f1 commented Jul 27, 2023

toebbel commented Jul 28, 2023

andrewzolotukhin commented Dec 7, 2023