N8N interface keeps staying on Executing node... and loosing connection sometimes on Digital Ocean's app platform

[matthijs166]

Describe the bug
I can't execute the workflow to test in the frontend interface.
It keeps loading.
Every workflow I create has this problem, no matter the configuration.

To Reproduce
Steps to reproduce the behavior:

1. Create an app in Digital Ocean
2. Create a resource from a docker image (n8nio/n8n)
3. Setup the env and add domain
4. Wait for a build
5. Open n8n and register
6. Create the first workflow and click on execute workflow
7. Workflow stays loading

Expected behavior

• To let the workflow run on digital ocean
• Give a clear error if it doesn't work
• A timeout with an error instead of waiting forever

Environment (please complete the following information):

• Digital Ocean
• Docker version latest
• Database system [SQLite]
• In the env file I have set N8N_HOST WEBHOOK_URL WEBHOOK_TUNNEL_URL VUE_APP_URL_BASE_API N8N_EDITOR_BASE_URL

Additional context
When I run the docker image local everything works

[Joffcom]

Hi @matthijs166,

Thanks for reporting this one, I suspect this is not a bug as such and is more than likely going to be a configuration issue. Are you just using the docker image or are you also using a reverse proxy and something like Cloudflare?

[matthijs166]

Hi @Joffcom,

It's indeed just the docker container. And I also suspect it to be a configuration issue.
Have used it with and without Cloudflare and it works on a VPS with Cloudflare.
I want it to run on the app platform in Digital Ocean so that there is no server management.

I also find this issue #959

Think it has something to do with EventSource...
But if it does there should be a clear error (in my opinion).

Also, the future is cloud and it is annoying that n8n cant is deployed on Digital Oceans app platform.
You cant configure the reverse proxy (what I imagine is also the case on other cloud app providers)

The thing is it should work right out of the box right?

[Joffcom]

Hey @matthijs166,

It should work fine on Digital Ocean I have an instance there myself and have used it previously with nginx, Caddy and Traefik with no issues, Are you using a reverse proxy and what environment options do you have set (and what are the values)?

If you check the browser console do you see any error messages?

[matthijs166]

Hey @Joffcom,

I also have it running on a droplet with an NGINX proxy in front.
And got it working locally without proxy.
But I need to run it on the app platform. (it's going to be part of a serverless infrastructure)

ENV values that are set are this (with current host/url as value):
N8N_HOST WEBHOOK_URL WEBHOOK_TUNNEL_URL VUE_APP_URL_BASE_API N8N_EDITOR_BASE_URL

[Joffcom]

Hey @matthijs166,

Perfect, the last few of these it was down to the nginx config. Can you share the config block for the proxy section.

[matthijs166]

Hey @Joffcom

Sorry for the wait but here is my digital ocean spec file.
All the settings of the app. You can also use this for reproducing the issue.

name: starfish-app
region: ams
services:
- envs:
  - key: N8N_HOST
    scope: RUN_AND_BUILD_TIME
    value: xxxxxxx.ondigitalocean.app
  - key: WEBHOOK_URL
    scope: RUN_AND_BUILD_TIME
    value: https://xxxxxxx.ondigitalocean.app
  - key: WEBHOOK_TUNNEL_URL
    scope: RUN_AND_BUILD_TIME
    value: https://xxxxxxx.ondigitalocean.app
  - key: VUE_APP_URL_BASE_API
    scope: RUN_AND_BUILD_TIME
    value: https://xxxxxxx.ondigitalocean.app
  - key: N8N_EDITOR_BASE_URL
    scope: RUN_AND_BUILD_TIME
    value: https://xxxxxxx.ondigitalocean.app
  http_port: 5678
  image:
    registry: n8nio
    registry_type: DOCKER_HUB
    repository: n8n
    tag: latest
  instance_count: 1
  instance_size_slug: basic-xxs
  name: n-8-nio-n-8-n
  routes:
  - path: /

[Joffcom]

Hey @matthijs166,

Ah I see the problem now, So this looks like the issue we see when n8n is used behind Cloudflare in proxy mode. Annoyingly Cloudflare attempts to cache everything so the response never fully makes it back to the browser. One way around this is to disable the cloudflare proxy option but in this case I can't see that as an option with a DO app.

I did find that when running the workflow it does actually run and logs correctly but I suspect you are going to see other issues if you don't have persisted storage or a database in use which I guess can be added later.

[matthijs166]

Hey @Joffcom,

There is no Cloudflare in this setup and in an other project where I have n8n on a VPS server it works fine with Cloudflare proxy mode.

Also if cache is the issue than this can be fixed by a simple Cache-Control header. Im trying to add this with my limited knowledge.

I have a more complex setup with persisted storage and a database so thats going to work fine in the final setup.

[Joffcom]

Hey @matthijs166,

When I set up a test using the Digital Ocean App and I viewed the headers there was one called "server" with a value of "cloudflare", I suspect Digital Ocean run their app service through Cloudflare. It could be that it isn't just a cache thing and maybe something else that Digital Ocean have set up is getting in the way of the browsers response but without knowing more about what they are doing it is hard to say.

I would say though if you have another VPS running and it is using Cloudflare in cache mode rather than direct then it won't be that and it will be something else DO are doing. Have you checked on our community support forum to see if anyone there has done a deployment on Digital Ocean Apps?

[matthijs166]

Hey @Joffcom,

First of all, I have to have n8n running on the app platform. We don't want to maintain the server and also want easy scaling.

The VPS instance is also running on a Cloudflare proxy so Cloudflare is not the issue.

I dived deeper into why I wasn't getting anything back:

Turns out the event stream (where all the responses are coming back from) didn't connect. (Top is DO and bottom a VPS with Cloudflare)

I also mentioned in a previous comment about issue #959 where they talk about setting the reverse proxy correct so that the event stream is not caught inside a buffer loop.

After digging into the code if found that n8n uses the sse-channel

n8n/packages/cli/src/Push.ts

Line 2 in eaf13cd

import sseChannel from 'sse-channel';

So in my case, I can't edit the reverse proxy settings (also something I don't want to worry about and in general hard to do with cloud setups). The solution that de documentation gives is that it needs to disable the proxy_buffering (on Nginx) to disable the buffer completely and so fixes the problem with the event stream getting stuck.

This seems to me like a hotfix... Why disable proxy_buffering completely for one failing request?

The logical solution is to somehow disable the proxy buffering for only the event stream.
https://serverfault.com/questions/801628/for-server-sent-events-sse-what-nginx-proxy-configuration-is-appropriate
But then there are maybe issues with keepalive

What I also want to propose is to convert the event stream to a WebSocket. It has its own schema wss:// and works way nicer with reverse proxys. Also, the package sse-channel is not actively maintained.

I think we can make the setup of the reverse proxy a lot easier if we fix this and will lower the bar to thinker with this. Im also prepared to make a blog about how to set up n8n within 2 minutes in Digital Ocean app platform with limited knowladige.

[Joffcom]

Hey @matthijs166,

That is a nice find, I have often wondered about the why but never found enough time to dig into it properly. In theory once this part is sorted we would be able to have a nice simple one click button to deploy as well.

What I will have to do from here is open a feature request internally to have this looked into and prioritised so we can schedule something for a future release. I have no idea what the timescales would be on this as we have our rough roadmap mostly in place for the last part of this year but who knows.

None of this is going to help get you up and running quickly though.

[matthijs166]

@Joffcom Awesome keep me uptodate!

Im trying to make a docker container with nginx build in.
Its a long shot but worth trying.

[matthijs166]

Are there any updates on this issue?

[Joffcom]

Hey @matthijs166,

Nothing yet as it is a feature request that isn't a quick change it has to go through prioritisation which can sometimes take a bit of depending on popularity and the current workflow of the team.

[matthijs166]

@Joffcom I made a PR that replaces the EventSource with a WebSocket #5082

[netroy]

@matthijs166 Websocket support is now available in 0.215.x, and can be enabled by setting the env variable N8N_PUSH_BACKEND to websocket.

Thanks for your contribution, and I hope this solves the connectivity issues 🤞🏽

[matthijs166]

@netroy awesome, I will use it immediately!

[alih70442]

I had the same issue, it fixed by using this nginx config code:

server {
    listen 80;
    server_name your-domain.com;

    location / {
        proxy_pass http://localhost:5678;  # Adjust this based on where n8n is running
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 86400;
        proxy_send_timeout 86400;
    }
}