Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caddy v2 - issues with dozzle, continuous spamming log reading: context canceled #2844

Closed
C8opmBM opened this issue Mar 23, 2024 · 10 comments
Closed
Assignees
Labels
enhancement New feature or request

Comments

@C8opmBM
Copy link

C8opmBM commented Mar 23, 2024

Is your feature request related to a problem? Please describe.

I'm using caddy and also using dozzle on my machines, which I love deeply.
However, they have a love-hate relationship, since using dozzle, my caddy log continuously spams some error I will show below. Do note this error is not new, I've had it for years.
I manage to fix all other containers, but I cannot with dozzle.
Other containers which had output this error were fixed by removing these headers which I previously had included.
But with dozzle, I don't have them, still the error is present.

(headers) {
        header {
                Permissions-Policy interest-cohort=()
                Strict-Transport-Security max-age=31536000;
                X-Content-Type-Options nosniff
                X-Frame-Options SAMEORIGIN
                Content-Security-Policy upgrade-insecure-requests
                Referrer-Policy strict-origin-when-cross-origin
                -Server
        }
}

Can you suggest a solution?

I think it's related to the headers (gzip) or something.

I've searched about this error in google, every time it pops up, dozzle is involved, mostly is a harmless error, but I narrowed it down to dozzle being the only container which still triggers it. (other offenders were fixed by removing my added headers in caddy. However, dozzle naked, without any header added, still won't stop the spam.

This is the log error, appearing everytime I click on a dozzle log container.
My guess is add Accept-Encoding=[ "gzip, deflate, br, zstd" ] but I could be wrong.

add duration=0.047730681
add error=reading: context canceled
add level=error
add logger=http.handlers.reverse_proxy
add msg=aborting with incomplete response
request=
add client_ip=192.168.1.153
headers=
add Accept=[ "text/event-stream" ]
add Accept-Encoding=[ "gzip, deflate, br, zstd" ]
add Accept-Language=[ "en-US,en;q=0.5" ]
add Cache-Control=[ "no-cache" ]
add Cookie=[ ]
add Referer=[ "https://log.domain.date/container/5b5975274b0d" ]
add Sec-Ch-Ua=[ ""Brave";v="123", "Not:A-Brand";v="8", "Chromium";v="123"" ]
add Sec-Ch-Ua-Mobile=[ "?0" ]
add Sec-Ch-Ua-Platform=[ ""Windows"" ]
add Sec-Fetch-Dest=[ "empty" ]
add Sec-Fetch-Mode=[ "cors" ]
add Sec-Fetch-Site=[ "same-origin" ]
add Sec-Gpc=[ "1" ]
add User-Agent=[ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" ]
add X-Forwarded-For=[ "192.168.1.153" ]
add X-Forwarded-Host=[ "log.domain.date" ]
add X-Forwarded-Proto=[ "https" ]
add host=log.inva.date
add method=GET
add proto=HTTP/2.0
add remote_ip=192.168.1.153
add remote_port=50591
tls=
add cipher_suite=4867
add proto=h2
add resumed=false
add server_name=log.inva.date
add version=772
add uri=/api/logs/stream/localhost/5b5975274b0d?stdout=1&stderr=1
add ts=1711181361.8220081
add upstream=dozzle:8080

Describe the solution you'd like
To not have the error every time.

Additional context
This is my setup with dozzle:

Caddy


{
        # Global options block
        order crowdsec first
        crowdsec {
                api_url http://crowdsec:8080/
                api_key {$BOUNCER_CADDY_TOKEN}
                ticker_interval 15s
        }
}

(cloudflare-tls) {
        tls {$EMAIL} {
                dns cloudflare {$CLOUDFLARE_API_TOKEN}
        }
}
log.{$DOMAIN} {
        reverse_proxy dozzle:8080
        import cloudflare-tls
}

Docker

  dozzle:
    container_name: dozzle
    image: amir20/dozzle:latest
    profiles:
      - monitor
    depends_on:
      - socky_proxy
    networks:
      - system
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped

x

@C8opmBM C8opmBM added the enhancement New feature or request label Mar 23, 2024
@amir20
Copy link
Owner

amir20 commented Mar 23, 2024

Interesting. I haven't been able to reproduce this. I used Caddy for a while before moving to Traefik.

I don't think Accept-Encoding=[ "gzip, deflate, br, zstd" ] is the problem. Those are headers from the browser. I imagine the problem is server side events which is a long poll HTTP call. I found caddyserver/caddy#1925 which suggests is a problem on Caddy. I suspect Caddy doesn't like a long connection.

Would you be able to provide a minimum example for me to test it?

Also have you tried disabling proxy buffering as suggested caddyserver/caddy#677 (comment)?

@amir20
Copy link
Owner

amir20 commented Mar 23, 2024

caddyserver/caddy#4500

Hmm seems like this is expected with SSE. So maybe there isn't anything to fix.

@C8opmBM
Copy link
Author

C8opmBM commented Mar 23, 2024

So you have used caddy before and had no error like this.. I don't know I've always have it.
So the weird thing is, there were other containers appearing in the caddy log with similar errors/aborted with incomplete, as said.
Removing my added headers (in Caddyfile) fixed them all, except for dozzle.

So to understand, that error is being printed only when I'm inside dozzle. Clicking on any container (I have about 50) will print one error. Clicking 5 times = 5 errors. Clicking back and forth between containers will print erorrs all the time.
If I kill dozzle and check logs, there are no more errors printed (eg in docker compose logs --tail 30 --timestamps caddy)

I got used to ignore the caddy container alltogether cause all I see is this error. But decided to post here, just in case there'll be a fix :)

As for those headers...
I don't know how to translate those nginx proxy buffering commands to caddy.

Thanks for your reply. In the end it's no big deal, maybe this should have been addressed by caddy long ago, but they choose to ignore it. If I know how to filter it, I would.

@amir20
Copy link
Owner

amir20 commented Mar 23, 2024

Nice I was able to reproduce it finally. I agree it is annoying. If there is a fix I can do on my side then I'll fix it. Let me take a look.

@amir20
Copy link
Owner

amir20 commented Mar 23, 2024

Alright, I did a lot research. Unfortunately this is an error that is being thrown at language level inside Caddy. https://caddy.community/t/filter-out-noisy-logs-aborting-with-incomplete-response/12834/3 suggests to just grep them out. There isn't anything I can do on my side to avoid these logs. It will have to be done from Caddy. I wish at the very least the errors could be moved to debug.

@amir20 amir20 closed this as not planned Won't fix, can't repro, duplicate, stale Mar 23, 2024
@C8opmBM
Copy link
Author

C8opmBM commented Mar 24, 2024

Amir, thanks for looking into it. I didn't have my hopes high either, knowing this was around for such long time.
I like caddy too much, at a time I was interested to switch to traefik also (many tutorials involved traefik), but for my small homeserver caddy is great.

Have a peaceful sunday and thanks for your lovely dozzle! It rocks (btw I read the docs after a long time and added forward auth envs and discovered I can restart/stop containers as well) 🥇 So nicee!

      DOZZLE_ENABLE_ACTIONS: true
      DOZZLE_AUTH_PROVIDER: forward-proxy

@amir20
Copy link
Owner

amir20 commented Mar 24, 2024

By the way, the reason why you see this error with Dozzle is that Dozzle uses server side events. Caddy throws this error every time a web socket or SSE is reset. Other apps don't use this type of connection as often.

@C8opmBM
Copy link
Author

C8opmBM commented Apr 30, 2024

There is no fix, it should come from caddy, they ve been aware of it for quite a while, but it's not an issue in their view.

@xd003
Copy link

xd003 commented Sep 3, 2024

Is there still no way to hide this error in Caddy ?

@C8opmBM
Copy link
Author

C8opmBM commented Sep 3, 2024

apparently not. If you find one, do post :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants