Requests with too large headers fields are not logged

[michael-pxr]

Describe the bug
A request to the webp server that contains a large Cookie header will return status code 431 - Request Header Fields Too Large. However, this request in not shown in the logs of the webp server.

To Reproduce
Start the webp server:

docker run -p 3333:3333 webpsh/webp-server-go:0.9.11

Send a request with a large Cookie header:

curl -H "Cookie: $(printf 'a%.0s' {1..5000})" -v http://localhost:3333
*   Trying 127.0.0.1:3333...
* Connected to localhost (127.0.0.1) port 3333 (#0)
> GET / HTTP/1.1
> Host: localhost:3333
> User-Agent: curl/8.1.2
> Accept: */*
> Cookie: aaa...aaa                                [I've reduced the number of characters]
>
< HTTP/1.1 431 Request Header Fields Too Large
< Server: WebP Server Go
< Date: Thu, 12 Oct 2023 12:20:16 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 31
< Connection: close
<
* Closing connection 0
Request Header Fields Too Large

where printf 'a%.0s' {1..5000} prints the character 'a' 5000 times.

The webp server does not log this request.

Expected behavior
I expect the webp server to log the request that returns 431 - Request Header Fields Too Large.

Screenshots and logs
N/A

Environment (please complete the following information):

• OS: macOS Sonoma 14.0
• version or branch: 0.9.11

Additional context
N/A

[n0vad3v]

In what scenario is it necessary to send such a large header to the WebP Server Go?🤔

[michael-pxr]

The website, say example.com, has a large Cookie header. When visiting webp.example.com to request the webp server, the Cookie header is also sent by the browser. Our webp server is behind an nginx proxy and the status code 431 is prevented by not forwarding the Cookie header from nginx to the webp server.

Note, however, that this issue is not about the webp server returning 431. It is about the webp server not logging a request that results in status code 431.

[n0vad3v]

Okay, I got the reason for this.

WebP Server Go is using https://github.com/gofiber/fiber as HTTP framework, it has a constant called ReadBufferSize:

ReadBufferSize specifies the I/O buffer size in bytes for incoming messages.

It's reflecting the same variable in FastHTTP, with description below:

// Per-connection buffer size for requests' reading.
// This also limits the maximum header size.
// Increase this buffer if your clients send multi-KB RequestURIs
// and/or multi-KB headers (for example, BIG cookies).
//
// Default: 4096
ReadBufferSize int `json:"read_buffer_size"`

Since is by default 4096, a large cookie of more characters will fail on FastHTTP first, so Fiber and WebP Server Go won't be able to log this request.

---

A quick fix for this(if you can compile by yourself), is to change

var app = fiber.New(fiber.Config{
	ServerHeader:          "WebP Server Go",
	AppName:               "WebP Server Go",
	DisableStartupMessage: true,
	ProxyHeader:           "X-Real-IP",
})

to something like(You can adjust ReadBufferSize to something bigger that fits your needs):

var app = fiber.New(fiber.Config{
	ServerHeader:          "WebP Server Go",
	AppName:               "WebP Server Go",
	DisableStartupMessage: true,
	ProxyHeader:           "X-Real-IP",
	ReadBufferSize: 8192,
})

in webp-server.go file and then compile the server(https://docs.webp.sh/build/), we are planning to make it changeable by supplying ENV in future version, stay tuned.

[BennyThink]

This issue should be fixed in 0.9.12 now. You need to customize your own env READ_BUFFER_SIZE