Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(bug) Panic concurrent map writes #2758

Closed
natenho opened this issue Dec 6, 2024 · 2 comments · Fixed by #2759 or #2794
Closed

(bug) Panic concurrent map writes #2758

natenho opened this issue Dec 6, 2024 · 2 comments · Fixed by #2759 or #2794
Assignees
@thomaspoignant
Labels
bug Something isn't working

Comments

@natenho
Copy link

natenho commented Dec 6, 2024 •
edited
Loading

Observed behaviour

image

Here is the full log:

echo: http: superfluous response.WriteHeader call from github.com/labstack/echo/v4/middleware.(*ignorableWriter).WriteHeader (timeout.go:211)
	echo: http: superfluous response.WriteHeader call from github.com/labstack/echo/v4/middleware.(*ignorableWriter).WriteHeader (timeout.go:211)
	echo: http: superfluous response.WriteHeader call from github.com/labstack/echo/v4/middleware.(*ignorableWriter).WriteHeader (timeout.go:211)
	echo: http: superfluous response.WriteHeader call from github.com/labstack/echo/v4/middleware.(*ignorableWriter).WriteHeader (timeout.go:211)
	fatal error: concurrent map writes
	
	goroutine 170694896 [running]:
	net/textproto.MIMEHeader.Set(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/textproto/header.go:22
	net/http.Header.Set(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/header.go:40
	github.com/labstack/echo/v4.(*context).writeContentType(0x65e2f9?, {0x366d3e5, 0x10})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/context.go:246 +0xc5
	github.com/labstack/echo/v4.(*context).json(0xc0001340a0, 0xc8, {0x33882a0, 0xc0015add10}, {0x0, 0x0})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/context.go:503 +0x45
	github.com/labstack/echo/v4.(*context).JSON(0xc0001340a0, 0xc8, {0x33882a0, 0xc0015add10})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/context.go:513 +0xc7
	github.com/thomaspoignant/go-feature-flag/cmd/relayproxy/ofrep.(*EvaluateCtrl).Evaluate(0xc000135ea0, {0x4424fc8, 0xc0001340a0})
		/home/runner/work/go-feature-flag/go-feature-flag/cmd/relayproxy/ofrep/evaluate.go:123 +0x1462
	github.com/thomaspoignant/go-feature-flag/cmd/relayproxy/api.(*Server).addOFREPRoutes.WithConfig.func3.1({0x4424fc8, 0xc0001340a0})
		/home/runner/go/pkg/mod/github.com/pablor21/echo-etag/v4@v4.0.3/etag.go:69 +0x676
	github.com/labstack/echo/v4.(*Echo).add.func1({0x4424fc8, 0xc0001340a0})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/echo.go:587 +0x4b
	github.com/labstack/echo/v4/middleware.echoHandlerFuncWrapper.ServeHTTP({0xc0004f9e40, {0x4424fc8, 0xc0001340a0}, 0xc0006e5d70, 0x0, 0xc001166ba0}, {0x43ccf70, 0xc001041dc0}, 0xc001478d80)
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/middleware/timeout.go:165 +0x1ac
	net/http.(*timeoutHandler).ServeHTTP.func1()
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3601 +0x71
	created by net/http.(*timeoutHandler).ServeHTTP in goroutine 170675051
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3595 +0x2d9
	
	goroutine 1 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c4c40fd0, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0x3?, 0xc0005cefc0?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Accept(0xc0004e4280)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:611 +0x2ac
	net.(*netFD).accept(0xc0004e4280)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_unix.go:172 +0x29
	net.(*TCPListener).accept(0xc0005a5180)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/tcpsock_posix.go:159 +0x1e
	net.(*TCPListener).AcceptTCP(0xc0005a5180)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/tcpsock.go:314 +0x30
	github.com/labstack/echo/v4.tcpKeepAliveListener.Accept({0x44ac60?})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/echo.go:994 +0x17
	net/http.(*Server).Serve(0xc0001bcb40, {0x43ccee0, 0xc0000a0af8})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3260 +0x33e
	github.com/labstack/echo/v4.(*Echo).Start(0xc0006d2b48, {0xc000013fe0, 0xc})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/echo.go:691 +0xd2
	github.com/thomaspoignant/go-feature-flag/cmd/relayproxy/api.(*Server).Start(0xc00011b800)
		/home/runner/work/go-feature-flag/go-feature-flag/cmd/relayproxy/api/server.go:141 +0x518
	main.main()
		/home/runner/work/go-feature-flag/go-feature-flag/cmd/relayproxy/main.go:110 +0xb6d
	
	goroutine 17 [select]:
	go.opencensus.io/stats/view.(*worker).start(0xc000225500)
		/home/runner/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
	created by go.opencensus.io/stats/view.init.0 in goroutine 1
		/home/runner/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d
	
	goroutine 170682913 [sync.Cond.Wait]:
	sync.runtime_notifyListWait(0xc001df6c90, 0x14)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/sema.go:569 +0x159
	sync.(*Cond).Wait(0xc000cd7d70?)
		/opt/hostedtoolcache/go/1.22.8/x64/src/sync/cond.go:70 +0x85
	net/http.(*connReader).abortPendingRead(0xc000cd7d70)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:729 +0xa6
	net/http.(*response).finishRequest(0xc00185d960)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:1676 +0x87
	net/http.(*conn).serve(0xc000bae510, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2050 +0x62b
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170694767 [runnable]:
	net/http.(*connReader).startBackgroundRead.gowrap2()
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677
	runtime.goexit({})
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/asm_amd64.s:1695 +0x1
	created by net/http.(*connReader).startBackgroundRead in goroutine 170694766
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677 +0xba
	
	goroutine 170694655 [runnable]:
	net/http.(*connReader).startBackgroundRead.gowrap2()
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677
	runtime.goexit({})
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/asm_amd64.s:1695 +0x1
	created by net/http.(*connReader).startBackgroundRead in goroutine 170694654
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677 +0xba
	
	goroutine 37 [chan receive]:
	github.com/thomaspoignant/go-feature-flag.retreiveFlags({0x12a05f200, 0x1, 0x0, 0x0, 0xc000633150, {0x43ddf28, 0x6309400}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
		/home/runner/work/go-feature-flag/go-feature-flag/feature_flag.go:256 +0x30a
	github.com/thomaspoignant/go-feature-flag.retrieveFlagsAndUpdateCache({0x12a05f200, 0x1, 0x0, 0x0, 0xc000633150, {0x43ddf28, 0x6309400}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
		/home/runner/work/go-feature-flag/go-feature-flag/feature_flag.go:277 +0x58
	github.com/thomaspoignant/go-feature-flag.(*GoFeatureFlag).startFlagUpdaterDaemon(0xc00058b8c0)
		/home/runner/work/go-feature-flag/go-feature-flag/feature_flag.go:195 +0x11a
	created by github.com/thomaspoignant/go-feature-flag.New in goroutine 1
		/home/runner/work/go-feature-flag/go-feature-flag/feature_flag.go:123 +0xae5
	
	goroutine 170682976 [sync.Cond.Wait]:
	sync.runtime_notifyListWait(0xc0010a6210, 0x29)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/sema.go:569 +0x159
	sync.(*Cond).Wait(0xc001665230?)
		/opt/hostedtoolcache/go/1.22.8/x64/src/sync/cond.go:70 +0x85
	net/http.(*connReader).abortPendingRead(0xc001665230)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:729 +0xa6
	net/http.(*response).finishRequest(0xc000fdfce0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:1676 +0x87
	net/http.(*conn).serve(0xc000a4def0, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2050 +0x62b
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170674879 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c49cef28, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc000cfcc00?, 0xc001f24000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc000cfcc00, {0xc001f24000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc000cfcc00, {0xc001f24000?, 0xc001f81a98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc001b250a8, {0xc001f24000?, 0x0?, 0xc0016f19b8?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc0016f19b0, {0xc001f24000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc0012c8000)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc0012c8000, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc0006d95f0, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170676319 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c4a07c08, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc000778c80?, 0xc002521000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc000778c80, {0xc002521000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc000778c80, {0xc002521000?, 0xc002511a98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc0013c1d18, {0xc002521000?, 0x0?, 0xc001b0f508?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc001b0f500, {0xc002521000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc0019874a0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc0019874a0, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc000fbe120, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170676324 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c48f1308, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc000778f00?, 0xc00253b000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc000778f00, {0xc00253b000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc000778f00, {0xc00253b000?, 0xc00252fa98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc0013c1d40, {0xc00253b000?, 0x0?, 0xc000d88b78?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc000d88b70, {0xc00253b000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc0012c86c0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc0012c86c0, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc000fbe480, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170682310 [sync.Cond.Wait]:
	sync.runtime_notifyListWait(0xc001194a10, 0x15)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/sema.go:569 +0x159
	sync.(*Cond).Wait(0xc001b9e750?)
		/opt/hostedtoolcache/go/1.22.8/x64/src/sync/cond.go:70 +0x85
	net/http.(*connReader).abortPendingRead(0xc001b9e750)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:729 +0xa6
	net/http.(*response).finishRequest(0xc0010e4c40)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:1676 +0x87
	net/http.(*conn).serve(0xc0019f33b0, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2050 +0x62b
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170682981 [runnable]:
	net/http.Header.Clone(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/header.go:114
	net/http.(*response).WriteHeader(0xc000c31a40, 0x1f7)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:1188 +0x795
	github.com/labstack/echo/v4/middleware.(*ignorableWriter).WriteHeader(0xc0018b6da0?, 0xc0018b6cec?)
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/middleware/timeout.go:211 +0xae
	net/http.(*timeoutHandler).ServeHTTP(0xc0012f1700, {0x43cca68, 0xc001539f00}, 0xc00172b320)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3628 +0x4a2
	github.com/thomaspoignant/go-feature-flag/cmd/relayproxy/api.(*Server).initRoutes.TimeoutWithConfig.TimeoutConfig.ToMiddleware.func7.1({0x4424fc8, 0xc000134960})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/middleware/timeout.go:126 +0x1ed
	github.com/thomaspoignant/go-feature-flag/cmd/relayproxy/api.(*Server).initRoutes.Recover.RecoverWithConfig.func6.1({0x4424fc8, 0xc000134960})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/middleware/recover.go:130 +0x114
	github.com/thomaspoignant/go-feature-flag/cmd/relayproxy/api.(*Server).initRoutes.VersionHeader.func5.1({0x4424fc8, 0xc000134960})
		/home/runner/work/go-feature-flag/go-feature-flag/cmd/relayproxy/api/middleware/version.go:13 +0x73
	github.com/labstack/echo/v4/middleware.CORSWithConfig.func1.1({0x4424fc8, 0xc000134960})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/middleware/cors.go:203 +0x463
	go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho.Middleware.func1.1({0x4424fc8, 0xc000134960})
		/home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho@v0.56.0/echo.go:82 +0x707
	github.com/labstack/echo-contrib/echoprometheus.MiddlewareConfig.ToMiddleware.func3.1({0x4424fc8, 0xc000134960})
		/home/runner/go/pkg/mod/github.com/labstack/echo-contrib@v0.17.1/echoprometheus/prometheus.go:245 +0x282
	github.com/thomaspoignant/go-feature-flag/cmd/relayproxy/api.(*Server).initRoutes.BodyDumpWithConfig.func4.1({0x4424fc8, 0xc000134960})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/middleware/body_dump.go:64 +0xd5
	github.com/labstack/echo/v4/middleware.RequestLoggerConfig.ToMiddleware.func1.1({0x4424fc8, 0xc000134960})
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/middleware/request_logger.go:286 +0x16b
	github.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc0006d2b48, {0x43cd0c0, 0xc000c31a40}, 0xc00172b0e0)
		/home/runner/go/pkg/mod/github.com/labstack/echo/v4@v4.12.0/echo.go:674 +0x327
	net/http.serverHandler.ServeHTTP({0x43b9b40?}, {0x43cd0c0?, 0xc000c31a40?}, 0x6?)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3142 +0x8e
	net/http.(*conn).serve(0xc00102c3f0, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2044 +0x5e8
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170694912 [IO wait]:
	internal/poll.runtime_pollWait(0x7fe9c491d800, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc000fbc100?, 0xc0019ecc11?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc000fbc100, {0xc0019ecc11, 0x1, 0x1})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc000fbc100, {0xc0019ecc11?, 0xc0006e5d70?, 0x0?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc0000a16d8, {0xc0019ecc11?, 0xc0006e5d70?, 0x0?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).backgroundRead(0xc0019ecc00)
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:681 +0x37
	created by net/http.(*connReader).startBackgroundRead in goroutine 170694911
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677 +0xba
	
	goroutine 170675038 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c48e02d0, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc001061e00?, 0xc000b31000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc001061e00, {0xc000b31000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc001061e00, {0xc000b31000?, 0xc0014dda98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc001b255a0, {0xc000b31000?, 0x0?, 0xc001201298?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc001201290, {0xc000b31000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc0009e81e0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc0009e81e0, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc001388750, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170675027 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c48e0d78, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc001061800?, 0xc000fb7000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc001061800, {0xc000fb7000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc001061800, {0xc000fb7000?, 0xc001891a98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc001b25548, {0xc000fb7000?, 0x0?, 0xc001838b18?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc001838b10, {0xc000fb7000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc0009e9200)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc0009e9200, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc001388120, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170694893 [runnable]:
	net/http.(*connReader).startBackgroundRead.gowrap2()
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677
	runtime.goexit({})
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/asm_amd64.s:1695 +0x1
	created by net/http.(*connReader).startBackgroundRead in goroutine 170694892
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677 +0xba
	
	goroutine 170694649 [runnable]:
	net/http.(*connReader).startBackgroundRead.gowrap2()
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677
	runtime.goexit({})
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/asm_amd64.s:1695 +0x1
	created by net/http.(*connReader).startBackgroundRead in goroutine 170694648
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677 +0xba
	
	goroutine 170694850 [runnable]:
	net/http.(*connReader).startBackgroundRead.gowrap2()
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677
	runtime.goexit({})
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/asm_amd64.s:1695 +0x1
	created by net/http.(*connReader).startBackgroundRead in goroutine 170694849
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677 +0xba
	
	goroutine 170668808 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c49ac120, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc001215580?, 0xc000c6d000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc001215580, {0xc000c6d000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc001215580, {0xc000c6d000?, 0xc000f6ba98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc0013c0950, {0xc000c6d000?, 0x0?, 0xc0018f9628?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc0018f9620, {0xc000c6d000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc0012585a0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc0012585a0, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc000d2bcb0, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170675018 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c4928508, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc001061380?, 0xc000e91000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc001061380, {0xc000e91000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc001061380, {0xc000e91000?, 0xc0023d3a98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc001b25500, {0xc000e91000?, 0x0?, 0xc00117ff28?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc00117ff20, {0xc000e91000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc00117a7e0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc00117a7e0, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc001b09a70, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170676307 [IO wait]:
	internal/poll.runtime_pollWait(0x7fe9c4966658, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc000778600?, 0xc002459000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc000778600, {0xc002459000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc000778600, {0xc002459000?, 0xc00244fa98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc0013c1cb8, {0xc002459000?, 0x0?, 0xc000287898?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc000287890, {0xc002459000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc000c124e0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc000c124e0, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc00102d680, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170694789 [runnable]:
	net/http.(*connReader).startBackgroundRead.gowrap2()
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677
	runtime.goexit({})
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/asm_amd64.s:1695 +0x1
	created by net/http.(*connReader).startBackgroundRead in goroutine 170694788
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677 +0xba
	
	goroutine 170672820 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c49d0c38, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc0017f0180?, 0xc0003a6000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc0017f0180, {0xc0003a6000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc0017f0180, {0xc0003a6000?, 0xc000dbba98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc0013c00b0, {0xc0003a6000?, 0x0?, 0xc0010aa1e8?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc0010aa1e0, {0xc0003a6000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc000b7f800)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc000b7f800, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc0011bf0e0, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170694761 [runnable]:
	net/http.(*connReader).startBackgroundRead.gowrap2()
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677
	runtime.goexit({})
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/asm_amd64.s:1695 +0x1
	created by net/http.(*connReader).startBackgroundRead in goroutine 170694760
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677 +0xba
	
	goroutine 170682264 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c49ac310, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc001340b80?, 0xc0014c6000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc001340b80, {0xc0014c6000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc001340b80, {0xc0014c6000?, 0xc002045a98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc000fa5478, {0xc0014c6000?, 0x0?, 0xc0019f7268?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc0019f7260, {0xc0014c6000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc000b97bc0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc000b97bc0, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc0019f22d0, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170682183 [IO wait]:
	internal/poll.runtime_pollWait(0x7fe9c498cd50, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc00133e280?, 0xc000d44000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
	internal/poll.(*FD).Read(0xc00133e280, {0xc000d44000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:164 +0x27a
	net.(*netFD).Read(0xc00133e280, {0xc000d44000?, 0xc00173da98?, 0x4f1e85?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_posix.go:55 +0x25
	net.(*conn).Read(0xc000fa51f0, {0xc000d44000?, 0x0?, 0xc0006e5208?})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/net.go:185 +0x45
	net/http.(*connReader).Read(0xc0006e5200, {0xc000d44000, 0x1000, 0x1000})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:789 +0x14b
	bufio.(*Reader).fill(0xc000fce4e0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:110 +0x103
	bufio.(*Reader).Peek(0xc000fce4e0, 0x4)
		/opt/hostedtoolcache/go/1.22.8/x64/src/bufio/bufio.go:148 +0x53
	net/http.(*conn).serve(0xc0018a9050, {0x43ddfd8, 0xc000587320})
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:2079 +0x749
	created by net/http.(*Server).Serve in goroutine 1
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3290 +0x4b4
	
	goroutine 170694827 [runnable]:
	net/http.(*connReader).startBackgroundRead.gowrap2()
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677
	runtime.goexit({})
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/asm_amd64.s:1695 +0x1
	created by net/http.(*connReader).startBackgroundRead in goroutine 170694826
		/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:677 +0xba
	
	goroutine 170675019 [runnable]:
	internal/poll.runtime_pollWait(0x7fe9c4a081d8, 0x72)
		/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
	internal/poll.(*pollDesc).wait(0xc001061400?, 0xc000eaf000?, 0x0)
		/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
	internal/poll.(*pollDesc).waitRead(...)

Expected Behavior

No panics at all

  • There should be a safe net recovery
  • The log should be structured even in that case

Steps to reproduce

Not easily reproducible
@natenho natenho added bug Something isn't working needs-triage A priority should be added to the issue labels Dec 6, 2024
@thomaspoignant
Copy link
Owner

thomaspoignant commented Dec 6, 2024 •
edited
Loading

Hey @natenho thanks for raising it, this is a behavior I have never seen.
I will try to understand what is happening here.

When it comes to panic we are using the recover middleware that should deal with that, I am not sure why it did not work that time.

Small question, does it kill the relay-proxy or is it staying up and running?

@thomaspoignant thomaspoignant added p1 High priority and removed needs-triage A priority should be added to the issue p1 High priority labels Dec 6, 2024
@thomaspoignant
Copy link
Owner

After looking at it, the issue seems to be in the Timeout middleware we are using https://github.com/labstack/echo/blob/3b017855b4d331002e2b8b28e903679b875ae3e9/middleware/timeout.go#L211

In the source there is this warning:

// WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
// WARNING: Timeout middleware causes more problems than it solves.
// WARNING: This middleware should be first middleware as it messes with request Writer and could cause data race if
// 					it is in other position

And for now the middleware is not on the 1st position.

I will open a PR to move the middleware as the 1st position.

@thomaspoignant thomaspoignant mentioned this issue Dec 6, 2024
Merged
2 tasks
thomaspoignant added a commit that referenced this issue Dec 13, 2024
@thomaspoignant
fix(timeout): Fix the swagger issue introduce after fixing #2758
5716293 
Signed-off-by: Thomas Poignant <thomas.poignant@gofeatureflag.org>
@thomaspoignant thomaspoignant mentioned this issue Dec 13, 2024
Merged
2 tasks
thomaspoignant added a commit that referenced this issue Dec 13, 2024
@thomaspoignant
fix(timeout): Fix the swagger issue introduce after fixing #2758
491023c 
Signed-off-by: Thomas Poignant <thomas.poignant@gofeatureflag.org>
kodiakhq bot pushed a commit that referenced this issue Dec 13, 2024
@thomaspoignant
fix(timeout): Fix the swagger issue introduce after fixing #2758 (#2794)
8ae0c6d 
Signed-off-by: Thomas Poignant <thomas.poignant@gofeatureflag.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thomaspoignant thomaspoignant

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
2 participants
@natenho @thomaspoignant