Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gitea 1.17-dev crashing Out of memory #19793

Closed
gboddin opened this issue May 24, 2022 · 11 comments · Fixed by #19799
Closed

Gitea 1.17-dev crashing Out of memory #19793

gboddin opened this issue May 24, 2022 · 11 comments · Fixed by #19799
Labels
issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP issue/regression Indicates a previously functioning feature or behavior that has broken or regressed after a change type/bug

Comments

@gboddin
Copy link

gboddin commented May 24, 2022

Description

Gitea starts what seems like an infinite loop and runs out of memory,

Issues happend 3 times, randomly browsing the instance, no pattern noticed.

Will update if a pattern is found

Gitea Version

7671a33

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

https://gist.github.com/gboddin/7956cd755b034c1b399672c1abc6ab2f

Screenshots

Nothing to see, gitea hangs until OOM kills it

Git Version

git version 2.30.2

Operating System

debian-bullseye

How are you running Gitea?

  • From binary local build 7671a33 (main while reporting )
  • Built with TAGS="bindata sqlite sqlite_unlock_notify" make build
  • Handled by systemd service

Database

SQLite

@gboddin
Copy link
Author

gboddin commented May 24, 2022

Suspecting /user/events ( which looks like a stream ) after checking logs at https://gist.github.com/gboddin/7956cd755b034c1b399672c1abc6ab2f

@gboddin
Copy link
Author

gboddin commented May 24, 2022

Happened 3 more times when checking an issue and hitting the browser Back button

@gboddin gboddin changed the title Gitea crashing Out of memory Gitea 1.17-dev crashing Out of memory May 24, 2022
@gboddin
Copy link
Author

gboddin commented May 24, 2022

Ok I crashed the test instance.
Was playing around https://try.gitea.io/gbo/testreoi/issues and linked it to milestone

@lafriks lafriks added the issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP label May 24, 2022
@a1012112796 a1012112796 modified the milestone: 1.17.0 May 24, 2022
@gboddin
Copy link
Author

gboddin commented May 24, 2022

Commented
https://github.com/go-gitea/gitea/blob/4266bd924bb8fff20271c17f8989b39b04ef6673/routers/web/web.go#L217..218
as a work-around for now.

We needed packages more than notifications and stopwatches :)

Let us know if we can provide anything else to help with 1.17 !

@silverwind
Copy link
Member

silverwind commented May 24, 2022

Can reproduce locally as well by just starting my dev instance gitea. A lot of GetNotificationCount: context canceled spam in the log and finally a crash:

runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc37e9806c8 stack=[0xc37e980000, 0xc39e9800002022/05/24 20:26:56 ...user/notification.go:39:func1() [E] [628d234b] GetNotificationCount: context canceled
]
fatal error: stack overflow

runtime stack:
runtime.throw({0x645c942?, 0x81c7420?})
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/panic.go:992 +0x71
runtime.newstack()
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/stack.go:1101 +0x5cc
runtime.morestack()
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:547 +0x8b
goroutines of crash
goroutine 1320 [running]:
reflect.funcLayout(0x5f245a0?, 0x0?)
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/type.go:3066 +0x6af fp=0xc37e9806d8 sp=0xc37e9806d0 pc=0x40989ef
reflect.Value.call({0x5f245a0?, 0x66293f8?, 0x198?}, {0x6364751, 0x4}, {0xc2dfffd470, 0x1, 0x6b1c700?})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:442 +0x450 fp=0xc37e980cc8 sp=0xc37e9806d8 pc=0x4099f30
reflect.Value.Call({0x5f245a0?, 0x66293f8?, 0xc2e2e38e60?}, {0xc2dfffd470, 0x1, 0x1})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:339 +0xbf fp=0xc37e980d40 sp=0xc37e980cc8 pc=0x409989f
text/template.safeCall({0x5f245a0?, 0x66293f8?, 0xc0067d8a80?}, {0xc2dfffd470?, 0x6b1c700?, 0x5f7b880?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/funcs.go:368 +0x9d fp=0xc37e980de0 sp=0xc37e980d40 pc=0x455739d
text/template.(*state).evalCall(0xc37e981398, {0x5fc09e0?, 0xc0067e34a0?, 0x2?}, {0x5f245a0?, 0x66293f8?, 0x1?}, 0x0, {0x6ae5338, 0xc003e7f350}, ...)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:781 +0x87c fp=0xc37e980f40 sp=0xc37e980de0 pc=0x455229c
text/template.(*state).evalFunction(0xc37e981398, {0x5fc09e0?, 0xc0067e34a0?, 0x40e807f?}, 0x40ec620?, {0x6ae5338, 0xc003e7f350}, {0xc0050cce50, 0x1, 0x1}, ...)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:601 +0x209 fp=0xc37e981038 sp=0xc37e980f40 pc=0x4550c69
text/template.(*state).evalCommand(0xc37e981398, {0x5fc09e0?, 0xc0067e34a0?, 0xc003ece000?}, 0xc37e981160?, {0x5eff460?, 0xc0067d8a80?, 0x6ac5900?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:488 +0xea fp=0xc37e981108 sp=0xc37e981038 pc=0x454f6ca
text/template.(*state).evalPipeline(0xc37e981398, {0x5fc09e0?, 0xc0067e34a0?, 0x3?}, 0xc00382dda0)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:457 +0x145 fp=0xc37e9811f8 sp=0xc37e981108 pc=0x454f265
text/template.(*state).walk(0xc37e981398, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, {0x6ae51d0?, 0xc0038491a0})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:261 +0x225 fp=0xc37e981298 sp=0xc37e9811f8 pc=0x454d985
text/template.(*state).walk(0xc37e981398, {0x5fc09e0?, 0xc0067e34a0?, 0x1?}, {0x6ae5530?, 0xc0038490b0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e981338 sp=0xc37e981298 pc=0x454db75
text/template.(*state).walkTemplate(0xc37e9815c0, {0x5fc09e0?, 0xc0067e34a0?, 0xc2dffa9930?}, 0xc005d5b540)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:439 +0x285 fp=0xc37e9813f0 sp=0xc37e981338 pc=0x454f0c5
text/template.(*state).walk(0xc37e9815c0, {0x5fc09e0?, 0xc0067e34a0?, 0x10000?}, {0x6ae56e0?, 0xc005d5b540?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:279 +0x119 fp=0xc37e981490 sp=0xc37e9813f0 pc=0x454d879
text/template.(*state).walk(0xc37e9815c0, {0x5fc09e0?, 0xc0067e34a0?, 0x5fbca20?}, {0x6ae5530?, 0xc005d6c7b0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e981530 sp=0xc37e981490 pc=0x454db75
text/template.(*Template).execute(0xc005d5b500, {0x6acd840?, 0xc4529c9890?}, {0x5fc09e0?, 0xc0067e34a0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:218 +0x2c5 fp=0xc37e981620 sp=0xc37e981530 pc=0x454d1a5
text/template.(*Template).Execute(...)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:201
html/template.(*Template).ExecuteTemplate(0xc00572c220?, {0x6acd840, 0xc4529c9890}, {0x63f3d2a?, 0x0?}, {0x5fc09e0, 0xc0067e34a0})
  /usr/local/Cellar/go/1.18.2/libexec/src/html/template/template.go:139 +0x6a fp=0xc37e981658 sp=0xc37e981620 pc=0x48774ea
github.com/unrolled/render.HTML.Render({{{0xc465df43c0, 0x18}, 0x1f4}, {0x63f3d2a, 0xa}, 0xc00572ea80, {0x6ad9338, 0xc00572c220}}, {0x11309b80, 0xc00679cdc0}, ...)
  /Users/silverwind/go/pkg/mod/github.com/unrolled/render@v1.4.1/engine.go:94 +0x145 fp=0xc37e9816f0 sp=0xc37e981658 pc=0x511f405
github.com/unrolled/render.(*HTML).Render(0x0?, {0x11309b80?, 0xc00679cdc0?}, {0x5fc09e0?, 0xc0067e34a0?})
  <autogenerated>:1 +0xdd fp=0xc37e9817a0 sp=0xc37e9816f0 pc=0x5123f3d
github.com/unrolled/render.(*Render).Render(0xc004bc8180, {0x11309b80, 0xc00679cdc0}, {0x6ad5160?, 0xc2df4255c0?}, {0x5fc09e0?, 0xc0067e34a0?})
  /Users/silverwind/go/pkg/mod/github.com/unrolled/render@v1.4.1/render.go:413 +0x50 fp=0xc37e9817f8 sp=0xc37e9817a0 pc=0x5122630
github.com/unrolled/render.(*Render).HTML(0xc004bc8180, {0x11309b80, 0xc00679cdc0}, 0x1f4, {0x63f3d2a, 0xa}, {0x5fc09e0, 0xc0067e34a0}, {0x0, 0x0, ...})
  /Users/silverwind/go/pkg/mod/github.com/unrolled/render@v1.4.1/render.go:470 +0x49c fp=0xc37e981990 sp=0xc37e9817f8 pc=0x5122cdc
code.gitea.io/gitea/modules/context.(*Context).HTML(0xc0067d6dd0, 0x1f4, {0x63f3d2a, 0xa})
  /Users/silverwind/git/gitea/modules/context/context.go:226 +0x25c fp=0xc37e981a30 sp=0xc37e981990 pc=0x55e389c
code.gitea.io/gitea/modules/context.(*Context).serverErrorInternal(0xc0067d6dd0, {0x64c01bb?, 0x14?}, {0x6acf040?, 0xc00003c230?})
  /Users/silverwind/git/gitea/modules/context/context.go:304 +0x214 fp=0xc37e981a88 sp=0xc37e981a30 pc=0x55e4334
code.gitea.io/gitea/modules/context.(*Context).ServerError(...)
  /Users/silverwind/git/gitea/modules/context/context.go:286
code.gitea.io/gitea/routers/web/user.GetNotificationCount.func1()
  /Users/silverwind/git/gitea/routers/web/user/notification.go:39 +0x57 fp=0xc37e981ac8 sp=0xc37e981a88 pc=0x5bd9c77
runtime.call16(0xc003e555c0, 0xc00612e850, 0x0, 0x0, 0x0, 0x0, 0xc37e981ff8)
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:701 +0x49 fp=0xc37e981ae8 sp=0xc37e981ac8 pc=0x406ca29
runtime.reflectcall(0x0?, 0x0?, 0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
  <autogenerated>:1 +0x3c fp=0xc37e981b28 sp=0xc37e981ae8 pc=0x4070e7c
reflect.Value.call({0x5ef4d60?, 0xc00612e850?, 0xc003c143d8?}, {0x6364751, 0x4}, {0x84402c8, 0x0, 0x487f8ed?})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:556 +0x845 fp=0xc37e982118 sp=0xc37e981b28 pc=0x409a325
reflect.Value.Call({0x5ef4d60?, 0xc00612e850?, 0xc002c2d2cb?}, {0x84402c8, 0x0, 0x0})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:339 +0xbf fp=0xc37e982190 sp=0xc37e982118 pc=0x409989f
text/template.safeCall({0x5ef4d60?, 0xc00612e850?, 0xc00406cd60?}, {0x84402c8?, 0x2?, 0x0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/funcs.go:368 +0x9d fp=0xc37e982230 sp=0xc37e982190 pc=0x455739d
text/template.call({0x5f7b880?, 0xc2e2e38d00?, 0xc002a1d500?}, {0x84402c8, 0x0, 0xc00406cd70?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/funcs.go:353 +0x3d5 fp=0xc37e982328 sp=0xc37e982230 pc=0x45570d5
runtime.call64(0xc003e55530, 0x662a350, 0x0, 0x0, 0x0, 0x30, 0xc37e982888)
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:703 +0x49 fp=0xc37e982378 sp=0xc37e982328 pc=0x406cb69
runtime.reflectcall(0x5ed4ae0?, 0xc2dfffd1b8?, 0x8?, 0x64a96de?, 0x0?, 0x12?, 0x5ed4ae0?)
  <autogenerated>:1 +0x3c fp=0xc37e9823b8 sp=0xc37e982378 pc=0x4070e7c
reflect.Value.call({0x5facd00?, 0x662a350?, 0x6330240?}, {0x6364751, 0x4}, {0xc2dfffd188, 0x1, 0x5fc09e0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:556 +0x845 fp=0xc37e9829a8 sp=0xc37e9823b8 pc=0x409a325
reflect.Value.Call({0x5facd00?, 0x662a350?, 0xc0067e34a0?}, {0xc2dfffd188, 0x1, 0x1})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:339 +0xbf fp=0xc37e982a20 sp=0xc37e9829a8 pc=0x409989f
text/template.safeCall({0x5facd00?, 0x662a350?, 0xc0067e34a0?}, {0xc2dfffd188?, 0x6b1c700?, 0x6330240?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/funcs.go:368 +0x9d fp=0xc37e982ac0 sp=0xc37e982a20 pc=0x455739d
text/template.(*state).evalCall(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0xc2e2e38cc0?}, {0x5facd00?, 0x662a350?, 0x98?}, 0x1, {0x6ae5338, 0xc00380d6e0}, ...)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:781 +0x87c fp=0xc37e982c20 sp=0xc37e982ac0 pc=0x455229c
text/template.(*state).evalFunction(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0x17?}, 0x6ae5458?, {0x6ae5338, 0xc00380d6e0}, {0xc0054bb600, 0x2, 0x2}, ...)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:601 +0x209 fp=0xc37e982d18 sp=0xc37e982c20 pc=0x4550c69
text/template.(*state).evalCommand(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0xffffffffffffffff?}, 0x5fc09e0?, {0x5fdd4e0?, 0x84402c8?, 0x5fc09e0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:488 +0xea fp=0xc37e982de8 sp=0xc37e982d18 pc=0x454f6ca
text/template.(*state).evalPipeline(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0xc2e2e38ce0?}, 0xc00382c4e0)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:457 +0x145 fp=0xc37e982ed8 sp=0xc37e982de8 pc=0x454f265
text/template.(*state).walk(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0x5ef4d60?}, {0x6ae51d0?, 0xc00380d770})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:261 +0x225 fp=0xc37e982f78 sp=0xc37e982ed8 pc=0x454d985
text/template.(*state).walk(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0x82?}, {0x6ae5530?, 0xc00380d680?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e983018 sp=0xc37e982f78 pc=0x454db75
text/template.(*state).walkIfOrWith(0xc37e983690, 0xa, {0x5fc09e0?, 0xc0067e34a0?, 0xc00382c3c0?}, 0x5fc10a0?, 0xc003e54a50?, 0x0)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:304 +0x1f8 fp=0xc37e9830e0 sp=0xc37e983018 pc=0x454ddd8
text/template.(*state).walk(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0x5ee6720?}, {0x6ae54e8?, 0xc005beaac0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:271 +0x1d6 fp=0xc37e983180 sp=0xc37e9830e0 pc=0x454d936
text/template.(*state).walk(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0xc0067e34a0?}, {0x6ae5530?, 0xc0037d3b60?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e983220 sp=0xc37e983180 pc=0x454db75
text/template.(*state).walkIfOrWith(0xc37e983690, 0xa, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, 0x6364751?, 0xc4529c91a0?, 0xc0038fe120)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:304 +0x1f8 fp=0xc37e9832e8 sp=0xc37e983220 pc=0x454ddd8
text/template.(*state).walk(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0x0?}, {0x6ae54e8?, 0xc005afac40?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:271 +0x1d6 fp=0xc37e983388 sp=0xc37e9832e8 pc=0x454d936
text/template.(*state).walk(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, {0x6ae5530?, 0xc0037d3ad0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e983428 sp=0xc37e983388 pc=0x454db75
text/template.(*state).walkIfOrWith(0xc37e983690, 0xa, {0x5fc09e0?, 0xc0067e34a0?, 0xc005ae2bc0?}, 0xc00561ee65?, 0x3?, 0xc0037d3ad0)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:307 +0x22a fp=0xc37e9834f0 sp=0xc37e983428 pc=0x454de0a
text/template.(*state).walk(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, {0x6ae54e8?, 0xc005afac80?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:271 +0x1d6 fp=0xc37e983590 sp=0xc37e9834f0 pc=0x454d936
text/template.(*state).walk(0xc37e983690, {0x5fc09e0?, 0xc0067e34a0?, 0xc001514301?}, {0x6ae5530?, 0xc0037d25d0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e983630 sp=0xc37e983590 pc=0x454db75
text/template.(*state).walkTemplate(0xc37e983a90, {0x5fc09e0?, 0xc0067e34a0?, 0x0?}, 0xc005c0ed40)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:439 +0x285 fp=0xc37e9836e8 sp=0xc37e983630 pc=0x454f0c5
text/template.(*state).walk(0xc37e983a90, {0x5fc09e0?, 0xc0067e34a0?, 0x5ee6720?}, {0x6ae56e0?, 0xc005c0ed40?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:279 +0x119 fp=0xc37e983788 sp=0xc37e9836e8 pc=0x454d879
text/template.(*state).walk(0xc37e983a90, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, {0x6ae5530?, 0xc003a0a5a0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e983828 sp=0xc37e983788 pc=0x454db75
text/template.(*state).walkIfOrWith(0xc37e983a90, 0xa, {0x5fc09e0?, 0xc0067e34a0?, 0xc005c0ecc0?}, 0xc37e983990?, 0x445433e?, 0x0)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:304 +0x1f8 fp=0xc37e9838f0 sp=0xc37e983828 pc=0x454ddd8
text/template.(*state).walk(0xc37e983a90, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, {0x6ae54e8?, 0xc005c0ef00?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:271 +0x1d6 fp=0xc37e983990 sp=0xc37e9838f0 pc=0x454d936
text/template.(*state).walk(0xc37e983a90, {0x5fc09e0?, 0xc0067e34a0?, 0x1?}, {0x6ae5530?, 0xc0038490b0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e983a30 sp=0xc37e983990 pc=0x454db75
text/template.(*state).walkTemplate(0xc37e983cb8, {0x5fc09e0?, 0xc0067e34a0?, 0xc2dffa9600?}, 0xc005d5b540)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:439 +0x285 fp=0xc37e983ae8 sp=0xc37e983a30 pc=0x454f0c5
text/template.(*state).walk(0xc37e983cb8, {0x5fc09e0?, 0xc0067e34a0?, 0x10000?}, {0x6ae56e0?, 0xc005d5b540?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:279 +0x119 fp=0xc37e983b88 sp=0xc37e983ae8 pc=0x454d879
text/template.(*state).walk(0xc37e983cb8, {0x5fc09e0?, 0xc0067e34a0?, 0x5fbca20?}, {0x6ae5530?, 0xc005d6c7b0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e983c28 sp=0xc37e983b88 pc=0x454db75
text/template.(*Template).execute(0xc005d5b500, {0x6acd840?, 0xc4529c8870?}, {0x5fc09e0?, 0xc0067e34a0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:218 +0x2c5 fp=0xc37e983d18 sp=0xc37e983c28 pc=0x454d1a5
text/template.(*Template).Execute(...)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:201
html/template.(*Template).ExecuteTemplate(0xc00572c220?, {0x6acd840, 0xc4529c8870}, {0x63f3d2a?, 0x0?}, {0x5fc09e0, 0xc0067e34a0})
  /usr/local/Cellar/go/1.18.2/libexec/src/html/template/template.go:139 +0x6a fp=0xc37e983d50 sp=0xc37e983d18 pc=0x48774ea
github.com/unrolled/render.HTML.Render({{{0xc465df42b8, 0x18}, 0x1f4}, {0x63f3d2a, 0xa}, 0xc00572ea80, {0x6ad9338, 0xc00572c220}}, {0x11309b80, 0xc00679cdc0}, ...)
  /Users/silverwind/go/pkg/mod/github.com/unrolled/render@v1.4.1/engine.go:94 +0x145 fp=0xc37e983de8 sp=0xc37e983d50 pc=0x511f405
github.com/unrolled/render.(*HTML).Render(0x0?, {0x11309b80?, 0xc00679cdc0?}, {0x5fc09e0?, 0xc0067e34a0?})
  <autogenerated>:1 +0xdd fp=0xc37e983e98 sp=0xc37e983de8 pc=0x5123f3d
github.com/unrolled/render.(*Render).Render(0xc004bc8180, {0x11309b80, 0xc00679cdc0}, {0x6ad5160?, 0xc2df425500?}, {0x5fc09e0?, 0xc0067e34a0?})
  /Users/silverwind/go/pkg/mod/github.com/unrolled/render@v1.4.1/render.go:413 +0x50 fp=0xc37e983ef0 sp=0xc37e983e98 pc=0x5122630
github.com/unrolled/render.(*Render).HTML(0xc004bc8180, {0x11309b80, 0xc00679cdc0}, 0x1f4, {0x63f3d2a, 0xa}, {0x5fc09e0, 0xc0067e34a0}, {0x0, 0x0, ...})
  /Users/silverwind/go/pkg/mod/github.com/unrolled/render@v1.4.1/render.go:470 +0x49c fp=0xc37e984088 sp=0xc37e983ef0 pc=0x5122cdc
code.gitea.io/gitea/modules/context.(*Context).HTML(0xc0067d6dd0, 0x1f4, {0x63f3d2a, 0xa})
  /Users/silverwind/git/gitea/modules/context/context.go:226 +0x25c fp=0xc37e984128 sp=0xc37e984088 pc=0x55e389c
code.gitea.io/gitea/modules/context.(*Context).serverErrorInternal(0xc0067d6dd0, {0x64c01bb?, 0x14?}, {0x6acf040?, 0xc00003c230?})
  /Users/silverwind/git/gitea/modules/context/context.go:304 +0x214 fp=0xc37e984180 sp=0xc37e984128 pc=0x55e4334
code.gitea.io/gitea/modules/context.(*Context).ServerError(...)
  /Users/silverwind/git/gitea/modules/context/context.go:286
code.gitea.io/gitea/routers/web/user.GetNotificationCount.func1()
  /Users/silverwind/git/gitea/routers/web/user/notification.go:39 +0x57 fp=0xc37e9841c0 sp=0xc37e984180 pc=0x5bd9c77
runtime.call16(0xc003e555c0, 0xc00612e850, 0x0, 0x0, 0x0, 0x0, 0xc37e9846f0)
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:701 +0x49 fp=0xc37e9841e0 sp=0xc37e9841c0 pc=0x406ca29
runtime.reflectcall(0x0?, 0x0?, 0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
  <autogenerated>:1 +0x3c fp=0xc37e984220 sp=0xc37e9841e0 pc=0x4070e7c
reflect.Value.call({0x5ef4d60?, 0xc00612e850?, 0xc003c143d8?}, {0x6364751, 0x4}, {0x84402c8, 0x0, 0x487f8ed?})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:556 +0x845 fp=0xc37e984810 sp=0xc37e984220 pc=0x409a325
reflect.Value.Call({0x5ef4d60?, 0xc00612e850?, 0xc002c2d2cb?}, {0x84402c8, 0x0, 0x0})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:339 +0xbf fp=0xc37e984888 sp=0xc37e984810 pc=0x409989f
text/template.safeCall({0x5ef4d60?, 0xc00612e850?, 0xc00406cd60?}, {0x84402c8?, 0x2?, 0x0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/funcs.go:368 +0x9d fp=0xc37e984928 sp=0xc37e984888 pc=0x455739d
text/template.call({0x5f7b880?, 0xc2e2e28de0?, 0xc002a1d500?}, {0x84402c8, 0x0, 0xc00406cd70?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/funcs.go:353 +0x3d5 fp=0xc37e984a20 sp=0xc37e984928 pc=0x45570d5
runtime.call64(0xc003e55530, 0x662a350, 0x0, 0x0, 0x0, 0x30, 0xc37e984f80)
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:703 +0x49 fp=0xc37e984a70 sp=0xc37e984a20 pc=0x406cb69
runtime.reflectcall(0x5ed4ae0?, 0xc2dfff7938?, 0x8?, 0x64a96de?, 0x0?, 0x12?, 0x5ed4ae0?)
  <autogenerated>:1 +0x3c fp=0xc37e984ab0 sp=0xc37e984a70 pc=0x4070e7c
reflect.Value.call({0x5facd00?, 0x662a350?, 0x6330240?}, {0x6364751, 0x4}, {0xc2dfff7908, 0x1, 0x5fc09e0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:556 +0x845 fp=0xc37e9850a0 sp=0xc37e984ab0 pc=0x409a325
reflect.Value.Call({0x5facd00?, 0x662a350?, 0xc0067e34a0?}, {0xc2dfff7908, 0x1, 0x1})
  /usr/local/Cellar/go/1.18.2/libexec/src/reflect/value.go:339 +0xbf fp=0xc37e985118 sp=0xc37e9850a0 pc=0x409989f
text/template.safeCall({0x5facd00?, 0x662a350?, 0xc0067e34a0?}, {0xc2dfff7908?, 0x6b1c700?, 0x6330240?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/funcs.go:368 +0x9d fp=0xc37e9851b8 sp=0xc37e985118 pc=0x455739d
text/template.(*state).evalCall(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0xc2e2e28da0?}, {0x5facd00?, 0x662a350?, 0x98?}, 0x1, {0x6ae5338, 0xc00380d6e0}, ...)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:781 +0x87c fp=0xc37e985318 sp=0xc37e9851b8 pc=0x455229c
text/template.(*state).evalFunction(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0x17?}, 0x6ae5458?, {0x6ae5338, 0xc00380d6e0}, {0xc0054bb600, 0x2, 0x2}, ...)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:601 +0x209 fp=0xc37e985410 sp=0xc37e985318 pc=0x4550c69
text/template.(*state).evalCommand(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0xffffffffffffffff?}, 0x5fc09e0?, {0x5fdd4e0?, 0x84402c8?, 0x5fc09e0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:488 +0xea fp=0xc37e9854e0 sp=0xc37e985410 pc=0x454f6ca
text/template.(*state).evalPipeline(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0xc2e2e28dc0?}, 0xc00382c4e0)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:457 +0x145 fp=0xc37e9855d0 sp=0xc37e9854e0 pc=0x454f265
text/template.(*state).walk(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0x5ef4d60?}, {0x6ae51d0?, 0xc00380d770})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:261 +0x225 fp=0xc37e985670 sp=0xc37e9855d0 pc=0x454d985
text/template.(*state).walk(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0x82?}, {0x6ae5530?, 0xc00380d680?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e985710 sp=0xc37e985670 pc=0x454db75
text/template.(*state).walkIfOrWith(0xc37e985d88, 0xa, {0x5fc09e0?, 0xc0067e34a0?, 0xc00382c3c0?}, 0x5fc10a0?, 0xc003e54a50?, 0x0)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:304 +0x1f8 fp=0xc37e9857d8 sp=0xc37e985710 pc=0x454ddd8
text/template.(*state).walk(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0x5ee6720?}, {0x6ae54e8?, 0xc005beaac0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:271 +0x1d6 fp=0xc37e985878 sp=0xc37e9857d8 pc=0x454d936
text/template.(*state).walk(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0xc0067e34a0?}, {0x6ae5530?, 0xc0037d3b60?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e985918 sp=0xc37e985878 pc=0x454db75
text/template.(*state).walkIfOrWith(0xc37e985d88, 0xa, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, 0x6364751?, 0xc4529c8180?, 0xc0038fe120)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:304 +0x1f8 fp=0xc37e9859e0 sp=0xc37e985918 pc=0x454ddd8
text/template.(*state).walk(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0x0?}, {0x6ae54e8?, 0xc005afac40?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:271 +0x1d6 fp=0xc37e985a80 sp=0xc37e9859e0 pc=0x454d936
text/template.(*state).walk(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, {0x6ae5530?, 0xc0037d3ad0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e985b20 sp=0xc37e985a80 pc=0x454db75
text/template.(*state).walkIfOrWith(0xc37e985d88, 0xa, {0x5fc09e0?, 0xc0067e34a0?, 0xc005ae2bc0?}, 0xc00561ee65?, 0x3?, 0xc0037d3ad0)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:307 +0x22a fp=0xc37e985be8 sp=0xc37e985b20 pc=0x454de0a
text/template.(*state).walk(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, {0x6ae54e8?, 0xc005afac80?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:271 +0x1d6 fp=0xc37e985c88 sp=0xc37e985be8 pc=0x454d936
text/template.(*state).walk(0xc37e985d88, {0x5fc09e0?, 0xc0067e34a0?, 0xc001514301?}, {0x6ae5530?, 0xc0037d25d0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e985d28 sp=0xc37e985c88 pc=0x454db75
text/template.(*state).walkTemplate(0xc37e986188, {0x5fc09e0?, 0xc0067e34a0?, 0x0?}, 0xc005c0ed40)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:439 +0x285 fp=0xc37e985de0 sp=0xc37e985d28 pc=0x454f0c5
text/template.(*state).walk(0xc37e986188, {0x5fc09e0?, 0xc0067e34a0?, 0x5ee6720?}, {0x6ae56e0?, 0xc005c0ed40?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:279 +0x119 fp=0xc37e985e80 sp=0xc37e985de0 pc=0x454d879
text/template.(*state).walk(0xc37e986188, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, {0x6ae5530?, 0xc003a0a5a0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e985f20 sp=0xc37e985e80 pc=0x454db75
text/template.(*state).walkIfOrWith(0xc37e986188, 0xa, {0x5fc09e0?, 0xc0067e34a0?, 0xc005c0ecc0?}, 0xc37e986088?, 0x445433e?, 0x0)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:304 +0x1f8 fp=0xc37e985fe8 sp=0xc37e985f20 pc=0x454ddd8
text/template.(*state).walk(0xc37e986188, {0x5fc09e0?, 0xc0067e34a0?, 0x5fc09e0?}, {0x6ae54e8?, 0xc005c0ef00?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:271 +0x1d6 fp=0xc37e986088 sp=0xc37e985fe8 pc=0x454d936
text/template.(*state).walk(0xc37e986188, {0x5fc09e0?, 0xc0067e34a0?, 0x1?}, {0x6ae5530?, 0xc0038490b0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:274 +0x415 fp=0xc37e986128 sp=0xc37e986088 pc=0x454db75
text/template.(*state).walkTemplate(0xc37e9863b0, {0x5fc09e0?, 0xc0067e34a0?, 0x20?}, 0xc005d5b540)
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:439 +0x285 fp=0xc37e9861e0 sp=0xc37e986128 pc=0x454f0c5
text/template.(*state).walk(0xc37e9863b0, {0x5fc09e0?, 0xc0067e34a0?, 0x454f265?}, {0x6ae56e0?, 0xc005d5b540?})
  /usr/local/Cellar/go/1.18.2/libexec/src/text/template/exec.go:279 +0x119 fp=0xc37e986280 sp=0xc37e9861e0 pc=0x454d879
created by net/http.(*Server).Serve
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:3071 +0x4db

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x1047ece8, 0x72)
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc004995c80?, 0xc007b15150?, 0x0)
  /usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
  /usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc004995c80)
  /usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc004995c80)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc001277170)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).AcceptTCP(0xc001277170)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/tcpsock.go:275 +0x3d
code.gitea.io/gitea/modules/graceful.(*wrappedListener).Accept(0xc0055cadc0)
  /Users/silverwind/git/gitea/modules/graceful/server.go:180 +0x47
net/http.(*Server).Serve(0xc00081a380, {0x6adb898, 0xc0055cadc0})
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:3039 +0x385
code.gitea.io/gitea/modules/graceful.(*Server).Serve(0xc003ca2070, 0xc005002ab0)
  /Users/silverwind/git/gitea/modules/graceful/server.go:133 +0xf7
code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe(0xc003ca2070, 0x4997a40?)
  /Users/silverwind/git/gitea/modules/graceful/server.go:95 +0x1bf
code.gitea.io/gitea/modules/graceful.HTTPListenAndServe({0x6360691?, 0x16?}, {0xc0053ec2c0?, 0x12?}, {0x635d9df?, 0x0?}, {0x6acdf20?, 0xc001dd0340?})
  /Users/silverwind/git/gitea/modules/graceful/server_http.go:33 +0x3d
code.gitea.io/gitea/cmd.runHTTP(...)
  /Users/silverwind/git/gitea/cmd/web_graceful.go:19
code.gitea.io/gitea/cmd.listen({0x6acdf20, 0xc001dd0340}, 0x1)
  /Users/silverwind/git/gitea/cmd/web.go:231 +0x4e5
code.gitea.io/gitea/cmd.runWeb(0xc001e25e40)
  /Users/silverwind/git/gitea/cmd/web.go:176 +0x831
github.com/urfave/cli.HandleAction({0x5f219c0?, 0x6622630?}, 0xc001e3b500?)
  /Users/silverwind/go/pkg/mod/github.com/urfave/cli@v1.22.9/app.go:524 +0xa8
github.com/urfave/cli.(*App).Run(0xc001e3b500, {0xc00003c200, 0x1, 0x1})
  /Users/silverwind/go/pkg/mod/github.com/urfave/cli@v1.22.9/app.go:286 +0x7c5
main.main()
  /Users/silverwind/git/gitea/main.go:115 +0xcf9

goroutine 375 [select]:
code.gitea.io/gitea/modules/eventsource.(*Manager).Run(0xc000968048, {0x6adf8c0, 0xc000327470})
  /Users/silverwind/git/gitea/modules/eventsource/manager_run.go:38 +0x147
code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext(0xc001e7e0f0, 0xc004722930)
  /Users/silverwind/git/gitea/modules/graceful/manager.go:114 +0xc7
created by code.gitea.io/gitea/modules/eventsource.(*Manager).Init
  /Users/silverwind/git/gitea/modules/eventsource/manager_run.go:26 +0xdb

goroutine 5 [select]:
code.gitea.io/gitea/modules/log.(*MultiChannelledLog).Start(0xc000120240)
  /Users/silverwind/git/gitea/modules/log/event.go:325 +0x2af
created by code.gitea.io/gitea/modules/log.(*MultiChannelledLog).AddLogger
  /Users/silverwind/git/gitea/modules/log/event.go:206 +0x176

goroutine 7 [select]:
github.com/blevesearch/bleve_index_api.AnalysisWorker({0xc001c1a1e0?, 0xc001c1a240?})
  /Users/silverwind/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:46 +0x8f
created by github.com/blevesearch/bleve_index_api.NewAnalysisQueue
  /Users/silverwind/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:38 +0x92

goroutine 8 [select]:
github.com/blevesearch/bleve_index_api.AnalysisWorker({0xc001c1a1e0?, 0xc001c1a240?})
  /Users/silverwind/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:46 +0x8f
created by github.com/blevesearch/bleve_index_api.NewAnalysisQueue
  /Users/silverwind/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:38 +0x92

goroutine 9 [select]:
github.com/blevesearch/bleve_index_api.AnalysisWorker({0xc001c1a1e0?, 0xc001c1a240?})
  /Users/silverwind/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:46 +0x8f
created by github.com/blevesearch/bleve_index_api.NewAnalysisQueue
  /Users/silverwind/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:38 +0x92

goroutine 10 [select]:
github.com/blevesearch/bleve_index_api.AnalysisWorker({0xc001c1a1e0?, 0xc001c1a240?})
  /Users/silverwind/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:46 +0x8f
created by github.com/blevesearch/bleve_index_api.NewAnalysisQueue
  /Users/silverwind/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:38 +0x92

goroutine 11 [select]:
code.gitea.io/gitea/modules/graceful.(*Manager).handleSignals(0x0?, {0x6adf8c0, 0xc000327530})
  /Users/silverwind/git/gitea/modules/graceful/manager_unix.go:132 +0x205
created by code.gitea.io/gitea/modules/graceful.(*Manager).start
  /Users/silverwind/git/gitea/modules/graceful/manager_unix.go:76 +0x9a5

goroutine 21 [syscall]:
os/signal.signal_recv()
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/sigqueue.go:148 +0x28
os/signal.loop()
  /usr/local/Cellar/go/1.18.2/libexec/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
  /usr/local/Cellar/go/1.18.2/libexec/src/os/signal/signal.go:151 +0x2a

goroutine 1414 [IO wait]:
internal/poll.runtime_pollWait(0x1047eb08, 0x72)
  /usr/local/Cellar/go/1.18.2/libexec/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc05d5f8d80?, 0xc06592f391?, 0x0)
  /usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
  /usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc05d5f8d80, {0xc06592f391, 0x1, 0x1})
  /usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc05d5f8d80, {0xc06592f391?, 0xc00030cea0?, 0xc001eaaf48?})
  /usr/local/Cellar/go/1.18.2/libexec/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc05cffb2f0, {0xc06592f391?, 0xc005013d50?, 0xc03975c9b0?})
  /usr/local/Cellar/go/1.18.2/libexec/src/net/net.go:183 +0x45
net/http.(*connReader).backgroundRead(0xc06592f380)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:672 +0x3f
created by net/http.(*connReader).startBackgroundRead
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:668 +0xca

goroutine 114 [select]:
code.gitea.io/gitea/modules/log.(*ChannelledLog).Start(0xc00097c690)
  /Users/silverwind/git/gitea/modules/log/event.go:80 +0xcc
created by code.gitea.io/gitea/modules/log.NewChannelledLog
  /Users/silverwind/git/gitea/modules/log/event.go:69 +0x365

goroutine 54 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc00081a1c0)
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xc7
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x18c

goroutine 55 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0002a3860)
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x158
created by github.com/syndtr/goleveldb/leveldb.openDB
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x4ea

goroutine 56 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0002a3860)
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xa8
created by github.com/syndtr/goleveldb/leveldb.openDB
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x52a

goroutine 57 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0002a3860)
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x657
created by github.com/syndtr/goleveldb/leveldb.openDB
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x598

goroutine 58 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0002a3860)
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x113
created by github.com/syndtr/goleveldb/leveldb.openDB
  /Users/silverwind/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5d6

goroutine 65 [select]:
database/sql.(*DB).connectionOpener(0xc001af9ee0, {0x6adf818, 0xc000021bc0})
  /usr/local/Cellar/go/1.18.2/libexec/src/database/sql/sql.go:1226 +0x8d
created by database/sql.OpenDB
  /usr/local/Cellar/go/1.18.2/libexec/src/database/sql/sql.go:794 +0x18d

goroutine 1020 [select]:
code.gitea.io/gitea/modules/graceful.(*Server).awaitShutdown(0xc000759290?)
  /Users/silverwind/git/gitea/modules/graceful/server_hooks.go:16 +0xab
created by code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe
  /Users/silverwind/git/gitea/modules/graceful/server.go:83 +0x65

goroutine 287 [select]:
github.com/gogs/cron.(*Cron).run(0xc000952af0)
  /Users/silverwind/go/pkg/mod/github.com/gogs/cron@v0.0.0-20171120032916-9f6c956d3e14/cron.go:197 +0x596
created by github.com/gogs/cron.(*Cron).Start
  /Users/silverwind/go/pkg/mod/github.com/gogs/cron@v0.0.0-20171120032916-9f6c956d3e14/cron.go:150 +0x65

goroutine 360 [select]:
code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).readToChan(0xc0047826e0)
  /Users/silverwind/git/gitea/modules/queue/queue_bytefifo.go:254 +0x5aa
code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).Run(0xc0047826e0, 0xc004791210, 0xc004791220)
  /Users/silverwind/git/gitea/modules/queue/queue_bytefifo.go:164 +0x23c
code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownFns(0xc001e7e0f0, 0xc004890198)
  /Users/silverwind/git/gitea/modules/graceful/manager.go:82 +0x123
created by code.gitea.io/gitea/modules/indexer/issues.InitIssueIndexer.func2
  /Users/silverwind/git/gitea/modules/indexer/issues/indexer.go:242 +0x6fb

goroutine 1351 [se]:
code.gitea.io/gitea/routers/web/events.Events(0xc0b48a6000)
  /Users/silverwind/git/gitea/routers/web/events/events.go:71 +0x698
code.gitea.io/gitea/modules/web.convertHandler.func3({0x6adb808?, 0xc0055e1400?}, 0xc0b3c8fb00, {0x0?, 0x4014e16?, 0x0?})
  /Users/silverwind/git/gitea/modules/web/wrap_convert.go:44 +0xb7
code.gitea.io/gitea/modules/web.wrapInternal.func1({0x6adb808, 0xc0055e1400}, 0xc003220990?)
  /Users/silverwind/git/gitea/modules/web/wrap.go:41 +0x12a
net/http.HandlerFunc.ServeHTTP(0x5fbf780?, {0x6adb808?, 0xc0055e1400?}, 0xc0b4892830?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc003d01440, {0x6adb808, 0xc0055e1400}, 0xc0b3c8fb00)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x216
net/http.HandlerFunc.ServeHTTP(0xc0b489dce0?, {0x6adb808?, 0xc0055e1400?}, 0xc0b489dbf0?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
code.gitea.io/gitea/modules/web.Middle.func1.1({0x606cde0?, 0x616bf00?}, 0xc0b3c8fb00)
  /Users/silverwind/git/gitea/modules/web/wrap.go:63 +0x130
net/http.HandlerFunc.ServeHTTP(0xc0b48a6000?, {0x6adb808?, 0xc0055e1400?}, 0x4010e47?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
code.gitea.io/gitea/modules/web.Middle.func1.1({0x606cde0?, 0x616bf00?}, 0xc0b3c8fb00)
  /Users/silverwind/git/gitea/modules/web/wrap.go:63 +0x130
net/http.HandlerFunc.ServeHTTP(0xc0b48a6000?, {0x6adb808?, 0xc0055e1400?}, 0xc000738000?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
code.gitea.io/gitea/modules/web.Middle.func1.1({0xc0b488fd20?, 0xc059fd4e80?}, 0xc0b3c8fb00)
  /Users/silverwind/git/gitea/modules/web/wrap.go:63 +0x130
net/http.HandlerFunc.ServeHTTP(0x6395c90?, {0x6adb808?, 0xc0055e1400?}, 0x400d7cd?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
github.com/go-chi/chi/v5/middleware.GetHead.func1({0x6adb808?, 0xc0055e1400?}, 0xc0b3c8fb00?)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/middleware/get_head.go:37 +0x1e5
net/http.HandlerFunc.ServeHTTP(0xc0b48a6000?, {0x6adb808?, 0xc0055e1400?}, 0x8a96108?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
code.gitea.io/gitea/modules/web.Middle.func1.1({0x606cde0?, 0x616bf00?}, 0xc0b3c8fb00)
  /Users/silverwind/git/gitea/modules/web/wrap.go:63 +0x130
net/http.HandlerFunc.ServeHTTP(0x5fc09e0?, {0x6adb808?, 0xc0055e1400?}, 0x8?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
code.gitea.io/gitea/modules/context.Contexter.func1.1({0x6adb808?, 0xc0055e1400}, 0xc0b3c8fa00)
  /Users/silverwind/git/gitea/modules/context/context.go:800 +0x19db
net/http.HandlerFunc.ServeHTTP(0x5f91520?, {0x6adb808?, 0xc0055e1400?}, 0x40561f0?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc003d01440, {0x6adb808, 0xc0055e1400}, 0xc0b3c8fa00)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:71 +0x48d
github.com/go-chi/chi/v5.(*Mux).Mount.func1({0x6adb808, 0xc0055e1400}, 0xc0b3c8fa00)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:314 +0x19c
net/http.HandlerFunc.ServeHTTP(0x5fbf780?, {0x6adb808?, 0xc0055e1400?}, 0xc0b4892720?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc0049610e0, {0x6adb808, 0xc0055e1400}, 0xc0b3c8fa00)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x216
net/http.HandlerFunc.ServeHTTP(0xc0b489dbf0?, {0x6adb808?, 0xc0055e1400?}, 0x113137c0?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
code.gitea.io/gitea/routers/web.Recovery.func1.1({0x6adb808?, 0xc0055e1400?}, 0x6ac9201?)
  /Users/silverwind/git/gitea/routers/web/base.go:173 +0x92
net/http.HandlerFunc.ServeHTTP(0x6adf738?, {0x6adb808?, 0xc0055e1400?}, 0x6ac9250?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
gitea.com/go-chi/session.Sessioner.func1.1({0x6adb808, 0xc0055e1400}, 0xc0b3c8f900)
  /Users/silverwind/go/pkg/mod/gitea.com/go-chi/session@v0.0.0-20211218221615-e3605d8b28b8/session.go:257 +0x2fe
net/http.HandlerFunc.ServeHTTP(0xc059fd5580?, {0x6adb808?, 0xc0055e1400?}, 0xc0055e1300?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
code.gitea.io/gitea/modules/web.WrapWithPrefix.func1.1({0x6adb808?, 0xc0055e1400?}, 0xc0b3c8f900?)
  /Users/silverwind/git/gitea/modules/web/wrap.go:110 +0x109
net/http.HandlerFunc.ServeHTTP(0x4100ab4?, {0x6adb808?, 0xc0055e1400?}, 0x40561f0?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc0049610e0, {0x6adb808, 0xc0055e1400}, 0xc0b3c8f900)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:71 +0x48d
github.com/go-chi/chi/v5.(*Mux).Mount.func1({0x6adb808, 0xc0055e1400}, 0xc0b3c8f900)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:314 +0x19c
net/http.HandlerFunc.ServeHTTP(0x5fbf780?, {0x6adb808?, 0xc0055e1400?}, 0xc0b3cdf1a4?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc004961080, {0x6adb808, 0xc0055e1400}, 0xc0b3c8f900)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:442 +0x216
net/http.HandlerFunc.ServeHTTP(0x835e3d0?, {0x6adb808?, 0xc0055e1400?}, 0x4100ab4?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
code.gitea.io/gitea/routers/common.Middlewares.func2.1({0x6adb808?, 0xc0055e1400?}, 0x444d365?)
  /Users/silverwind/git/gitea/routers/common/middleware.go:79 +0x83
net/http.HandlerFunc.ServeHTTP(0x6466ac4?, {0x6adb808?, 0xc0055e1400?}, 0x5a6a966?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
github.com/go-chi/chi/v5/middleware.StripSlashes.func1({0x6adb808, 0xc0055e1400}, 0xc0b3c8f900)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/middleware/strip.go:30 +0x139
net/http.HandlerFunc.ServeHTTP(0xc0b3c8f900?, {0x6adb808?, 0xc0055e1400?}, 0x63a2b4f?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
github.com/chi-middleware/proxy.ForwardedHeaders.func1.1({0x6adb808, 0xc0055e1400}, 0xc0b3c8f900)
  /Users/silverwind/go/pkg/mod/github.com/chi-middleware/proxy@v1.1.1/middleware.go:37 +0x197
net/http.HandlerFunc.ServeHTTP(0x637c6ba?, {0x6adb808?, 0xc0055e1400?}, 0xc0b3ca3338?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
code.gitea.io/gitea/routers/common.Middlewares.func1.1({0x6adddb8?, 0xc0047f01c0}, 0xc0b3c8f800)
  /Users/silverwind/git/gitea/routers/common/middleware.go:32 +0x473
net/http.HandlerFunc.ServeHTTP(0x6adf818?, {0x6adddb8?, 0xc0047f01c0?}, 0x835e3d0?)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2084 +0x2f
github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc004961080, {0x6adddb8, 0xc0047f01c0}, 0xc038b3f500)
  /Users/silverwind/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.7/mux.go:88 +0x442
code.gitea.io/gitea/modules/web.(*Route).ServeHTTP(0x0?, {0x6adddb8?, 0xc0047f01c0?}, 0xc000738000?)
  /Users/silverwind/git/gitea/modules/web/route.go:200 +0x2e
net/http.serverHandler.ServeHTTP({0xc06592f380?}, {0x6adddb8, 0xc0047f01c0}, 0xc038b3f500)
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc003dc0a00, {0x6adf8c0, 0xc003ca01e0})
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
  /usr/local/Cellar/go/1.18.2/libexec/src/net/http/server.go:3071 +0x4db

Interestingly, crash did not occur again on second startup.

@zeripath
Copy link
Contributor

zeripath commented May 24, 2022

Agh I hit this a few days ago whilst I was in debug mode but thought it was due to too many checkpoints.

This is a regression from #19748 but it is looping because of a mistake in #10961

The problem is in here:

	c.Data["NotificationUnreadCount"] = func() int64 {
		count, err := models.GetNotificationCount(c, c.Doer, models.NotificationStatusUnread)
		if err != nil {
			c.ServerError("GetNotificationCount", err)
			return -1
		}

		return count
	}

The function here is called during rendering.

If there is an err it's too late really to use ServerError as we're already rendering - and we should simply return -1 here and log the problem.

I'll pop up a PR.

zeripath added a commit to zeripath/gitea that referenced this issue May 24, 2022
If the context is cancelled `.NotificationUnreadCount` in a template can
cause an infinite loop with `ctx.ServerError()` being called, which
creates a template that then calls `.NotificationUnreadCount` calling
`GetNotificationCount()` with the cancelled context resulting in an
error that calls `ctx.ServerError`... and so on...

This PR simply stops calling `ctx.ServerError` in the error handler code
for `.NotificationUnreadCount` as we have already started rendering and
so it is too late to call `ctx.ServerError`. Additionally we skip
logging the error if it's a context cancelled error.

Fix go-gitea#19793

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath added the issue/regression Indicates a previously functioning feature or behavior that has broken or regressed after a change label May 24, 2022
@zeripath
Copy link
Contributor

As the bug is a regression due to unreleased code the PR to fix it has been marked skip-changelog.

lunny pushed a commit that referenced this issue May 25, 2022
If the context is cancelled `.NotificationUnreadCount` in a template can
cause an infinite loop with `ctx.ServerError()` being called, which
creates a template that then calls `.NotificationUnreadCount` calling
`GetNotificationCount()` with the cancelled context resulting in an
error that calls `ctx.ServerError`... and so on...

This PR simply stops calling `ctx.ServerError` in the error handler code
for `.NotificationUnreadCount` as we have already started rendering and
so it is too late to call `ctx.ServerError`. Additionally we skip
logging the error if it's a context cancelled error.

Fix #19793

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
@lunny
Copy link
Member

lunny commented May 25, 2022

I think the problem is c.ServerError("GetNotificationCount", err) MUST not be called in a render function.

@silverwind
Copy link
Member

I think the problem is c.ServerError("GetNotificationCount", err) MUST not be called in a render function.

If we could add a linter rule for that, that'd be great.

@gboddin
Copy link
Author

gboddin commented May 25, 2022

Thank you so much for the quick response and the details !

I know a bit more if I need to help on a PR next time :)

@zeripath
Copy link
Contributor

I think the problem is c.ServerError("GetNotificationCount", err) MUST not be called in a render function.

If we could add a linter rule for that, that'd be great.

I don't think that would be very easy.

I think we have a couple of options:

  1. We could make sure that ServerError doesn't re-render within a render.
  2. This particular error is actually due context cancellation - it makes no sense to continue rendering if the request context has been cancelled so we could test this in ServerError and not attempt to render if the request context is cancelled.
  3. A much more involved option is to change the framework so ServerError throws a panic which is caught upwards and handled then any panic that gets thrown within its render can be handled in a different way.

AbdulrhmnGhanem pushed a commit to kitspace/gitea that referenced this issue Aug 24, 2022
…9799)

If the context is cancelled `.NotificationUnreadCount` in a template can
cause an infinite loop with `ctx.ServerError()` being called, which
creates a template that then calls `.NotificationUnreadCount` calling
`GetNotificationCount()` with the cancelled context resulting in an
error that calls `ctx.ServerError`... and so on...

This PR simply stops calling `ctx.ServerError` in the error handler code
for `.NotificationUnreadCount` as we have already started rendering and
so it is too late to call `ctx.ServerError`. Additionally we skip
logging the error if it's a context cancelled error.

Fix go-gitea#19793

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
@go-gitea go-gitea locked and limited conversation to collaborators May 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP issue/regression Indicates a previously functioning feature or behavior that has broken or regressed after a change type/bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants