Intermittent deadlock in queue tests

[zeripath]

• Gitea version (or commit ref): 1.15+
• Log gist:

2021/08/14 08:28:16 .../queue/workerpool.go:93:zeroBoost() [W] WorkerPool: 1 (for TestChannelQueue) has zero workers - adding 5 temporary workers for 5m0s
645s
	2021/08/14 08:28:16 ...eue/queue_channel.go:98:func1() [W] ChannelQueue: first-channel Terminated before completed flushing
645s
	2021/08/14 08:28:16 .../queue/workerpool.go:93:zeroBoost() [W] WorkerPool: 7 (for second-level) has zero workers - adding 1 temporary workers for 5m0s
645s
	2021/08/14 08:28:16 ...ueue_disk_channel.go:163:func3() [W] LevelQueue: second-level shut down before completely flushed
645s
	panic: test timed out after 10m0s
645s
69	
645s
70	goroutine 162 [running]:
646s
71	testing.(*M).startAlarm.func1()
646s
72		/usr/local/go/src/testing/testing.go:1700 +0x11f
646s
73	created by time.goFunc
646s
74		/usr/local/go/src/time/sleep.go:180 +0x52
646s
75	
646s
76	goroutine 1 [chan receive, 9 minutes]:
646s
77	testing.(*T).Run(0xc000001e00, 0xe7f214, 0xe, 0xe9d238, 0x1)
646s
78		/usr/local/go/src/testing/testing.go:1239 +0x610
646s
79	testing.runTests.func1(0xc000001e00)
646s
80		/usr/local/go/src/testing/testing.go:1511 +0xa7
646s
81	testing.tRunner(0xc000001e00, 0xc000175ce0)
646s
82		/usr/local/go/src/testing/testing.go:1193 +0x203
646s
83	testing.runTests(0xc00000f488, 0x12d8a60, 0x5, 0x5, 0xc03dfe1e37809e52, 0x8bb35251c3, 0x1326640, 0xc0000da1a8)
646s
84		/usr/local/go/src/testing/testing.go:1509 +0x613
646s
85	testing.(*M).Run(0xc00017f280, 0x0)
646s
86		/usr/local/go/src/testing/testing.go:1417 +0x3b4
646s
87	main.main()
646s
88		_testmain.go:51 +0x237
646s
89	
646s
90	goroutine 6 [select, 9 minutes]:
646s
91	code.gitea.io/gitea/modules/log.(*ChannelledLog).Start(0xc0000e43c0)
646s
92		/drone/src/modules/log/event.go:70 +0x173
646s
93	created by code.gitea.io/gitea/modules/log.NewChannelledLog
646s
94		/drone/src/modules/log/event.go:61 +0x3e8
646s
95	
646s
96	goroutine 7 [select, 9 minutes]:
646s
97	code.gitea.io/gitea/modules/log.(*MultiChannelledLog).Start(0xc000156100)
646s
98		/drone/src/modules/log/event.go:300 +0x538
646s
99	created by code.gitea.io/gitea/modules/log.(*MultiChannelledLog).AddLogger
646s
100		/drone/src/modules/log/event.go:190 +0x22e
646s
101	
646s
102	goroutine 115 [semacquire, 9 minutes]:
646s
103	sync.runtime_SemacquireMutex(0xc00017e194, 0x900000000, 0x1)
646s
104		/usr/local/go/src/runtime/sema.go:71 +0x47
646s
105	sync.(*Mutex).lockSlow(0xc00017e190)
646s
106		/usr/local/go/src/sync/mutex.go:138 +0x1d4
646s
107	sync.(*Mutex).Lock(0xc00017e190)
646s
108		/usr/local/go/src/sync/mutex.go:81 +0x85
646s
109	gitea.com/lunny/levelqueue.(*Queue).Close(0xc00017e180, 0x0, 0x0)
646s
110		/drone/src/vendor/gitea.com/lunny/levelqueue/queue.go:299 +0x8c
646s
111	code.gitea.io/gitea/modules/queue.(*LevelQueueByteFIFO).Close(0xc0002540c0, 0xc00021a120, 0xe80d00)
646s
112		/drone/src/modules/queue/queue_disk.go:108 +0x50
646s
113	code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).Terminate(0xc00011e500)
646s
114		/drone/src/modules/queue/queue_bytefifo.go:262 +0x27f
646s
115	code.gitea.io/gitea/modules/queue.TestLevelQueue(0xc0001caf00)

Description

Unfortunately there is a deadlock in gitea.com/lunny/levelqueue.(*Queue).Close introduced by my commit in 0.4.0.

The locking order in the queue.Close() function is incorrect as the queue.Len() function locks the lowLock before the highLock. We therefore must lock the lowLock before the highLock.

This will be fixed upstream by: https://gitea.com/lunny/levelqueue/pulls/5