Race fix: Acquire lock in fetchAndLoadDatabase before calling _fetchAndLoadDatabase

[bbrks]

This race appears to have cropped up now we're running more tests with Rosmar.

• https://mobile.jenkins.couchbase.com/job/sgw-unix-build/30766/consoleFull#-193976283857ab531-e808-4d97-b44b-e8a7cb55e9a7

fetchAndLoadDatabase wasn't acquiring the lock before falling into _fetchAndLoadDatabase which seems obviously wrong. I've scanned usages of both methods and the locking seems correct based on _ prefix usage (i.e. we're not going to be accidentally double locking)

22:05:17 ==================
22:05:17 
WARNING: DATA RACE
22:05:17 Read at 0x00c001e25e98 by goroutine 37168:
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext)._applyConfig()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/config.go:1830 +0x375
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext)._applyConfigs()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/config.go:1770 +0x17c
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext).fetchAndLoadConfigs()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/config.go:1446 +0xa64
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext).initializeBootstrapConnection.func1()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/server_context.go:2093 +0x325
22:05:17 
22:05:17 Previous write at 0x00c001e25e98 by goroutine 37166:
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext)._getOrAddDatabaseFromConfig()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/server_context.go:911 +0x3f46
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext)._reloadDatabaseWithConfig()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/server_context.go:438 +0x159
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext)._applyConfig()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/config.go:1853 +0x6f9
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext)._applyConfigs()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/config.go:1770 +0x17c
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext)._fetchAndLoadDatabase()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/config.go:1472 +0x1cd
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext).fetchAndLoadDatabase()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/config.go:1462 +0x255
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext).GetInactiveDatabase()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/server_context.go:294 +0x256
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext).GetDatabase()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/server_context.go:256 +0xd7
22:05:17   github.com/couchbase/sync_gateway/rest/adminapitest.TestDbConfigPersistentSGVersions.func1.1()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/adminapitest/admin_api_test.go:3356 +0x84
22:05:17   github.com/couchbase/sync_gateway/rest.WaitAndAssertCondition()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/utilities_testing.go:2283 +0x116
22:05:17   github.com/couchbase/sync_gateway/rest/adminapitest.TestDbConfigPersistentSGVersions.func1()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/adminapitest/admin_api_test.go:3355 +0x14d
22:05:17   github.com/couchbase/sync_gateway/rest/adminapitest.TestDbConfigPersistentSGVersions()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/adminapitest/admin_api_test.go:3369 +0x80b
22:05:17   testing.tRunner()
22:05:17       /home/couchbase/cbdeps/go1.21.4/src/testing/testing.go:1595 +0x238
22:05:17   testing.(*T).Run.func1()
22:05:17       /home/couchbase/cbdeps/go1.21.4/src/testing/testing.go:1648 +0x44
22:05:17 
22:05:17 Goroutine 37168 (running) created at:
22:05:17   github.com/couchbase/sync_gateway/rest.(*ServerContext).initializeBootstrapConnection()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/server_context.go:2082 +0x784
22:05:17   github.com/couchbase/sync_gateway/rest.SetupServerContext()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/config.go:1387 +0x50b
22:05:17   github.com/couchbase/sync_gateway/rest.StartServerWithConfig()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/utilities_testing_bootstrap.go:144 +0x86
22:05:17   github.com/couchbase/sync_gateway/rest/adminapitest.TestDbConfigPersistentSGVersions()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/adminapitest/admin_api_test.go:3323 +0x1f2
22:05:17   testing.tRunner()
22:05:17       /home/couchbase/cbdeps/go1.21.4/src/testing/testing.go:1595 +0x238
22:05:17   testing.(*T).Run.func1()
22:05:17       /home/couchbase/cbdeps/go1.21.4/src/testing/testing.go:1648 +0x44
22:05:17 
22:05:17 Goroutine 37166 (running) created at:
22:05:17   testing.(*T).Run()
22:05:17       /home/couchbase/cbdeps/go1.21.4/src/testing/testing.go:1648 +0x82a
22:05:17   testing.runTests.func1()
22:05:17       /home/couchbase/cbdeps/go1.21.4/src/testing/testing.go:2054 +0x84
22:05:17   testing.tRunner()
22:05:17       /home/couchbase/cbdeps/go1.21.4/src/testing/testing.go:1595 +0x238
22:05:17   testing.runTests()
22:05:17       /home/couchbase/cbdeps/go1.21.4/src/testing/testing.go:2052 +0x896
22:05:17   testing.(*M).Run()
22:05:17       /home/couchbase/cbdeps/go1.21.4/src/testing/testing.go:1925 +0xb57
22:05:17   github.com/couchbase/sync_gateway/base.TestBucketPoolMain()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/base/main_test_bucket_pool.go:689 +0x424
22:05:17   github.com/couchbase/sync_gateway/db.TestBucketPoolWithIndexes()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/db/util_testing.go:510 +0x71
22:05:17   github.com/couchbase/sync_gateway/rest/adminapitest.TestMain()
22:05:17       /home/couchbase/jenkins/workspace/sgw-unix-build/3.2.0/community/sync_gateway/rest/adminapitest/main_test.go:24 +0x1d
22:05:17   main.main()
22:05:17       _testmain.go:235 +0x307
22:05:17 ==================

Integration Tests

• GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/2254/

[bbrks]

Retrospective Jira ticket: CBG-3743