Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

migration/migrations: TestEnsureSpanConfigReconciliation failed [SettingsWatcher data race] #75849

Closed
cockroach-teamcity opened this issue Feb 2, 2022 · 4 comments
Closed

migration/migrations: TestEnsureSpanConfigReconciliation failed [SettingsWatcher data race] #75849

cockroach-teamcity opened this issue Feb 2, 2022 · 4 comments
Assignees
@irfansharif
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot. S-3-productivity Severe issues that impede the productivity of CockroachDB developers. T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Feb 2, 2022 •
edited by cockroach-jira-scripts
Loading

migration/migrations.TestEnsureSpanConfigReconciliation failed with artifacts on master @ a434c8418c36dbeb64e73588bcd4dd5b248c3238:

      /go/src/github.com/cockroachdb/cockroach/pkg/settings/updater.go:118 +0x77b
  github.com/cockroachdb/cockroach/pkg/settings.(*updater).Set()
      <autogenerated>:1 +0xf1
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).setLocked()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:292 +0x4b0
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).handleKV()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:263 +0x56d
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start.func4()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:192 +0x4e
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:240 +0x9a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:334 +0x1b3
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:287 +0x64
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x96

Goroutine 159 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:483 +0x615
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:354 +0xffb
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:320 +0xc7d
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeClaimedJobs.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:205 +0xd3
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeClaimedJobs·dwrap·1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:208 +0x47

Goroutine 342 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0xf0
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:165 +0x104
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:187 +0xb1
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:290 +0x7fb
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:214 +0x4ac
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494 +0x235
==================
    migrate_span_configs_test.go:98: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestEnsureSpanConfigReconciliation1527205484
    testing.go:1152: race detected during execution of test
--- FAIL: TestEnsureSpanConfigReconciliation (4.35s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-race -parallel=4

/cc @cockroachdb/kv @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-12857
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Feb 2, 2022
@blathers-crl blathers-crl bot added the T-kv KV Team label Feb 2, 2022
@irfansharif
Copy link
Contributor 

WARNING: DATA RACE
Write at 0x00c00217fc58 by goroutine 159:
  sync/atomic.CompareAndSwapInt32()
      /usr/local/go/src/runtime/race_amd64.s:334 +0xb
  sync/atomic.CompareAndSwapInt32()
      <autogenerated>:1 +0x1e
  github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync_race.go:32 +0x30
  github.com/cockroachdb/cockroach/pkg/util.(*EveryN).ShouldProcess()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/every_n.go:46 +0x52
  github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob.(*resumer).Resume.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob/job.go:91 +0xa4
  github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*Reconciler).Reconcile()
      /go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:159 +0x281
  github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob.(*resumer).Resume()
      /go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob/job.go:87 +0x667
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1123 +0x164
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1124 +0x8db
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:401 +0x74e
  github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:323 +0x125
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494 +0x235

Previous write at 0x00c00217fc58 by goroutine 342:
  github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob.(*resumer).Resume.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob/job.go:80 +0x9e
  github.com/cockroachdb/cockroach/pkg/settings.(*Values).settingChanged()
      /go/src/github.com/cockroachdb/cockroach/pkg/settings/values.go:148 +0xf4
  github.com/cockroachdb/cockroach/pkg/settings.(*Values).setInt64()
      /go/src/github.com/cockroachdb/cockroach/pkg/settings/values.go:154 +0x66
  github.com/cockroachdb/cockroach/pkg/settings.(*DurationSetting).set()
      /go/src/github.com/cockroachdb/cockroach/pkg/settings/duration.go:99 +0xe5
  github.com/cockroachdb/cockroach/pkg/settings.updater.Set()
      /go/src/github.com/cockroachdb/cockroach/pkg/settings/updater.go:118 +0x77b
  github.com/cockroachdb/cockroach/pkg/settings.(*updater).Set()
      <autogenerated>:1 +0xf1
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).setLocked()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:292 +0x4b0
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).handleKV()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:263 +0x56d
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start.func4()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:192 +0x4e
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run.func3()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:240 +0x9a
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:334 +0x1b3
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:287 +0x64
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x4c
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x96

@irfansharif irfansharif self-assigned this Feb 2, 2022
@cockroach-teamcity
Copy link
Member Author

migration/migrations.TestEnsureSpanConfigReconciliation failed with artifacts on master @ 0e5eb08368020e7633142fd60f0f818fcd46510d:

=== RUN   TestEnsureSpanConfigReconciliation
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/cf8619971a16ee574fbac84534c7eb1f/logTestEnsureSpanConfigReconciliation1750809104
    test_log_scope.go:80: use -show-logs to present logs inline
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • TAGS=bazel,gss

This test on roachdash | Improve this report!

otan added a commit to otan-cockroach/cockroach that referenced this issue Feb 3, 2022
@otan
migration/migrations: skip TestEnsureSpanConfigReconciliation
b81accf 
Refs: cockroachdb#75849

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None
@otan otan mentioned this issue Feb 3, 2022
Merged
craig bot pushed a commit that referenced this issue Feb 3, 2022
@ecwall @andreimatei @otan
Merge #75854 #75892 #75901 #75910
7499c27 
75854: sql: update SHOW GRANTS ON DATABASE to include grant options r=RichardJCai a=ecwall

refs #73394

Release note (sql): SHOW GRANTS ON DATABASE includes is_grantable column

75892: distsql: cleanup log tag annotation r=andreimatei a=andreimatei

A context annotation was not doing what it wanted; it was inadvertently
a no-op.

This patch also makes FlowCtx.AmbientCtx not be embedded. Embedded
AmbientCtxs are annoying because one can't easily find where they're
used.

Release note: None

75901: distsql: improve flow cleanup r=andreimatei a=andreimatei

FlowBase.Cleanup(ctx) was closing the tracing span from ctx, if any.
This was assuming that the respective ctx is the one associated with the
flow, and that the flow always has a span. These are brittle
assumptions; in particular, I intend to avoid opening spans for local
flows when there's no particular reason to (i.e. when not collecting
stats), to save on their cost.

This patch has the Flow explicitly keep track of which Span it owns, if
any.

Release note: None

75910: migration/migrations: skip TestEnsureSpanConfigReconciliation r=irfansharif a=otan

Refs: #75849

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None

Co-authored-by: Evan Wall <wall@cockroachlabs.com>
Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Co-authored-by: Oliver Tan <otan@cockroachlabs.com>
@sumeerbhola sumeerbhola mentioned this issue Feb 3, 2022
Merged
kvoli pushed a commit to kvoli/cockroach that referenced this issue Feb 7, 2022
@otan @kvoli
migration/migrations: skip TestEnsureSpanConfigReconciliation
b5f4988 
Refs: cockroachdb#75849

Reason: flaky test

Generated by bin/skip-test.

Release justification: non-production code changes

Release note: None
@AlexTalks AlexTalks added the S-3-productivity Severe issues that impede the productivity of CockroachDB developers. label Feb 8, 2022
irfansharif added a commit to irfansharif/cockroach that referenced this issue Feb 12, 2022
@irfansharif
spanconfig: squash benign data race
2a29e55 
Fixes cockroachdb#75849.

Release note: None
@tbg
Copy link
Member

tbg commented Mar 23, 2022

@irfansharif I see that you pushed a commit fixing this on Feb 12 but it doesn't seem to have made it into a PR. Is that coming?

@nvanbenschoten should this be KV Stability?

@tbg tbg changed the title migration/migrations: TestEnsureSpanConfigReconciliation failed migration/migrations: TestEnsureSpanConfigReconciliation failed [SettingsWatcher data race] Mar 23, 2022
@nvanbenschoten
Copy link
Member

The fix made it into #76466, but that hasn't been merged. I'll mark as a GA-blocker for now.

@craig craig bot closed this as completed in d893711 Mar 30, 2022
blathers-crl bot pushed a commit that referenced this issue Mar 30, 2022
@irfansharif
spanconfig: squash benign data race
f82ce57 
Fixes #75849.

Release note: None
fqazi pushed a commit to fqazi/cockroach that referenced this issue Apr 4, 2022
@irfansharif @fqazi
spanconfig: squash benign data race
cc5d44a 
Fixes cockroachdb#75849.

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot. S-3-productivity Severe issues that impede the productivity of CockroachDB developers. T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@AlexTalks @tbg @nvanbenschoten @irfansharif @cockroach-teamcity