roachtest: schemachange/index/tpcc/w=1000 failed

[cockroach-teamcity]

roachtest.schemachange/index/tpcc/w=1000 failed with artifacts on master @ cc6296c24ddb048215dabe5cc41339f306db4f41:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/index/tpcc/w=1000/run_1
	schemachange.go:491,schemachange.go:318,monitor.go:106,errgroup.go:57: pq: polling for queued jobs to complete: poll-show-jobs: scan with start key /Table/15/1/699727443913834497: root: memory budget exceeded: 10240 bytes requested, 3773286400 currently allocated, 3773295616 bytes in budget

	monitor.go:128,tpcc.go:245,schemachange.go:312,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:245
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange.go:312
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6309
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:208
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

• roachtest: schemachange/index/tpcc/w=1000 failed #54304 roachtest: schemachange/index/tpcc/w=1000 failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-20.2]

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

[ajwerner]

No open sessions. A bunch of the debug zip commands failed. Nothing of interest in the heap profile. Smells like a leak in a memory monitor somewhere.

[stevendanna]

Might be fixed by #71280

[ajwerner]

Looks right to me ✅. Assuming fixed by #71280.