Automatic micro-testing

[neverchanje]

As we are heading toward delivering a production-ready database, the micro testing frameworks can be used to ensure that we have covered as many cases as possible. In the meantime, we will conduct manual testing (or "stress testing") and observe the key metrics when processing several selected queries in order to ensure that the system does not behave unexpectedly, such as OOM, under a relatively high level of load. Yet, the manual testing steps are not the topic of this issue.

Source micro-testing framework

Source reading/parsing

The parsing part will be more deterministic than the rest. We need to generate random data in a specific format (with a probability of generating false data, the expected behavior is to drop it) and verify the correctness of the parsed output.

• MockSource x Protobuf
• MockSource x JSON test: a test framework for source parsing #5512
• MockSource x Debezium JSON
• MockSource x Avro

Failover

In this test, we need to ensure that RisingWave can achieve at-least-once delivery. There must be no data loss or wrong data ordering. The program will include a Meta and a simplified CN, which only runs a wrapped SourceExecutor that consumes the upstream and checks the data ordering. When the CN receives a periodic barrier, it flushes its state to ensure that no data older than the checkpointed offset has been consumed. Also, there shouldn’t be a hole (gap) between the consumed data and the new incoming data. The checking can be implemented as debug_assert, which can be turned off on production (or enabled if necessary).

The simplified CN will be killed periodically, like for 30s at a time. We can run the testing daily, each for 1h.

• micro-testing: Redpanda x JSON (Partition number: 16, Parallelism: 4)NOTE: Kafka/Redpanda is the highest priority #5221
  NOTE: Kafka/Redpanda is the highest priority
• Kinesis x JSON
• Pulsar x JSON (Partition number: 16, Parallelism: 4)

SQL testing

• Now that we have sqlogictest and sqlsmith, the stability of our SQL support has been quite satisfying.
  The issues are tracked in Tracking Issue: Fix Sqlsmith workarounds / refinements / bugs #3896

Hummock testing

Compaction / Vacuum

We will conduct four types of test to improve compaction/vacuum coverage:

• Functionality test. Target coverage ≥ 85%.
  1. Compaction/vacuum task generation (i.e. compaction strategy) refactor(compaction): add more tests #5208
  2. Compaction/vacuum task scheduling and assignment refactor(compaction): unify task cancellation #5240
  3. Compaction worker logic (build SST, compaction filter, bloom filter generation, …)
• Deterministic replay Hummock MANIFEST. We will have a framework implemented to reply Hummock MANIFEST to verify foreground reads produce the same result before and after compaction. feat(storage): Deterministic Compaction Test Tool #5464
• (Write-only) Stress test. We will verify the whole compaction process can sustain under reasonable stress without crashing by putting excessive write load into RisingWave.

Read-path

The testing can be covered by batch-query testing.

Batch-query testing

The motivation is to verify the query correctness and the stability of the query path while querying a large dataset.

The approach is to first generate the dataset via tpch and ch-benchmark and load the data into both Postgres and RisingWave. The Postgres is used as a source of truth. We will then perform different queries and compare the query results of both databases.

Streaming testing

Deterministic testing

By the end of 9/30, we will heavily rely on MadSim for testing the streaming execution since it’s currently the most effective way to achieve high coverage. The development plan:

• Enable mocking Etcd RPC feat(test): add etcd simulator #5084
• Enable mocking S3 test: add S3 simulator #5161
• Enable mocking Kafka test: build a kafka simulator #5181
• Recovery testing: The basic idea is to set up a madsim cluster that will randomly kill one of the instances, and while consuming the source, the service must remain available after recovery. After it finishes processing, there will be a checker to verify if the computation results are correct. test(recovery): add recovery test for nexmark stream #7623

Sink testing

TBD

[fuyufjh]

Hi @neverchanje Any updates or future plan on this? It has been silent for a few months and I am considering to close it...

[neverchanje]

Checked through the items two weeks ago with @wangrunji0408 I think this issue is now blocked by #5161 and #5170 . Once it's done, we can close this issue and track other issues separately.

[neverchanje]

I've re-assigned this issue to @wangrunji0408. Please decide whether to close this issue and the depending issues if you think they add little value to our QA process. Personally, I think both the simulated s3 and the random killer can greatly help us ensure stability under extreme situations.

[wangrunji0408]

Okay. The S3 simulator is under final review and integration. madsim-rs/madsim#107. Will close this issue when it is done.

[wangrunji0408]

I'm closing this issue as the main work on simulation testing is done.