Compatibility test suit

[waynexia]

What type of enhancement is this?

Tech debt reduction, User experience

What does the enhancement do?

With DB going stable, we should care about backward compatibility and forward compatibility between essential versions.

There is an existing compatibility test suit which leverages sqlness as harness. But it has some problems:

• Depending on a "bug" behavior of sqlness feat: add preserve arg to sqlness runner #3724
• Doesn't aware of feature version. I.e., some features are not available in the old version, and we may deprecate some features in the new version. Mechanisms like #[since(v0.7.0)] and #[till(v0.8.0)] looks necessary to compatibility test.
• It's based on bash script 😣 Prefer Rust/Python/Go etc that are easy to maintain

Implementation challenges

No response