Charset conversion does not work properly

[cyliu0]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

CREATE TABLE t1 (
  a ENUM('ä','ö','ü') character set latin1 default 'ü'
);
insert into t1 values ('ä'),('ö'),('ü');
select * from t1;
set charset latin1;
select * from t1;

2. What did you expect to see? (Required)

On MySQL:

mysql root@127.0.0.1:test> select * from t1
+----+
| a  |
+----+
| ä |
| ö |
| ü |
+----+
3 rows in set
Time: 0.016s
mysql root@127.0.0.1:test> set charset latin1;
Query OK, 0 rows affected
Time: 0.002s
mysql root@127.0.0.1:test> select * from t1
+---+
| a |
+---+
| ä |
| ö |
| ü |
+---+
3 rows in set
Time: 0.015s

3. What did you see instead (Required)

On TiDB:

mysql root@127.0.0.1:test> select * from t1
+----+
| a  |
+----+
| ä |
| ö |
| ü |
+----+
3 rows in set
Time: 0.014s
mysql root@127.0.0.1:test> set charset latin1
Query OK, 0 rows affected
Time: 0.001s
mysql root@127.0.0.1:test> select * from t1
+----+
| a  |
+----+
| ä |
| ö |
| ü |
+----+
3 rows in set
Time: 0.014s

4. Affected version (Required)

mysql root@127.0.0.1:test> select tidb_version()
+-------------------------------------------------------------------+
| tidb_version()                                                    |
+-------------------------------------------------------------------+
| Release Version: v4.0.0-beta.2-893-g4e829aaee                     |
| Edition: Community                                                |
| Git Commit Hash: 4e829aaee7b656aa807814708ae05af5233302af         |
| Git Branch: master                                                |
| UTC Build Time: 2020-08-05 02:23:17                               |
| GoVersion: go1.14.4                                               |
| Race Enabled: false                                               |
| TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 |
| Check Table Before Drop: false                                    |
+-------------------------------------------------------------------+
1 row in set
Time: 0.014s

5. Root Cause Analysis

[ghost]

I believe this relates to #18955

TiDB does not support latin1 correctly - it treats it as a subset of utf8. This is only safe for the lower 128 characters of latin1.

[wjhuang2016]

I think this problem can be solved after we implement GBK support.

[wjhuang2016]

Presently, it's the "expected behavior". Because we don't truly support "character_set_results". So I think it's should be a feature request: support "character_set_results".