Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YSQL] ERROR: The column already exists when trying to add column after dropping with drop domain cascade #22874

Closed
1 task done
archit-rastogi opened this issue Jun 14, 2024 · 0 comments
Assignees
Labels
2.20 Backport Required 2024.1 Backport Required area/docdb YugabyteDB core features area/ysql Yugabyte SQL (YSQL) kind/bug This issue is a bug priority/medium Medium priority issue

Comments

@archit-rastogi
Copy link

archit-rastogi commented Jun 14, 2024

Jira Link: DB-11776

Description

Step to reproduce:

  1. create domain
  2. create table with one or more columns using the domain type created in step 1.
  3. drop domain cascade
  4. check table schema with \d, column(s) must be dropped.
  5. create domain again
  6. alter table add column with same name and domain type in step (5). ALTER table fails with column already exists.

Example:

yugabyte=# CREATE DOMAIN postal_code AS TEXT CHECK(VALUE ~ '^\d{5}$' OR VALUE ~ '^\d{5}-\d{4}$');
CREATE DOMAIN
yugabyte=# CREATE TABLE person(first_name TEXT, last_name TEXT, post postal_code, joining date);
CREATE TABLE
yugabyte=# insert into person values ('arc', 'ras', 12345, '12-12-2000');
yugabyte=# drop domain postal_code cascade;
NOTICE:  drop cascades to column post of table person
DROP DOMAIN
yugabyte=# \d person
               Table "public.person"
   Column   | Type | Collation | Nullable | Default 
------------+------+-----------+----------+---------
 first_name | text |           |          | 
 last_name  | text |           |          | 
 joining    | date |           |          | 

yugabyte=# CREATE DOMAIN postal_code AS TEXT CHECK(VALUE ~ '^\d{5}$' OR VALUE ~ '^\d{5}-\d{4}$');
CREATE DOMAIN
yugabyte=# alter table person add column post postal_code;
ERROR:  The column already exists: post
yugabyte=# alter table person add column post postal_code;
ERROR:  The column already exists: post
yugabyte=# \d
         List of relations
 Schema |  Name  | Type  |  Owner   
--------+--------+-------+----------
 public | person | table | yugabyte
(1 row)

yugabyte=# \d person
               Table "public.person"
   Column   | Type | Collation | Nullable | Default 
------------+------+-----------+----------+---------
 first_name | text |           |          | 
 last_name  | text |           |          | 
 joining    | date |           |          | 

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@archit-rastogi archit-rastogi added area/ysql Yugabyte SQL (YSQL) status/awaiting-triage Issue awaiting triage labels Jun 14, 2024
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Jun 14, 2024
@sushantrmishra sushantrmishra removed the status/awaiting-triage Issue awaiting triage label Jun 14, 2024
fizaaluthra added a commit that referenced this issue Jun 27, 2024
Summary:
Currently, when an object that a column depends on is dropped, the drop isn't
correctly cascaded to the column in the DocDB table itself.

Example:

```
yugabyte=# CREATE TABLE test_dropcolumn(a int, b int, c int);
CREATE TABLE
yugabyte=# CREATE TYPE test_dropcolumn_type AS (a int, b int);
CREATE TYPE
yugabyte=# ALTER TABLE test_dropcolumn ADD COLUMN d test_dropcolumn_type;
ALTER TABLE
yugabyte=# DROP TYPE test_dropcolumn_type CASCADE;
NOTICE:  drop cascades to column t of table person
DROP TYPE
yugabyte=# ALTER TABLE test_dropcolumn ADD COLUMN d int;
ERROR:  The column already exists: d
```
Although the PG metadata for the column is dropped, the column itself is not
dropped in DocDB. This leads to an inconsistency between DocDB and PG.

This diff fixes the issue by adding a YB drop for the column inside `doDeletion`.
Note: this function is also used during `ALTER TABLE... DROP COLUMN`; however,
alter table commands execute YB schema changes separately (see `YBCPrepareAlterTableCmd`
and `YBCExecAlterTable`). So, an additional flag is added to skip the YB column
drop in `doDeletion` in case it is being invoked by the alter table flow.
Jira: DB-11776

Test Plan: yb_alter_table

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D35866
@yugabyte-ci yugabyte-ci added the area/docdb YugabyteDB core features label Jun 28, 2024
jasonyb pushed a commit that referenced this issue Jun 28, 2024
Summary:
 9c637e2 [PLAT-14429]: Modify Troubleshooting Platform registration workflow in YBA
 0a1406d [PLAT-14098]: Updating yb.runtime_conf_ui.tag_filter with appropriate tags value does not display the flags accordingly
 70a87f9 [PLAT-13605][PLAT-13609]: Edit Volume controls and storage type in FULL_MOVE but not in case of UPDATE
 26fbfe0 [PLAT-14515][UI] Clicking preview doesn't show the correct info and clears up the data provided while setting up the ysql_ident or ysql_hba multiline flags.- [PLAT-14514] [PLAT-14513]
 a07946b [#18233] Initial commit for yugabyted unit test framework.
 b2e8ee7 [#22842] docdb: Improve usability of stack trace tracking endpoints
 508f26e [docs] Added RN 2.20.2.3-b2 (#23042)
 214d44a [#22935] YSQL: Use db oid in the tserver's sequence cache entry key
 c47b2d9 [#22802] YSQL: Avoid renaming DocDb tables during legacy rewrite
 Excluded: 7c8343d [#22874] YSQL: Fix cascaded drops on columns
 58c8d4e [#23046] xCluster: Remove ns_replication from the code
 a70681d [#22816] YSQL: Bug fixes for replication connections in ysql connection manager
 b239e07 Doc upgrade versions (#22988)
 Excluded: ec76062 [#16408] YSQL: Split TestPgRegressIndex.testPgRegressIndex

Test Plan: Jenkins: rebase: pg15-cherrypicks

Reviewers: jason, tfoucher

Subscribers: yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D36247
fizaaluthra added a commit that referenced this issue Jul 2, 2024
Summary:
Cherry-pick YB master commit 7c8343d titled

    [#22874] YSQL: Fix cascaded drops on columns

and committed 2024-06-27T12:41:11-04:00 into YB pg15.

Upstream PG commit 1df5875d39383b3981b804666ee1f4b0ff65943f changes
the `ATExecDropColumn` to use `performMultipleDeletions` (instead of `performDeletion`).

YB master commit 7c8343d changes the
`performDeletion` call to use `YB_SKIP_YB_DROP_COLUMN` as the flag parameter.

Resolve the conflict by using `YB_SKIP_YB_DROP_COLUMN` as the flag parameter
in the performMultipleDeletions call so that we skip YB column drops in `doDeletion`.
Jira: DB-11776

Test Plan:
./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressTable'
./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressTypesUDT'

Manually verified the test introduced by PG commit 1df5875d39383b3981b804666ee1f4b0ff65943f:
```
-- check that dropping a column takes with it any partitioned indexes
-- depending on it.
create table parted_index_col_drop(a int, b int, c int)
  partition by list (a);
create table parted_index_col_drop1 partition of parted_index_col_drop
  for values in (1) partition by list (a);
-- leave this partition without children.
create table parted_index_col_drop2 partition of parted_index_col_drop
  for values in (2) partition by list (a);
create table parted_index_col_drop11 partition of parted_index_col_drop1
  for values in (1);
create index on parted_index_col_drop (b);
create index on parted_index_col_drop (c);
create index on parted_index_col_drop (b, c);
alter table parted_index_col_drop drop column c;
\d parted_index_col_drop
\d parted_index_col_drop1
\d parted_index_col_drop2
\d parted_index_col_drop11
drop table parted_index_col_drop;
```

Reviewers: jason, tfoucher

Reviewed By: jason

Subscribers: yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D36309
fizaaluthra added a commit that referenced this issue Jul 12, 2024
Summary:
Commit 7c8343d / D35866 summary:

Currently, when an object that a column depends on is dropped, the drop isn't
correctly cascaded to the column in the DocDB table itself.

Example:

```
yugabyte=# CREATE TABLE test_dropcolumn(a int, b int, c int);
CREATE TABLE
yugabyte=# CREATE TYPE test_dropcolumn_type AS (a int, b int);
CREATE TYPE
yugabyte=# ALTER TABLE test_dropcolumn ADD COLUMN d test_dropcolumn_type;
ALTER TABLE
yugabyte=# DROP TYPE test_dropcolumn_type CASCADE;
NOTICE:  drop cascades to column t of table person
DROP TYPE
yugabyte=# ALTER TABLE test_dropcolumn ADD COLUMN d int;
ERROR:  The column already exists: d
```
Although the PG metadata for the column is dropped, the column itself is not
dropped in DocDB. This leads to an inconsistency between DocDB and PG.

This diff fixes the issue by adding a YB drop for the column inside `doDeletion`.
Note: this function is also used during `ALTER TABLE... DROP COLUMN`; however,
alter table commands execute YB schema changes separately (see `YBCPrepareAlterTableCmd`
and `YBCExecAlterTable`). So, an additional flag is added to skip the YB column
drop in `doDeletion` in case it is being invoked by the alter table flow.
Jira: DB-11776

Commit b81dbaa / D36329 summary:

The test became flaky after commit 7c8343d fixed the logic for cascaded drops on columns.

The test yb_feature_hash_types creates a domain and creates a table with a key column that has the domain as its type. However, since we currently don't support key column drops (#22902), the test fails when the cleanUpCustomEntities() function in BasePgSQLTest.java attempts to drop the domain (that has the key column as a dependant object) during cleanup.
Moreover, the test is flaky because sometimes the table may be dropped before the domain, as the order of the drops depends on the pg_depend scan output, which isn't deterministic because pg_depend doesn't have a primary key and internally uses ybctid.

Fix the test flakiness by dropping the table in yb_feature_hash_types before the test clean up logic.
Jira: DB-12029

Original commits:
7c8343d / D35866
b81dbaa / D36329

Test Plan: yb_alter_table

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D36286
fizaaluthra added a commit that referenced this issue Jul 12, 2024
Summary:
Commit 7c8343d / D35866 summary:

Currently, when an object that a column depends on is dropped, the drop isn't
correctly cascaded to the column in the DocDB table itself.

Example:

yugabyte=# CREATE TABLE test_dropcolumn(a int, b int, c int);
CREATE TABLE
yugabyte=# CREATE TYPE test_dropcolumn_type AS (a int, b int);
CREATE TYPE
yugabyte=# ALTER TABLE test_dropcolumn ADD COLUMN d test_dropcolumn_type;
ALTER TABLE
yugabyte=# DROP TYPE test_dropcolumn_type CASCADE;
NOTICE:  drop cascades to column t of table person
DROP TYPE
yugabyte=# ALTER TABLE test_dropcolumn ADD COLUMN d int;
ERROR:  The column already exists: d
Although the PG metadata for the column is dropped, the column itself is not
dropped in DocDB. This leads to an inconsistency between DocDB and PG.

This diff fixes the issue by adding a YB drop for the column inside doDeletion.
Note: this function is also used during ALTER TABLE... DROP COLUMN; however,
alter table commands execute YB schema changes separately (see YBCPrepareAlterTableCmd
and YBCExecAlterTable). So, an additional flag is added to skip the YB column
drop in doDeletion in case it is being invoked by the alter table flow.
Jira: DB-11776

Commit b81dbaa / D36329 summary:

The test became flaky after commit 7c8343d fixed the logic for cascaded drops on columns.

The test yb_feature_hash_types creates a domain and creates a table with a key column that has the domain as its type. However, since we currently don't support key column drops (#22902), the test fails when the cleanUpCustomEntities() function in BasePgSQLTest.java attempts to drop the domain (that has the key column as a dependant object) during cleanup.
Moreover, the test is flaky because sometimes the table may be dropped before the domain, as the order of the drops depends on the pg_depend scan output, which isn't deterministic because pg_depend doesn't have a primary key and internally uses ybctid.

Fix the test flakiness by dropping the table in yb_feature_hash_types before the test clean up logic.
Jira: DB-12029

Original commits:
7c8343d / D35866
b81dbaa / D36329

Test Plan: yb_alter_table

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D36289
fizaaluthra added a commit that referenced this issue Jul 12, 2024
Summary:
Commit 7c8343d / D35866 summary:

Currently, when an object that a column depends on is dropped, the drop isn't
correctly cascaded to the column in the DocDB table itself.

Example:

```
yugabyte=# CREATE TABLE test_dropcolumn(a int, b int, c int);
CREATE TABLE
yugabyte=# CREATE TYPE test_dropcolumn_type AS (a int, b int);
CREATE TYPE
yugabyte=# ALTER TABLE test_dropcolumn ADD COLUMN d test_dropcolumn_type;
ALTER TABLE
yugabyte=# DROP TYPE test_dropcolumn_type CASCADE;
NOTICE:  drop cascades to column t of table person
DROP TYPE
yugabyte=# ALTER TABLE test_dropcolumn ADD COLUMN d int;
ERROR:  The column already exists: d
```
Although the PG metadata for the column is dropped, the column itself is not
dropped in DocDB. This leads to an inconsistency between DocDB and PG.

This diff fixes the issue by adding a YB drop for the column inside `doDeletion`.
Note: this function is also used during `ALTER TABLE... DROP COLUMN`; however,
alter table commands execute YB schema changes separately (see `YBCPrepareAlterTableCmd`
and `YBCExecAlterTable`). So, an additional flag is added to skip the YB column
drop in `doDeletion` in case it is being invoked by the alter table flow.
Jira: DB-11776

Commit b81dbaa / D36329 summary:

The test became flaky after commit 7c8343d fixed the logic for cascaded drops on columns.

The test yb_feature_hash_types creates a domain and creates a table with a key column that has the domain as its type. However, since we currently don't support key column drops (#22902), the test fails when the cleanUpCustomEntities() function in BasePgSQLTest.java attempts to drop the domain (that has the key column as a dependant object) during cleanup.
Moreover, the test is flaky because sometimes the table may be dropped before the domain, as the order of the drops depends on the pg_depend scan output, which isn't deterministic because pg_depend doesn't have a primary key and internally uses ybctid.

Fix the test flakiness by dropping the table in yb_feature_hash_types before the test clean up logic.
Jira: DB-12029

Original commits:
7c8343d / D35866
b81dbaa / D36329

Test Plan: yb_alter_table

Reviewers: myang

Reviewed By: myang

Subscribers: yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D36290
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.20 Backport Required 2024.1 Backport Required area/docdb YugabyteDB core features area/ysql Yugabyte SQL (YSQL) kind/bug This issue is a bug priority/medium Medium priority issue
Projects
None yet
Development

No branches or pull requests

4 participants