after migrating from version 2.1.4 to 2.3.0 I'm unable to delete dags

[tronlightracer]

Apache Airflow version

main (development)

What happened

After changing the dockerfile in the astro cli to that of version 2.3.0 from version 2.1.4 I am unable to delete dags and am greeted with the 'Oops something bad had happened screen'

Ooops!
Something bad has happened.

Airflow is used by many users, and it is very likely that others had similar problems and you can easily find
a solution to your problem.

Consider following these steps:

  * gather the relevant information (detailed logs with errors, reproduction steps, details of your deployment)

  * find similar issues using:
     * [GitHub Discussions](https://github.com/apache/airflow/discussions)
     * [GitHub Issues](https://github.com/apache/airflow/issues)
     * [Stack Overflow](https://stackoverflow.com/questions/tagged/airflow)
     * the usual search engine you use on a daily basis

  * if you run Airflow on a Managed Service, consider opening an issue using the service support channels

  * if you tried and have difficulty with diagnosing and fixing the problem yourself, consider creating a [bug report](https://github.com/apache/airflow/issues/new/choose).
    Make sure however, to include all relevant details and results of your investigation so far.

Python version: 3.9.12
Airflow version: 2.3.0.dev20220411
Node: 5f22e42f2768
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.ForeignKeyViolation: update or delete on table "dag" violates foreign key constraint "dag_tag_dag_id_fkey" on table "dag_tag"
DETAIL:  Key (dag_id)=(builtin_macros_UNDRY) is still referenced from table "dag_tag".


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.9/site-packages/airflow/www/auth.py", line 40, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/www/decorators.py", line 80, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/www/views.py", line 1830, in delete
    delete_dag.delete_dag(dag_id)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/api/common/delete_dag.py", line 78, in delete_dag
    session.query(model)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3108, in delete
    result = self.session.execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1670, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1520, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 313, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1389, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1748, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1929, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.ForeignKeyViolation) update or delete on table "dag" violates foreign key constraint "dag_tag_dag_id_fkey" on table "dag_tag"
DETAIL:  Key (dag_id)=(builtin_macros_UNDRY) is still referenced from table "dag_tag".

[SQL: DELETE FROM dag WHERE dag.dag_id IN (%(dag_id_1_1)s) RETURNING dag.dag_id]
[parameters: {'dag_id_1_1': 'builtin_macros_UNDRY'}]
(Background on this error at: http://sqlalche.me/e/14/gkpj)

After trying to upgrade the database in the scheduler container with the airflow db upgrade command I'm given this error:

alembic.util.exc.CommandError: Requested revision 75d5ed6c2b43 overlaps with other requested revisions ccde3e26fe78
astro@7f8aa67

What you think should happen instead

I think airflow should be able to handle the upgrade from 2.1.4 to 2.3.0 seamlessly.

How to reproduce

run dags in 2.1.4
export the database to a sql file
stop astro cli
change dockerfile in astro cli to that of version 2.3.0
start astro cli
import the database previously exported
delete dags in webserver ui
Oops screen should appear

Operating System

Docker (debian:buster)

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

Astro cli with images:

FROM quay.io/astronomer/ap-airflow-dev:2.1.4-7-buster-nightly-20220411

FROM quay.io/astronomer/ap-airflow-dev:2.3.0.dev20220411

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[uranusjr]

Hmm I don’t seem to recall we changed delete_dag logic recently on cleaning related objects. The tags relationship on DagModel does declare cascade (unchanged for the last 2 years):

airflow/airflow/models/dag.py

Line 2663 in a0f7e61

tags = relationship('DagTag', cascade='all,delete-orphan', backref=backref('dag'))

Not sure why this suddenly happens.

[ephraimbuddy]

@tronlightracer can you provide the exact images you used(links to them).
That said, the way you did your upgrade is not in line with how you do upgrade with astro cli. Using astro cli, you just change the image on the dockerfile. No need to export and reimport database