fix: added scoped session for SQL-based alerting

[JasonD28]

SUMMARY

Multiple errors such as sqlalchemy.exc.NoSuchColumnError, sqlalchemy.exc.ResourceClosedError and MySQLdb._exceptions.OperationalError were being seen when multiple alert celery tasks were run concurrently. These errors occurred during the first few iterations of a celery beat schedule after initialization, however they would stop after after a few iterations of the beat schedule. This PR uses db.create_scoped_session() for alerting celery tasks to fix these errors. Retrying is implemented along with the celery task as a temporary fix for #10530

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

• local test
• dropbox staging
• dropbox prod

ADDITIONAL INFORMATION

• Has associated issue: celery SQLAlchemy errors in celery tasks that use sqlalchemy pools #10530

[villebro]

FYI @john-bodley , interesting empirical observations for when scoped sessions can be helpful. This might explain the presence of some of the scoped sessions that were removed in #10427

[villebro]

@JasonD28 Did this change solve the errors completely?

[JasonD28]

@villebro yes it did, but a better solution than retrying would be ideal to solve #10530

[villebro]

I'm trying to determine what the effect of scoped sessions vs vanilla sessions is. Did you try with scoped sessions without the retry, did that improve reliability?

[JasonD28]

scoped sessions without retry did improve reliability with the exception of #10530

for reference, before adding scoped sessions sqlalchemy.exc.NoSuchColumnError, sqlalchemy.exc.ResourceClosedError and MySQLdb._exceptions.OperationalError were seen at these places: https://github.com/apache/incubator-superset/blob/ece91928a9339190163c0bc72b96e51217a90d1e/superset/tasks/schedules.py#L544 https://github.com/apache/incubator-superset/blob/ece91928a9339190163c0bc72b96e51217a90d1e/superset/tasks/schedules.py#L570 https://github.com/apache/incubator-superset/blob/ece91928a9339190163c0bc72b96e51217a90d1e/superset/tasks/schedules.py#L626

[villebro]

@JasonD28 thanks for the context, really helpful! 👍

[john-bodley]

@villebro thanks for being looped in. I do wonder if the scoped session is merely masking some other issue we haven’t addresses, i.e., it seems somewhat nonintuitive that a scoped session is necessary here.

[JasonD28]

@villebro @john-bodley further testing showed that just implementing retrying was sufficient enough for the celery task to run without any errors. While non-scoped sessions invoked retrying more often than scoped sessions, I'm going to close this PR and move forward with non-scoped sessions #10542 until more information is know about whats causing the error.