fix(sqllab): Inefficient SqlaTable query (#24343)

[giftig]

Use a combination of lazyload and load_only to ensure sqlalchemy doesn't eagerly join to additional tables, which was happening in a very inefficient way for a large number of tables and resulting in this endpoint never returning (in addition to high db load)

SUMMARY

Makes the lookup for extra for tables listed by schema much more efficient, especially when there is a large number of tables present.

TESTING INSTRUCTIONS

1. Populate a few thousand datasources for tables in a schema you want to list
2. Go to SQL Lab and try to get the table list for your selected schema
3. Observe the call times out as the API responds in a timely manner

More info on the associated issue: #24343

ADDITIONAL INFORMATION

• Has associated issue: SqlaTable query hangs on list tables API call with large number of tables #24343
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #24344 (7220a47) into master (0e3f1f6) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head 7220a47 differs from pull request most recent head bb5d308. Consider uploading reports for the commit bb5d308 to get more accurate results

@@           Coverage Diff           @@
##           master   #24344   +/-   ##
=======================================
  Coverage   69.09%   69.09%           
=======================================
  Files        1903     1903           
  Lines       74608    74610    +2     
  Branches     8107     8107           
=======================================
+ Hits        51550    51552    +2     
  Misses      20947    20947           
  Partials     2111     2111           

Flag	Coverage Δ
hive	53.79% <100.00%> (+<0.01%)	⬆️
mysql	79.39% <100.00%> (+<0.01%)	⬆️
postgres	79.47% <100.00%> (+<0.01%)	⬆️
presto	53.71% <100.00%> (+<0.01%)	⬆️
python	83.38% <100.00%> (+<0.01%)	⬆️
sqlite	77.99% <100.00%> (+<0.01%)	⬆️
unit	54.20% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/databases/commands/tables.py	100.00% <100.00%> (ø)
superset/views/core.py	76.26% <100.00%> (+0.01%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[john-bodley]

Thanks @giftig for the change. It seems this endpoint is deprecated (and slated for deletion in #24342) and thus I was wondering whether you could make said change to the replacement API endpoint?

[villebro]

LGTM, big thanks for this perf improvement!

[villebro]

Thanks @giftig for the change. It seems this endpoint is deprecated (and slated for deletion in #24342) and thus I was wondering whether you could make said change to the replacement API endpoint?

@john-bodley as this is fixing a serious perf regression affecting master branch right now, I suggest we defer the API endpoint move to a follow-up PR.

Edit: Never mind, I read this too hastily; @giftig , can you apply the same fix to this code to ensure this doesn't break again when #24342 is merged?

superset/superset/databases/commands/tables.py

Lines 74 to 82 in 0e3f1f6

	extra_dict_by_name = {
	table.name: table.extra_dict
	for table in (
	db.session.query(SqlaTable).filter(
	SqlaTable.database_id == self._model.id,
	SqlaTable.schema == self._schema_name,
	)
	).all()
	}