Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply new naming convention to ferc714 and eia861 output assets #2882

Merged
merged 6 commits into from
Sep 26, 2023
Merged

Apply new naming convention to ferc714 and eia861 output assets #2882

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
239eb4d
Apply new naming convention to ferc714 respondents, hourly demand and…
bendnorman Sep 21, 2023
5ebedcd
Fix refs to renamed tables in release notes
bendnorman Sep 21, 2023
6ffe6a5
Rename ferc714 and eia861 output tables in integration tests
bendnorman Sep 21, 2023
d257e52
Merge branch 'rename-core-assets' into rename-annualized-respondents-…
e-belfer Sep 21, 2023
97149cb
Add missing balance authority fk migration
bendnorman Sep 25, 2023
6a5411a
Rename out_ferc714__fipsified_respondents to out_ferc714__respondents…
bendnorman Sep 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions docs/release_notes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,8 +173,8 @@ Data Coverage

* :ref:`core_ferc714__respondent_id` (linking FERC-714 respondents to EIA utilities)
* :ref:`core_ferc714__hourly_demand_pa` (hourly electricity demand by planning area)
* :ref:`fipsified_respondents_ferc714` (annual respondents with county FIPS IDs)
* :ref:`summarized_demand_ferc714` (annual demand for FERC-714 respondents)
* :ref:`out_ferc714__fipsified_respondents` (annual respondents with county FIPS IDs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

non-blocking: respondents_with_fips might be clearer

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, but I want to preserve the PudlTabl API for now.

Copy link
Member

@e-belfer e-belfer Sep 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% following. It should be possible to change the underlying table asset name and keep the PudlTabl method name as it was without altering access (the same as for all the other assets). In which case we'd still want to update the release notes to refer to the actual table, as you have here.

Copy link
Member Author

@bendnorman bendnorman Sep 26, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I'm sorry I thought you were referring to the PudlTabl method. I just renamed the table to out_ferc714__respondents_with_fips .

* :ref:`out_ferc714__summarized_demand` (annual demand for FERC-714 respondents)

* Added new table :ref:`core_epa__assn_epacamd_eia_subplant_ids`, which aguments the
:ref:`core_epa__assn_epacamd_eia` glue table. This table incorporates all
Expand Down Expand Up @@ -272,9 +272,10 @@ Analysis
* Added outputs from :mod:`pudl.analysis.service_territory` and
:mod:`pudl.analysis.state_demand` into PUDL. These outputs include the US Census
geometries associated with balancing authority and utility data from EIA 861
(:ref:`compiled_geometry_balancing_authority_eia861` and
:ref:`compiled_geometry_utility_eia861`), and the estimated total hourly electricity
demand for each US state in :ref:`predicted_state_hourly_demand`. See :issue:`1973`
(:ref:`out_eia861__compiled_geometry_balancing_authorities` and
:ref:`out_eia861__compiled_geometry_utilities`), and the estimated total hourly
electricity demand for each US state in
:ref:`out_ferc714__hourly_predicted_state_demand`. See :issue:`1973`
and :pr:`2550`.

Deprecations
Expand Down
94 changes: 94 additions & 0 deletions migrations/versions/027d56be4d4e_rename_service_territory_and_demand_.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
"""Rename service territory and demand tables

Revision ID: 027d56be4d4e
Revises: 79158fe4f428
Create Date: 2023-09-21 08:38:10.224197

"""
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import sqlite

# revision identifiers, used by Alembic.
revision = '027d56be4d4e'
down_revision = '79158fe4f428'
branch_labels = None
depends_on = None


def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('out_eia861__compiled_geometry_balancing_authority',
sa.Column('county_id_fips', sa.Text(), nullable=False, comment='County ID from the Federal Information Processing Standard Publication 6-4.'),
sa.Column('county_name_census', sa.Text(), nullable=True, comment='County name as specified in Census DP1 Data.'),
sa.Column('population', sa.Float(), nullable=True, comment='County population, sourced from Census DP1 data.'),
sa.Column('area_km2', sa.Float(), nullable=True, comment='County area in km2.'),
sa.Column('report_date', sa.Date(), nullable=False, comment='Date reported.'),
sa.Column('balancing_authority_id_eia', sa.Integer(), nullable=False, comment='EIA balancing authority ID. This is often (but not always!) the same as the utility ID associated with the same legal entity.'),
sa.Column('state', sa.Text(), nullable=True, comment='Two letter US state abbreviation.'),
sa.Column('county', sa.Text(), nullable=False, comment='County name.'),
sa.Column('state_id_fips', sa.Text(), nullable=True, comment='Two digit state FIPS code.'),
sa.PrimaryKeyConstraint('balancing_authority_id_eia', 'report_date', 'county_id_fips', 'county', name=op.f('pk_out_eia861__compiled_geometry_balancing_authority'))
)
op.create_table('out_eia861__compiled_geometry_utility',
sa.Column('county_id_fips', sa.Text(), nullable=False, comment='County ID from the Federal Information Processing Standard Publication 6-4.'),
sa.Column('county_name_census', sa.Text(), nullable=True, comment='County name as specified in Census DP1 Data.'),
sa.Column('population', sa.Float(), nullable=True, comment='County population, sourced from Census DP1 data.'),
sa.Column('area_km2', sa.Float(), nullable=True, comment='County area in km2.'),
sa.Column('report_date', sa.Date(), nullable=False, comment='Date reported.'),
sa.Column('utility_id_eia', sa.Integer(), nullable=False, comment='The EIA Utility Identification number.'),
sa.Column('state', sa.Text(), nullable=True, comment='Two letter US state abbreviation.'),
sa.Column('county', sa.Text(), nullable=True, comment='County name.'),
sa.Column('state_id_fips', sa.Text(), nullable=True, comment='Two digit state FIPS code.'),
sa.PrimaryKeyConstraint('utility_id_eia', 'report_date', 'county_id_fips', name=op.f('pk_out_eia861__compiled_geometry_utility'))
)
op.create_table('out_ferc714__hourly_predicted_state_demand',
sa.Column('state_id_fips', sa.Text(), nullable=False, comment='Two digit state FIPS code.'),
sa.Column('utc_datetime', sqlite.DATETIME(), nullable=False),
sa.Column('demand_mwh', sa.Float(), nullable=True),
sa.Column('scaled_demand_mwh', sa.Float(), nullable=True, comment='Estimated electricity demand scaled by the total sales within a state.'),
sa.PrimaryKeyConstraint('state_id_fips', 'utc_datetime', name=op.f('pk_out_ferc714__hourly_predicted_state_demand'))
)
op.drop_table('compiled_geometry_balancing_authority_eia861')
op.drop_table('predicted_state_hourly_demand')
op.drop_table('compiled_geometry_utility_eia861')
# ### end Alembic commands ###


def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('compiled_geometry_utility_eia861',
sa.Column('county_id_fips', sa.TEXT(), nullable=False),
sa.Column('county_name_census', sa.TEXT(), nullable=True),
sa.Column('population', sa.FLOAT(), nullable=True),
sa.Column('area_km2', sa.FLOAT(), nullable=True),
sa.Column('report_date', sa.DATE(), nullable=False),
sa.Column('utility_id_eia', sa.INTEGER(), nullable=False),
sa.Column('state', sa.TEXT(), nullable=True),
sa.Column('county', sa.TEXT(), nullable=True),
sa.Column('state_id_fips', sa.TEXT(), nullable=True),
sa.PrimaryKeyConstraint('utility_id_eia', 'report_date', 'county_id_fips', name='pk_compiled_geometry_utility_eia861')
)
op.create_table('predicted_state_hourly_demand',
sa.Column('state_id_fips', sa.TEXT(), nullable=False),
sa.Column('utc_datetime', sa.DATETIME(), nullable=False),
sa.Column('demand_mwh', sa.FLOAT(), nullable=True),
sa.Column('scaled_demand_mwh', sa.FLOAT(), nullable=True),
sa.PrimaryKeyConstraint('state_id_fips', 'utc_datetime', name='pk_predicted_state_hourly_demand')
)
op.create_table('compiled_geometry_balancing_authority_eia861',
sa.Column('county_id_fips', sa.TEXT(), nullable=False),
sa.Column('county_name_census', sa.TEXT(), nullable=True),
sa.Column('population', sa.FLOAT(), nullable=True),
sa.Column('area_km2', sa.FLOAT(), nullable=True),
sa.Column('report_date', sa.DATE(), nullable=False),
sa.Column('balancing_authority_id_eia', sa.INTEGER(), nullable=False),
sa.Column('state', sa.TEXT(), nullable=True),
sa.Column('county', sa.TEXT(), nullable=False),
sa.Column('state_id_fips', sa.TEXT(), nullable=True),
sa.PrimaryKeyConstraint('balancing_authority_id_eia', 'report_date', 'county_id_fips', 'county', name='pk_compiled_geometry_balancing_authority_eia861')
)
op.drop_table('out_ferc714__hourly_predicted_state_demand')
op.drop_table('out_eia861__compiled_geometry_utility')
op.drop_table('out_eia861__compiled_geometry_balancing_authority')
# ### end Alembic commands ###
83 changes: 83 additions & 0 deletions migrations/versions/232c79b90224_make_service_territories_plural.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
"""Make service territories plural

Revision ID: 232c79b90224
Revises: 027d56be4d4e
Create Date: 2023-09-21 09:02:11.211316

"""
import sqlalchemy as sa
from alembic import op

# revision identifiers, used by Alembic.
revision = '232c79b90224'
down_revision = '027d56be4d4e'
branch_labels = None
depends_on = None


def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('out_eia861__compiled_geometry_balancing_authorities',
sa.Column('county_id_fips', sa.Text(), nullable=False, comment='County ID from the Federal Information Processing Standard Publication 6-4.'),
sa.Column('county_name_census', sa.Text(), nullable=True, comment='County name as specified in Census DP1 Data.'),
sa.Column('population', sa.Float(), nullable=True, comment='County population, sourced from Census DP1 data.'),
sa.Column('area_km2', sa.Float(), nullable=True, comment='County area in km2.'),
sa.Column('report_date', sa.Date(), nullable=False, comment='Date reported.'),
sa.Column('balancing_authority_id_eia', sa.Integer(), nullable=False, comment='EIA balancing authority ID. This is often (but not always!) the same as the utility ID associated with the same legal entity.'),
sa.Column('state', sa.Text(), nullable=True, comment='Two letter US state abbreviation.'),
sa.Column('county', sa.Text(), nullable=False, comment='County name.'),
sa.Column('state_id_fips', sa.Text(), nullable=True, comment='Two digit state FIPS code.'),
sa.PrimaryKeyConstraint('balancing_authority_id_eia', 'report_date', 'county_id_fips', 'county', name=op.f('pk_out_eia861__compiled_geometry_balancing_authorities'))
)
op.create_table('out_eia861__compiled_geometry_utilities',
sa.Column('county_id_fips', sa.Text(), nullable=False, comment='County ID from the Federal Information Processing Standard Publication 6-4.'),
sa.Column('county_name_census', sa.Text(), nullable=True, comment='County name as specified in Census DP1 Data.'),
sa.Column('population', sa.Float(), nullable=True, comment='County population, sourced from Census DP1 data.'),
sa.Column('area_km2', sa.Float(), nullable=True, comment='County area in km2.'),
sa.Column('report_date', sa.Date(), nullable=False, comment='Date reported.'),
sa.Column('utility_id_eia', sa.Integer(), nullable=False, comment='The EIA Utility Identification number.'),
sa.Column('state', sa.Text(), nullable=True, comment='Two letter US state abbreviation.'),
sa.Column('county', sa.Text(), nullable=True, comment='County name.'),
sa.Column('state_id_fips', sa.Text(), nullable=True, comment='Two digit state FIPS code.'),
sa.PrimaryKeyConstraint('utility_id_eia', 'report_date', 'county_id_fips', name=op.f('pk_out_eia861__compiled_geometry_utilities'))
)
op.drop_table('out_eia861__compiled_geometry_balancing_authority')
op.drop_table('out_eia861__compiled_geometry_utility')
with op.batch_alter_table('core_eia861__assn_balancing_authority', schema=None) as batch_op:
batch_op.create_foreign_key(batch_op.f('fk_core_eia861__assn_balancing_authority_utility_id_eia_core_eia__entity_utilities'), 'core_eia__entity_utilities', ['utility_id_eia'], ['utility_id_eia'])

# ### end Alembic commands ###


def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table('core_eia861__assn_balancing_authority', schema=None) as batch_op:
batch_op.drop_constraint(batch_op.f('fk_core_eia861__assn_balancing_authority_utility_id_eia_core_eia__entity_utilities'), type_='foreignkey')

op.create_table('out_eia861__compiled_geometry_utility',
sa.Column('county_id_fips', sa.TEXT(), nullable=False),
sa.Column('county_name_census', sa.TEXT(), nullable=True),
sa.Column('population', sa.FLOAT(), nullable=True),
sa.Column('area_km2', sa.FLOAT(), nullable=True),
sa.Column('report_date', sa.DATE(), nullable=False),
sa.Column('utility_id_eia', sa.INTEGER(), nullable=False),
sa.Column('state', sa.TEXT(), nullable=True),
sa.Column('county', sa.TEXT(), nullable=True),
sa.Column('state_id_fips', sa.TEXT(), nullable=True),
sa.PrimaryKeyConstraint('utility_id_eia', 'report_date', 'county_id_fips', name='pk_out_eia861__compiled_geometry_utility')
)
op.create_table('out_eia861__compiled_geometry_balancing_authority',
sa.Column('county_id_fips', sa.TEXT(), nullable=False),
sa.Column('county_name_census', sa.TEXT(), nullable=True),
sa.Column('population', sa.FLOAT(), nullable=True),
sa.Column('area_km2', sa.FLOAT(), nullable=True),
sa.Column('report_date', sa.DATE(), nullable=False),
sa.Column('balancing_authority_id_eia', sa.INTEGER(), nullable=False),
sa.Column('state', sa.TEXT(), nullable=True),
sa.Column('county', sa.TEXT(), nullable=False),
sa.Column('state_id_fips', sa.TEXT(), nullable=True),
sa.PrimaryKeyConstraint('balancing_authority_id_eia', 'report_date', 'county_id_fips', 'county', name='pk_out_eia861__compiled_geometry_balancing_authority')
)
op.drop_table('out_eia861__compiled_geometry_utilities')
op.drop_table('out_eia861__compiled_geometry_balancing_authorities')
# ### end Alembic commands ###
103 changes: 103 additions & 0 deletions migrations/versions/79158fe4f428_rename_ferc714_respondent_tables.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
"""Rename ferc714 respondent tables

Revision ID: 79158fe4f428
Revises: dfbc19b1cb34
Create Date: 2023-09-20 16:27:40.621325

"""
import sqlalchemy as sa
from alembic import op

# revision identifiers, used by Alembic.
revision = '79158fe4f428'
down_revision = 'dfbc19b1cb34'
branch_labels = None
depends_on = None


def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('out_ferc714__fipsified_respondents',
sa.Column('eia_code', sa.Integer(), nullable=True),
sa.Column('respondent_type', sa.Enum('utility', 'balancing_authority'), nullable=True),
sa.Column('respondent_id_ferc714', sa.Integer(), nullable=True),
sa.Column('respondent_name_ferc714', sa.Text(), nullable=True),
sa.Column('report_date', sa.Date(), nullable=True, comment='Date reported.'),
sa.Column('balancing_authority_id_eia', sa.Integer(), nullable=True, comment='EIA balancing authority ID. This is often (but not always!) the same as the utility ID associated with the same legal entity.'),
sa.Column('balancing_authority_code_eia', sa.Text(), nullable=True, comment='EIA short code identifying a balancing authority.'),
sa.Column('balancing_authority_name_eia', sa.Text(), nullable=True, comment='Name of the balancing authority.'),
sa.Column('utility_id_eia', sa.Integer(), nullable=True, comment='The EIA Utility Identification number.'),
sa.Column('utility_name_eia', sa.Text(), nullable=True, comment='The name of the utility.'),
sa.Column('state', sa.Text(), nullable=True, comment='Two letter US state abbreviation.'),
sa.Column('county', sa.Text(), nullable=True, comment='County name.'),
sa.Column('state_id_fips', sa.Text(), nullable=True, comment='Two digit state FIPS code.'),
sa.Column('county_id_fips', sa.Text(), nullable=True, comment='County ID from the Federal Information Processing Standard Publication 6-4.'),
sa.ForeignKeyConstraint(['respondent_id_ferc714'], ['core_ferc714__respondent_id.respondent_id_ferc714'], name=op.f('fk_out_ferc714__fipsified_respondents_respondent_id_ferc714_core_ferc714__respondent_id'))
)
op.create_table('out_ferc714__summarized_demand',
sa.Column('report_date', sa.Date(), nullable=False, comment='Date reported.'),
sa.Column('respondent_id_ferc714', sa.Integer(), nullable=False),
sa.Column('demand_annual_mwh', sa.Float(), nullable=True),
sa.Column('population', sa.Float(), nullable=True, comment='County population, sourced from Census DP1 data.'),
sa.Column('area_km2', sa.Float(), nullable=True, comment='County area in km2.'),
sa.Column('population_density_km2', sa.Float(), nullable=True, comment='Average population per sq. km area of a service territory.'),
sa.Column('demand_annual_per_capita_mwh', sa.Float(), nullable=True, comment='Per-capita annual demand, averaged using Census county-level population estimates.'),
sa.Column('demand_density_mwh_km2', sa.Float(), nullable=True, comment='Annual demand per km2 of a given service territory.'),
sa.Column('eia_code', sa.Integer(), nullable=True),
sa.Column('respondent_type', sa.Enum('utility', 'balancing_authority'), nullable=True),
sa.Column('respondent_name_ferc714', sa.Text(), nullable=True),
sa.Column('balancing_authority_id_eia', sa.Integer(), nullable=True, comment='EIA balancing authority ID. This is often (but not always!) the same as the utility ID associated with the same legal entity.'),
sa.Column('balancing_authority_code_eia', sa.Text(), nullable=True, comment='EIA short code identifying a balancing authority.'),
sa.Column('balancing_authority_name_eia', sa.Text(), nullable=True, comment='Name of the balancing authority.'),
sa.Column('utility_id_eia', sa.Integer(), nullable=True, comment='The EIA Utility Identification number.'),
sa.Column('utility_name_eia', sa.Text(), nullable=True, comment='The name of the utility.'),
sa.ForeignKeyConstraint(['respondent_id_ferc714'], ['core_ferc714__respondent_id.respondent_id_ferc714'], name=op.f('fk_out_ferc714__summarized_demand_respondent_id_ferc714_core_ferc714__respondent_id')),
sa.PrimaryKeyConstraint('respondent_id_ferc714', 'report_date', name=op.f('pk_out_ferc714__summarized_demand'))
)
op.drop_table('fipsified_respondents_ferc714')
op.drop_table('summarized_demand_ferc714')
# ### end Alembic commands ###


def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('summarized_demand_ferc714',
sa.Column('report_date', sa.DATE(), nullable=False),
sa.Column('respondent_id_ferc714', sa.INTEGER(), nullable=False),
sa.Column('demand_annual_mwh', sa.FLOAT(), nullable=True),
sa.Column('population', sa.FLOAT(), nullable=True),
sa.Column('area_km2', sa.FLOAT(), nullable=True),
sa.Column('population_density_km2', sa.FLOAT(), nullable=True),
sa.Column('demand_annual_per_capita_mwh', sa.FLOAT(), nullable=True),
sa.Column('demand_density_mwh_km2', sa.FLOAT(), nullable=True),
sa.Column('eia_code', sa.INTEGER(), nullable=True),
sa.Column('respondent_type', sa.VARCHAR(length=19), nullable=True),
sa.Column('respondent_name_ferc714', sa.TEXT(), nullable=True),
sa.Column('balancing_authority_id_eia', sa.INTEGER(), nullable=True),
sa.Column('balancing_authority_code_eia', sa.TEXT(), nullable=True),
sa.Column('balancing_authority_name_eia', sa.TEXT(), nullable=True),
sa.Column('utility_id_eia', sa.INTEGER(), nullable=True),
sa.Column('utility_name_eia', sa.TEXT(), nullable=True),
sa.ForeignKeyConstraint(['respondent_id_ferc714'], ['core_ferc714__respondent_id.respondent_id_ferc714'], name='fk_summarized_demand_ferc714_respondent_id_ferc714_core_ferc714__respondent_id'),
sa.PrimaryKeyConstraint('respondent_id_ferc714', 'report_date', name='pk_summarized_demand_ferc714')
)
op.create_table('fipsified_respondents_ferc714',
sa.Column('eia_code', sa.INTEGER(), nullable=True),
sa.Column('respondent_type', sa.VARCHAR(length=19), nullable=True),
sa.Column('respondent_id_ferc714', sa.INTEGER(), nullable=True),
sa.Column('respondent_name_ferc714', sa.TEXT(), nullable=True),
sa.Column('report_date', sa.DATE(), nullable=True),
sa.Column('balancing_authority_id_eia', sa.INTEGER(), nullable=True),
sa.Column('balancing_authority_code_eia', sa.TEXT(), nullable=True),
sa.Column('balancing_authority_name_eia', sa.TEXT(), nullable=True),
sa.Column('utility_id_eia', sa.INTEGER(), nullable=True),
sa.Column('utility_name_eia', sa.TEXT(), nullable=True),
sa.Column('state', sa.TEXT(), nullable=True),
sa.Column('county', sa.TEXT(), nullable=True),
sa.Column('state_id_fips', sa.TEXT(), nullable=True),
sa.Column('county_id_fips', sa.TEXT(), nullable=True),
sa.ForeignKeyConstraint(['respondent_id_ferc714'], ['core_ferc714__respondent_id.respondent_id_ferc714'], name='fk_fipsified_respondents_ferc714_respondent_id_ferc714_core_ferc714__respondent_id')
)
op.drop_table('out_ferc714__summarized_demand')
op.drop_table('out_ferc714__fipsified_respondents')
# ### end Alembic commands ###
Loading
Loading