Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove twitter-blocked logic #333

Merged
merged 2 commits into from
Oct 3, 2023
Merged

Remove twitter-blocked logic #333

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bf291a2
Remove twitter-blocked logic
Phaeton Oct 2, 2023
2f55f27
remove spaces before ?
Phaeton Oct 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion plane-alert-db.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13296,4 +13296,15 @@ AD18D5,N943FL,Ron Desantis,Cessna Citation Latitude,C68A,Civ,Bizjet,Fly Navy,Flo
AB772B,N838MF,Ronald Perelman,Gulfstream G650,GLF6,Civ,Bizjet,Revlon,Political Donor,Don't you know who I am?,https://en.wikipedia.org/wiki/Ronald_Perelman
A97659,N709DS,Steve Ballmer,Gulfstream G650,GLF6,Civ,Bizjet,Microsoft,LA Clippers,Don't you know who I am?,https://en.wikipedia.org/wiki/Steve_Ballmer
AC701E,N900KS,Steven Spielberg,Gulfstream G650,GLF6,Civ,Bizjet,Jurassic Park,Indiana Jones,Don't you know who I am?,https://en.wikipedia.org/wiki/Steven_Spielberg
AD5B90,N96UA,Under Armour Corporation,Gulfstream G-V,GLF5,Civ,Bizjet,Undies,As Seen on TV,Vanity Plate,https://en.wikipedia.org/wiki/Under_Armour
AD5B90,N96UA,Under Armour Corporation,Gulfstream G-V,GLF5,Civ,Bizjet,Undies,As Seen on TV,Vanity Plate,https://en.wikipedia.org/wiki/Under_Armour
A2AE0A,N272BG,Falcon Landing LLC,Gulfstream G550,GLF5,Civ,Elon Musk,SpaceX,Toys4Billionaires,Don't you know who I am?,https://youtu.be/bvim4rsNHkQ
Phaeton marked this conversation as resolved.
Show resolved Hide resolved
A64304,N502SX,Falcon Landing LLC,Gulfstream G550,GLF5,Civ,Elon Musk,SpaceX,Toys4Billionaires,Don't you know who I am?,https://youtu.be/bvim4rsNHkQ
A835AF,N629TS,Falcon Landing LLC,Gulfstream G650,GLF6,Civ,Elon Musk,SpaceX,Toys4Billionaires,Don't you know who I am?,https://youtu.be/bvim4rsNHkQ
A34877,N310SX,Falcon Landing LLC,Not Yet Allocated,ZZZZ,Civ,Elon Musk,SpaceX,Toys4Billionaires,Don't you know who I am?,https://youtu.be/bvim4rsNHkQ
AB1270,N812SX,Falcon Landing LLC,Not Yet Allocated,ZZZZ,Civ,Elon Musk,SpaceX,Toys4Billionaires,Don't you know who I am?,https://youtu.be/bvim4rsNHkQ
AB817F,N840SX,Falcon Landing LLC,Not Yet Allocated,ZZZZ,Civ,Elon Musk,SpaceX,Toys4Billionaires,Don't you know who I am?,https://youtu.be/bvim4rsNHkQ
ACDD88,N928SX,Falcon Landing LLC,Not Yet Allocated,ZZZZ,Civ,Elon Musk,SpaceX,Toys4Billionaires,Don't you know who I am?,https://youtu.be/bvim4rsNHkQ
A0FE3B,N16DJ,Elon Musk,Gulfstream G-IV,GLF4,Civ,PIA,Hive Mind,You Can Run But You Cannot Hide,PIA,https://en.wikipedia.org/wiki/Elon_Musk
A0FEBB,N628TS,Elon Musk,Gulfstream G650,GLF6,Civ,PIA,Hive Mind,You Can Run But You Cannot Hide,PIA,https://en.wikipedia.org/wiki/Elon_Musk
AA3410,N757AF,DJT Operations,Boeing 757,B752,Civ,Original Nuttah,Orange Danger,Lock Him Up,Dictator Alert,https://en.wikipedia.org/wiki/Veracity_of_statements_by_Donald_Trump
A9B67C,N725DT,Donald Trump,Cessna Citation X,C750,Civ,Bad Human,Orange Danger,Lock Him Up,Dictator Alert,https://en.wikipedia.org/wiki/Veracity_of_statements_by_Donald_Trump
Phaeton marked this conversation as resolved.
Show resolved Hide resolved
12 changes: 0 additions & 12 deletions plane-alert-twitter-blocked-images.csv
View file
Open in desktop

This file was deleted.

12 changes: 0 additions & 12 deletions plane-alert-twitter-blocked.csv
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion scripts/check_invalid_derivatives.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
MAIN_DB_FILES = [
"plane-alert-db.csv",
"plane-alert-pia.csv",
"plane-alert-twitter-blocked.csv",
"plane-alert-ukraine.csv",
"plane_images.csv",
]
Expand Down
25 changes: 0 additions & 25 deletions scripts/check_main_databases.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
)

MAIN_DATABASE_NAME = "plane-alert-db.csv"
TWITTER_BLOCKED_DATABASE_NAME = "plane-alert-twitter-blocked.csv"
UKRAINE_DATABASE_NAME = "plane-alert-ukraine.csv"
PLANE_IMAGES_DATABASE_NAME = "plane_images.csv"

Expand Down Expand Up @@ -170,28 +169,6 @@ def contains_valid_ICAO_hexes(df):
# contains_bad_links(main_df)
logging.info("The main database is valid.")

##########################################
# Check 'plane-alert-twitter-blocked' db.#
##########################################
logging.info("Checking the 'plane-alert-twitter-blocked' database...")
try:
twitter_blocked_df = pd.read_csv(TWITTER_BLOCKED_DATABASE_NAME)
main_df.name = TWITTER_BLOCKED_DATABASE_NAME
except Exception as e:
logging.error(
f"The '{TWITTER_BLOCKED_DATABASE_NAME}' database is not a valid CSV."
)
sys.stdout.write(
f"The '{TWITTER_BLOCKED_DATABASE_NAME}' database is not a valid CSV: {e}\n"
)
sys.exit(1)

# Preform database checks.
contains_duplicate_ICAOs(twitter_blocked_df)
contains_valid_ICAO_hexes(twitter_blocked_df)
contains_duplicate_regs(twitter_blocked_df)
contains_bad_links(twitter_blocked_df)
logging.info("The 'plane-alert-twitter-blocked' database is valid.")

##########################################
# Check 'plane-alert-ukraine' db. #
Expand All @@ -209,7 +186,6 @@ def contains_valid_ICAO_hexes(df):

# Preform database checks.
contains_duplicate_ICAOs(ukraine_df)
contains_valid_ICAO_hexes(twitter_blocked_df)
contains_duplicate_regs(ukraine_df)
contains_bad_links(ukraine_df)
logging.info("The 'plane-alert-ukraine' database is valid.")
Expand All @@ -232,7 +208,6 @@ def contains_valid_ICAO_hexes(df):

# Perform database checks.
contains_duplicate_ICAOs(images_df)
contains_valid_ICAO_hexes(twitter_blocked_df)
bad_links = pd.DataFrame()
for col in images_df.columns: # Check all link columns for bad links.
if col != "$ICAO":
Expand Down
19 changes: 2 additions & 17 deletions scripts/create_db_derivatives.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
"""This script creates (derivative) category and images CSV database files from the main
'plane-alert-db.csv' database file. The categories are created based on the 'CMPG'
column, while images are added using the 'plane_images.csv' reference file. It also
creates the 'plane-alert-twitter-blocked-images.csv' and
'plane-alert-ukraine-images.csv' database files.
creates the 'plane-alert-ukraine-images.csv' database file.
"""

import logging
Expand Down Expand Up @@ -50,20 +49,6 @@
)
logging.info("Category and category images CSV files created successfully.")

logging.info("Creating the twitter blocked database images CSV file...")
twitter_blocked_df = pd.read_csv("plane-alert-twitter-blocked.csv")
twitter_blocked_df_images = pd.merge(
twitter_blocked_df, images_df, how="left", on="$ICAO"
)
twitter_blocked_df_images.to_csv(
"plane-alert-twitter-blocked-images.csv",
index=False,
mode="wb",
encoding="utf8",
lineterminator="\n",
)
logging.info("Twitter blocked database images CSV file created successfully.")

logging.info("Creating the ukraine database images CSV file...")
ukraine_df = pd.read_csv("plane-alert-ukraine.csv")
ukraine_df_images = pd.merge(ukraine_df, images_df, how="left", on="$ICAO")
Expand Down Expand Up @@ -92,7 +77,7 @@
"Check for new ICAOs in DB files and add them to the images reference file..."
)
plane_alert_df = (
pd.concat([df["$ICAO"], twitter_blocked_df["$ICAO"], ukraine_df["$ICAO"]])
pd.concat([df["$ICAO"], ukraine_df["$ICAO"]])
.drop_duplicates()
.reset_index(drop=True)
)
Expand Down