Skip to content

Commit

Permalink
Autoscaling: Refactor code to top-level code environment
Browse files Browse the repository at this point in the history
For making `autoscale.py` a real program, and good Python citizen, it is
important to consider your file might be considered as a module, and
_imported_. In this case, you don't want to have your main code executed.

https://docs.python.org/3/library/__main__.html

In this case, the program would have caused any sort of test runner,
which scans directory trees for functions to executed, by, well,
importing all modules, to eventually run into an eternal loop.

When putting code into the top-level code environment, as exercised on
behalf of the `__name__ == '__main__'` code guard, it will safely
execute as a Python program, but not when used as a Python module.

In this spirit, a Python file can be both a program and a module at the
same time.
  • Loading branch information
amotl committed Mar 6, 2024
1 parent 5e1070d commit 4c9056e
Showing 1 changed file with 19 additions and 13 deletions.
32 changes: 19 additions & 13 deletions topic/autoscaling/autoscale.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,16 +131,22 @@ def scale_cluster(num, cluster_status, max_num_shard):
DELAY_SECONDS = 5
MAX_NUM_SHARDS = 30

# Main loop to monitor and adjust cluster size based on shard count
while True:
try:
status = get_cluster_status() # Fetch current cluster status
number_shards = num_shards() # Calculate average shard count
if number_shards is not None:
logging.info(f"Current avg number of shards: {number_shards}")
scale_cluster(number_shards, status, MAX_NUM_SHARDS)
else:
logging.error("Failed to retrieve shard metrics.")
except Exception as e:
logging.error(f"An error occurred: {e}")
time.sleep(DELAY_SECONDS)

def main():
# Main loop to monitor and adjust cluster size based on shard count
while True:
try:
status = get_cluster_status() # Fetch current cluster status
number_shards = num_shards() # Calculate average shard count
if number_shards is not None:
logging.info(f"Current avg number of shards: {number_shards}")
scale_cluster(number_shards, status, MAX_NUM_SHARDS)
else:
logging.error("Failed to retrieve shard metrics.")
except Exception as e:
logging.error(f"An error occurred: {e}")
time.sleep(DELAY_SECONDS)


if __name__ == "__main__":
main()

0 comments on commit 4c9056e

Please sign in to comment.