Project Meeting 2023.06.22

Agenda

• New meeting invites and recordings
• Roadmap Update - revisit desired work products (WSP)
• Sharrow-related performance update

Meeting Notes

Admin

• We know that some people did not receive the new meetings notice. We also learned that people outside of the CS organization was not able to access the chat during the meeting. CS will work with their IT folks to understand and correct. This may be a case of - it'll get worse before it gets better - and hope folks can be patient as we figure out the best path forward.

Roadmap Update

Presentation: activitysim-roadmap-wip-deliverables-22-june-2023.pptx

• There were two options for delivering the roadmap:
  1. Revisit the outline and develop a final report
  2. A set of working documents that can be updated over time. There are options for how those documents would be maintained, including: google docs, wiki, quarto (markdown in Github repo). There are no huge pros or cons for any of these options, mostly personal preference. If there can be an AMPO-sponsored Google Drive, Google Docs would fit nicely.
• The path to building consensus could be two different ways:
  1. WSP develops a draft, it's reviewed, comments consolidated, a new draft is produced, review/comments/update, and then a final report is developed.
  2. WSP develops the working documents with a list of potential tasks, with high level levels of effort but limited details on approach, and partners would rank and prioritize. There is a challenge in consultants developing this long-term list in that there is a short-term competition on future task orders, if current structure is retained. Right now there are not clearly defined steps to reach ActivitySim 2.0 but a schedule that hints at its delivery.
• Recommendation to make a clearer trajectory of how to get to ActivitySim 2.0.
  • WSP noted that virtually everyone they talked to during interviews identified non-behavioral enhancements (documentation, usability, performance) as priorities.
  • Discussion on how far incrementalism would get the group to ActivitySim 2.0. At some point, rethinking the framework / broader refactoring would be needed, and the group needs to make a decision. An ActivitySim 2.0 platform that supports ActivitySimple, the State of the Practice, and ActivitySim 2.0 is a significant effort that wouldn't fall under roadmap activities. Finalizing management and governance are key components of the roadmap that would help move things forward.
• Consider the possibility of one of the roadmap deliverables as marketing material for potential partners.
• Recommendation to add a concise list of limitations of the current structure, somewhere in the deliverables. It was noted that we wouldn't want this in any marketing material.
• Recommendation to add workforce development aspects to the roadmap.

Sharrow-related performance update

• SANDAG and PSRC noticed some unexpected performance issues while running with sharrow (runtimes taking longer than the legacy version, for example).
• Jeff reviewed the SANDAG model and it now runs with Sharrow with the speed improvements expected. (There are are still some outstanding issues with string comparisons, in the vehicle model for example, and SANDAG/RSG can work on those if they want better performance for the component.)
• ACTION ITEM: Jeff will work on developing functionality to turn sharrow on/off for each component. That enhancement is on CS’s to do list. It is currently available for a handful of components, but should be available for all.
• The basic issue is that sharrow has a number of optimizations in it – including parallelization of computations. Sharrow multithreading and ActivitySim multiprocessing were layered on top of each other, such that if there were 40 processors, you’d have 1600 threads running computations. All are competing for resources - each asking the CPU for their turn was slowing things down.
• The solution was to disable multithreading. Parallelism is only controlled by multiprocessing. We should be turning off multithreading for mkl, numba, etc. All of these things have to be turned off before any of the libraries are loaded. Disabling has to be called in either python script or command environment, before loading ActivitySim. It can’t be built into the settings files, you need to do this yourself BEFORE loading the settings.
  • Automating enable/disable of multithreading is complicated. Jeff can make multithreading disabled as a default before anything gets loaded. This would assume that the naïve user would always want to turn off multithreading. This will make it so that anyone doing model development will experience slow speeds unless you intervene and turn it on.
  • Jeff will do some research to include a flag to turn multithreading on, but suspects it would require refactoring a lot of the code to automate.
• Best performance would be turning off multithreading and using multiprocessing. Anything you are running that isn’t compatible with sharrow (for example pre-processing), those things will run serial single thread/single process and it will be very slow. Most stable option is to have everything run through numba but huge amount of effort in refactoring to get there. This isn’t clearly laid out in the roadmap, because there could be a better method. ActivitySim 2.0 could restructure things where we could be doing something completely different. Incremental development could get us to eventually refactor everything.