Network Building with NetworkWrangler
Home > TravelModel > UsersGuide > NetworkCoding > Network-Building-with-NetworkWrangler
This is a step-by-step guide for Travel Model 1.5 users who want to build or modify the roadway and transit networks using the tool NetworkWrangler.
The basic premise is that we start with a base network (representing 2015), layer network projects on top of that, and output networks representing future years.
CONTENTS
How a transportation project is reflected in the Travel Model One network is specified by the following two sets of modeling files:
- project definitions
Project definition represents how a project will change the network, e.g. modifying the transit services or freeway lanes in a certain year (referred to as the “opening year”). Information of project definition generally comes from the Fact Sheet of each project. The Fact Sheet is usually created by MTC planning staff in collaboration with project sponsors. Project definitions are stored in local Git repositories in the Box folder "TM1_NetworkProjects".
In the TM1 network project Box folder, each subfolder represents one project, coded as a local Git repository (that is not on GitHub). A local Git repository saves the commit history locally, in the .git folder, instead of remotely on GitHub. Occasionally, even with the Box Driver installed locally, Box has challenges syncing the complete Git history down to the local drive, causing the local files to miss the latest project coding changes committed by other modelers. To resolve this, MTC staff recommend installing Visual Studio Code and open the entire TM1 network project Box folder in VS Code, which would force Box to sync down all the commits.
- network specification
Network specification (e.g. net_spec_blueprint.py) represents which projects will be added to the network in which year, and whether it will affect the roadway and/or transit network. This is usually informed by policy discussions around which projects will get funding and when will get the funding.
For projects with both roadway and transit components, include them in both the hwy and trn parts of the network specification, e.g. the project “FBP_NP_036_SR29_Imola_PNR”.
Some projects have multiple phases, with different components with different opening years, then they may need to be added to multiple years in the network specification, each year accounting for the opening component of that year, e.g. the “Bridge_Toll_Updates_rtp2025” project applies in years 2015, 2020, 2023, 2025, 2030.
While a project can open in any year, Travel Model One network is built at 5-year intervals (except for 2023), therefore, a project that opens in the between years should be in the end year of the 5-year interval in network specification. E.g., the BART Irvington Station project opens in year 2043, then the project should have an opening year of 2043 in the project definition, and be in year 2045 in the network specification.
Code/Update a Network Project
When a project is coded into the network, both sets of modeling files should be consistent with the Fact Sheet.
- When coding a project for the first time, a new local Git repository needs to be created, and the project needs to be added to the network specification in the correct year and type (hwy – roadway projects or trn – transit projects).
- There are cases when a project was already coded in a previous planning/modeling effort, but the project definitions have changed, necessitating an update to the existing project coding. The update may involve the project definition, or the network specification, or both.
Additional information on Project Name
Projects typically have a different name in Travel Model One from the name in the Plan (or RTP ID) and Project Performance Assessment (or PPA ID). Modeled project name is used in network coding, including in project definition and network specification. In the PBA50+ TM Project Coding Asana project, the “NetworkProjects Folder Name” tag shows the project name in the model. For example, the San Pablo BRT project has PBA50_RTPID "21-T10-077", PBA50_PPA ID "2100", and NetworkProjects Folder Name (modeled name) "MAJ_SanPablo_BRT".
To build model networks, the following software needs to be installed on your computer. These instructions are written assuming installation on Windows (tested on Windows 10).
- Install conda to manage your NetworkWrangler python environment.
-
Install Citilabs Cube 6.4.4 or newer. After installing when you run the application, you'll be asked where the license server is located -- there is a note in Lisa's office with this information. Cube is typically installed in
C:\Program Files (x86)\Citilabs, andRUNTPP.EXEis typically found inC:\Program Files (x86)\Citilabs\CubeVoyager. - Install Github Desktop. You can use this for the next step, cloning NetworkWrangler from Github, and also for committing changes to the NetworkWrangler network configuration back to github.
-
Clone NetworkWrangler from Github and keep track of where you cloned it. We typically clone it into our personal
Documentsfolder orDocuments\GitHub. For example, Flavia's installation is inC:\Users\ftsang\Documents\GitHub\NetworkWrangler. -
Install Git. This is related to, but not the same as the GitHub Desktop application; we'll need it because projects are coded as local git repositories (that are not on Github). This is typically installed in
C:\Program Files\Git. -
Verify that the M drive is mounted and available. 2015 base networks are available here: TM1_2015_Base_Network which is cloned to
M:\Application\Model One\Networks\TM1_2015_Base_Network
This is so that Windows will know where this executable is when NetworkWrangler tries to run it in order to build networks. Again, you’ll need to know where Cube's RUNTPP.EXE is installed; it's most likely installed in C:\Program Files (x86)\Citilabs\CubeVoyager. Add this to your PATH variable using the Windows GUI, since this will make it persistent in your environment across machine restarts, etc. Conda environments will inherit these environment variables as well.
Open an Anaconda Powershell Prompt and create an environment for NetworkWrangler as follows:
(base) PS C:\> conda create --name NetworkWrangler python=3.10
(base) PS C:\> conda activate NetworkWrangler
(NetworkWrangler) PS C:\>This will automatically put the environment's python into your PATH when you activate the environment.
In your Anaconda Powershell Prompt, activate your NetworkWrangler environment and then install the required packages through pip. The required modules are:
In some cases, additional packages are required. For example, some network projects require geopandas and its dependencies. To use the --create_project_diffs or --create_project_diff feature, geopandas is required.
Note: Output has not been copied into the below example because it's long, but the pip install commands will result in output.
(NetworkWrangler) C:\> pip install xlrd SimpleParse numpy pywin32 pandas
(NetworkWrangler) C:\> pip install geopandas partridge
(NetworkWrangler) C:\> rem Navigate to where your NetworkWrangler GitHub repo is cloned
(NetworkWrangler) C:\> E:
(NetworkWrangler) E:\> cd GitHub\NetworkWrangler
(NetworkWrangler) E:\GitHub\NetworkWrangler>pip install -e .In your Anaconda Powershell Prompt window, when you run the command of python, you'll enter into the python interpreter. This means that subsequent commands are interpretted as lines of Python rather than as Windows DOS commands. When you enter into the python interpreter, the command window should display a message about the version of your python and the prompt will change to be >>> instead of [your current working directory]>.
To exit the python interpreter, type the python command, quit().
Note that python, unlike Windows DOS, is case-sensitive.
(NetworkWrangler) PS E:\GitHub\NetworkWrangler> rem Note we have activated the NetworkWrangler conda environment
(NetworkWrangler) PS E:\GitHub\NetworkWrangler> python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:29:51) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> print("This is a python print command")
This is a python print command
>>> print("Type quit() to exit the python interpreter")
Type quit() to exit the python interpreter
>>> quit()
(NetworkWrangler) PS E:\GitHub\NetworkWrangler>While in the python interpreter, try to import NetworkWrangler. A successful import will look like the following.
(NetworkWrangler) PS C:\>python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:29:51) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import Wrangler
Importing E:\GitHub\NetworkWrangler\_static\dataTable.py
Importing e:\github\networkwrangler\Wrangler\TransitAssignmentData.py
>>>The most likely type of error you'll get here is an import error if a module required to NetworkWrangler fails to load. For example, if the required module, xlrd, is not installed, it'll look like this:
C:\>python
Python 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import Wrangler
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\lzorn\Documents\NetworkWrangler\Wrangler\__init__.py", line 9, in <module>
from .TransitAssignmentData import TransitAssignmentData ##
File "C:\Users\lzorn\Documents\NetworkWrangler\Wrangler\TransitAssignmentData.py", line 5, in <module>
import csv,os,logging,string,sys,xlrd
ImportError: No module named xlrd
>>>To resolve, quit the python interpreter and install the missing python module using pip and the instructions in Step 3.
Now that we can import NetworkWrangler, it's time to build a network!
There are two network building scripts and they each have their own configuration. The following documentation assumes you'll use your local NetworkWrangler\scripts as your working directory, so navigate there in the Anaconda Powershell Prompt:
(NetworkWrangler) PS C:\>
(NetworkWrangler) PS C:\>E:
(NetworkWrangler) PS C:\>cd GitHub\NetworkWrangler\scripts
(NetworkWrangler) PS E:\GitHub\NetworkWrangler>There are a number of network build scripts. Note that each of these take different arguments; run the script with --help to see the script usage documentation.
-
build_network_mtc_futures.py- The futures network script and configuration exists to build the networks for each of the three Horizon futures. -
build_network_mtc_blueprint.py- This script builds the PBA50 Blueprint networks, including the Baseline, Blueprint, EIR Alt1 and EIR Alt2 scenarios. It's also used to create other networks that are based on the Blueprint scenario, such as the Next Gen Freeway No Project network. The network specification used with this script isnet_spec_blueprint.py. -
build_network_mtc.pyis the generic version of building a network, so it's used for other network specs, such asnet_spec_NGF.py, etc. -
build_network_mtc_add_project.pyis the script to apply 1 or more (but a small number) of projects to a network. For example, this is used to build Project Performance Assessment networks, adding networks on top of the baseline.
For build_network_mtc_futures.py and build_network_mtc_blueprint.py, two environment variables are required: TM1_2015_Base_Network and TM1_NetworkProjects.
:: Set this to the location where the 2015 base network is cloned from https://github.com/BayAreaMetro/TM1_2015_Base_Network
(NetworkWrangler) PS E:\GitHub\NetworkWrangler\scripts> conda env config vars set "TM1_2015_Base_Network=M:\Application\Model One\Networks\TM1_2015_Base_Network"
To make your changes take effect please reactivate your environment
:: Set this to your BOX path where the TM1_NetworkProjects are located (https://mtcdrive.box.com/s/cs0dmr987kaasmi83a6irru6ts6g4y1x)
(NetworkWrangler) PS E:\GitHub\NetworkWrangler\scripts> conda env config vars set "TM1_NetworkProjects=E:\Box\Modeling and Surveys\TM1_NetworkProjects"
To make your changes take effect please reactivate your environment
(NetworkWrangler) PS E:\GitHub\NetworkWrangler\scripts> conda activate NetworkWrangler
To build the Blueprint network series, run:
(NetworkWrangler) PS E:\GitHub\NetworkWrangler\scripts>python build_network_mtc_blueprint.py .\net_spec_blueprint.py Blueprint
Importing E:\GitHub\NetworkWrangler\_static\dataTable.py
Importing e:\github\networkwrangler\Wrangler\TransitAssignmentData.py
WranglerLogger: DEBUG Args: Namespace(configword=None, model_type='TravelModelOne', base_network=None, continue_on_warning=True, skip_precheck_requirements=True, restart_year=None, restart_mode=None, create_project_diffs=True, net_spec='.\\net_spec_blueprint.py', netvariant='Blueprint')
WranglerLogger: INFO Removing 2025 hwy {'name': 'EIR2_ReXBlue', 'variants_include': ['Alt2']}
WranglerLogger: INFO Removing 2025 hwy {'name': 'EIR2_Val_Link_ExpressBus', 'variants_include': ['Alt2']}
[lots of output omitted]
WranglerLogger: INFO
WranglerLogger: DEBUG Successfully completed running E:\GitHub\NetworkWrangler\scripts\build_network_mtc_blueprint.py
(NetworkWrangler) PS E:\GitHub\NetworkWrangler\scripts>dir .\BlueprintNetworks\
Directory: E:\GitHub\NetworkWrangler\scripts\BlueprintNetworks
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/1/2023 10:02 AM net_2015_Blueprint
d----- 6/1/2023 10:14 AM net_2020_Blueprint
d----- 6/1/2023 10:30 AM net_2025_Blueprint
d----- 6/1/2023 10:40 AM net_2030_Blueprint
d----- 6/1/2023 10:46 AM net_2035_Blueprint
d----- 6/1/2023 10:56 AM net_2040_Blueprint
d----- 6/1/2023 11:00 AM net_2045_Blueprint
d----- 6/1/2023 11:05 AM net_2050_Blueprint
-a---- 6/1/2023 11:05 AM 4006311 buildnetwork_Blueprint_Blueprint_2023Jun01.100121.debug.LOG
-a---- 6/1/2023 11:05 AM 115997 buildnetwork_Blueprint_Blueprint_2023Jun01.100121.info.LOGThis creates the 2015 network and applies the 2015 projects, and outputs that network; it then applies the 2020 projects and outputs that network, etc. The script does not create an intermediate year network unless a new project has been applied.
Each time a build network python script is run, a scratch folder is created to house copies of the repositories for each network project included in the accompanying netspec file. If you are running several iterations of the build network script, you should regularly delete the "scratch" folder to save space on your local drive and avoid potential errors.
The test network script and configuration are very similar, but they exist as a convenience to test the project coding that you're working on. As the network projects get coded and the Futures network configuration becomes more filled out, building a full network series for each future will become slower. To make project coding easier, the test network script and configuration exist to build a bare-bones network so you can add only the project you're working on (and it's required projects, if any) for testing.
The generic network building script is build_network_mtc.py and its configuration is net_spec_test.py.
Similar to above, in order to use the script, you will need to set the PIVOT_DIR location to point to the location of the 2015 base network inputs.
(NetworkWrangler) PS E:\GitHub\NetworkWrangler\scripts> python build_network_mtc.py test_network net_spec_test.py
Importing E:\GitHub\NetworkWrangler\_static\dataTable.py
Importing e:\github\networkwrangler\Wrangler\TransitAssignmentData.py
WranglerLogger: DEBUG Using tier network M:\\Application\\Model One\\Networks\\TM1_2015_Base_Network\hwy\freeflow.net
WranglerLogger: INFO Reading M:\\Application\\Model One\\Networks\\TM1_2015_Base_Network\trn\transit_lines\Transit_Lines.block
[lots of output omitted]
WranglerLogger: INFO Building 2050 trn networks
WranglerLogger: INFO No applied projects for this year -- skipping output
WranglerLogger: DEBUG Successfully completed running E:\GitHub\NetworkWrangler\scripts\build_network_mtc.py
(NetworkWrangler) PS E:\GitHub\NetworkWrangler\scripts> dir .\Test_Scenario_network_2020\
Directory: E:\GitHub\NetworkWrangler\scripts\Test_Scenario_network_2020
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 6/29/2023 11:05 AM hwy
d----- 6/29/2023 11:05 AM trn
Network projects are coded as folders within M:\Application\Model One\NetworkProjects (configured in build_network_mtc.py and build_network_mtc_blueprint.py. Each project or folder is a local Git repository (that will not be attached to a remote repository -- so it will be standalone).
To code a project, you'll likely want to start with a similar seed project (more on that below) and create a new folder, name it appropriately and copy the contents of the seed project into your folder to start. Make sure you do not copy the .git folder, as that is the git version history of the seed project, and your project will have its own git version history.
The basic files required in a project are:
-
README.txt- While not technically required, it's good practice to keep basic documentation and notes here. Since we plan to keep much of our notes/conversations on Asana, assuming we continue that practice, a link the relevant Asana task is sufficient. -
__init__.py- This makes the project serve as a python module, so that NetworkWrangler can import it.- The
desc()method should return a short string description of the project - The
year()method should return the project's opening year - For transit projects, the
applyEdits()method will be meaningful; more on that later.
- The
-
apply.s- This Cube script is run for roadway projects. More on that below. - data files - These names can vary depending on what the project does to the network, although it'd be nice for them to have intuitive names, like
mod_nodes.csv,new_links.csv, etc.
Initially, just make the minimal changes to README.txt and __init__.py and get things ready to test. Once you're able to build a test network with this new project, you can fill out the details.
- In Windows Explorer, go to your project folder and right click in the blank space of this folder
- Choose Git GUI Here
- Choose Create New Repository
- Select the current folder and click Create
- On the top left quadrant, click on each file to move the files from unstaged to staged. Alternatively, click on the button that says Stage Changed to staget all of them at once.
- Supply a commit message. For the first commit, it doesn't need to be that meaningful, so I typically start with initial commit.
- Click Commit
Now your project is ready to be added to a network.
Modify your local copy of net_spec_test.py in a text editor to add your project. You can add it to any year by adding the string to the hwy and\or trn list for that year (depending on what kind of project it is).
For example, if your project is called ALA050014_SR84_Livermore, the test spec specifies:
NETWORK_PROJECTS = collections.OrderedDict([
(2015, {'hwy':['PROJ_attributes'], 'trn':[]}), # adds PROJ attributes to NODE and LINK
(2020, {
'hwy':['ALA050014_SR84_Livermore'],
'trn':[]
}),
...Now, build a test network as described above.
Roadway projects typically consist of one (or more) of the following actions:
| Action | Example Project |
|---|---|
| Create new nodes | FBP_AL_021_South_Bay_Connect, SF_Park_Presidio_HOV_Pilot, Transform_SR37_Widening_Interim |
| Modify attributes for nodes | To be created. |
| Delete nodes | None yet and this may not ever be necessary |
| Create new links | FBP_AL_055_DubBlvd_NCanyons_Ext, FBP_SC_074_US101_BuenaVista_Int, SF_991030_101_Doyle_Repl |
| Widen Roadway Segment | ALA050014_SR84_Widening, FBP_SM_022_I380_Widening, CC_070011_Brentwood_Blvd_Widening |
| Roadway diet | ALA150043_Claremont_road_diet, SF_130011_2ndSt_Road_Diet, FBP_SL_022_ConomaBlvd_Diet |
| Delete links | SeaLevelRise_1foot, SF_070027_Yerba_Buena_Ramp_Imp, SCL130001_237_101_MAT_Int_Mod |
| Roadway BRT treatment | FBP_AL_065_Bancroft_Bus_Only, CC_170061_Bus_On_Shoulder_680BRT, ALA210028_I80_HOV_Bus_Lane_Ext |
| HOV ramp coding | AL_044_I880_Whipple_Imps, ALA210027_I80_Powell_Transit_Access |
| FT==8 and TOS==2 | MAJ_Bay_Area_Forward_all |
| SIGCOR==1 and TOS==1 | FBP_SC_054_SR17_Corridor_Relief |
| Add new freeway on/off ramps | FBP_CC_038_SR242_Clayton_OnOffRamps, FBP_SM_035_Peninsula_101_OnOffRamps |
When adding a node, it's important to do the following:
- within apply.s, check whether the node number is already in use and error if so; otherwise, the project could be moving an existing node and creating an error.
- add the new node number to the ledger of all nodes in Node Description.xls (This file is managed in the TM1 2015 Base Network repo https://github.com/BayAreaMetro/TM1_2015_Base_Network/blob/master/Node%20Description.xls)
Apply.s example:
RUN PGM=NETWORK
NETI[1] =FREEFLOW.BLD
NETO =FREEFLOW.BLDOUT, exclude=_PROJECT
NODEI[2]="new_nodes.csv" VAR=N,X,Y,_PROJECT(c) START=(substr(record,1,4)='N,X,')
_NODES_ADDED = 0
PHASE=NODEMERGE
IF (ni.2._PROJECT=='my_project')
X = ni.2.X
Y = ni.2.Y
; Is the node in freeflow.bld being moved?
IF (NI.1.X <> 0)
PRINT LIST="BAD node number: ",ni.2.N
ABORT MSG="Node collision: node number already in use"
ENDIF
_NODES_ADDED = _NODES_ADDED + 1
ENDIF
ENDPHASE
ENDRUN
*copy /y FREEFLOW.BLDOUT FREEFLOW.BLD
Transit projects are more complicated. Some example actions:
| Action | Example Project |
|---|---|
| Adding a new line (including transfer links and funnel links) | CC_050025_EBart_to_Antioch |
| Removing a line | Earthquake |
| Rerouting a bus line | Earthquake |
| Extending an existing line | SOL030002_FairfieldVacaville_Stn |
| Splitting a project into phases (frequency changes) | RRSP_Alameda_Point_Transit_Improvements |
| Splitting a project into phases (via multiple .lin files) | FBP_SM_020_Regional_Express_Buses |
| Bus headway adjustment | FBP_MU_059_ACTransbay_Freq_Incr |
| Rail headway adjustment using model year | FBP_MU_046_ACE_Freq_Inc |
| Light station removal and headway change | FBP_SC_106_VTA_LRT_Modernization |
| Deleting bus route | FBP_SL_026_SolExpressBus |
Similar to Code Contribution Best Practices > Source Control, this bullet also applies:
- Communicate effects of commit in the commit comment: update scripts or code cleanup are not useful commit messages because if someone is trying to find a revision before a particular change was made, they would need to read the entire commit/diff to understand the effect of the change. The commit comment should be more specific -- what's the effect of the update or cleanup? Does it change results? And ideally, the diff should be helpful too -- so when making a bunch of functional changes, I think it can be helpful to do one functional change, and commit it once it's working, and then the next in another commit.
Additionally, because project updates must be committed in order to be applied, this means that many commits won't actually be used; they'll simply be fixing bugs in the coding as the coder is testing the project. In this context, it makes sense to Amend Last Commit rather than creating a new commit for every change. That is, if the previous commit doesn't work, it can be amended. Once a project has been applied to create a built network, however, the commit should stand and further updates to the project should be done in a new commit.
The Readme.txt and the Asana task for the project should have notes on how the project was implemented in a human-readable form.
For example, a project may be described as Extend service hours for operator X. There are many ways this may be done, so the project coder should explain how they interpreted this (which often involves discussion with other staff and/or the project sponsor). In the end, an explanation might be something like:
This project goes through every line for operator X applies the maximum headway from the AM, MD and PM periods so that they apply to EA and EV as well. For lines that already had EA or EV service, the headway is not downgraded (but it may be upgraded).
When coding a project with new off-street nodes, see TransitNetworkCoding documentation for how the nodes connect with WNRs, PNRs, transfers and street access. Also, please include information about these nodes in the project's Readme.txt. For example:
The Readme.txt for MAJ_SMART_to_Cloverdale:
SMART extension to Cloverdale (adds Healdsburg station as well - see discussion on asana)
https://app.asana.com/0/741988522701299/928782229623017/f
Information for Cloverdale/Healdsburg SMART fare, transfer, PNR, and zone access is already in the network
Station wnr pnr xfer/network
Healdsburg 10460 11460 13460 8784
Cloverdale 15800 15801 15802 8480
station to walk funnel node: commuter_rail.zac
station to pnr node: commuter_rail.pnr
network to walk funnel node: smart_cloverdale.access
network to pnr node: smart_cloverdale.access
other stops to station xfer: commuter_rail_neti_xfer_links.dat
In addition to documenting this as described above, it's also good practice to keep this information up-to-date in TM1_2015_Base_Network's Node Description.xls
In June 2024, NetworkWrangler was updated to include a feature for optionally visualizing the effects of a project using Tableau. (See details in Asana task: Create tool to visualize project diffs This feature adds two additional optional arguments to the network building scripts referenced above:
--create_project_diffs: This will create Project Diffs folders for every application of every project — which is slow, but sometimes useful.
Alternatively, you may specify, -—create_project_diff [project_name] to create a Project Diff for a single project when it gets applied. (If it gets applied multiple times, then it'll get reported for each application).
Additionally, if a project has additional complications requiring a special tableau for visualizing the differences (beyond the default versions), then that tableau can be added to the network project and it will be used instead.
This way, a visualization of the project can be added to the project's Asana task for QA as a screenshot. Alternatively, if the project is too complicated for a screenshot to suffice, the project visualization tableau may be published to MTC's tableau online for others to review the changes.