River Charts is a Python
, Django
, Plotly
, and Pandas
web application that visualizes river data for a specific river/site/location.
- The line graph is driven by data pulled using an
API
from the United States Geological Survey (USGS).- The data is captured by the USGS using a gage height sensor every 15 minutes.
- The data is updated with the most recent river height data every time the application is loaded.
- Please be patient, the loading time of the application is around 30+ seconds (depending on your internet connection), due to the large
API
JSON
response and the amount of data being processed from the USGS. - The application can be configured to pull live data in via the
API
or use a static.json
file to display a snapshot of data.
- Please be patient, the loading time of the application is around 30+ seconds (depending on your internet connection), due to the large
- The loading screen screen contains fun wave and tube graphics as well as 20 "Fun Facts" regarding Tubing and Rivers on a 7 second interval to keep the user occupied while the data is loading in the background.
- I recommend using the application on a desktop since the chart is interactive has a wider view, but it can be used on a mobile device as well.
- Visit the application here.
- Features
- Background Story
- Definitions
- Getting Started
- What's Inside?
- API Documentation
- Disclaimer
- What's Next?
- Project
- Contributing
- Resources
- License
- Credit
- Responsive: The application is responsive and can be used on a mobile device.
- Interactive: The application uses
Plotly
to create an interactive line graph. - Data Visualization: The application visualizes river data for a specific river/site/location.
- Data Source: The application sources data using an
API
that returnsJSON
output. - Data Processing: The application processes the
JSON
output and converts it to aPandas
DataFrame
. - Data Manipulation: The application manipulates the
DataFrame
to create aPlotly
line graph. - Data Update: The application updates the data with the most recent river height data every time the application is loaded.
- Data Capture: The data is captured by the USGS using a gage height sensor every 15 minutes.
Every year, my friends and I float 2 miles down the Susquehanna River in NEPA on river tubes (a 2 hour float). I wanted to create a web application that would allow us to visualize past river data in order to see the river height on the days we floated down the river. Some float dates, we still got together, but we didn't float due to the dangerous river levels.
Below is the key to use when viewing the plots on the graph for Float dates:
Color | Hex | Floated Status | Floated Key |
---|---|---|---|
Green | #008000 |
Yes (Met up and did Float) | Y |
Red | #FF0000 |
No (Met up, but did not Float) | N |
Orange | #FFA500 |
Skipped (Skipped the event altogether) | S |
Note
The closest river gage to our float location is the Susquehanna River at Meshoppen, PA, which is the default site code in the application.
You can change the site code in the application config.py
file to visualize data for a different river/site code.
Here are some definitions to help you understand the terminology used in this document:
- USGS: The United States Geological Survey. The USGS is a science organization that provides impartial information on the health of our ecosystems and environment, the natural hazards that threaten us, the natural resources we rely on, the impacts of climate and land-use change, and the core science systems that help us provide timely, relevant, and useable information.
- Gage Height: The height of the water surface above the gage datum (zero point). Gage height is often used interchangeably with the more general term, stage, although gage height is more appropriate when used with a gage reading. Stage is more appropriate when used with a recorded or calculated gage height.
This project makes use of several libraries and frameworks:
- Python: For the application logic.
- Django: For web application functionality.
- Plotly: For creating interactive visualizations.
- Pandas: For data manipulation and analysis.
- Requests: For making
API
calls. - Python-Decouple: For storing sensitive information in a
.env
file. - See requirements.txt for a full list of dependencies.
- Edit
config.py
to add your own USGSAPI
(and other) information. - Toggle
USE_DUMMY_DATA
toTrue
inconfig.py
to use dummy data instead of theAPI
.- This is useful for testing the application without making
API
calls.
- This is useful for testing the application without making
- Toggle
USE_SNAPSHOT_DATA
toTrue
inconfig.py
to use snapshot/static data instead of theAPI
.- This is useful for avoiding
API
calls and using a static data set in thedata/snapshot.json
file. - You will need to populate the file with
JSON
by calling the "Historical River Levels"API
call inPostman
, included in this project (docs/api/River Charts.postman_collection.json
); see Calling the API below for more information. - Your app will esentially be a snapshot, and not live updates using this method, but it can be helpful if you want to avoid server costs making
API
calls.
- This is useful for avoiding
- The float data plots are driven from a
.csv
file located instatic/data/river_charts.csv
.- This file can be edited to add/remove float dates.
- The file is read in
views.py
and passed to the template as acontext
variable.
To install and run the project locally, follow the steps below:
-
Clone the repository:
git clone https://github.com/scottgriv/River-Charts
-
Navigate to the project directory:
cd [YOUR PROJECT DIRECTORY]
-
Create a virtual environment:
python -m venv venv
-
Activate the virtual environment:
source venv/bin/activate
-
Install the required packages using
requirements.txt
:pip install -r requirements.txt
Note: If you wany to generate a new
requirements.txt
file, run the following command:pip freeze > requirements.txt
-
Run the Django server:
python manage.py runserver
Now, you can visit http://127.0.0.1:8000/
in your browser to access the application.
- The application is hosted here on PythonAnywhere.
- The application is deployed using a
WSGI
configuration file. - First, make sure you adjust your
settings.py
fileALLOWED_HOSTS
to include your deployment host. - Second, make sure you adjust your
settings.py
fileDEBUG
toFalse
for production. - Finnaly, be sure to create a
.env
file where you host your application to store your sensitive information (excluded from this repository).
Below is a list of the main files and folders in this repository and their specific purposes:
River-Charts # Root folder
ββ .github # GitHub related files such as CHANGELOG, CREDITS, etc.
ββ docs # Included resources.
βΒ Β ββ api # API resources.
βΒ Β βΒ Β ββ River Charts.postman_collection.json # A Postman Collection used to manually call the API, and to gather data if you want to use this as a static site.
βΒ Β ββ assets # Misc. resources.
βΒ Β βΒ Β ββ stage versus flow slide.pdf # A PDF file that explains stage vs. flow slide.
βΒ Β ββ images # Image resources.
ββ River_Charts # The Django project directory.
βΒ Β ββ __init__.py # An empty file that tells Python that this directory should be considered a Python package.
βΒ Β ββ asgi.py # An entry-point for ASGI-compatible web servers to serve your project.
βΒ Β ββ settings.py # Settings/configuration for this Django project.
βΒ Β ββ urls.py # The URL declarations for this Django project.
βΒ Β ββ wsgi.py # An entry-point for WSGI-compatible web servers to serve your project.
ββ rivercharts # A directory for the rivercharts app.
βΒ Β ββ templates # A directory for HTML templates.
βΒ Β βΒ Β ββ rivercharts # A directory for HTML templates specific to the river_charts app.
βΒ Β βΒ ββ error.html # An HTML template that displays an error message.
βΒ Β βΒ ββ index.html # An HTML template that displays the application.
βΒ Β ββ __init__.py # An empty file that tells Python that this directory should be considered a Python package.
βΒ Β ββ admin.py # A file that registers models to be displayed in the Django admin site.
βΒ Β ββ apps.py # A file that contains the application configuration.
βΒ Β ββ config.py # A file that contains sensitive information (excluded from this repository).
βΒ Β ββ models.py # A file that contains the database models.
βΒ Β ββ tests.py # A file that contains the tests for the application.
βΒ Β ββ urls.py # A file that contains the URL declarations for the application.
βΒ Β ββ views.py # A file that contains the application logic.
ββ static # A directory for static files that are used in this Django project.
βΒ Β ββ assets # A directory images used throughout the application.
βΒ Β ββ css # A directory for CSS files.
βΒ Β βΒ Β ββ styles.css # A CSS file that contains the styles for the application.
βΒ Β ββ data # A directory for data files.
βΒ Β βΒ Β ββ fun_facts.json # A JSON file used to display Fun Facts while the application is loading, add or remove more facts here.
βΒ Β βΒ Β ββ snapshot.json # A JSON file used for static data when USE_SNAPSHOT_DATA is set to True in config.py.
βΒ Β βΒ Β ββ river_charts.csv # A CSV file that contains the float dates for the application.
βΒ Β ββ fonts # A directory fonts used throughout the application.
βΒ Β ββ js # A directory for JavaScript files.
βΒ Β Β Β ββ script.js # The JavaScript file used to render the loading screen.
ββ requirements.txt # A list of Python packages required to run this project.
ββ db.sqllite3 # A database file Django uses for this project (Do not delete).
ββ manage.py # A command-line utility that lets you interact with this Django project in various ways.
ββ .github # GitHub folder
ββ .gitignore # Git ignore file
ββ .gitattributes # Git attributes file
ββ PRG.md # PRG Connection File
ββ LICENSE # A file that contains the license for this project.
ββ VERSION # A file used to keep the repository release and the PythonAnywhere deployment in sync.
ββ README.md # This file.
Below is the documentation for the API
used in this application.
To call the API
and retrieve the data:
-
Make a
GET
request to:http://nwis.waterservices.usgs.gov/nwis/...
(based on your requirements).- ex.
https://waterservices.usgs.gov/nwis/iv?format=json&sites=01533400&startDT=2015-07-01&endDT=2023-08-16¶meterCd=00065&siteStatus=active&siteType=ST
- ex.
-
Pass the necessary parameters in the request.
- ex.
params = { "format": "json", # Set your interchange format. "sites": "01533400", # Site Code: Susquehanna River at Meshoppen, PA. "startDT": "2015-07-01", # Set the date you want to start collecting data from. "endDT": "2023-09-14", # This is based on the current date in the application. "parameterCd": "00065", # Parameter Code: Gage height, ft. "siteStatus": "active", # Selects sites based on whether or not they are currently active. Each USGS Water Science Center determines whether a site is active or inactive. The default is all (show both active and inactive sites). "siteType": "ST", # ST = A body of running water moving under gravity flow in a defined channel. The channel may be entirely natural, or altered by engineering practices through straightening, dredging, and (or) lining. An entirely artificial channel should be qualified with the "canal" or "ditch" secondary site type. }
-
Process the JSON response as demonstrated in the example above.
Notes:
- The application uses the
API
to source data for the graph. If theAPI
is down, the graph will not render. - Errors are handled in the application by redirecting the user to an error page with the appropriate error message.
HTTP
errors are handled in the application by redirecting the user to an error page with the appropriate error message.- The application will also display an error message if the
API
returns an empty response or a timeout error (default is set to 90 seconds inconfig.py
). - The
API
is rate limited to 30 calls per minute. If you exceed this limit, you will receive a429
error.
The application sources data using an API
that returns JSON
output. Here's an example of what the API response looks like:
{
"name": "ns1:timeSeriesResponseType",
...
"timeSeries": [
{
"sourceInfo": {
"siteName": "Susquehanna River at Meshoppen, PA",
...
},
"variable": {
"variableName": "Gage height, ft",
...
},
"values": [
{
"value": [
{
"value": "15.13",
"dateTime": "2015-07-01T00:00:00.000-04:00"
},
...
]
}
]
}
]
}
(For the sake of brevity, the full output is abbreviated with ...
)
- The data provided by this application is sourced from the USGS.
- It's subject to revision, and for more information, please refer to their official disclaimer.
- Software is provided as-is and no warranty is given about its usability.
Thank you for taking the time to read through this document and I hope you find it useful! If you have any questions or suggestions, please feel free to reach out to me.
Please reference the SUPPORT file in this repository for more details.
I'm looking forward to seeing how this project evolves over time and how it can help others with their GitHub Portfolio.
Please reference the CHANGELOG file in this repository for more details.
Please reference the GitHub Project tab inside this repository to get a good understanding of where I'm currently at with the overall project.
- Issues and Enhancements will also be tracked there as well.
Feel free to submit a pull request if you find any issues or have any suggestions on how to improve this project. You can also open an issue with the tag "bug" or "enhancement".
- How to contribute:
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/River-Charts
) - Commit your Changes (
git commit -m 'Add new feature'
) - Push to the Branch (
git push origin feature/River-Charts
) - Open a Pull Request
Please reference the CONTRIBUTING file in this repository for more details.
Below are some external resources I found helpful when creating River Charts:
- Python - A programming language that lets you work quickly and integrate systems more effectively.
- Django - A high-level Python Web framework that encourages rapid development and clean, pragmatic design.
- Plotly - A Python graphing library that makes interactive, publication-quality graphs online.
- Pandas - A fast, powerful, flexible and easy to use open source data analysis and manipulation tool.
- Requests - A simple, yet elegant HTTP library.
- Python-Decouple - A Python library for separating the settings of your Django/Flask/FastAPI project from the source code.
- Deploying Django - A tutorial on how to deploy Django.
- Django Settings - A list of all settings available in Django.
- Running the Django server - A tutorial on how to run the Django server.
- Creating Virtual Environments - A tutorial on how to create virtual environments.
- Activating a virtual environment - A tutorial on how to activate a virtual environment.
- Installing Packages - A tutorial on how to install packages using
pip
. - Requirements Files - A file containing a list of items to be installed using pip install like so:
pip install -r requirements.txt
. - PythonAnywhere - A Python hosting provider with a free tier.
Note
To use external API's (like this application does), PythonAnywhere requires a basic paid tier.
- USGS - The United States Geological Survey.
- USGS API - The USGS Instantaneous Values Web Service.
- USGS API Documentation - The USGS Instantaneous Values Web Service Documentation.
- USGS Site Web Service - The USGS Site Web Service.
- Codes and Parameters - The USGS Codes and Parameters Documentation.
- How to Get Real-time Flood Data from the USGS API - A tutorial on how to get real-time flood data from the USGS API.
- USGS Client Library - Client library for interfacing with USGS datasets (GitHub repository).
- USGS Client Library Repository - Client library for interfacing with USGS datasets.
- USGS API Homepage - The USGS API Homepage.
- USGS Rest API web services - The USGS Rest API web services.
- USGS Instantaneous Water Service data URL generator - The USGS Instantaneous Water Service data URL generator.
- National Water Dashboard - The National Water Dashboard.
- Rivers vs. Streams vs. Creeks (USGS) - A USGS article on the difference between rivers, streams, and creeks.
- How to Make Colored Country Maps with GeoPandas and Python - A tutorial on how to make colored country maps with GeoPandas and Python.
- 5 Ways to Use Free Data on the Internet for Science - A tutorial on how to use free data on the internet for science.
- GitHub for Biologists - A tutorial on how to use GitHub for biologists.
- Why does the USGS use the spelling "gage" instead of "gauge"? - Fun nerd fact.
This project is released under the terms of the GNU General Public License, version 3 (GNU GPLv3), which ensures that derivatives of the software remain open source.
- The GNU GPLv3 is a "copyleft" license, ensuring that derivatives of the software remain open source and under the GPL.
- For more details and to understand all requirements and conditions, see the LICENSE file in this repository.
Author: Scott Grivner
Email: scott.grivner@gmail.com
Website: scottgrivner.dev
Reference: Main Branch