diff --git a/paper/Feedback on Decision Support Modelling with Python Self-Guided Course (Responses) - Form Responses 1.csv b/paper/Feedback on Decision Support Modelling with Python Self-Guided Course (Responses) - Form Responses 1.csv index 24d6b5ff..26878443 100644 --- a/paper/Feedback on Decision Support Modelling with Python Self-Guided Course (Responses) - Form Responses 1.csv +++ b/paper/Feedback on Decision Support Modelling with Python Self-Guided Course (Responses) - Form Responses 1.csv @@ -73,4 +73,5 @@ José Miguel Espinoza" Thanks for all your effort and work. The course was very solid and educational. A thing that I would like to see in the course is more practical examples of applications. Regards and good luck!" -10/24/2023 8:38:21,in-person intensive workshop (one week - ~40 hours),I did a bit but fell behind,4,4,"Enjoyed the class, but don't use Flopy (I model pretty much entirely in MF-USG using custom Fortran/Python script and unstructured grids) so would have preferred instructionals re: file setup that could be applied to any model, rather than a row/column model built using Flopy. Really enjoyed the session on MOU and would like to learn more about this for my projects." \ No newline at end of file +10/24/2023 8:38:21,in-person intensive workshop (one week - ~40 hours),I did a bit but fell behind,4,4,"Enjoyed the class, but don't use Flopy (I model pretty much entirely in MF-USG using custom Fortran/Python script and unstructured grids) so would have preferred instructionals re: file setup that could be applied to any model, rather than a row/column model built using Flopy. Really enjoyed the session on MOU and would like to learn more about this for my projects." +10/25/2023 6:45:30,"the ""self-guided"" approach.",I did a bit but fell behind,3,3, \ No newline at end of file diff --git a/paper/feedback.ipynb b/paper/feedback.ipynb index bcff2ce7..609aadde 100644 --- a/paper/feedback.ipynb +++ b/paper/feedback.ipynb @@ -78,6 +78,16 @@ " )['inperson']/len(df)*100" ] }, + { + "cell_type": "code", + "execution_count": null, + "id": "118877b1-54e6-43c7-b109-d34dabe1ef36", + "metadata": {}, + "outputs": [], + "source": [ + "inperson" + ] + }, { "cell_type": "code", "execution_count": null, @@ -90,7 +100,8 @@ "outputs": [], "source": [ "inperson.index = [\n", - " 'In-person intensive\\nworkshop\\n(~40 hours)',\n", + " '\\n\\n\\nIn-person intensive\\nworkshop\\n(~40 hours)',\n", + " 'The \"self-guided\"\\napproach',\n", " 'This hybrid self-guided\\napproach with\\nfew online meetings\\n(as experienced now)'\n", "]" ] @@ -401,7 +412,7 @@ ], "metadata": { "kernelspec": { - "display_name": "gmdsitut", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, diff --git a/paper/paper.bib b/paper/paper.bib index 0c06f84d..62d9bb65 100644 --- a/paper/paper.bib +++ b/paper/paper.bib @@ -3,7 +3,7 @@ @techreport{whitepestpp title={Approaches to highly parameterized inversion: PEST++ version 5, a software suite for parameter estimation, uncertainty analysis, management optimization and sensitivity analysis}, author={White, Jeremy T and Hunt, Randall J and Fienen, Michael N and Doherty, John E}, year={2020}, - institution={US Geological Survey} + institution={US Geological Survey Techniques and Methods 7-C26} } @article{whitepyemu, @@ -27,11 +27,11 @@ @article{White_A_python_framework_2016 year = {2016} } -@article{White2020DR, +@techreport{White2020DR, author={White, J.T. and Foster, L.K. and Fienen, M.N. and Knowling, M.J. and Hemmings, B. and Winterle, J.R.}, year = {2020}, title={Towards reproducible environmental modeling for decision support: a worked example}, -journal = {{U.S. Geological Survey data release}}, +institution = {U.S. Geological Survey data release}, doi={10.5066/P9AUZMI7} } @article{white2020r3, @@ -144,5 +144,6 @@ @article{glaveski2019companies author={Glaveski, Steve}, journal={Harvard Business Review}, volume={2}, - year={2019} + year={2019}, + url={https://hbr.org/2019/10/where-companies-go-wrong-with-learning-and-development} } \ No newline at end of file diff --git a/paper/paper.md b/paper/paper.md index 3690a5a6..08f803fb 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -3,7 +3,7 @@ title: 'Self-Guided Decision Support Groundwater Modelling with Python' tags: - Python - groundwater modelling - - environemntal modelling + - environmental modelling - decision-support - uncertainty analysis authors: @@ -62,19 +62,18 @@ bibliography: paper.bib # Summary -The [GMDSI tutorial notebooks repository](https://github.com/gmdsi/GMDSI_notebooks/tree/main) provide learners with a comprehensive set of tutorials for self-guided training on decision-support groundwater modelling using Python-based tools. Although targeted at groundwater modelling, they are based around model-agnostic tools and readily transferable to other environmental modelling workflows. The tutorials are divided into three parts. The first covers fundamental theoretical concepts. These are intended as background reading for reference on an as-needed basis. Tutorials in the second part introduce learners to some of the core concepts parameter estimation in a groundwater modelling context, as well as providing a gentle introduction to the ``PEST``, ``PEST++`` and ``pyEMU`` software. Lastly, the third part demonstrates how to implement highly-parameterized applied decision-support modelling workflows. Their aim is to provide examples of both “how to use” the software as well as “how to think” about using the software. +The [GMDSI tutorial notebooks repository](https://github.com/gmdsi/GMDSI_notebooks/tree/main) provides learners with a comprehensive set of tutorials for self-guided training on decision-support groundwater modelling using Python-based tools. Although targeted at groundwater modelling, they are based around model-agnostic tools and readily transferable to other environmental modelling workflows. The tutorials are divided into three parts. The first covers fundamental theoretical concepts. These are intended as background reading for reference on an as-needed basis. Tutorials in the second part introduce learners to some of the core concepts parameter estimation in a groundwater modelling context, as well as providing a gentle introduction to the ``PEST``, ``PEST++`` and ``pyEMU`` software. Lastly, the third part demonstrates how to implement highly-parameterized applied decision-support modelling workflows. Their aim is to provide examples of both “how to use” the software as well as “how to think” about using the software. A key advantage to using notebooks in this context is that the workflows described run the same code as practitioners would run on a large-scale real-world application. Using a small synthetic model facilitates rapid progression through the workflow. - # Statement of Need Effective environmental management necessitates transparent acknowledgment of uncertainties in critical decision-making predictions, coupled with efforts to mitigate these uncertainties, especially when significant risks accompany management outcomes. The significance of uncertainty quantification (UQ) and parameter estimation (PE) in environmental modeling for decision support is widely acknowledged. UQ provides estimates of outcome uncertainty, while PE reduces this uncertainty through assimilating data. -Implementing highly-parameterized UQ and PE in real-world modeling can be challenging due to both theoretical complexity and practical logistics. Limited project time and funding also often hinder their application. Open-source software such as ``PEST`` `[@pest]` and ``PEST++`` `[@whitepestpp]` provide tools for undertaking UQ and PE analyses. However, the steep learning curve associated with their use and the lack of user-friendly training materials have been a barrier to uptake. +Implementing highly-parameterized UQ and PE in real-world modeling can be challenging due to both theoretical complexity and practical logistics. Limited project time and funding also often hinder their application. Open-source software such as ``PEST`` [@pest] and ``PEST++`` [@whitepestpp] provide tools for undertaking UQ and PE analyses. However, the steep learning curve associated with their use and the lack of user-friendly training materials have been a barrier to uptake. -There is a growing demand within the environmental modelling community for transparent, reproducible, and accountable modeling processes, driven by the need for increased credibility and rigor in computational science and environmental simulation `[@white11rapid; @fienen2016hess]`. While some script-based tools enhance the reproducibility of forward model construction `[@flopy]`, they often overlook UQ and PE analyses. In decision-support scenarios, these analyses are equally vital for robust model deployment as the forward model itself. +There is a growing demand within the environmental modelling community for transparent, reproducible, and accountable modeling processes, driven by the need for increased credibility and rigor in computational science and environmental simulation [@white11rapid; @fienen2016hess]. While some script-based tools enhance the reproducibility of forward model construction [@flopy], they often overlook UQ and PE analyses. In decision-support scenarios, these analyses are equally vital for robust model deployment as the forward model itself. -The uptake of Python for environmental modeling has increased in recent years, due to its open-source nature, user-friendly syntax, and extensive scientific libraries. Python-based tools have been developed to facilitate UQ and PE analyses, such as ``pyEMU`` `[@White_A_python_framework_2016; @white2021towards]`. ``pyEMU`` is a Python package that provides a framework for implementing UQ and PE analyses with PEST and PEST++. It offers a range of capabilities, including parameter estimation, uncertainty analysis, and management optimization. Although initially designed for groundwater modeling, ``pyEMU``'s methodologies are versatile and can be applied to diverse numerical environmental models, as long as they can be manipulated using text files and generate outputs that can be automatically extracted without manual interference. +The uptake of Python for environmental modeling has increased in recent years, due to its open-source nature, user-friendly syntax, and extensive scientific libraries. Python-based tools have been developed to facilitate UQ and PE analyses, such as ``pyEMU`` [@White_A_python_framework_2016; @white2021towards]. ``pyEMU`` is a Python package that provides a framework for implementing UQ and PE analyses with PEST and PEST++. It offers a range of capabilities, including parameter estimation, uncertainty analysis, and management optimization. Although initially designed for groundwater modeling, ``pyEMU``'s methodologies are versatile and can be applied to diverse numerical environmental models, as long as they can be manipulated using text files and generate outputs that can be automatically extracted without manual interference. The tutorial notebooks discussed herein provide a comprehensive, self-guided, and open-source resource for learning decision-support modeling workflows with Python. They are designed to be accessible to a broad audience, including students, researchers, and practitioners who aim to undertake applied environmental decision-support modelling. @@ -92,9 +91,9 @@ Decision-support modeling often demands capabilities that surpass what current G Over time, more modelers are turning to Python packages like ``FloPy`` and ``pyEMU`` for model and ``PEST++`` setup. Unfortunately, the adoption of this approach is hindered by a steep learning curve primarily due to the scarcity of user-friendly training materials. The [GMDSI tutorial notebooks](https://github.com/gmdsi/GMDSI_notebooks/tree/main) aim to address this gap by providing a comprehensive, self-guided, and open-source resource for learning decision-support modeling workflows with Python. -The roots of the materials making up the tutorial notebooks were from a traditional, week-long classroom course curriculum developed for internal training at the USGS by a subset of the authors of this paper. For this course, the instructors leveraged the power of jupyter notebooks as a mechanism to teach both the fundamental background and application of inverse theory. High-level mathematical libraries in python (and other high-level languages with easy plotting utilities such as MATLAB and R) provide an opportunity for students to explore linear algebra and statistical modeling principles that underlie the PE and UQ techniques implemented in ``PEST`` and ``PEST++``. Furthermore, the combination of text, code, and graphics provide an interactive platform for mixing theory and applications and, potentially, providing a template for application on real-world applications. The native support for python makes the connection between worked examples and notebooks seamless and has connections with other worked examples [`@white2020r3`;`@white2020DR`;`@fienen2022risk`;https://github.com/doi-usgs/neversink_workflow] +The roots of the materials making up the tutorial notebooks were from a traditional, week-long classroom course curriculum developed for internal training at the USGS by a subset of the authors of this paper. For this course, the instructors leveraged the power of jupyter notebooks as a mechanism to teach both the fundamental background and application of inverse theory. High-level mathematical libraries in python (and other high-level languages with easy plotting utilities such as MATLAB and R) provide an opportunity for students to explore linear algebra and statistical modeling principles that underlie the PE and UQ techniques implemented in ``PEST`` and ``PEST++``. Furthermore, the combination of text, code, and graphics provide an interactive platform for mixing theory and applications and, potentially, providing a template for application on real-world applications. The native support for python makes the connection between worked examples and notebooks seamless and has connections with other worked examples [@white2020r3; @White2020DR; @fienen2022risk; https://github.com/doi-usgs/neversink_workflow] -After three iterations of teaching the in-person class, the instructors concluded that the materials and approach were valuable, but came to question the level of retention by students in a 40-hour intensive setting. It is well-documented that without repetition and rapid adoption of new techniques, they can fade quickly from memory [`@glaveski2019companies`]. As a result, the authors, with support from the GMDSI, endeavored to build on the positive aspects of using jupyter notebooks and explore alternative teaching environments instead of week-long classes. The first major change was to add sufficient narration and explanation to the notebooks to improve possibilities for self-study. The initial design through in-person instruction was to have the notebooks serve as illustrations to assist in a narrative discussion, so bolstering of the explanatory text was necessary to help them stand alone. The next change was to refactor the organization from a strictly linear progression to the current three-part organization discussed below. This led to a hybrid model of self-study punctuated by discussion and background lectures online. +After three iterations of teaching the in-person class, the instructors concluded that the materials and approach were valuable, but came to question the level of retention by students in a 40-hour intensive setting. It is well-documented that without repetition and rapid adoption of new techniques, they can fade quickly from memory [@glaveski2019companies]. As a result, the authors, with support from the GMDSI, endeavored to build on the positive aspects of using jupyter notebooks and explore alternative teaching environments instead of week-long classes. The first major change was to add sufficient narration and explanation to the notebooks to improve possibilities for self-study. The initial design through in-person instruction was to have the notebooks serve as illustrations to assist in a narrative discussion, so bolstering of the explanatory text was necessary to help them stand alone. The next change was to refactor the organization from a strictly linear progression to the current three-part organization discussed below. This led to a hybrid model of self-study punctuated by discussion and background lectures online. # Resources @@ -153,14 +152,14 @@ In summary, the tutorial notebooks are organized to guide learners through a str The notebooks were employed during the [Applied Decision Support Groundwater Modeling With Python: A Guided Self-Study Course](https://gmdsi.org/blog/guided-self-study-course/) hosted by GMDSI. This self-guided course comprised 5 online sessions, each lasting 1 to 2 hours and focused on the workflows of Part 2. During each session the instructors go through a section of the tutorials and expand on some of the concepts. Learners were tasked with going through the notebooks in between sessions to stimulate discussion and questions. Sessions were recorded and can be accessed [on the GMDSI website](https://gmdsi.org/education/videos/). Beyond the live online sessions, learners were incentivized to make use of the GitHub [Discussions](https://github.com/gmdsi/GMDSI_notebooks/discussions) feature to retain a search-engine findable record of common questions. -Feedback from the 65 students who participated in the course was anecdotal but informative. Figure (@fig-responses) summarizes the responses by 33 respondents to four questions, comprising 51%. The majority of respondents indicated a preference for this hybrid self-guided/online instruction approach over an in-person week-long intensive class with zero respondents indicating preference for self-guided study of the course materials only. More than 60% of the respondents reported being able to keep up with most or all of the assigned self-study notebooks, while 39% reported falling behind. Given 5 categories of comfort level working with PEST++ (1 being most comfortable, and 5 being least) before and after the class, there was a notable shift toward higher comfort level. Interestingly, when evaluating individual responses, the majority (60%) reported being more comfortable with PEST++ after the course (defined as an increase of one level) and 15% reported being much more comfortable (an increase of two levels). However, 18% reported the same comfort level before and after while 24% reported being less or much less comfortable (a decrease or one or two levels, respectively). Without further questions, we cannot know whether these decreases reflect a humble realization that their mastery was less complete than they thought, _a priori_, or whether the material was confounding. +Feedback from the 65 students who participated in the course was anecdotal but informative. Figure (@fig-responses) summarizes the responses by 34 respondents to four questions, comprising 52%. The majority of respondents indicated a preference for this hybrid self-guided/online instruction approach over an in-person week-long intensive class with only one respondent indicating preference for self-guided study of the course materials only. Just under 60% of the respondents reported being able to keep up with most or all of the assigned self-study notebooks, while 41% reported falling behind. Given 5 categories of comfort level working with PEST++ (1 being most comfortable, and 5 being least) before and after the class, there was a notable shift toward higher comfort level. Interestingly, when evaluating individual responses, the majority (56%) reported being more comfortable with PEST++ after the course (defined as an increase of one level) and 15% reported being much more comfortable (an increase of two levels). However, 21% reported the same comfort level before and after while 24% reported being less or much less comfortable (a decrease or one or two levels, respectively). Without further questions, we cannot know whether these decreases reflect a humble realization that their mastery was less complete than they thought, _a priori_, or whether the material was confounding. Open-ended feedback from the participants was generally positive and also included some constructive criticism. Participants appreciated the opportunity to ask questions and several reported hearing the discussion around other peoples' questions as being valuable and clarifying aspects of the material. The main critical suggestions included incorporating more real-world examples rather than relying, as we 100% did in the notebook design, on the synthetic model. Participants also noted the twin challenges of a large amount of information coupled with trying to be accountable to keep up in the class as potentially limiting the value relative to a week-long course. We conclude from this experience that the hybrid approach has value but there may still be a better approach for future educational opportunities. -![Summary of responses to post-course survey based on 29 responses. Panel A summarizes whether respondents would prefer and intensive in-person workshop or this hybrid option. Panel B summarizes how much of the notebooks respondents were able to complete throughout the course. Panel C summarizes respondent comfort level with PEST++ before and after the course. Panel D highlights individual changes in comfort level reported due to the course.](./responses.png){#fig-responses} +![Summary of responses to post-course survey based on 34 responses. Panel A summarizes whether respondents would prefer and intensive in-person workshop or this hybrid option. Panel B summarizes how much of the notebooks respondents were able to complete throughout the course. Panel C summarizes respondent comfort level with PEST++ before and after the course. Panel D highlights individual changes in comfort level reported due to the course.](./responses.png){#fig-responses} # Acknowledgements -The tutorials were originally developed with support from the U.S Geological Survey (USGS) and support from USGS continues through the HyTest training project. Continued development and support is funded by the Groundwater Modelling Decision Support Initiative (GMDSI). GMDSI is jointly funded by BHP and Rio Tinto. We thank Dr. John Doherty for his tireless and pioneering efforts starting `PEST` and continuing to innovate and Dr. Randall Hunt for his leadership in `PEST` and `PEST++` applications and contributions to the initial curriculum for this material and the early version of the notebooks. We finally thank users and stress-testers for their valuable feedback and continued community contributions to the repository. +The tutorials were originally developed with support from the U.S Geological Survey (USGS) and support from USGS continues through the HyTest training project. Continued development and support is funded by the Groundwater Modelling Decision Support Initiative (GMDSI). GMDSI is jointly funded by BHP and Rio Tinto. We thank Dr. John Doherty for his tireless and pioneering efforts starting `PEST` and continuing to innovate and Dr. Randall Hunt for his leadership in `PEST` and `PEST++` applications and development and contributions to the initial curriculum for this material and the early version of the notebooks. We finally thank users and stress-testers for their valuable feedback and continued community contributions to the repository. # References diff --git a/paper/responses.png b/paper/responses.png index 78968390..9c3c2f1b 100644 Binary files a/paper/responses.png and b/paper/responses.png differ