From 625998eab946c0c7e28b42da26104778d18b2de0 Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Sun, 15 Dec 2024 17:04:33 +0100 Subject: [PATCH 01/13] Initial converted qmd from docx --- .../Causal Indicator_ Cost Savings_v2.qmd | 148 ++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 sections/0_causality/Causal Indicator_ Cost Savings_v2.qmd diff --git a/sections/0_causality/Causal Indicator_ Cost Savings_v2.qmd b/sections/0_causality/Causal Indicator_ Cost Savings_v2.qmd new file mode 100644 index 0000000..c513eef --- /dev/null +++ b/sections/0_causality/Causal Indicator_ Cost Savings_v2.qmd @@ -0,0 +1,148 @@ +# The effect of Open Data on cost savings {#the-effect-of-open-data-on-cost-savings .unnumbered} + +History (please fill out in reverse chronological order, latest revision on top): + +| | | | | +|----|----|----|----| +| Version | Revision date | Revision | Author | +| | | | | +| 1.3 | 2024-12-09 | Second draft | M. Scorrano, E. Delugas, G. Catalano (reviewer) | +| 1.2 | 2024-11-27 | Peer review | V.A. Traag | +| 1.1 | 2024-11-22 | First draft | M. Scorrano, E. Delugas, S. Vignetti (reviewer) | +| 1.0 | 2024-10-07 | Template outline | E. Delugas | + +## Literature background + +Measuring the economic impact of open science and open data has proven to be challenging. Many theoretical studies highlight the benefits of making research results public, with strong support for Open Science from economic research on technological change (Chataway et al., 2017; Yozwiak et al., 2015; Mazzucato, 2011). However, only few studies have attempted to measure the impacts of open science compared to closed science, and more robust evidence on how Open Science drives innovation and economic outcomes is needed to strengthen support and counter emerging criticisms (Karasz et al., 2024; Ali-Khan et al., 2018). The existing literature mainly concentrates on specific sectors, particularly health, medicine, and biosciences, which receive more attention due to early regulation by funders and significant interest in clinical trial outcomes. Another important stream of literature is focused on highlighting the economic value of Open Science through personal industry experiences, though lacking precise quantitative evidence, with contributions from McManamay and Utz (2014) on fisheries, Harding (2017) on medicine, Chan (2015) on the transition to an Open Science model, and Chen et al. (2017) on the role of open data in AI and machine learning applications. Although directly linking economic outcomes to open data initiatives can be challenging, with authors combining theoretical arguments and the limited quantitative evidence available at the time of their publication (Ali and Dahlhaus, 2022; Arzberger, 2004; Tennant et al., 2016; Fell, 2019; Wehn et al., 2021), open access to findings and data is considered to lead to significant savings in access costs. By removing paywalls and subscription fees, open data allows researchers and businesses to access valuable information without incurring additional costs. A major economic benefit of lowering the cost of knowledge is the availability of an extra budget that can be reallocated for other purposes (Tennant et al., 2016). + +By eliminating barriers to data access, organisations can reduce the time spent on data collection and focus on core activities. Fell (2019) notes that open science reduces the time associated with accessing new knowledge, directly contributing to enhanced research quality and productivity increases. Although the specific time savings associated with open access were not directly tested, Parsons et al. (2011) conducted an interview-based study that supports this potential benefit. Beagrie and Houghton (2014) demonstrate that data sharing and curation significantly enhance research efficiency, with labour cost savings ranging from two to over twenty times the operational costs of the data centres. While there are indirect costs associated with preparing data for sharing, such as time and expenses incurred by depositors, the benefits realised by users through improved efficiency and their willingness to pay for access far outweigh these costs (Beagrie and Houghton, 2012). Additionally, findings from Beagrie and Houghton (2016;2021) provide compelling evidence of the timesaving and efficiency gains associated with using European Bioinformatics Institute services (EMBL-EBI). + +Open data minimises redundant data collection and mitigates the "file drawer effect," where valuable findings remain unpublished and inaccessible, ultimately impeding research effectiveness (van Assen et al., 2014). According to SPARC Europe (2019), making research data openly available can save up to 9% of a project's costs by preventing unnecessary data collection and facilitating the efficient reuse of existing data. Additionally, Houghton, Swan, and Brown (2011) estimate that access barriers to academic research in Denmark cost DKK 540 million annually. This figure is based on the average time spent (51-63 minutes) attempting to access research articles and the study highlights that delays in accessing academic research can prolong product and process development by an average of 2.2 years, resulting in significant financial losses for firms. + +The broader implications of open data extend beyond mere cost and time savings. Scientific literature is widely recognised as an important source of strategic knowledge, facilitating the exploration of new ideas in industrial research and innovation, particularly for small and medium enterprises that may struggle to obtain data independently (European Union, 2020). However, inefficiencies in traditional publishing models, such as delays and biases in data dissemination, can negatively impact private research productivity, as discussed by Harding (2017). Open data benefits various sectors, including agriculture, the environment, forensics, and industrial biotechnology, by providing access to information that helps researchers understand their fields and build on existing work (Fell, 2019; Tennant et al., 2016; Tripp et al.,2011; Yozwiak et al., 2015; Arshad et al., 2016). + +To fully harness the potential of open data, it is important to develop the necessary skills and capacities to manage it effectively. Zeleti and Ojo (2014) emphasise the need to streamline data generation processes to produce insights that inform and shape business strategies. Given that open data initiatives contribute to greater transparency and accountability, businesses that leverage open data achieve cost savings by co-creating and integrating data from multiple sources to enhance their services (Davies & Perini, 2016; Lindman et al., 2014). Fell (2019) suggests that adopting an open approach fosters connections and encourages collaborations that might not occur or would take longer in a closed environment. This is exemplified by the work of Van Vlijmen et al. (2020), who show how the integration of diverse types of open data, specifically scientific, clinical, and experimental public evidence, can be achieved using advanced AI platforms like Euretos. By combining these various data sources, the platform enhances the depth of information available for analysis, enabling more accurate predictions regarding drug efficacy, as demonstrated by a machine learning model that improved prediction accuracy by 12 percentage points over previous state-of-the-art. + +Despite the theoretical benefits of open data, several limitations hinder a comprehensive assessment of its economic impact fully. Implementing open data practices requires significant investments in infrastructure, technology, and training, potentially offsetting some cost savings (van Vlijmen et al., 2020). Moreover, the European Commission study emphasises that the benefits of open data are contingent on the quality and standardisation of the data provided (European Commission and PwC EU Services, 2018). Finally, a major limitation is the scarcity of empirical evidence; few studies have attempted to measure the impacts of open science compared to closed science, making it challenging to generalise findings (Karasz et al., 2024). Herala et al. (2016) review the benefits and challenges of open data initiatives in the private sector, highlighting advantages like enhanced collaboration and innovation, but caution that these are often based on speculative assumptions rather than empirical evidence, emphasising the need for further research to inform best practices and mitigate risks associated with increased costs and data privacy concerns. + +## Directed Acyclic Graph (DAG) + +As discussed in the general introduction on causal inference, we use DAGs to represent structural causal models. In the following, a DAG (Figure 1) is employed to examine the causal relationship between *Open Data* and *Cost Savings*. The visual illustrates multiple potential pathways, including a direct path from Open Data and Cost Savings, an indirect one involving Time Savings (i.e., a mediator), and additional paths that incorporate factors affecting either Open Data or Time Savings (i.e., confounders). These additional factors, such as technological infrastructure, data quality and availability, standardisation, user skills, innovation, and collaboration introduce layers of complexity to the model. As we will show in the subsequent sections, they are essential to discuss the causal and non-causal, open and closed, relationships among all these variables. + +Figure 1: Hypothetical structural causal model on Open Data + +![Immagine che contiene testo, linea, diagramma, Carattere +Descrizione generata automaticamente](figures/1d1cf98828cbe8eab6cfc7690c05d6db29154e28.png) + +## The effect of Open data on Cost Saving + +In this section, we apply the concepts presented in the section Causality in Science Studies to potential research questions. We present a specific perspective on causal inference through the lens of structural causal models (Pearl 2009). + +Suppose we are interested in assessing the *total causal effect* of *Open Data* on *Cost Saving*. According to our model (Figure 1), there are multiple pathways from Open Data to Cost Saving, some are causal, some are not. To estimate the causal effect of interest, we need to make sure that all causal paths are open, and all non-causal paths are closed. Within the DAG representation, two causal pathways can be identified: a *direct* pathway of *Open Data → Cost Saving,* representing the direct effect of Open Data on Cost Savings*,* and an *indirect* pathway *Open Data → Time Saving → Cost Saving*, where the effect is indirect and mediated by Time Saving. The *direct* effect captures the immediate benefits of providing free access to datasets, while the *indirect* effect, mediated by Time Savings, strengthens the relationship by triggering additional efficiencies that also lead to Cost Savings. + +To properly estimate the *total* causal effect of Open Data on Cost Saving, an empirical model should not control for Time saving. On the contrary, if the model conditions on Time Savings, even implicitly (e.g., by accounting for approaches and tools that optimise and speed up data access and processing), it closes the causal path and introduces biases into the estimation of the total effect (Figure 2). + +Figure 2: Misleading structural causal model on Open Data, conditioning on mediator (Time Saving) + +![Immagine che contiene testo, linea, schermata, diagramma +Descrizione generata automaticamente](figures/f0d9bcfde3955951768ec4951070567661000e25.png) + +*Note: DAG illustrating the misleading effect of conditioning on the mediator variable, Time Saving. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship.* + +As mentioned before, the proposed model accounts for additional variables such as *Data Availability, Quality and Standardisation*, *Users' Skills, Collaboration, and Technological Infrastructure*. These variables can act as *confounders* along different pathways illustrated in Figure 1. Examples of non-causal paths represented in the DAG are: + +- *Open Data* *←* *Technological Infrastructure* → *Time Saving* → *Cost Saving* +- *Open Data* ← *Data Availability, Quality and Standardisation* → *Time Saving* → *Cost Saving* +- *Open Data* ← *User’s skills* → *Time Saving* → *Cost Saving* + +In these pathways, to correctly identify the causal effect of *Open Data* on *Time Saving*, and by extension, on *Cost Savings*, it is required to control for these confounders. This is because the confounders jointly affect Open Data and Time Savings and omitting them from empirical models results in omitted variable bias. In the proposed example, to correct identify the causal effect one should control for *Technological infrastructure*, *Data availability, quality and standardisation, and the skills of users.* + +There might be instances where a confounder is not observed because is not included in the dataset or not observable at all. In such cases, the non-causal path remains open, resulting in biased conclusions. In fact, unobservable factors might be correlated with observable variables, leaving causal paths unexplored. As a result, if these confounders are not accounted for, we are unable to fully isolate the causal effect of *Open Data* influences *Cost saving*. + +Another case that makes not possible the identification of the causal effect is erroneously controlling for a collider (see the introduction for further details). In the pathway *Open Data* → *Innovation* ← *Collaboration*→ *Cost Saving,* the variable *Innovation* acts as a collider. Hence, this path is already closed, and the bias arises when controlling for innovation. + +Empirically speaking, in a model where to estimate the causal effect of Open Data on Cost savings we condition on Innovation (and not for Collaboration), it is likely to get a downward estimation of the causal effect, since both Collaboration and Open Data have a positive impact on Innovation. This conditioning opens up the non-causal pathway, Open Data → Innovation ← Collaboration → Cost Saving, which connect *Open Data* and *Cost Saving* through *Collaboration*, creating a spurious association and distorting the true effect of *Open Data* on *Cost Saving*. This is an example of *bad controls (*Angrist and Pischke 2009), a concept explained in the general introduction. Only by ignoring the collider, meaning non-conditioning on it in empirical models, we can effectively isolate the causal effect. This non-causal path is open because *Innovation* is open (because it is a collider that is conditioned on), and because *Collaboration* is open (because it is a confounder that is not conditioned on) (see Figure 3). + +[]{#_Ref182214498}Figure 3: Misleading structural causal model on Open Data, not conditioning on Collaboration (confounder) and Innovation (collider) + +![A diagram of cost saving and cost saving +Description automatically generated](figures/26de0c31a0e5f229ff6935be961496367a3b6cd4.png) + +Note: DAG illustrating the misleading effect of conditioning on a collider variable, *Innovation,* and not conditioning on a confounder, *Collaboration*. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship. + +In addition, *Collaboration* acts as a confounder on the non-causal path *Open data* ← *Collaboration* → *Cost saving*. To identify the causal effect, we hence need to close this non-causal path by conditioning on *Collaboration*. After controlling for Collaboration, whether *Innovation* is conditioned on is then irrelevant for the identification of the causal effect. When all non-causal paths are closed, the research design is said to meet the *backdoor* criterion, a formal requirement that ensures the design blocks all non-causal paths between the treatment (*Open Data*) and the outcome (*Cost Saving*), enabling us to identify the causal effect in question (Cunningham, 2021) (Figure 4). + +[]{#_Ref180136180}Figure 4: Hypothetical structural causal model on Open Data, conditioning on confounders and not on mediator and collider + +![A diagram of cost saving and standardisation +Description automatically generated](figures/0558967f8b3c0c345051326ee1dea0137c4caf1e.png) Note: DAG illustrating the total effect *Open Data* on *Cost Saving*, conditioning on confounders and not on mediator and collider. Nodes that are controlled for have a thick outline. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship. + +This example highlights key components of causal inference: controlling for confounders *(Data availability, quality and standardisation, User skills, Collaboration,* and *Technological Infrastructure*), not controlling for mediators *(Time saving),* and not controlling for colliders *(Innovation),* as shown in Figure 4*.* Constructing an appropriate DAG is important when aiming to draw causal conclusions. Without making assumptions explicit via a DAG, it would be unclear which variables should be controlled for and which not. Omitting important variables weakens the study's ability to draw accurate conclusions about cause and effect. Moreover, adding complexity to a DAG does not always change the variables that need to be controlled for when identifying the causal effect. In some cases, such as when adding confounders between unrelated variables, the identification of the relationship between Open data and Cost savings remains unaffected. However, if a confounder is introduced between Open data and Cost savings directly, it becomes necessary to control for it. + +Klebel and Traag (2023) emphasise the importance of carefully selecting variables when analysing causal relationships. They warn against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which they refer to as "causal salad" (McElreath, 2020). Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, Pearl proposes using the *do-operator^[\[1\]](#footnote-2){#footnote-ref-2}^* to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system (Pearl, 2009). + +This approach is effective in predicting how probability distributions shift under controlled changes when the causal structure is known. However, this approach depends on having an established causal graph, limiting its use for exploring causality from scratch. Critics suggest an alternative approach that allows causal discovery through experimentation without prior assumptions about mechanisms (Woodward, 2003). This is especially useful in complex fields, such as social and biomedical sciences, where causal relationships are less understood. Considering this, Cunningham (2021) argues that sample selection problems have been recognised long before the introduction of DAGs, with early solutions like Heckman (1979), and emphasizes that an atheoretical approach to empiricism is inadequate. He asserts that causal inference requires a deep understanding of the behavioural processes behind the phenomenon being studied, and while DAGs are useful, they cannot replace the need for theoretical knowledge in creating credible identification strategies. Thus, causal inference is not solved by simply collecting more data, but by integrating theory with empirical analysis. + +## Discussing empirical issues + +The presented model illustrates how open data might drive cost savings by focusing on a limited set of variables commonly discussed in the literature. However, this approach presents challenges, as the existing literature reveals a significant gap in empirical studies that specifically measure the economic impact of open data on cost savings. + +Other relevant factors could be included based on additional evidence but are currently excluded to maintain a coherent link with existing empirical findings and to preserve simplicity. While there is general agreement on the potential benefits of open data for cost savings, no study to date has estimated the total effect using a causal identification strategy such as the one presented here. Moreover, the specific causal pathways remain unclear, as little attention has been paid to the intermediate factors influencing the total effect. This lack of empirical evidence on these pathways limits the ability to draw strong conclusions from the model and underscores the need for more focused empirical research. + +It is also important to acknowledge that causal pathways are not static and may evolve depending on the context or field of application. Cunningham (2021) highlights that causal inference requires a deeper understanding of the underlying processes governing the system being studied. It is not solely about the data but also about the theoretical and contextual knowledge of how behaviours, choices, or events interact to produce stable outcomes. Timing and the evolution of causality play a critical role, as interactions between variables change over time, leading to different outcomes in different contexts. This complexity, arising from reverse causality and cyclical interactions, represents a key limitation of this DAG. The relationships between variables are often more intricate, with the potential for bidirectional influences. For example, when organisations share open data, they enable collaborative efforts that can lead to innovative solutions, such as new products or services. Increased collaboration, in turn, may prompt organisations to adopt more open data practices, creating a bidirectional relationship. This innovation can then lead to cost savings by streamlining processes or reducing redundancies. Conversely, independent cost savings may provide the resources necessary for further investment in innovation, illustrating a cyclical relationship where each factor reinforces the other. This dynamic interplay complicates the one-way causal pathways represented in the DAG. Feedback loops and time-dependent changes are often critical in real-world scenarios, making the DAG an initial framework rather than a definitive model. Further empirical research is essential to refine and validate the relationships it proposes, capturing the complexities and nuances of the interactions between these variables. + +A targeted survey could be a viable approach to address the empirical challenges arising from the lack of data and counterfactual evidence in studying the economic impact of open data. This survey should be carefully designed to measure the direct effects of open data on cost savings while accounting for confounding factors. It should aim to answer specific research questions, such as the extent of open data sharing, the perceived efficiency gains from data use, and the direct cost reductions attributed to open data initiatives. The survey would need to collect both longitudinal and cross-sectional data from a diverse sample of organisations that share open data as well as those that do not. This would allow for comparisons of outcomes and the establishment of causal relationships. Key variables to be measured should include the extent of open data sharing, cost reductions, new products or services enabled or accelerated by open data, and changes in collaboration. Questions investigating the perceived causal pathways would also be essential. Incorporating temporal data would provide further insight, enabling a better understanding of causal direction and distinguishing between short-term and long-term effects. + +Access to such detailed data would enable the use of causal inference techniques, such as Propensity Score Matching (PSM) or the Difference-in-Differences (DID) estimator, to identify the true impact of open data on cost savings. By addressing current data gaps, this survey could provide the empirical evidence needed to validate and refine the DAG, enhancing our understanding of how open data drives economic outcomes like cost savings. For instance, in the PSM context, a rich dataset would support the development of a more comprehensive DAG, helping to identify variables to include in the matching process (e.g., data availability, technological infrastructure, and user skills) and those to exclude (e.g., time savings and innovation). This approach would strengthen the empirical basis for analysing the causal impact of open data. + +## References + +Angrist, J. D., & Pischke, J.-S. (2009). *Mostly Harmless Econometrics: An Empiricist’s Companion*. Princeton University Press. + +Beagrie, N. and Houghton, J. (2014). The Value and Impact of Data Sharing and Curation. + +Beagrie, N. and Houghton, J. (2016). The Value and Impact of the European Bioinformatics Institute. . https://www.beagrie.com/static/resource/EBI-impact-report.pdf . + +Cunningham, S. (2021). *Causal Inference: The Mixtape.* Yale University Press. https://www.jstor.org/stable/j.ctv1c29t27. + +Davies, T. and Perini, F. (2016). Researching the emerging impacts of open data: revisiting the ODDC conceptual framework. The Journal of Community Informatics, 12(2). https://openjournals.uwaterloo.ca/index.php/JoCI/article/view/3246. + +European Commission and PwC EU Services. (2018). Cost-benefit analysis for FAIR research data: cost of not having FAIR research data. LU: Publications Office. https://data.europa.eu/doi/10.2777/02999. + +European Commission. (2020). The economic impact of open data: opportunities for value creation in Europe. LU: Publications Office. . + +Fell, M.J. (2019). The Economic Impacts of Open Science: A Rapid Evidence Assessment. Publications, 7(3), p.46. + +Harding, R. J. (2017). Open Science and Accelerating Discovery in Rare and Neglected Diseases. Proceedings of the 21st International Conference on Electronic Publishing, 1–5. + +Herala, A., Vanhala, E., Porras, J. and Krri, T. (2016). Experiences about opening data in private sector: A systematic literature review. In: 2016 SAI Computing Conference (SAI). 2016 SAI Computing Conference (SAI). London: IEEE, pp.715–724. https://ieeexplore.ieee.org/document/7556060/ . + +Houghton, J., Swan, A. and Brown, S. (2011). Access to research and technical information in Denmark. https://eprints.soton.ac.uk/272603/. + +Klebel, T., Cole, N. L., Tsipouri, L., Kormann, E., Karasz, I., Liarti, S., Stoy, L., Traag, V., Vignetti, S., & Ross-Hellauer, T. (2023). PathOS - D1.2 Scoping Review of Open Science Impact (Economic Impact of Open Science). Zenodo. + +Klebel, T., Traag V., (2024). Causality in science studies. + +Lindman, J., Kinnari, T. and Rossi, M. (2014). Industrial Open Data: Case Studies of Early Open Data Entrepreneurs. In: 2014 47th Hawaii International Conference on System Sciences. 2014 47th Hawaii International Conference on System Sciences. pp.739–748. . + +McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in r and Stan. 2nd ed. CRC Texts in Statistical Science. Boca Raton: Taylor; Francis, CRC Press. + +Parsons, D.D., Willis, D. and Jane, D. (2011). Benefits to the Private Sector of Open Access to Higher Education and Scholarly Research. + +Pearl, J. (2009). Causality: Models, Reasoning, and Inference (2nd ed.). Cambridge University Press. + +SPARC Europe. (2019). Using open and FAIR data to increase research efficiency \[online\]. https://sparceurope.org/wp-content/uploads/dlm_uploads/2019/04/SPARC-Europe_Brief_ODEfficiency.pdf . + +Tennant, J.P., Waldner, F., Jacques, D.C., Masuzzo, P., Collister, L.B. and Hartgerink, C.H.J. (2016). The academic, economic and societal impacts of Open Access: an evidence-based review. F1000Research, 5, p.632. + +van Assen, M.A.L.M., van Aert, R.C.M., Nuijten, M.B. and Wicherts, J.M. (2014). Why Publishing Everything Is More Effective than Selective Publishing of Statistically Significant Results. PLoS ONE, 9(1), p.e84896. + +van Vlijmen, H., Mons, A., Waalkens, A., Franke, W., Baak, A., Ruiter, G., Kirkpatrick, C., da Silva Santos, L. O. B., Meerman, B., Jellema, R., Arts, D., Kersloot, M., Knijnenburg, S., Lusher, S., Verbeeck, R., & Neefs, J.-M. (2020). The Need of Industry to Go FAIR. Data Intelligence, 2(1–2), 276–284. + +Woodward, J. (2003). CRITICAL NOTICE: *CAUSALITY* BY JUDEA PEARL. *Economics and Philosophy*, *19*(2), 321–340. + +Zeleti, F.A. and Ojo, A. (2014). Capability Matrix for Open Data. In: Camarinha-Matos, L. M. and Afsarmanesh, H., eds. Collaborative Systems for Smart Networked Environments. Berlin, Heidelberg: Springer, pp.498–509 + +1. ::: {#footnote-2} + The *do-operator* is a notation used in causal inference to denote an intervention in a system. Written as , it represents setting variable to a specific value , simulating the effect of this intervention on other variables in the system while breaking any causal connections that usually determine . This approach allows us to differentiate causation from correlation, estimate causal effects, and answer hypothetical “what-if” scenarios. Through *do-calculus*, a set of rules introduced by Pearl, interventional distributions involving the *do-operator* can be converted into observational distributions. [↑](#footnote-ref-2) + ::: From 5400d753b0c30da6348642bdaaf99424ec775267 Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Sun, 15 Dec 2024 17:11:56 +0100 Subject: [PATCH 02/13] Rename file and include in quarto --- _quarto.yml | 1 + ...ings_v2.qmd => open_data_cost_savings.qmd} | 20 ++++++++----------- 2 files changed, 9 insertions(+), 12 deletions(-) rename sections/0_causality/{Causal Indicator_ Cost Savings_v2.qmd => open_data_cost_savings.qmd} (95%) diff --git a/_quarto.yml b/_quarto.yml index c6396e7..b0cdd7c 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -41,6 +41,7 @@ website: contents: - sections/0_causality/causal_intro/article/intro-causality.qmd - sections/0_causality/open_data_citation_advantage.qmd + - sections/0_causality/open_data_cost_savings.qmd - title: Open Science contents: diff --git a/sections/0_causality/Causal Indicator_ Cost Savings_v2.qmd b/sections/0_causality/open_data_cost_savings.qmd similarity index 95% rename from sections/0_causality/Causal Indicator_ Cost Savings_v2.qmd rename to sections/0_causality/open_data_cost_savings.qmd index c513eef..cec8cec 100644 --- a/sections/0_causality/Causal Indicator_ Cost Savings_v2.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -3,7 +3,7 @@ History (please fill out in reverse chronological order, latest revision on top): | | | | | -|----|----|----|----| +|------------------|------------------|------------------|------------------| | Version | Revision date | Revision | Author | | | | | | | 1.3 | 2024-12-09 | Second draft | M. Scorrano, E. Delugas, G. Catalano (reviewer) | @@ -31,8 +31,7 @@ As discussed in the general introduction on causal inference, we use DAGs to rep Figure 1: Hypothetical structural causal model on Open Data -![Immagine che contiene testo, linea, diagramma, Carattere -Descrizione generata automaticamente](figures/1d1cf98828cbe8eab6cfc7690c05d6db29154e28.png) +![Immagine che contiene testo, linea, diagramma, Carattere Descrizione generata automaticamente](figures/1d1cf98828cbe8eab6cfc7690c05d6db29154e28.png) ## The effect of Open data on Cost Saving @@ -44,8 +43,7 @@ To properly estimate the *total* causal effect of Open Data on Cost Saving, an e Figure 2: Misleading structural causal model on Open Data, conditioning on mediator (Time Saving) -![Immagine che contiene testo, linea, schermata, diagramma -Descrizione generata automaticamente](figures/f0d9bcfde3955951768ec4951070567661000e25.png) +![Immagine che contiene testo, linea, schermata, diagramma Descrizione generata automaticamente](figures/f0d9bcfde3955951768ec4951070567661000e25.png) *Note: DAG illustrating the misleading effect of conditioning on the mediator variable, Time Saving. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship.* @@ -63,19 +61,17 @@ Another case that makes not possible the identification of the causal effect is Empirically speaking, in a model where to estimate the causal effect of Open Data on Cost savings we condition on Innovation (and not for Collaboration), it is likely to get a downward estimation of the causal effect, since both Collaboration and Open Data have a positive impact on Innovation. This conditioning opens up the non-causal pathway, Open Data → Innovation ← Collaboration → Cost Saving, which connect *Open Data* and *Cost Saving* through *Collaboration*, creating a spurious association and distorting the true effect of *Open Data* on *Cost Saving*. This is an example of *bad controls (*Angrist and Pischke 2009), a concept explained in the general introduction. Only by ignoring the collider, meaning non-conditioning on it in empirical models, we can effectively isolate the causal effect. This non-causal path is open because *Innovation* is open (because it is a collider that is conditioned on), and because *Collaboration* is open (because it is a confounder that is not conditioned on) (see Figure 3). -[]{#_Ref182214498}Figure 3: Misleading structural causal model on Open Data, not conditioning on Collaboration (confounder) and Innovation (collider) +Figure 3: Misleading structural causal model on Open Data, not conditioning on Collaboration (confounder) and Innovation (collider) -![A diagram of cost saving and cost saving -Description automatically generated](figures/26de0c31a0e5f229ff6935be961496367a3b6cd4.png) +![A diagram of cost saving and cost saving Description automatically generated](figures/26de0c31a0e5f229ff6935be961496367a3b6cd4.png) Note: DAG illustrating the misleading effect of conditioning on a collider variable, *Innovation,* and not conditioning on a confounder, *Collaboration*. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship. In addition, *Collaboration* acts as a confounder on the non-causal path *Open data* ← *Collaboration* → *Cost saving*. To identify the causal effect, we hence need to close this non-causal path by conditioning on *Collaboration*. After controlling for Collaboration, whether *Innovation* is conditioned on is then irrelevant for the identification of the causal effect. When all non-causal paths are closed, the research design is said to meet the *backdoor* criterion, a formal requirement that ensures the design blocks all non-causal paths between the treatment (*Open Data*) and the outcome (*Cost Saving*), enabling us to identify the causal effect in question (Cunningham, 2021) (Figure 4). -[]{#_Ref180136180}Figure 4: Hypothetical structural causal model on Open Data, conditioning on confounders and not on mediator and collider +Figure 4: Hypothetical structural causal model on Open Data, conditioning on confounders and not on mediator and collider -![A diagram of cost saving and standardisation -Description automatically generated](figures/0558967f8b3c0c345051326ee1dea0137c4caf1e.png) Note: DAG illustrating the total effect *Open Data* on *Cost Saving*, conditioning on confounders and not on mediator and collider. Nodes that are controlled for have a thick outline. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship. +![A diagram of cost saving and standardisation Description automatically generated](figures/0558967f8b3c0c345051326ee1dea0137c4caf1e.png) Note: DAG illustrating the total effect *Open Data* on *Cost Saving*, conditioning on confounders and not on mediator and collider. Nodes that are controlled for have a thick outline. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship. This example highlights key components of causal inference: controlling for confounders *(Data availability, quality and standardisation, User skills, Collaboration,* and *Technological Infrastructure*), not controlling for mediators *(Time saving),* and not controlling for colliders *(Innovation),* as shown in Figure 4*.* Constructing an appropriate DAG is important when aiming to draw causal conclusions. Without making assumptions explicit via a DAG, it would be unclear which variables should be controlled for and which not. Omitting important variables weakens the study's ability to draw accurate conclusions about cause and effect. Moreover, adding complexity to a DAG does not always change the variables that need to be controlled for when identifying the causal effect. In some cases, such as when adding confounders between unrelated variables, the identification of the relationship between Open data and Cost savings remains unaffected. However, if a confounder is introduced between Open data and Cost savings directly, it becomes necessary to control for it. @@ -145,4 +141,4 @@ Zeleti, F.A. and Ojo, A. (2014). Capability Matrix for Open Data. In: Camarinha- 1. ::: {#footnote-2} The *do-operator* is a notation used in causal inference to denote an intervention in a system. Written as , it represents setting variable to a specific value , simulating the effect of this intervention on other variables in the system while breaking any causal connections that usually determine . This approach allows us to differentiate causation from correlation, estimate causal effects, and answer hypothetical “what-if” scenarios. Through *do-calculus*, a set of rules introduced by Pearl, interventional distributions involving the *do-operator* can be converted into observational distributions. [↑](#footnote-ref-2) - ::: + ::: \ No newline at end of file From 49c00a12cb1c5f1debbc14b90764a9fb23782d5b Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 10:04:08 +0100 Subject: [PATCH 03/13] Added original LateX for figures --- .../figures/DAG_open_data_cost_savings.tex | 174 ++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 sections/0_causality/figures/DAG_open_data_cost_savings.tex diff --git a/sections/0_causality/figures/DAG_open_data_cost_savings.tex b/sections/0_causality/figures/DAG_open_data_cost_savings.tex new file mode 100644 index 0000000..e859f31 --- /dev/null +++ b/sections/0_causality/figures/DAG_open_data_cost_savings.tex @@ -0,0 +1,174 @@ +\documentclass{article} +\usepackage{graphicx,tikz} + +\title{Directed Acyclic Graphs (DAGs)} +\author{} +\date{December 2024} + +\begin{document} + +\maketitle % Displays the title, author, and date +\clearpage +% FIRST DAG +\begin{figure}[h!] +\centering +\begin{tikzpicture} + +% Nodes +\node (v0) at (-3, -1.5) {Open Data}; +\node (v1) at (-3,-4) {Cost Saving}; +\node (v2) at (2, -1.5) {Time Saving}; +\node (v3) at (2,-3) {User Skills}; +\node (v4) at (-8, -3) {Innovation}; +\node (v5) at (-8, -1.5) {Collaboration}; +\node (v6) at (2, 0) {Technological Infrastructure}; +\node (v7) at (-5, 0) {Data Availability, Quality and Standardisation}; + +% Arrows +\draw[->] (v7) -- (v0); +\draw[->] (v6) -- (v0); +\draw[->] (v5) -- (v0); +\draw[->] (v0) -- (v1); +\draw[->] (v0) -- (v2); +\draw[->] (v5) -- (v4); +\draw[->] (v0) -- (v4); +\draw[->] (v5) -- (v1); +\draw[->] (v2) -- (v1); +\draw[->] (v3) -- (v0); +\draw[->] (v3) -- (v2); +\draw[->] (v6) -- (v2); +\draw[->] (v7) -- (v2); + +\end{tikzpicture} +\caption{First DAG} +\label{fig:first_dag} +\end{figure} + +\clearpage + +% SECOND DAG +\begin{figure}[h!] +\centering +\begin{tikzpicture}[ + box/.style={rectangle, fill=gray!10, minimum height=0.5cm, minimum width=1.5cm}, + highlight/.style={rectangle, draw, fill=red!10, minimum height=0.5cm, minimum width=1.5cm} +] + +% Nodes +\node (v0) at (-3, -1.5) {Open Data}; +\node (v1) at (-3,-4) {Cost Saving}; +\node[highlight] (v2) at (2, -1.5) {Time Saving}; +\node[box] (v3) at (2,-3) {User Skills}; +\node[box] (v4) at (-8, -3) {Innovation}; +\node[box] (v5) at (-8, -1.5) {Collaboration}; +\node[box] (v6) at (2, 0) {Technological Infrastructure}; +\node[box] (v7) at (-5, 0) {Data Availability, Quality and Standardisation}; + +% Arrows +\draw[dashed,->] (v7) -- (v0); +\draw[dashed,->] (v6) -- (v0); +\draw[dashed,->] (v5) -- (v0); +\draw[->] (v0) -- (v1); +\draw[dashed,->] (v0) -- (v2); +\draw[dashed,->] (v5) -- (v4); +\draw[dashed,->] (v0) -- (v4); +\draw[dashed,->] (v5) -- (v1); +\draw[dashed,->] (v2) -- (v1); +\draw[dashed,->] (v3) -- (v0); +\draw[dashed,->] (v3) -- (v2); +\draw[dashed,->] (v6) -- (v2); +\draw[dashed,->] (v7) -- (v2); + +\end{tikzpicture} +\caption{Second DAG} +\label{fig:second_dag} +\end{figure} + +\clearpage + +% THIRD DAG +\begin{figure}[h!] +\centering +\begin{tikzpicture}[ + node distance=2cm and 3cm, + every node/.style={align=center}, + box/.style={rectangle, fill=gray!10, minimum height=0.5cm, minimum width=1.5cm}, + highlightGreen/.style={rectangle, draw, fill=green!10, minimum height=0.5cm, minimum width=1.5cm}, + borderHighlight/.style={rectangle, fill=green!10, minimum height=0.5cm, minimum width=1.5cm} +] + +% Nodes +\node (v0) at (-3, -1.5) {Open Data}; +\node (v1) at (-3,-4) {Cost Saving}; +\node[box] (v2) at (2, -1.5) {Time Saving}; +\node[box] (v3) at (2,-3) {User Skills}; +\node[highlightGreen] (v4) at (-8, -3) {Innovation}; +\node[borderHighlight] (v5) at (-8, -1.5) {Collaboration}; +\node[box] (v6) at (2, 0) {Technological Infrastructure}; +\node[box] (v7) at (-5, 0) {Data Availability, Quality and Standardisation}; + +% Arrows +\draw[dashed,->] (v7) -- (v0); +\draw[dashed,->] (v6) -- (v0); +\draw[->] (v5) -- (v0); +\draw[->] (v0) -- (v1); +\draw[dashed,->] (v0) -- (v2); +\draw[->] (v5) -- (v4); +\draw[->] (v0) -- (v4); +\draw[->] (v5) -- (v1); +\draw[dashed,->] (v2) -- (v1); +\draw[dashed,->] (v3) -- (v0); +\draw[dashed,->] (v3) -- (v2); +\draw[dashed,->] (v6) -- (v2); +\draw[dashed,->] (v7) -- (v2); + +\end{tikzpicture} +\caption{Third DAG} +\label{fig:third_dag} +\end{figure} + +\clearpage + +% FOURTH DAG +\begin{figure}[h!] +\centering +\begin{tikzpicture}[ + node distance=2cm and 3cm, + every node/.style={align=center}, + box/.style={rectangle, fill=gray!10, minimum height=0.5cm, minimum width=1.5cm}, + borderRed/.style={rectangle, draw, fill=red!10, minimum height=0.5cm, minimum width=1.5cm}, + highlightGreen/.style={rectangle, fill=green!10, minimum height=0.5cm, minimum width=1.5cm}, + highlightRed/.style={rectangle, fill=red!10, minimum height=0.5cm, minimum width=1.5cm} +] + +% Nodes +\node (v0) at (-3, -1.5) {Open Data}; +\node (v1) at (-3,-4) {Cost Saving}; +\node[highlightGreen] (v2) at (2, -1.5) {Time Saving}; +\node[borderRed] (v3) at (2,-3) {User Skills}; +\node[highlightRed] (v4) at (-8, -3) {Innovation}; +\node[borderRed] (v5) at (-8, -1.5) {Collaboration}; +\node[borderRed] (v6) at (2, 0) {Technological Infrastructure}; +\node[borderRed] (v7) at (-5, 0) {Data Availability, Quality and Standardisation}; + +% Arrows +\draw[dashed,->] (v7) -- (v0); +\draw[dashed,->] (v6) -- (v0); +\draw[dashed,->] (v5) -- (v0); +\draw[->] (v0) -- (v1); +\draw[->] (v0) -- (v2); +\draw[dashed,->] (v5) -- (v4); +\draw[dashed,->] (v0) -- (v4); +\draw[dashed,->] (v5) -- (v1); +\draw[->] (v2) -- (v1); +\draw[dashed,->] (v3) -- (v0); +\draw[dashed,->] (v3) -- (v2); +\draw[dashed,->] (v6) -- (v2); +\draw[dashed,->] (v7) -- (v2); + +\end{tikzpicture} +\caption{Fourth DAG} +\label{fig:fourth_dag} +\end{figure} + +\end{document} From 8155898226598902ba1a011e1336fa08b27729d8 Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 11:10:22 +0100 Subject: [PATCH 04/13] adjust styling, standardize and make more readable --- .../figures/DAG_open_data_cost_savings.tex | 249 ++++++++---------- sections/0_causality/figures/ODCA_model.tex | 2 +- sections/0_causality/figures/styling.tex | 3 +- 3 files changed, 108 insertions(+), 146 deletions(-) diff --git a/sections/0_causality/figures/DAG_open_data_cost_savings.tex b/sections/0_causality/figures/DAG_open_data_cost_savings.tex index e859f31..e8b1052 100644 --- a/sections/0_causality/figures/DAG_open_data_cost_savings.tex +++ b/sections/0_causality/figures/DAG_open_data_cost_savings.tex @@ -1,174 +1,135 @@ -\documentclass{article} -\usepackage{graphicx,tikz} +\documentclass[tikz]{standalone} -\title{Directed Acyclic Graphs (DAGs)} -\author{} -\date{December 2024} +\input{figure.tex} +\input{styling.tex} + +\usetikzlibrary{calc} \begin{document} -\maketitle % Displays the title, author, and date -\clearpage +\tikzset{every node/.style={n square}} +\tikzset{every edge/.style={e}} +\tikzstyle{grayed}=[fill=gray!10] +\tikzset{x=1cm,y=1.5cm} + % FIRST DAG -\begin{figure}[h!] -\centering \begin{tikzpicture} % Nodes -\node (v0) at (-3, -1.5) {Open Data}; -\node (v1) at (-3,-4) {Cost Saving}; -\node (v2) at (2, -1.5) {Time Saving}; -\node (v3) at (2,-3) {User Skills}; -\node (v4) at (-8, -3) {Innovation}; -\node (v5) at (-8, -1.5) {Collaboration}; -\node (v6) at (2, 0) {Technological Infrastructure}; -\node (v7) at (-5, 0) {Data Availability, Quality and Standardisation}; - -% Arrows -\draw[->] (v7) -- (v0); -\draw[->] (v6) -- (v0); -\draw[->] (v5) -- (v0); -\draw[->] (v0) -- (v1); -\draw[->] (v0) -- (v2); -\draw[->] (v5) -- (v4); -\draw[->] (v0) -- (v4); -\draw[->] (v5) -- (v1); -\draw[->] (v2) -- (v1); -\draw[->] (v3) -- (v0); -\draw[->] (v3) -- (v2); -\draw[->] (v6) -- (v2); -\draw[->] (v7) -- (v2); +\node (open data) at (-3, -1.5) {Open Data}; +\node (cost saving) at (-3,-4) {Cost Saving}; +\node (time saving) at (2, -1.5) {Time Saving}; +\node (user skills) at (2,-3) {User Skills}; +\node (innovation) at (-8, -3) {Innovation}; +\node (collaboration) at (-8, -1.5) {Collaboration}; +\node (infrastructure) at (2, 0) {Technological Infrastructure}; +\node (quality) at (-5, 0) {Data Availability, Quality and Standardisation}; + +% Edges +\draw (quality) edge (open data) + (infrastructure) edge (open data) + (collaboration) edge (open data) + (open data) edge (cost saving) + (open data) edge (time saving) + (collaboration) edge (innovation) + (open data) edge (innovation) + (collaboration) edge (cost saving) + (time saving) edge (cost saving) + (user skills) edge (open data) + (user skills) edge (time saving) + (infrastructure) edge (time saving) + (quality) edge (time saving); \end{tikzpicture} -\caption{First DAG} -\label{fig:first_dag} -\end{figure} - -\clearpage % SECOND DAG -\begin{figure}[h!] -\centering -\begin{tikzpicture}[ - box/.style={rectangle, fill=gray!10, minimum height=0.5cm, minimum width=1.5cm}, - highlight/.style={rectangle, draw, fill=red!10, minimum height=0.5cm, minimum width=1.5cm} -] +\begin{tikzpicture} % Nodes -\node (v0) at (-3, -1.5) {Open Data}; -\node (v1) at (-3,-4) {Cost Saving}; -\node[highlight] (v2) at (2, -1.5) {Time Saving}; -\node[box] (v3) at (2,-3) {User Skills}; -\node[box] (v4) at (-8, -3) {Innovation}; -\node[box] (v5) at (-8, -1.5) {Collaboration}; -\node[box] (v6) at (2, 0) {Technological Infrastructure}; -\node[box] (v7) at (-5, 0) {Data Availability, Quality and Standardisation}; - -% Arrows -\draw[dashed,->] (v7) -- (v0); -\draw[dashed,->] (v6) -- (v0); -\draw[dashed,->] (v5) -- (v0); -\draw[->] (v0) -- (v1); -\draw[dashed,->] (v0) -- (v2); -\draw[dashed,->] (v5) -- (v4); -\draw[dashed,->] (v0) -- (v4); -\draw[dashed,->] (v5) -- (v1); -\draw[dashed,->] (v2) -- (v1); -\draw[dashed,->] (v3) -- (v0); -\draw[dashed,->] (v3) -- (v2); -\draw[dashed,->] (v6) -- (v2); -\draw[dashed,->] (v7) -- (v2); +\node (open data) at (-3, -1.5) {Open Data}; +\node (cost saving) at (-3,-4) {Cost Saving}; +\node[closed] (time saving) at (2, -1.5) {Time Saving}; +\node[grayed] (user skills) at (2,-3) {User Skills}; +\node[grayed] (innovation) at (-8, -3) {Innovation}; +\node[grayed] (collaboration) at (-8, -1.5) {Collaboration}; +\node[grayed] (infrastructure) at (2, 0) {Technological Infrastructure}; +\node[grayed] (quality) at (-5, 0) {Data Availability, Quality and Standardisation}; + +% Edges +\draw (quality) edge[e-gray] (open data) + (infrastructure) edge[e-gray] (open data) + (collaboration) edge[e-gray] (open data) + (open data) edge (cost saving) + (open data) edge[e-gray] (time saving) + (collaboration) edge[e-gray] (innovation) + (open data) edge[e-gray] (innovation) + (collaboration) edge[e-gray] (cost saving) + (time saving) edge[e-gray] (cost saving) + (user skills) edge[e-gray] (open data) + (user skills) edge[e-gray] (time saving) + (infrastructure) edge[e-gray] (time saving) + (quality) edge[e-gray] (time saving); \end{tikzpicture} -\caption{Second DAG} -\label{fig:second_dag} -\end{figure} - -\clearpage % THIRD DAG -\begin{figure}[h!] -\centering -\begin{tikzpicture}[ - node distance=2cm and 3cm, - every node/.style={align=center}, - box/.style={rectangle, fill=gray!10, minimum height=0.5cm, minimum width=1.5cm}, - highlightGreen/.style={rectangle, draw, fill=green!10, minimum height=0.5cm, minimum width=1.5cm}, - borderHighlight/.style={rectangle, fill=green!10, minimum height=0.5cm, minimum width=1.5cm} -] +\begin{tikzpicture} % Nodes -\node (v0) at (-3, -1.5) {Open Data}; -\node (v1) at (-3,-4) {Cost Saving}; -\node[box] (v2) at (2, -1.5) {Time Saving}; -\node[box] (v3) at (2,-3) {User Skills}; -\node[highlightGreen] (v4) at (-8, -3) {Innovation}; -\node[borderHighlight] (v5) at (-8, -1.5) {Collaboration}; -\node[box] (v6) at (2, 0) {Technological Infrastructure}; -\node[box] (v7) at (-5, 0) {Data Availability, Quality and Standardisation}; - -% Arrows -\draw[dashed,->] (v7) -- (v0); -\draw[dashed,->] (v6) -- (v0); -\draw[->] (v5) -- (v0); -\draw[->] (v0) -- (v1); -\draw[dashed,->] (v0) -- (v2); -\draw[->] (v5) -- (v4); -\draw[->] (v0) -- (v4); -\draw[->] (v5) -- (v1); -\draw[dashed,->] (v2) -- (v1); -\draw[dashed,->] (v3) -- (v0); -\draw[dashed,->] (v3) -- (v2); -\draw[dashed,->] (v6) -- (v2); -\draw[dashed,->] (v7) -- (v2); +\node (open data) at (-3, -1.5) {Open Data}; +\node (cost saving) at (-3,-4) {Cost Saving}; +\node[grayed] (time saving) at (2, -1.5) {Time Saving}; +\node[grayed] (user skills) at (2,-3) {User Skills}; +\node[open,conditioned] (innovation) at (-8, -3) {Innovation}; +\node[open] (collaboration) at (-8, -1.5) {Collaboration}; +\node[grayed] (infrastructure) at (2, 0) {Technological Infrastructure}; +\node[grayed] (quality) at (-5, 0) {Data Availability, Quality and Standardisation}; + +% Edges +\draw (quality) edge[e-gray] (open data) + (infrastructure) edge[e-gray] (open data) + (collaboration) edge (open data) + (open data) edge (cost saving) + (open data) edge[e-gray] (time saving) + (collaboration) edge (innovation) + (open data) edge (innovation) + (collaboration) edge (cost saving) + (time saving) edge[e-gray] (cost saving) + (user skills) edge[e-gray] (open data) + (user skills) edge[e-gray] (time saving) + (infrastructure) edge[e-gray] (time saving) + (quality) edge[e-gray] (time saving); \end{tikzpicture} -\caption{Third DAG} -\label{fig:third_dag} -\end{figure} - -\clearpage % FOURTH DAG -\begin{figure}[h!] -\centering -\begin{tikzpicture}[ - node distance=2cm and 3cm, - every node/.style={align=center}, - box/.style={rectangle, fill=gray!10, minimum height=0.5cm, minimum width=1.5cm}, - borderRed/.style={rectangle, draw, fill=red!10, minimum height=0.5cm, minimum width=1.5cm}, - highlightGreen/.style={rectangle, fill=green!10, minimum height=0.5cm, minimum width=1.5cm}, - highlightRed/.style={rectangle, fill=red!10, minimum height=0.5cm, minimum width=1.5cm} -] +\begin{tikzpicture} % Nodes -\node (v0) at (-3, -1.5) {Open Data}; -\node (v1) at (-3,-4) {Cost Saving}; -\node[highlightGreen] (v2) at (2, -1.5) {Time Saving}; -\node[borderRed] (v3) at (2,-3) {User Skills}; -\node[highlightRed] (v4) at (-8, -3) {Innovation}; -\node[borderRed] (v5) at (-8, -1.5) {Collaboration}; -\node[borderRed] (v6) at (2, 0) {Technological Infrastructure}; -\node[borderRed] (v7) at (-5, 0) {Data Availability, Quality and Standardisation}; - -% Arrows -\draw[dashed,->] (v7) -- (v0); -\draw[dashed,->] (v6) -- (v0); -\draw[dashed,->] (v5) -- (v0); -\draw[->] (v0) -- (v1); -\draw[->] (v0) -- (v2); -\draw[dashed,->] (v5) -- (v4); -\draw[dashed,->] (v0) -- (v4); -\draw[dashed,->] (v5) -- (v1); -\draw[->] (v2) -- (v1); -\draw[dashed,->] (v3) -- (v0); -\draw[dashed,->] (v3) -- (v2); -\draw[dashed,->] (v6) -- (v2); -\draw[dashed,->] (v7) -- (v2); +\node (open data) at (-3, -1.5) {Open Data}; +\node (cost saving) at (-3,-4) {Cost Saving}; +\node[open] (time saving) at (2, -1.5) {Time Saving}; +\node[closed,conditioned] (user skills) at (2,-3) {User Skills}; +\node[closed] (innovation) at (-8, -3) {Innovation}; +\node[closed,conditioned] (collaboration) at (-8, -1.5) {Collaboration}; +\node[closed,conditioned] (infrastructure) at (2, 0) {Technological Infrastructure}; +\node[closed,conditioned] (quality) at (-5, 0) {Data Availability, Quality and Standardisation}; + +% Edges +\draw (quality) edge[e-gray] (open data) + (infrastructure) edge[e-gray] (open data) + (collaboration) edge[e-gray] (open data) + (open data) edge (cost saving) + (open data) edge (time saving) + (collaboration) edge[e-gray] (innovation) + (open data) edge[e-gray] (innovation) + (collaboration) edge[e-gray] (cost saving) + (time saving) edge (cost saving) + (user skills) edge[e-gray] (open data) + (user skills) edge[e-gray] (time saving) + (infrastructure) edge[e-gray] (time saving) + (quality) edge[e-gray] (time saving); \end{tikzpicture} -\caption{Fourth DAG} -\label{fig:fourth_dag} -\end{figure} \end{document} diff --git a/sections/0_causality/figures/ODCA_model.tex b/sections/0_causality/figures/ODCA_model.tex index ced6407..12e57bb 100644 --- a/sections/0_causality/figures/ODCA_model.tex +++ b/sections/0_causality/figures/ODCA_model.tex @@ -8,7 +8,7 @@ \begin{document} \begin{tikzpicture}[x=100,y=100, every node/.style={n square}, - every edge/.style={e}] + every edge/.style={e-gray}] \node (OD policy) at (-2,0.5) {OD policy}; \node (open data) at (-1.1,0.5) {open data}; diff --git a/sections/0_causality/figures/styling.tex b/sections/0_causality/figures/styling.tex index b3f2bf5..a7562ec 100644 --- a/sections/0_causality/figures/styling.tex +++ b/sections/0_causality/figures/styling.tex @@ -9,7 +9,8 @@ \definecolor{Set1-A}{RGB}{228, 26, 28} \definecolor{Set1-B}{RGB}{ 55, 126, 184} -\tikzstyle{e}=[rounded corners,very thick,draw=white,double=gray,arrows={-latex[gray]}] +\tikzstyle{e}=[rounded corners,very thick,draw=white,double=black,arrows={-latex[black]}] +\tikzstyle{e-gray}=[e,draw=white,double=lightgray,arrows={-latex[lightgray]}] \tikzset{every label/.style={rectangle,fill=none,draw=none, label distance=4pt}} \tikzstyle{n}=[circle,fill=white,draw=white,line width=2pt,outer sep=0pt,inner sep=0pt, minimum size=4ex] From 645388d015895b99a9d33d3c9059af31bdab5fc2 Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 11:45:44 +0100 Subject: [PATCH 05/13] Added figures --- .../figures/DAG_open_data_cost_savings-0.png | Bin 0 -> 34024 bytes .../figures/DAG_open_data_cost_savings-1.png | Bin 0 -> 27453 bytes .../figures/DAG_open_data_cost_savings-2.png | Bin 0 -> 27685 bytes .../figures/DAG_open_data_cost_savings-3.png | Bin 0 -> 27519 bytes .../0_causality/open_data_cost_savings.qmd | 30 ++++++------------ 5 files changed, 9 insertions(+), 21 deletions(-) create mode 100644 sections/0_causality/figures/DAG_open_data_cost_savings-0.png create mode 100644 sections/0_causality/figures/DAG_open_data_cost_savings-1.png create mode 100644 sections/0_causality/figures/DAG_open_data_cost_savings-2.png create mode 100644 sections/0_causality/figures/DAG_open_data_cost_savings-3.png diff --git a/sections/0_causality/figures/DAG_open_data_cost_savings-0.png b/sections/0_causality/figures/DAG_open_data_cost_savings-0.png new file mode 100644 index 0000000000000000000000000000000000000000..98227bd7be08c535249879e63f61722e31f407bc GIT binary patch literal 34024 zcmc$`XH=6-)Gi!FL6M>Yf>aAgFM`sGK8Q${E(8cDNKfb;qM{%opi}|rO-ksYCmxcAkdrlAP^CN-YkPa?jj%%&Kv}i{ssauI(@Iv zmIWHlzf@Cx0y+(j_+1bDciHKgkt+yvPT=2P*N-c}_#g^56%8edWvc6x46NLs01Pl1 zNae|6JufWb82&>Remb=)x7RVmmmSvKnnw@TSZXt@e|gJ!@>%8we)#Xl(f^D!B$tyb zY43f6b7c~Qny>S{tL?vTf2B((m-j>Lyu%p%?Q5gQOW?}^;O_FrzhGa!KYaWyC|0cH z=+@imgVx`x-4U48-@wmN%@lbdetR-WjD#H>9rfzbrwQg=l66o=~HOz*_1z-FqLWkWnLW=uQtve*K(vhw|EZ z5;Fgg5t<)2q7l&EvQ30*LZY@7`y0OF!G%QOtR3_NBYa-YvLeNVQNvC0RBp12*#9 z7qFAGFd9nV6jl{R?~Iy+#K68F>hPKKV*^jERg_KLn7h($1j&TUswg)Go%<9R6jKqB zVV9c~>k}mmDY`&IRvhv!$bS}{byu0+bf-IlcH;qfPI%PbO~S`1w~vVmLZQ0|YnN$N#Ta+ z0wb=jc(JsY;ZivKHNlFdfe77nEn1Z(1zM0u{H(7_+f?pk_Tx7N*%bY`mp43J?~rW4&+V1i7WaDIF@tD&ZwC+T1?4PljB8y9rzyIgL! zibnux`EaBal!xCpx&M88$jLZT;%D(=V`Z`9;iS$qS_@pWpu^R6da_xiw1#7$0Y^93 z{H6#)mv@)MIS^>=o8`#Wr6W^v_-#flV>^DBRL=z@`#R%iuGZxnmi=<$#I7tVWbLv=jc|{C@#O*#|oS)gcb-JW2LF@GK6io zMO!kA@5*;X&6|Z58`@7jjvIt9G zmzyOEnGCG?(3}&WTk;Vh=!erF=2D$=#cK}huI&iu9Xwk*8w zhj9C_j=kcv8NBBeD~?g2@Sbp(H!6=Q=Fo`HpO};;j`i5^S6bclQ`qTPX1 z)fVjG@b-yXZ+OCO z`Fqf{Buw7-qBUe{chT{&*7BxU&dSi*=~KN^bQZ*>6h-aa?56p|(xNr`u3?3PX}Zk` z;&pDJN5!3A{?zRsowGdS^CVsvuiRR(@d{HMOdgYg=KDuDtQRD4f-3|}3$JaX;y+@| zZL^Xx182}`Ue>Mt)0(P&+^HP_SJs*2u$!ZM*24s|ya8utl2*8 zg=cxbZV6MSqmWkosIh9FyyVmR+zR?#N&5U!&*be;T{vx=SQ6o{&CLn^$RTcy-Ed9J z514ZCsngw@KEC2A3${Y(PiNohmF#zMg|jHP-W#fPfq20&suLj*>*R$l_HfHJv2g)0 zZQr3iebf@f-Pif(`)30!j7&0#!fcdPH&G$6XxV+svomQNZ-~tEd3JB|_^#&#?CI~C z3AG_({8Qz?cZA&%|DT&|Q=2c$z=NRp;)(6Y7bvDL)B~v?I!x^m?Y7ujHMV= z{;?w%W1-5SW#Jg@vkHAYGfAO3SR-j7ds~TRY707Pg*2@HaLyxJ-`$Mg_oyIjDZX`2 zbAqG(VDXs;iMwfTXunQ-s@`?rglR^?3Jn3cOWM^Z3&WvowVHxb&WWz(>)#Fp=442*RynZBUDGsp^tYVq`Z{pi57v?T|;OX zc}_@orrRrqBhR*Q=^r>Z8e|!R?kAnCsYBZraUt~`P^)+weK5h-!B;c8v*%OLl!=JXu(bBF0b>TZYcbu1B*s77VwQ)&?rh|5JX9}QTO<#ci?x}>WeO9-6umqj%EaOW z?hw$OK1I-#yW0qRwDLMfk11QxO!U>U@x3Nn2ERE0qXl2~xvK#+qqiGp8hTt`BaWW1 zM6Og7xVl9nNNs&ar$wkAAO{yzv7kO0tiRvA9pD^^Jt$o4@o=BiU~zpKZ~WX8HCHK^ zEpA9c$E(Xb@C*oWA8#VwGq4nlXBqp|v_ULek>~o0T%O5dc5D3{eAl9057I(fdseIM zNUa5Czh;PVR|uf9AIB)SdZfon+dD75#(PD{@S&JJVm}5+50TAzLia1BO>EL?Gan|V zY+8n)gjgGrNZ2DU>Kst{XF_<>M8TF^g!3i%HOHo$dc7^tdvc^{fA{e4Qg;q1X^unu zMpCx$LgTt;%(_7mDr%@pxLZSJYRK`|)W~YB_fFlO*OlK1e*ZXDRO9x$EK3#B=K?co z4-oO`S01XVvC-4u)Zx7qvC^v2oY`?yT2DQKo4#SEI|jrkoj;Iuc!r>SKCX>FkTEs2 zJ34+IQJGk{7in}TJ~x*h%6=Pck3fd3wIKH7S>5$*Pg@ zQ)U;-_PCUU*?;m%m+sSS(tk-D7ehni;G2G=p>Tq7DEQd9mnv)j4kCqJw!-OPI9dREBA7B$P8MeN{6EMd^_RA1dXXpT; zk!JLW%8F3eYF*bkYbc>FXR~EoQK9`0=DxI=l(F0bt80ka)bBo{!?Lzw*tb+^Vd7N! znA*%CpI-&8#B^zy2fY6^WBY5p$8~(pzQzzFtrdMZ!T1Q^!FU(7FxB;H3E2uZAD*Z4 z`M$E7=RfuP9*g*Ug`{lKhBI&J>$sAx3YX6(Ym;mTetnf2@79NDXQoy*?3PR`%23uH zs6X9fk~33FVPnvGwg2(Jel7<36Sgp0N$4pUL+um{S6Hy^{)mQ0D*8kWnW2tJ1b!nQ zt)o>RAI#$%6D{L0-<%GsPNr~ix14%yejtj%!3k}{O09Yvo?gDisusgOEp9Zl6Eb5l zsKmTKbUrUknX4pBeGCbr4tSh1{kZl9`VJ7lO-%{r!M#6ED(>CkMo)*@pDa*lA@_ku z&dhVR=E6)v*#oc3vVg(Ygv{a9Jfl|y1mZBFXWKL&%khTPhP{|0u#3pgHO?O%ff z_BaX;2W3!;1z2aspJv!6ZYP^|G}6z}Eq1=*`-mt4=7jXr4KPbgY(>N9s99!>E=Sfj zW590JYi%+7YT=of1Tl*85(b4D-(q1&lR2Z{@St-8n-^0oHHB;exz(c92A;p%Lkdi) zZeDbbN=XxOs9OHA+_$-N7rQ3-P4a2c$qNaGziyN;ET1EsH+dYpr|?yJy>unNvP5~# z;RR&rhx&IjfuR;rWKP<%)e0P=$PBF=5^KGAA8{`>?AaRr4 zd%~KPxVl8&C8YK8`axITp1)Y4<;b*>g`y8-i^yxwbfyz+M+YtiTe}2*R3~BSwVqQt zE~xX}e5A*uRqa@93TCu$xjzh10z*jzC7MXm?jz@K77fmj*Bg+Rxnf2Hs@^CmWl*GAAM*ny3sdX794ftxE z!h~ya>bia5ua&Kn>L+>87Va!*)kQl2`D;l_r?YM=Up-$*7s)BYTVV#s^~BK&+~&I( zZ-ou`keqc*u$95(IXi293rG`6sC5=cJ+pu}S_BBkV%GMK@SsfVOV-aM%^*nIR;X9R zQe*xcA>E$F3d;or!MR9j}M!EHSOl(&e5ACD!`@QOvy~qh0AS znKUnU&w(egGp{t)6)KihM-1^XCT-1fHNli%2g9~6p^*cXB9H%^JEw;$Tm zcAgkVdZRKz2k>zcGn^sHcv({UHvY z*2Yf$gd(Qp;tric7_1L>xkQtmONqu{AB@=TzAXqpGOgIe5mx&I@H2b*c*=;LYlj}b zf_gW8_I^lNYw;>Sh)2@2grbC^6cX5yhEB0sN52e63Hs;rR+EI_&dEiGwe6=?V6r%k zr7@pYSW@huT{~=xzeT0Hi|Xp`e4ShcK)>q~<|>sMgqe2w$oP zJ*SrsujJ&c%-Bj#kJz{su;JRHF6ir%2KzCwEhpOH4{oV0{0<4)Sj+KlWx`gZE>R7&nm%9&L`)2Lw!nkV@3lR89 zm%V%1XrIw;Q4{R<(}>(s&l;)v-7H)!uT0SgTv8D> z^{4Ae36(`Fdo)|fNPex!huI!rDwaW=pO4O)FiL{72m#hNmhRgYNW= zWc6Wj%~7m`cU%_))y}@_K6szmgz1Fzy?Uq*kVkXBO3~KB$^voQ2Yt`k@4?Xp5jPH> z5cy-IiBw+^@xJRY6=o~rl}sJeemb%{j1(98P)0ic#9i_nUbFVMjC0|umAus^l5bAv zHW>bI;GLki@8_<2j6vfzeT%r<%?QbMQxbvZHfp)0)CGaCxha2Uj7NIC3aJB8>1WP= zkE4&cunHt?`5QO4U5Cb&415ZRd&7NV;TD8-AmBb4LG|3*V|(%T{KfotuW~w;q9V?O} z8mll}(w6lHarA;|U1weq8twbQa3W*WvzYR?uakbQ|Ng#nADZ+aApEeW2uRs@NjEKW zO9$s;Pe$Wr_V4=dOn^J7(d?4axPr0O<$^uTSgMWZ*iYpThP+cU8a*&8Nd78uS0PLn z+WG6>X!EG<##-&_b|)PLR?~#Gd&911w8;f=}QwhS)zWDNwk~w7LeK zTs5uO#zn04p){-Agur9Xn*-hP)AH!)y^*dspCiS&=*M}6mI!< z8egET)60=|{8F^-uKsBBa?3!8d6zVJMRAPaYw2t4y+zl4hroB@@DB@NetBTy_doru zP22nvxaT$6OBP~?V0+nw zKz0<-^ju-hhXk~rw|17V8iZjAe~c2_$^Yh)SD#2)LOO*hcN<2m_La`Tkg*^DM0_-*@@crUcRb5jHz5b7{}upZ2bcc)zE1oW|5xS#X`(cI1I5xP6th z&VVdk=Hfk<7I^foU0m*A(fpUaYZF{v98VTnr9bUo2TdO~CO%&*zLnhU?oj{W8*?MM zdbhUH!z$rCnNXdZGBv{Uc*J_8sqUuPdC*Q4qUPKq#!DI#w*O!yoqS-9|;B_(T}4JVsi^D@Q;FAh?%HgchldMQzd)IG;-0mX)Ou3^c>2!T$ zO5IdGzvf9OqxDJ)rM@?%KV`l4TG$ba$YFL$D#B^4C=8|aL-4iQ4No@gO&4Lk zQ#n(4^-7UD);r2XC{6E()wP|83!tV~w)ruBs1AHBFZC+2L(XH+NcqrTg1RAd)Yz7b zf3q#W+h2mx=2P*xN3mH^tSXU3an~R(;W+{2QQ^|~0<|gX3nGszhJ(O2C{X(oBB{JB z;-F`vN3&BWblSaF$*&q@bnPzgpM=7O(|8t*?23f_?F4JrJqkA9v+wLCnVS zuqUkUy+-{CM0@zr`g&~$_2ii7Q3U*O{Rn`E4%5=$8X0x`Nalw?0lL5 zKUN^c@8lJYeW=JdW|G0c32}|53b-Gh)h@^fo`RAtsl&I4QoI!)V&+*xRE#>|Jmb%g+H#g&iGMS>U`a zaQa*W8-J2pM+#$)&rm2$Uw!OK%<9)9<+oHpPsqMRMiufM%|yA3swg~-BCRhE>^KF0 z0ufo`lD3#;j5d1=JIKVyIDtZe!>uYALTOO5oVC8FV9CP$!m&KcttRPVuJV2U9e@~4 z2&_`=l>{Xt4aWC%Wafa))`{cx)u1C#P^gE`kd_aouZ~NNpRkLF@Vj&V&1ea7C~i;b z?#;&3pZ(Z!9WR#iO!s|QK+WGRV}g>zBSlA5Uh$nSwhfg^_HJgp>yt1T+;-xYPE}_W zfh>wbYnD^NVc*`xl;(#~qo}_;83EmDA6nUq*#A*9BWC(*#$8XruOfE$9R;5psLdHE zZY7rNuD;WLqkz+Uv~aBNw!<6FMskpC?hKX1w7Koh!jyyH0zIwMp8K$r*qbz+4JY<4 z8(L5VAMf0aqH@uxlxIca2JipQgO|?Hc~So}%95+IUM2W`EL`{d2RCbv%vVBQPUQl8 zbd2c$jixajRYt~KcFmcrpVWSR2K)#ops+SID(jDA!Gp!Yo8&Y)A8u34463wG`Mjer z_i#XHu!W;Na=%jXZ#!H@(_k8@bsvp@RDjoh>et^kDJjhJOXRGO+Qi(|%D!xoF{!q^ zNTg5&F>x5GZBH+m+Hp3O+H7n?ca~ZO8p)sCaR8Y|GIw8%c-O7x%mkmlrBBJ;U9C!c zfwsXCQT~87V3MmaBF;FTLLtTd+k9n8@wv!LETC^1`A6XRNC8gQc*&(nk;?--_8`UK z%+FvCWbT>7lpOpg*BV;`Z;I-(nsM*mHInh-VSDAsY(;JEMM=(kfaQv0a1=xeF}FK? zYn!$f`%BAn2jd069pQyKn!WXWK;(AaR^xeh{_0>~z32$6A00xrXukIUpc-X=(d_(} zH!I(Lr5fOwGhp~DMINzZsRO zikCG1!812Za|TYxcK!ul03iDr*y?u%UP_2@Rx2WJB!2=BF?fId?|jR&R^wRZzv{)M z|K&6}R1|7@Cgxr369i31^SPc!H!Fp%i9fn-LH0)Z`3XWEz)aIVgE5y_ZiN(|Yq-mE zy2k@xg!Ip2Vpsq6YX$mU2Cy#e7J~O#zYLz!VS20CH_8B3eTK~e7&*{y_jwE9@Gtg= zE*frs1?a~IutNKnm0qfaDeRMY$y6q*aiZd1k`n-u$9U){G6C7Z&IBgaQSbg%)cMVm zo0S%~{;Jk7^$6_ALl=AqQ1|?dI&pwH`{W#KCIDb= z+AWUX&M;p9QwOj?Nn-#_Jwp=#)B!*r08fvcVYL8n6dlbLzj@}w{muw zhZnz6MLV-FoiQliQNJ4^>dX}Uj?(e)-xknW%SE6i0%#F3>mJoJag-Vq8%q0(EY!r4?vS*yyQ->D0J@WUlGqX@9UlQRu*#@z*E;{W8>9YxJI{5mN60^97yF7X}Qf94}c844`|amm1~6 z%KLze#6G@R$$MsBJ&z76Hvx=Y`DSJAf7qtVVWoDOIx7y)O!V3MT51~%qfvgWd*V0^ z=nPl^;{JanH*KUdm&$u8ekVFa@qS0uf9B?uJSe&RuHla;0F$s}h;<(0dihUBmB#tM z=IE6@C~4DCA9mGh_9`$<-Prg`hucKK5 zu(U@bZ!Q4_B>()xu>&hir<@fmuO~2PP^{zg@H5NVU{@BVvJ3y&_`=@sD8A}k+GiHP zY5Xyp^lKPoN{p_iF z;3@8k@tPR+#^hj`D5Q_H?n)Y0S4UNrVN>~;B+%L?j!+&t3E61prZZ8e!U3x9@+ZjV zid^AUcmvo|l8r$NAsx{B>oF21-IE(%>{{;P5((T@6=``f&9$^iFgTL_alfH~^|YXrcqob{9L zs85UMjST@vPd;-1R2&Oaz2j~j&7;PN%Ax?3NoS7x)uTg+a;tHxS%98^GkTm;k65q| zT11_rofY9{3Xpt$LQP;%`21Mt zhJE{kF>A(GWf=hL$Nay!(vDbaF#qyyVRP1V1nAk*2e=C0=`&Z_aPSA)uKPn%lhc_4 zxDUuJ88Ig6e31bw%LXuT`%JTQfarJ90DYV$pMbLkby}#&{`srI7GUWwYXN4f19X4; zM+Pg_{t_S%`!x5GnoGLV%$II;VwGnUqt~~bFteSE<@AYR;}l~5o{z12-(wh2Wj($f zynXWNBo6qqFUS$Go8>5Pvw&kTU1>w&%SLtud*z<}hki=lwZ9iL&k%rtD-Z z4$+hZn(eCgf>Hq&1Zr9RLc_}Xp^5!E?*Go=K;EyQQ zaOam90WadLrs=?;Q8bSijE|3wH^00b4X0+OS-em!;w5S8&xSJ*Z80WN6PH9Y3eHWh z0Gs}2)AWA=85TILf*usi)|pJJ__6I2N*v25@mFDSYAMyzGfIhr^N7n8!_q5Y60>Xd z`A7DD9-AT&r13W-9U|==x}y3)Nk%KW+{aD(hAxFa6=jp9+3#sH$-XD!0xC#kff}(l zyk02NS5nd{9aO$w_A))ERRj>b4Ae-K!zf}z7<}4>W#N`pU~r?w^=$27P%mJeAYEXs zQUxLqx&I^`I zal@lm4uE_lePVZ`d}j(>_7w>7`|l6`JF@hjk!MzR_adqq*L+=uRn<7gqm;N~rB}oh z-YphLpiG4KuNa87(VAaseGwePPWdkQ2#7Ont^qu(0G2{5?vIn-dS%^6bZ8~BNHg5%JyMq6-3rykWPKtt?9h+Z#pVAIfq#YoJr7g8~j$6mfizl)Zi-2=vKgk zUpm=QZyU~a0Y<$QCM3<-|B=#wQh3=j$laXZA&( zbv~H!u;U_4;tr?UBf&z0?0|E^`sH=zS*v!hEqW9!Hl|m;U1oOx$cT%ZSDs*a^q>Dq zkcVNkoPRCr?djxIJb2!u%0Do4XjI$0tNT-Rbf`mMp+j)(*@oMoR7~KLjW?QSHf~$$ zrbNsQ<%UPi)o%=krqD?|G{&MUkrfqK#{*ascV&+h&W{&W|YxA!8vM4EV| zJkfClg4FW@NKpo`B>qd4q%g338vEK&&}O|V(8wrw%k{CC4rXIq$h4oS2cuYvJf&GtS# z+jjh;;fib%(av~g0OUFYDFE={MBHES;u%N_fbrI`&Uj9*-;6+%WTy$pdhb!ZWSlDP zz**qA_<0L9Al3sgo1s2FeztDB^G#12EiS&l)?{fbwGFiS0)dEYeLoPQkUKbsQJ*v@ zY;P&hoZ@M55znwI9G{a9B~F%3$4+58DRFOZIc))6@HZF;bFZjzNK>eHNdtkBFMCIG zSIbLS_Z?=f5~{yyl_E8d!;D`ApwYwfjeS#l^ec!aV{X#O4G5KlTU^ zZD^UirP@|t3K1$)XoAoPr01F2dk`IxSgvH}QeA8IOgn)L@LfEDr;97~q_|UTQN+pS z-COZ5FH!Q9v$>~!9xKzn2Ti|`8dNh+m(6I3Xu8zXnzh=p_(Jk;BfBiOZA}FVL@oJF3F2 zS*BLUz;x61?$cl~u>KUWBBeM#gUiXir}?bP{=aeOU8J`_=Z)t3U`(VKw{Ieg>y^Fwf<+QtR&pq_v=D zWA+Ox$0x{3wyFqlvz(lB2aw`qe-w4jC_fN50!@?$q$?2HF=9pTcgo>`+ z=pCtB$6Up{bK=g?AuKUUrJpr`L&Z1Keop~PZ&{V}SGYAk?vFN(NuJ+?Z=I6rj~ox( z0UMqz1=K_Om{C*j(O=SmNd zexj3&??Gp9oLWF#?YCJs;(r+4o|9OE}BYFSDu&Xy$4v@QMGv3JgbY zOs~fqE^=s2X7#v97_T5Ar+%~28&{w{J{!q|Q=4`&T$EA!)K3Ml)nFPb<1OlQxoYY7 z3m@G?a_QVnAUGQCY(CaH#hnsi>W*(3(s~>F_jfrmdAfiz1(06x2(vGn$ItBb*f4Em z2vqm0QVo`65%EjZS$Nbz0#Y=1hbQuzMhJbD5pbmRM)mV$HMCikgppz2s*1OEzm(Ou zc|VZzY6Uftp(mM;ma+f0Z^w<;;ds*gj>MpzoXwYj9>_7|7{=88rj|J-WbB9e4Xd>* z7E^28^xiS`KN=Cd0wz7_mm4S^E-v1&5Nde-W|CEJ+dRJN6m??Rm*S+@z?0zl2_>cr zDCGIr%Ur@ksduDE20?)&JG~hiXYY47b{;y<)_826w>Io{lvWQtz@F@_uZ2(GidKsNr9J$qX+(RZPxau zzQjOfy-VyT-p8k%6|?qDn{nmEJ6-en+e(1;s0AvFquxTY%^UR<5*D)$0t)Z8-_^KG zTdR9-E&UUl5ryMHz?6Wz`=X4LAwY9)TVr2xO5e*6i(W?4c@=3j%cpa>R93Sdc}H1X zM!Xg!n6K;WCVwQMcty>g?FXY{sZ%*k(?y}$-der*DvZxAnTpb9f#KDVLqHQC(DsZ? zZWau~KIA(rBS9eoN|Rr%q)NDz7Z46C@PHEM6-Lc@!m+f=rw!TSDL7DRYOsOf`l3#;!YOanTuM@MeQ zb?fB7)Ux_aAg`Jyp@zAlWO3o&fP*v;NJQA0@a;O&4*aS(?tI0ri+@SvFZo+PxX2r1a?nE&~-3|FU z8{VAt#dBZWhN~ALBbaUNgSg>a^<5MMw|3drwz~S}}B8WE~ z5apl$?+pR%$$A$t1aM*vyljUV+X?5dlb`?g4a4)W{l6;%?$Mcj6K@`y(te;$K3cAC zyFxDF=iFP|FYe5X8q`rk;qO3-GN@Qt9bF^5vC9H0KSDJx&^(H$2!j3CVV*$H5}}i1 z;@Z8%TAZHJF6F>0{VmBkb+fMG8K*FwSFt8im|9Qw4;0g)F<=Lu0v}R1?WrTYZJ4Zv zKEh^%rJJfyewR-RJ-H(lk|L|j%~t+Z_>Zq_c!uDjCaj@2lGG}LTyqh+i$o`Ok|osc zJ+-ow@Ns6g_IVz?-nZ>*O}d#SM%Ut?BolA!fok#nC%OOan0hLG>$FzzFK)A z)IsgPn*>WS8{)q`+^qG?DJ!}p`f4)#R6N`clQFv=^KP$2Rp&B^A7D+aZ09TY54_~M7`!-EEsRE%pc4zqh95Jk1h>DZ?w3wDa+WB~)R;E9 zVb2z*woUjQjPWGvtyR@epGsG>%IX=9$jd^Yzu}6$B8QF3Xzxo@?1c`b&*67Ogi2KF zAv0OK+PXDCm*_4F;bg_hTH1#&FYC<@w5(> zd4aFaejTX%@17XN0u}cXm(=xjo8o61En2Mq*b6_nQ?oxK7ONuNsr5kvO%sAc9vZY9 z)6LXM-cyx9T*t>aUSztgW{+T`AOyt})8Esq$rTG+Ihtux`L58op2gk%HDCWHbri`u z_obTSyc+h|ho}xkSHFSd%!~6>s!&a)Xb%DS?zqqXfLXwDdY1Yv>?6;YYUQI$5UxRsQAL z0s?Zv(pKyvo!;g$W5$pPyiw}vR{jC;$H7rakDst(lXu>cfO`)i{paP1pVli_tEG_j zW$o?Q#`=?*GA&$^cA34=Ptn~XN)GI5T;ycXL1C0k+S_*`#nl=gtw~+*{S#-mWL@3PK&&KF*vUOrIsEU-X>8lykT42sLfh1{;J9Z{B#`Vcnjm1yX164ON zC7Rp;`_9%;UUXrEe%MVw-G^=Cgpy6U=o zysW0FO0raibk^EWqdrTxI_vGV}demM>?SZqf;+9=% zw^81Uy)h1`l(&^$dG?BINi?F^%?ZoJy!(8CSk`g#VS$q)SIbHe{xAtsksKRYd|PPI z;54zU&wlVH=LfQU2(vm_)qonV5sP`Oux7&S>oY`4a2r-ahx=d>?>pr7cZ7CpD~}lo z`wZOPtXUw`*yJR6^t}pRhR6`EGd^f-W_AZJDIvCZ^Ds7F28<|BqAPb=x`5iD(4t=+ zeZG95Q7iJEg$e`hDTyK<+lGUkbaOQ%R(S6D z)lug$VZiPJ`B+i$N>%w{gIzb^7hth~`e<^DLrklJnIup49z9&&I~; zeupM1wa#YQJC&6~z5>zLPris#gR!HOB0Oq>NC>O@us4sG;AniAp5N{7MfqG@^-3@6S_+BrhzyVadNFvp+Rl+zAtPCr#5X^WeJASK6m(!%15HJR z+&jf`(-1Ny7x1^1(ufx-Ob~RCkgQ3!*rG7!8UZbXbecvfJFPt=9x2LBU)|4F65w5H zaxpT~Fo{dC)l7!ubQdy)KMs~`kTM8BLXHzp`^i!$dz({o(Rp{3@xPR;y(5m@3=fSy zVsbsuU|g!@Xoi<)lb-|%=5=g+kvbx|DdlY$4RO!Ze<&-qACSnBQ1+)nZr>9pO7@6+ zfo5D%R$k?61X8^3VgKUWOb@UcP;P!5#bcpyxpr1?gwI-UK>7mJZp^Ubk9DSnh$K^i&f#)|m*l5$LF<5AVoW2>ySy;^A{&o$4GPC^ZWiX<#AT4YdUIzec1FiTu;N_3)-{yRTLIq`$+`k%T?f(o5=%}Mquv%Ic&J%yyaeB-5L;Uf|Nw_sb z+h?%rsc91UaI>J6p<9|MX@mUf0UbP+amQ_FWzbL7U|m4J$6&_2U#Yd$8oce@IHcWw z-4uNwg?>ho&o>y#UEjl7BbLcN~yU1mCW1U0J>3LRG-j=_B zHwg%0Q;XefqOo3*E7yb7XtE>tm6M_StQ@ur6|)zWw*SyzpTK(@aTE_BBErN$gM0kM zAs_g(vWhCQP_wBUrWF^t-ga0MFx2wRf!{#WKl~s^f|TL)Orw3{NA{on0)6~dIyX{B zRT|RKuw@~c}b(2=W< zdzeF8kyG_wV`cN5u)%A4JQ9>F?2cJOc7wi<1Z{%DMW%CF%51@+!8&55DOrtW%Y^UGmt<$!zH&p9 z1vB2e)b18ry-DtH`(!BW*NutWa+OX4XbSec1TMrJnPawZRA#p(*pR{*Jyfk7h2B*i~@PFzf`>sve3 zpeL;g>BpE}ndJrJ2FGVXSRwOfSrnej3wGD$%lHr48HXQr*U>MS4Cp-#mdFkdB4uyB zet(L(A9UQKX|`2JMmwM|x>tWnjzRFf(hi%Ky&?+3! zc_i%0kD@hCo7vr2+!pD#uS>>$lza*Yr6y>zj;FE{q~<|D(9iyfhN`F1 zroQ=ZqzBT>vG14^tQg_a%fgLWgjQ%VnQd;H)wre0z@49Y{4K4>F`upHBZV4jQCq(a zZ6|^d0nZ}Ei3R09eFq|ZXbqB}cO%5*%u>18B5@ZO+Tv!uKbKAEOOi7FPTa!MMez|M)E{p=D7vY+aK#PTD*?FV5zm zs{(dDIR|Y9&&Xc82dcB>GAtD8lM%hrV=rR_1bYQd?liH|VBTgR^dUXurW60}Zj%$? zT9xoU%g>b4bf+*W)@A;wb^qK$CMO-n1@|7hPMhqujry@g;8u$aK~&-=o7C?tJw@A* zX&XdR>HgR!!QH~*;2!$0AaH@_*NLa>h?Mqf`;3^0h5d?p+P80OZ5Rj9YX#l?DlBaD zC-W%o#lA}^6H~{dPMmCFxe4L&R7G3465d;`Y=@>@O&&Z)*UTW++CB0CdS0;mM#sN+ zzp2R;l9;#IM<_)oDKvz*+B4gYI{L*E@)?bEL0e?6yu8Po4 z{ZuZd8UESsBrib_-cE3@h;*?#2q`NrFJI1ll0rw8s~>-t?o)!vXAnx9LGIggc~*BMU{*3ljY{ij6TpJd zz}TvQoV1E<(bS(6y?$BITq&>_Y*H^BWX=<*xLxsZzDdPG)5JsQwlXCCoOi|E#nFZZ z++wRfKsL6+qx5}f<>aC~@cks;ibA^cNqV8Tevr5=>>@d(N>!x^JwOf$P3xSD=ua~l zjj_H)2{{&5dBKP%7!cBGO;}Hz@V6@ zl;T3{OMqMOReigu%9+%yKnVkIf+j)}tLcN;lr|gunx*#gK~n;*j{!JU1P%uq1^kx_ z6d#y|$9ehLvQ}OJYV$xm#S1j_AochBhUhxH$VD|8+v$gYdy@-1{VcBY>bwE)C}{i7 zarOVw*Ny!Dlta4$N09GtlGupWT*vp1A0KBW8#}G|GJ>d&ZI3OE|2geihSWpgz#(MX z`zcSdHwJ8O0kiqXH;#pOAQ*T$99@YI^4=ms8=eciLxV*i;qn=$TRYviFAu)9&x$E% zzH1GA5BgSs8193!?GAgs5>b9n?O(FPOCkzGSlEcr>mqjwV~uaJ!~Jnd3Lq~UH4YcL z(@7hQXOxJ-XUFCC)9GzJ*;>WkHtmTm3R$@b`$GQ z(x&!VNH)J;4*9V#pJH)Xb67=^UXb z1D(=1o7=q1!I5>ov+Z$~-0u{X&5Bn@av(Y9-LI?G8@o6B9a;U`PCg9FeRgtM5DRiB zyu^|W69yZ14kZc!Nu1W$ypaW0aD2=ge#8IM+{&v{3<;Sbq3q18 zLS!Z@8Fyv8ij12ZWslq5dt{HgQTAS;%#36u^gU0#-;c-l`~3qxkI(&!hijbcT<1F1 z^*rZX=bX!qVa?&0B**-@+j*9@`~o7xmp#oq#}-)Kss0-O81Yznk!at4NcN~sX;*RR z$^5L|!h)$FAD_gXDH$^k9_ewT&J9^+pId)bO3rJV<1`XBW3b9~%O9`Ve9DLXHlv?N zDYUGWOne*pp2ns%f8$WQfS*ynppmHFcJEnM82*yIXN98a&T8)Z(lzP7S44O%)@CEf zs%;4=ysF*%jg8{;N=RyhxWyl>hwI+#K}&RW$5UIePdZB6;*<8MAyCGYn-j*oL!fk> zR9(sPekEwFM|i0M)<#lx?^#;X7VCCc5U<0phS+-d^l~ztuY(lK-t!qB$rv7~V;qOc zvNIFo2AJFVwyt+kTeSSLa>3lzBRqe9bsOhaQMj7wahPsGZA6zxUD%#Z_=;w4v|#Yh z5#EfBZ^@%APxr1mCVlc%wjZT&Yd_6dF@T_OL0CeCgO>oa2A8jco!fC*McU#<8v#+# zDWnHq>-ICZ{`kJynOEmD$K-Hv{q|b5*bJ&7xsxgd7d3|)DloKaM^(alp=<6=pjH;6 zIA4nm8*B5;MRaydA_)S~z|Kt6*^_c$k;}dT z%4sZBmQAi?vJyN|A>a7>qUQfAO7Y|1weUT2UOm+7?p7JifLEv9^A;QIDGD&ga~BKAd_G3+Y?%6_$`E^91ob!K|SMFR$T_RJsZWqw9T z*AqJvg*$j|l&7cD4btT;M5yyNz$eTZAE47~-P zjq21UHFOEv3!S1cd-sQ9#cn;9@P+~1uJUNNg9K^ktKc}9{~8J(DR4dA;!0e0>mAw? znyz}sUw1t@$m)~DD?$v`AfwEn>B=7_dgVIx->OT4^F>{v_NO_Nrz`!ABK?}Yg(eC# z^UO`WyPVltgtx48@hUTzgTTwtU}`cD{wP09zG%VVH6mAXzEd>%rTm$t8y{&sLL)ov zTLR_aAa~B$*35tNJd5awUvxlQQ3#&|&;7D1p0l;<(Zy08OLfQ;DVc8TJz)6rfbOm3 zZAxE(E*-P(t*pc+BjHs-5=voJzjL!THg1On{pOZ@QNJZD%y!-R)xt`iOLIRCr5_r<`j=aAa=tNX9WN6JQb`q18`mFQ}@o?>T z@a}nN@oY!9?_0Xu3rlQX(RytFU$?QBlN@DdY%|d*7@W)x{Lzg&f3yT*aE64;Rx<|s zMz`L_QwyBZDZEVQxZh=cOs|Ke$Mo8~$xtaq*BrRn#v?P!P5M!|(W>=$R##%Zu2uI; z3&kCCW{9BuMZ?!tv1`O7y)G##= zZOIwU#t$Ap)@=4!_0$wm*;4NVABN*rtOj6&D?9)1=61*P7NT9-a<4oqHeiBaGGO%$ zR!FIst_3o9?iG)fpT!nTK=0L z1rv36UTfcQGoRR`pg(z-z~_$Ut8sK{wVjKRk`V( zopu{ibB$_i=nc$7pJTJ9FMMVCee!5&QDph3ONve6fGkx^AHt|~WCXw%KcIjTci_GOlhRsG!i`Yt1y$!Pu$j3*L5_1BsfoLjqBLkeHYy4YD` zW8jOO2#c_mVthV4XCXWZ2Ue-0!oiy~fi?QbnP%R2j1&Bii`w^aJ%WeZeT0(f!y zC-&WYZwZcBQhn!h*v1`RbLIY&QCc>n@197~i2laWb<(0`FCx$1eI(m1JJ*xr6w^p`O@E4ZAu4k)rMav*HD7fbE=Qag z8#hee5UFM=-}n$$+ntxnEZub$&|w2~*bGSq4Izk&!mAf9(VouS4*d4GF?MfkG4p_a zaZA&$0pP2ejY>AgvS1>Gz!|&$X36;+@ROG_6qgDGQhbKue?LlU-* zDuVB8Q2~F_`Iv936Dp1fIdJg45O$Pfi3)hdfLB4EX37Y)#7O|mjsI>~&D8UdUf1B@ zLL!6a(NLZm!qlU9pSHlf?1a^fwf{GcuWyPhC~oV$rvkvMx9NndRfnIa@lCWw#Gm8< zx#$F5^%r>ACp(;?nU0PMh&zFaL{m2h{=R5vI|fs6#U)#-)J!p1&0G%r#sQv+7U}@& z3cxa{rN#z~YtuTu0W`^A^Z-j-HaO95NX4Y~QUD}+OeBgOAh8+RLI?(tl$k}_^^WGd zC-EwBKyZmmhPDUH07v>t*4{0{s$N*Ks#ir8iMRC_ML^~O*X%_@d8Ef%db~=bi9h)- zrIJ2+t#)$y^8keypzr_`c0=1cz?yd%uA4h^lU_dslt83CCcNsL`7{$AW@e6pa%(8Z zo)rTESv=j=(rsS_fu=EBlh!c=kiWuikHV|A(CFD1TVKcsVG=Tk3SgoGIxvE}X+)l= z)X$5j6c{S$0NED6wD_U~Nfx+9Ud3PoQ@{GAB{3Wi(+I=3!Z79M#J&bv;*KDSV zMC1+70M<{H4XYL|Ns!nOSe#g6Gt72~dUE0{@IYyJ{K+!#2eG=X_n6db0%cjQL+fa` zFZoLe`a{rk6uV(|HH4EP%k*oYjBkS2YUT{m2cm)Qh35?@eiwF30$B6GCC;m`el`%+ zmUGE)=^T`4n*1tpo{kC#M%srK5e!D;=_*6}X{dA7$>oc;D5A2O;Q#`FGjK8m2Ah+8 zul=(S23Wmu;vOIAZdw)J#6rk(9~E^%N%;|U&27D!LI|J>n-z8x1Q^K~uzB7^2f(n4 z5RcE_`pIzp?#(TWjXre380pndM$94Ts zSN)8Ww;Uqa5^7CQjH!I_iZ16jbp6R~TJB@8u!!ac^Crh*WWyNI*<8H9_KiZHOaJvh zCzdSTe^(E7I@%)6*iam3$O74q%vbYET`a&7++@0gHr79iTwdeL$Gq6>>lt<8OA-smKg&k2DS;>6y?e`Eo z%~60Su9)K(&$3A#(p`yz5@%r6R}ipNKrVgUem*5RFhu0Xzj9d7HS+8K$~p7)^k6IA z_;6eW0U78=^rj`}fg!uFX1cVF(q;epAQcSi|0+x@+A1yUDos0^VTsAWSuyd^X}l`% z1$v}7i*G`C`WRg#Y^{61siY4B_=am~{>&f#2TBn@t#BD;o6kQ1-j)*`-xBILhC~aH z)~o+Pa{0MYaKEiG^O(u#g_?=BW(@JFOQt&zcS(TOl5eZz!Xo@fFA7z-6-(U9CNEaN z`fH-^KVaa}yWek8qTMurhOas?R3&$n9%CGWf~o=E5^W2%O9%q|-MpmAF6V{L_>)7T z8@#GVN7W&OGNG~x{~RNn2J|{Ahb^St{#hgMxCU2yT*D4jVY7+O(=}*c1^!3%gbExF z9&*OwMBr?(C9c+LBJ?%FR4DTDW1cp5gB+ZgOoxNk#s$T$QNp?a|H9nFW`u)^)& zjIN-~Ww?~ow=Nv>WE_qpRLvml{f3)>pNu~_{rhFSDi1>G%*T4PYj0rn`2e8QU?lbu zQfey7OUkt*h{x~)UtYav-*|RVMX#@+5CRNQ$%1cTnBlUam?Jab4tC&T7J$rW2OC>Lc^EI`?1owz8nmG7j6d+fVzc zXgZs#vj00+X2XDSB$OkB|qCuW#bC>fb=k3z`J{P&v^G#@EXy5Qyen#jCfxKH>K5SJ`c4BOVK-=UCOX)+oYL5lh zUt-f`)BY~3UH}_Oqa1v_-7%O-&r5zb;jyM|9J$e?mv5sT-e%WE{Fb=wbV>*QeuQ}6LGL|_kS#i?Tzm7hMV$7;Z=@jt}QKr zAI=ra_H*i@nTEDxfgbk z{mbvJq4-xGIH$_tFgU8i&JL3GdzwY$okSfz1p_ZH|&-!`1%H^LnK@)Pe9nA!X zuzG(WTAKHOpIg4iU%ae>&Z=ulfLd8**f$Ol zKuem(Waa)fke{Bm0KGV15*LSMZGWM?&&ZC&b--}OCMZE~6!b%atV^-@RG+ul2FNNsB`@G!L|*?Ye%+kK zz>4=&lSdKx{%E24#yiJPeTY$sICpASxHCS6_tcr8NxCg~!`_mb@w&O10gu(e{_z_1k|fy|gnW7R@E_OGP^;QmKJ7(XG%@YM*azh#Fy> z*c6yMjjp#EA}2HV4A?gZHWtRK`H{H$RJxlZaWd2*yaaMShu!hI3HtYG*FgG_)BIr)klz(K2x-~q_(y(S5%_vW0%zg9 zoM0VUucj5r5AHK9+Jy%i|IboP#9|`5&RnUlhI+=>dn1}D4NaMa@&)Rz1pXncX&(Yy z0OC9B2&u<<7O!*}#kx3b47{nO>27x_zO@3o+J#O^qJ)1V)2aW!dWt=1Fx?15U9@ zbwmy{+{Q_bXNH<$Y;Ugkn8y@GnL03cUNJG9{k2b|H(EoL^rHW{o#A}3X5|aqv{jtU z?}UOj_o30JN7-k@ClGR_)rOqFXQgB;LbBLCDI??y=%=_bo1z5dv4 z)Ska|lR-Q>h2Ni??~wl?_sWoK3g1K;M5YE4eipJno!;G?Y3;0G7+Rn9m#ksgvFLgN zOM~_>KJ*#rbWZ}wW*%5gk5nwazBm%LpHLF=kj>tF$?i%2Za{sZ{wwvJqd&W!1gR5d z&7^bfaXI-23c?A#JYyo%JYkGJ%le#+E4 zY?)H|Lg|I|==j;a_MEmOxyKA1HpLp#{2DP?^dM#;9ME|GuETu#GEjU#tffePUKICG zXx=jJ4DGmmMlXMZ&6d@~dpOt{nLqRV+nVI1WK4wJ&NDX-9-ny2r?<2zosw`_7Z{|{*Dy)xaB#2467!BNpg0OwM@7(}d&di(FuZHkRG6~Q(! z$5=Cpos8+)%OD9NZb-L~HN_e^0&rEB?7Cgiban~7BVSJC{GjARd+ zGBy8kI;hGjgK*}2dgm|_uB!9i=8ZNsSzeOM4|MWt-NM%ukkQX$BA3;=)qSmOZ#Noz7PIfk53@)u<-SMCn9lw9J&5|qnI}ZU!6gk8YaEPk34A#Jm&>ZSfKl`| z@!|4?SeyT80vh8r(9PvaDVPWaYnJ28%~~ELtv;z=zi>qAs!WG0Pi#c|Ks3ll1ngpS zMe^!Vj{;tN@CV2eK0W3BIb$tV6N? zxz?wP0EZsn1YEY7are2-YV%;qqQ#JE#m+!{-+Kdty>StgFhZfr99(WI5hcz?qRKiC zWPGPQPV7_Pr;XtWAP;sEu-i-xp6zAwo#CcXjxH-Y5vVx5uyk+SSk8P&;fBMehCg2Q zi+$&Jg2klVFiY82es!L1TM=uIHa>q3)oRjGmN2q%qONvv>s43&y@Z`NIY&Yzv;qn2 zuVV>%N!6b7b3UtSRr@Z4z7Ue*UDv5byJ3i2Nd4wY$1?x2_eS*;JJcl26D>>ksQBlm;h#Dj{7AZ$hN&pViOOY%E|;u^Oh@yqW!{`Tj%- zDBJtxeI26B8+osOOjjIY>*I(;Mb_DR8y_Nj^6b^8+S6w`n|)=*%g z(AKcnaPt!x$e*{jv#{?-)B|}{l6f==@q9z$8gBEiosU|4zyyh6bl%3^VcFScOB&-jot~* zJNcCXg!>){3rscveqB6|S3!9}oLLzd{!k`t5+rg?NJBV5l1V|nUW6Ft9&$J#%3got zKuLpvqz24g=n8?O!}O$uZn|Js6k7vLj(2PPwA4v5G?RXAwR=of(FK7u1yi%>*iMFfkK&<^Yis1V5F(Ibe# zqMw1TmgX^VtI%sOcdD$nczr%uMApeKvJjL0-M-MNd0dY6dY~3IQ{TbKj9%@+m~Eur zr8{~qWiurNv1Ixe@V-Q5PQ8WyrG9)cFIbPEY`a9M1brX@EG2G!GW_hjx2vmNZrF}2G9BPcX=sWy58liYHmn(P}} zNw?aH{APor^uc4>!Uj&dg{O;p%zS2lU@rxi4(6<2v|Rk%yPB7u@@IT#Y34=meI;@R zix!3E7t+QL)hBz1cr7fEEzM#SDzjbi@aVOWp>5<^AWFzHS_X4S)^~aNQPKG&mI#(Z zGWr}OZbgz%-C$_cfKXpc;TEP(K|pe)i~OSND;TQ=#!~bz<+{r8%oJ99CWVYy=x&r- zi)A?b4RBFt6aw}Z;_r(0S0P!~KPT1IY1vHFFgPw;|D}4Cv&Ek0atHvCysdsS=);-U z{idD6*DTgYI3+`aa2-b=$%=7E#%J&{D|!!UB}!G7l-~&>%PqHF%zG#*q+C)yAnCUK zZHXIyg}&rz6c@hEadcRIVa3X)zGH^7#^6_KL;WdK?ur+&iS~HjSl`Df>C127KHk=R zp2M|ySbd@O^>jpB73*sj*Z+pD-gEk*{>x>+Fy_o)Rt&r|QWkM0NGax5Dnm^4u>JUK zZdSz(G2`iFf$?O;sxir5SxnrT&SyZ%sp0<UdSgJ=64(t36 zpLwhN4dL)5&-=>$oMIs|aoOdJs()_&dfqwK>yEP#8ZGyNUvAN896i3|w_02;_LF*q z#t0*OKf&DCIRWc~-Ewb8^(a!6vaWfPvtxigjF=+|^GOvkNt7a3QF7X}>WnB<8!*`s z7(c6{XV~8`?x$?AROt@7m>g|zZtcdO1JC1UVPA8P)u z{|dj4y{rY1RNZB+_KK}@pF3ioA)70eNe&g*IH+7=+%s%Eps~8h?nAe=L#>H77Cp7B z8p@#oE}q;HwOZcW7PdUfbSAD*W}ebddYP z8Pk5{(L>L7J=|qFsblVKBphVP7Vi2BYn*I)^mS{>DEedcfd0FwKoY)pW?lW;jy7wi zYuR6K@xX;Eg%{J#JHL;_+Ad5pZlO;<=ln-0I$rd>iq8Jtm{a)Q@vT|LX~zl(A=%G% z5pJ*u+Y!?py+E`jHej^;R|RB!2%WLJO^h!|S?r(w)_kr@kyy+@0wF{#fvE<;k(f;aPDnu*)qrLqnXc5W2c93oF3r7SX;Iq5~0Kx4iCh~J8Wses&e*X- zJht)eT~v-)dQ3ipyTWLIpO0QiZpX3X{N=)-;htO7;a>dtwMjw>U3n+T=tQ_?RO}tn z6q6sizUu3}@{{zd(qCiO-NX3ml+mgCZ;b;;wCPy-NuFU`>zdpmhL;b*t^S;Fpm)G7 zUKy6J4E~Ab<7|-fb-n$Lf=h#`Zdr+xzr@2SsYg2*TmsM*@EvDSn@TdnJ=A^XH5?YS zPF4PH;cQsuCo_fB)NCV8E5Ce3I}g7`%Q6cElb?G;N`KdRrI{sNdxCv!GDYY%=_L^; z@L7NR4A+elT#Ko0`eZuaGe-O6b-xuSH}6=@??drkZ-SVi1w!uW3pUmO$MZI|$wUzG z<4sySFV_}{p9WpSxl?J)7wrBb=r+CY)C}5raf>diM z*Ql1&Vg(iU%`Wkn&{z>$fu;Eyww90aT{DcZ})*oXKEj zOgZ)Qno2LHqj#P2-!%=-i|tEyLOBXc8)uGQFjK02sTnLXdYtWSjb7vUAE=4k_Axh{ z>s}gSnj^@R@#4&qN0>qUT$cM<;k0%#NG<52@OXc{s$PpS5mV#6;Xa3Ztz5r1mLKR8 zwPj4LF@K5Ub~c@wyt(((RgAXIflAhEmBC@;ta4sVS3Z+QA!hbWmx>T;0Xx^# zZf$Kt2Zz7 zdnGHM6YjEq2X0`MV-WxPjC&<;DBp6)r2dOnudw8sFT)IUpNB~kKit7|UwPs&*_S!> z4d1Y+&tu>F-pyI16fHfyeAHGk`(J#fWM`yct!wBD zVPERu+7h)#*Ee1Bs;uew`HL1Gq|q}d|M><<^baL7+hlwuevb*IY$xB>_(ZC*GCo$tsdXC()#zx(vcCwf0~vBF=Y&b+&sD)r`%J}o3)w!+&U zPSw(xjNW&RU^iA@iVbwUH-16C(0|+Q4u`!Qqyy3(c^<#~BDB!NMdLOx2212$WU&Wg z7yIJ?a?eZ6fkw9y(@!>n;Frtk|mZzYj zXtIEO9u9fax`=_vuOXe__6B5FUV>dv8c?^O(*$&)24Niq8ER0!6iUvU-vS8KFw*>M zOU{dLK@+Hgko#C(7>NlaeB|*!Av1u$hWrER5`>(<)ol%Q1hguMmep{&JVhYrg7YuT zwu=T((1hH3D1g>QiTnd*DD+uycB4W7Tn2YyxZD+@Ly&WTbOhjc*gSwBM+u5bIOsPS zA%KD>GKRuvD z6%aZEoB&Wg=K)8s7&IFoXypT1ny2up^vA@w2N>so{`U|rY~T?YNc?vXAo3v)hd?99 z0^yqsS%e-EfUylO3v39fB|#ZXh7}glMP~g7hL(B>4at#)NQy1Q6s>?jbOnfwX)v@V zvc?IZcJ-XZt1dD@!wqT}`aLxCfkOn92+SY{cE=RZPY4>}))Qp|&IX=hK;o}(uZb!H zM5N&M8ns~w-Nrz7OAJ8Bilp6g0R7JdLCXs`1<_bj)DW~ZkZz!{x*`o1!TOID0Q@Tu z8q@)IAR24WWn>8-0YpS&9YZkQg6n6w7%&p(Ac$N9L>6iBsx~hH2K{Lm%mnx&48fQP zQqwew07TGD2+{ID)eUAC8V&}Oxj8}+QoL#wTzH}AnW0;^V#)aqga)sGt-e+RMG%Fx z5z)F}lGtN~l^#;qqI}?VF+{48p=uG>{`T=;2v%l*6~asOQb;?PG}4S4!H^F8G7z2( z0p3Q}gBJpFVb*8^RrDYs%m*QzXrL2lfPEhjEZ8A|kvTUwXHX*O!W|!qj0CURuZ+Mz zlCD~SAI2CmuLz)&kSGW5T{@t>kP%QS@IOo35(o>ZTW^5?LNFLVGT0|Tm;MR}oUf3< zq+u|HTP1ya2(=?2Uy%6-xN2NS2D^j|mT=V&56&*^;Cq0&1KEJkS3hy`CO{KOq=_~( z34!IGLl)!?T-c+F!NWuf!3Pg?7P<^B6WApZ40azmEbbr(Kf!bIwlRRmE;%8Bs8(?D zIxR}hhu0yJ;6)~6M`rt_xS1_)R8<-LV|$zRP`%G-P%zc+FsNZ)*^krrGW|}#j@V?k z^N{htyP_{UA`^*($EE=E6PGfLP}7a_*ZYjZ8XQTC$pQI=jJX$H?$#UM|MdUepSH@f zCr}{>pB0ys-3>Dpx1svp<5b$hIGagiQ(SnJXpXaLYjTRPKn61@RX>#)@g-EI?^Uvs z1_7L7#gLBg%Jmo8Qa;*)@36Ej^t+nU$)n`#HbOtHBO4kITSv$mgli-F(xsil(+6*O zbbX_K^_K!ZO2X$s*F?ph!bG7$CKd3i!8jSw@bshrT;UlJnbp9n%wEOAs-_$4`&-{XM~EdQ zEr|e?;m^T|($S60vn;z`^*zp;@aYwdYzQ^y(lR-wlYf-IbFNm`!e%=<(Q$B?w?Ch-aSLgh)77Qww9nhNA{k6T7D6G=?i;|y!ELxp znCnwA+6DW93Jg@{9R+yRxaSX0t*lA^d~e zqUh|>GUXk4RT#?~GY8lO@bAM!tic(+4}}&Q$Xe)6rzZ&I)hd z9^;npXk6SkS5#z+fO?LQ<-zrXT)!EQ-AQc}*yXedg?uhxUlQ5Mq7`wjUuVfL+8F4Q zFsA=K@ZgWWHw5fV1?>FV*#5ij{lT?{2d>U{-jXl+EDtIr_>`E9V24kkkR0D=6=OcT z0KydlU1xFH0}t7qoT(EJX@;taP$5X7GTW!F1=lLOmDgXZR*K@LbfR)DbGNNx8?weq zPoU@-ARBvZKmf0DMf@2OwYu&sSLv(kFHJi$?!A*&ssG@-bfc93P?^5nVZ=P;wgQX z3&{hs%@yjuNKy%)8V``wDG%V+FDTIo^7110%60!xw9UD6_QP(KS#uBPQfk_lL;VUA zY75CeR#sHjb&|3!)Y-rrKlI#rado>tjZr{ydF0VwNXh62C*oN{1Ea@oca4{y__sVL zyyv%_LY2no&Ej^^9EF0RzR-}se39H4o+d!igYOhd z$RH113LfrR49IsdQ_&8tiqA~0Mv-Yb_gKt4t{d5U@|*cStu!YQzux9{ z-?Xzcs!_A|-4gWE zQSjYYA%TW+$fq5CoWMKqi-NraKTT*3fBw%_&MqKXRBIOSVMm>-wLG}c&2K6gzR`d0 zE2F?ar?HjAb?EB$f1LWE(ItNz-xrtvk1ci5)~>sEMck*s2i1t?hu9`~G~2$spk`Ss zZbHektIuP4oudUF2P1IW9Gy`*yovd?Jn(Vcbn9m-3cbJRfD%%v7n4-{>tM%&A5y~5 znY^c5|FUlv_5RyqPW@;Ojj8fN%l~}XQD_O|`y+p}eQBasOeC>sSUspIfIKd%?sj?Q`GcM`HfbinL+agEWRk~FU6?egy7PvoxwRq9~R2K85c+dnq*>8fJ-^X8dPF?Jd%bwd<(p4U<4@9D?kBRqR!C(NJx5ldPV z-XcH9P2e*gjG2msXm)+x(VA@+|85qll#e;h%u4?<*3jgR{+3+y_Q85{+G;4P=rs+6 zZn~~SNGzt>|Ggv5W-ng$CofG36canck>IC8@`(pFNd}v414@bBhKJOc6R0v?a|;U# zhW--rwGzTpXX2CV-PUk<=1E3#B4b|e;N$;QcjT)MPJh~T*yE+chG9*Gm5swDww}s? zcTCN!w_!r=#Y?xxMAyyn+K>2v=XP7>nLTjEI6YNmF0QZm%f%1#scvIVneZ4W-yi-#=mbjbj#N z{>G^fSNwa;)n<}ye`V7&{C}TiwP-d7uYASRn@k|teto;u@hYlM`=3p%%=A@|Lcon(y|sjoJ|~vyp4^twpNii6Nx9IP`!W7L zZ0LuO7}WgcEJvE858=$I>Ssn;%F(l`6rGC=ev5Z&xh7O9%ZG-qDcSt$p>3``3Tl05 z5pJo^+-|}u8avEpjzTXXdza@)j`}v~uuv;^jhuVI%E+*PLYkP@;7wU=x^I&sw_dd1 z(i@r%LMOB8uDk6s=6pWqHm-2pBJ2(}`2lCR<)|i~mp@cY)*$6j^LgnvJb&8THfm33?FSEjBJyg#58@IapnUBgN`9b72+EV_bdt273+s7Rs-%?l8TRq*`(?rHCoNf083%LiV$X(Ui9Z_RW$Zjh z#9U9Y$x)rZd%~c1iU4(I{P>ex*4?M#B&jC3qSVRVVpg;0vihhheaIIps3B;Czi053 zUtM$mzaco^v_dBqn-440H;;Tgwdx}|xr`n@=O5K;{olvplpS4pd_NGK_~O!oD% z+AF?{!)`CvR~U0&^5<)h{}%qBUA>wRond^gV(>#W;VHYhomCr_&b>UlJ+ZuLIzRo4tuSkH(KJ>)0u1e^KS~v1Ikwa^ugVMQmZS5q0XXbyGYO z9IR8dZVXnM+nLy2>MO4=yOu#|Xq0JmuEH(JcFD-hBRIG1kj-S`P*&Mi_cC3bbP7fc z^_Ah++<}(tc{aQ2JFKgybe^}fWtoand z&qbHjdcZxp9HDR9&zTch7tOmMS<(_C2stE#c8BX`#)DazC{bD9%hp&0l6mqC$Zeeu zq4E#veHSm;`Z?_&h8cPiP>E2Kq(L!pEiXMEk6(GJLVyZ{qX^ZbIVnklLjQy#0VT!9 zJo(d+4BqT2674S{Y@~)yCyKQqnjPOCEcW`6N1x&Pa8sY|TOu@%If&s;EN*W#?m@8m zzMyin!0TSJ`KCS+ve!iDh-6h!R5Af%?Oiz=Kw&!j6IcA02xT6R$W`RkF5f$Ya~|G~ zj>qiNqtM03z-5gXwJa2hsvRC~g5UoI0@(jg=sQyL@g|@SPS02Y^r&2w3|yTpttIuW hT%OuGIifTbZ}DFhxTYg!)P$g~qNJ`^aMS$d{{tkR^|Sy0 literal 0 HcmV?d00001 diff --git a/sections/0_causality/figures/DAG_open_data_cost_savings-1.png b/sections/0_causality/figures/DAG_open_data_cost_savings-1.png new file mode 100644 index 0000000000000000000000000000000000000000..b7a6c5b1750b3a2d7bb09aa18607061b7b2180ca GIT binary patch literal 27453 zcmb@t2UL^I*Do4Hf&u|6El3kYqzXdlh+v^N0qI4gh=kBP2xtI7dX*+f73mz;ML_1$}J)_O5}X3uZ$-`;!Xnane9fU@E%2r&&Y2n2%2yp~o4 zfxu7@h=B4s0nlTUz@55v3HstO>G>q8Le(`OLq5NP_e00KGifIv%zAdqk@2tU@({8|DXQW*VkWm;qmz4;bB0(r>Cd0v$LtGX>)TEz;ADF|KIWdXUo9A z05GbhrDb+@c6xexadB~EWaNMLUVi!e_b>1Th!yZrQBk@4a`_h+_`iz(>vw=GFzV#w z!xh0|XqPM%d3&afw>FVk#Kz3qcVry$_e}8{yXb6aK zb93|9*jQUzTSrF+;KuG~fGHlfyy)2&;#xF67X>PZ6` z1n3_g9_|32!Ac;`Krpwrw>LI6fD{M{3gU1$XLo!g!0Lm&s-haG(!V7+f3K%3WNpIq zR@^GzFDB_uiD4fnos(hg@Jj2!v3mVROtbDS=}ZTvN~gHaU#omr|7~^C+@-zGnO!k0 z6Y(X=44L!64x89NhYXc<8@b!VJVz6g(*7+>ako|tE7d!(roZ@{LiBk|_yDU{jOAWQ zgZv5azXpUGh}T+nKpV%!e|G+hzytvi*~=)rAX*^3O?>aZGLfPr2xQhEBQ5#Hc@$sm z;)@yno;+fl(0C{Cc7B?SNr-50esSm>)wI`tni_3Z8fB_X9Hu!OSDnM#f+DstnAKlP zOX;qC8lx3syUg7gV?!UqrkPZj*kpI*cil?|>s>)rk3e7MXhA(&N+MC;tT z$=_;a`}2byNvqdZ(>$pTsH5U8ymN)B!M>%Zho-+C0?OdQZfq4J?GXf@*yNp#kAE}X z$AAop2kuKqSiUBLfnxtF)x;(2x|;;iC&J^TGo2?FK+ITaU0aDZVTs;jV1O_N}d z)Filo3K)F-zZx7}{Ddm_WjwlY(eX{&@bQB#iGk=v$5=4TJ{rG)+L8DN#_;r0uMX~M z{lRri=&j~%S@&E1ts0|AtA#V1sJq(j>z0RmDpQhL1T;o0_OwWB6UJXC{ls9PhREZBx<>Gg*vYT;c{-s#*d+UQW zv?46a;?^O1CB-H1<@HKi`MkXm)alcSVY$SlalZA*n*mf?Dcx{ zH^~>j?jiWDCDQD$V-;mE?ElW2FQnHPFjk$OA+4&Y10!q z@UvP$_uzkT?bcVbDAKOjqOZfuCy^}m5Cx7#e`qtc*gkLaM>k%fw@w-Lw2`9>e^5r6 zf&{|4#=&`daH@>^_a?y|^$AcNWXDxZT(p&rw2 z#XP2fJ1F5Ju5F-ETUR}TxAqfN_6%f`_3b>nK~;N|wbJ64WAhU}++Udo`Q3#}s})AC z{_bk;{zgpDjU{aC;YLRnyKdD^R$v%bV!}ME7U6nl&pNo6H>=}pd<8fPYy`hbo0R#X zr;g+FSjh`HYotwj^rrvR{7IbR;bk*j(06_QcHD5mAnN%T%jc1D_+RSa-CY<_;P{Fd z<;l~nUD7YR{s!UB9~>q{_;u5ZiO<%a2j-Zo+7?Qg{kE)fEajmW3q<*^DsUK3`LMl<66g9R}reSNB=8oQ+*DWbM`LbeNah6uo zBmpxxMhB!im_7F`9rUs-yQ6UjI}(1RZhvL7M|{_tVl`oE%((XYlpEg+a(%%wEl7xp z{e5{G*G9w(1E#`dFQH#**#0Dlrr0`?DQzy@el1`Gf_2B5Z!usBH!B&170e3EAFnB| z==hq=B{Gk&vX3(x6>epMk0PqO`t;ZNbKT4cBl_mePc!er_DAl-COZ<|Zr_Wde4D=R zxFbg;$WF!O98;OTi_u+95x&Y%5aAl{B88`kM@!+H5l%)kl?&jVjXftKv;yAl*P6Ta zNHs@8Qf=w(?RY87Ym0ch`eBV8JS35#(<^(+cj4Ot_rtz~AbQ2fEa)_Z0&{cKvEf7N z(f1W(>s4l$w1n~%*k`%u8WaYed$lcKv!&_M_`e!V;e|9n~J)aH-z zz<^(gXtrC7K8&jQ|y^rAqZ~5zob0Lsa ziGOVw+JBo7yfV;K(h!@MqcI65mrNZRGROZ!oN{>`nfuV<#$`&ersG#omj6b{1u`-F zTopzGwd*6!6B6Et zyA2WkR$fYB(X2Z^`j?R@8!)nxpTP-mKvC9-Kn|GGe%kYB#Uu+#O#PG_*GBt31Z@$(t<>!wlExe+IF2+l_3OLwhSmW0PZD863GdJJF{^7F&%nwej{qdk=*k{_$zL6FWPKdGx>|ag^1j~Qvd;HL^EkI&wvytwNGgesy zL3=cma}~f?3u03$f`S9Fp+^y%L(Cq9Ki5CL`|Xq$_AR0!7GIy|l(;$xZ98p`(SE<~ zi2JN`6mVm$9}_A}&Edd0M4~!H&YxxR_1ju1^kvwp^T62k2U7}dU5sM!6+#1LyMljn zo3*Q-I(Mu#1%8k~J|HUH(`k#P8RN`}*wbn?gnNEk~?Ei4cd+V7BF z#oYc!LqxT7mH=N{d>WO;#x$J)uA)`<;M+UZrxWA z+b_%)n+#8&ckm46DbNJ#`QARX25L4a-#uF$Y(0{{5wpp4R_aI6T&n7%dPXkddXMio zdGNe~bR@QDQS_b$q5>2Ge(N6(=jA6xr*3t>EV;HIURk>yZW${G^Cn{WC;a+S+Uel~NSnRyp2cnhi&+BrGY+hd!^Ovb7YswTY^qQppd|kubFwCrI-smy%{8xWTSWVSk7_&aBBl{Q3q(bW7?uvoW+gh-P zfrs-~lYb*3rUP@EAxm<%%*;4W8C*1|y(&FTBX=SD^AB1l^P)vtrriA{$sXhH!zA?hL)1274q~IPXoq zfxYGqD9FhW^!vEff*F6Fo<;6Jyb3T!^`>iSsQQVLjpD6&EDs{{9b6i0L0M_EaY~UA1Wh}6 zRFfZPD`k87zIhEYq8xUrjH|hQRjk zERz-GvE+4s0iUwI4gv*EI9{w#{l@7#uU5N4dmvK(Ak_5CmQnpUpXacl;}Ktu zv~6Io>9Y!;;C?^;DP*Dy^S`3ReNdRv!&+(9CrFwALE5bQFLPr2JMPd08(1b9anNz?@nd9%$0j40wtb?y+^EfARJn!tb zDvfY>uo2m?9X_p|9=s99w<@V@fGJ3*7{E#AFh`2KKzNFeVjgTg15dE z?SyVf`j`b_p|JJc_9m`0(y4GSy#fv9dC5ek$iekHrnN&( zJlr6&(&$|{m3imDt(-G!T2lj78(N|2MEWa>5q9a;BIh|}wkez*|C6uqRnA6grI9(@ zzH7hTE??E%Z8nAh_O#rod_=pwFyf_S{6f&t3p=6;9C$}3XzfJVuH}UCtHoq8lHur{*;ik(49m(LVJt8@#+|tqo+VZpSGBIE{fbs)L|4OCrTKM98JBg-gd@B#*9zvbA%$lU z2DTgLK3m{8>mj*nwtYWa_h6)6oc|15R@J{+htSvEfZX%SKPq?ZUsyCPSXno&sJ1~f z*z)Z|*4V7b&U^+QE&dDYVmon3aMa8;G*$jB6!$+|9_w|zV^u5bAcdMB9)nro1}7-= ziq@<7SCQYQYE}>k181n`{zL6 z>DEk@6@`);rRZ%E2l{K4%0p?bjwC$>2_!-kMyQJ`qI=_>$TwnX zS5-5kyX+o$IFI_}t2DC;b~CDX3ojzca~p}JwQ~6ukRVR)dh`$J_HV*CnIlK$Nm^5T zEir4SBdS>3k^9eMccz`~afDoM8?`41SsJjLIG+tk``=igzaR)Y(xLH+Wvf|$?Xd99 zR(vhSqf+2WAuC+Ddgs)AVD)46*yC$T3XWyXGxd3t>1}Js-vDDRg`1<~Qx-EO@e?u6!ZCtK+}_Cl>m7EteW5-D}-Yct<=ps_V}QPC(7zb`cBY=gOkP z!_xc}ZvFIL->>;<{OP`viz`PkX7O)l{MXPmB$&z02D@E5r|@`3-CDf9iMB_lvDzl52_fed^}7@<`3Gp@>N~kR$>_V-`Gjj!MhXRSSLEloD!te zU~FK&t?3&D47S$6T3Oi&z>LtBcXq^OAs?2y&^v|p+f+F z+lj1zyx%1Ibr`WJ7tOl1*>3<7w_j!w`WL`ymgTfXP5BGq6Zx|XZ6wcRJhN$?{Y63B z(YKsitTWcT)GiB-w7n@I`}UBXQX_5#9|*1BGpI9Uedi*w$R4-G;QM9-Y+nyyGq5q9 z5mvgl-*xw2LYPF3?`%qOM&d^FW^+x+@LRiItH!D~HSdg#P^@Y$LcUo~EIhaMtXbGI z#Xvkv44^_(9h5V&wmfYW7!`E|ALPPssQhg)y79m91!mkg;O0r*o{#KeseQKo2G5IH ze!Ttk8a&HefHBHMP`L7Tog85b@;z|0!0Y&M zNQd9Lq2Z#Be5Q-xd2xqiPyZzH;Xf*JKR-u&E`>lgY!X?a?6$+hXF)*ne|Vtw-#Lv_ zONN3N=rcq-J%>ZdfOEVri^0js{As`mpS)4U49*O)2pr$3#jfrjXh{hJ2YYw_o2o3$|vDG+-0nRH6w{Zte>=3A>ilT6DxXpUdFV!LrDHw7;~oyz|XCwpeF zKjSt1)1j;~&xWN|AW^Z8;4W^itU+vkMk!;!AjipyyXzpL_H@}x_Qb7i+l+{(ZTW1) z{;-e8AvynSF&oXA>`Bpw)}iEzJ3TghLdpHvbbs=sXNYS%(=yjXU)fUlyJ86S*r%GU z%~2xOfkl}vTW1ga47r5U4#J0yfj+`}fQ+2+TwmU+%DPnnCQ`e_;*WG~`D!yB%Gpz7U!eybL^qb0-C?x<-n$Pb3SQd3SV z{#%{Q^S$B8LgNq`w{V4GqeuYcaU4+GQ{|+)b|vU43?wL6tcjDTv0(6%bND?UZtC45 z4%tS~4iMB6Ay8T5&uZ6JxamEpBAJ#&ZR#K(uKe zWF2Wspsp}X&OIY5HJj9T+hSD_b3dMXW#Ni4ajkqd(`X0(KqVhvydH5VVu=CWoL1CO zed2e>hhV2P2wpPc>?Nr8^L#Li5f|=v4AA^87}p2^(d1vV%@QHN@r zyA`0H;HKeM;h78d>c_V3anB#Y!o8fGqpdfp|%M)tB#5B6rrkw~v zAy|_f#PbC3@3|B;N-%~i?Hawf#8)uAKFfZuIt30hJgi?nXkv*Vcou2Oi@2KY%0M|n z^f_S+7nY!dE{?Mycou}@MNI$N!cAWnw84KecdP70CpT>Ge z00X(x65<6qd!KbH0C{A#cuK%3$(6@#GKZ}XsVx%0owbD^8GYS6mG{|(4xwXmRQ&7T~+)JV#`~bhlNWOE7(QkLdg2H67(KI5^ z(qAsO>%wAH3DZRwuZ$*KcvNm7b523Gx~}MoE^wRtbrtjPTrg8kxQxWy zxT;>~1u+(LmkT0f!rw25b@ZNIfPAqRxB!h#CAk2xuot|P8^~XmiJr%TOQ!6?F+(9P zN0y6*=OdR5GF&XPz1jNVy1E4{z~uM)kE!?Nv%3&%>v^Vh7GGUfhC&NatsXs1JC})% zZi$d`0$NVGngWDiRSU^-MJV@FJABB2NyTg{RNd`WCXTxDgCh~>v*)E88M(}n0y*GjemaZ>dFR&)fExrINrizrK^l=yUNx02Sz-qHG_qTI+hGKpiP_%Xsy`b^yhXml? zpsucn4|#RZMMh*7)WC|FdJ-VSTnM9l)y+u+kbdr(;2lxGdoEG}qHGCi z!Wg#7qT)qHA;)5wsLjgp^*jQMSNk4KkvN<^oqCE zan^5^3);f(`9MJ*K9gcpIik)x*?>+(pi{11HzCPRhZUzm2PB5wWHOr7i)*#VoSPY^ zaRboNXswH4^5W{I|MU>20RbBHfd*zTE;xSgluM}W7qBezOM&yA9@yEmyh@I+%ee-? zz)U6|Gk9?+LQM@nnz;Z0!H0V1d_FE0bc*#R%OwR+JX`D3rB8J+5_>%cyx6~v!0wI) z2+I4XwzYx)&I*fe}p|*1vDOajF$W{72_D)1KXto0o*K z_v2McqVrm(2Haz{ktmB$nsI&PP%C&HNB~3poYdb9f^*=?=iGIrM4_Ut;m zY6Z7LiQypgyS#|YPUEg5Iel2*7SPfRl(kED?jy>!rgUIJboOnL>WTpl z<}y&S0y_=TejKUpH-Ln$JVZRfy@GB)%D{&l*z7Y`chwCg*|8IsqaFd`qm#}${s=qv zb|vo6KoS7=<)RTduMM!>+H?T?Oeo&wBp7i9?95Kn0awm}c@p(R0E6$R zCX=M|n6Lduy0pM=4aE=O)uP2{^|luSLS@qs#jYlg(M4U%y(Vkgo`aLcZMB`X+O(wd z4;6Z#YN_rzy_(%Wi%509`{$mozUNv!d^7uX(ecVQIU9k~LG$+g0sJrGKp@VGMqH~) z3Nnzyf7Dk-rFjXQmUr!42keV~O*vS_)_(>3J&X?2iNZXSxlwm*gA{X^tX|uqlM%0e z#kCv%&hi{VoolrSF^$05{{$V%a1iX(=nTV8KB1dPcF*1asf&4>2?71N*;ZFQ@r@>b z2Z-G^3uTO`&14Di6A4&vy{>zPXF%1FBM*eiveVDIA5fOHTM&HWH>XB@a#Dc(x~{o1 z{O1vnU%J`pyyihfm%T6OFbHb5Kk4;xIs5n<^z^5BH@@){@K!iEFz`KXs{klgx^2q$ z&I#;ahSu*KPs;voptB+|u=2JF+G?Qd%7Re-@7!c*^1q{MZz^V@=(H1n;K$h6z_zQ` zgBOQE+lgKWKmT--m(R&tm5&~t5B?rl`7^oNaj8Kc#5t>IylSZPRd)P$i`NVG!V&?% zIr#iE{Leaw@W)R~+(V03DvL!1KI?w>w1J|KC9<;_HMggnJ^*pPtO`;PnYLcr?-H{G z|ELTFdPi(+%6#gklh|mrNIXAr8hv8b<9rB z$M*^g?f+YFe8Yc5xMyhSuL;W-U-rhWZ89KDs)Cf6-ITczX#TRc&rlG@l0?PK9gLYq%^)f#_-({sS54T9w<_AK$i59k#(^MXOy*{dP9D zIKI9%DDwjIKQpi)mJ>)4#8f0^fZ5|Mi5BaLzCnkBUe}I&Uj`9uhw8vSRv}()+D=DP zbXh&f_B&%yi7bA9q-P3d|J3z+RI_1ZK5x9r5kLB?`?xkE-VD~Ki!t-j&DYuVn3#;e zRuHEXeeyC4M>e}2?ufA9dqC!h}8$ffm@-v{6`pn2|U zFiHilCI_3g?PxqGCX+0vyaPK2wnXxN%7cKkFEI1n(f(W^3X?Qs0GPJ@GC zv+Jx4Hf-hz3^X&zDCNBdy56kKL12&uf*_Ir23)3kmV4>aHTWm( zdWbR^=KMxgsn4a;s4E``|7$=CxS)7qVj?y%F@AO0#up!JkB-?}ov1w9BHBg{j}$>1D8>>cI#p zHGjCXaG$f|n;7Ur6*}=22q*Ir(k&(bEx zDwmP<8D=vlU1&U^#|q*pO(VSwDaC2&T{~a=B-m1>Lh@c@i6nByY%9W_I0QFoGZq6Dw-Mr2RN5SIUzd()Qe+6T$HwnDb zjNYMYv(#E3)$`qnoiUAA1e1LaxuvkpYgGNMJMaSkvV7B;#vthG3^ENmnB9;!{0+|W zYhGs(Idy!E2o`JP;EI=RmYaq<^Tbv8j~ow89jiQm3-$;CM?VldE7ud-X1Sn5n7c1D z6;9PWSU~jvP8IaB=~c3g6y&#Jrk10B<*VGU}+T>T1 zY$(Rl+Zbh*k&J=(oIt#C4zxzGqeLu8x+V32`&pLx-8>TbUkxb;)Und2rl2Uo(ngBq z5zOQ-<2>??3V~6GhqziypWydIF?SmZu{sHljMSUt$HA`iK{Uw{&H_Sip^jPHD5LQM zrK*AGnamFwmZ;XB|2_(M&3i}q3EnpFf&QBRUQDEC@J8$LG_qxHRm_$iTj_PS%w6NW zjVF(wuW?8^JZ(8rO2V9}gs_R8GKy{fo@WcNjpb$yk32mbuWogJm`^Av*MP*F?S|F- zcxB)*Q-V7Jkj@abgmC4VUP57gOs}ObOg^)0pyj?XT<~5?B2H{Q@W)z`kj%9vt+(#N zyUA`gG*xq=&b;aJ%%~~Rw`T*Cr_xoFxccO)-Ayr*$Y162c*zm#^qMq(=)4k^giee& zyekCfv2$l{3g%7gQ!9RpHU2c~bQ>MAA-El))bx>nRuan+xqoKyi$$r zS>Xg^jPR9XBOD#D*ht1FP3Cz3&Uufq0YUy~g6TjoT8A2>s<%|%ngxce` zC=+U8MXTI)c^2BYP*abb;WeMkiU;f*gr`l{!r`y0m{-A$iABej^>p@}_qw7bk=wyeQk~F-MsMTAY^NQQFAR_3#@zZ(3DW+=g`CM$D zvi9(9?+v{bvU}ZjzEg;V8dOR#YGbwDq$EcMefAfXa`zT2i{7b-_of|M>ksr;%AWt} zl!BlbIo~Z@63?JdRX{$u&(}9WcVQ@|8{PVY*`K{MZaVnx-=@bX6DCXV{eU`!&-BYx z)+c{n+TJiAB4lu~pHz9P@0vE^YESn?bBv^$)F=43(lBM~jt@6*7cNiV$#){unjp~q ztkDZwHrnpB=%vbzove~K)h8^GY)7X>{fIoWC3HdO~=((baftVc@_nbH-BIQua+zYK9^SZk8 z+RZLft;GoEV~%vu9S$L%VCb*Oh(@|?MP2fa%BkfaYve`5VX8{|c_oj~P9e2Bc>^a_ z)7BFE-}{VhCBWQ4sHw^V#`>SuzOU*pyoCbQB9mOVWHK$_=Z7Z1zX*rUhmawCHw;fJ zP7(_SlFpp0OR~plt8}!}Y|(WG0?7U%igHP9%yGW&N!$3mIkz(bJSzxT8w(9)S0t&d zU+>m9`KnyULO#^w41~}x29&DpF-yu7MmeCh_TPMmc89MuK-AX*N{FiMUP;seWi;We zNV%kKC!s*r6uBpX8+OZt5@U!B##M~Rn2aWFE+xwL+xSvmc&>Dw>fTal>fjJ zh%^+-lav-hveI2?QI<0?N={bK0$T~jaJ{{scQiAKo_y=+ev__FF7S;R)i28l8NFyb zuA6`4bP5Yl2L^kreoKAN_L9yDqoAh^PdM7vl-|P)$UU(>(C{*q>pc&d;LejLxN@31 zyDpP8Z=hF!2bcvJeiLpu!qy?vPG|p)KbESx1nMKM2_>k(iJhd=sO}yJu(PK8Qk`%Q ztH{eg=JU@-$1K9vp|>lhY}UuugeQ@?F09KVKF#&-+8~lmuN7QJZ(t^qrLuk=Rd2r# zexD+C<}RhJ@g3UOlx^E@`XhL<$h!i1I(_%UWb;y@jtne8u4U75DiT=cl3a1%VwuP- zb8EduNM>7@@0M8{yw@PL(pxW6F)vrYT(=ogPbKNBOOImogkHDFP<=yIc*pOR#5K$7 z4_Q5+FPVUFLT+H!9h7AbRoOL5hahgQ<(?y6|E7h0+mXh<2B*-zDxPZO_k#~BUU;a--$)d1!SeG%aX}__b0tccg%>GipB#Z-uDt{umpX*zrqS zwv}4C_k_8ISXN?6Aq~y%j3x>xwV5M@i8`9~dYC!(V4}&){nuN#8+;G^p~q(l#12_=7#@<& zS(9JMUENCkEzsGReI5!u!s)l$-^zCWk(Q#S3hCZ@Mtq|?AyvLdU*J^qt#B$-%jPE2JH4uc-*;iH zfdk*aKp(R%BBdJ{ZO)K>0e6UojFOQdQZUi2r?$+4DbcoQnMWB!p5M8!kWD#vgw#EE zCVxf&k_rl#JC6L;?5vLU+esaXCzN_<@^<2|a-}$amOX9ih$HQV!tK0I@-zw{u)8<}0ygeB3J{={gPYchAaNfnc^3o(C&4(@$redY} zz1W)|io^$og5zy8)}fTtHomBfG;~ZMN4V?4&A06Pxzgf*S+d1Sew?SDEm;}-;Zg8R zWyEX!e=+Edm6SdCCotL`Cd=>F%&upScC5ORCyKmi<=$`!(967f>VS`AoD(41LDiR= zCuM$KTl0um$gGWXzoY+p#)fB=R?&huSuIP>U;*9!4+m*L_|3tU`J&QtOi8DUKVjlh ze4uZxT8Vga-LgSvh)~Pwn>0eEi&QZT++2Q8Lkk~E`^<@?7YN_SczL>{Up1%F3M&2F zg?M?*YPTB4&FvI@H&yfycT&>tL{YDGrq-G{8?>*TAFjBvos=AHj@ENfsNE|4i6h^Z zdHrEWsR!31rr;5<3Vsym^=;L1Eu$vwAuQ`grjqCJVPJmk@vh9~BYiz*i_NZT?kKSu z`{raUuJ(KF;`mSQnxoAznMTOcXC_;%j1{5Z5l38mqfh)C#CB?|p%HIgCmDv{9fb&# zT>HKP_EOnoa)7mV$lCZ;31XC4^Pq6<3d@p7Wa+!i;zUsqyVa2O=RGnx-~0J${j-L` zf_+%4x4tI8{dorR$UJY0=Ms(uALCXtYOcZF$TaQkvP~iZ`;5n)5t?$uYXg&$$;R%v zV)ul0H4{(nWU~}OJqIaIyJGMNbcJU*bD0u$LwIb`6CrX3AAa`F`$ zw{h6?V6{yT`d^>?^1j)4M9U&~J>;G(X9wv{xzQ>8nXqer*7lwRQWD8J3Zc<8bGKWo zbh`yRrs9)tc9t3x9lznX14rb&g%NY*S6466rCDdml~m}hb$cQx7zQI7t^`V&aEww0 zR}?42BPb}Tv)KYVh)r`=r8~5zrx8+#zfI#9yCC|L`zxy!D=#I+Mbo!3i*W%uphVF` zF@?$#+-+Kkew)xUp^D;_K7}rr+5~h zS{t7;HU6M32+IE579Y8k0l8VLP_dI7Z*tFWT4IOq%}<;h0C`b=jDLQ%?wwApKXI8S^O5j(_>M~9we#~fO*(Q{h@H!p-^IDL}1R2@wwP=d-$;n}s znmbnXc8!~Z37*GOtddy&wut-@XN7|R7Qz*z=pZXpA@-K&#>`LWGIN9L!qk%itz z2;Srhr+NR*QZNWRE#!VQ{4(W5d&j7Zj$}8*G_n~ukO(HLn9^r<-rKpObkE90s;vI< zMB~0adQT#&Txm5M{U9n5@roYd3zj>NqhbMhu-Kp} z={&9DrAq_*3xb<}^?{p{qf!_v7Lf<#=m!q@AH?*%p&vo8r-Z=ai2V>m=6~Y;zdiT( z|KT8IKli{_V&xuid;y%rJV2niE4$C&nNU|^cor0hCrbIW@&goAfA#;vsnLJ?u&HYN z!|Auu(%Nn7XstTSj506WLD?z3_V?~?t+A_Sc7j|+*+&CstMWBm{wbiQ`^%cA^~Jzg z)rqcZbvHCd9!8+*f^W#7eeZiEq4-;JES!*PdRj-ZY-_0_1A3J=zt(unf_K|Dkh|Os z8xORY?mLa|9VTiD5*3Fg*WCR+wO?((hoxO0&v!G+p?BYGQGgMCow27)me1^VLJi(N z#V&56$uS%w3a(Kdg~W`(oPn!RFV)5%X0g zD{v<_;rDg**XM9-E<+?svKl{d81XG6^*DL5YtB4(25>7eqJ*!$ZEEh|L3JAP`W-a63Cc>TwaWWY@(^2iOl6m~;mF7k2>($?3;>;Lv56j+TuH zjfsb>-VYzGftO^XLp#eABo29KP2alBsh(KzOo3&{*2??xYEg#M+sDeT;dg z*LGF^jxGGvA&=C9leAl~>$dAk(fZSeDhjGLViqsD4Apr8hpt7Tyeq;jW)pM-JJ(I6 z48Bmqm}l3$p28ZgGOCY6XdGWB;Uke({szrrr+(2>et%gAwq|Lb=k|0K`I%PSOG5O$ zYtC?-)=gy>F^g`IYt-LVqRv&N=uQ3SFn1*^ihdinL)wZ|(|$D9Nu9hgP}1Izlf4@i zq#SpyCfQT&0kGj0qqXdW+!8uYPGa1_*VRc?4=MwSrfg5t+DM1Wy(?6=s&>EDC&Ygi zg6$6xv9yq(J;6QiKzkC zAkM3JdYh^2Gv6=r9lpoN^qjOj*M8aUQ(E1uN}I<{1PIi0NLA((}w{ee5omdAkf2A~&1zSAb?!Cov~q z^zNMZvW>zznvG#5^HVFai$hmG0E=2$f&g&9DDSl^u2KfTbAF4o8~8fg3|v6bsAOaz zr-Z9*B81F;Bt11#-=GYcKSnQZ?+w*@>5EC7=1P7zPhC3jy87Q}pw=ofs#7!}s<|`( zyd-mlQj|~#*Y5sAL9G^mz4xlUyfpRDs1`cW)D&3l=@gLACUco?g;WUq z<`zODWL-y!Zzm0eF^qf(OxA2rZj#kMGFXX~X#wX`EhW`S@Xx0+z!)J5HDC-UJah|T zHQEn0FG;%l`z9XXCRNUgz_6)4p;&PY-(7T z0X<*fZyCvBz67|2UxiVnYXPQZ`$rSvrA4s4QD5~HsImd9W&rDaE~J(Kldw+%7V0P0 zq%0=m4Gro?0dcXghf} zHG-k?Mh!rpMultSC>-|NeqBEAE_hjn7-M+%0YGo?3t)R0`;1O;;(s*r!53WgvMY;? zfdiJ{E?Rc)K3COVAh^V8@C6&aY<3}7fFdeIEg)j9%=_O3jiy+rtG4EnbN;|Bz{1$1 z3(sYsV(p~pmt_<4HSAj;dU?llbZI6}dVR#kG_50SnkNPJ6VU6@3XckL}&4P^~qsv>$|s#Tv~U zDF8l^0zN_D3kbbz>*86FWn?$28m7lw=_2IDU!cMOxhd>JkriarYW#S{C1*02ePP~U zN9CZ?803EbO>K20Vw%rDw1I2p8k$RsL!yuv|K#{*<< z4AVPZp_+CbTmNs(Wg!Y$5$0E1atYTB;ml><;Q?xm5B9r6g51pVl6f*P@idp-16At< zH+Ny_8wH?%O*%0Y+xw?NRS8IkhrLwb7J_2rB1;+-De4aC@Gesh83}Cfa9oGLWxHcV z3co|TIPjD>TtaHhPyNngC4@9WG75$uxmoO`Tjqs|hs(&Q6~5Nw?{LFw9hXEbv4&oH zGOgu_C8_PY52_udayAivE&+Ed>H&^9W|@O8d$7Gpq6s@YX+OeH^>+m->!&Mj3&G;X zD;fF7%S@&E(+Mx*JOnQ&$zz_!l)RIIdY%BUFp1g7%iw)sQqZM~xloBUe5c3u0Jj^> zsZQ`<5*!osbq4yf$3sU2Nb6T1t(DM^IqWh*Wg+Oyk2F?KufW#}1;|IjFKfXS_Eue! z2^W0@e*KoIpqGg!(oW|R_XumKpeNH_{!gIQoCfpboe^CgKj3U3;H(wG_nrfbE!?FJ z^5ipL*Iwt`l=hp8X;9JtnO~L#uI|Keifw+N!GzYOit5H+07g2Z>JfSY2|D@+-%pW# zM}yDk^3(yq`g1@V-}(yGm)-b#j-xj*48~G)dG9Y5GE0nMR-SZs?oAXUWWWXWXXqkd zcpKjUAX;TUlSuoZ`6+Epfa*7>eBVRZQ$2u<1{J2@`a2gC#qfo8uC3J1b9d!FK`Gun zBp+$MT*36<=Dz`GD6 z0@v4wV6&<9qPjX4xsKc|PnkdWgm)5|@HEQL(h8W9qHvxUP^|L1bFu(9eEa{Z?YkeE zTADy}RZzr$qM#DWML|HsMhU$m5EL+!C<+P|nubsU5~@UkfPw;1l-{cmDG~xm30Ub> zA)(h$r3whVP4eCM7rd9B&Y82jQ+9TCc6Mi@cGE`JT9(9aXNvHa0op;}Vc}-pBBUlZ zpQ>~-17>-SL4y4xf8$Et66PIHHc<$$^6$_&6s8RLFuGTre^5&+S-S7^4tDw-pfevN zOe7deFLmTCVjjL{;pm+vd}rdw+YLXX3-uv)_rwJAJ_ho+54S)vAbI0EZc%PRl!~rFbU(JnY{w8x@F;I5t0m>%}sjYWgDXe3zOz|YUEV4%tcHh zm^MT?0hiopDQi@_2YMRN4T3VY?LK54c5f$QbW(FT=drpDEM@dXf zM`>pK#-l?jz>;Yk1caezFY-y^SNOmIKnZfHPWaf6iWN&gJwHQ#F;!s&4?CR+6wrlD zmN)IiRTLjj-PdwU9&}Dzr4PBe3y3TPh)e>0CI_H!0f{HdK8qOP0+xhT#t5;LcF}cW zLyX3;AkqPlm2%yOT-{sCmgNmJSlomt+yIDzVGOa8ygmQ>;DUZJjc60)S1#+Vfns02 z44uBo=0q8u(MkMTuCNQ86ZK4u)-qv=&0%%GR1yryKr%^qnc6uYtEK=B@8ogE=k==v{CbWS2@v^S8JXmSM1-7)A^ z9#Y0S&=6;k$fISs7FsL;e-2Wt`LhVyFdb1GW=3 zCEz7UYdsP;rU_lci1GG>Nk0OU9Nk^llkbn_A1&ZeRf-D5f|Q;5kkWl2^SnL#nGK8* z>bGdY(unoBy?Zljq0ceDGfS1)w}yH(7J`6hAPhHT9)YyCnIgX0xeIQ%k=K5tJ{MJT zVe;=k6*SAu)bnX_7BXy-;&spJU{zg7V3t-YCTU7f_K?G(p1U; zyX$gJYu?j$MLa}tKRWGYj>;1P zp~v?Qv;onj;A{$^VB)dD9e6;|Yq&c&lU ziOzfBoqCZtUf?xBIJdN0Hed^&*6 z!T((m-@5W`lUgdv8w#X;8g79(uV<5M1(LFk04ho}Sl|=z(I4gav~6-tFuG$$-b*^o zKlElvL>8TcUgW@L3f4j`i!IcR(wlt1j0&uJYOhfrpldhWqOk~IUuDZ_kWi;w28K=- z2RbMAB7lEn!Dqaw7uj%-P8MARSoC7TrLFjPezhe8%4^zKDz(rq($6M>Lz&mjP>KX%8`44mqFzUVagZzu&hHb@B10j@+_92}+$@E0ph*q$m&14W=L)K;RLqCo`^o>8w_%fsPv{DP6<# zw$9vrz(5d#VW(3_xadn`FH<)Zfy=1-w|*#Vic>*!ID=#XtdTsnJ4U|uMJN@@SuW^7 z&3mj^nXazrR%C3GtAo+K&LByK{7eR4PleqRI&c?eg;#m?LHi~H<#&%J;5Sj`tmoN;?A zg98HDMJtqV0s_JIy#eIi=Q2o-ul}A?J;z&x{xx}*i-F(|g1u@m1QdQ|QussG<7As1 zMdw@1UvU=(0*rfOd*Jfh-D0*r{xSEZ1jnx9)P!4tx|yUv&1TZo__jWAR`;DWaoN&)S?>_OEt zk|uGgo@#n$hC00$E%i(S$b&m8wqlU8gRGwkdrp#b>pmDM7x!uX2sV>%kdXG%98?x( zRW4nRM(RL2Oa(Qij6x~xAcZjaCzw$_oL~+NkMyZ%Bof-c=DIO#E7D1P#Q{H3_X!u> zt+>P-2MJvS2WcbcxBy40{$qb&2f5Hv!9;+(gZ1@B(<_h(9R8OhA7{%z7(a&XzYRQ9 z>I1-0+{C!&2KtUn#o7GYul^wdS~gC{fz$ASBzk-s6Iq)vr$3C_qkYN?yQMF?Hjkx}V!3IB zK|Gx?^0omkv#xP-rg<6I@XyeXS~k+W%~rKgKH*$Rr%503DjwEokMnp1q6j<*4})QQ zDCVa7YoVuVp#s`A(uW;l1_|QyJV~EH&X_^Yc(B+GDb~m6Cd_rk2x0IbrfvzStobEE z7Q1wmck@Xa_T6SY8Jgx)91hL|{I0qFbjNnQp*kKJpN6U!$n!SFr9C zcnll%1bi>h7d$$IFLP}N0OzUyegN=D%uv=4^wT!5ggt@5Pw`~!3496+`UU2SoCB^m ziD9)B3@dm-LGCAOXaD#Q-4O=j1X(-1F~YYg zsje&H`gPE-ey8(-(0m4t)Aw-H7JU}89yWJ=CXG;gD>RpWKa_TU;3Dzi%)Pkxffj$R zR52rKM3IVA@BiwOFMEp>U3feBr>eSDm|)W>^=DeDw67it>@ygt0)+NJHfe@M!3Y=tgyTnWx=U3Jv-6NG1=w+P31 zLL0uQ;zkJlJw)6T!4@oFn#%3&1YT6`+@9w(Fh=jIkOF=RzAfj^q(`hsdFli_Yw|$8kLKg$x;QF}e>zjJXwAOhFa);L1gNsg6hBv-0ezT~CHl%Nqg5^j1 z2n9J$YwcrwdWO;6MQgpZaRbG81H;_}BE|Jsne+!(_fvDto~_lle@*R37X9pSnSlP8^!_%yYhP`;o~jBs_qu)8)uZI-GELePEoF)3 z;T66|fniWPyT%va{jLPNPWJ-eaWvzNVf|9-pH?>8_#4fhY3+gBca8J=;* z;(>WGy|tKvpYY8ZQQQ)<%8Vo=QJv84K6$ra?MH?N`qdNQkZe74bw+I zEla1O7ep`74hFi*ZG@~lcx;b~TOJH=5fTvv#NSJ`kH^gt0=kG45Y;yl;~1BQR=y10 z{m@ZNWADY;G@~@3kGtSPljbMnn94jrKacPnnG0N_BcsEKPrB*giovuI^&%KjARM*_ zXMti=0v?@4-SIti*Y(<6*&8wA1iv0XGn;NRkH&z}db?DZ{cH`i2v+&tKINB6rGyMR z=f!p>QDqKbKLjp}JC~fXvKO|kRm6x5MSfQ&q26aU3^-o?Wlu;j#K>8YvoqQe`J9B5wCv7aj=Qp&U!MP{@YI;IRnqSx+~ zjALBz8I$D=&*648Ti1ZWM)Pn)MjXs*Yw1a<_n-IK@7&2^j-Z{w3t{?){<_{v`M6;B z=+FVT%`7sIFInt)0N`&1@W;i6KKA8)s>mt+!XYp2y>(rQN6BJdTiyP$XD80ymn*nG z=7H~tOsddcq1Bo-MXt_1oN0;L;$1GZUeLcpPtoVXYkuW!faWfP7^V$SPDC^YALF`z z^Y`;Dy#v3LciDJ57&)J7bk|+}UO*j`QpDiBbf;+OpI@NOUQ3_$VUO2gN*w#EU5*~R z&YK+!E$RAGZ0Vh*f6^zFZ1<0PjtsUNjG#Ty=%2rOE`^j%4?y8tGNZz{-NbY=9vHRi z41DIwYTeSIci#J5&&u&2A98!Ng?TO0pv~})nnF=!UXq7w^5r;Y#W5W|T^k>liWU?P zId@*FS~*Yga|0+2(cjBpI{^y4Xf$SH_GRDBoTJ;AjYD)q&-G_pcYbV96IT*tv4`Uh zz#k}~tOe1^y4=mr-V}NWhVqVJXGhcTs&342Zsm>b#Z8%98Q&io>s`lHV7>aVFAy#F z(a#B>5K9CoIsl5hd*I~t$4x`6VHq4-b);AK7Mf)~@(!EID!D3kqAOC)p**OcZS6&> zCJ8?gzTmD^r}zrDwX_}?Vj?ga+0A+?u-IOmj0z7&|V?vsd-_ z`(I}8y;o>;SOeU}9UV8kN!c_)*c0fNg&sUdl`D$pRC<&41NXM(N{=wCfNCg}>gWBz z+qM9G+^f)s{BliL`bSvFZg?CA+|iRl>QrZHfMf&o9q>w6?v+ih2hjTxu(1G~5`7Ao z4dcLIl-!&%jrJo8AM0nF;Wtm^*==gE|U4VRexLCoqKTHXwVtT(8x!>VrE z`i6q6plhk;9%EIP^voc2I)mz`LA1|kkZ`NyP4WogGUY1D`2*mX|MP%-wF3n&T`O8L z0HxO~kC(hL=_K|}19lpGa%)TnaMo*MZ3etsL*mx62$&FQ5?p?l2DO3KCl(CWmQs;hF;qDF+1!Ctj$qHeXJ(#lw^?ZmI6usqc!1ZVDrtM)!Nzi za0v7{rDOQ3l3LGI5E0NDG5SoaUh+r3ZO2kU|vZ3K2fClzs zgg>BxUEmoO(7=X9-UBqSAERsl4XnqSB1*ImlOG7CFo$%P#$*_xHj^uJ99VjBNYFtL z6S&W!_TOL$tVBAC--FahO0+hsmZ81A&9?-mY#>jsnV~L@Gld6{j(ab2oYikZp3i!S zjbx^vYjh6u5TCIgwK(rjBRE+E{0jpkZ=8+cDCacdIjd8&Od7k0f@~809?O?eU@9vF z5^~-G<;*mh5XgRe5V=cXAwKfO&?Z-zMb5SG6q0BAHJe;{5g|xhFu+l?hurwkF;g~^MhVQ(x&x*Ue zeQTv9a7fMkw?iuEbhATOI*9>_ht``AC>_xADJpMsWVW&<3CWh_`s0-8O z&xZD7!#g#Z>^h6RBYCcjds3`x7h@!O_!AUwdch2s?VSQ~XZZL|#r4{!y7qZU{+L*+ ziRba#uUWNp3uJP)KfXO2`o|)3|0(I*4#L#3sUM3Djc9CUl(52OSF7{G3&n3uVB_h<(YbEM25MuKdg_8qqw1tYXquw&@T1-hO@PRP~4#iMW9Z7a(T2zP>d#z(rdah_&q=>uVeLzszLV*1~2= zs&~!mP|&8$Eoim*r?C7F|M~u@s5%F`yz3L+#U9%9?-M?+Ag=?^`l_Wyl5X zu9oIV3eMcwmLIhW{tZIDKk<7k>Galp1&K@-_a$^(KNkP4gzFsx6DudrG@2@kb$$%Wf;ps@hkpX)mZPh+D=}CYZWH z>)I9YzY8gQUi+)Z3c3GDjD9`wX)NNgn*#&_Bl}HUS{Dl)I0MIb9e+Uf5xxL{{G@hu z#Y-%GslG9%;_IPmQ*j3;12R&dsrDeA8|=)?$6jAwiO)Zvu^aO9E7AM=a?8E5ac&N! ze?#j20sjW@zh8>At#6fRTz@kJ@^HA$TJwBW;y;kyJwOgYN@P$D#2+9K=B);hf5Dv< z|IdNVyWj~4ExDC30UD~RsuilfkpNSHL=!aITk`C=ISutZQiO)T((l)sDTDSCX)54i z0?3trD~5bpLsaRHEbl96bR1Qc?(S+Czz{&qw_5Jh-@Cg!G!vRk9EywGZQNrWob6ho zdb&*#I->8;AduS2`&JYwJ4;_F*ASHM*EE3UdZ0!e@`(O}2>*)&?$;RSt z36mS9&82*uKq4k-ciKKQh=3clAjVHC?A+!C|JHI#t{jaz-pj})2|QXHBcsh{+D;(|I z!%bdJeWy73GfmXC)qZjFz1^%|jhzNQh{Ext1j=z*=jA-mP6!W$eHlID z*R^~8gUX!u(Y2eJjOyt`m>zARPSzyR`E!VOOR2^h&0lRk za%5<{0ewxC_x;DC1*ayzMg?29Q!%v$+s-G^3l3C`eP`VN*17iYc9N$rdvQV>53kb~ z%;w$cc`av@Kl;5QMk=ZJ588|>T(7{5pEHg4roM9P7?yhVXa8Uue(>r=13R4RZb(U3 z9Lxmy1!i(0y<%=BiSy*1b^73+q1^P)dJban+>TU&>`JTV>!;*eUc?i097mwWD?Y)j zGC7D={L>b5#Pd%Lq({GXlm`u)&QmrmUTdF)lOv2z4b{!}gq=`7BSywo2s?zN0XjIL3+1a~th1G5G^w z^JaK6bmfeCnE=dnA5U{saxw>s`gNZ0rYBU@ZQP-FDaJ5+9_l{;Ma57-I!;0pY;|}a z7(B6JL!1icMil4IPJT)4dh?GRw&NEoh91MuEYzs?M`1Ud2S>_PBDgpUid_)iRo4-N)zhPNI z9rCT^VzN-PshaSfHN0wA)HTsf2mWEL!k$b69jsgn*Tma`FD)oUD8WWmoYJzv0 z!1#@LrbT(@|CdbSbTqgQn7D?6v_YGhjk%7cr6u!;wEXPHhin_0Taq;wvL)0poG+^! znzuS0e=}dv8@>e1o00cX@+I|89gmQ^W56CPUI>%KH@(ykJ8B&kT?Nd!pazptJN%oq*yt5P97JvS3`B3hs zg-|zD_UxboI=Jo3S0TqTE7jG6L!AWLW=*a`aL1mti?rT%LnU&d*>*36LcH*tuBpZ8 z*XyCC=_4Z7N1#`i>UaD7uhZS|?bP)LioPPro}SyDUV?A9JQw{0{KxMq8dt9^eKW@z zBsq8|GZ;$X-b~qbv)5j4mEj}}>EXX&upB$`$L?=^Eqh?vxKXdm zN36+fxO`&+gx&GBKbex0m<)&AAF9t*@V;azFZz%@B1vRzj#b-7JwKEJUsM)@dkLAg zkTMK%S=JD~EtN3sdbxHs#gorq<3OhPg?fDYAJHo;3(_VAZ)_2^5Wll(K{;5!t_;p6 zaInP$uD63|ftz+95H}3?Gpc&#q;U{jAN2nNSQ57*dMM-c_ZlVnR)FxZr~Wlh%-#Dc jR`=XJoLtcmGrdc)7tSh}D_Ykwr+XuP6Frj7?co0dT>kJ^ literal 0 HcmV?d00001 diff --git a/sections/0_causality/figures/DAG_open_data_cost_savings-2.png b/sections/0_causality/figures/DAG_open_data_cost_savings-2.png new file mode 100644 index 0000000000000000000000000000000000000000..7ff3c1af66db734ff53130d2ea3d9f7b21c0c798 GIT binary patch literal 27685 zcmb@u2{@GD_cuNgDu#-PLC8`ON*b~yB3V<&z9hRb_MMoFgi?sC*|QTd_GOTL-_2lb z!`SzIybr$b?|QH6cm3b%egFUeTvyX`-_N9{7vx-3xVP5XgfY1p4q11UdkwKFota&VnEi!3YF;5)A?| zz~jnaiGx5F4Ahj~JU{vW_8A`^Ut3!Pz5zB^EViMcp{J(@xW?gd9UUF3tE>I}{msqI z|6NGU*=KHU4uip*U;jUS*VWaX-&$T?9vmD5=)1bQ09^m;M^F#|c=~B|Ni&y zA8@g;vGMBFD?k(AIyxFK0jRaL^}nn1Uo$EyDhdh;+S=N#uC4%wnVFg3;NXRYg`uIL z_V)JwB0Ik@H8lk;05S>+3IINEeSCa;baX@{68HD_0nt-aQ{TRQv$waGmX-$8C@d^Y zN=oYO?X9n`H#Rl~lwo6IyL$C10Kw1C4^VP;cJ}b_aAafzjYb2&2?RoScQ;_qot+)P zl7Nv{R#pHtCMG5}H#hh8_67z90Gl^8HI0ppwY0RfwY33i;5L?9=O+#h4njLCwzs!` z{`?7Gk(HIz(9o!;s4z1#2N-Qz** z;8OGckw)EmRFmE{*$iin3fG_QSt~-=Pg@#hxrE)1W}hl#u^B5I9a$Pe?NL9Y{{3^! zxHb`6tj3lx_r-YwJ6Ixhz*bSao{_sXD0nzAX|(l~&i_j-Ae+QQQC=4GiHzk1Ft`kV zq3sCF@tyr~{ImdUA`pd>g7R~Ud8+GIZm_6Ps7QlA7DxqI={Ihp%T?|OjAnVF{9P53 zf5{EsW@Jld7vFY16P@~WJLubWdFB>%Gdazr7EYBZF3-4zk;8G!N)~~T=D~Ph;509>!4NchO1pqm$<9~uF9GLz&5H!&ciOBfJ6pb3>9q}BvZ`hguz`j!e< zeDQy`xJ{dpNLc86qN%)+)H7&gzhIBmb^`v;6NP8GuZUMHU7n&a2s{%Zpkwj~rcRGQ{A*_az}QrPbmN*xTiD7WaZ7%R_= zh%F-{mt&rw(k(~9ZEc9I#vmqTW4I=@YC{CLR#M{La<{T+JqMp_|OQu`-3xa6~4 z=}WClkVDjiL#%98LF6EVVtG>kj`n;9%2=A-%@-$NVGLcel9$R&cZc%XncnjfWV#{s zOtW0`eHxj0iZ?x6rYV-;;8796O2fM^ad`NPMByG@8y@+Y&6N>R?5BUj`b{sGEYm2| z9_=6D?fdh+X}PiU!@G{3)wyaqUNSwHB*%Yn%hL`Fw!!*6;b8f}Z>+V743G@%8{BzW znKX4@qwMALu!<1lnDMN)DP~x;I;LqPS>Y%+#X(H&oCA*h@AjMuCEr$y9DzA4ly7;) zvgr!c=F;Kh;002O3XEVqm4X^$33p)i~Y&sF6XYwNTW6ia*j={{_jPVQ+1 zWn7AN6a;B0Pll>E9>cr{&v->@qYRFU{Dpg-l2PbX-dPriFj&s3Di2SsaOvDEy(1gVL1g7k(Q4|XU=JQ|F- zm@2U@fpqGgxGz^FzZ+y=3UEB0$5e@1W~;BZ`hF)oIJ(3;=7gfdQqq0rZ2nwb1z$QsPL`vkx;Kp7%3`Su)f%65@mS!&BiwAqvt9`9+QM*jp*P06|-wJGlPIuigJ5$erW1j8%#9=QY z4bSJn_{+RGX~gh~_=4#9>K~Z=WN@?8;V?x^g(K1~_?YA4YDWm9KIQP+!#Sgzh*|E6 z2330Rpoa95+21P8%ASw)wrXQPhm9&OjML^r%J<8A2r8&|v#v5*kK*epU%LmIOfZS_ zu5$hdQM`okaDq+lmlFSc%Mr*8CAsXBkwkUOq@k+O=^#Q&d-1qEy&v&}c_7o<%la*7rfsK+7IZHmIDvy51jBk5zfA0jfR)I zE;yrWv!_{(-x^1chY&3zXc7GO=Hnla%(5W8HOrX9oXLBaJF;kPu%q2<2jTHKSxuVA zaD`F2_iiCVz11^I`IWUC0j1d_i5_Ee2Ng%lDVH7MJb4COVzntI`QO`$4aaS?y7qvx zi2*fDV?>aF?YsOhBCNRwcGqj#X5?Ou#W|MjbbD?-m7>&Z`@pTqGaX3FMId*N3h8I8 z$MTm&&s5Wokj?Q~yF_CS-Nbi1;KwU5gY2P*e^aM;rUfgd zb`A*JK|T$I->XGa&0j;!4+g2f(n3?nvT+Wr1k)pUvMiR(mkJBU!Fc$%%ksIfzl7O{ zOKCI_U0|KI`du?G%+)`%D(dMSTdS!F=|WdZ?PuQH%r*HFE56QSqq-y=Fa-Wf`mfNi z>fR(0zS}7>tKfn*s)jiD{t3`juydjN^0>(Z&om;E*^xu@xN6afV&cV)R&_TAqEw7j za+!y_QqfU{&|;J+mT3@n#rL{BuG$vB^b%*&z3jqi6B zb(UWYe0MFD+CzNGo%wY;sXjqlrtPV*XCe5)JVwmwR$Z1Y`SiF42GkMBQ0BMlmSn#di3S#HD|;Tp!Usn?T`Y% z-hwYG@BewaKsxIr<2Im(^Awsvh6c^I1z@j=PGb8NRkMlysU8Xn(7ltpz1Juh=$P8O zL}+-_rQ#nj#2<#=FMiqf3VP95-FN@njTR@wu4PI6k0Pv^*SB&=Nh1R{pb~eMmTS?@ z(|tTc#hqpskxmF#)VKsRCv!wIQ5i3A*D$z1^rLy-X=JcooaiZ^M24^w{f5kZO{y^V z96vQI4!Jxt_M^{-)Zw-z?2;~hHS{&?YG*g|s%g2juR_EK`@9O=y6nEUAF)ND^=8Gw zatD^Hc<4Dsu)DK7x=(4VZ?9sPaKzj6g{OByo|w@Bml@>IwTRp@OPiH1QD%jTIJ^xX zv>;Nn)HJUxdkB2r$TK1#wR>#JW8V-}9e!YE5V|5sRMRI`m*bIpp)5_ENri7V!c>cb zrD6+38>Q?n8gAE{CF)bzW%1X!edN+RIEio0O_IgU1!<7PVbt!Xy`4O${dW zCc|$Sq`4IhprpO;PK_{bO1(&U-ZYW-m9Gdh8MiS-G|^Jm;+e%mUv6oxXB^HUjTxb) zHY)pFy2LQu*ns{(rew(6vfZ`M<;EgFdi1{Up3r-={8$eImhpb@10^lh^{CPlo&`yb zk{y1NArhEF_=g$i?q1}2=4nE6uQIO^Qpp&G&9R)&NXnoN)V&L9K5V&?r~YTe3iC_S z)J@%{Nz1_WTT3Z}1A61Ll=jN8QIRWGNoAr+=0W1&|hU*lT`ZJ|`(Yr6lI(Rwq&nt{Zd82_ZB^9A_V?{j=sN=KeSc!|e z-e-sjY50p?;c4Eksu%0a-wgdLlUu6Vss`1+#6S8oCfkKdbX|G6i}~dfyk7M}1i@Jf zdH6VE5C4;+UwG$nAY_WeH8tOzNTfG5hPJP^_%C|ga7Gk+Q&vbHj8mcN%GyWd3ev^u zDQ-URl8Q1&E=&+n<-2aL_CR5~X9)i{%8RoG_FJMNrgBYRN1plBkdh|iRg+S2DWBax zLswW~t}V1`Kgs24{9^bGJ+8u$VtYln%wmMz)}4)GFxpUYM=Qbe+tk=J1-$BhGo->;F|# zMSQ2)a&GLnTxVgpm34-XPf=8b-H?jXQV`^gU6sUGF~pisa;5Hdc*oy$w}OS|mKzo9 z){ZZIH8YOgv@?vSm^J_BK9>HUppP}rKo5S}D~X1DT+L;K%68k;7haRhjqDga#%sPd zy#9w` z8>^SLXTMY!yL
?f;=ZlcLBVShyfq*e(P!-G0g_kq*v=mduny$hlsx}lnWB0u>s zcklFUmve31cHKd7!_#^gteaiYlhWsRH~<3=y)*x%>aQ!>BGBVI*JASN2g-RjvSU9u z!b5GrxpTh#Zg2-RNpa#oAP^5MD>bO)o`j3S*77d0E~)jp==Q)|f9!_8&rxSi8j1Mw zWcU%UCH`*i5;DWhZt<4*=J_#wU0>c!s{phsXYxg)x76Au6bO?+Wgd_H9C=u@?ah;C zpQjsMXK$t(s#oLoV}FcYxB+z#A=UC)JcM6STg?+Bz@Y1TSp-@+B`3sfDCd*XYFv;m>H&ntLip#}96lC?v zj&!Y7?%gl1!$hr^#rgd$%!VQV&`Q00yT{8w|8shIQ_JbyV6D;Zl&a9JfOeNK=`3qf z;7}2fFH&e9pK;~L?6Xa0bK-#XMz?lvlbf>pJ6#h#O_ylPg*ksnjI6$y^?M$N5XsIO zeBSSn7o>n9eh{nLGS08ARa87R^#!LDgz(>{2QJ=4b{(?KG zB!#9B7vs4?!|HaSL(W_p0_Sy790SN$ZCTbq$4u)<|Imp)J3t`_W^v_mj9>m>)| z{@)eR{$Et!(j}pwr;zhSLC!m;j z_y3SI%MsnzO+u`eQro;h^f50zq#)Kw?FB0NzM%w|m9Co~2NEZx9*y}C3l;|ENxGuP zcWXKLB`dZtX&wygHt@F_2y^}zVrBQ^1m6Qu*H!nrtr*bOr(l6jAE&N?v3F7#Uv+%f z+Begsgo;1m(YJ%mmGdc|`tIF6DcH>REL0Iax?OWL)A!=lW@A)0>C99>Mm^SSzevi& z!x?l?f9v>NqCf@KIh1mL+SAi>p5*gpdmD~*Guv==`FPzUYNtO$YF)Qxn`fXVO3Bv+ z6kfUZb!FL`Xo;QPmimidTNJ1cna9m$2>c`-89iV~r0!T-;Hi04(Vdv}cW1v?rLx<$zP={4Xl2H zRyS}LHSv2&ljysX-F&OL(eH<&V>)BplNtOW?k=orKos`@0utvht3eAq6YcL6GxS(! z{B%+BcqYn^RWfwJPe}K$ee>WE0!nfm`3pU z=4&1b0X@TnN4{xKB2r-jg1cn53+U$kCS_`_IlvNEtt%Rel&QGwNk23d`7skZYen0) z*7V-c9L`J~e2H`HQ*D??@Y^h?N3uP@|FBn!tK3ojX6B#$B%@n{+ZE9 z@(*`9OeZg@Z$IVFqq!tnIw9)-11AF29+1hS2tQhecqs286jIYoLt3U zPmd-H!22lCF}xNGy53=a%FCI_`fS z2Qj`oV;x^#5*G_p>|fLwk*nyjIM}n)Ri96wEzX!wG&h$?ymz{wLkb0@nuw6{Nb~1m zhku^QUAWm*s}L4!%{w@U;e?acGV?W`joIn(XJhfD^YIPM|BQ25&hAtGI3MfrwVsZP zZXG(F!ZniAJhO?mGUOaSY1>2i)V=S)MG-DDrwpIc51tde&^;%hr4)i+pP5=pmN++J zXyh5FlJ2ZC49$LNXJh*AKj&ja)!A6iQomFA0u(e;M>sXQ$m?Q3JhvYN!kVwzI7K2w=ZAom>NhsS4-p2Bp#pPE@qV7c?m%}2%D zXC~w4C-hO-wx7*iD86$xN4^y|7PWjn$5G6CHm7dTP5v(TJVZE(`OoI=J^6lS?f?tp zGj-hgt5fk9pT4OxPVY z@p-@z)wd{nW(m5CfiugLmA+v(_XUYg!YPm_#Fum5auiFQG2xFtS1w@w*%=cimUEy1 zYoxvR&%!~mclk__w;{p$XZ{ZE-aelQQ@kwdloiFvz+iJ(0>7jEA5pLP%e-Zv4^KVG zzjgb}()8&-cm`GDrw+cIsT@4rk|YMq5a*^E;$k0nbpJfZDop`kf9_x(zaZxkaYp%m zZKk`BDo(b`!Ct*nS?qbX?pbK4eaJMtL-O9hFea>X)~a_IOkVaOsNxznGwT8keXdIsZ8vIOy!Frl*0qz3XaIv|7oL-=B#V`y49+8l zY>ShW$A10i@XuU|`|XYIXD|jUZf82ak@f|Ql39jwxp>Rte5XftX8^2uyQ-P{WONM*hr&dpPVjk1E3fqVzr^X@0LaTxGds^sCpXj;_*ZT?z#@T%pcR5j&$x9CP*&pku#7~SVfrBbMic@ zeMkGzN2HEo!GNQ!0;%-4X1W^7`ly^N-UEi(mLbpM*uYN_3w@8iFGSix0c(&n*J{Y! z-VG+jfVwZ4PTrdd@a52y%-Zv@hXZbTbdeF2W zarGg*@!Bdr-E|8$8LX%04DjV)kc!eo|O7GfMT&%AYG&fMS zEK!$u*q9Xf1~6Vqm|vSRCth>S2e!WoJd34)ciNAD@bltyfED>1Q+i)`lC9X~TO+c? zcFG4yrAG~HThUegj}ssJnW?E$AZ0Ty@}`sOExCX?^bZU|t{y1VM}tJd#IOEr*hT{kFEFA+ zf@d+Gzc)8}1G{-~oA2uR?EKuFZ;re#t@yu8B2L!ovw}tiq1WLfx_V7sX%NE3I(n-@Kh6kT+_$D?|E%(mcI}9z2X#i1s&nA_g z1fZm~G`|Y)OuU)e#fV~Zggq~dX7Um_G7(J=tiE^)I3_B1#QJ4R0Gu4`Ujf5c2S5F+ zx$&A2S_#|?3;?5$UNn_n_!*o{0KD0PE_vR1obIdUMo~aL^wvF6^HcIIJw0E48lb7* zQ4|K2PQfUG0G4b?G6Vt~!ejg@Y&<%$W|9};bk}dFbToSREa+jfKTA0ZPTMUndnXZ( zGx~+{e;NpU7-u)13{7#;IOZ&_)eDMue6Zk+0HpTSkxfs~62MB5+)Z#yYI zkf^rw_wy(htIV4iyny zTZuDc=~`Kr@)xYfl27G@1$*94NAjj?UHa99dJw335?ew&n{Tjv3rJFdJ9t3mzMdZ6 z^wScOTF#f$iL8X0mdEjp9v!RCMt=hFQtD`*FW0~JV_|?~UbKt<{w+D0;nCM`JpS=Y z6pH|!;Kxtp5j=5X=|kkdR*!@dOSg~0pz*JD4XnL^@X+sQtV;Q6Am(f=@PU3qWeQVQ zLKj(C#CoEg+t51-s8{|o*fS-7;tq&qHol5;`OZE49mT7};%99@>?|1ZtHg$`JQKh5 ztMOA*p{4xMvOJNnB1YS-4`lW<>oOD+p}&^3pg}SZ&DW2_6bvDC?A${3_cK^pf%~%j zE1$g$9Sg)th^_ZQw)nr+c3jwQf5UX2hey6+xqxu`_QtAm*MTRZM+M6AJRp7i@~uX! z(Mj2Ckj^kZ9^mo*n;yo7R)>c|3WEKT#r+iTzO1uk$~7jGxz4yn4fso4gdw9Tc(4Wc zxuW$W1jL7P=%@aXD1B24?VI81RTXV^^=?G4Bi`FK94!_R0pgq95Ibl<3=GvOCAcnj zJ#rENEC?JQq$Q|PM6=>vW=1EHstSo-82qy;kuls z4iJ4U_+z@xMQJcWyZ2`dhyYXk)DML6E>a_0Q7Sd{qS5WBV}ye>1$8ph4$^rYMz3TKoDaKxk-%{; zv_%+Ee#Eew#x^u`f4@3^L;1pRd+d*6xp?jQQZqTXNQW)|j~;u^nB2A1wLIci8EUQg z4U6tFAJ2Wwt9mCtF<@1xhsmEUkGt_+mRIuv=$AQ=(PmZqxNY->okW19Eai=^w9fgO zx>sVLr>grs22ZG1s5YO~;98;1jy+pidJIuYb|chqcRT)}INBcx&|_ujSC%M!Z30v^ zcAOZGC+jn`*B5{9(dP1DjR|ju%D8Z{rg01JH90iUjkD~VaM~QUXEf~7trJ8g!QiB+ z8!1&)*;3^@o!O40K$ZV$da1uLTn0Ny{(@_$rS^ZFxI+OG{J4;7bQ7ZnJWE^sAXLU4 zyf?|A{b1M!he>jo=B8FWnts&e% zCEbOSM%;#@9ftP3F-9ckV^S#4eDySUlKfQgMc4}Nek1@Kh0R8R%{c#H%ZvFVv?=?5c z4zQx4ylYkjDa_&%2W$<`(pvOK;^&gpiNF(BHSS33&f+NV%4mga|J1r5$zURhU^pDXn5XJ^l+0w8w~v+Su?HU z5VO!oi86ZVn|f!wX+W9Fr!t&c|I^`jiAn?SAMIrPgRU~m)2w}kW!`DXoOq=guRR^! zryD-qS1jXQ2QKRLodWApclK-N8iybDEjaD8XNCt3Gn~pcn?o|@cV+Y4utyt-Fjss< zrt~yzY`8g^|0!A!|NBR5bewgW@F1X0#bZW0K7+H>cu))h#JMJH*Sh*wdlERW(fu=C zU#Rt6>3p^JBYd(Y;^DVRWVq43x$6<_`bykI-zy^5PIYT?Mz>SjrB|nZf|#j-5%SY) zBjVB917j=b+^5uahNX?+%B};6E-usd^VsFdCrmKan(G}(UBpUO?eM+N4L_=@2*h_a zkm|c4MPp=8@skUD&54u;y)=2gcUFxaKrQ|)7n&pFt-Rn%I|tJH@&RlsajeRA7xfR9 z$mCtQp?1s@ROK7uuwj~nSYf!ehf^~0Hf)D~0oecqgwfF5HmMrC<7|8uwxC34M2AIm zNS29)JG_T*r&%?-(_NnBhz6Nu$0|T+a}d94BvJk`9%qz;WFCVM7m+wM(;fVLOgn3P zC0)*^lC9jNqP1Kgv54ptj)M)m2Ax;hSEeoU;|R!qyBy!j++l8U13mvnTj4Thj`mRT zs*98l2P*m!j~)d=R7fcs#zZf(Vt=y!cK9O6hQb-TEX;gyP5#DYg&N3A2@(vzv7y+# zRajkU$9Flb&-T3P_MV0rLke|z1asX2P_+cDxuyR>rJ z>0o*}LL0xf!JMMT8aw{Z8MA6-)*(jN)n(zt{K{=M8{SFZ#kdH5FB}8kqmGvxgRdS2 zs%)@(1@kbess(FJDAf;J;RT?oARg+*Ltyb~V0E{C~GQ%4o~?hcTIcFI7ksY?pzL~^&MLmPHCKRK`_W+3Rv@AtXT-T(eYud zpyCOsiM0gNDb_e&Z=$T{w;AL}3J-pF6`NSKg2NW{B^>Wo$kX)4B3u=f??ExH)uE1= zx$dyAnAqcOBD%$uhSi&)k_7rjt`{sM`8kC3<37@K)$?|AKsWYRr>pb9y0=83%MA7P zVme)ID084{S{Y7cpC6+}rT`W3=nht-tfSWtZ&`!>qv*9x*m(2r?v z{-8oiv0Qq4I_TYgnnHJvJO(}&3Z6oThg;a`H-N3Co;-GhJMi{_4-}oCM~xD{zw!zL~}D-Zowk$bjm4MLI04xhlxO=5M|{lmi_qhEYnA@_a$3<#IVRN-8`0Zph^ zM$}@>uHoYo=E0w2)zp})%sbfuUa_O8LiJZox{P%#VOL{c!}hrK_+b_yyS^KnyWe4N zV~%GvD;MKKDUb4?Q{**$`Q54ycFKkP|iR7 z!DTq3>j|qKPC|M(I_L|59_*U=P7?Ep<7b9(k7HZkm*xD>RD)|WCc<})KSIAzM@=f< zqAGTur2?9Oge;OYYz2`>F$2)T#ReG1}XoXBP;yz-iMJ<`*GkoB0n*#(^w-(&NukcE43$psN} zhkg@GldOModv($WIB`jQ!@?&deniEFIIlkgx}3sd`!p$`Z&Z50L7!x%h+rr!DaAnR zt344X^nKOahtF87*rCiwlPBAKP7AQGvt&0ph6XkV2a>_x1ju5_Jq4G!R1I zOz=JwVGLD4NpH`53tLZ;BulDbuN*+<&d{xMz~Fh6xCCC$U;6jN8^L!4*6Y9;z)?%L z{+QXy9zp29cRt9GJ*ikuAg@SnjNay=ig@C(UHD>7ueiKvd`{4zjJpCG<{^>B0Yl|u z)r)f=A4}i}VOh*tPesjDmjZ?csHH|4TI^upb9xL9jLaRM(j1x)FHecWt!awn+b7@W zAs{EI#v-EjCXx3(=?P75Yo+mLu>$8(-{F&dJ|3p85pl=_+NM#sjG=$4C@=OS;=?}d zX0Un3=rTFJl6Apfc7i8seR6W7mx zKQ`$qRLG7LN5j}>A5(7wJx3M^86P8!H~e4fhUcz5V?r2>i;Ab+^-@XV_fAsfHUA3v zxIo>IkmYnM@-$_@QZt5WI;{Q(_( zTpBLF4!pVy&&9?3`4=N}vLspa#Ip}xSy^2o8ibMt-{FXQ-zpjC{Lq21WSV&}Uw!p( z$L+czjd;-ZFfrwy&IM%`#6$!pb%ng_%*Z(86MO6$v9dyr68829{K;5L?mIm3lIvDn zan@bz#y<`n7E39&UWnF=lg!6V7w0wfn&tiBl2s|!+uK~Iq?h-qOI1mreEF*%C+s$? z6mZ^5%t!1^$elMc!-mc~W4$rWu;Bs7gpPF}q`FriYGxJ|nY>JJ3-t8k)E9)_Q6f3B);F6%j+86_+N zUYSs921D)Qp07geXU0+ZIVayN=cCuFEr_kCN#vEX@9&WZ7{SA@zAO@!njY_=R1a;e zp%(D?LSemI!1Ed+q}1dk(=R^$0qp@5(aQ0p(=#}XrC51ZDd>rKY(g5A=tx2hQ_X1cl)gn!Z3U9|fIU|ml_N%t1X6*m zBt>Ui^+A(6(AL+2@HE6$tw#ToDrblZK}p-RSuR;`r{||+vk3!f=S1$vX?$;{i44RY$83gf|j$^L(0G}V_Y48HQTXn&XlvU~iurLl0H zAmFSbhhZz3jk8$H)6Usth(UvaGJ@I{>T=`M#N~44i?>H7k@?l4G$v%D*j`r8&OQFP zjmQn~Zk1NNuOn-Gu*(Ed&|$G#HoV6R(f`Aj^497K%p;F|gc&AC1=Iwb`oJFZxf(Dv z4;1jzl4vVVfUhoY9@alsPZ1~X9ErOGDSOF@xA*vKc)YO!rNoZqOqQrMg{L z0bYHWmuoLm3OpQvG6FM;6T0=wlie1UCm|Z@_3_@d;Kv^zF+iO{^vOJO_Qvi@PqFb( zw#p#O^BMq9bw~+@MS~eM5o-&rX$oy9|>9oXBzoiM-{frbS!U$H~ph7?hez#krJ$;Hv;-=#IT)?UC{>D!rXD%W5||n8h**y6=&ex-8DgU zjnB-sSl2`HqA;z5D;;o$=?THAC*8LU*5tAii3eX}3~t={(r3@C%H8RTQlmVXGj;Ho zTp{g`w3(g~5(qRCDHr~!M9kHR!tU%Pc=UX|5B%2vc57;0~$_b0*OH?Q`;&Cy3_;?|dk=#G>$ zyw$NklM_0CGJlm!ugqaK20fNveeh*-vh{B{1I^~8MTMe#{94=f*VJ0R76(PAoc8LL zYy%0OBFew(SN>8KVotdCK9u}G)gjB3bSxa_@R@ns$ba-^i$`xr20eCpe01zx zech$muGbq`4E>Qgz-x_V&ps-7+-rfMqsB)oNXjTf2JG5UqVbv)d1k^J6LUA^cuS)L zq3!UEePl4Og8jRrV+JEE?xU*2_Dd{pcO`@)gEzoS6vAd7$xqt6o!U6PC-r868}XDj z!j;sSOYy}VhTByIZHY`X7Z|9a=EktK;G$M|ny7*9S_#Xh8=ay;&u_p003=>Gs|Z#0cWW6pfiH_5*>!Xj zJbf{+ol5ERF+hk?Ne!q;@iIQ|DvO^Ng|1p#}Us>KRH|bJ{E!aXt&>q1VHxD_v_EOZoioWDCN2x zVH7QKx_y0xO_;0S&=ZG$H@nOYt;YO<0zER55}fM96oZ5!+tuO$4TQTkyO@B!j%Y`O zp|=`EZX!R6*PQr)GwS}21QROV+s^&rW*1xJKELdEopwqdjEjr?C=LC1*)qfWU_)p? zahb&@+3qTN;kKHh#DYJ>%dyPr;Py_~OYBqqOzDIN@s8nw^GG#n1L^6$;|YO}g}gQ(t~xr2Zne}JuU|`8B3*v! z=fw|;XjJ%>tCPwKWSHN>a~t=A84R2~zCpLI4fsLt=c@GC=!h57*KZNJFubSbOV-)L}uq!l_HFoz{nulnG}$y&0e z+hKpA8yk(*UE#rSe$I`M{1{@fcMAQf4rFbB4yV!HQTq2}p8NQtxZZ5Pw(1q39?ASz zBFa03^^}Tr5uj?qUX{tO{T7%V{RHqV2Ty?mQ}$(o1F3VGRUy#*V;l=HJ4`CUHKR3N zRBfk@6yQQ7$f@NrI{=05$u^h8Gj)|I4C9gWj5Ra3tmJWzUNYn1+S(*zf+N7p!eQ=BCVfe1BwPG}$faTn^Zm4Co1qJ#A!K0vs~m94xMqanRDFYZXZJ8>oIizi)7Q|G8XeM}3hEpNotcYv&iC zDxIgxK`P}10oqG)Tx4Fd^2Y(zi$ahS6om5b5~qkE zC*F3ecA35>-rH3BrR14d03todZM>PafLR%Y-*gU_^C(aGk(%lix6#*OuVItHmN5sX z(IleYx!er2i+wzX0q*$BO;G3wz>&Xxgah_RqlH4JaxxfpbrW9GlaXc{iz`K%5fn&J zb$3Eh0C(u__s`JIqj3zoCQ01_VQ0K3v6 zADC{5&f>WXN4$zTO@d!b05^;-A>CM4!o=h1Jk$%|7^5q+fB>f?%2NRqxC`#6JLHQB zS1?9&w}JWofVf!MMeI{nl?l54Obfy0-Ssme068^^-D7SzL!uMSn9D=P`vIb4+zOV+ z<8|i3bN5_^%F^<9cc2;b&g6%bC1p6LQO{yHpG+YsUm;Gi0^LRW&JcbUg>Dt*^=AT- zTru67;&JPMV%Jf1bllkP(sK2Gz<$vO{t;)0-{do|QHH@mA@YZXzs|5eARr9}BA-M# zhR^?i9K-|`C7n+%BFFn9%cFtm7>LP)p40|h6END1I&iw+GSUFJ1#}F~CqVeO?$ekn zoH@({1IFg_>Juyg^M}Jeb<#PD)0ius-x>$AQO+4TD-FiAcKD3EF?5hNh!`;$FOh~An7^{gBi*b1B)`x6e~o7^>cU^G(H{k1HD}3 z%g8TyTuv5Xx{rYAf+6LaTD|#mBRzm2hOpf@@woTr_7R57ztPWR04%;zk(BK&GDbM_ zk0+-4XSw>!it+d(nbYPqZ@3K4CVX%p+&{7&oL$@S1J#??VePE<_47dk@I^Cl&e|L^F3Zi46ML3OedDy%PYiBFq2ot~&C1pvC&S}lF zTOMJ6;UXdB-vWz@&k=Qk@8|Mv139Zl2&!k{%i1}3E=3Yr4EVGSfO3NA-Vl%byPaw@ z3rF0wagrG$yp#ai_O^P!_dC(rdS~JPb^)mg{O1I*0Q(HcDB#JmbM}4U+)UnWQGmS` zwD=*7xiesUX~6cn*zV8e>NNoBW8et1B#r?Z69oNd&+YXWR#212IfH?1-vt(0LyG}~ zp63YjA$<9~Ko{S98Dum(u&6qLE{78)7|;PLK;;3=hv~bp=63Y`kuWDq>GK7DA$+;K zL84IGm^!3k4Lcn1tQCD<*J-93d5#kX9e%6T+FVv;qc~7|3022+n`Sq;r_hdMot*McPn=Wsge=crf7vk5O}Fb`|N?9W(N95>xjCH-eXv{O+R zQ~n)Do%|W*jhrc#aKx7Ychn#d#gV631OdtU4Zvt4g?(b5!TDJ7(+nk_>63N?CTOVD z+FB;@ZsD6&-z4%P{VZxMxQFcdkOa9p9O24GvM7cd&J$* z8D#pYh${%loG~BG(0RzEX}~Os0rT3Dz|mgqGq};W=d9bnp#zc4W%tQp&GkS|jRcr9 zo-&z3%AJa&Cxg!X8WGo1JWEFrgJns?^6R3djsK&zcaLZK`vb>ULe!8-Wg+E~Bq}v` ziQIC_ZMiITlP)utk+~J`rYkkg+a-51x0aI3kXVG=Hz{o{xo@Fd%Ur%^+vojzJbwRw zzx(U;dY$KaUeD!u&Urr1>Y$a^v?O~44UHIyDyRgwUv8x+s@jeYxqH})y8`ZVo}da| z^S&3_LjiRU8jh8SGFY50x~tblL{&d8n@gco)_TomHQCerdvmA1GVq(#zYDG;nOdh}A_|?E< zN_bR+nGQ+57q+^TTFR5v1@j4)RceW}GA?|<3k}50K5wC6@-ucs>}GmDJVMF7hP3La zl*9A*a}eGF2npIwa6R-M+j==G{e?%vAny8kU|XV~!NE{XNS@j>DcOf)00Tt;*FJ!Y zOi@H2)l!41Oxs@uqD}AZ_)No;@R^%@412><8(S3aYz-#IUjA_Qe43(|%Nv(~!{I7P z$-X!{Hl&9DQtkj!{>#)N@ub{?HXR*lreR72h@Qh7+1Am6l7959z9Y=*o8Sx+SNZ8z zA^$5*;uUWI1t1i{)Lfn%D^Nf$b6+pP!0TyHF^uW3Yfl)ZT`+K`2bo$UAH485Fo;`j zYND;S|16Xcl(2XulQS$JrZFYk?#(QE-6BrF^%1HGSzYc0P?#Lk;W$7xE$G1Mz^!_; ze%BKrPTHdvc=-y9k{xnv7JYWzCsgko)BCv{)kL3yZ|7+a6m2M?8oyWoZa{D+6noB9 z#|yCWWDsXw@M8_F0fFx(-avKQk39v#4jTh!E?oq!-Qe{%foA9lH2!0U0YHeVRKyxr ztk0bSjz^8c-sj*f{v-YDHt_)kGKd(>4Q_@$;8qHg4vtoVLQucHft2*aaw+Zx_oEQ- zGD4aASRGUA(j4oKcPpxwj#9G4>Nx??T4PTS;8;xFhrO8|H%Ar%BM6z1Kdr(S z>SEL4zCU6rkQ*Wo0UWA)9EX!O=a3njfV?A|0MV+jvDG2>O`uei`82?|!gz&LsD45O zY(xBBLq$(%`fZn-emn0ovWA{weh-K}=RPAc6orD%f1a$t2N>%VfM$5yZz+^>SzZFD zVHDLAYFshTturYf#JQ)0MpsCT4FT__@2802xC*O7!@$b)JqnH?wt#q-+0%`ytYg!X zy;K9R<76Gql4`3ATMD9IFX=j`jUk$Yge<(PefQI8Ns=w1;;>{KZ~|T7ApEk6Td5qq zi9tw(z!{CzSiE^75P=X!1I`r*miHnpMAtCjmkqLT1KJdMxhL^7k46rk#unC7LeSat z;9Rn^J|$Zvkdb)$A1Vv~`BantU9S#GaH`J8 ztEph|(Y>Q6Mz?E}DYLB??t=J&Y#>D192z*RQ)tuIFFa(x!bS-0+?unX^kh+>!9g5| z{yYp8N5ZIr=f}YJcl_@^Luva3-#4KO@NX9Ko`&$gtC7{$5+@(O{PxOi$YAa`okJ`` zw`*)+TbJK~9m)pM=C#wCi#E3&6{M!Us2*#5$&hGZz*Ian5 z!cVq(K5aq_>`)dNi^s|Dc>@PV+bXS~6XA`$w_mZ{IuPXsmT#avrh+w_ z?5!r+kVtIA?jRaQi1TavRDc{`OD>yH2hXw=@}#&tzLj1-j=g)6wvZL?;vv9)7iQgk z%Fv27%QIhd__h-f%oXvK;A4NA!W?}0AT9u;%K?C}c_1woR9I@QXJZHB=AZDcH0P-* z`c{`Q=9?_C^EOKM*0!ER?ZfSWlicd!!l6%3G*@@c=zqSokFRIz4^ZW6?=p5=yv5Vf z8RIAAf485nId@j4(7{VU>JCgM|1TE*?xsGE+rvXQgXa+6=KhUjV?LJ)h-{$voC6CE z?VGKDNZCp&(0b+bCZ=ab_t*22eiC zE2T$V&dJ+>Fuf!oy8*tV5f8}n?JHYRFn=!hv(01K`Y;hXJTr&vypid!Bb;p|o5X8> zenvOn=RJ3zjZl8a;)B786wIRck-$XiJic@kP9Pnvi+qmh`cad0McLMmVqoGvUWQg4 z@Hw7PFF4RlqEewx;C0%WyOE$aFmMb3NJZzDW}fhHT>fh|(&5oF=ZJ9M%yfWwfYx!9 zr%wX~8x`YQH+>X7U)Kus%GPy2xNl-Ug{6V=lrW0yzfWMTa{)o23x6Bz9U0z(fp&A* za`FjO=^gWS$J8g=ZKB(SL0bfJS#jA`ebi5L=;-P%4Tf|7zg))KcXl%f;&)(g zu3a&-I)IPkjw&3j(SxXDlb^$wPfd6lbC;i1qjNTT%doA(>Yzb>z$rV>L@W2D&`tWQ z`qgaw+s^>#?M$r%o_3f9%;@FdxYDtUsIW?-$sPId@gEL|7GYGfsVJ~}M|Jf)JW0=t zvjc4Fg9W-g+VqfIp{gHnw-#ezXj%?{-GH*p)d#JB=`GFl-o_vttAnZq7iWyzh1tZw zZpMIBJ;fw1d_hT+53myee1J+LOL!ic6*~5c10r<`N@{|?cfhLG;0%J zz)f;MSPQ|;CeS>e6yPoShw3O!F(G~CyDnm=(j9E;{S=d2!O9uD?_^s`Qxvm?|0@Uj z+eoP+asGSX&9)Xp$6Ac@U0)#SaRvdT4#dgxy%Z%%-Sq3;ve>d*y)FyZ_%SPUaL+Eo%uiVf#OJ`yd$Q%#5mRi~o_oCd?ohEKlK# z_=3XP&!i&EhC3ZO@@i)c9j-gh$tNmJ@7q^p!)YA;jpJV1 zy@EMdO%6gO8ydhTao=VYhD7$TtyeLoGi)n>_$oKjS(w&*+4;#HSM%+1wMz+_ok%hN zl-$8r2Zk_|4RCEy3^9Y69S3M7g6K}ZN}c>ENZtzoDj2f z^OAz$AtNr+13q6(W|+d~!D9MaI~0Ulp2bEb_l*zZM3ON-m(ZqBSR{g-30ns3M|&LS zre91^6h~cC1bZwC?a-Q+s=&XdZH6m-fZoch1x#0Fe*#SB4vSp3K3Wt^8;(G(42KWw zK%0`y!Tm66UkJ;iI3ut=Im0SFdf~xBD~$8Ga<@%RP}e9k>y-tZg`T#N9^kGuk8~}W zNNO1UC1A&fe!d(?Hx5}3Us_qYG4*O7BU4{9YGvU-d1>50IQGxj%1Xpy>(aRRcqO-A zooMc~Yd{}9|BlR1htE?p@(%vM5r*RF!auhP$!!hUzsGlHpzE`Dy}9S}Q&eld;vg9S%|Wp>>DVT=BOnT~KrPPziM!!W~;(QcHA`CT-=zU=esSjDAU zlEt=}u&GSaiB#$zc`9#XNxm|B9EiRFe)Hz1<83CLVc}BmZ&w@ir&?aRizFEjuq-jO z<(nJQ3e7jmD}Tu?Ls8bzv#eN)0b#??mc)e2R(WHc^<~dhZp|EH344pZc6s(=_*~)~ zcWJug!s0Jf66_yyCz*k>@cA+_LjgWt17sV-xh2$s%MoWJd?h_k{46+1^Da2uKHLcv zB4tXk&C)U~T}3%vi*Y7FLDK$pxgW8?U!anzUg42d1Nx8b+s%G`lv9r2cIv-a4_Tys zfJ#=Y)<%ykj%olm+accTFPE6` zwV=A{I=FkYS_AZA?yTNI57e+kec>aNk+^rZ%4gAkzNK&Y+i$M*FSOL+n#Y?x3{HLV z6tsLX?&yhDjOj%~y`MP$nD#LI+DPbfYpj=zo}}LISMMbfenn>3sCdOaddA#$^3W?M ze^zHL=qk)phRwvMu$<$~2c~8%4`^0pF-y~naCAvcF^#SAiOv{+sy%tgVfcbk zhs+RygRSMJ84-+3*JTVm`QUjc)o05$vGn?~UH+Qmr*zHIavZLI?T*<#$CUC)kf<6m z=#t!GZ*;$LR6eX>-};D>i&S@+M7u}kK*I{rz;e&7d$+^_WJ!HWy7f5JN{+t`@u9^)&30l`yp~y z9_(!L(8yLY2Qf=GqDns$uH{uK4nWOZq)`RBSM`sflR3Dr*g=tR(DCV5t~+L8Ep({& z9_&!AF@_jlbr7?|CY4byKK1G;X=Ym-ftfx>sbkD)ynLcfow|(G&p7#~o^f)T# zc?e#%X$F@)jdN3YmA@?}$muv!%8iFHYUTO<^}}@4BGYx17Y|(h-uX+nHK8p1cEIyG z6RD0Q)+s|Z^Lpr^>Jl$I|0_Sd6isKogua-2o+GU`hO>=+p*1hD-m8*0W$+h_3-c@} zyU*&`!TRmBF?oSduee=xo!vK61n9aFsE%;o1Xl@Ib+Wu z*ko9Nsteh=zOrO%BwJL4@xh9({E!Wd5*&X%{eol*UgJB~u@&BTo_O-FK9l_ok@ zpT)S!Vh5|)#r`k;y;t!d*>KIhW`T2!doh>dBM)}6F$~a0%!@o1qqYPL zEaB6Qv^B9}_NnOb&`V*LZUyPLz9&*lcHU9(pq=o&Zen|YODO7u)@`cyge&#t&VcFx zlD5$Ho?~y-MyrOzO57KxVC$8luNR3x9#1_l1Unzd)4Umlo)apjWpB6I(qf+Go>T`z zl+1k0Og#<)flyUh6~$l<{2U&RgI(M*RRepEjhGG@8Mrt1>ESEYXP{_S?XYy0pg_-v|Gw6ErnlE(+~t)u*5h6SSGee^%GyulWb+E>QR%2&g5Ct^QHA$W z$=`f*`cfXdioQ;Kbn*+1W-|Zy%M;cR>PIk9`(y~h!Xq)ghefEKvGh}YBM+8FJeLbY zoUQ~M>8YWv?F4SZd9yfBG^hcFmsClV zmKbW{pxjG!Obajza3odgph#jGI1k|=RN;e{1-~IPP01W0||Fho~g zbrVp1Bm6~0iUmb65yc?5{{%qyeph_;Uq^-i)t?GAg=^L5-pbz&$noqK5CIR4`4ZVf z$+idv$4Tx$Kz9L2)jlu8^+G^W1MCg~{p@`CO1+#8^dlq0XW?;DFTs5hVBZqM^Msep zt2ovg!9d;M{v5zkO>Pe$0rb}m{8^xW`R*T7M-EVbAuqeyt!wjx4GoYuld+^`JF*FgM(H3jd$X^W2pFYKH~I-H7cGc zpwYNY#i#QhKNM^Dqtb%aStIC!^eAicai!DaPkUx6g~=iKqsH@!%KN& z1~|b5LL4@8Ku~+mBMXo689o1WjAISuGxFhw_65ZCMt%kBE;O=m8z17Jl>|8bwLo2tRai@QuI0Zwp9URiiL9Rguz6&rD0(es&ZZ&r9%w43&S;-QETi|&1`xMo z1MN7-*)j-5GOS-dghxDWIjHTy5WSvz)oU6_F&xfVO}<&b!>iWN`cfVFDrY{(un;GC z2;F|ezj|!tEYMCCc8o!Ya<`hY*sbGy^yclznWE)5UJZ@B9)i(IR_X-SM=W} zH8nSQ`o4aENS;4;rYG}bp|y*15xLE?bSd*3Fi!#WHCg**$;x(Pm%^CS1lf~6r&eEU z76SD!EZ&fVzo(rfXMrn_@_lGFoB4>8E+G19oqX)qhVR4r)dvP|gv)aU1+ECH%N5}X z5>?iy$oq=d603A+-H}Hk(GsaPqPkOi_5GLrs?oIYAkB?D!!O?l5f3jn_>o?_28_{n zvbOTkG`v!uT94kj>+aic6XIi<{qc`wjdFQP(nKNE+?`Wbsw1k5#bmi|lkbky?$l6Q zeaI8MrhwZ0cKNeH4X0uu>aR<+j0>%Q+>1~B-$xz&cAmpIF_ROLhhVWERdRa#tofwI zXzK=HLyRTs3bt?V=ge*4AZ@N@y?*|r5M)A6LAggOtTxzXlH}5MV;yZjc#95$LbASZ zuJ9-rd1I~M?Bs}@f;L0@YxY7QSu#<2&%$2aq$wbr9I00fA_@h4+^Krim)+FNXTsKKIN7Z}kbkHbU;G8AZS;%m$0m{Jlnp zBQ=PjF&k**2wVSMkUJT3Tf8KdNqhS>!@{QFkw>*=wnDsxYHu&)&N@jb{Hq_^lXCO+ zFy{5>zt?ty$T z@8?>jym3guGxU3pq^{nZ%4AA zPX<&T(QVP_`8pL#B%1o34+NJHKuXNew7cwj|G&qfqI7-wLxxs*dHBe>?>c1=4ekno zXoEXCPB`oIAkXErz6wJ79@C|-d7<^NdazI6wt_nLYWbDheL&>$wC!#)!xl&jovHq{uFl@FSa z2*{|#>`5QoX)%{MvSXH2-MNhk5?i}lITr;ZRmyFZO^y~vkAusA_K8E0wZL%bu~Xp z3ytrPNRo)p|5TBat3dz02a;v`k)_3{^(Pkj$SM(QvKxlv3`Zk!;xg`a(NZowEBg~> z<7;vtlnIwJJ5Db%R`+XN84}jGnqU`mAEVRsJChn@Y8jN)-wuJ?0?A#mx~ipmDHIs} zxNpJW`-!oeQUM`>fhW@q&$k4}9`Be6?zdPj_;DVpA8!|bwXI)w-fP#`RJ?)5874Z| zH*l`;w{4`!^#zn2$$Be1!N1j9>)B9(K&=;14yhNt!*n3!#9av5aKN8cMUCl}gK7RA|fY;5B(6w)+q`PM%rvDU&9%{?~pNO?E z6dOtQSV#dUCkbWU+qf2_{xcU6%Zojt!rg7ATcbgE9Cv70B*8A$qIvJV|9IE2N%z-z z+&WEN*WYq=^K4b!Z(2%j+(%(T%yhJ|fd@O^7-TloNm~DdN*0MWdMYg|=zsS!NzY{? zA0dDLh!6bmUXM1>m~tf66NOGYt1?tSy^b@E(b;-ICcspyZ5w6i;Ga1o6a?H*#DrFK z&qCoj4f&Ty^}!=;a!kGMjE>NTCw_s0AfxI$E;K8&enC3G`{@&gBX+A^* za&3D6+-&;wJ_R~wy=KaHJ4$XxaIAH`FjU?y!Qb4sV1`0eS}2NblBP4W9@=x)G86w@ zKUtH$0n%qS%~5LxxeLEYNrt*09Z&ZD?JauU)2G;)v@|0fadkI0kVyO_aw#8EFRCS| z@k6iV(`04&*9cQIIttCzp~#v?!EI`cm=BH*q-N|j*NgOLbImEqdr`^dK?WYL^?5xZ zYpWQ%;5Z?#`0Cz<4yV)!!(WVOrznusM&C%9{vB09M{HN9=sBp8T-r`nWBs~_iIGjR z@%M$?Ir)tH-d{BF{QDu-l~-=4`?dk$-`GMLj-;v~%a;u)i)M@bgq%s|_fszfM*D|* zX5iuHp^;9y3EG%K`}26=u5po+YD~=$L+PS4tesBR;oK^d(HA{MKULQcQiqgwpW+P- zc(C*$7w(d|y68_iumRKZ6h2gPq_${GJRiP-NdH^5ZLN?T$9)juZ->Ge>57k~)BZlm zePA^3^1tMl?jxJ-ff4GVBBieC>%x<=#Kgn{<2l6^cehR3*xR8pmr2U$dWlW_$5q$R z-hufUmdpYZLb?i^RM7)VZ@%GALDT#&eghzec%bLbX@&F=oj6S3&ts!|T_JalS%MVK zUwx5#t|t9zg!HjwnHxS9F7AINUanBB)-~al*R^k3K!M>--No8HHDabou=e>z|HdhA ztiBNBt=Ko{SuTBFP!`L5>O+vZ11xYxh?F6Q6Pvz5DfgiH9f&K_YC>C7nZ))^1V_sT z(Ab|g}&#Xq=&Yx}y z;xwj^<72sM;+JM{5gOZX8n!re9#$IB!FhDiJgm zy-}R5fIV)G<^NmrCMn4Y%>pE)0lzI;{B^-5rnpFaxXU}mmy X^vDUPlkRoAdEV-bof-MGcg+6*RxF1e literal 0 HcmV?d00001 diff --git a/sections/0_causality/figures/DAG_open_data_cost_savings-3.png b/sections/0_causality/figures/DAG_open_data_cost_savings-3.png new file mode 100644 index 0000000000000000000000000000000000000000..f90d48c6157f78fbb07cb94307358615d39d4900 GIT binary patch literal 27519 zcmc$_cUV)=*De@EL6HE8f)b<&B27?A=!mHFBGNl5O(Z1rj(`Ld0qMO;lMYfsFQNBd zlhAwb9qs|X@3}K`=l-6V`DgMxVzSR(YrX4TYwvxs&dLu(c?mEfB_Rj|0!vB0Q38SR zAs`SQ#VtHwHkDOZp&?Km-Ux0gowr zF9ZTzH&>KZetUj?4$J{wIXO8>N=nYo&U<@%6%`f!{{Ga|)GI41gM)*>RbbkGc;Rrk zjg1Z9@6{YE7F%Cm-_z3re4Lz|?C9uNUtjOd%mBiXkr7^A z-v1R6Fw)o8XK85(SOG|Ib#^m zTwH*{wY9a#$jI{Yasc`4?CkOJG2jRQ7>z~)BrGj0b$53I+ycP@SOo|NTs}QL9UmXx z-rhbqI2afh0QhfgY#bXKYiVg|Yik4iZCx1eS)TFhEYBS4u(Pv!^X85250}^aYEPjr zIo^uU)6*M=|4?y;BT~Z1AdhYn6Y(j@IQ~oW@$t#X$oO89d-v{LDrQC**yoooUji|u zrKPQ#9cyiE1-RPT+4=VE8^Cj2T^$gV&HAMM@%FvZ`lBT*A0J`V~q>zv!h~^y|n-R4ZU6D-8HVXj`oc3@TR6Ttjc(Fp(1VCoWISMB37&T zh_<|DGd>`DXOQ!Fyl{MCQ2#XGP1l+sVE@`j8HqO_pPP>*fe*Lgk{VXP7~ADPE1yMR zU4aO!rR3fcED+rzeDGM2Kwb<4GVYLiBc^OWx>|YYiFsEw3{`qAs3EQWqhBU@rll)C z#?xBz2R#otnXz5bag2+9EH*~OvZ-`EIW3J}SQyIh-H8k34z8%E?@V01%N%s?FCBA3 zX>5JzS*mp+;{G=d%2(6r*EbGN>gXtSo>A%0^p`B%VPg+{LjtmVKWLHrT9$^Mv(ZZX z4G77J-Przz{0ztY#w-gxImx1XgaPRj_JKu3%_Q-mpoltabXON;P%Rjg`k$tzIzzK| zuIUrA;w&02C8n}e=a9ARpa^t=J8Z%j`{)RDt%eA7jSW!x@H+Hc!vsD^d;&l3Az%RbmN&k~Do3T0H!u}A!lfK<&!G4R+ z<@Qa{*21!$5Y4tZlm3JFuVC0M;eDGt6>Qa4`L=Q0mCQb_Mb)pPU%|mc5b1bnJ>swj zV=3nA@Ejrv*2Ena!_SV-?} z^L4qZZD_$uY!u2lo2HDiwNRz^DVJ+z_ol`|o1YCjA^cV2r*4knbM=YiQU%3c-j_SE z#k-#dw3iU#FqAS*;cQYDNH?YJpneHP6V$O(PeU+|hTy?0OH#G;1ALb+<0*rEQIpz@!A zh8g`xh;rtXT?_O$ljrNp zEv1Vy8?2?5uuET)B=8a+o-H>>tVFbG>iCj}R42d83?V3OdC0Js{fly78F}vu0IdEC zbjMfcVaf@w40HW=NdLQe;yY}G#x6V(rmS*;8b9;vz*==YsDf8bwa=kD2yxUcOx`V2 zqW|k(D1(q*4~_!k=t^8CyD??Ed=K77FG#T$_3tJ4Y;5zFa0tGX-Az=jCEW84EWIDR zgs)D6P=|PF2^n-HWlBnjgDs8BMabHGweJP)E?J>7I4zTqv`X_0Z@*}(hAJt%`d+_@ z??6J%Kf4W$++DtHJ~sI5;~Wu%smUr)G%OcV{QXnzY}_nt?1?wm(W$J2V5z{=?VMI{ zoz0`NL2}sO?qkiu-`x&e)A;7<`sPD_J>u!h`5zLc$`r>s`^X_41cqw-h&~!D#`o+_ zj;LMfA$#qBcHdVfTn!=}j4m}W>(C}*pH_8k#@Kk{LcTG3p!mk0AMeyuy! zPDeaa`AH!NZ!CK^CRll9yKo16wj}LlXN40=iJ&To-4cra<%WG6Gd8?N5>)kos;{V) zE!Ughu}1>cd+P;$TJ%m7L$(DShsO0m^ zZE(GtzireAe&VKVV>h7XW;5e9!*x$B@S%hISK$VOYzpZLomd;@jK5$cPOv;tr_Hw2>Z{U0nbi{>&CQ&*i&|%aqnhB1f1#W716ud0Tl5}x<~p-t zC)SPf_8oID#dd{L-cKfp+PtlBJ$r;2I`OB$2M=;BM9#g*-U;Hr_A1IcKQLta`qVkQ zAvY6#ZL3S9K-`L>bHXf8?>ijVj-o@Aud{w4mXOx?*Zv&QRhKFG)rm=hCj&CttKTPX z_v|$bN1{VOHZ)>(Xm`lY;|^;%q^{2zxt+R2Twgq($D+p&xyR-f`u$^?FEX<-bocD1 zTNJFc(nHs+QuH)EoD_-Mb1fFvdp#oW;4eq0r&j$v(M{*6i`4TZ_)7vHL zIUXC+EE3z;Vm1{-A}th)sNodLXuM6R-HYnZL0Y4 z;MD5=MY$c#0}8TC`BoAp*{8cT+`e?ds;EY;R*nH`e|=Sk4hi-b$X+(X^OwvMKyC&Po*I+q3_U zzsKU9-5j{tf~kz|@T)QIUyRDVpDJWujq?)?4w%NY)V#96j+M!DtrOH~xv?`lAL2d* z*A1z7E4Y}m9%|t>dv5TzItP9U?{#9bBQQY-r`FS^&rW+J{`X^$=@K z4m^vkHx)RYc<7y4XL2_fdr%*rAe4Xe!63v0>c7Nk!QjH5NJsp9J%m}I0D|1c{&wgh z&j`}}D!O3)Yp0RN8h+1rgbG&PwYb+MdBm(|5_~okOB}yPMpl(lN{4u-an9j(EprnG@+zUHFu@xMq|+UvR))b9L4`&5Hi>);&zxOG(if&bv^b^1GIv z_iyd4epyDRY`IgRF@yY`%4!hG8Lhb@*$`w`NKzqRFI~|W*qeT5 zc@d__WdA!cq8oSPee^(Suz7G3WRSkWHyi#yrPp_XK$ytw*i3l!sYwnbhU?$*g29t# z{Iek(g;M9sM_gz3s%Epx1&y^?H|cjphI{`+)Sf3_5FYK=WyLpksIrMu=cB6Yok2%B z7kUcD6v1!7i>+tHnl#Cd^nvXU3D)kSd4<6ohx+)$S2&^ue?96MB-H8sH;`(}TgWc_f%;bSZz; zIS8)#9Zkn3Clb>l;k2M;`2jB8>J>Qz-hwT_)MHB?O=|j<8t|MIX-QRIEB(y?y}@{A zz9R(k{ZZ7PSY_x!=J!aP$J#+|m{t#TK~7$&aaiuT#TsK`HiacxG1}%uTEqsn+B$h(6t|_Y`;O0K5&O66Hgn66 zmd^#b-Pk-4JFZPL_Bs4)kg4Yskjv<@s>lTgu34nb=~G?9Ag}6sX3wTFzYj0Gz5NR% z@bC9e?rg^J@CoFwv?GEL481r(|FH0W5N6MP^_#Azp+KZj-q-_h;nAQzAh0JI9& z!}f966;0o+4^eIyeBW||1dGOd(1mST&k_QrGBHiY`r@c5s=Swe{jDGEN4{VB&k>%jZy-0?XSqq&XWhNby> z^!6vDLip!i=dv9d&imMkom*0H9K4-dLi!d%t0U2421L*{<@J)CIH{~%(55Z^(tBDd zx$uggothjc>Sf*0B$#U%w7T?ROec&vYW5Yi(ds3^du)gS3Q9*OQph#(F55xjHVwwV z>ZZquc3hvW(64O8Tm6&Gm5$2JWF*r~W|lO^6dP7>t=54@G+&$OCj$R|UA0AXR@F6P z+^gr>;9{v~={JVSnTm;56F%24vTwkT{I+)F%bi&h-fAcGArdA-!) z{Uk^_%H{ANcKG3!XAK4tI-HsnZ`c$LWVAVEkT$EgK`l4mK|V53l|`Nr(~Fw+k+cz% zNyfB?nF2gXvGAHIwu>%}VDhr(!_s25R-!|ya?bDFV;K!!Gq8mjmPwwHAbTRY{zWI` zpbWUq9?mFQ8qSztrLDwUCqqPa7dzrYdzLL4q0A~wtB!Z_QINUf&v@p~UW7xEAGfX| zcyiKw2y1duNK#(qXfDX4e{;wfhmzPlNK~?s*Y* zXP&|THFj+A@oy1+HGwgWeKzCJ`i~8n<|@(mfh?44VhCA`vTWtDP&#p32&Yx!cD3n< zD+-yU2kY=oKHA@&BQER$FO@v3{ApXyVR(V`zGxiFYvE58PI-(lO9_FFi)aIP2?kmTv;F>tQ~*989j3u7%;IlJ$V7rf`=Wim~^HqQO+qaF}g44;q4) z$P7)BRLOkt5nqaG{W7fZ=!b2|tnNW{Cu?!hbSlMqmsv-dF@Gild33+Urk+XcZwf3) zY_-^inUv^kB$U}z*b$FOS^1HY@VK+%s0>*L)mcW;OFwEW|Ki)+p6V|LHx|JWv4Sl zfBdHjIXSDir_Xbf=ss^F0qop59hdd@iALxHOH$ zkL))KcBFqVvw1TjC@*rjh`chV&M*4lABF8RUC9%(K%);L7Ca~erUPvc)avqB zCxx-b4nOl#)=u=FcYTKCKQc{j-R}CMTR)`vk0p$$Fk#BEGuOez+N{lC&_Q&w?Y+ve z6IVHrs>e4wj>&r!vm&1gswY*S_PgM0a{m>Ld~*A52cMSMu$F6;y!+>!P3V+No=e2r z9~LtSJ%7h?8e%-2yt6xO!KVA({F{|=Le8N1KCLlIvS{>g{J8b*$WYObACD29U$WKG zylu_ID^+Xt`p0k&{g7iY7CBpJ@?-hMPduwl9X6Epb9vGOQsubUY-{)iT={0~jyhhs zTARikD~idan`U>Rx8}2{q2Igp>hd434b~Y&;|g}`Rj%_U8-zR%?o7@J9XO1>uc?R- ze(+&*lOnCUnywg%Gu`lZnMc(|dd+Lk|C54^x}46f(Di+Rde?S6n~qCSVfIkYI?3Fi zuF0mUk)??*NO8J>pQRv>knNX!rfVvJDG2vtqJ5sPm24 zoTP$D3dYnml^-z{7u-FT&Jr*MHfLNNjFgwGd@yN9<)G|=+g=^eICgv*}XGxc~MUZ(`yeNrz^yP)ZS2|L1;2)%4#q zp{A!^xe^?YC$-M-jX3UE)dxCjKj9f9S!08Z_Mwd*TcWK?DhQbUKl%>NgR2>0CJ(*2 z&NiPppWmg~{PExWQ*uXrPEoVX1G(S7T`&A_o#-8!@-yA37??mFX8j#fY#dL>;0RIE zh{>8|%!Q7|HoSdV@POGk0lLjhk^#TRe7s!l*-ZCPS+PhUCwm7~Kyd@w-=$a`IwB|MVKqWD@1UyAn0_0hc_rjy}L1n_;HY&RmiD7IVp=-F{-d8*}`FX{LCZ@|()^MOKZ+b?5I z9xR)EWkn^*D;phsvH63~{#VJk2CS0DD{~)<_~)^bo0mO-6nOtH!6{R3Y$()kUN@mgoVvtJehnEroPDf|CWW{X(DfIuVv zsr>igL2&<))&b~!e2>#}|<@X7&Z zPpw%qz5`XOaIZGh+blUVcUvHd#;M%7n7CV4ilE>ijJjJR)L0VK-lvg-ae#>wEk zVS4~J<>eAUQ)S(`bN0H={;$$Vid^Q$0}FW&6-1X>|-@5cM~11%UjviO{+`1 z#-_`uL-B2GL(45~JJTsen*T&+-EAWg=6$NB``%s;|4%u@0mB# z^rlpUB?@((86XH&eqN^`_S_X*XA~ZctI{3X7f=F=Lw^bhc|(laHCv#H|I}!#PgZM?G7E+<6V}P zCux%i?$Ojt#)yDU-K{E}xAJLzW))0-gMf@GgnSoVIZt<8IL#8f)bc{1x+EGkLue+N zbC#WJ4v2)=Qhqd%Jk?h4wEoXAUln3Lpcov!{cn6S&7F$HG`6eyNLunbjq65Lt~bdO zM|~r+a0d+5)N%he|FoSRm)I~eNHpv0@<)Li!zYX*f#0&!><&4fs z?xo7;a@P{rS5hZAuA-K7Uf_KlH^$U7%DDOZajX;ul(6Z+;hkYML6z!g|02u#)7Dg( zf6RO;_|*`X)HNt*p?yYCoH(Wn!ts6gm9BQ0RIAnY+Fh1vYGI(J?H@g-=t3MLuT3)3 zU=z&m0Q))gMd%cDjrNTkKm2{Co}{Gn!3kf?!3O;g5_3b>xUsXdJM+s~U>NAu2Lw)toYC`7-65c){xCcyFlKSJ>{eby)@1fI(LI z&j{o_D)^4`MXv;!GA;$OQg2@h(B5pboTz6BqA+IeXugy%RS&-~5#z6a@3Njw14mtYDtvU$i=Knr?(E@6L3Xt~nC1#Y>dS2OA= z!UcU1mlHCkVno`z>#vY{;?G_hy5<-+X8d2q986R%(WO|vU;f3zd+Fj#BH|KZ=v$d9 zm!4BzcU%(U+kJ8=HP<_OHG$N;`NG@h zKgj1V4SQ|eONPWvNX{kj(O9xHI*y`HUFtve&r?IXcQh(PizIs|7zarLOwv$N>|&Hj3!yA z`-9|SMO$aO*MmU~i*EpQPFqs~ zSBWB>@C0ydUiWAFS78%$AYv_YNn*(BaLWk>oc-jk z4|P-!g+yHREg0yTmp_nK4D$^m(Fd>Mw18!oKvhVEQZR-#&&#R(b)^ggCThI|h8d># z;jv#e=4KjP^wo5!9AMkPk~B=5t?8H=&;def^5$|7c)FKzx66Gwooc1a_emE!7z0wI z)+EMcB`(TGiBmakPc>V$fdIpS#K6i?kGxR-6ksCKgdhHIZce3-=cS*a17fDs@CFnV zB0U6fx6*Q!YTy@!4*@ic;>6nYb6Xb{fLo|%5X*@< zeFhGHp^_6|yXmDPfIYXFZwQ<(z6D4+1CohV6?|J5a=?%>h#&s9KZs(#K%^a@ff@kx z;5S~l*Ut;MtN`5f8$I}UlD5?Wn93>eg-I`Y<6!GYjHqK3Y91{TWNs+ zpYP7p%qCS_=x+r9`fu{X`I#9x*1%gBY2aW{#^;BZGDlrFuK{3Nq9uR<<&+C`ZvfL? zzwv-??Xv(>RTe%Mrv z`#Zq<<_0&~dm=1{E{MGE5YQJ7DpY~}VyKJz7LI~|7yw5Ofb;4;#<-qW-_l6sH$_`5VVmuK-{B0qWk~s_Zf!JgIie z<0$Apo}vXFC8vWeC&-_!y1>r$Yxf((?gL0@W@^~qEV#hp8^`B)onmlP0KjX&_!A%$ z63+3_xNd@2j;*OyJV5DfARbZ<@nfv{+<`IczMeM#PqTrD69{*8IK@W~)-hFdu^%5# zb&Jyj_*H@U%0BC2pFhSFRCD;(697-t6**>V44;I}^O_Hd4iF^tMeVGt`-+nQcfbio z!1PzraaxW0rG{>LR1S#ufOagP{T;AE&G_HlzEDzu0-k;+*fzUI%V@@F;gBqVwPxQt zKu~rs(s*PGV;L^3tE~&%Xthm3fEY*)5J9nD_0n-0|JQZ1vVw zpjd|(c?#z~-Y~hhNri%}|7seNQ_G}!{sm2>3NV&p=74Amg4|-h7C{?@x8D=5T-^jo z%~ZJ#tkvAC7OOUyw5_SlF=~pnk zZ&D#tB0@cG!E#XzWh*>%z%50aOgX@scmIa}#n+-3iak+*+v9mWOzGBrU}0`PAOz-} zzU8TUUQoFq{DkEbbUe3>YctsROIv6o5Z{z+2LyB%XO>C5a*y zWqkUAv)fLGf9g6Aqw zqCCa#iNk@68TKBuQ|HwQcNq9Z64#}xy^qcTb2Wc%^R)_nQ3X?bTFdiV8nk!!O&FW& z=US~#?p>p?z#8AFKn0wL+hH?mQ=qp2CC!ZMshClJ&vfKw{L%)er>wLtCf|1BYnhxH zSE!^jUfyq3HS@Se$Hk0v{ODu;xrlIzeZ6Igu2hJ--AV>A@O|po-y9k&5B4#xP-1k)fxHiIm_uet+5+KvEo-i~K(3_+iqow!l^7c^ z!Bn91+7SG#bv~-f*Fg!NVyZWeyrn#O)y&Ht15JzqhZ1BGJGoXv#Gvl?IBF*Gp;}JS zC0;qPxFAhz!aQ zjbL^H=qvijr+Cx&;t~HfPmu_tf2+@r3ueNPy7trk|1j|d+>=fg+KEb>oxp#F599p4 z*!T!q2%NtSHDKfhWy@K@hY4g^4Oo$A9DHs*cDTM`jh;|Vs{z}%LlPcVL8?pyt82mJ z-yloZp|38K!SR#{Vc$MZ-~$a<>A+!lp$8CWzOoAlDZ%mX<3WKotyd{mCU3yLN!J1f z0Eie+s0W}HRSTR*EI0qNLMa#@UrbCKE{-Y^Hiht9jj%?C+pS~E_Xn^X2yr_AAzk^T zeU~8KK1rrom4eMvCRAeJckIRl>^e!T0*%l?Sw->DBowD|w6IyiUtKOK2loU6zLH|$ zsnz1sod^|}Pes=B85ySgRUHf!y~7gbwDe(p{H47khq7yDm}_9v1U}@#OsSX+GF1hUbf6+Hg8uFCnSfkIF8R!3uV8+pjse)UK^B%D&2q9|%GF96Urhx{~P zc8fiNccf$)jz0TA__E`)adOdgP=pAU{zC}%EWo=G2Mv9IQ-@}mXWnG>tc7zfq!m`6 zuFa8Ip-r=*69n#swj27D(Ty7Z;A+t&!zhv&Vc;c0$Ar)~n7m@H({+DnZyv9D>`?kN zQr>FYXo=-LH0D;v_UTUn;ulc9k9A=3&0w0eoR5v*eYiLVoROKA&GW}>JL!-dWx%;Q zF@ev;7Lcneufe#4tkhlC;GWBc%?Fqv(C?+2u0i|Ttt$6jqY8fJyW+h_@qysqYi$@9# zEuXF&9@+wZReDUbg~BUdO-bTHF2A7@y{C{;3}GxsG&j?PW!bhx#jz7E~gC?B-Cz0$7}ij78dvS zsfEk-E8|i(6jL}El7%uIq?4#pXLojTU0n8ICL+OFc-2qE-mVF|CQ$55WLm@=3Axzs z{?c3rWAL;RL zQ9S+r;|7%Ka7@%`PUl9K|4D$ee?7p@h|oOs#J(#O5h?_WcwV-AzFX7v)s@dYlcwYB zq%qv{eDMjQU~zxx&w5ors<_*EbNN&14$Y&qO2y8dooNeko*$|V1-lY@9cTID;Anmc z+p!5`I7gkw$zp!f?kwxpO>%^%?Kri@(s{}eLE1mnagy_N1|MT5@8B`lJ`tVC4UP4) z)xKlb4{;6Prn3UmYj#h)B{BqGwpIF;(-lkigS7?T9yXmkIm3{&+M>h9cdHJU;?i#LgzuytMB7YtuSWr$53njzm@e-!AcUYk{HN|O|SsTEA z8{T5!1~lX|_F0>I<&Q9V^IETrr=aUgxyHeG4?24UZV#!C! zTjF`!#`*ouJSA@b>8zdRU%RR5u>zP*z;_wnCu%M=2uq|T4ntfL*VVBNRW$K(!2&06dE#hc^yPG3?W zsqD-0NtF1w)=by!=cNXA19}$8E;e}sb`-g}9n8Ip_kXEkSQ=U>7u9;7(K9BZ7SsOl zSfJfQ;o{71FbVIdlk!teV0;7NBAZn<4@2N%tPA$VGm>mpqABB~aHfaUqH?zsus;G4 zrjajwPzJ5*Z+G)hkgB~gS5Y;(`%Aa0LPJ_NR?$ssR%J9zBFCM5zrvpGS5|*)29M)E z*gSV!IiIySf#Tvu7m!7|TQ^jhm|!k>g2O$kK^V?cGh6Q9zl{g;Np>Y*V zi!R5Nf4qnBLQl;9Qhy(dJ!qr|TEn-HafhrO?K)=&2d`h@e$U~ct?x7{+ZA=>r*clN zZHcyZ8@!P@W@Dl4G%?;5VCw;~|0_9wR+Kwd{iq?YN1GmAY;ZwzW~I*Vkm7#a8dB5> zbZ3j;Q=}cPY|)1122|*-)~$1(jN`QC7MK3QAfR$2W0dq2HB{2d1i0N6b-mGWu~XF(8#mOKrU_vkAHw7*f%qS}Ff z4Szs){0DDkpR4=O4iYoLbHrFSSxg-#5ZP{2(}@swet=P=5Ou@-U=N+b&koe92eXiM za;z zZ}ntLPi|jTRy|u!ZSC>pWQOud1uC@1Nzn}sZMS1Ni}uNn6$pP}LDSs%@pf-aLd@jz zU$aUab4X8J&hgz1*KTeGl$hpQAbv00Kc0}>);(x&s@_2+XIhF+k-cE3+7cJ7l}Yww zN1+KwiV(B+p+Yeo2`)$KZa04qz$9JM7i&m5__7-DJ*-f*rvchCw3A+U|H6Lz_BApG zVQRij)%kqtS+7W85_p5%NTz6e$7fHD0dP#5&MW$ev(F)aSUhD)JL;TSU~s4q({1p~ zUwBE{l;n0sCM;8S4=U78Hu!SFrAx~AY;DamS6u7Bk(Ym$HJjx38q$=1KlQPngqRWb zYKQ!ipx$nwP69o3*`IUGfxHt08Vdh8Ig~a_l5vEDPT-gFWox1mt*2ls5dQZICD#7&ncHu0P` zj(uRo{yq>&5rRGZl6#+h2|tkyqYtKoYL`1mp=y9KSKRkM(o9`DX%FPM6)>CIu_WRg zj<#o-HDD$?4}{+nb3p^B(8zdv`A-7aGtX`=F5Gco{~MKQ{0AE(qvvUQ^@Jo|F>7V= z_6au79rkn~dn1=X(->eV=}=#^QNRj$ZZ(B;I|$ApX|a-{L<#% zkotmq7+Q+(S}+AijfcsH3S1eTo<0Ja&RbnF&$BBH9<@@&Lr|r^H)Gcu2}ak>RFyNt zLx`UyvhSWy*CiH@n(r&j%iY~#q0U7thSZ4uwRv<;cUsX865~BrDsArwJ$sd6;W&PS zq}P5SKyd$L?OmSiru}Gcw|8jYF}M!yJquIh3QbNkBKq_1n-Q1zO=vn%VTXwx{S0z< zsGp@354#sSHl$e*T<35RsmRrHloxcXWI=g#4XDuTx!h6Zc%=p;Ym3W~M zr@8qmLj)Sf*09tvQ~VNrN?SBp?3g;d#`wgc_#|^Nd`;wFYK((s|20hU@z3s0 zk-hVIdutu2v0waICHKaUiwAu_ldQba{JNgY!;U9s>!}u^oxFx$nso!(+%Sn`i7cIr zb+GUw$K<(9a2Hn!rga|QjT7z>QF@Ugh#Tpdl59GAU7{D*1EzaA+E=UK(V+QpEWcz0 zXsSjf%1XPIC=f%PH&^Xy5B)^hne7!kJ%<_&G_~%R?W_5nKKc8!08%w{=X|cs2sIgY zhDz3mE}P{oK0BT$b{u`eOb>f#dg}iha+0!)16QeEz+&5-e}}nR3SoWiwB-h2CM!id zi9`H+pL&8Lx$3ev>pOUcx5vkSw}JO9;=RTp2tFr7*J;b<8?ytZwK~rk;9mBJP`D9t z2{?-b_NjzTm5&zp#puybR&Rp2jGe9a$I9uT&NMG%TkXY%g<#%jM<|~{*79+{5KcQD zo1FVsqOdaxm#tDwL0ccixH{fb{;2a~U}8j&dYGt|4J*~)Jzq3fr69%# z$pA&&Zb~a5dBSmom}h<*D;HO;TSC76jFwI1xlc5BgKM=1s7HrLj?=*3~AZI*;%Dte<} zH8B=qsB%jO^n;p|rTBHSNp=C?Xm6#pPu8VZDXS%&eBRV&f${e|g+=gIWjeYUyKtE} z>ex>G0iEe-)k4KfEd&MgEo9yzF|=W`vW2k&Y`sT!8Yc14z}NnFcd&4U_|+q?_=i0= zpqNpr1Jj$MW4(jg##uMStCU>P-8yD@{I>$$c2!kTI@*|IHN#y!mBaQ4Y4%rPj=JI% zd>DP;*If-JX^@@dl6R$Iiw4{eiFlL#692*0D%wkai74``rQP~DKd}rkEBzVLHYz>( zs*!REISSMwevy|=!k%reneD&G@k5B0*IpGf*salh>;0HCjiJff={V@oH^^5IwEreh z%qZ*z>j}jQo30D)za?F(;fCd10}ffBqS5mXDqDU>@34g|pb}w<(Z@q2KDR7OX9Af4 zlwf|{Y3NevtGOuWWZ+*_U;ekXAxNZjKtKWN zj%BeoWaYoqx%Qi!{XIQ(JBnvkstJ&zIUn5aIbA+T zdVc7J+j(0+J1-Xy+l_HLV$KmIp4ZfwOj-qAK2GAHcx6z3h=RyVT7LiT|G|>lb8*->k|oT{RewON?~*e&upU#3+B$4cBkH}g%8Fpg+ksJ~J_cwTq>T$zWz zJ~V6E&^%)D-?T!=Baw6c^9E@JML(FjrlZ-4)$Y=Op|>4}Ue)3W!lrbo11;oX->VF^ z^S?vHukF;gQ)L~J1Us1AF{Ak_s5B*Qi+B$)I#~FWcI|XmY*x-T8>U{yABUHCE7)X{ zVc(LkT3;%>(IOfb>95oc&h4CfP#?-RouYP~>ttz2Uh6R5$yGPkvV6D9U~y~r4UcxN z>+)D-5X3&@eT9exs7oh)rPw1P~53@^b-&5)P&XJY^RD2vc2KPurqlvM4dU(7r? z5m$v~-(SvYWCYb&+R;08+JqJR-k|Why7ud2A=CI%((iha zR|g2kRG=&37m$wyeLpdzj>4i0-T5s0q&vwGkLGg

$0X~RCJ zF&-8KU|Twm4X|AZ34>K+o-%RkD~M<>DnC*nv`iAly{s(P`9lfN$U?Jx;Ym3i5GsFX z6g@COlTc3IkTUajh9#iHGM^Jr!V8tnMR&VDp~>OS@MDrI2M;w_Q~gqk1lS0b9LN0TE%LK;;@x}g*A~9}~&Ba%A5He*>9OI-O=d4`UN%{gfsrit(OYTk(1ZWG% z8wKA!gx9`cq)Fa?NHs6@x%<(+LMG!(^?>9ou=EN^ftHiv;*VkF2FyG( zozl~zZ@`-jG-;leLezpThVrsl<%_T#25w;}@1Ew3gXMnpYrJ{9nY_^m_hepz13o!( zGjz!%-G-5FPi9e1mLMIcq(o4)&x%m)C?#S_;jw~8Fq6@CeXVq^xa1n}yT47qYI@lV zG~N72MTvEKoAn0rzRKQYaPvgf-nS_Qgz%%>pkq`}Hl7M^RY9Jn;QJTaE-p~YwOIw5 zgg~0T+r+S&3^!q%(k_4x+~KzEai$R$!_upOqCXdk%7FRwazLY&NzBCs>t9O&Ow4f4 z_g`%A5R8ZE+*|-=cbGao&2uo>m8wcW)%~=5Ac!5-;#%~M4-6RMW-J~;Ba&bMw}gE_ zGu4mOGIwXx8={6QOy%yVpm)ng?|Fy7<`H7rH&dsi2r>F0N5C3uNdX?9-vfU8$W4>; z*f^QCaZ2i}5#h-!3$W6wq_wUTy^2gB-x0p;iGq-nJLDJegKSMowWW!% zr&_>?w-=2|z4DnDTKFQTzon$VWDLw6K&tsbVT<)UMApC;GD@ztw9qFswuU61M{AH@ zQlUPLRBZ??<5_iS9jf&Z5AlEXwLmst7FvLo`ql5jo|E7?|&~YJ^_vg=xQxHM$h9f_55sHtV^h*fK<9XsQLpn`Y!gH7*g7H_*wl4(R^zC<(l0quZ=}qRxg3%Yr2??9u3rexmwef^}R1YB;O1`v}@(9LB zbzzGYCHAV-`?&v=i1wc;DOWFGu}4kedm>Uogi0t-CU|g9AMha@?0XqxO5n*cf)Qx@ zafbwP#*z#b{*T(e{2%J?dwg0DH54Mo5?Rs^36p&fgJ@nwm=;u)GG^>iWE)XYm=Y1$ zk_<(RrG>FHc2c$(`#QF4ErjoVj^2;&U-0qMbDsB{bI&>VJkLG%+;gvOTIx8Z0}w4) ziiA)m;Osu&tR8R{k6^|^oFV8+JkEf~gKZ+?MEg7RO@vu1gF&N(wFus=Fh6=3rY^ba01Gg!}ZSaTQhO4aC9H^@=>u*zX{!#HZtb<<1 z+(js3Y;~~Cqh`V5kW5oVE~wLkyHHn8w?*Zp0y`?*W7{%4 z!@ZIOM&Q6XM7%w9pgFD)xO^Wi+TafQm6)80j9CirEnN0p#1h6%2TL7gInerwj4tT4 zbSPqpkLy~ny^Rj`>vq#bMMeqKG#-i&$OyYlsc%U5oBA23=p6zb4OCoeti?g{{ZtJsHr{4-3BRC;*YZSd*(nsP1X}QSdVS9cSR!I%Kd;}eso2pd?E@>Grx}bwi z*`7uGC=%X~I^C<`*WLo7$_K=_hXJuV__Gmor<6XZHr$O$H2rv+@??8HDj*mJLKw_y zcy>I9VLZ0SC5Zlr*OnJo1Bx3(>DJ12pl?B|C4Vk3v037h_<+3?h~W*VuhOD!IBqNE z7)q*^w7DAMyqU$ro^HMibxz-2hkp6yzvkmmb6AqXE!?*-#A^v@adPB#PyO`64Wt^~ zV@vd70LK`ugPqxy<$Jk=1f({FAtKY17PY=nyzUXhvf;i-Vg#jg%gZ3mVReXj%&aMD zJzwEp(-gZpd1bg^to%GT?p0FaZOYH>Z@9pR+F2g*Uo4W1x=*3J_#K>XtuFTmzrJM? zBzzXkzu-EQ%JTRLXvBpm8SIULy3bd^1PD?Lfz~!jg@BL9A)@`!(G5HthB}n(foqrb zHYH?-tPh^JNVNe+bw(8bR#qQ8K1O}-#3crTj7s|cwB4JwI{YJO??Da@wraR4!BN*} z{ufnsC+JVDT;it=DkKI|`m0*s5pSsQJjg>gW+j-C!=nZD+W_wb<&c+wt+oGn@t_Wz z6WWv3urVt?C3iJM7yBQGiQJ+Qodc7nv3X!1;1Qtl6Eigmd=EFoPCP(7=;T6WSul@~ z&mkxoUSpGlb?u2R(DfeBwE#t89F#>4h|98C-D`hUZe)jC-r{`Fzl1gf0QHp;RCC|S zk38JKSk75=t~2bP@k9Zdc+}OP7YM(nzN;h0iQ8U_NUWR;bFD1hue#A_;bA7nP)=pJkoEzvf z4|i|8JbPmgSV8Q-y?StKQR?CCG7G6s(jmGvTkojZ+S8GRs2lb=B4F4u>DDzZ`@sSM5PXD8%j)=j8|1VS*~=vy(j0^V zIq>`aDKZt?cbZ=TrPzn-IS z*Cu=Hz_Gp$J_4=Y=HV0!@Fu)+Kw>=kd7j9sIE;SfZdpBVx8gIhtsW`i2k2o3?&w2E z(GQ!VsPG($&qXh4N>2Xxe@dX}dIWv&E~4>e70^qaZr;0Uc>M(dsICuvs|%lCh|TlC z8=55wTXnjggS0~IB2e5aFejOuvAnj&5Oiy<@rHzwclp9KNOAg=*lU`V@Kwv#fO%y~ z#=c|GvTo;2tQKb zl7OM{=QILt_?Cb{>61Yn%FT3c&l@2rQ?sDU19a;|u93awb{Mpuc`62mc$i?5UBRilKV`C=pIr;RZKj0y$E-9VE=Rvd2lyM z{w~|16#PDD)`gRG|<6JjctLtHGp@k!2%Na+Og_4A&CKqHcnMkehU64?}C`m*QR z@TKFmBmp+mSk1?PcpKC*#P+Es)iz9gm(perF=}TJSaR;CKjJ`+nhEDX-h+VaLSO>D z<(FLExZjFKD(uF62?a|#crk>k)!i`af1|(0;hpeKzj%3YW)PT6n7nLWPd$; z;#U60vNdd=raqiqY*Iyh1ikkz;+<2FmfV$K9s0~coDeE$^18>#3x!yOtdm^>SW4|2 zB0f+7>y;C-;^f9O6`WEVsiJQj(Vg(kdfia7Q-Yi*NPiSf1j5HJ4PSbLaiLW294N;& zUH^M_%Ab7c;-yQ$%kSz_nwo4a#EFM$NnT-yckR!?AU)@?oi?k73;ggCeH*OWg`b-n zPkB1$t)~oNLJ`XCZvrrP6M1ai3JBqBUCQQeeO5{zUhLJw%JEPnNE3AlB^4&i!@-rE zEgx3;5_Oj}KP{Vl`!s2}pryBa1g-uV?xv!gmUW;pDrgIo3v+x1Gpcep0^{ngWoFnE z71ZK$5!P&gl|xb-k6Xsdy8ZrxZ<(Gid30aUK608`eiT55AtpLj0x+ND#piWze>DEv zvvQ)PDRRTcaSeZ*n~-fWL{u*UidpQOS>HNaX#Drf%8r)3xFWU@wB2WTbO|t+#d`0Y zes6lan1y(&Kg+~VDHcH1NI0jOaN{w1$Tu@sRm4rQ2IqW3;|B! zKnoa|YPb#1%no|`S716jQAxW|7q|%02k7Y?%zwXoz;90a(ATB^weKZJ@1PHLu+}RO z{60c9sDR8~WrZ!AxFVqu^jUnkAqOj30m7@+k|c-NjRpVr8zhMhhKSj*toTEUQtfvS zIXy%q|L#S_1+?6a-9e%m+b8w`JxaDh^p;x~7P3`r5k3 zN_sED1xAUrnU_z7S0COAFaaAFe<&DPMmunVFLl6ebUyiwXeg`B}N+=Zs^ zwS^DW47Xk^5G^A8KqliM(zgkIF;-S?slxF_iGZ6>sha0Cr=GoW#y7?lhuA-e&G}G2 z3rUH2EUc{c2k%ulelF@tknh?zQfQDhbm>~r)|H_*Ri*tr2Xtq{p?z+rDO|_0KM$KGv)=V=mn2jsOQ0@&LQH~7hwuY6y(>d@6@PB_+CQ>?l_Yk{H_Oj<#>Im$JBb{8YcDuFtXP%VVg}QkLRg~(wd#mt* zw*3ro*xmEmhfgDAbk4oqiO0G*Dr1#hIPea-JHCqb$F7g3R^zj7e-zfihan;i_|yhL z#^@xuOBCg!otE`wnCKQ%^iETFjyoshcS{5FC%#Cmgbx%{MFQM= z?9BZQtHcU?!%=T<^m|TcM>g&nc=Y+pdJxl6u^j)^VImH3(_JEWH|kzwqc8I{LIY$~ z?ji2Us$cb|9(MKkE8~2vWy||e$tE&7x-@ghEhz9p(EhVjEUK2-;e@)$u9;MYHx|Fu z!yOxsat=UIsK0tRPX~j-2!UhWdBwP?OV~UF?r@sQk*A27BeXjOf2JxG9(z%lRW3?5 zxyu-nKAs%oJd*CF-iB9=D&ONSboeO|hM$P(EN$ghu+I-;IUkyv#JYze#tgg;51_L>d4zUHU9ASjqA#d z5`Ft4G1QnRr%trNI|563;JYAl z=k}>Oa1>sxgG+a&QZP*nJuSJ2G2)NWgp!|ES`el-50~^By;^7455Aw`!1bKaT+jKe}QZV-+I;WfAnP23)7?hpguea31{_vSp$KBKVX*^0hW+d= zUVR^Vs7K7mO;LTpbr2@_r5yK7M@l!7R=d4JY%?vH=C?*iwo%6YwO zx{s)RSEO6u$Pc7+-IwprR5x8a9cbL4ZHQq>};f`W&)O8?=;46AkTxT zsLxt4)Qk|$1zWXYN3^zwophFQeSW(7MF++2Z()FT#XS2)-neGq)EUbOE@EO?80Q=N z`Tm@;o7$L<=YSOEGo$D^A0UNLHN5z5=xX(6GqdIXNDqal)pNEt=Zyqgrctx*g%{hK zzJZE*-V_tX3W;|et(A^_9x0-9(vM}dUr2ISmgUw z;1MS7LIGQ!Y0xNI!F$(YRyiwdvFYriU@GZ)T5e!F2OwpasF1skq4ryWL#-f)Dk1sR zlk7pvW|qeS|K8b1!2|(tIZ-G+Y;@5j@b&jji1688IT-QR$>AmggUiJRhW zi9@P(0R;-M!cd=>`UX<+H0b0$D+1OyO&vu)1eTQrGDd*siV^ z6wETDU;;}94mUhDj{vwi2OTO_Y1HZw*-;lf-cA3i{;f{ir*oS~vb?mEN*uJ5d(U<_Hn$rhs{=oUN+_!Rs;QWcs~z?B#( zNOBj91+3QoiW0v8dfyFG$j~Q9&j8=U39~o!9zj0=dgg_dMAz4nX8l+sgy7LgJ4l_d zc8%l%w(oja)E=rxlCKs5{p`>)cLPDMJ&W*nrw}X+v)zaD9W3n(>+y;6Ei4Va4dCpL zr5Uguk6-ZxMIEfi*ee7qO@sB=dc_k*eXL`8Vn{0N;Mvo zL(0?JIWdYRvWDWY&?rIrBumjsf>eU^5!U0QuwF|14i;I{23KreFzfM4L^TC7w+(Q> zn~1^1n3^$S4RSSnt;c<&y%8&-=5xIXF`OkCe?uJd&JVLyLN=4tc{;y) zG;(9=R-IAD%*G(|h3=%h+^x%y4ib$-3EASmj0TXtMqia1J5`otI5UKF#e2qX_T>h|&uEM3+xDd6OAZRR(zECG*`!|dt z>rjV%L>i~ms=C+9*P{7aC3MbSLS9C4uOud-2k4SJ{{_v#qIkkJ3 z4P8j^*HZkpTmh94FGDiQhdJu*uQSq1ww^*vXGe^ncRaoS^o^yn!jT09{!B^mxmn9y z!cG@|B3BOEFNVGEnBqPC(oo{{z?KR4{?hHQ6SDEvZhx;YdWE!kHinep?Q}Pczra7p zvU)PEBgInPj~>}E&vD6c^Zv;B2NQO>nlcbog~?05znwyJQac6=Nx>LO*5I6RxgkW; ziTszAgg_sP{dqlApe;K1_ylnv#C#+$BRUv|LslKdN#BQwzpBtA?aSr&8pohl3_Rdjo`Vv1Pf zO$xR{^KB;VR@Jh-?GCQgZysGr@vP$zR@d)S;{8%@6bYm9O^_%Gdwq%YgURFTu zn*S9e`_cH_K=9%KPy0@o9s_&n`x;NTCqHtQ<7>*_u+@%Byeq|n)V(lR4#jKPW2+op zM-&Oh{(6aLA12WOiu+-(#KxBV$okBXzd^=on*(iWEa>uZA$aNgi@B-lN0 zsvdTj3kQy_f;%K&u$TWAFPjNeT1R3hVp=u<{1b-$1XGb?^e$M8Vz|;i;kSH46bp;2a4q2_P%@>`4$fMfqFB#UI?W0qcq! zV*hr&=j+9@hnow%f4u3|{JOa%m1~!?0qRO&)*J+sg{1qJ25{Ul9?c!}Vf{X8p6$e) z=R07@M$PaB_g4~wrgvMr20uIjPSagHobMQmI78S4(>r;uVwf^imGx(X z-3bkYSsrE*Pu@BHxK{tow61!mfweas53Sx!VzeRjC;NS=f2+bXKe(Kk2>(=HJrd!Ya|BgQJ7oTHHr_M(j z`S`4F7Jn@c-w3DDrc?0?4J!n8m@DsDltur7X0&_K?`NI3?vb%)Y2&xEWVQExh&tII z>C$n_!X;Ru?fPYnJ(Ctzx;C9jYgEgUq)p$k+zTA3BAy+^{RV1f+lC{FJC$@Y^cyX@ zQd`}-f~%M`Z>4#_#l*xVVOo4W4~MTWPpx0KUO4~!HLhW}W4NTpMRp)i|05&gM*brN z_lwneaI;#Wr|J2H&8shK@mniDOE~zLn@KvYKtqxv?BA+nv-PbUW;KgT*J;JQ8*7eA zxI0;lud+Wi!(vSqR4&|T+i+m~YKsYaIk-CLx6Uqs(n?{=hRn`b2c2->444*G-@!WG%npf8L6lSBUf9p7l^0WA{{>P4j#l?s94M zXI6*L_9p>W+IK$CQjPD(EWRE8S~7cYl&bwpqHwY2n`ohyg5`X(!--kqX!^&^%R!r$ z7qz&yv5e^W_)`OTUSUS+#zM2W#Fu&Y1|FAKu{5Kn>n&RZB?som?x%aotIZSl(3KW- zbsj0JSctp8?#diU%}q>N6Sy@}cXIr$vnqA!=dgb(CStl>RFmTUN}=s%?L)JwQM^yXAQ7c*##Y$6 z8x?S#la3#$oQdvQ{b#;i@C<+XFTYmSLS6)vbNX|8Rp?)<`NNiGab_XP#oSv;DvsD#rQ$I?3!CLLeh|q!>xus!Q@sBTYnp|aCoF6P>}Yv4E6Z3 zrpwx?T`(2S;X57vFTGA_5PXL|@yttUINcdqJysifrOb>+`kN(IR>bW|2P1vy`T6s) zqvA2RQ%joQSUN2Ia2Vpb3zIGM*=T=p+TCR3G450O4TZ1YhS5Ac3r>2f)TA&2y+n|| zf)|2^5)4D=>W2nT%}$Rpm>q&^MP7lQB&PMP0iDC*VF-HE#X%iS4Gj$+v5LSc#BlG} z)D$fZrQp^kWaRf$y>hPrq^r6>jMEBonws$dc? zAf>GJ^-}wgfQEWn1RXCR_RAFExhF+7`SZ3;!TI2$+9H-R39=n#kXN9#Tcjt*Cj5v8 zF6$4YP1c3^P|F|pVw!m>KzRY#YaQp8d`gk5qaOH@U*J|$_`lgn{yV>e38jgE*X<+x zvm^&PUbwYR^M0$n^0sN+r>Aq2;(@j7=o=xLDJ{B%sAV2|F7Ck^+UQb>zVxiN_lu{5 z62Xw-yluJwa3j1o;paR>G3?%iX$s5hyi~;{`3d6IX?#Hq9Xyt-PqIT2WwFkEOxoE? zMtjfqZ68|TMHSipV8O*V|L3#i|7Kv}<}IFL^9sgdANmTI$oYJXFZkfEUe~a>=H>0? eiGx`iomMz;T=hRS`)X*OH#0tCL^ik*_WuC=hfp5? literal 0 HcmV?d00001 diff --git a/sections/0_causality/open_data_cost_savings.qmd b/sections/0_causality/open_data_cost_savings.qmd index cec8cec..0ff5fa4 100644 --- a/sections/0_causality/open_data_cost_savings.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -27,11 +27,9 @@ Despite the theoretical benefits of open data, several limitations hinder a comp ## Directed Acyclic Graph (DAG) -As discussed in the general introduction on causal inference, we use DAGs to represent structural causal models. In the following, a DAG (Figure 1) is employed to examine the causal relationship between *Open Data* and *Cost Savings*. The visual illustrates multiple potential pathways, including a direct path from Open Data and Cost Savings, an indirect one involving Time Savings (i.e., a mediator), and additional paths that incorporate factors affecting either Open Data or Time Savings (i.e., confounders). These additional factors, such as technological infrastructure, data quality and availability, standardisation, user skills, innovation, and collaboration introduce layers of complexity to the model. As we will show in the subsequent sections, they are essential to discuss the causal and non-causal, open and closed, relationships among all these variables. +As discussed in the general introduction on causal inference, we use DAGs to represent structural causal models. In the following, a DAG ([@fig-model]) is employed to examine the causal relationship between *Open Data* and *Cost Savings*. The visual illustrates multiple potential pathways, including a direct path from Open Data and Cost Savings, an indirect one involving Time Savings (i.e., a mediator), and additional paths that incorporate factors affecting either Open Data or Time Savings (i.e., confounders). These additional factors, such as technological infrastructure, data quality and availability, standardisation, user skills, innovation, and collaboration introduce layers of complexity to the model. As we will show in the subsequent sections, they are essential to discuss the causal and non-causal, open and closed, relationships among all these variables. -Figure 1: Hypothetical structural causal model on Open Data - -![Immagine che contiene testo, linea, diagramma, Carattere Descrizione generata automaticamente](figures/1d1cf98828cbe8eab6cfc7690c05d6db29154e28.png) +![Hypothetical structural causal model on Open Data](figures/DAG_open_data_cost_savings-0.png){#fig-model} ## The effect of Open data on Cost Saving @@ -39,13 +37,9 @@ In this section, we apply the concepts presented in the section Causality in Sci Suppose we are interested in assessing the *total causal effect* of *Open Data* on *Cost Saving*. According to our model (Figure 1), there are multiple pathways from Open Data to Cost Saving, some are causal, some are not. To estimate the causal effect of interest, we need to make sure that all causal paths are open, and all non-causal paths are closed. Within the DAG representation, two causal pathways can be identified: a *direct* pathway of *Open Data → Cost Saving,* representing the direct effect of Open Data on Cost Savings*,* and an *indirect* pathway *Open Data → Time Saving → Cost Saving*, where the effect is indirect and mediated by Time Saving. The *direct* effect captures the immediate benefits of providing free access to datasets, while the *indirect* effect, mediated by Time Savings, strengthens the relationship by triggering additional efficiencies that also lead to Cost Savings. -To properly estimate the *total* causal effect of Open Data on Cost Saving, an empirical model should not control for Time saving. On the contrary, if the model conditions on Time Savings, even implicitly (e.g., by accounting for approaches and tools that optimise and speed up data access and processing), it closes the causal path and introduces biases into the estimation of the total effect (Figure 2). - -Figure 2: Misleading structural causal model on Open Data, conditioning on mediator (Time Saving) - -![Immagine che contiene testo, linea, schermata, diagramma Descrizione generata automaticamente](figures/f0d9bcfde3955951768ec4951070567661000e25.png) +To properly estimate the *total* causal effect of Open Data on Cost Saving, an empirical model should not control for Time saving. On the contrary, if the model conditions on Time Savings, even implicitly (e.g., by accounting for approaches and tools that optimise and speed up data access and processing), it closes the causal path and introduces biases into the estimation of the total effect ([@fig-mediator]). -*Note: DAG illustrating the misleading effect of conditioning on the mediator variable, Time Saving. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship.* +![DAG illustrating the misleading effect of conditioning on the mediator variable, Time Saving. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey arrows indicate indirect association that may involve non-causal relationship.](figures/DAG_open_data_cost_savings-1.png){#fig-mediator} As mentioned before, the proposed model accounts for additional variables such as *Data Availability, Quality and Standardisation*, *Users' Skills, Collaboration, and Technological Infrastructure*. These variables can act as *confounders* along different pathways illustrated in Figure 1. Examples of non-causal paths represented in the DAG are: @@ -59,21 +53,15 @@ There might be instances where a confounder is not observed because is not inclu Another case that makes not possible the identification of the causal effect is erroneously controlling for a collider (see the introduction for further details). In the pathway *Open Data* → *Innovation* ← *Collaboration*→ *Cost Saving,* the variable *Innovation* acts as a collider. Hence, this path is already closed, and the bias arises when controlling for innovation. -Empirically speaking, in a model where to estimate the causal effect of Open Data on Cost savings we condition on Innovation (and not for Collaboration), it is likely to get a downward estimation of the causal effect, since both Collaboration and Open Data have a positive impact on Innovation. This conditioning opens up the non-causal pathway, Open Data → Innovation ← Collaboration → Cost Saving, which connect *Open Data* and *Cost Saving* through *Collaboration*, creating a spurious association and distorting the true effect of *Open Data* on *Cost Saving*. This is an example of *bad controls (*Angrist and Pischke 2009), a concept explained in the general introduction. Only by ignoring the collider, meaning non-conditioning on it in empirical models, we can effectively isolate the causal effect. This non-causal path is open because *Innovation* is open (because it is a collider that is conditioned on), and because *Collaboration* is open (because it is a confounder that is not conditioned on) (see Figure 3). - -Figure 3: Misleading structural causal model on Open Data, not conditioning on Collaboration (confounder) and Innovation (collider) - -![A diagram of cost saving and cost saving Description automatically generated](figures/26de0c31a0e5f229ff6935be961496367a3b6cd4.png) - -Note: DAG illustrating the misleading effect of conditioning on a collider variable, *Innovation,* and not conditioning on a confounder, *Collaboration*. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship. +Empirically speaking, in a model where to estimate the causal effect of Open Data on Cost savings we condition on Innovation (and not for Collaboration), it is likely to get a downward estimation of the causal effect, since both Collaboration and Open Data have a positive impact on Innovation. This conditioning opens up the non-causal pathway, Open Data → Innovation ← Collaboration → Cost Saving, which connect *Open Data* and *Cost Saving* through *Collaboration*, creating a spurious association and distorting the true effect of *Open Data* on *Cost Saving*. This is an example of *bad controls (*Angrist and Pischke 2009), a concept explained in the general introduction. Only by ignoring the collider, meaning non-conditioning on it in empirical models, we can effectively isolate the causal effect. This non-causal path is open because *Innovation* is open (because it is a collider that is conditioned on), and because *Collaboration* is open (because it is a confounder that is not conditioned on) (see @fig-collider). -In addition, *Collaboration* acts as a confounder on the non-causal path *Open data* ← *Collaboration* → *Cost saving*. To identify the causal effect, we hence need to close this non-causal path by conditioning on *Collaboration*. After controlling for Collaboration, whether *Innovation* is conditioned on is then irrelevant for the identification of the causal effect. When all non-causal paths are closed, the research design is said to meet the *backdoor* criterion, a formal requirement that ensures the design blocks all non-causal paths between the treatment (*Open Data*) and the outcome (*Cost Saving*), enabling us to identify the causal effect in question (Cunningham, 2021) (Figure 4). +![DAG illustrating the misleading effect of conditioning on a collider variable, *Innovation,* and not conditioning on a confounder, *Collaboration*. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey arrows indicate indirect association that may involve non-causal relationship.](figures/DAG_open_data_cost_savings-2.png){#fig-collider} -Figure 4: Hypothetical structural causal model on Open Data, conditioning on confounders and not on mediator and collider +In addition, *Collaboration* acts as a confounder on the non-causal path *Open data* ← *Collaboration* → *Cost saving*. To identify the causal effect, we hence need to close this non-causal path by conditioning on *Collaboration*. After controlling for Collaboration, whether *Innovation* is conditioned on is then irrelevant for the identification of the causal effect. When all non-causal paths are closed, the research design is said to meet the *backdoor* criterion, a formal requirement that ensures the design blocks all non-causal paths between the treatment (*Open Data*) and the outcome (*Cost Saving*), enabling us to identify the causal effect in question (Cunningham, 2021) ([@fig-adjustment]). -![A diagram of cost saving and standardisation Description automatically generated](figures/0558967f8b3c0c345051326ee1dea0137c4caf1e.png) Note: DAG illustrating the total effect *Open Data* on *Cost Saving*, conditioning on confounders and not on mediator and collider. Nodes that are controlled for have a thick outline. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship. +![DAG illustrating the total effect *Open Data* on *Cost Saving*, conditioning on confounders and not on mediator and collider. Nodes that are controlled for have a thick outline. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship.](figures/DAG_open_data_cost_savings-3.png){#fig-adjustment} -This example highlights key components of causal inference: controlling for confounders *(Data availability, quality and standardisation, User skills, Collaboration,* and *Technological Infrastructure*), not controlling for mediators *(Time saving),* and not controlling for colliders *(Innovation),* as shown in Figure 4*.* Constructing an appropriate DAG is important when aiming to draw causal conclusions. Without making assumptions explicit via a DAG, it would be unclear which variables should be controlled for and which not. Omitting important variables weakens the study's ability to draw accurate conclusions about cause and effect. Moreover, adding complexity to a DAG does not always change the variables that need to be controlled for when identifying the causal effect. In some cases, such as when adding confounders between unrelated variables, the identification of the relationship between Open data and Cost savings remains unaffected. However, if a confounder is introduced between Open data and Cost savings directly, it becomes necessary to control for it. +This example highlights key components of causal inference: controlling for confounders *(Data availability, quality and standardisation, User skills, Collaboration,* and *Technological Infrastructure*), not controlling for mediators *(Time saving),* and not controlling for colliders *(Innovation),* as shown in @fig-adjustment. Constructing an appropriate DAG is important when aiming to draw causal conclusions. Without making assumptions explicit via a DAG, it would be unclear which variables should be controlled for and which not. Omitting important variables weakens the study's ability to draw accurate conclusions about cause and effect. Moreover, adding complexity to a DAG does not always change the variables that need to be controlled for when identifying the causal effect. In some cases, such as when adding confounders between unrelated variables, the identification of the relationship between Open data and Cost savings remains unaffected. However, if a confounder is introduced between Open data and Cost savings directly, it becomes necessary to control for it. Klebel and Traag (2023) emphasise the importance of carefully selecting variables when analysing causal relationships. They warn against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which they refer to as "causal salad" (McElreath, 2020). Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, Pearl proposes using the *do-operator^[\[1\]](#footnote-2){#footnote-ref-2}^* to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system (Pearl, 2009). From ca24df16ed606177b84901924419c1b2f0731566 Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 15:13:40 +0100 Subject: [PATCH 06/13] Clean up references --- references.bib | 290 ++++++++++++++++-- .../0_causality/open_data_cost_savings.qmd | 76 +---- 2 files changed, 272 insertions(+), 94 deletions(-) diff --git a/references.bib b/references.bib index fa53afe..e1caf19 100644 --- a/references.bib +++ b/references.bib @@ -41,6 +41,17 @@ @article{aksnes2019 langid = {en} } +@article{ali2022, + title = {The Role of FAIR Data towards Sustainable Agricultural Performance: A Systematic Literature Review}, + author = {Ali, Basharat and Dahlhaus, Peter}, + year = {2022}, + journal = {Agriculture}, + pages = {309}, + volume = {12}, + number = {2}, + doi = {10.3390/agriculture12020309} +} + @book{almerud2022, title = {Knowledge ecosystems in the new ERA: using a competence based approach for career development in academia and beyond}, author = {Almerud, Mikaela and Ricksten, Maria and {O{\textquoteright}Neill}, Gareth and Weijden, Inge van der and Kaltenbrunner, Wolfgang and {Núñez}, Lidia and De Coen, An}, @@ -128,6 +139,28 @@ @article{armitage2020 url = {https://doi.org/10.1162/qss_a_00071} } +@article{arshad2016, + title = {Open Access Could Transform Drug Discovery: A Case Study of JQ1}, + author = {Arshad, Zeeshaan and Smith, James and Roberts, Mackenna and Lee, Wen Hwa and Davies, Ben and Bure, Kim and Hollander, Georg A. and Dopson, Sue and Bountra, Chas and Brindley, David}, + year = {2016}, + journal = {Expert Opinion on Drug Discovery}, + pages = {321--332}, + volume = {11}, + number = {3}, + doi = {10.1517/17460441.2016.1144587} +} + +@article{assen2014, + title = {Why Publishing Everything Is More Effective than Selective Publishing of Statistically Significant Results}, + author = {Assen, Marcel A. L. M. van and Aert, Robbie C. M. van and Nuijten, {Michèle B.} and Wicherts, Jelte M.}, + year = {2014}, + journal = {PLOS ONE}, + pages = {e84896}, + volume = {9}, + number = {1}, + doi = {10.1371/journal.pone.0084896} +} + @article{baker2016, title = {1,500 scientists lift the lid on reproducibility}, author = {Baker, Monya}, @@ -155,6 +188,32 @@ @article{balke2012 url = {https://www.tandfonline.com/doi/abs/10.1080/01621459.1997.10474074} } +@techreport{beagrie2012, + title = {Economic Impact Evaluation of the Economic and Social Data Service}, + author = {Beagrie, N. and Houghton, J.}, + year = {2012} +} + +@techreport{beagrie2014, + title = {The Value and Impact of Data Sharing and Curation. A synthesis of three recent studies of UK research data centres}, + author = {Beagrie, Neil and Houghton, John}, + year = {2014} +} + +@techreport{beagrie2016, + title = {The Value and Impact of the European Bioinformatics Institute}, + author = {Beagrie, N. and Houghton, J.}, + year = {2016}, + url = {https://www.embl.org/documents/wp-content/uploads/2021/09/EMBL-EBI_Impact_report-2016-summary.pdf} +} + +@techreport{beagrie2021, + title = {Data-driven discovery: The value and impact of EMBL-EBI managed data resources}, + author = {Beagrie, Neil and Houghton, John}, + year = {2021} +} + + @book{bellis2009, title = {Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics}, author = {Bellis, Nicola De}, @@ -166,6 +225,7 @@ @book{bellis2009 langid = {en} } + @article{bornmann_scientific_2011, title = {Scientific peer review}, volume = {45}, @@ -240,7 +300,6 @@ @article{brembs2019 url = {http://dx.doi.org/10.1371/journal.pbio.3000117} } - @techreport{brown2016, title = {The GRIM test: A simple technique detects numerous anomalies in the reporting of results in psychology}, author = {Brown, Nicholas J. L. and Heathers, James A.}, @@ -252,7 +311,6 @@ @techreport{brown2016 langid = {en} } - @article{bryan2021, title = {The impact of open access mandates on invention}, author = {Bryan, Kevin A. and Ozcan, Yasin}, @@ -303,6 +361,7 @@ @article{carlin2023 note = {Publisher: PeerJ Inc.} } + @article{castelnovo2023, title = {A quasi-experimental design to assess the innovative impact of public procurement: An application to the Italian space industry}, author = {Castelnovo, Paolo and {Clò}, Stefano and Florio, Massimo}, @@ -315,12 +374,12 @@ @article{castelnovo2023 note = {Publisher: Elsevier} } + @unpublished{catalano_forthcoming, author = {Catalano, G. and Florio, M. and Articolo, R. and Consiglio, G. and Eggleton, D.}, note = {forthcoming}, title = {Diamond contribution to the development of an innovative vaccine for the Foot-and-Mouth Disease: a veterinary pathology of global significance, case study report} } - @article{catalano2021, title = {From scientific experiments to innovation: Impact pathways of a Synchrotron Light Facility}, author = {Catalano, Gelsomina and {López}, {Gaston García} and {Sánchez}, Alejandro and Vignetti, Silvia}, @@ -350,6 +409,30 @@ @inproceedings{cazacu2023 address = {New York, NY, USA} } +@inbook{chataway2018, + title = {How Will Open Science Impact on University/Industry Collaborations?}, + author = {Chataway, Joanna and Parks, Sarah and Smith, Elta}, + editor = {Meissner, Dirk and Erdil, Erkan and Chataway, Joanna}, + year = {2018}, + publisher = {Springer International Publishing}, + pages = {265--282}, + series = {Science, Technology and Innovation Studies}, + url = {https://doi.org/10.1007/978-3-319-62649-9_12}, + note = {DOI: 10.1007/978-3-319-62649-9{\_}12}, + address = {Cham} +} + +@inproceedings{chen2017, + title = {2017 IEEE International Symposium on Multimedia (ISM)}, + author = {Chen, Sheng-Chih and Chen, Yi-Cheng and Chen, Wei-Lin}, + year = {2017}, + publisher = {IEEE}, + pages = {469--474}, + doi = {10.1109/ISM.2017.93}, + url = {http://ieeexplore.ieee.org/document/8241659/}, + address = {Taichung} +} + @inproceedings{cinelli_sensitivity_2019, title = {Sensitivity {Analysis} of {Linear} {Structural} {Causal} {Models}}, url = {https://proceedings.mlr.press/v97/cinelli19a.html}, @@ -381,7 +464,6 @@ @misc{codeof langid = {en} } - @article{cohen2002, title = {Links and Impacts: The Influence of Public Research on Industrial R&D}, author = {Cohen, Wesley M. and Nelson, Richard R. and Walsh, John P.}, @@ -397,7 +479,6 @@ @article{cohen2002 langid = {en} } - @article{colavizza2020, title = {The citation advantage of linking publications to research data}, author = {Colavizza, Giovanni and Hrynaszkiewicz, Iain and Staden, Isla and Whitaker, Kirstie and McGillivray, Barbara}, @@ -413,6 +494,7 @@ @article{colavizza2020 note = {Publisher: Public Library of Science}, langid = {en} } + @article{cole_chance_1981, title = {Chance and consensus in peer review}, volume = {214}, @@ -495,6 +577,18 @@ @misc{datasci url = {https://edison-project.eu/data-science-competence-framework-cf-ds/} } +@article{davies2016, + title = {Researching the emerging impacts of open data: revisiting the ODDC conceptual framework}, + author = {Davies, Tim and Perini, Fernando}, + year = {2016}, + month = {06}, + date = {2016-06-21}, + journal = {The Journal of Community Informatics}, + volume = {12}, + number = {2}, + doi = {10.15353/joci.v12i2.3246} +} + @article{davis_open_2008, title = {Open access publishing, article downloads, and citations: {Randomised} controlled trial}, volume = {337}, @@ -713,6 +807,25 @@ @article{estimati2015 langid = {en} } +@book{EU-DGRI2018, + title = {Cost-benefit analysis for FAIR research data: cost of not having FAIR research data.}, + author = {{European Commission. Directorate General for Research and Innovation.} and {PwC EU Services.}}, + year = {2018}, + publisher = {Publications Office}, + doi = {10.2777/02999}, + url = {https://data.europa.eu/doi/10.2777/02999} +} + +@book{EU2020, + title = {The economic impact of open data: opportunities for value creation in Europe}, + author = {Publications Office of the European Union, and Huyer, Esther and Knippenberg, Laura van}, + year = {2020}, + publisher = {Publications Office of the European Union}, + doi = {10.2830/63132}, + url = {https://data.europa.eu/doi/10.2830/63132}, + langid = {eng} +} + @article{fang2020, title = {An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics}, author = {Fang, Zhichao and Costas, Rodrigo and Tian, Wencan and Wang, Xianwen and Wouters, Paul}, @@ -1028,6 +1141,7 @@ @article{grimme langid = {en} } + @article{group_fair_2020, title = {{FAIR} {Data} {Maturity} {Model}. {Specification} and {Guidelines}}, doi = {10.15497/rda00050}, @@ -1036,6 +1150,7 @@ @article{group_fair_2020 year = {2020} } + @techreport{gruenpeter2021, title = {Defining Research Software: a controversial discussion}, author = {Gruenpeter, Morane and Katz, Daniel S. and Lamprecht, Anna-Lena and Honeyman, Tom and Garijo, Daniel and Struck, Alexander and Niehues, Anna and Martinez, Paula Andrea and Castro, Leyla Jael and Rabemanantsoa, Tovo and Chue Hong, Neil P. and Martinez-Ortiz, Carlos and Sesink, Laurents and Liffers, Matthias and Fouilloux, Anne Claire and Erdmann, Chris and Peroni, Silvio and Martinez Lavanchy, Paula and Todorov, Ilian and Sinha, Manodeep}, @@ -1055,6 +1170,7 @@ @book{guideto2015 langid = {en} } + @article{guyan2022, title = {Queer data}, author = {Guyan, Kevin}, @@ -1077,6 +1193,17 @@ @article{hagen2008 note = {Publisher: Public Library of Science San Francisco, USA} } +@inproceedings{harding2017, + title = {Expanding Perspectives on Open Science: Communities, Cultures and Diversity in Concepts and Practices}, + booktitle = {Proceedings of the 21st International Conference on Electronic Publishing}, + editor = {Chan, Leslie and Loizides, Fernando}, + author = {Harding, Rachel J.}, + year = {2017}, + date = {2017}, + pages = {1--5}, + address = {Limassol, Cyprus} +} + @article{hardwicke2018, title = {Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal {\emph{Cognition}}}, author = {Hardwicke, Tom E. and Mathur, Maya B. and MacDonald, Kyle and Nilsonne, Gustav and Banks, George C. and Kidwell, Mallory C. and Hofelich Mohr, Alicia and Clayton, Elizabeth and Yoon, Erica J. and Henry Tessler, Michael and Lenne, Richie L. and Altman, Sara and Long, Bria and Frank, Michael C.}, @@ -1119,6 +1246,7 @@ @article{healy2017 url = {https://doi.org/10.1177/0735275117709046} } + @techreport{heathers2018, title = {Recovering data from summary statistics: Sample Parameter Reconstruction via Iterative TEchniques (SPRITE)}, author = {Heathers, James A. and Anaya, Jordan and Zee, Tim van der and Brown, Nicholas J. L.}, @@ -1131,6 +1259,7 @@ @techreport{heathers2018 langid = {en} } + @book{heathers2018a, title = {SPRITE}, author = {Heathers, James A. and Brown, Nicholas J. L. and Zee, Tim van der and Anaya, Jordan}, @@ -1141,6 +1270,21 @@ @book{heathers2018a note = {Publisher: OSF} } +@inproceedings{herala2016, + title = {2016 SAI Computing Conference (SAI)}, + author = {Herala, Antti and Vanhala, Erno and Porras, Jari and Krri, Timo}, + year = {2016}, + pages = {715--724}, + doi = {10.1109/SAI.2016.7556060} +} + +@techreport{houghton2011, + title = {Access to research and technical information in Denmark}, + author = {Houghton, John and Swan, Alma and Brown, Sheridan}, + year = {2011}, + url = {https://eprints.soton.ac.uk/272603/} +} + @misc{howison_softcite_2023, title = {Softcite {Dataset} {Version} 2}, doi = {10.5281/zenodo.7995565}, @@ -1162,7 +1306,6 @@ @misc{hunermund_causal_2023 year = {2023} } - @inproceedings{hunter2015, title = {Formal Acknowledgement of Citizen Scientists{\textquoteright} Contributions via Dynamic Data Citations}, author = {Hunter, Jane and Hsu, Chih-Hsiang}, @@ -1176,7 +1319,6 @@ @inproceedings{hunter2015 langid = {en} } - @book{huntington-klein_effect_2021, title = {The {Effect}: {An} {Introduction} to {Research} {Design} and {Causality}}, isbn = {978-1-00-050914-4}, @@ -1198,7 +1340,6 @@ @book{huyer2020 langid = {eng} } - @article{istrate, title = {A large dataset of software mentions in the biomedical literature}, author = {Istrate, Ana-Maria and Li, Donghui and Taraborelli, Dario and Torkar, Michaela and Veytsman, Boris and Williams, Ivana}, @@ -1255,7 +1396,6 @@ @article{janssens langid = {en} } - @article{johnston2017, title = {Contemporary Guidance for Stated Preference Studies}, author = {Johnston, Robert J. and Boyle, Kevin J. and Adamowicz, {Wiktor (Vic)} and Bennett, Jeff and Brouwer, Roy and Cameron, Trudy Ann and Hanemann, W. Michael and Hanley, Nick and Ryan, Mandy and Scarpa, Riccardo and Tourangeau, Roger and Vossler, Christian A.}, @@ -1271,7 +1411,6 @@ @article{johnston2017 note = {Publisher: The University of Chicago Press} } - @book{jung2023, title = {scrutiny: Error Detection in Science}, author = {Jung, Lukas and Allard, {Aurélien}}, @@ -1281,6 +1420,15 @@ @book{jung2023 url = {https://cran.r-project.org/web/packages/scrutiny/index.html} } +@article{karasz2024, + title = {PathOS - D1.3 Key Impact Pathways for the Open Science Framework}, + author = {Karasz, Istvan and Stoy, Lennart and Seminaroti, Elisa and Grapengiesser, Izabella Martins}, + year = {2024}, + doi = {10.5281/ZENODO.11108567}, + url = {https://zenodo.org/doi/10.5281/zenodo.11108567}, + langid = {en} +} + @article{keller2014, title = {Re-use of public sector information in cultural heritage institutions}, author = {Keller, Paul and Margoni, Thomas and Rybicka, Katarzyna and Tarkowski, Alek}, @@ -1541,6 +1689,14 @@ @article{Levontin2022 pages = {25} } +@inproceedings{lindman2014, + title = {2014 47th Hawaii International Conference on System Sciences}, + author = {Lindman, Juho and Kinnari, Tomi and Rossi, Matti}, + year = {2014}, + pages = {739--748}, + doi = {10.1109/HICSS.2014.99} +} + @article{liu2023, title = {Data, measurement and empirical methods in the science of science}, author = {Liu, Lu and Jones, Benjamin F. and Uzzi, Brian and Wang, Dashun}, @@ -1946,6 +2102,7 @@ @misc{oneill2022_data url = {https://doi.org/10.5281/zenodo.7431678} } + @techreport{oneill2022a, title = {Monitoring Framework for National Contributions to EOSC}, author = {{O'Neill}, Gareth}, @@ -2004,6 +2161,7 @@ @article{opitz2017 doi = {10.1080/13803611.2017.1338586} } + @article{osborne2023, title = {Science education in an age of misinformation}, author = {Osborne, Jonathan and Pimentel, Daniel}, @@ -2015,6 +2173,7 @@ @article{osborne2023 doi = {10.1002/sce.21790} } + @article{ozolinciute2022, title = {Guidelines for research ethics and research integrity in citizen science}, author = {{Ozolin{\v{c}}i{\={u}}t{\.{e}}}, {Egl{\.{e}}} and {Bülow}, William and Bjelobaba, Sonja and {Gai{\v{z}}auskait{\.{e}}}, Inga and {Krásni{\v{c}}an}, Veronika and {Dlabolová}, Dita and {Umbrasait{\.{e}}}, Julija}, @@ -2037,6 +2196,14 @@ @article{page2021 note = {Publisher: British Medical Journal Publishing Group} } +@techreport{parsons_benefits_2011, + address = {Horsham, United Kingdom}, + title = {Benefits to the {Private} {Sector} of {Open} {Access} to {Higher} {Education} and {Scholarly} {Research}}, + institution = {HOST Policy Research}, + author = {Parsons, David and Willis, Dick and Holland, Jane}, + year = {2011} +} + @article{pasquetto2017, title = {On the Reuse of Scientific Data}, author = {Pasquetto, Irene V. and Randles, Bernadette M. and Borgman, Christine L.}, @@ -2161,7 +2328,6 @@ @article{radicchi2008 url = {http://www.pnas.org/cgi/content/abstract/105/45/17268} } - @article{rafols_monitoring_2024, title = {Monitoring {Open} {Science} as transformative change: {Towards} a systemic framework}, shorttitle = {Monitoring {Open} {Science} as transformative change}, @@ -2214,7 +2380,6 @@ @inbook{roberts2007 url = {https://api.taylorfrancis.com/content/chapters/edit/download?identifierName=doi&identifierValue=10.4324/9780203824696-29&type=chapterpdf} } - @inbook{roberts2013, title = {Scientific literacy/science literacy}, author = {Roberts, Douglas A.}, @@ -2225,7 +2390,6 @@ @inbook{roberts2013 url = {https://api.taylorfrancis.com/content/chapters/edit/download?identifierName=doi&identifierValue=10.4324/9780203824696-29&type=chapterpdf} } - @inbook{roberts2013a, title = {Scientific literacy/science literacy}, author = {Roberts, Douglas A.}, @@ -2320,13 +2484,13 @@ @article{ross-hellauer2022 } @techreport{ruiter2023, - title = {Automatically Finding and Categorizing Replication Studies}, - author = {Ruiter, Bob de}, - year = {2023}, - month = {11}, - publisher = {arXiv}, - url = {http://arxiv.org/abs/2311.15055}, - doi = {10.48550/arXiv.2311.15055} + title = {Automatically Finding and Categorizing Replication Studies}, + author = {Ruiter, Bob de}, + year = {2023}, + month = {11}, + publisher = {arXiv}, + url = {http://arxiv.org/abs/2311.15055}, + doi = {10.48550/arXiv.2311.15055} } @article{schmidt2009, @@ -2385,6 +2549,14 @@ @book{smaldino2023 address = {Princeton} } +@techreport{sparceurope2019, + title = {Using open and FAIR data to increase research efficiency}, + author = {SPARC Europe}, + year = {2019}, + date = {2019}, + url = {https://sparceurope.org/wp-content/uploads/dlm_uploads/2019/04/SPARC-Europe_Brief_ODEfficiency.pdf} +} + @article{squicciarini2021, title = {Demand for AI skills in jobs: Evidence from online job postings}, author = {Squicciarini, Mariagrazia and Nachtigall, Heike}, @@ -2450,6 +2622,7 @@ @techreport{sveinsdottir2021 note = {DOI: 10.5281/zenodo.4725817} } + @article{szomszor2022, title = {Overton: A bibliometric database of policy document citations}, author = {Szomszor, Martin and Adie, Euan}, @@ -2464,6 +2637,7 @@ @article{szomszor2022 url = {https://doi.org/10.1162/qss_a_00204} } + @book{tashakkori2021, title = {Foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences}, author = {Tashakkori, Abbas and Johnson, R. Burke and Teddlie, Charles}, @@ -2474,6 +2648,7 @@ @book{tashakkori2021 address = {Los Angeles London New Delhi Singapore Washington DC Melbourne} } + @article{tattersall2018, title = {What Can Altmetric.com Tell Us About Policy Citations of Research? An Analysis of Altmetric.com Data for Research Articles from the University of Sheffield}, author = {Tattersall, Andy and Carroll, Christopher}, @@ -2488,6 +2663,7 @@ @article{tattersall2018 langid = {English} } + @article{tennant2016, title = {The academic, economic and societal impacts of Open Access: an evidence-based review}, author = {Tennant, Jonathan P. and Waldner, {François} and Jacques, Damien C. and Masuzzo, Paola and Collister, Lauren B. and Hartgerink, Chris H. J.}, @@ -2498,6 +2674,7 @@ @article{tennant2016 url = {https://f1000research.com/articles/5-632} } + @article{tiokhin_shifting_2023, title = {Shifting the {Level} of {Selection} in {Science}}, issn = {1745-6916}, @@ -2512,6 +2689,7 @@ @article{tiokhin_shifting_2023 pages = {17456916231182568} } + @article{tomkins_reviewer_2017, title = {Reviewer bias in single- versus double-blind peer review}, volume = {114}, @@ -2526,6 +2704,7 @@ @article{tomkins_reviewer_2017 pages = {12708--12713} } + @misc{traag_causal_2022, title = {Causal foundations of bias, disparity and fairness}, url = {http://arxiv.org/abs/2207.13665}, @@ -2537,6 +2716,8 @@ @misc{traag_causal_2022 year = {2022} } + + @misc{traag_metrics_2023, title = {Metrics and peer review agreement at the institutional level}, doi = {10.48550/arXiv.2006.14830}, @@ -2546,6 +2727,7 @@ @misc{traag_metrics_2023 year = {2023} } + @article{traag2021, title = {Inferring the causal effect of journals on citations}, author = {Traag, V. A.}, @@ -2561,6 +2743,16 @@ @article{traag2021 langid = {en} } + + +@techreport{tripp2011, + title = {The Economic Impacts of Human Genome Project}, + author = {Tripp, Simon and Grueber, Martin}, + year = {2011}, + date = {2011} +} + + @article{valladares2021, title = {Scientific Literacy and Social Transformation}, author = {Valladares, Liliana}, @@ -2586,6 +2778,19 @@ @article{van_den_besselaar_perverse_2017 pages = {905--918} } + +@article{vanvlijmen2020, + title = {The Need of Industry to Go FAIR}, + author = {van Vlijmen, Herman and Mons, Albert and Waalkens, Arne and Franke, Wouter and Baak, Arie and Ruiter, Gerbrand and Kirkpatrick, Christine and da Silva Santos, Luiz Olavo Bonino and Meerman, Bert and Jellema, Renger and Arts, Derk and Kersloot, Martijn and Knijnenburg, Sebastiaan and Lusher, Scott and Verbeeck, Rudi and Neefs, Jean-Marc}, + year = {2020}, + date = {2020}, + journal = {Data Intelligence}, + pages = {276--284}, + volume = {2}, + number = {1-2}, + doi = {10.1162/dint_a_00050} +} + @book{venturini2021, title = {Controversy Mapping: A Field Guide}, author = {Venturini, Tommaso and Munk, Anders Kristian}, @@ -2683,7 +2888,6 @@ @article{waltman2019 doi = {10.1007/978-3-030-02511-3_11} } - @article{wang2020, title = {Consistency and validity of interdisciplinarity measures}, author = {Wang, Qi and Schneider, Jesper Wiborg}, @@ -2697,7 +2901,6 @@ @article{wang2020 note = {Publisher: MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info {\ldots}} } - @article{way2019, title = {Productivity, prominence, and the effects of academic environment}, author = {Way, Samuel F. and Morgan, Allison C. and Larremore, Daniel B. and Clauset, Aaron}, @@ -2709,6 +2912,17 @@ @article{way2019 doi = {10.1073/pnas.1817431116} } +@article{wehn2021, + title = {Impact assessment of citizen science: state of the art and guiding principles for a consolidated approach}, + author = {Wehn, Uta and Gharesifard, Mohammad and Ceccaroni, Luigi and Joyce, Hannah and Ajates, Raquel and Woods, Sasha and Bilbao, Ane and Parkinson, Stephen and Gold, Margaret and Wheatland, Jonathan}, + year = {2021}, + date = {2021}, + journal = {Sustainability Science}, + pages = {1683--1699}, + volume = {16}, + number = {5}, + doi = {10.1007/s11625-021-00959-2} +} @article{westreich2013, title = {The Table 2 Fallacy: Presenting and Interpreting Confounder and Modifier Coefficients}, @@ -2724,14 +2938,12 @@ @article{westreich2013 url = {https://doi.org/10.1093/aje/kws412} } - @misc{whatper, title = {What personal data is considered sensitive? - European Commission}, url = {https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/sensitive-data/what-personal-data-considered-sensitive_en}, langid = {en} } - @article{wilkinson2016, title = {The FAIR Guiding Principles for scientific data management and stewardship}, author = {Wilkinson, Mark D. and Dumontier, Michel and Aalbersberg, IJsbrand Jan and Appleton, Gabrielle and Axton, Myles and Baak, Arie and Blomberg, Niklas and Boiten, Jan-Willem and da Silva Santos, Luiz Bonino and Bourne, Philip E. and Bouwman, Jildau and Brookes, Anthony J. and Clark, Tim and Crosas, {Mercè} and Dillo, Ingrid and Dumon, Olivier and Edmunds, Scott and Evelo, Chris T. and Finkers, Richard and Gonzalez-Beltran, Alejandra and Gray, Alasdair J. G. and Groth, Paul and Goble, Carole and Grethe, Jeffrey S. and Heringa, Jaap and {{\textquoteright}t Hoen}, Peter A. C. and Hooft, Rob and Kuhn, Tobias and Kok, Ruben and Kok, Joost and Lusher, Scott J. and Martone, Maryann E. and Mons, Albert and Packer, Abel L. and Persson, Bengt and Rocca-Serra, Philippe and Roos, Marco and van Schaik, Rene and Sansone, Susanna-Assunta and Schultes, Erik and Sengstag, Thierry and Slater, Ted and Strawn, George and Swertz, Morris A. and Thompson, Mark and van der Lei, Johan and van Mulligen, Erik and Velterop, Jan and Waagmeester, Andra and Wittenburg, Peter and Wolstencroft, Katherine and Zhao, Jun and Mons, Barend}, @@ -2748,7 +2960,6 @@ @article{wilkinson2016 langid = {en} } - @article{wilkinson2016a, title = {The FAIR Guiding Principles for scientific data management and stewardship}, author = {Wilkinson, Mark D. and Dumontier, Michel and Aalbersberg, IJsbrand Jan and Appleton, Gabrielle and Axton, Myles and Baak, Arie and Blomberg, Niklas and Boiten, Jan-Willem and da Silva Santos, Luiz Bonino and Bourne, Philip E. and Bouwman, Jildau and Brookes, Anthony J. and Clark, Tim and Crosas, {Mercè} and Dillo, Ingrid and Dumon, Olivier and Edmunds, Scott and Evelo, Chris T. and Finkers, Richard and Gonzalez-Beltran, Alejandra and Gray, Alasdair J. G. and Groth, Paul and Goble, Carole and Grethe, Jeffrey S. and Heringa, Jaap and {{\textquoteright}t Hoen}, Peter A. C. and Hooft, Rob and Kuhn, Tobias and Kok, Ruben and Kok, Joost and Lusher, Scott J. and Martone, Maryann E. and Mons, Albert and Packer, Abel L. and Persson, Bengt and Rocca-Serra, Philippe and Roos, Marco and van Schaik, Rene and Sansone, Susanna-Assunta and Schultes, Erik and Sengstag, Thierry and Slater, Ted and Strawn, George and Swertz, Morris A. and Thompson, Mark and van der Lei, Johan and van Mulligen, Erik and Velterop, Jan and Waagmeester, Andra and Wittenburg, Peter and Wolstencroft, Katherine and Zhao, Jun and Mons, Barend}, @@ -2765,7 +2976,6 @@ @article{wilkinson2016a langid = {en} } - @article{wilner, title = {Complete recovery of values in Diophantine systems (CORVIDS)}, author = {Wilner, Sean and Wood, Katherine and Simons, Daniel J.}, @@ -2773,8 +2983,6 @@ @article{wilner langid = {en-us} } - - @techreport{wilsdon_metric_2015, title = {Metric {Tide}: {Report} of the {Independent} {Review} of the {Role} of {Metrics} in {Research} {Assessment} and {Management}}, institution = {Higher Education Funding Council for England}, @@ -2784,7 +2992,6 @@ @techreport{wilsdon_metric_2015 pages = {163} } - @book{wood2021, title = {CORVIDS}, author = {Wood, Katherine}, @@ -2795,8 +3002,6 @@ @book{wood2021 note = {original-date: 2018-01-29T16:15:29Z} } - - @article{woods2022, title = {Incentivising research data sharing: a scoping review}, author = {Woods, Helen Buckley and Pinfield, Stephen}, @@ -2810,6 +3015,16 @@ @article{woods2022 url = {https://wellcomeopenresearch.org/articles/6-355/v2} } +@article{woodward2003, + title = {Critical Notice: Causality by Judea Pearl}, + author = {Woodward, James}, + year = {2003}, + journal = {Economics & Philosophy}, + pages = {321--340}, + volume = {19}, + number = {2}, + doi = {10.1017/S0266267103001184} +} @article{wuchty2007, title = {The Increasing Dominance of Teams in Production of Knowledge}, @@ -2836,7 +3051,6 @@ @article{yarkoni2019 url = {https://psyarxiv.com/jqw35/} } - @article{zahedi2017, title = {Mendeley readership as a filtering tool to identify highly cited publications}, author = {Zahedi, Zohreh and Costas, Rodrigo and Wouters, Paul}, @@ -2864,3 +3078,15 @@ @article{zahedi2020 url = {https://ll-j-sar.ubiquityjournal.website/articles/10.29024/sar.20}, langid = {canadian} } + +@inproceedings{zeleti2014, + title = {Capability Matrix for Open Data}, + author = {Zeleti, Fatemeh Ahmadi and Ojo, Adegboyega}, + editor = {Camarinha-Matos, Luis M. and Afsarmanesh, Hamideh}, + year = {2014}, + publisher = {Springer}, + pages = {498--509}, + doi = {10.1007/978-3-662-44745-1_50}, + address = {Berlin, Heidelberg} +} + diff --git a/sections/0_causality/open_data_cost_savings.qmd b/sections/0_causality/open_data_cost_savings.qmd index 0ff5fa4..59073ca 100644 --- a/sections/0_causality/open_data_cost_savings.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -13,17 +13,17 @@ History (please fill out in reverse chronological order, latest revision on top) ## Literature background -Measuring the economic impact of open science and open data has proven to be challenging. Many theoretical studies highlight the benefits of making research results public, with strong support for Open Science from economic research on technological change (Chataway et al., 2017; Yozwiak et al., 2015; Mazzucato, 2011). However, only few studies have attempted to measure the impacts of open science compared to closed science, and more robust evidence on how Open Science drives innovation and economic outcomes is needed to strengthen support and counter emerging criticisms (Karasz et al., 2024; Ali-Khan et al., 2018). The existing literature mainly concentrates on specific sectors, particularly health, medicine, and biosciences, which receive more attention due to early regulation by funders and significant interest in clinical trial outcomes. Another important stream of literature is focused on highlighting the economic value of Open Science through personal industry experiences, though lacking precise quantitative evidence, with contributions from McManamay and Utz (2014) on fisheries, Harding (2017) on medicine, Chan (2015) on the transition to an Open Science model, and Chen et al. (2017) on the role of open data in AI and machine learning applications. Although directly linking economic outcomes to open data initiatives can be challenging, with authors combining theoretical arguments and the limited quantitative evidence available at the time of their publication (Ali and Dahlhaus, 2022; Arzberger, 2004; Tennant et al., 2016; Fell, 2019; Wehn et al., 2021), open access to findings and data is considered to lead to significant savings in access costs. By removing paywalls and subscription fees, open data allows researchers and businesses to access valuable information without incurring additional costs. A major economic benefit of lowering the cost of knowledge is the availability of an extra budget that can be reallocated for other purposes (Tennant et al., 2016). +Measuring the economic impact of open science and open data has proven to be challenging. Many theoretical studies highlight the benefits of making research results public, with strong support for Open Science from economic research on technological change [\@chataway2018](Yozwiak%20et%20al.,%202015;%20Mazzucato,%202011). However, only few studies have attempted to measure the impacts of open science compared to closed science, and more robust evidence on how Open Science drives innovation and economic outcomes is needed to strengthen support and counter emerging criticisms [@karasz2024] (Ali-Khan et al., 2018). The existing literature mainly concentrates on specific sectors, particularly health, medicine, and biosciences, which receive more attention due to early regulation by funders and significant interest in clinical trial outcomes. Another important stream of literature is focused on highlighting the economic value of Open Science through personal industry experiences, though lacking precise quantitative evidence, with contributions from McManamay and Utz (2014) on fisheries, @harding2017 on medicine, Chan (2015) on the transition to an Open Science model, and @chen2017 on the role of open data in AI and machine learning applications. Although directly linking economic outcomes to open data initiatives can be challenging, with authors combining theoretical arguments and the limited quantitative evidence available at the time of their publication [@ali2022,@tennant2016,@fell2019,@wehn2021] (Arzberger, 2004), open access to findings and data is considered to lead to significant savings in access costs. By removing paywalls and subscription fees, open data allows researchers and businesses to access valuable information without incurring additional costs. A major economic benefit of lowering the cost of knowledge is the availability of an extra budget that can be reallocated for other purposes [@tennant2016]. -By eliminating barriers to data access, organisations can reduce the time spent on data collection and focus on core activities. Fell (2019) notes that open science reduces the time associated with accessing new knowledge, directly contributing to enhanced research quality and productivity increases. Although the specific time savings associated with open access were not directly tested, Parsons et al. (2011) conducted an interview-based study that supports this potential benefit. Beagrie and Houghton (2014) demonstrate that data sharing and curation significantly enhance research efficiency, with labour cost savings ranging from two to over twenty times the operational costs of the data centres. While there are indirect costs associated with preparing data for sharing, such as time and expenses incurred by depositors, the benefits realised by users through improved efficiency and their willingness to pay for access far outweigh these costs (Beagrie and Houghton, 2012). Additionally, findings from Beagrie and Houghton (2016;2021) provide compelling evidence of the timesaving and efficiency gains associated with using European Bioinformatics Institute services (EMBL-EBI). +By eliminating barriers to data access, organisations can reduce the time spent on data collection and focus on core activities. @fell2019 notes that open science reduces the time associated with accessing new knowledge, directly contributing to enhanced research quality and productivity increases. Although the specific time savings associated with open access were not directly tested, @parsons_benefits_2011 conducted an interview-based study that supports this potential benefit. [@beagrie2014] demonstrate that data sharing and curation significantly enhance research efficiency, with labour cost savings ranging from two to over twenty times the operational costs of the data centres. While there are indirect costs associated with preparing data for sharing, such as time and expenses incurred by depositors, the benefits realised by users through improved efficiency and their willingness to pay for access far outweigh these costs [@beagrie2012]. Additionally, findings from [@beagrie2016,@beagrie2021] provide compelling evidence of the timesaving and efficiency gains associated with using European Bioinformatics Institute services (EMBL-EBI). -Open data minimises redundant data collection and mitigates the "file drawer effect," where valuable findings remain unpublished and inaccessible, ultimately impeding research effectiveness (van Assen et al., 2014). According to SPARC Europe (2019), making research data openly available can save up to 9% of a project's costs by preventing unnecessary data collection and facilitating the efficient reuse of existing data. Additionally, Houghton, Swan, and Brown (2011) estimate that access barriers to academic research in Denmark cost DKK 540 million annually. This figure is based on the average time spent (51-63 minutes) attempting to access research articles and the study highlights that delays in accessing academic research can prolong product and process development by an average of 2.2 years, resulting in significant financial losses for firms. +Open data minimises redundant data collection and mitigates the "file drawer effect," where valuable findings remain unpublished and inaccessible, ultimately impeding research effectiveness [@assen2014]. According to [@sparceurope2019], making research data openly available can save up to 9% of a project's costs by preventing unnecessary data collection and facilitating the efficient reuse of existing data. Additionally, @houghton2011 estimate that access barriers to academic research in Denmark cost DKK 540 million annually. This figure is based on the average time spent (51-63 minutes) attempting to access research articles and the study highlights that delays in accessing academic research can prolong product and process development by an average of 2.2 years, resulting in significant financial losses for firms. -The broader implications of open data extend beyond mere cost and time savings. Scientific literature is widely recognised as an important source of strategic knowledge, facilitating the exploration of new ideas in industrial research and innovation, particularly for small and medium enterprises that may struggle to obtain data independently (European Union, 2020). However, inefficiencies in traditional publishing models, such as delays and biases in data dissemination, can negatively impact private research productivity, as discussed by Harding (2017). Open data benefits various sectors, including agriculture, the environment, forensics, and industrial biotechnology, by providing access to information that helps researchers understand their fields and build on existing work (Fell, 2019; Tennant et al., 2016; Tripp et al.,2011; Yozwiak et al., 2015; Arshad et al., 2016). +The broader implications of open data extend beyond mere cost and time savings. Scientific literature is widely recognised as an important source of strategic knowledge, facilitating the exploration of new ideas in industrial research and innovation, particularly for small and medium enterprises that may struggle to obtain data independently [@EU2020]. However, inefficiencies in traditional publishing models, such as delays and biases in data dissemination, can negatively impact private research productivity, as discussed by @harding2017. Open data benefits various sectors, including agriculture, the environment, forensics, and industrial biotechnology, by providing access to information that helps researchers understand their fields and build on existing work [@fell2019,@tennant2016,@tripp2011,@arshad2016] (Yozwiak et al., 2015). -To fully harness the potential of open data, it is important to develop the necessary skills and capacities to manage it effectively. Zeleti and Ojo (2014) emphasise the need to streamline data generation processes to produce insights that inform and shape business strategies. Given that open data initiatives contribute to greater transparency and accountability, businesses that leverage open data achieve cost savings by co-creating and integrating data from multiple sources to enhance their services (Davies & Perini, 2016; Lindman et al., 2014). Fell (2019) suggests that adopting an open approach fosters connections and encourages collaborations that might not occur or would take longer in a closed environment. This is exemplified by the work of Van Vlijmen et al. (2020), who show how the integration of diverse types of open data, specifically scientific, clinical, and experimental public evidence, can be achieved using advanced AI platforms like Euretos. By combining these various data sources, the platform enhances the depth of information available for analysis, enabling more accurate predictions regarding drug efficacy, as demonstrated by a machine learning model that improved prediction accuracy by 12 percentage points over previous state-of-the-art. +To fully harness the potential of open data, it is important to develop the necessary skills and capacities to manage it effectively. @zeleti2014 emphasise the need to streamline data generation processes to produce insights that inform and shape business strategies. Given that open data initiatives contribute to greater transparency and accountability, businesses that leverage open data achieve cost savings by co-creating and integrating data from multiple sources to enhance their services [@davies2016,@lindman2014]. @fell2019 suggests that adopting an open approach fosters connections and encourages collaborations that might not occur or would take longer in a closed environment. This is exemplified by the work of [@vanvlijmen2020], who show how the integration of diverse types of open data, specifically scientific, clinical, and experimental public evidence, can be achieved using advanced AI platforms like Euretos. By combining these various data sources, the platform enhances the depth of information available for analysis, enabling more accurate predictions regarding drug efficacy, as demonstrated by a machine learning model that improved prediction accuracy by 12 percentage points over previous state-of-the-art. -Despite the theoretical benefits of open data, several limitations hinder a comprehensive assessment of its economic impact fully. Implementing open data practices requires significant investments in infrastructure, technology, and training, potentially offsetting some cost savings (van Vlijmen et al., 2020). Moreover, the European Commission study emphasises that the benefits of open data are contingent on the quality and standardisation of the data provided (European Commission and PwC EU Services, 2018). Finally, a major limitation is the scarcity of empirical evidence; few studies have attempted to measure the impacts of open science compared to closed science, making it challenging to generalise findings (Karasz et al., 2024). Herala et al. (2016) review the benefits and challenges of open data initiatives in the private sector, highlighting advantages like enhanced collaboration and innovation, but caution that these are often based on speculative assumptions rather than empirical evidence, emphasising the need for further research to inform best practices and mitigate risks associated with increased costs and data privacy concerns. +Despite the theoretical benefits of open data, several limitations hinder a comprehensive assessment of its economic impact fully. Implementing open data practices requires significant investments in infrastructure, technology, and training, potentially offsetting some cost savings [@vanvlijmen2020]. Moreover, the European Commission study emphasises that the benefits of open data are contingent on the quality and standardisation of the data provided [@EC-DGRI2018]. Finally, a major limitation is the scarcity of empirical evidence; few studies have attempted to measure the impacts of open science compared to closed science, making it challenging to generalise findings [@karasz2024]. @herala2016 review the benefits and challenges of open data initiatives in the private sector, highlighting advantages like enhanced collaboration and innovation, but caution that these are often based on speculative assumptions rather than empirical evidence, emphasising the need for further research to inform best practices and mitigate risks associated with increased costs and data privacy concerns. ## Directed Acyclic Graph (DAG) @@ -33,15 +33,15 @@ As discussed in the general introduction on causal inference, we use DAGs to rep ## The effect of Open data on Cost Saving -In this section, we apply the concepts presented in the section Causality in Science Studies to potential research questions. We present a specific perspective on causal inference through the lens of structural causal models (Pearl 2009). +In this section, we apply the concepts presented in the section Causality in Science Studies to potential research questions. We present a specific perspective on causal inference through the lens of structural causal models [@pearl2009]. -Suppose we are interested in assessing the *total causal effect* of *Open Data* on *Cost Saving*. According to our model (Figure 1), there are multiple pathways from Open Data to Cost Saving, some are causal, some are not. To estimate the causal effect of interest, we need to make sure that all causal paths are open, and all non-causal paths are closed. Within the DAG representation, two causal pathways can be identified: a *direct* pathway of *Open Data → Cost Saving,* representing the direct effect of Open Data on Cost Savings*,* and an *indirect* pathway *Open Data → Time Saving → Cost Saving*, where the effect is indirect and mediated by Time Saving. The *direct* effect captures the immediate benefits of providing free access to datasets, while the *indirect* effect, mediated by Time Savings, strengthens the relationship by triggering additional efficiencies that also lead to Cost Savings. +Suppose we are interested in assessing the *total causal effect* of *Open Data* on *Cost Saving*. According to our model ([@fig-model]), there are multiple pathways from Open Data to Cost Saving, some are causal, some are not. To estimate the causal effect of interest, we need to make sure that all causal paths are open, and all non-causal paths are closed. Within the DAG representation, two causal pathways can be identified: a *direct* pathway of *Open Data → Cost Saving,* representing the direct effect of Open Data on Cost Savings*,* and an *indirect* pathway *Open Data → Time Saving → Cost Saving*, where the effect is indirect and mediated by Time Saving. The *direct* effect captures the immediate benefits of providing free access to datasets, while the *indirect* effect, mediated by Time Savings, strengthens the relationship by triggering additional efficiencies that also lead to Cost Savings. To properly estimate the *total* causal effect of Open Data on Cost Saving, an empirical model should not control for Time saving. On the contrary, if the model conditions on Time Savings, even implicitly (e.g., by accounting for approaches and tools that optimise and speed up data access and processing), it closes the causal path and introduces biases into the estimation of the total effect ([@fig-mediator]). ![DAG illustrating the misleading effect of conditioning on the mediator variable, Time Saving. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey arrows indicate indirect association that may involve non-causal relationship.](figures/DAG_open_data_cost_savings-1.png){#fig-mediator} -As mentioned before, the proposed model accounts for additional variables such as *Data Availability, Quality and Standardisation*, *Users' Skills, Collaboration, and Technological Infrastructure*. These variables can act as *confounders* along different pathways illustrated in Figure 1. Examples of non-causal paths represented in the DAG are: +As mentioned before, the proposed model accounts for additional variables such as *Data Availability, Quality and Standardisation*, *Users' Skills, Collaboration, and Technological Infrastructure*. These variables can act as *confounders* along different pathways illustrated in [@fig-model]. Examples of non-causal paths represented in the DAG are: - *Open Data* *←* *Technological Infrastructure* → *Time Saving* → *Cost Saving* - *Open Data* ← *Data Availability, Quality and Standardisation* → *Time Saving* → *Cost Saving* @@ -53,19 +53,19 @@ There might be instances where a confounder is not observed because is not inclu Another case that makes not possible the identification of the causal effect is erroneously controlling for a collider (see the introduction for further details). In the pathway *Open Data* → *Innovation* ← *Collaboration*→ *Cost Saving,* the variable *Innovation* acts as a collider. Hence, this path is already closed, and the bias arises when controlling for innovation. -Empirically speaking, in a model where to estimate the causal effect of Open Data on Cost savings we condition on Innovation (and not for Collaboration), it is likely to get a downward estimation of the causal effect, since both Collaboration and Open Data have a positive impact on Innovation. This conditioning opens up the non-causal pathway, Open Data → Innovation ← Collaboration → Cost Saving, which connect *Open Data* and *Cost Saving* through *Collaboration*, creating a spurious association and distorting the true effect of *Open Data* on *Cost Saving*. This is an example of *bad controls (*Angrist and Pischke 2009), a concept explained in the general introduction. Only by ignoring the collider, meaning non-conditioning on it in empirical models, we can effectively isolate the causal effect. This non-causal path is open because *Innovation* is open (because it is a collider that is conditioned on), and because *Collaboration* is open (because it is a confounder that is not conditioned on) (see @fig-collider). +Empirically speaking, in a model where to estimate the causal effect of Open Data on Cost savings we condition on Innovation (and not for Collaboration), it is likely to get a downward estimation of the causal effect, since both Collaboration and Open Data have a positive impact on Innovation. This conditioning opens up the non-causal pathway, Open Data → Innovation ← Collaboration → Cost Saving, which connect *Open Data* and *Cost Saving* through *Collaboration*, creating a spurious association and distorting the true effect of *Open Data* on *Cost Saving*. This is an example of *bad controls* [@angrist2009], a concept explained in the general introduction. Only by ignoring the collider, meaning non-conditioning on it in empirical models, we can effectively isolate the causal effect. This non-causal path is open because *Innovation* is open (because it is a collider that is conditioned on), and because *Collaboration* is open (because it is a confounder that is not conditioned on) (see @fig-collider). ![DAG illustrating the misleading effect of conditioning on a collider variable, *Innovation,* and not conditioning on a confounder, *Collaboration*. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey arrows indicate indirect association that may involve non-causal relationship.](figures/DAG_open_data_cost_savings-2.png){#fig-collider} -In addition, *Collaboration* acts as a confounder on the non-causal path *Open data* ← *Collaboration* → *Cost saving*. To identify the causal effect, we hence need to close this non-causal path by conditioning on *Collaboration*. After controlling for Collaboration, whether *Innovation* is conditioned on is then irrelevant for the identification of the causal effect. When all non-causal paths are closed, the research design is said to meet the *backdoor* criterion, a formal requirement that ensures the design blocks all non-causal paths between the treatment (*Open Data*) and the outcome (*Cost Saving*), enabling us to identify the causal effect in question (Cunningham, 2021) ([@fig-adjustment]). +In addition, *Collaboration* acts as a confounder on the non-causal path *Open data* ← *Collaboration* → *Cost saving*. To identify the causal effect, we hence need to close this non-causal path by conditioning on *Collaboration*. After controlling for Collaboration, whether *Innovation* is conditioned on is then irrelevant for the identification of the causal effect. When all non-causal paths are closed, the research design is said to meet the *backdoor* criterion, a formal requirement that ensures the design blocks all non-causal paths between the treatment (*Open Data*) and the outcome (*Cost Saving*), enabling us to identify the causal effect in question [@cunningham_causal_2021] ([@fig-adjustment]). ![DAG illustrating the total effect *Open Data* on *Cost Saving*, conditioning on confounders and not on mediator and collider. Nodes that are controlled for have a thick outline. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship.](figures/DAG_open_data_cost_savings-3.png){#fig-adjustment} This example highlights key components of causal inference: controlling for confounders *(Data availability, quality and standardisation, User skills, Collaboration,* and *Technological Infrastructure*), not controlling for mediators *(Time saving),* and not controlling for colliders *(Innovation),* as shown in @fig-adjustment. Constructing an appropriate DAG is important when aiming to draw causal conclusions. Without making assumptions explicit via a DAG, it would be unclear which variables should be controlled for and which not. Omitting important variables weakens the study's ability to draw accurate conclusions about cause and effect. Moreover, adding complexity to a DAG does not always change the variables that need to be controlled for when identifying the causal effect. In some cases, such as when adding confounders between unrelated variables, the identification of the relationship between Open data and Cost savings remains unaffected. However, if a confounder is introduced between Open data and Cost savings directly, it becomes necessary to control for it. -Klebel and Traag (2023) emphasise the importance of carefully selecting variables when analysing causal relationships. They warn against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which they refer to as "causal salad" (McElreath, 2020). Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, Pearl proposes using the *do-operator^[\[1\]](#footnote-2){#footnote-ref-2}^* to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system (Pearl, 2009). +In the [causality introduction](causal_intro/article/intro-causality.qmd) we emphasise the importance of carefully selecting variables when analysing causal relationships. The introduction warns against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which @mcelreath2020 refers to as "causal salad". Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, @pearl2009 proposes using the *do-operator^[\[1\]](#footnote-2){#footnote-ref-2}^* to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system. -This approach is effective in predicting how probability distributions shift under controlled changes when the causal structure is known. However, this approach depends on having an established causal graph, limiting its use for exploring causality from scratch. Critics suggest an alternative approach that allows causal discovery through experimentation without prior assumptions about mechanisms (Woodward, 2003). This is especially useful in complex fields, such as social and biomedical sciences, where causal relationships are less understood. Considering this, Cunningham (2021) argues that sample selection problems have been recognised long before the introduction of DAGs, with early solutions like Heckman (1979), and emphasizes that an atheoretical approach to empiricism is inadequate. He asserts that causal inference requires a deep understanding of the behavioural processes behind the phenomenon being studied, and while DAGs are useful, they cannot replace the need for theoretical knowledge in creating credible identification strategies. Thus, causal inference is not solved by simply collecting more data, but by integrating theory with empirical analysis. +This approach is effective in predicting how probability distributions shift under controlled changes when the causal structure is known. However, this approach depends on having an established causal graph, limiting its use for exploring causality from scratch. Critics suggest an alternative approach that allows causal discovery through experimentation without prior assumptions about mechanisms [@woodward2003]. This is especially useful in complex fields, such as social and biomedical sciences, where causal relationships are less understood. Considering this, @cunningham_causal_2021 argues that sample selection problems have been recognised long before the introduction of DAGs, with early solutions like Heckman (1979), and emphasizes that an atheoretical approach to empiricism is inadequate. He asserts that causal inference requires a deep understanding of the behavioural processes behind the phenomenon being studied, and while DAGs are useful, they cannot replace the need for theoretical knowledge in creating credible identification strategies. Thus, causal inference is not solved by simply collecting more data, but by integrating theory with empirical analysis. ## Discussing empirical issues @@ -73,60 +73,12 @@ The presented model illustrates how open data might drive cost savings by focusi Other relevant factors could be included based on additional evidence but are currently excluded to maintain a coherent link with existing empirical findings and to preserve simplicity. While there is general agreement on the potential benefits of open data for cost savings, no study to date has estimated the total effect using a causal identification strategy such as the one presented here. Moreover, the specific causal pathways remain unclear, as little attention has been paid to the intermediate factors influencing the total effect. This lack of empirical evidence on these pathways limits the ability to draw strong conclusions from the model and underscores the need for more focused empirical research. -It is also important to acknowledge that causal pathways are not static and may evolve depending on the context or field of application. Cunningham (2021) highlights that causal inference requires a deeper understanding of the underlying processes governing the system being studied. It is not solely about the data but also about the theoretical and contextual knowledge of how behaviours, choices, or events interact to produce stable outcomes. Timing and the evolution of causality play a critical role, as interactions between variables change over time, leading to different outcomes in different contexts. This complexity, arising from reverse causality and cyclical interactions, represents a key limitation of this DAG. The relationships between variables are often more intricate, with the potential for bidirectional influences. For example, when organisations share open data, they enable collaborative efforts that can lead to innovative solutions, such as new products or services. Increased collaboration, in turn, may prompt organisations to adopt more open data practices, creating a bidirectional relationship. This innovation can then lead to cost savings by streamlining processes or reducing redundancies. Conversely, independent cost savings may provide the resources necessary for further investment in innovation, illustrating a cyclical relationship where each factor reinforces the other. This dynamic interplay complicates the one-way causal pathways represented in the DAG. Feedback loops and time-dependent changes are often critical in real-world scenarios, making the DAG an initial framework rather than a definitive model. Further empirical research is essential to refine and validate the relationships it proposes, capturing the complexities and nuances of the interactions between these variables. +It is also important to acknowledge that causal pathways are not static and may evolve depending on the context or field of application. @cunningham_causal_2021 highlights that causal inference requires a deeper understanding of the underlying processes governing the system being studied. It is not solely about the data but also about the theoretical and contextual knowledge of how behaviours, choices, or events interact to produce stable outcomes. Timing and the evolution of causality play a critical role, as interactions between variables change over time, leading to different outcomes in different contexts. This complexity, arising from reverse causality and cyclical interactions, represents a key limitation of this DAG. The relationships between variables are often more intricate, with the potential for bidirectional influences. For example, when organisations share open data, they enable collaborative efforts that can lead to innovative solutions, such as new products or services. Increased collaboration, in turn, may prompt organisations to adopt more open data practices, creating a bidirectional relationship. This innovation can then lead to cost savings by streamlining processes or reducing redundancies. Conversely, independent cost savings may provide the resources necessary for further investment in innovation, illustrating a cyclical relationship where each factor reinforces the other. This dynamic interplay complicates the one-way causal pathways represented in the DAG. Feedback loops and time-dependent changes are often critical in real-world scenarios, making the DAG an initial framework rather than a definitive model. Further empirical research is essential to refine and validate the relationships it proposes, capturing the complexities and nuances of the interactions between these variables. A targeted survey could be a viable approach to address the empirical challenges arising from the lack of data and counterfactual evidence in studying the economic impact of open data. This survey should be carefully designed to measure the direct effects of open data on cost savings while accounting for confounding factors. It should aim to answer specific research questions, such as the extent of open data sharing, the perceived efficiency gains from data use, and the direct cost reductions attributed to open data initiatives. The survey would need to collect both longitudinal and cross-sectional data from a diverse sample of organisations that share open data as well as those that do not. This would allow for comparisons of outcomes and the establishment of causal relationships. Key variables to be measured should include the extent of open data sharing, cost reductions, new products or services enabled or accelerated by open data, and changes in collaboration. Questions investigating the perceived causal pathways would also be essential. Incorporating temporal data would provide further insight, enabling a better understanding of causal direction and distinguishing between short-term and long-term effects. Access to such detailed data would enable the use of causal inference techniques, such as Propensity Score Matching (PSM) or the Difference-in-Differences (DID) estimator, to identify the true impact of open data on cost savings. By addressing current data gaps, this survey could provide the empirical evidence needed to validate and refine the DAG, enhancing our understanding of how open data drives economic outcomes like cost savings. For instance, in the PSM context, a rich dataset would support the development of a more comprehensive DAG, helping to identify variables to include in the matching process (e.g., data availability, technological infrastructure, and user skills) and those to exclude (e.g., time savings and innovation). This approach would strengthen the empirical basis for analysing the causal impact of open data. -## References - -Angrist, J. D., & Pischke, J.-S. (2009). *Mostly Harmless Econometrics: An Empiricist’s Companion*. Princeton University Press. - -Beagrie, N. and Houghton, J. (2014). The Value and Impact of Data Sharing and Curation. - -Beagrie, N. and Houghton, J. (2016). The Value and Impact of the European Bioinformatics Institute. . https://www.beagrie.com/static/resource/EBI-impact-report.pdf . - -Cunningham, S. (2021). *Causal Inference: The Mixtape.* Yale University Press. https://www.jstor.org/stable/j.ctv1c29t27. - -Davies, T. and Perini, F. (2016). Researching the emerging impacts of open data: revisiting the ODDC conceptual framework. The Journal of Community Informatics, 12(2). https://openjournals.uwaterloo.ca/index.php/JoCI/article/view/3246. - -European Commission and PwC EU Services. (2018). Cost-benefit analysis for FAIR research data: cost of not having FAIR research data. LU: Publications Office. https://data.europa.eu/doi/10.2777/02999. - -European Commission. (2020). The economic impact of open data: opportunities for value creation in Europe. LU: Publications Office. . - -Fell, M.J. (2019). The Economic Impacts of Open Science: A Rapid Evidence Assessment. Publications, 7(3), p.46. - -Harding, R. J. (2017). Open Science and Accelerating Discovery in Rare and Neglected Diseases. Proceedings of the 21st International Conference on Electronic Publishing, 1–5. - -Herala, A., Vanhala, E., Porras, J. and Krri, T. (2016). Experiences about opening data in private sector: A systematic literature review. In: 2016 SAI Computing Conference (SAI). 2016 SAI Computing Conference (SAI). London: IEEE, pp.715–724. https://ieeexplore.ieee.org/document/7556060/ . - -Houghton, J., Swan, A. and Brown, S. (2011). Access to research and technical information in Denmark. https://eprints.soton.ac.uk/272603/. - -Klebel, T., Cole, N. L., Tsipouri, L., Kormann, E., Karasz, I., Liarti, S., Stoy, L., Traag, V., Vignetti, S., & Ross-Hellauer, T. (2023). PathOS - D1.2 Scoping Review of Open Science Impact (Economic Impact of Open Science). Zenodo. - -Klebel, T., Traag V., (2024). Causality in science studies. - -Lindman, J., Kinnari, T. and Rossi, M. (2014). Industrial Open Data: Case Studies of Early Open Data Entrepreneurs. In: 2014 47th Hawaii International Conference on System Sciences. 2014 47th Hawaii International Conference on System Sciences. pp.739–748. . - -McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in r and Stan. 2nd ed. CRC Texts in Statistical Science. Boca Raton: Taylor; Francis, CRC Press. - -Parsons, D.D., Willis, D. and Jane, D. (2011). Benefits to the Private Sector of Open Access to Higher Education and Scholarly Research. - -Pearl, J. (2009). Causality: Models, Reasoning, and Inference (2nd ed.). Cambridge University Press. - -SPARC Europe. (2019). Using open and FAIR data to increase research efficiency \[online\]. https://sparceurope.org/wp-content/uploads/dlm_uploads/2019/04/SPARC-Europe_Brief_ODEfficiency.pdf . - -Tennant, J.P., Waldner, F., Jacques, D.C., Masuzzo, P., Collister, L.B. and Hartgerink, C.H.J. (2016). The academic, economic and societal impacts of Open Access: an evidence-based review. F1000Research, 5, p.632. - -van Assen, M.A.L.M., van Aert, R.C.M., Nuijten, M.B. and Wicherts, J.M. (2014). Why Publishing Everything Is More Effective than Selective Publishing of Statistically Significant Results. PLoS ONE, 9(1), p.e84896. - -van Vlijmen, H., Mons, A., Waalkens, A., Franke, W., Baak, A., Ruiter, G., Kirkpatrick, C., da Silva Santos, L. O. B., Meerman, B., Jellema, R., Arts, D., Kersloot, M., Knijnenburg, S., Lusher, S., Verbeeck, R., & Neefs, J.-M. (2020). The Need of Industry to Go FAIR. Data Intelligence, 2(1–2), 276–284. - -Woodward, J. (2003). CRITICAL NOTICE: *CAUSALITY* BY JUDEA PEARL. *Economics and Philosophy*, *19*(2), 321–340. - -Zeleti, F.A. and Ojo, A. (2014). Capability Matrix for Open Data. In: Camarinha-Matos, L. M. and Afsarmanesh, H., eds. Collaborative Systems for Smart Networked Environments. Berlin, Heidelberg: Springer, pp.498–509 - 1. ::: {#footnote-2} The *do-operator* is a notation used in causal inference to denote an intervention in a system. Written as , it represents setting variable to a specific value , simulating the effect of this intervention on other variables in the system while breaking any causal connections that usually determine . This approach allows us to differentiate causation from correlation, estimate causal effects, and answer hypothetical “what-if” scenarios. Through *do-calculus*, a set of rules introduced by Pearl, interventional distributions involving the *do-operator* can be converted into observational distributions. [↑](#footnote-ref-2) ::: \ No newline at end of file From d3347468f0aa0aeb7490b73808a036fc5dfeb393 Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 15:17:31 +0100 Subject: [PATCH 07/13] Fix footnote --- sections/0_causality/open_data_cost_savings.qmd | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/sections/0_causality/open_data_cost_savings.qmd b/sections/0_causality/open_data_cost_savings.qmd index 59073ca..221ee0a 100644 --- a/sections/0_causality/open_data_cost_savings.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -63,7 +63,10 @@ In addition, *Collaboration* acts as a confounder on the non-causal path *Open d This example highlights key components of causal inference: controlling for confounders *(Data availability, quality and standardisation, User skills, Collaboration,* and *Technological Infrastructure*), not controlling for mediators *(Time saving),* and not controlling for colliders *(Innovation),* as shown in @fig-adjustment. Constructing an appropriate DAG is important when aiming to draw causal conclusions. Without making assumptions explicit via a DAG, it would be unclear which variables should be controlled for and which not. Omitting important variables weakens the study's ability to draw accurate conclusions about cause and effect. Moreover, adding complexity to a DAG does not always change the variables that need to be controlled for when identifying the causal effect. In some cases, such as when adding confounders between unrelated variables, the identification of the relationship between Open data and Cost savings remains unaffected. However, if a confounder is introduced between Open data and Cost savings directly, it becomes necessary to control for it. -In the [causality introduction](causal_intro/article/intro-causality.qmd) we emphasise the importance of carefully selecting variables when analysing causal relationships. The introduction warns against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which @mcelreath2020 refers to as "causal salad". Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, @pearl2009 proposes using the *do-operator^[\[1\]](#footnote-2){#footnote-ref-2}^* to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system. +In the [causality introduction](causal_intro/article/intro-causality.qmd) we emphasise the importance of carefully selecting variables when analysing causal relationships. The introduction warns against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which @mcelreath2020 refers to as "causal salad". Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, @pearl2009 proposes using the *do-operator*[^note-do-op] to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system. + +[^note-do-op]: + The *do-operator* is a notation used in causal inference to denote an intervention in a system. Written as , it represents setting variable to a specific value , simulating the effect of this intervention on other variables in the system while breaking any causal connections that usually determine . This approach allows us to differentiate causation from correlation, estimate causal effects, and answer hypothetical “what-if” scenarios. Through *do-calculus*, a set of rules introduced by Pearl, interventional distributions involving the *do-operator* can be converted into observational distributions. This approach is effective in predicting how probability distributions shift under controlled changes when the causal structure is known. However, this approach depends on having an established causal graph, limiting its use for exploring causality from scratch. Critics suggest an alternative approach that allows causal discovery through experimentation without prior assumptions about mechanisms [@woodward2003]. This is especially useful in complex fields, such as social and biomedical sciences, where causal relationships are less understood. Considering this, @cunningham_causal_2021 argues that sample selection problems have been recognised long before the introduction of DAGs, with early solutions like Heckman (1979), and emphasizes that an atheoretical approach to empiricism is inadequate. He asserts that causal inference requires a deep understanding of the behavioural processes behind the phenomenon being studied, and while DAGs are useful, they cannot replace the need for theoretical knowledge in creating credible identification strategies. Thus, causal inference is not solved by simply collecting more data, but by integrating theory with empirical analysis. @@ -78,7 +81,3 @@ It is also important to acknowledge that causal pathways are not static and may A targeted survey could be a viable approach to address the empirical challenges arising from the lack of data and counterfactual evidence in studying the economic impact of open data. This survey should be carefully designed to measure the direct effects of open data on cost savings while accounting for confounding factors. It should aim to answer specific research questions, such as the extent of open data sharing, the perceived efficiency gains from data use, and the direct cost reductions attributed to open data initiatives. The survey would need to collect both longitudinal and cross-sectional data from a diverse sample of organisations that share open data as well as those that do not. This would allow for comparisons of outcomes and the establishment of causal relationships. Key variables to be measured should include the extent of open data sharing, cost reductions, new products or services enabled or accelerated by open data, and changes in collaboration. Questions investigating the perceived causal pathways would also be essential. Incorporating temporal data would provide further insight, enabling a better understanding of causal direction and distinguishing between short-term and long-term effects. Access to such detailed data would enable the use of causal inference techniques, such as Propensity Score Matching (PSM) or the Difference-in-Differences (DID) estimator, to identify the true impact of open data on cost savings. By addressing current data gaps, this survey could provide the empirical evidence needed to validate and refine the DAG, enhancing our understanding of how open data drives economic outcomes like cost savings. For instance, in the PSM context, a rich dataset would support the development of a more comprehensive DAG, helping to identify variables to include in the matching process (e.g., data availability, technological infrastructure, and user skills) and those to exclude (e.g., time savings and innovation). This approach would strengthen the empirical basis for analysing the causal impact of open data. - -1. ::: {#footnote-2} - The *do-operator* is a notation used in causal inference to denote an intervention in a system. Written as , it represents setting variable to a specific value , simulating the effect of this intervention on other variables in the system while breaking any causal connections that usually determine . This approach allows us to differentiate causation from correlation, estimate causal effects, and answer hypothetical “what-if” scenarios. Through *do-calculus*, a set of rules introduced by Pearl, interventional distributions involving the *do-operator* can be converted into observational distributions. [↑](#footnote-ref-2) - ::: \ No newline at end of file From f233002c4566bf46e00a86563022b438aafe7608 Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 15:21:24 +0100 Subject: [PATCH 08/13] fix few references --- references.bib | 2 +- sections/0_causality/open_data_cost_savings.qmd | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/references.bib b/references.bib index e1caf19..188ae2c 100644 --- a/references.bib +++ b/references.bib @@ -807,7 +807,7 @@ @article{estimati2015 langid = {en} } -@book{EU-DGRI2018, +@book{EC-DGRI2018, title = {Cost-benefit analysis for FAIR research data: cost of not having FAIR research data.}, author = {{European Commission. Directorate General for Research and Innovation.} and {PwC EU Services.}}, year = {2018}, diff --git a/sections/0_causality/open_data_cost_savings.qmd b/sections/0_causality/open_data_cost_savings.qmd index 221ee0a..29b582c 100644 --- a/sections/0_causality/open_data_cost_savings.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -13,7 +13,7 @@ History (please fill out in reverse chronological order, latest revision on top) ## Literature background -Measuring the economic impact of open science and open data has proven to be challenging. Many theoretical studies highlight the benefits of making research results public, with strong support for Open Science from economic research on technological change [\@chataway2018](Yozwiak%20et%20al.,%202015;%20Mazzucato,%202011). However, only few studies have attempted to measure the impacts of open science compared to closed science, and more robust evidence on how Open Science drives innovation and economic outcomes is needed to strengthen support and counter emerging criticisms [@karasz2024] (Ali-Khan et al., 2018). The existing literature mainly concentrates on specific sectors, particularly health, medicine, and biosciences, which receive more attention due to early regulation by funders and significant interest in clinical trial outcomes. Another important stream of literature is focused on highlighting the economic value of Open Science through personal industry experiences, though lacking precise quantitative evidence, with contributions from McManamay and Utz (2014) on fisheries, @harding2017 on medicine, Chan (2015) on the transition to an Open Science model, and @chen2017 on the role of open data in AI and machine learning applications. Although directly linking economic outcomes to open data initiatives can be challenging, with authors combining theoretical arguments and the limited quantitative evidence available at the time of their publication [@ali2022,@tennant2016,@fell2019,@wehn2021] (Arzberger, 2004), open access to findings and data is considered to lead to significant savings in access costs. By removing paywalls and subscription fees, open data allows researchers and businesses to access valuable information without incurring additional costs. A major economic benefit of lowering the cost of knowledge is the availability of an extra budget that can be reallocated for other purposes [@tennant2016]. +Measuring the economic impact of open science and open data has proven to be challenging. Many theoretical studies highlight the benefits of making research results public, with strong support for Open Science from economic research on technological change [@chataway2018] (Yozwiak et al., 2015; Mazzucato,2011). However, only few studies have attempted to measure the impacts of open science compared to closed science, and more robust evidence on how Open Science drives innovation and economic outcomes is needed to strengthen support and counter emerging criticisms [@karasz2024] (Ali-Khan et al., 2018). The existing literature mainly concentrates on specific sectors, particularly health, medicine, and biosciences, which receive more attention due to early regulation by funders and significant interest in clinical trial outcomes. Another important stream of literature is focused on highlighting the economic value of Open Science through personal industry experiences, though lacking precise quantitative evidence, with contributions from McManamay and Utz (2014) on fisheries, @harding2017 on medicine, Chan (2015) on the transition to an Open Science model, and @chen2017 on the role of open data in AI and machine learning applications. Although directly linking economic outcomes to open data initiatives can be challenging, with authors combining theoretical arguments and the limited quantitative evidence available at the time of their publication [@ali2022,@tennant2016,@fell2019,@wehn2021] (Arzberger, 2004), open access to findings and data is considered to lead to significant savings in access costs. By removing paywalls and subscription fees, open data allows researchers and businesses to access valuable information without incurring additional costs. A major economic benefit of lowering the cost of knowledge is the availability of an extra budget that can be reallocated for other purposes [@tennant2016]. By eliminating barriers to data access, organisations can reduce the time spent on data collection and focus on core activities. @fell2019 notes that open science reduces the time associated with accessing new knowledge, directly contributing to enhanced research quality and productivity increases. Although the specific time savings associated with open access were not directly tested, @parsons_benefits_2011 conducted an interview-based study that supports this potential benefit. [@beagrie2014] demonstrate that data sharing and curation significantly enhance research efficiency, with labour cost savings ranging from two to over twenty times the operational costs of the data centres. While there are indirect costs associated with preparing data for sharing, such as time and expenses incurred by depositors, the benefits realised by users through improved efficiency and their willingness to pay for access far outweigh these costs [@beagrie2012]. Additionally, findings from [@beagrie2016,@beagrie2021] provide compelling evidence of the timesaving and efficiency gains associated with using European Bioinformatics Institute services (EMBL-EBI). @@ -33,7 +33,7 @@ As discussed in the general introduction on causal inference, we use DAGs to rep ## The effect of Open data on Cost Saving -In this section, we apply the concepts presented in the section Causality in Science Studies to potential research questions. We present a specific perspective on causal inference through the lens of structural causal models [@pearl2009]. +In this section, we apply the concepts presented in the section Causality in Science Studies to potential research questions. We present a specific perspective on causal inference through the lens of structural causal models [@pearl_causality_2009]. Suppose we are interested in assessing the *total causal effect* of *Open Data* on *Cost Saving*. According to our model ([@fig-model]), there are multiple pathways from Open Data to Cost Saving, some are causal, some are not. To estimate the causal effect of interest, we need to make sure that all causal paths are open, and all non-causal paths are closed. Within the DAG representation, two causal pathways can be identified: a *direct* pathway of *Open Data → Cost Saving,* representing the direct effect of Open Data on Cost Savings*,* and an *indirect* pathway *Open Data → Time Saving → Cost Saving*, where the effect is indirect and mediated by Time Saving. The *direct* effect captures the immediate benefits of providing free access to datasets, while the *indirect* effect, mediated by Time Savings, strengthens the relationship by triggering additional efficiencies that also lead to Cost Savings. @@ -63,7 +63,7 @@ In addition, *Collaboration* acts as a confounder on the non-causal path *Open d This example highlights key components of causal inference: controlling for confounders *(Data availability, quality and standardisation, User skills, Collaboration,* and *Technological Infrastructure*), not controlling for mediators *(Time saving),* and not controlling for colliders *(Innovation),* as shown in @fig-adjustment. Constructing an appropriate DAG is important when aiming to draw causal conclusions. Without making assumptions explicit via a DAG, it would be unclear which variables should be controlled for and which not. Omitting important variables weakens the study's ability to draw accurate conclusions about cause and effect. Moreover, adding complexity to a DAG does not always change the variables that need to be controlled for when identifying the causal effect. In some cases, such as when adding confounders between unrelated variables, the identification of the relationship between Open data and Cost savings remains unaffected. However, if a confounder is introduced between Open data and Cost savings directly, it becomes necessary to control for it. -In the [causality introduction](causal_intro/article/intro-causality.qmd) we emphasise the importance of carefully selecting variables when analysing causal relationships. The introduction warns against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which @mcelreath2020 refers to as "causal salad". Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, @pearl2009 proposes using the *do-operator*[^note-do-op] to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system. +In the [causality introduction](causal_intro/article/intro-causality.qmd) we emphasise the importance of carefully selecting variables when analysing causal relationships. The introduction warns against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which @mcelreath2020 refers to as "causal salad". Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, @pearl_causality_2009 proposes using the *do-operator*[^note-do-op] to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system. [^note-do-op]: The *do-operator* is a notation used in causal inference to denote an intervention in a system. Written as , it represents setting variable to a specific value , simulating the effect of this intervention on other variables in the system while breaking any causal connections that usually determine . This approach allows us to differentiate causation from correlation, estimate causal effects, and answer hypothetical “what-if” scenarios. Through *do-calculus*, a set of rules introduced by Pearl, interventional distributions involving the *do-operator* can be converted into observational distributions. From f9b966c5c8c377d02ea428b51bbbac32c0ee0a0a Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 15:24:14 +0100 Subject: [PATCH 09/13] Add authors --- sections/0_causality/open_data_cost_savings.qmd | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/sections/0_causality/open_data_cost_savings.qmd b/sections/0_causality/open_data_cost_savings.qmd index 29b582c..b1ff111 100644 --- a/sections/0_causality/open_data_cost_savings.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -1,3 +1,20 @@ +--- +author: + - name: Marla Scorrano + affiliations: + - ref: csil + - name: E. Delugas + orcid: 0009-0000-4638-1062 + affiliations: + - ref: csil + +affiliations: +- id: csil + name: Centre for Industrial Studies + city: Milan + country: Italy +--- + # The effect of Open Data on cost savings {#the-effect-of-open-data-on-cost-savings .unnumbered} History (please fill out in reverse chronological order, latest revision on top): From ee5472703788a91ce08af46dfa45f4de1e01d39a Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 15:28:18 +0100 Subject: [PATCH 10/13] Add link to intro --- sections/0_causality/open_data_cost_savings.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sections/0_causality/open_data_cost_savings.qmd b/sections/0_causality/open_data_cost_savings.qmd index b1ff111..ec1fda9 100644 --- a/sections/0_causality/open_data_cost_savings.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -44,7 +44,7 @@ Despite the theoretical benefits of open data, several limitations hinder a comp ## Directed Acyclic Graph (DAG) -As discussed in the general introduction on causal inference, we use DAGs to represent structural causal models. In the following, a DAG ([@fig-model]) is employed to examine the causal relationship between *Open Data* and *Cost Savings*. The visual illustrates multiple potential pathways, including a direct path from Open Data and Cost Savings, an indirect one involving Time Savings (i.e., a mediator), and additional paths that incorporate factors affecting either Open Data or Time Savings (i.e., confounders). These additional factors, such as technological infrastructure, data quality and availability, standardisation, user skills, innovation, and collaboration introduce layers of complexity to the model. As we will show in the subsequent sections, they are essential to discuss the causal and non-causal, open and closed, relationships among all these variables. +As discussed in the general [introduction on causal inference](causal_intro/article/intro-causality.qmd), we use DAGs to represent structural causal models. In the following, a DAG ([@fig-model]) is employed to examine the causal relationship between *Open Data* and *Cost Savings*. The visual illustrates multiple potential pathways, including a direct path from Open Data and Cost Savings, an indirect one involving Time Savings (i.e., a mediator), and additional paths that incorporate factors affecting either Open Data or Time Savings (i.e., confounders). These additional factors, such as technological infrastructure, data quality and availability, standardisation, user skills, innovation, and collaboration introduce layers of complexity to the model. As we will show in the subsequent sections, they are essential to discuss the causal and non-causal, open and closed, relationships among all these variables. ![Hypothetical structural causal model on Open Data](figures/DAG_open_data_cost_savings-0.png){#fig-model} From 99dcbf117c2527fa7db0309035297b904c86dac5 Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 16:06:50 +0100 Subject: [PATCH 11/13] Correct arrows --- .../0_causality/open_data_cost_savings.qmd | 25 +++++++++---------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/sections/0_causality/open_data_cost_savings.qmd b/sections/0_causality/open_data_cost_savings.qmd index ec1fda9..917e771 100644 --- a/sections/0_causality/open_data_cost_savings.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -52,38 +52,37 @@ As discussed in the general [introduction on causal inference](causal_intro/arti In this section, we apply the concepts presented in the section Causality in Science Studies to potential research questions. We present a specific perspective on causal inference through the lens of structural causal models [@pearl_causality_2009]. -Suppose we are interested in assessing the *total causal effect* of *Open Data* on *Cost Saving*. According to our model ([@fig-model]), there are multiple pathways from Open Data to Cost Saving, some are causal, some are not. To estimate the causal effect of interest, we need to make sure that all causal paths are open, and all non-causal paths are closed. Within the DAG representation, two causal pathways can be identified: a *direct* pathway of *Open Data → Cost Saving,* representing the direct effect of Open Data on Cost Savings*,* and an *indirect* pathway *Open Data → Time Saving → Cost Saving*, where the effect is indirect and mediated by Time Saving. The *direct* effect captures the immediate benefits of providing free access to datasets, while the *indirect* effect, mediated by Time Savings, strengthens the relationship by triggering additional efficiencies that also lead to Cost Savings. +Suppose we are interested in assessing the *total causal effect* of *Open Data* on *Cost Saving*. According to our model ([@fig-model]), there are multiple pathways from Open Data to Cost Saving, some are causal, some are not. To estimate the causal effect of interest, we need to make sure that all causal paths are open, and all non-causal paths are closed. Within the DAG representation, two causal pathways can be identified: a *direct* pathway of *Open Data* $\rightarrow$ *Cost Saving*, representing the direct effect of Open Data on Cost Savings, and an *indirect* pathway *Open Data* $\rightarrow$ *Time Saving* $\rightarrow$ *Cost Saving*, where the effect is indirect and mediated by *Time Saving*. The *direct* effect captures the immediate benefits of providing free access to datasets, while the *indirect* effect, mediated by Time Savings, strengthens the relationship by triggering additional efficiencies that also lead to Cost Savings. To properly estimate the *total* causal effect of Open Data on Cost Saving, an empirical model should not control for Time saving. On the contrary, if the model conditions on Time Savings, even implicitly (e.g., by accounting for approaches and tools that optimise and speed up data access and processing), it closes the causal path and introduces biases into the estimation of the total effect ([@fig-mediator]). ![DAG illustrating the misleading effect of conditioning on the mediator variable, Time Saving. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey arrows indicate indirect association that may involve non-causal relationship.](figures/DAG_open_data_cost_savings-1.png){#fig-mediator} -As mentioned before, the proposed model accounts for additional variables such as *Data Availability, Quality and Standardisation*, *Users' Skills, Collaboration, and Technological Infrastructure*. These variables can act as *confounders* along different pathways illustrated in [@fig-model]. Examples of non-causal paths represented in the DAG are: +As mentioned before, the proposed model accounts for additional variables such as *Data Availability, Quality and Standardisation*, *Users' Skills*, *Collaboration*, and *Technological Infrastructure*. These variables can act as *confounders* along different pathways illustrated in [@fig-model]. Examples of non-causal paths represented in the DAG are: -- *Open Data* *←* *Technological Infrastructure* → *Time Saving* → *Cost Saving* -- *Open Data* ← *Data Availability, Quality and Standardisation* → *Time Saving* → *Cost Saving* -- *Open Data* ← *User’s skills* → *Time Saving* → *Cost Saving* +- *Open Data* $\leftarrow$ *Technological Infrastructure* $\rightarrow$ *Time Saving* $\rightarrow$ *Cost Saving* +- *Open Data* $\leftarrow$ *Data Availability, Quality and Standardisation* $\rightarrow$ *Time Saving* $\rightarrow$ *Cost Saving* +- *Open Data* $\leftarrow$ *User’s skills* $\rightarrow$ *Time Saving* $\rightarrow$ *Cost Saving* In these pathways, to correctly identify the causal effect of *Open Data* on *Time Saving*, and by extension, on *Cost Savings*, it is required to control for these confounders. This is because the confounders jointly affect Open Data and Time Savings and omitting them from empirical models results in omitted variable bias. In the proposed example, to correct identify the causal effect one should control for *Technological infrastructure*, *Data availability, quality and standardisation, and the skills of users.* There might be instances where a confounder is not observed because is not included in the dataset or not observable at all. In such cases, the non-causal path remains open, resulting in biased conclusions. In fact, unobservable factors might be correlated with observable variables, leaving causal paths unexplored. As a result, if these confounders are not accounted for, we are unable to fully isolate the causal effect of *Open Data* influences *Cost saving*. -Another case that makes not possible the identification of the causal effect is erroneously controlling for a collider (see the introduction for further details). In the pathway *Open Data* → *Innovation* ← *Collaboration*→ *Cost Saving,* the variable *Innovation* acts as a collider. Hence, this path is already closed, and the bias arises when controlling for innovation. +Another case that makes not possible the identification of the causal effect is erroneously controlling for a collider (see the introduction for further details). In the pathway *Open Data* $\rightarrow$ *Innovation* $\leftarrow$ *Collaboration*$\rightarrow$ *Cost Saving,* the variable *Innovation* acts as a collider. Hence, this path is already closed, and the bias arises when controlling for innovation. -Empirically speaking, in a model where to estimate the causal effect of Open Data on Cost savings we condition on Innovation (and not for Collaboration), it is likely to get a downward estimation of the causal effect, since both Collaboration and Open Data have a positive impact on Innovation. This conditioning opens up the non-causal pathway, Open Data → Innovation ← Collaboration → Cost Saving, which connect *Open Data* and *Cost Saving* through *Collaboration*, creating a spurious association and distorting the true effect of *Open Data* on *Cost Saving*. This is an example of *bad controls* [@angrist2009], a concept explained in the general introduction. Only by ignoring the collider, meaning non-conditioning on it in empirical models, we can effectively isolate the causal effect. This non-causal path is open because *Innovation* is open (because it is a collider that is conditioned on), and because *Collaboration* is open (because it is a confounder that is not conditioned on) (see @fig-collider). +Empirically speaking, in a model where to estimate the causal effect of Open Data on Cost savings we condition on Innovation (and not for Collaboration), it is likely to get a downward estimation of the causal effect, since both Collaboration and Open Data have a positive impact on Innovation. This conditioning opens up the non-causal pathway, *Open Data* $\rightarrow$ *Innovation* $\leftarrow$ *Collaboration* $\rightarrow$ *Cost Saving*, which connect *Open Data* and *Cost Saving* through *Collaboration*, creating a spurious association and distorting the true effect of *Open Data* on *Cost Saving*. This is an example of *bad controls* [@angrist2009], a concept explained in the general introduction. Only by ignoring the collider, meaning non-conditioning on it in empirical models, we can effectively isolate the causal effect. This non-causal path is open because *Innovation* is open (because it is a collider that is conditioned on), and because *Collaboration* is open (because it is a confounder that is not conditioned on) (see @fig-collider). ![DAG illustrating the misleading effect of conditioning on a collider variable, *Innovation,* and not conditioning on a confounder, *Collaboration*. Nodes that are controlled for have a thick outline. Grey nodes represent variables not considered in this figure. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey arrows indicate indirect association that may involve non-causal relationship.](figures/DAG_open_data_cost_savings-2.png){#fig-collider} -In addition, *Collaboration* acts as a confounder on the non-causal path *Open data* ← *Collaboration* → *Cost saving*. To identify the causal effect, we hence need to close this non-causal path by conditioning on *Collaboration*. After controlling for Collaboration, whether *Innovation* is conditioned on is then irrelevant for the identification of the causal effect. When all non-causal paths are closed, the research design is said to meet the *backdoor* criterion, a formal requirement that ensures the design blocks all non-causal paths between the treatment (*Open Data*) and the outcome (*Cost Saving*), enabling us to identify the causal effect in question [@cunningham_causal_2021] ([@fig-adjustment]). +In addition, *Collaboration* acts as a confounder on the non-causal path *Open data* $\leftarrow$ *Collaboration* $\rightarrow$ *Cost saving*. To identify the causal effect, we hence need to close this non-causal path by conditioning on *Collaboration*. After controlling for Collaboration, whether *Innovation* is conditioned on is then irrelevant for the identification of the causal effect. When all non-causal paths are closed, the research design is said to meet the *backdoor* criterion, a formal requirement that ensures the design blocks all non-causal paths between the treatment (*Open Data*) and the outcome (*Cost Saving*), enabling us to identify the causal effect in question [@cunningham_causal_2021] ([@fig-adjustment]). ![DAG illustrating the total effect *Open Data* on *Cost Saving*, conditioning on confounders and not on mediator and collider. Nodes that are controlled for have a thick outline. Green nodes are open, indicating they allow associations or relationships to flow through them along the paths they connect, while orange nodes are closed, blocking associations or relationships from flowing through the paths they connect. Black arrows represent potential causal influence, whereas grey dashed arrows indicate indirect association that may involve non-causal relationship.](figures/DAG_open_data_cost_savings-3.png){#fig-adjustment} -This example highlights key components of causal inference: controlling for confounders *(Data availability, quality and standardisation, User skills, Collaboration,* and *Technological Infrastructure*), not controlling for mediators *(Time saving),* and not controlling for colliders *(Innovation),* as shown in @fig-adjustment. Constructing an appropriate DAG is important when aiming to draw causal conclusions. Without making assumptions explicit via a DAG, it would be unclear which variables should be controlled for and which not. Omitting important variables weakens the study's ability to draw accurate conclusions about cause and effect. Moreover, adding complexity to a DAG does not always change the variables that need to be controlled for when identifying the causal effect. In some cases, such as when adding confounders between unrelated variables, the identification of the relationship between Open data and Cost savings remains unaffected. However, if a confounder is introduced between Open data and Cost savings directly, it becomes necessary to control for it. +This example highlights key components of causal inference: controlling for confounders (*Data availability, quality and standardisation*, *User skills*, *Collaboration*, and *Technological Infrastructure*), not controlling for mediators (*Time saving*), and not controlling for colliders (*Innovation*), as shown in @fig-adjustment. Constructing an appropriate DAG is important when aiming to draw causal conclusions. Without making assumptions explicit via a DAG, it would be unclear which variables should be controlled for and which not. Omitting important variables weakens the study's ability to draw accurate conclusions about cause and effect. Moreover, adding complexity to a DAG does not always change the variables that need to be controlled for when identifying the causal effect. In some cases, such as when adding confounders between unrelated variables, the identification of the relationship between Open data and Cost savings remains unaffected. However, if a confounder is introduced between Open data and Cost savings directly, it becomes necessary to control for it. -In the [causality introduction](causal_intro/article/intro-causality.qmd) we emphasise the importance of carefully selecting variables when analysing causal relationships. The introduction warns against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which @mcelreath2020 refers to as "causal salad". Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, @pearl_causality_2009 proposes using the *do-operator*[^note-do-op] to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system. +In the [causality introduction](causal_intro/article/intro-causality.qmd) we emphasise the importance of carefully selecting variables when analysing causal relationships. The introduction warns against two common errors: relying on the data (e.g., through stepwise regression) to decide which variables to control for, or including all available variables, which @mcelreath2020 refers to as "causal salad". Both approaches can lead to incorrect conclusions. Specifically, including mediating variables or focusing only on certain cases could obscure the true effect of open data on outcomes like cost savings. In this regard, @pearl_causality_2009 proposes using the *do-operator*[^1] to define causal effects, where an intervention on one variable allows us to observe changes in another, thereby illuminating causal connections within a system. -[^note-do-op]: - The *do-operator* is a notation used in causal inference to denote an intervention in a system. Written as , it represents setting variable to a specific value , simulating the effect of this intervention on other variables in the system while breaking any causal connections that usually determine . This approach allows us to differentiate causation from correlation, estimate causal effects, and answer hypothetical “what-if” scenarios. Through *do-calculus*, a set of rules introduced by Pearl, interventional distributions involving the *do-operator* can be converted into observational distributions. +[^1]: The *do-operator* is a notation used in causal inference to denote an intervention in a system. Written as , it represents setting variable to a specific value, simulating the effect of this intervention on other variables in the system while breaking any causal connections that usually determine . This approach allows us to differentiate causation from correlation, estimate causal effects, and answer hypothetical "what-if" scenarios. Through *do-calculus*, a set of rules introduced by Pearl, interventional distributions involving the *do-operator* can be converted into observational distributions. This approach is effective in predicting how probability distributions shift under controlled changes when the causal structure is known. However, this approach depends on having an established causal graph, limiting its use for exploring causality from scratch. Critics suggest an alternative approach that allows causal discovery through experimentation without prior assumptions about mechanisms [@woodward2003]. This is especially useful in complex fields, such as social and biomedical sciences, where causal relationships are less understood. Considering this, @cunningham_causal_2021 argues that sample selection problems have been recognised long before the introduction of DAGs, with early solutions like Heckman (1979), and emphasizes that an atheoretical approach to empiricism is inadequate. He asserts that causal inference requires a deep understanding of the behavioural processes behind the phenomenon being studied, and while DAGs are useful, they cannot replace the need for theoretical knowledge in creating credible identification strategies. Thus, causal inference is not solved by simply collecting more data, but by integrating theory with empirical analysis. @@ -97,4 +96,4 @@ It is also important to acknowledge that causal pathways are not static and may A targeted survey could be a viable approach to address the empirical challenges arising from the lack of data and counterfactual evidence in studying the economic impact of open data. This survey should be carefully designed to measure the direct effects of open data on cost savings while accounting for confounding factors. It should aim to answer specific research questions, such as the extent of open data sharing, the perceived efficiency gains from data use, and the direct cost reductions attributed to open data initiatives. The survey would need to collect both longitudinal and cross-sectional data from a diverse sample of organisations that share open data as well as those that do not. This would allow for comparisons of outcomes and the establishment of causal relationships. Key variables to be measured should include the extent of open data sharing, cost reductions, new products or services enabled or accelerated by open data, and changes in collaboration. Questions investigating the perceived causal pathways would also be essential. Incorporating temporal data would provide further insight, enabling a better understanding of causal direction and distinguishing between short-term and long-term effects. -Access to such detailed data would enable the use of causal inference techniques, such as Propensity Score Matching (PSM) or the Difference-in-Differences (DID) estimator, to identify the true impact of open data on cost savings. By addressing current data gaps, this survey could provide the empirical evidence needed to validate and refine the DAG, enhancing our understanding of how open data drives economic outcomes like cost savings. For instance, in the PSM context, a rich dataset would support the development of a more comprehensive DAG, helping to identify variables to include in the matching process (e.g., data availability, technological infrastructure, and user skills) and those to exclude (e.g., time savings and innovation). This approach would strengthen the empirical basis for analysing the causal impact of open data. +Access to such detailed data would enable the use of causal inference techniques, such as Propensity Score Matching (PSM) or the Difference-in-Differences (DID) estimator, to identify the true impact of open data on cost savings. By addressing current data gaps, this survey could provide the empirical evidence needed to validate and refine the DAG, enhancing our understanding of how open data drives economic outcomes like cost savings. For instance, in the PSM context, a rich dataset would support the development of a more comprehensive DAG, helping to identify variables to include in the matching process (e.g., data availability, technological infrastructure, and user skills) and those to exclude (e.g., time savings and innovation). This approach would strengthen the empirical basis for analysing the causal impact of open data. \ No newline at end of file From cde43d55aad700e67a568a257209a1764976dff2 Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Wed, 18 Dec 2024 16:07:56 +0100 Subject: [PATCH 12/13] Minor fix --- sections/0_causality/open_data_cost_savings.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sections/0_causality/open_data_cost_savings.qmd b/sections/0_causality/open_data_cost_savings.qmd index 917e771..2dc8969 100644 --- a/sections/0_causality/open_data_cost_savings.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -68,7 +68,7 @@ In these pathways, to correctly identify the causal effect of *Open Data* on *Ti There might be instances where a confounder is not observed because is not included in the dataset or not observable at all. In such cases, the non-causal path remains open, resulting in biased conclusions. In fact, unobservable factors might be correlated with observable variables, leaving causal paths unexplored. As a result, if these confounders are not accounted for, we are unable to fully isolate the causal effect of *Open Data* influences *Cost saving*. -Another case that makes not possible the identification of the causal effect is erroneously controlling for a collider (see the introduction for further details). In the pathway *Open Data* $\rightarrow$ *Innovation* $\leftarrow$ *Collaboration*$\rightarrow$ *Cost Saving,* the variable *Innovation* acts as a collider. Hence, this path is already closed, and the bias arises when controlling for innovation. +Another case that makes not possible the identification of the causal effect is erroneously controlling for a collider (see the introduction for further details). In the pathway *Open Data* $\rightarrow$ *Innovation* $\leftarrow$ *Collaboration* $\rightarrow$ *Cost Saving,* the variable *Innovation* acts as a collider. Hence, this path is already closed, and the bias arises when controlling for innovation. Empirically speaking, in a model where to estimate the causal effect of Open Data on Cost savings we condition on Innovation (and not for Collaboration), it is likely to get a downward estimation of the causal effect, since both Collaboration and Open Data have a positive impact on Innovation. This conditioning opens up the non-causal pathway, *Open Data* $\rightarrow$ *Innovation* $\leftarrow$ *Collaboration* $\rightarrow$ *Cost Saving*, which connect *Open Data* and *Cost Saving* through *Collaboration*, creating a spurious association and distorting the true effect of *Open Data* on *Cost Saving*. This is an example of *bad controls* [@angrist2009], a concept explained in the general introduction. Only by ignoring the collider, meaning non-conditioning on it in empirical models, we can effectively isolate the causal effect. This non-causal path is open because *Innovation* is open (because it is a collider that is conditioned on), and because *Collaboration* is open (because it is a confounder that is not conditioned on) (see @fig-collider). From 49144d4456d0bb44016c9da86b53f220aaddff2f Mon Sep 17 00:00:00 2001 From: Vincent Traag Date: Fri, 20 Dec 2024 23:54:13 +0100 Subject: [PATCH 13/13] Added remaining references --- references.bib | 146 ++++++++++++++---- .../0_causality/open_data_cost_savings.qmd | 6 +- 2 files changed, 120 insertions(+), 32 deletions(-) diff --git a/references.bib b/references.bib index 188ae2c..da932c4 100644 --- a/references.bib +++ b/references.bib @@ -41,6 +41,17 @@ @article{aksnes2019 langid = {en} } +@article{ali-khan2018, + title = {Defining Success in Open Science}, + author = {Ali-Khan, Sarah E. and Jean, Antoine and MacDonald, Emily and Gold, E. Richard}, + year = {2018}, + month = {03}, + journal = {MNI Open Research}, + pages = {2}, + volume = {2}, + doi = {10.12688/mniopenres.12780.2} +} + @article{ali2022, title = {The Role of FAIR Data towards Sustainable Agricultural Performance: A Systematic Literature Review}, author = {Ali, Basharat and Dahlhaus, Peter}, @@ -150,6 +161,18 @@ @article{arshad2016 doi = {10.1517/17460441.2016.1144587} } +@article{arzberger_promoting_2004, + title = {Promoting {Access} to {Public} {Research} {Data} for {Scientific}, {Economic}, and {Social} {Development}}, + volume = {3}, + issn = {1683-1470}, + doi = {10.2481/dsj.3.135}, + language = {en}, + journal = {Data Science Journal}, + author = {Arzberger, P and Schroeder, P and Beaulieu, A and Bowker, G and Casey, K and Laaksonen, L and Moorman, D and Uhlir, P and Wouters, P}, + year = {2004}, + pages = {135--152} +} + @article{assen2014, title = {Why Publishing Everything Is More Effective than Selective Publishing of Statistically Significant Results}, author = {Assen, Marcel A. L. M. van and Aert, Robbie C. M. van and Nuijten, {Michèle B.} and Wicherts, Jelte M.}, @@ -200,6 +223,7 @@ @techreport{beagrie2014 year = {2014} } + @techreport{beagrie2016, title = {The Value and Impact of the European Bioinformatics Institute}, author = {Beagrie, N. and Houghton, J.}, @@ -207,13 +231,13 @@ @techreport{beagrie2016 url = {https://www.embl.org/documents/wp-content/uploads/2021/09/EMBL-EBI_Impact_report-2016-summary.pdf} } + @techreport{beagrie2021, title = {Data-driven discovery: The value and impact of EMBL-EBI managed data resources}, author = {Beagrie, Neil and Houghton, John}, year = {2021} } - @book{bellis2009, title = {Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics}, author = {Bellis, Nicola De}, @@ -225,7 +249,6 @@ @book{bellis2009 langid = {en} } - @article{bornmann_scientific_2011, title = {Scientific peer review}, volume = {45}, @@ -336,6 +359,7 @@ @article{budi2022 doi = {10.1007/s11192-022-04567-4} } + @article{butler_explaining_2003, title = {Explaining {Australia}'s increased share of {ISI} publications—the effects of a funding formula based on publication counts}, volume = {32}, @@ -349,6 +373,7 @@ @article{butler_explaining_2003 pages = {143--155} } + @article{carlin2023, title = {Where is all the research software? An analysis of software in UK academic repositories}, author = {Carlin, Domhnall and Rainer, Austen and Wilson, David}, @@ -360,8 +385,6 @@ @article{carlin2023 url = {https://peerj.com/articles/cs-1546/}, note = {Publisher: PeerJ Inc.} } - - @article{castelnovo2023, title = {A quasi-experimental design to assess the innovative impact of public procurement: An application to the Italian space industry}, author = {Castelnovo, Paolo and {Clò}, Stefano and Florio, Massimo}, @@ -374,12 +397,12 @@ @article{castelnovo2023 note = {Publisher: Elsevier} } - @unpublished{catalano_forthcoming, author = {Catalano, G. and Florio, M. and Articolo, R. and Consiglio, G. and Eggleton, D.}, note = {forthcoming}, title = {Diamond contribution to the development of an innovative vaccine for the Foot-and-Mouth Disease: a veterinary pathology of global significance, case study report} } + @article{catalano2021, title = {From scientific experiments to innovation: Impact pathways of a Synchrotron Light Facility}, author = {Catalano, Gelsomina and {López}, {Gaston García} and {Sánchez}, Alejandro and Vignetti, Silvia}, @@ -409,6 +432,18 @@ @inproceedings{cazacu2023 address = {New York, NY, USA} } +@inproceedings{chan_cost_2015, + title = {Cost {Impact} in {Managing} the {Transition} to an {Open} {Access} {Model}}, + isbn = {978-1-941269-03-9}, + doi = {10.5703/1288284315595}, + booktitle = {The {Importance} of {Being} {Earnest}}, + publisher = {Against the Grain}, + author = {Chan, Gayle Rosemary}, + month = sep, + year = {2015}, + pages = {358--362} +} + @inbook{chataway2018, title = {How Will Open Science Impact on University/Industry Collaborations?}, author = {Chataway, Joanna and Parks, Sarah and Smith, Elta}, @@ -765,6 +800,15 @@ @article{dulongderosnay2014 note = {Publisher: Universidad de Talca} } +@book{EC-DGRI2018, + title = {Cost-benefit analysis for FAIR research data: cost of not having FAIR research data.}, + author = {{European Commission. Directorate General for Research and Innovation.} and {PwC EU Services.}}, + year = {2018}, + publisher = {Publications Office}, + doi = {10.2777/02999}, + url = {https://data.europa.eu/doi/10.2777/02999} +} + @book{EC2018, title = {Cost-benefit analysis for FAIR research data: cost of not having FAIR research data}, year = {2018}, @@ -807,15 +851,6 @@ @article{estimati2015 langid = {en} } -@book{EC-DGRI2018, - title = {Cost-benefit analysis for FAIR research data: cost of not having FAIR research data.}, - author = {{European Commission. Directorate General for Research and Innovation.} and {PwC EU Services.}}, - year = {2018}, - publisher = {Publications Office}, - doi = {10.2777/02999}, - url = {https://data.europa.eu/doi/10.2777/02999} -} - @book{EU2020, title = {The economic impact of open data: opportunities for value creation in Europe}, author = {Publications Office of the European Union, and Huyer, Esther and Knippenberg, Laura van}, @@ -1113,6 +1148,7 @@ @article{gordon2021 langid = {en} } + @article{gormally2012, title = {Developing a Test of Scientific Literacy Skills (TOSLS): Measuring Undergraduates{\textquoteright} Evaluation of Scientific Information and Arguments}, author = {Gormally, Cara and Brickman, Peggy and Lutz, Mary}, @@ -1124,6 +1160,7 @@ @article{gormally2012 doi = {10.1187/cbe.12-03-0026} } + @misc{goyal_causal_2024, title = {Causal {Effect} of {Group} {Diversity} on {Redundancy} and {Coverage} in {Peer}-{Reviewing}}, doi = {10.48550/arXiv.2411.11437}, @@ -1150,7 +1187,6 @@ @article{group_fair_2020 year = {2020} } - @techreport{gruenpeter2021, title = {Defining Research Software: a controversial discussion}, author = {Gruenpeter, Morane and Katz, Daniel S. and Lamprecht, Anna-Lena and Honeyman, Tom and Garijo, Daniel and Struck, Alexander and Niehues, Anna and Martinez, Paula Andrea and Castro, Leyla Jael and Rabemanantsoa, Tovo and Chue Hong, Neil P. and Martinez-Ortiz, Carlos and Sesink, Laurents and Liffers, Matthias and Fouilloux, Anne Claire and Erdmann, Chris and Peroni, Silvio and Martinez Lavanchy, Paula and Todorov, Ilian and Sinha, Manodeep}, @@ -1170,7 +1206,6 @@ @book{guideto2015 langid = {en} } - @article{guyan2022, title = {Queer data}, author = {Guyan, Kevin}, @@ -1204,6 +1239,7 @@ @inproceedings{harding2017 address = {Limassol, Cyprus} } + @article{hardwicke2018, title = {Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal {\emph{Cognition}}}, author = {Hardwicke, Tom E. and Mathur, Maya B. and MacDonald, Kyle and Nilsonne, Gustav and Banks, George C. and Kidwell, Mallory C. and Hofelich Mohr, Alicia and Clayton, Elizabeth and Yoon, Erica J. and Henry Tessler, Michael and Lenne, Richie L. and Altman, Sara and Long, Bria and Frank, Michael C.}, @@ -1218,6 +1254,7 @@ @article{hardwicke2018 url = {https://royalsocietypublishing.org/doi/10.1098/rsos.180448} } + @article{hardwicke2021, title = {Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study}, author = {Hardwicke, Tom E. and Bohn, Manuel and MacDonald, Kyle and Hembacher, Emily and Nuijten, {Michèle B.} and Peloquin, Benjamin N. and deMayo, Benjamin E. and Long, Bria and Yoon, Erica J. and Frank, Michael C.}, @@ -1246,7 +1283,6 @@ @article{healy2017 url = {https://doi.org/10.1177/0735275117709046} } - @techreport{heathers2018, title = {Recovering data from summary statistics: Sample Parameter Reconstruction via Iterative TEchniques (SPRITE)}, author = {Heathers, James A. and Anaya, Jordan and Zee, Tim van der and Brown, Nicholas J. L.}, @@ -1259,7 +1295,6 @@ @techreport{heathers2018 langid = {en} } - @book{heathers2018a, title = {SPRITE}, author = {Heathers, James A. and Brown, Nicholas J. L. and Zee, Tim van der and Anaya, Jordan}, @@ -1270,6 +1305,17 @@ @book{heathers2018a note = {Publisher: OSF} } +@article{heckman1979, + title = {Sample Selection Bias as a Specification Error}, + author = {Heckman, James J.}, + year = {1979}, + journal = {Econometrica}, + pages = {153--161}, + volume = {47}, + number = {1}, + doi = {10.2307/1912352} +} + @inproceedings{herala2016, title = {2016 SAI Computing Conference (SAI)}, author = {Herala, Antti and Vanhala, Erno and Porras, Jari and Krri, Timo}, @@ -1816,6 +1862,20 @@ @article{marino2018 langid = {en} } +@article{mazzucato_entrepreneurial_2011, + title = {The entrepreneurial state}, + volume = {49}, + issn = {1362-6620}, + doi = {10.3898/136266211798411183}, + language = {en}, + number = {49}, + journal = {Soundings}, + author = {Mazzucato, Mariana}, + month = nov, + year = {2011}, + pages = {131--142} +} + @article{mccaffrey2020, title = {Open Science Skills Visualisation}, author = {McCaffrey, Ciara and Meyer, Thorsten and Riera Quintero, Clara and Swiatek, Cecile and Marcerou-Ramel, Nathalie and {Gillén}, Camilla and Clavel, Karin and Wojciechowska, Anna and Brinken, Helene and Prevoo, {Mariëlle} and Egerton, Frank}, @@ -1839,6 +1899,19 @@ @book{mcelreath2020 address = {Boca Raton} } +@article{mcmanamay_openaccess_2014, + title = {Open‐{Access} {Databases} as {Unprecedented} {Resources} and {Drivers} of {Cultural} {Change} in {Fisheries} {Science}}, + volume = {39}, + issn = {0363-2415, 1548-8446}, + doi = {10.1080/03632415.2014.946128}, + number = {9}, + journal = {Fisheries}, + author = {McManamay, Ryan A. and Utz, Ryan M.}, + month = sep, + year = {2014}, + pages = {417--425} +} + @book{merton1973, title = {The Sociology of Science: Theoretical and Empirical Investigations}, author = {Merton, Robert K.}, @@ -2048,6 +2121,7 @@ @book{nuijten2023 url = {https://cran.r-project.org/web/packages/statcheck/} } + @book{ocarroll2017, title = {Providing researchers with the skills and competencies they need to practise Open Science}, author = {{O'Carroll}, Conor and Hyllseth, Berit and Berg, Rinske van den and Kohl, Ulrike and Kamerlin, Caroline Lynn and Brennan, Niamh and {O{\textquoteright}Neill}, Gareth}, @@ -2094,6 +2168,7 @@ @techreport{oneill2022 langid = {eng} } + @misc{oneill2022_data, author = {Gareth O'Neill and Stefania Martziou}, howpublished = {Zenodo}, @@ -2161,7 +2236,6 @@ @article{opitz2017 doi = {10.1080/13803611.2017.1338586} } - @article{osborne2023, title = {Science education in an age of misinformation}, author = {Osborne, Jonathan and Pimentel, Daniel}, @@ -2173,7 +2247,6 @@ @article{osborne2023 doi = {10.1002/sce.21790} } - @article{ozolinciute2022, title = {Guidelines for research ethics and research integrity in citizen science}, author = {{Ozolin{\v{c}}i{\={u}}t{\.{e}}}, {Egl{\.{e}}} and {Bülow}, William and Bjelobaba, Sonja and {Gai{\v{z}}auskait{\.{e}}}, Inga and {Krásni{\v{c}}an}, Veronika and {Dlabolová}, Dita and {Umbrasait{\.{e}}}, Julija}, @@ -2557,6 +2630,7 @@ @techreport{sparceurope2019 url = {https://sparceurope.org/wp-content/uploads/dlm_uploads/2019/04/SPARC-Europe_Brief_ODEfficiency.pdf} } + @article{squicciarini2021, title = {Demand for AI skills in jobs: Evidence from online job postings}, author = {Squicciarini, Mariagrazia and Nachtigall, Heike}, @@ -2566,6 +2640,7 @@ @article{squicciarini2021 note = {Publisher: OECD} } + @inproceedings{stavropoulos2023, title = {Empowering Knowledge Discovery from Scientific Literature: A novel approach to Research Artifact Analysis}, author = {Stavropoulos, Petros and Lyris, Ioannis and Manola, Natalia and Grypari, Ioanna and Papageorgiou, Harris}, @@ -2575,6 +2650,7 @@ @inproceedings{stavropoulos2023 url = {https://aclanthology.org/2023.nlposs-1.5/} } + @article{stroebe2019, title = {What Can We Learn from Many Labs Replications?}, author = {Stroebe, Wolfgang}, @@ -2590,6 +2666,7 @@ @article{stroebe2019 langid = {en} } + @book{suber2012, title = {Open access}, author = {Suber, Peter}, @@ -2599,6 +2676,7 @@ @book{suber2012 url = {https://library.oapen.org/handle/20.500.12657/26065} } + @article{sugimoto2011, title = {Academic genealogy as an indicator of interdisciplinarity: An examination of dissertation networks in Library and Information Science}, author = {Sugimoto, Cassidy R. and Ni, Chaoqun and Russell, Terrell G. and Bychowski, Brenna}, @@ -2612,6 +2690,7 @@ @article{sugimoto2011 url = {https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.21568} } + @techreport{sveinsdottir2021, title = {An Analysis of Open Science Policies in Europe, v7}, author = {Sveinsdottir, Thordis and Davidson, Joy and Proudman, Vanessa}, @@ -2638,6 +2717,7 @@ @article{szomszor2022 } + @book{tashakkori2021, title = {Foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences}, author = {Tashakkori, Abbas and Johnson, R. Burke and Teddlie, Charles}, @@ -2664,6 +2744,7 @@ @article{tattersall2018 } + @article{tennant2016, title = {The academic, economic and societal impacts of Open Access: an evidence-based review}, author = {Tennant, Jonathan P. and Waldner, {François} and Jacques, Damien C. and Masuzzo, Paola and Collister, Lauren B. and Hartgerink, Chris H. J.}, @@ -2689,7 +2770,6 @@ @article{tiokhin_shifting_2023 pages = {17456916231182568} } - @article{tomkins_reviewer_2017, title = {Reviewer bias in single- versus double-blind peer review}, volume = {114}, @@ -2716,8 +2796,6 @@ @misc{traag_causal_2022 year = {2022} } - - @misc{traag_metrics_2023, title = {Metrics and peer review agreement at the institutional level}, doi = {10.48550/arXiv.2006.14830}, @@ -2727,7 +2805,6 @@ @misc{traag_metrics_2023 year = {2023} } - @article{traag2021, title = {Inferring the causal effect of journals on citations}, author = {Traag, V. A.}, @@ -2743,8 +2820,6 @@ @article{traag2021 langid = {en} } - - @techreport{tripp2011, title = {The Economic Impacts of Human Genome Project}, author = {Tripp, Simon and Grueber, Martin}, @@ -2752,7 +2827,6 @@ @techreport{tripp2011 date = {2011} } - @article{valladares2021, title = {Scientific Literacy and Social Transformation}, author = {Valladares, Liliana}, @@ -2778,7 +2852,6 @@ @article{van_den_besselaar_perverse_2017 pages = {905--918} } - @article{vanvlijmen2020, title = {The Need of Industry to Go FAIR}, author = {van Vlijmen, Herman and Mons, Albert and Waalkens, Arne and Franke, Wouter and Baak, Arie and Ruiter, Gerbrand and Kirkpatrick, Christine and da Silva Santos, Luiz Olavo Bonino and Meerman, Bert and Jellema, Renger and Arts, Derk and Kersloot, Martijn and Knijnenburg, Sebastiaan and Lusher, Scott and Verbeeck, Rudi and Neefs, Jean-Marc}, @@ -3051,6 +3124,22 @@ @article{yarkoni2019 url = {https://psyarxiv.com/jqw35/} } +@article{yozwiak_data_2015, + title = {Data sharing: {Make} outbreak research open access}, + volume = {518}, + issn = {1476-4687}, + shorttitle = {Data sharing}, + doi = {10.1038/518477a}, + language = {en}, + number = {7540}, + urldate = {2024-12-19}, + journal = {Nature}, + author = {Yozwiak, Nathan L. and Schaffner, Stephen F. and Sabeti, Pardis C.}, + month = feb, + year = {2015}, + pages = {477--479} +} + @article{zahedi2017, title = {Mendeley readership as a filtering tool to identify highly cited publications}, author = {Zahedi, Zohreh and Costas, Rodrigo and Wouters, Paul}, @@ -3089,4 +3178,3 @@ @inproceedings{zeleti2014 doi = {10.1007/978-3-662-44745-1_50}, address = {Berlin, Heidelberg} } - diff --git a/sections/0_causality/open_data_cost_savings.qmd b/sections/0_causality/open_data_cost_savings.qmd index 2dc8969..b69bd45 100644 --- a/sections/0_causality/open_data_cost_savings.qmd +++ b/sections/0_causality/open_data_cost_savings.qmd @@ -30,13 +30,13 @@ History (please fill out in reverse chronological order, latest revision on top) ## Literature background -Measuring the economic impact of open science and open data has proven to be challenging. Many theoretical studies highlight the benefits of making research results public, with strong support for Open Science from economic research on technological change [@chataway2018] (Yozwiak et al., 2015; Mazzucato,2011). However, only few studies have attempted to measure the impacts of open science compared to closed science, and more robust evidence on how Open Science drives innovation and economic outcomes is needed to strengthen support and counter emerging criticisms [@karasz2024] (Ali-Khan et al., 2018). The existing literature mainly concentrates on specific sectors, particularly health, medicine, and biosciences, which receive more attention due to early regulation by funders and significant interest in clinical trial outcomes. Another important stream of literature is focused on highlighting the economic value of Open Science through personal industry experiences, though lacking precise quantitative evidence, with contributions from McManamay and Utz (2014) on fisheries, @harding2017 on medicine, Chan (2015) on the transition to an Open Science model, and @chen2017 on the role of open data in AI and machine learning applications. Although directly linking economic outcomes to open data initiatives can be challenging, with authors combining theoretical arguments and the limited quantitative evidence available at the time of their publication [@ali2022,@tennant2016,@fell2019,@wehn2021] (Arzberger, 2004), open access to findings and data is considered to lead to significant savings in access costs. By removing paywalls and subscription fees, open data allows researchers and businesses to access valuable information without incurring additional costs. A major economic benefit of lowering the cost of knowledge is the availability of an extra budget that can be reallocated for other purposes [@tennant2016]. +Measuring the economic impact of open science and open data has proven to be challenging. Many theoretical studies highlight the benefits of making research results public, with strong support for Open Science from economic research on technological change [@chataway2018,@yozwiak_data_2015,@mazzucato_entrepreneurial_2011]. However, only few studies have attempted to measure the impacts of open science compared to closed science, and more robust evidence on how Open Science drives innovation and economic outcomes is needed to strengthen support and counter emerging criticisms [@karasz2024,@ali-khan2018]. The existing literature mainly concentrates on specific sectors, particularly health, medicine, and biosciences, which receive more attention due to early regulation by funders and significant interest in clinical trial outcomes. Another important stream of literature is focused on highlighting the economic value of Open Science through personal industry experiences, though lacking precise quantitative evidence, with contributions from @mcmanamay_openaccess_2014 on fisheries, @harding2017 on medicine, @chan_cost_2015 on the transition to an Open Science model, and @chen2017 on the role of open data in AI and machine learning applications. Although directly linking economic outcomes to open data initiatives can be challenging, with authors combining theoretical arguments and the limited quantitative evidence available at the time of their publication [@ali2022,@tennant2016,@fell2019,@wehn2021,@arzberger_promoting_2004], open access to findings and data is considered to lead to significant savings in access costs. By removing paywalls and subscription fees, open data allows researchers and businesses to access valuable information without incurring additional costs. A major economic benefit of lowering the cost of knowledge is the availability of an extra budget that can be reallocated for other purposes [@tennant2016]. By eliminating barriers to data access, organisations can reduce the time spent on data collection and focus on core activities. @fell2019 notes that open science reduces the time associated with accessing new knowledge, directly contributing to enhanced research quality and productivity increases. Although the specific time savings associated with open access were not directly tested, @parsons_benefits_2011 conducted an interview-based study that supports this potential benefit. [@beagrie2014] demonstrate that data sharing and curation significantly enhance research efficiency, with labour cost savings ranging from two to over twenty times the operational costs of the data centres. While there are indirect costs associated with preparing data for sharing, such as time and expenses incurred by depositors, the benefits realised by users through improved efficiency and their willingness to pay for access far outweigh these costs [@beagrie2012]. Additionally, findings from [@beagrie2016,@beagrie2021] provide compelling evidence of the timesaving and efficiency gains associated with using European Bioinformatics Institute services (EMBL-EBI). Open data minimises redundant data collection and mitigates the "file drawer effect," where valuable findings remain unpublished and inaccessible, ultimately impeding research effectiveness [@assen2014]. According to [@sparceurope2019], making research data openly available can save up to 9% of a project's costs by preventing unnecessary data collection and facilitating the efficient reuse of existing data. Additionally, @houghton2011 estimate that access barriers to academic research in Denmark cost DKK 540 million annually. This figure is based on the average time spent (51-63 minutes) attempting to access research articles and the study highlights that delays in accessing academic research can prolong product and process development by an average of 2.2 years, resulting in significant financial losses for firms. -The broader implications of open data extend beyond mere cost and time savings. Scientific literature is widely recognised as an important source of strategic knowledge, facilitating the exploration of new ideas in industrial research and innovation, particularly for small and medium enterprises that may struggle to obtain data independently [@EU2020]. However, inefficiencies in traditional publishing models, such as delays and biases in data dissemination, can negatively impact private research productivity, as discussed by @harding2017. Open data benefits various sectors, including agriculture, the environment, forensics, and industrial biotechnology, by providing access to information that helps researchers understand their fields and build on existing work [@fell2019,@tennant2016,@tripp2011,@arshad2016] (Yozwiak et al., 2015). +The broader implications of open data extend beyond mere cost and time savings. Scientific literature is widely recognised as an important source of strategic knowledge, facilitating the exploration of new ideas in industrial research and innovation, particularly for small and medium enterprises that may struggle to obtain data independently [@EU2020]. However, inefficiencies in traditional publishing models, such as delays and biases in data dissemination, can negatively impact private research productivity, as discussed by @harding2017. Open data benefits various sectors, including agriculture, the environment, forensics, and industrial biotechnology, by providing access to information that helps researchers understand their fields and build on existing work [@fell2019,@tennant2016,@tripp2011,@arshad2016,@yozwiak_data_2015]. To fully harness the potential of open data, it is important to develop the necessary skills and capacities to manage it effectively. @zeleti2014 emphasise the need to streamline data generation processes to produce insights that inform and shape business strategies. Given that open data initiatives contribute to greater transparency and accountability, businesses that leverage open data achieve cost savings by co-creating and integrating data from multiple sources to enhance their services [@davies2016,@lindman2014]. @fell2019 suggests that adopting an open approach fosters connections and encourages collaborations that might not occur or would take longer in a closed environment. This is exemplified by the work of [@vanvlijmen2020], who show how the integration of diverse types of open data, specifically scientific, clinical, and experimental public evidence, can be achieved using advanced AI platforms like Euretos. By combining these various data sources, the platform enhances the depth of information available for analysis, enabling more accurate predictions regarding drug efficacy, as demonstrated by a machine learning model that improved prediction accuracy by 12 percentage points over previous state-of-the-art. @@ -84,7 +84,7 @@ In the [causality introduction](causal_intro/article/intro-causality.qmd) we emp [^1]: The *do-operator* is a notation used in causal inference to denote an intervention in a system. Written as , it represents setting variable to a specific value, simulating the effect of this intervention on other variables in the system while breaking any causal connections that usually determine . This approach allows us to differentiate causation from correlation, estimate causal effects, and answer hypothetical "what-if" scenarios. Through *do-calculus*, a set of rules introduced by Pearl, interventional distributions involving the *do-operator* can be converted into observational distributions. -This approach is effective in predicting how probability distributions shift under controlled changes when the causal structure is known. However, this approach depends on having an established causal graph, limiting its use for exploring causality from scratch. Critics suggest an alternative approach that allows causal discovery through experimentation without prior assumptions about mechanisms [@woodward2003]. This is especially useful in complex fields, such as social and biomedical sciences, where causal relationships are less understood. Considering this, @cunningham_causal_2021 argues that sample selection problems have been recognised long before the introduction of DAGs, with early solutions like Heckman (1979), and emphasizes that an atheoretical approach to empiricism is inadequate. He asserts that causal inference requires a deep understanding of the behavioural processes behind the phenomenon being studied, and while DAGs are useful, they cannot replace the need for theoretical knowledge in creating credible identification strategies. Thus, causal inference is not solved by simply collecting more data, but by integrating theory with empirical analysis. +This approach is effective in predicting how probability distributions shift under controlled changes when the causal structure is known. However, this approach depends on having an established causal graph, limiting its use for exploring causality from scratch. Critics suggest an alternative approach that allows causal discovery through experimentation without prior assumptions about mechanisms [@woodward2003]. This is especially useful in complex fields, such as social and biomedical sciences, where causal relationships are less understood. Considering this, @cunningham_causal_2021 argues that sample selection problems have been recognised long before the introduction of DAGs, with early solutions like @heckman1979, and emphasizes that an atheoretical approach to empiricism is inadequate. He asserts that causal inference requires a deep understanding of the behavioural processes behind the phenomenon being studied, and while DAGs are useful, they cannot replace the need for theoretical knowledge in creating credible identification strategies. Thus, causal inference is not solved by simply collecting more data, but by integrating theory with empirical analysis. ## Discussing empirical issues