From b05f6472e89bd783076c61ac02fbd09ad5052e79 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 29 Oct 2019 13:07:41 +0100 Subject: [PATCH 01/33] add initial outline for section on datalad extensions, with a focus on datalad containers --- docs/basics/101-144-intro_extensions.rst | 56 ++++++++++++++++++++++++ docs/basics/101-145-containersrun.rst | 25 +++++++++++ docs/contents.rst.inc | 12 +++++ 3 files changed, 93 insertions(+) create mode 100644 docs/basics/101-144-intro_extensions.rst create mode 100644 docs/basics/101-145-containersrun.rst diff --git a/docs/basics/101-144-intro_extensions.rst b/docs/basics/101-144-intro_extensions.rst new file mode 100644 index 000000000..949f9bd6d --- /dev/null +++ b/docs/basics/101-144-intro_extensions.rst @@ -0,0 +1,56 @@ +.. _extensions_intro: + +DataLad's extensions +-------------------- + +The commands DataLad provides cover a broad range of domain-agnostic use cases. +However, there is a vast supply of extension packages that can add +(domain-specific) functionality and new commands. + +Such extensions are shipped as proper Python packages, and are *not* included in +DataLad itself. Instead, users with the need for a particular extension can +install the extension package on top of DataLad. The installation is done with +standard Python package managers, such as :term:`pip`, and beyond installation +of the package, no additional setup is required. + +The following DataLad extensions are currently available: + +.. todo:: + + Which extensions do you want to see mentioned (and which not)? + +.. list-table:: + :widths: 50 100 + :header-rows: 1 + + * - Extension name + - Description + * - `DataLad Container `_ + - Equips DataLad's :command:`run`/:command:`rerun` functionality with + the ability to transparently execute commands in containerized + computational environments. + * - `DataLad Neuroimaging `_ + - Metadata extraction support for a range of standards common to + neuroimaging data. + * - `DataLad Metalad `_ + - Equips DataLad with an alternative command suite for metadata handling + (extraction, aggregation, reporting). + * - `DataLad Webapp `_ + - DataLad extension for exposing a REST API of selected functionality. + * - `DataLad Crawler `_ + - Datalad for crawling web resources and automated data distributions. + * - `DataLad HTCondor `_ + - Remote code execution for DataLad via HTCondor. + + +To install a DataLad extension, use + +.. code-block:: bash + + $ pip install + +such as in + +.. code-block:: bash + + $ pip install datalad-container \ No newline at end of file diff --git a/docs/basics/101-145-containersrun.rst b/docs/basics/101-145-containersrun.rst new file mode 100644 index 000000000..29a303a2e --- /dev/null +++ b/docs/basics/101-145-containersrun.rst @@ -0,0 +1,25 @@ +.. _containersrun: + +Ensure reproducibility with software containers +----------------------------------------------- + +Introductory words about datalad container. + +Something about full provenance capture, including software environment + +On the importance of software environments for reproducibility +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +- Why is it important to share software environments for computations? + +Capture and share software environment with containers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- What is a software container (maybe only Singularity?) + + +Using ``datalad containers-run`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Concrete example of ``datalad containers-run`` \ No newline at end of file diff --git a/docs/contents.rst.inc b/docs/contents.rst.inc index a301fca66..b82787c55 100644 --- a/docs/contents.rst.inc +++ b/docs/contents.rst.inc @@ -106,6 +106,18 @@ Make the most out of datasets basics/101-123-yoda basics/101-124-summary +################# +Extending DataLad +################# + +.. toctree:: + :maxdepth: 1 + :caption: An overview on DataLads extensions + + basics/101-144-intro_extensions + basics/101-145-containersrun + + ######### Use Cases ######### From 29ebc5c08d885aa1648b9dec1e8bb02da42f88d6 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 29 Oct 2019 13:17:30 +0100 Subject: [PATCH 02/33] add pip to glossary --- docs/glossary.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/glossary.rst b/docs/glossary.rst index f54a517b7..6cff18de4 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -110,6 +110,11 @@ Glossary Git-annex concept: The place where :term:`Git-annex` stores available file contents. Files that are annexed get a :term:`symlink` added to :term:`Git` that points to the file content. A different word for :term:`annex`. + pip + A Python package manager. Short for "Pip installs Python". ``pip install `` + searches the Python package index `PyPi `_ for a + package and installs it while resolving any potential dependencies. + provenance A record that describes entities and processes that were involved in producinng or influencing a digital resource. It provides a critical foundation for assessing authenticity, enables trust, From 2d46e39d0a0864581d661b1dccef52c606b38219 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 5 Nov 2019 13:19:31 +0100 Subject: [PATCH 03/33] draft outline on datalad containers-run --- docs/basics/101-145-containersrun.rst | 201 ++++++++++++++++++++++++-- 1 file changed, 188 insertions(+), 13 deletions(-) diff --git a/docs/basics/101-145-containersrun.rst b/docs/basics/101-145-containersrun.rst index 29a303a2e..5f3bc79a1 100644 --- a/docs/basics/101-145-containersrun.rst +++ b/docs/basics/101-145-containersrun.rst @@ -1,25 +1,200 @@ .. _containersrun: -Ensure reproducibility with software containers ------------------------------------------------ +Computational reproducibility with software containers +------------------------------------------------------ -Introductory words about datalad container. +Just after submitting your midterm data analysis project, you get together +with your friends. "I'm curious: So what kind of analyses did y'all carry out?" +you ask. The variety of methods and datasets the others used is huge, and +one analysis interests you in particular. Later that day, you decide to +install this particular analysis dataset to learn more about the methods used +in there. However, when you :command:`re-run` your friends analysis script, +it throws an error. Hastily, you call her -- maybe she can quickly fix her +script and resubmit the project with only minor delays. "I don't know what +you mean", you hear in return, however, "on my machine, everything works fine!" -Something about full provenance capture, including software environment +On its own, DataLad datasets can contain almost everything that is relevant to +ensure reproducibility: Data, code, human-readable analysis descriptions +(e.g., ``README.md`` files), provenance on the origin of all files +obtained from elsewhere, and machine-readable records that link generated +outputs to the commands, scripts, and data they were created from. -On the importance of software environments for reproducibility -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +This however may not be sufficient to ensure that an analysis *reproduces* +(i.e., produces the same or highly similar results), let alone *works* on a +computer different than the one it was initially composed on. This is because +the analysis does not only depend on data and code, but also the +*software environment* that it is conducted in. +A lack of information about the operating system of the computer, the precise +versions of installed software, or their configurations may +make it impossible to replicate your analysis on a different machine, or even +on your own machine once a new software update is installed. Therefore, it is +important to communicate all details about the computational environment for +an analysis as thoroughly as possible. Luckily, DataLad provides an extension +that can link computational environments to datasets, the +`datalad containers `_ +extension. -- Why is it important to share software environments for computations? +This section will give a quick overview on what containers are and +demonstrate how ``datalad-containers`` helps to capture full provenance of an +analysis by linking containers to datasets and analyses. -Capture and share software environment with containers -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. todo:: -- What is a software container (maybe only Singularity?) + - link to neuroimaging use case + - maybe mention the 10 year code challenge by ReScience -Using ``datalad containers-run`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Containers +^^^^^^^^^^ -- Concrete example of ``datalad containers-run`` \ No newline at end of file +To put it simple, computational containers are cut-down virtual machines that +allow you to package all software libraries and their dependencies (all in the +precise version your analysis requires) into a bundle you can share with +others. On your own and others machines, the container constitutes a secluded +software environment that + +- contains the exact software environment that you specified, ready to run + analysis in +- does not effect any software outside of the container + +Unlike virtual machines, software containers do not have their own operating +system. Instead, they use basic services of the underlying operating system +of the computer they run on (in a read-only fashion). This makes them +lightweight and portable. By sharing software environments with containers, +others (and also yourself) have easy access to the correct software +without the need to modify the software environment of the machine the +container runs on. Thus, containers are ideal to encapsulate the software +environment and share it together with the analysis code and data to ensure +computational reproducibility of your analyses. + +There are a number of different tools to create and use containers, with +`Docker `_ being one of the most well-known of them. +While being a powerful tool, it can not be used on high performance computing +(HPC) infrastructure [#f1]_. An alternative is `Singularity `_. +Both of these tools share core terminology: + +**Image** + A template or "recipe" from which containers are build. It lists all + required components of the computational environment, and is made by a + human user. If you want to create your own container, you start by writing + an Image file, but you can also *pull* a publicly shared image from the + *Hub* of the tool you are using. Based on images, containers are *build*. + +**Container** + The instance that you can actually use for your computations, *build* from + an image. + +**Hub** + A storage resource to share and consume images. There is + `Singularity-Hub `_ and + `Docker-Hub `_. + +Note that as of now, the ``datalad-containers`` extensions only supports +Singularity images, but support for Docker is being actively developed. +Singularity is however very compatible with Docker -- you can, for example, use +Docker images as a basis for Singularity images, or run Docker images with +Singularity (even without having Docker installed). Note further that in order +to use Singularity containers, you have to +`install `_ the software +singularity. + +Using ``datalad containers`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +One core feature of the ``datalad containers`` extension is that it registers +computational containers to a dataset. This is done with the +:command:`datalad containers-add` command. The command takes an arbitrary +name to give to the container, and a path or url to a container image. + +Let's see this in action for the ``midterm_analysis`` dataset. + +.. todo:: + + based on analysis in section on Python, build a singularity image. + The recipe can be displayed (added to the dataset as a how to), but the + container should be build and pulled from shub to show the workflow + +.. runrecord:: _examples/DL-101-145-101 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + +This command downloaded the container image from singularity hub, added it to +the ``midterm_project`` dataset, and recorded basic information on the +container under its name "" in the dataset's configuration at +``.datalad/config``. + +.. findoutmore:: What has been added to .datalad/config? + + .. runrecord:: _examples/DL-101-145-102 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ cat .datalad/config + + This recorded the images origin on Singularity-Hub, the location of the + image in the dataset, and + + .. todo:: + + what exactly is ``cmdexec = singularity exec {img} {cmd}``? + + Note that the image is saved under ``.datalad/environments`` and the + configuration is done in ``.datalad/config`` -- as these files are version + controlled and shared with together with a dataset, your software + container and the information where it can be re-obtained from are linked + to your dataset. + + This is how the ``containers-add`` command is recorded in your history: + + .. runrecord:: _examples/DL-101-145-103 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ git log -n 1 -p + +Once a container is registered, arbitrary commands can be executed inside of +it, i.e., in the precise software environment the container encapsulates. All it +needs for this it to swap the :command:`datalad run` command introduced in +section :ref:`run` with the :command:`datalad containers-run` command. + +.. todo:: + + re-do a previous (yet) to write run command for a data analysis in + ``midterm_project``. + + +Note: If your dataset contains more than one container, you will also need to +specify the name of the container you want to use in your command. The +complete command's structure looks like this:: + + $ datalad containers-run --name [--input ...] [--output ...] + +.. findoutmore:: How can I list available containers or remove them? + + The command :command:`datalad containers-list` will list all containers in + the current dataset: + + .. runrecord:: _examples/DL-101-145-110 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + The command :command:`datalad containers-remove` will remove a container + from the dataset, if there exists a container with name given to the + command. Note that this will remove not only the image from the dataset, + but also the configuration for it in ``.datalad/config``. + +.. todo:: + + take care of demonstrating more peculiarities of working and modifying a + subdataset from the perspective of the superdataset! + +.. rubric:: Footnotes + +.. [#f1] The main reason why Docker is not deployed on HPC systems is because + it grants users "`superuser privileges `_". + On multi-user systems such as HPC, users should not have those + privileges, as it would enable them to temper with other's or shared + data and resources, posing a severe security threat. \ No newline at end of file From 7256523734970bf44dcdd017e012a992b38a3fa6 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 5 Nov 2019 13:19:31 +0100 Subject: [PATCH 04/33] draft outline on datalad containers-run --- docs/basics/101-145-containersrun.rst | 201 ++++++++++++++++++++++++-- 1 file changed, 188 insertions(+), 13 deletions(-) diff --git a/docs/basics/101-145-containersrun.rst b/docs/basics/101-145-containersrun.rst index 29a303a2e..a1e9306c2 100644 --- a/docs/basics/101-145-containersrun.rst +++ b/docs/basics/101-145-containersrun.rst @@ -1,25 +1,200 @@ .. _containersrun: -Ensure reproducibility with software containers ------------------------------------------------ +Computational reproducibility with software containers +------------------------------------------------------ -Introductory words about datalad container. +Just after submitting your midterm data analysis project, you get together +with your friends. "I'm curious: So what kind of analyses did y'all carry out?" +you ask. The variety of methods and datasets the others used is huge, and +one analysis interests you in particular. Later that day, you decide to +install this particular analysis dataset to learn more about the methods used +in there. However, when you :command:`re-run` your friends analysis script, +it throws an error. Hastily, you call her -- maybe she can quickly fix her +script and resubmit the project with only minor delays. "I don't know what +you mean", you hear in return, "on my machine, everything works fine!" -Something about full provenance capture, including software environment +On its own, DataLad datasets can contain almost everything that is relevant to +ensure reproducibility: Data, code, human-readable analysis descriptions +(e.g., ``README.md`` files), provenance on the origin of all files +obtained from elsewhere, and machine-readable records that link generated +outputs to the commands, scripts, and data they were created from. -On the importance of software environments for reproducibility -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +This however may not be sufficient to ensure that an analysis *reproduces* +(i.e., produces the same or highly similar results), let alone *works* on a +computer different than the one it was initially composed on. This is because +the analysis does not only depend on data and code, but also the +*software environment* that it is conducted in. +A lack of information about the operating system of the computer, the precise +versions of installed software, or their configurations may +make it impossible to replicate your analysis on a different machine, or even +on your own machine once a new software update is installed. Therefore, it is +important to communicate all details about the computational environment for +an analysis as thoroughly as possible. Luckily, DataLad provides an extension +that can link computational environments to datasets, the +`datalad containers `_ +extension. -- Why is it important to share software environments for computations? +This section will give a quick overview on what containers are and +demonstrate how ``datalad-containers`` helps to capture full provenance of an +analysis by linking containers to datasets and analyses. -Capture and share software environment with containers -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. todo:: -- What is a software container (maybe only Singularity?) + - link to neuroimaging use case + - maybe mention the 10 year code challenge by ReScience -Using ``datalad containers-run`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Containers +^^^^^^^^^^ -- Concrete example of ``datalad containers-run`` \ No newline at end of file +To put it simple, computational containers are cut-down virtual machines that +allow you to package all software libraries and their dependencies (all in the +precise version your analysis requires) into a bundle you can share with +others. On your own and others machines, the container constitutes a secluded +software environment that + +- contains the exact software environment that you specified, ready to run + analysis in +- does not effect any software outside of the container + +Unlike virtual machines, software containers do not have their own operating +system. Instead, they use basic services of the underlying operating system +of the computer they run on (in a read-only fashion). This makes them +lightweight and portable. By sharing software environments with containers, +others (and also yourself) have easy access to the correct software +without the need to modify the software environment of the machine the +container runs on. Thus, containers are ideal to encapsulate the software +environment and share it together with the analysis code and data to ensure +computational reproducibility of your analyses. + +There are a number of different tools to create and use containers, with +`Docker `_ being one of the most well-known of them. +While being a powerful tool, it can not be used on high performance computing +(HPC) infrastructure [#f1]_. An alternative is `Singularity `_. +Both of these tools share core terminology: + +**Image** + A template or "recipe" from which containers are build. It lists all + required components of the computational environment, and is made by a + human user. If you want to create your own container, you start by writing + an Image file, but you can also *pull* a publicly shared image from the + *Hub* of the tool you are using. Based on images, containers are *build*. + +**Container** + The instance that you can actually use for your computations, *build* from + an image. + +**Hub** + A storage resource to share and consume images. There is + `Singularity-Hub `_ and + `Docker-Hub `_. + +Note that as of now, the ``datalad-containers`` extensions only supports +Singularity images, but support for Docker is being actively developed. +Singularity is however very compatible with Docker -- you can, for example, use +Docker images as a basis for Singularity images, or run Docker images with +Singularity (even without having Docker installed). Note further that in order +to use Singularity containers, you have to +`install `_ the software +singularity. + +Using ``datalad containers`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +One core feature of the ``datalad containers`` extension is that it registers +computational containers to a dataset. This is done with the +:command:`datalad containers-add` command. The command takes an arbitrary +name to give to the container, and a path or url to a container image. + +Let's see this in action for the ``midterm_analysis`` dataset. + +.. todo:: + + based on analysis in section on Python, build a singularity image. + The recipe can be displayed (added to the dataset as a how to), but the + container should be build and pulled from shub to show the workflow + +.. runrecord:: _examples/DL-101-145-101 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + +This command downloaded the container image from singularity hub, added it to +the ``midterm_project`` dataset, and recorded basic information on the +container under its name "" in the dataset's configuration at +``.datalad/config``. + +.. findoutmore:: What has been added to .datalad/config? + + .. runrecord:: _examples/DL-101-145-102 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ cat .datalad/config + + This recorded the images origin on Singularity-Hub, the location of the + image in the dataset, and + + .. todo:: + + what exactly is ``cmdexec = singularity exec {img} {cmd}``? + + Note that the image is saved under ``.datalad/environments`` and the + configuration is done in ``.datalad/config`` -- as these files are version + controlled and shared with together with a dataset, your software + container and the information where it can be re-obtained from are linked + to your dataset. + + This is how the ``containers-add`` command is recorded in your history: + + .. runrecord:: _examples/DL-101-145-103 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ git log -n 1 -p + +Once a container is registered, arbitrary commands can be executed inside of +it, i.e., in the precise software environment the container encapsulates. All it +needs for this it to swap the :command:`datalad run` command introduced in +section :ref:`run` with the :command:`datalad containers-run` command. + +.. todo:: + + re-do a previous (yet) to write run command for a data analysis in + ``midterm_project``. + + +Note: If your dataset contains more than one container, you will also need to +specify the name of the container you want to use in your command. The +complete command's structure looks like this:: + + $ datalad containers-run --name [--input ...] [--output ...] + +.. findoutmore:: How can I list available containers or remove them? + + The command :command:`datalad containers-list` will list all containers in + the current dataset: + + .. runrecord:: _examples/DL-101-145-110 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + The command :command:`datalad containers-remove` will remove a container + from the dataset, if there exists a container with name given to the + command. Note that this will remove not only the image from the dataset, + but also the configuration for it in ``.datalad/config``. + +.. todo:: + + take care of demonstrating more peculiarities of working and modifying a + subdataset from the perspective of the superdataset! + +.. rubric:: Footnotes + +.. [#f1] The main reason why Docker is not deployed on HPC systems is because + it grants users "`superuser privileges `_". + On multi-user systems such as HPC, users should not have those + privileges, as it would enable them to temper with other's or shared + data and resources, posing a severe security threat. \ No newline at end of file From ebf6a32a71dfd405bebb5b6e95540abdaa3f9436 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Mon, 11 Nov 2019 16:48:12 +0100 Subject: [PATCH 05/33] add container information --- docs/basics/101-145-containersrun.rst | 63 +++++++++++++++++++-------- 1 file changed, 46 insertions(+), 17 deletions(-) diff --git a/docs/basics/101-145-containersrun.rst b/docs/basics/101-145-containersrun.rst index a1e9306c2..5fa58fa88 100644 --- a/docs/basics/101-145-containersrun.rst +++ b/docs/basics/101-145-containersrun.rst @@ -108,22 +108,49 @@ computational containers to a dataset. This is done with the :command:`datalad containers-add` command. The command takes an arbitrary name to give to the container, and a path or url to a container image. -Let's see this in action for the ``midterm_analysis`` dataset. - -.. todo:: - - based on analysis in section on Python, build a singularity image. - The recipe can be displayed (added to the dataset as a how to), but the - container should be build and pulled from shub to show the workflow +Let's see this in action for the ``midterm_analysis`` dataset by rerunning +the analysis you did for the midterm project within a Singularity container. +For this we will pull a container from Singularity hub. This container has +been build for the handbook, and it contains the relevant Python setup for +the analysis. If you're curious how to create a Singularity image, the hidden +section below has some pointers: + +.. findoutmore:: How to make a Singularity image + + A singularity containers are build from definition files, often + called "recipes". The + `singularity documentation `_ + has its own tutorial. One other way to build one is with + `Neurodocker `_. This + command-line program can help you generate custom Singularity recipes (and + also Dockerfiles, from which Docker images are build). A wonderful tutorial + on how to use Neurodocker is + `this introduction `_ + by Michael Notter. + + Once a recipe exists, the command + + .. code-block:: bash + + sudo singularity build + + will build a container (callen ````) from the recipe. Alternatively, + ``Singularity Hub `_ integrates with Github + and builds containers from recipes pushed to repositories on Github. + `The docs `_ can + give you an easy set of instructions for this. .. runrecord:: _examples/DL-101-145-101 :language: console :workdir: dl-101/DataLad-101/midterm_project + # we are in the midterm_project subdataset + $ datalad containers-add python shub://adswa/resources:1 -This command downloaded the container image from singularity hub, added it to + +This command downloaded the container from Singularity Hub, added it to the ``midterm_project`` dataset, and recorded basic information on the -container under its name "" in the dataset's configuration at +container under its name "python" in the dataset's configuration at ``.datalad/config``. .. findoutmore:: What has been added to .datalad/config? @@ -160,15 +187,20 @@ it, i.e., in the precise software environment the container encapsulates. All it needs for this it to swap the :command:`datalad run` command introduced in section :ref:`run` with the :command:`datalad containers-run` command. +.. runrecord:: _examples/DL-101-145-104 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + :realcommand: echo "datalad containers-run -m "rerun analysis in container" \--container-name python \ datalad rerun $(git rev-parse HEAD~2)" && datalad containers-run -m "rerun analysis in container" \--container-name python \ datalad rerun $(git rev-parse HEAD~2) + .. todo:: re-do a previous (yet) to write run command for a data analysis in ``midterm_project``. -Note: If your dataset contains more than one container, you will also need to -specify the name of the container you want to use in your command. The -complete command's structure looks like this:: +Note: The ``--container-name`` flag is optional at this point. Only if your dataset +contains more than one container, you will *need* to specify the name of the container +you want to use in your command. The complete command's structure looks like this:: $ datalad containers-run --name [--input ...] [--output ...] @@ -181,16 +213,13 @@ complete command's structure looks like this:: :language: console :workdir: dl-101/DataLad-101/midterm_project + $ datalad containers-list + The command :command:`datalad containers-remove` will remove a container from the dataset, if there exists a container with name given to the command. Note that this will remove not only the image from the dataset, but also the configuration for it in ``.datalad/config``. -.. todo:: - - take care of demonstrating more peculiarities of working and modifying a - subdataset from the perspective of the superdataset! - .. rubric:: Footnotes .. [#f1] The main reason why Docker is not deployed on HPC systems is because From 7c3da4d75f9ea773577d8e6543151829159d1d5a Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Mon, 11 Nov 2019 17:34:56 +0100 Subject: [PATCH 06/33] fix command calls --- docs/basics/101-145-containersrun.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/basics/101-145-containersrun.rst b/docs/basics/101-145-containersrun.rst index 5fa58fa88..4fde3a547 100644 --- a/docs/basics/101-145-containersrun.rst +++ b/docs/basics/101-145-containersrun.rst @@ -134,8 +134,8 @@ section below has some pointers: sudo singularity build - will build a container (callen ````) from the recipe. Alternatively, - ``Singularity Hub `_ integrates with Github + will build a container (called ````) from the recipe. Alternatively, + `Singularity Hub `_ integrates with Github and builds containers from recipes pushed to repositories on Github. `The docs `_ can give you an easy set of instructions for this. @@ -145,7 +145,7 @@ section below has some pointers: :workdir: dl-101/DataLad-101/midterm_project # we are in the midterm_project subdataset - $ datalad containers-add python shub://adswa/resources:1 + $ datalad containers-add python --url shub://adswa/resources:1 This command downloaded the container from Singularity Hub, added it to @@ -190,7 +190,7 @@ section :ref:`run` with the :command:`datalad containers-run` command. .. runrecord:: _examples/DL-101-145-104 :language: console :workdir: dl-101/DataLad-101/midterm_project - :realcommand: echo "datalad containers-run -m "rerun analysis in container" \--container-name python \ datalad rerun $(git rev-parse HEAD~2)" && datalad containers-run -m "rerun analysis in container" \--container-name python \ datalad rerun $(git rev-parse HEAD~2) + :realcommand: echo "datalad containers-run -m "rerun analysis in container" --container-name python datalad rerun $(git rev-parse HEAD~3)" && datalad containers-run -m "rerun analysis in container" --container-name python datalad rerun $(git rev-parse HEAD~3) .. todo:: From 68e1f9a3b54a07ee2b2c666bd7976d0bab47c88a Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Mon, 11 Nov 2019 18:23:40 +0100 Subject: [PATCH 07/33] add examples --- docs/basics/_examples/DL-101-145-101 | 9 +++++++++ docs/basics/_examples/DL-101-145-102 | 7 +++++++ docs/basics/_examples/DL-101-145-103 | 26 ++++++++++++++++++++++++++ docs/basics/_examples/DL-101-145-104 | 23 +++++++++++++++++++++++ docs/basics/_examples/DL-101-145-110 | 2 ++ 5 files changed, 67 insertions(+) create mode 100644 docs/basics/_examples/DL-101-145-101 create mode 100644 docs/basics/_examples/DL-101-145-102 create mode 100644 docs/basics/_examples/DL-101-145-103 create mode 100644 docs/basics/_examples/DL-101-145-104 create mode 100644 docs/basics/_examples/DL-101-145-110 diff --git a/docs/basics/_examples/DL-101-145-101 b/docs/basics/_examples/DL-101-145-101 new file mode 100644 index 000000000..d46fa9cc9 --- /dev/null +++ b/docs/basics/_examples/DL-101-145-101 @@ -0,0 +1,9 @@ +# we are in the midterm_project subdataset +$ datalad containers-add python --url shub://adswa/resources:1 +add(ok): .datalad/config (file) +save(ok): . (dataset) +containers_add(ok): /home/me/dl-101/DataLad-101/midterm_project/.datalad/environments/python/image (file) +action summary: + add (ok: 1) + containers_add (ok: 1) + save (ok: 1) diff --git a/docs/basics/_examples/DL-101-145-102 b/docs/basics/_examples/DL-101-145-102 new file mode 100644 index 000000000..d24f71716 --- /dev/null +++ b/docs/basics/_examples/DL-101-145-102 @@ -0,0 +1,7 @@ +$ cat .datalad/config +[datalad "dataset"] + id = 942371e6-04a0-11ea-99ca-d0c637c523bc +[datalad "containers.python"] + updateurl = shub://adswa/resources:1 + image = .datalad/environments/python/image + cmdexec = singularity exec {img} {cmd} diff --git a/docs/basics/_examples/DL-101-145-103 b/docs/basics/_examples/DL-101-145-103 new file mode 100644 index 000000000..f82332308 --- /dev/null +++ b/docs/basics/_examples/DL-101-145-103 @@ -0,0 +1,26 @@ +$ git log -n 1 -p +commit 3cdab542d2749d3812d0d71ece030acabfc9f6d9 +Author: Elena Piscopia +Date: Mon Nov 11 17:33:49 2019 +0100 + + [DATALAD] Configure containerized environment 'python' + +diff --git a/.datalad/config b/.datalad/config +index 71b5f13..0500889 100644 +--- a/.datalad/config ++++ b/.datalad/config +@@ -1,2 +1,6 @@ + [datalad "dataset"] + id = 942371e6-04a0-11ea-99ca-d0c637c523bc ++[datalad "containers.python"] ++ updateurl = shub://adswa/resources:1 ++ image = .datalad/environments/python/image ++ cmdexec = singularity exec {img} {cmd} +diff --git a/.datalad/environments/python/image b/.datalad/environments/python/image +new file mode 120000 +index 0000000..800282a +--- /dev/null ++++ b/.datalad/environments/python/image +@@ -0,0 +1 @@ ++../../../.git/annex/objects/zJ/8f/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10 +\ No newline at end of file diff --git a/docs/basics/_examples/DL-101-145-104 b/docs/basics/_examples/DL-101-145-104 new file mode 100644 index 000000000..457d879f3 --- /dev/null +++ b/docs/basics/_examples/DL-101-145-104 @@ -0,0 +1,23 @@ + +datalad containers-run -m rerun analysis in container --container-name python datalad rerun a39d7907214b1669612281a23a50656288c87735 +[INFO] Making sure inputs are available (this may take some time) +[INFO] == Command start (output follows) ===== +[INFO] Making sure inputs are available (this may take some time) +[INFO] == Command start (output follows) ===== +/usr/lib/python3/dist-packages/sklearn/externals/joblib.py:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses + import imp +[INFO] == Command exit (modification check follows) ===== +unlock(ok): output/prediction_report.csv (file) +unlock(ok): output/pairwise_relationships.png (file) +add(ok): output/pairwise_relationships.png (file) +add(ok): output/prediction_report.csv (file) +save(ok): /home/me/dl-101/DataLad-101/midterm_project (dataset) +action summary: + add (ok: 2) + get (notneeded: 2) + save (ok: 1) + unlock (ok: 2) +[INFO] == Command exit (modification check follows) ===== +action summary: + get (notneeded: 1) + save (notneeded: 2) diff --git a/docs/basics/_examples/DL-101-145-110 b/docs/basics/_examples/DL-101-145-110 new file mode 100644 index 000000000..ddba701f3 --- /dev/null +++ b/docs/basics/_examples/DL-101-145-110 @@ -0,0 +1,2 @@ +$ datalad containers-list +python -> .datalad/environments/python/image From 65e7559c9ea18234f2f7a9cd285303c94225f1ad Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 12 Nov 2019 11:00:37 +0100 Subject: [PATCH 08/33] a few language tweaks --- docs/basics/101-144-intro_extensions.rst | 4 ++- docs/basics/101-145-containersrun.rst | 38 +++++++++++++++++------- 2 files changed, 30 insertions(+), 12 deletions(-) diff --git a/docs/basics/101-144-intro_extensions.rst b/docs/basics/101-144-intro_extensions.rst index 949f9bd6d..588f45943 100644 --- a/docs/basics/101-144-intro_extensions.rst +++ b/docs/basics/101-144-intro_extensions.rst @@ -53,4 +53,6 @@ such as in .. code-block:: bash - $ pip install datalad-container \ No newline at end of file + $ pip install datalad-container + +Afterwards, the new DataLad functionality the extension provides is readily available. \ No newline at end of file diff --git a/docs/basics/101-145-containersrun.rst b/docs/basics/101-145-containersrun.rst index 043343924..c04c26bcc 100644 --- a/docs/basics/101-145-containersrun.rst +++ b/docs/basics/101-145-containersrun.rst @@ -117,13 +117,15 @@ section below has some pointers: .. findoutmore:: How to make a Singularity image - A singularity containers are build from definition files, often - called "recipes". The + Singularity containers are build from Image files, often + called "recipes", that hold a "definition" of the software container and its + contents and components. The `singularity documentation `_ - has its own tutorial. One other way to build one is with + has its own tutorial on how to build such Images from scratch. + An alternative to writing the Image file by hand is to use `Neurodocker `_. This command-line program can help you generate custom Singularity recipes (and - also Dockerfiles, from which Docker images are build). A wonderful tutorial + also ``Dockerfiles``, from which Docker images are build). A wonderful tutorial on how to use Neurodocker is `this introduction `_ by Michael Notter. @@ -136,9 +138,9 @@ section below has some pointers: will build a container (called ````) from the recipe. Alternatively, `Singularity Hub `_ integrates with Github - and builds containers from recipes pushed to repositories on Github. + and builds containers from Images pushed to repositories on Github. `The docs `_ can - give you an easy set of instructions for this. + give you an easy set of instructions on how to do this. .. runrecord:: _examples/DL-101-145-101 :language: console @@ -162,11 +164,17 @@ container under its name "python" in the dataset's configuration at $ cat .datalad/config This recorded the images origin on Singularity-Hub, the location of the - image in the dataset, and + image in the dataset under ``.datalad/environments//image``, and it + specifies the way in which the container should be used: The line - .. todo:: + .. code-block:: bash + + cmdexec = singularity exec {img} {cmd} - what exactly is ``cmdexec = singularity exec {img} {cmd}``? + can be read as: "If this container is used, take the ``cmd`` (what you wrap in a + :command:`datalad containers-run` command) and plug it into a + :command:`singularity exec` command. The mode of calling Singularity, + namely ``exec``, means that the command will be executed inside of the container. Note that the image is saved under ``.datalad/environments`` and the configuration is done in ``.datalad/config`` -- as these files are version @@ -192,8 +200,9 @@ section :ref:`run` with the :command:`datalad containers-run` command. :workdir: dl-101/DataLad-101/midterm_project :realcommand: echo "datalad containers-run -m "rerun analysis in container" --container-name python datalad rerun $(git rev-parse HEAD~3)" && datalad containers-run -m "rerun analysis in container" --container-name python datalad rerun $(git rev-parse HEAD~3) -Note: The ``--container-name`` flag is optional at this point. Only if your dataset -contains more than one container, you will *need* to specify the name of the container +Note that at this point, the ``--container-name`` flag is *optional* because there +is only a single container registered to the dataset. But if your dataset +contains more than one container you will *need* to specify the name of the container you want to use in your command. The complete command's structure looks like this:: $ datalad containers-run --name [--input ...] [--output ...] @@ -215,6 +224,13 @@ you want to use in your command. The complete command's structure looks like thi command. Note that this will remove not only the image from the dataset, but also the configuration for it in ``.datalad/config``. +Software containers, the ``datalad-containers``, and DataLad thus work well together +to make your analysis completely reproducible -- by not only linking code, data, +and outputs, but also the software environment of an analysis. And this does not +only benefit your future self, but also whomever you share your dataset with, as +the information about the container is shared together with the dataset. How cool +is that? + .. rubric:: Footnotes .. [#f1] The main reason why Docker is not deployed on HPC systems is because From f88b5556e9178653d97ca87ad309d8dbcfc365bf Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 11:38:28 +0100 Subject: [PATCH 09/33] restructure toc tree, put containers-run closer to yoda project --- ...inersrun.rst => 101-133-containersrun.rst} | 0 docs/contents.rst.inc | 21 ++++++------------- 2 files changed, 6 insertions(+), 15 deletions(-) rename docs/basics/{101-145-containersrun.rst => 101-133-containersrun.rst} (100%) diff --git a/docs/basics/101-145-containersrun.rst b/docs/basics/101-133-containersrun.rst similarity index 100% rename from docs/basics/101-145-containersrun.rst rename to docs/basics/101-133-containersrun.rst diff --git a/docs/contents.rst.inc b/docs/contents.rst.inc index dd1cec874..b9ee8f754 100644 --- a/docs/contents.rst.inc +++ b/docs/contents.rst.inc @@ -89,7 +89,7 @@ Make the most out of datasets .. toctree:: :maxdepth: 1 - :caption: Organizational principles and best practices for data analyses + :caption: Organizational principles and best practices for reproducible data analyses basics/101-126-intro basics/101-127-yoda @@ -97,15 +97,16 @@ Make the most out of datasets basics/101-128-summary_yoda -########################### -Dataset nesting -- Advanced -########################### +################ +One step further +################ .. toctree:: :maxdepth: 1 - :caption: Manipulating subdatasets and its impact on the superdataset(s) + :caption: Advanced nesting and "extended" reproducibility basics/101-132-advancednesting + basics/101-133-containersrun basics/101-132-gitC @@ -133,17 +134,7 @@ Further options :caption: Small pieces of advice and helpful additional options basics/101-179-gitignore - -################# -Extending DataLad -################# - -.. toctree:: - :maxdepth: 1 - :caption: An overview on DataLads extensions - basics/101-144-intro_extensions - basics/101-145-containersrun ######### From 10c8ecc515025e982f1fd4932ff82a927ea5940d Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 11:38:59 +0100 Subject: [PATCH 10/33] provide ref to yoda project section --- docs/basics/101-130-yodaproject.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/basics/101-130-yodaproject.rst b/docs/basics/101-130-yodaproject.rst index 65cbfc009..53e912e62 100644 --- a/docs/basics/101-130-yodaproject.rst +++ b/docs/basics/101-130-yodaproject.rst @@ -1,3 +1,5 @@ +.. _yoda_project: + YODA-compliant data analysis projects ------------------------------------- From c6e0a8ab9e2b3f8fab4b78aba8fcbc0ffc7475ef Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 11:39:42 +0100 Subject: [PATCH 11/33] some language tweaks --- docs/basics/101-132-advancednesting.rst | 7 +++-- docs/basics/101-133-containersrun.rst | 42 ++++++++++++++++--------- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/docs/basics/101-132-advancednesting.rst b/docs/basics/101-132-advancednesting.rst index 88bb20708..33c005664 100644 --- a/docs/basics/101-132-advancednesting.rst +++ b/docs/basics/101-132-advancednesting.rst @@ -39,8 +39,11 @@ evolved. Let's query the superdataset what it thinks about this. $ datalad status From the superdataset's perspective, the subdataset appears as being -"modified". Note that these modifications of the subdataset are not -automatically recorded to the superdataset! This makes sense -- after all it +"modified". Note how it is not individual files that show up as "modified", but +indeed the complete subdataset as a single entity. + +What this shows you is that the modifications of the subdataset you performed are not +automatically recorded to the superdataset. This makes sense -- after all it should be up to you to decide whether you want record something or not --, but it is worth repeating: If you modify a subdataset, you will need to save this *in the superdataset* in order to have a clean superdataset status. diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index c04c26bcc..aefd905bc 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -11,7 +11,9 @@ install this particular analysis dataset to learn more about the methods used in there. However, when you :command:`re-run` your friends analysis script, it throws an error. Hastily, you call her -- maybe she can quickly fix her script and resubmit the project with only minor delays. "I don't know what -you mean", you hear in return, "on my machine, everything works fine!" +you mean", you hear in return. +"On my machine, everything works fine!" + On its own, DataLad datasets can contain almost everything that is relevant to ensure reproducibility: Data, code, human-readable analysis descriptions @@ -33,7 +35,7 @@ important to communicate all details about the computational environment for an analysis as thoroughly as possible. Luckily, DataLad provides an extension that can link computational environments to datasets, the `datalad containers `_ -extension. +extension [#f1]_. This section will give a quick overview on what containers are and demonstrate how ``datalad-containers`` helps to capture full provenance of an @@ -55,7 +57,7 @@ others. On your own and others machines, the container constitutes a secluded software environment that - contains the exact software environment that you specified, ready to run - analysis in + analyses in - does not effect any software outside of the container Unlike virtual machines, software containers do not have their own operating @@ -71,7 +73,7 @@ computational reproducibility of your analyses. There are a number of different tools to create and use containers, with `Docker `_ being one of the most well-known of them. While being a powerful tool, it can not be used on high performance computing -(HPC) infrastructure [#f1]_. An alternative is `Singularity `_. Both of these tools share core terminology: @@ -95,21 +97,27 @@ Note that as of now, the ``datalad-containers`` extensions only supports Singularity images, but support for Docker is being actively developed. Singularity is however very compatible with Docker -- you can, for example, use Docker images as a basis for Singularity images, or run Docker images with -Singularity (even without having Docker installed). Note further that in order -to use Singularity containers, you have to -`install `_ the software -singularity. +Singularity (even without having Docker installed). + +.. note:: + + In order to use Singularity containers (and thus ``datalad containers``), you have to + `install `_ the software singularity. Using ``datalad containers`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ One core feature of the ``datalad containers`` extension is that it registers computational containers to a dataset. This is done with the -:command:`datalad containers-add` command. The command takes an arbitrary -name to give to the container, and a path or url to a container image. +:command:`datalad containers-add` command. +Once a container is registered, arbitrary commands can be executed inside of +it, i.e., in the precise software environment the container encapsulates. All it +needs for this it to swap the :command:`datalad run` command introduced in +section :ref:`run` with the :command:`datalad containers-run` command. Let's see this in action for the ``midterm_analysis`` dataset by rerunning the analysis you did for the midterm project within a Singularity container. +We start by registering a container to the dataset. For this we will pull a container from Singularity hub. This container has been build for the handbook, and it contains the relevant Python setup for the analysis. If you're curious how to create a Singularity image, the hidden @@ -139,8 +147,11 @@ section below has some pointers: will build a container (called ````) from the recipe. Alternatively, `Singularity Hub `_ integrates with Github and builds containers from Images pushed to repositories on Github. - `The docs `_ can - give you an easy set of instructions on how to do this. + `The docs `_ + give you a set of instructions on how to do this. + +The :command:`datalad containers-add` command takes an arbitrary +name to give to the container, and a path or url to a container image: .. runrecord:: _examples/DL-101-145-101 :language: console @@ -163,7 +174,7 @@ container under its name "python" in the dataset's configuration at $ cat .datalad/config - This recorded the images origin on Singularity-Hub, the location of the + This recorded the image's origin on Singularity-Hub, the location of the image in the dataset under ``.datalad/environments//image``, and it specifies the way in which the container should be used: The line @@ -205,7 +216,7 @@ is only a single container registered to the dataset. But if your dataset contains more than one container you will *need* to specify the name of the container you want to use in your command. The complete command's structure looks like this:: - $ datalad containers-run --name [--input ...] [--output ...] + $ datalad containers-run --name [-m ...] [--input ...] [--output ...] .. findoutmore:: How can I list available containers or remove them? @@ -233,7 +244,8 @@ is that? .. rubric:: Footnotes -.. [#f1] The main reason why Docker is not deployed on HPC systems is because +.. [#f1] To read more about DataLad's extensions, see section :ref:`extensions_intro`. +.. [#f2] The main reason why Docker is not deployed on HPC systems is because it grants users "`superuser privileges `_". On multi-user systems such as HPC, users should not have those privileges, as it would enable them to temper with other's or shared From 59b6145d6ff5d4fbee089dc195541e0f005f86db Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 11:39:59 +0100 Subject: [PATCH 12/33] define proper containers-run command --- docs/basics/101-133-containersrun.rst | 36 ++++++++++++++++++++------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index aefd905bc..9c4ca69a9 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -201,20 +201,38 @@ container under its name "python" in the dataset's configuration at $ git log -n 1 -p -Once a container is registered, arbitrary commands can be executed inside of -it, i.e., in the precise software environment the container encapsulates. All it -needs for this it to swap the :command:`datalad run` command introduced in -section :ref:`run` with the :command:`datalad containers-run` command. +Now that we have a complete computational environment linked to the ``midterm_project`` +dataset, we can execute commands in this environment. Let us for example try to repeat +the :command:`datalad run` command from the section :ref:`yoda_project` as a +:command:`datalad containers-run` command. + +The previous ``run`` command looked like this:: + + $ datalad run -m "analyze iris data with classification analysis" \ + --input "input/iris.csv" \ + --output "prediction_report.csv" \ + --output "pairwise_relationships.png" \ + "python3 code/script.py" + +How would it look like as a ``containers-run`` command? .. runrecord:: _examples/DL-101-145-104 :language: console :workdir: dl-101/DataLad-101/midterm_project - :realcommand: echo "datalad containers-run -m "rerun analysis in container" --container-name python datalad rerun $(git rev-parse HEAD~3)" && datalad containers-run -m "rerun analysis in container" --container-name python datalad rerun $(git rev-parse HEAD~3) -Note that at this point, the ``--container-name`` flag is *optional* because there -is only a single container registered to the dataset. But if your dataset -contains more than one container you will *need* to specify the name of the container -you want to use in your command. The complete command's structure looks like this:: + $ datalad containers-run -m "rerun analysis in container" \ + --container-name python \ + --input "input/iris.csv" \ + --output "prediction_report.csv" \ + --output "pairwise_relationships.png" \ + "python3 code/script.py" + +Almost exactly like a :command:`datalad run` command! The only additional parameter +is ``container-name``. At this point, though, the ``--container-name`` +flag is even *optional* because there is only a single container registered to the dataset. +But if your dataset contains more than one container you will *need* to specify +the name of the container you want to use in your command. +The complete command's structure looks like this:: $ datalad containers-run --name [-m ...] [--input ...] [--output ...] From d175414848c4f4b12c89f1d99939e01885843569 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 12:29:14 +0100 Subject: [PATCH 13/33] update runrecord numbering --- docs/basics/101-133-containersrun.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index 9c4ca69a9..ea04116fe 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -153,7 +153,7 @@ section below has some pointers: The :command:`datalad containers-add` command takes an arbitrary name to give to the container, and a path or url to a container image: -.. runrecord:: _examples/DL-101-145-101 +.. runrecord:: _examples/DL-101-133-101 :language: console :workdir: dl-101/DataLad-101/midterm_project @@ -168,7 +168,7 @@ container under its name "python" in the dataset's configuration at .. findoutmore:: What has been added to .datalad/config? - .. runrecord:: _examples/DL-101-145-102 + .. runrecord:: _examples/DL-101-133-102 :language: console :workdir: dl-101/DataLad-101/midterm_project @@ -195,7 +195,7 @@ container under its name "python" in the dataset's configuration at This is how the ``containers-add`` command is recorded in your history: - .. runrecord:: _examples/DL-101-145-103 + .. runrecord:: _examples/DL-101-133-103 :language: console :workdir: dl-101/DataLad-101/midterm_project @@ -216,7 +216,7 @@ The previous ``run`` command looked like this:: How would it look like as a ``containers-run`` command? -.. runrecord:: _examples/DL-101-145-104 +.. runrecord:: _examples/DL-101-133-104 :language: console :workdir: dl-101/DataLad-101/midterm_project @@ -241,7 +241,7 @@ The complete command's structure looks like this:: The command :command:`datalad containers-list` will list all containers in the current dataset: - .. runrecord:: _examples/DL-101-145-110 + .. runrecord:: _examples/DL-101-133-110 :language: console :workdir: dl-101/DataLad-101/midterm_project From cb6a0ba113ec8be373c06287889d3816b4b8b67f Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 12:49:50 +0100 Subject: [PATCH 14/33] save subds state to superds --- docs/basics/101-133-containersrun.rst | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index ea04116fe..738cebdf8 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -253,6 +253,32 @@ The complete command's structure looks like this:: command. Note that this will remove not only the image from the dataset, but also the configuration for it in ``.datalad/config``. + +Here is how the history entry looks like: + +.. runrecord:: _examples/DL-101-133-111 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ git log -p -n 1 + +Because this modified the ``midterm_project`` subdirectory, we need to also save +the most recent state of the subdataset to the superdataset ``DataLad-101``. + +.. runrecord:: _examples/DL-101-133-112 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ cd ../ + $ datalad status + +.. runrecord:: _examples/DL-101-133-113 + :language: console + :workdir: dl-101/DataLad-101 + + $ datalad save -m "add container and execute analysis within container" + + Software containers, the ``datalad-containers``, and DataLad thus work well together to make your analysis completely reproducible -- by not only linking code, data, and outputs, but also the software environment of an analysis. And this does not From 732017104dd41dcc59baa256f97d6adff95a9f9b Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 12:58:37 +0100 Subject: [PATCH 15/33] rebuild all examples --- docs/basics/_examples/DL-101-101-103 | 8 ++--- docs/basics/_examples/DL-101-102-103 | 4 +-- docs/basics/_examples/DL-101-102-107 | 4 +-- docs/basics/_examples/DL-101-102-108 | 2 +- docs/basics/_examples/DL-101-102-111 | 8 ++--- docs/basics/_examples/DL-101-103-107 | 8 ++--- docs/basics/_examples/DL-101-106-101 | 36 +++++++++---------- docs/basics/_examples/DL-101-108-106 | 6 ++-- docs/basics/_examples/DL-101-108-108 | 20 +++++------ docs/basics/_examples/DL-101-108-113 | 10 +++--- docs/basics/_examples/DL-101-108-114 | 2 +- docs/basics/_examples/DL-101-108-115 | 8 ++--- docs/basics/_examples/DL-101-108-120 | 14 ++++---- docs/basics/_examples/DL-101-112-109 | 10 +++--- docs/basics/_examples/DL-101-114-101 | 42 +++++++++++----------- docs/basics/_examples/DL-101-115-103 | 6 ++-- docs/basics/_examples/DL-101-116-105 | 2 +- docs/basics/_examples/DL-101-117-101 | 2 +- docs/basics/_examples/DL-101-118-102 | 42 +++++++++++----------- docs/basics/_examples/DL-101-118-103 | 2 +- docs/basics/_examples/DL-101-121-103 | 5 ++- docs/basics/_examples/DL-101-121-112 | 53 ++++++++++++++-------------- docs/basics/_examples/DL-101-122-101 | 4 +-- docs/basics/_examples/DL-101-122-103 | 4 +-- docs/basics/_examples/DL-101-123-105 | 2 +- docs/basics/_examples/DL-101-125-106 | 3 -- docs/basics/_examples/DL-101-130-106 | 1 + docs/basics/_examples/DL-101-130-110 | 4 +-- docs/basics/_examples/DL-101-130-112 | 10 +++--- docs/basics/_examples/DL-101-130-116 | 3 +- docs/basics/_examples/DL-101-132-101 | 12 +++---- docs/basics/_examples/DL-101-132-104 | 10 +++--- docs/basics/_examples/DL-101-135-101 | 2 +- docs/basics/_examples/DL-101-135-102 | 14 ++++---- docs/basics/_examples/DL-101-136-101 | 6 ++-- docs/basics/_examples/DL-101-136-104 | 4 +-- docs/basics/_examples/DL-101-136-105 | 3 +- docs/basics/_examples/DL-101-136-108 | 3 +- docs/basics/_examples/DL-101-136-109 | 2 +- docs/basics/_examples/DL-101-136-110 | 3 +- docs/basics/_examples/DL-101-136-121 | 2 +- docs/basics/_examples/DL-101-136-122 | 2 +- docs/basics/_examples/DL-101-136-123 | 4 +-- docs/basics/_examples/DL-101-136-124 | 3 +- docs/basics/_examples/DL-101-136-132 | 4 +-- docs/basics/_examples/DL-101-136-133 | 4 +-- docs/basics/_examples/DL-101-136-134 | 2 +- docs/basics/_examples/DL-101-136-140 | 4 +-- docs/basics/_examples/DL-101-136-144 | 4 +-- docs/basics/_examples/DL-101-136-146 | 3 +- docs/basics/_examples/DL-101-136-152 | 4 +-- docs/basics/_examples/DL-101-136-156 | 2 +- docs/basics/_examples/DL-101-136-164 | 3 +- docs/basics/_examples/DL-101-136-174 | 2 +- docs/basics/_examples/DL-101-136-180 | 2 +- docs/basics/_examples/DL-101-137-101 | 30 ++++++++-------- docs/basics/_examples/DL-101-137-105 | 4 +-- docs/basics/_examples/DL-101-137-106 | 6 ++-- docs/basics/_examples/DL-101-137-107 | 3 +- docs/basics/_examples/DL-101-137-108 | 4 +-- docs/basics/_examples/DL-101-137-112 | 12 +++---- docs/basics/_examples/DL-101-137-114 | 2 +- docs/basics/_examples/DL-101-137-116 | 2 +- docs/basics/_examples/DL-101-137-117 | 6 ++-- docs/basics/_examples/DL-101-137-118 | 3 +- docs/basics/_examples/DL-101-137-119 | 4 +-- docs/basics/_examples/DL-101-137-121 | 40 ++++++++++----------- docs/basics/_examples/DL-101-137-122 | 19 ++++++---- docs/basics/_examples/DL-101-137-124 | 2 +- docs/basics/_examples/DL-101-137-127 | 3 +- docs/basics/_examples/DL-101-137-132 | 4 +-- docs/basics/_examples/DL-101-137-133 | 5 +-- docs/basics/_examples/DL-101-137-144 | 6 ++-- docs/basics/_examples/DL-101-137-146 | 14 ++++---- docs/basics/_examples/DL-101-145-101 | 9 ----- docs/basics/_examples/DL-101-145-102 | 7 ---- docs/basics/_examples/DL-101-145-103 | 26 -------------- docs/basics/_examples/DL-101-145-104 | 23 ------------ docs/basics/_examples/DL-101-145-110 | 2 -- docs/intro/_examples/how-to-1 | 4 +-- docs/intro/_examples/how-to-2 | 2 +- 81 files changed, 302 insertions(+), 369 deletions(-) delete mode 100644 docs/basics/_examples/DL-101-125-106 delete mode 100644 docs/basics/_examples/DL-101-145-101 delete mode 100644 docs/basics/_examples/DL-101-145-102 delete mode 100644 docs/basics/_examples/DL-101-145-103 delete mode 100644 docs/basics/_examples/DL-101-145-104 delete mode 100644 docs/basics/_examples/DL-101-145-110 diff --git a/docs/basics/_examples/DL-101-101-103 b/docs/basics/_examples/DL-101-101-103 index 1d6d17cd1..803ad9bd6 100644 --- a/docs/basics/_examples/DL-101-101-103 +++ b/docs/basics/_examples/DL-101-101-103 @@ -1,12 +1,12 @@ $ git log -commit 758f0af5a5d8865db0f8aac4717d8fa93771d8a4 +commit 4cc7c21a716368eda466b94630522d19abfef72f Author: Elena Piscopia -Date: Mon Nov 18 10:49:52 2019 +0100 +Date: Tue Nov 26 12:52:00 2019 +0100 Instruct annex to add text files to Git -commit 715acb08af2fdab8d901180c3d9985a5335b0584 +commit fca4a608c6f8c749a2f3f83a3da8042b61e496ec Author: Elena Piscopia -Date: Mon Nov 18 10:49:52 2019 +0100 +Date: Tue Nov 26 12:52:00 2019 +0100 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-102-103 b/docs/basics/_examples/DL-101-102-103 index 935d1f3ac..4a967d6ee 100644 --- a/docs/basics/_examples/DL-101-102-103 +++ b/docs/basics/_examples/DL-101-102-103 @@ -3,5 +3,5 @@ $ wget https://sourceforge.net/projects/linuxcommand/files/TLCL/19.01/TLCL-19.01 $ wget https://www.gitbook.com/download/pdf/book/swaroopch/byte-of-python -O byte-of-python.pdf # get back into the root of the dataset $ cd ../ -2019-11-18 10:50:31 URL:https://vorboss.dl.sourceforge.net/project/linuxcommand/TLCL/19.01/TLCL-19.01.pdf [2120211/2120211] -> "TLCL.pdf" [1] -2019-11-18 10:50:35 URL:https://legacy.gitbook.com/download/pdf/book/swaroopch/byte-of-python [4407669] -> "byte-of-python.pdf" [1] +2019-11-26 12:52:56 URL:https://netcologne.dl.sourceforge.net/project/linuxcommand/TLCL/19.01/TLCL-19.01.pdf [2120211/2120211] -> "TLCL.pdf" [1] +2019-11-26 12:52:57 URL:https://legacy.gitbook.com/download/pdf/book/swaroopch/byte-of-python [4407669] -> "byte-of-python.pdf" [1] diff --git a/docs/basics/_examples/DL-101-102-107 b/docs/basics/_examples/DL-101-102-107 index fe7591bac..41351976b 100644 --- a/docs/basics/_examples/DL-101-102-107 +++ b/docs/basics/_examples/DL-101-102-107 @@ -1,7 +1,7 @@ $ git log -p -n 1 -commit 464b1e50a588101cff01de9d0a95ad1f7cfc0457 +commit 5d5a8014fa0f87fe11cc24223094a01e082c4b9b Author: Elena Piscopia -Date: Mon Nov 18 10:50:36 2019 +0100 +Date: Tue Nov 26 12:52:58 2019 +0100 add books on Python and Unix to read later diff --git a/docs/basics/_examples/DL-101-102-108 b/docs/basics/_examples/DL-101-102-108 index 92b366f4d..bd7c064f4 100644 --- a/docs/basics/_examples/DL-101-102-108 +++ b/docs/basics/_examples/DL-101-102-108 @@ -1,4 +1,4 @@ $ cd books $ wget https://github.com/progit/progit2/releases/download/2.1.154/progit.pdf $ cd ../ -2019-11-18 10:51:14 URL:https://github-production-release-asset-2e65be.s3.amazonaws.com/15400220/57552a00-9a49-11e9-9144-d9607ed4c2db?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191118T095037Z&X-Amz-Expires=300&X-Amz-Signature=ee3c1f736354cc22e799692c8f5ffe9a7501eac269b98a4f0d915e1d08264530&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprogit.pdf&response-content-type=application%2Foctet-stream [12465653/12465653] -> "progit.pdf" [1] +2019-11-26 12:53:00 URL:https://github-production-release-asset-2e65be.s3.amazonaws.com/15400220/57552a00-9a49-11e9-9144-d9607ed4c2db?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191126%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191126T115258Z&X-Amz-Expires=300&X-Amz-Signature=834f5e39852c20a317fae48510e80cb3f40172896a77497f97aeb1bb86e2b802&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprogit.pdf&response-content-type=application%2Foctet-stream [12465653/12465653] -> "progit.pdf" [1] diff --git a/docs/basics/_examples/DL-101-102-111 b/docs/basics/_examples/DL-101-102-111 index 2678b3f23..d5dee43f7 100644 --- a/docs/basics/_examples/DL-101-102-111 +++ b/docs/basics/_examples/DL-101-102-111 @@ -1,6 +1,6 @@ # lets make the output a bit more concise with the --oneline option $ git log --oneline -8d6a681 add reference book about git -464b1e5 add books on Python and Unix to read later -758f0af Instruct annex to add text files to Git -715acb0 [DATALAD] new dataset +90ade2e add reference book about git +5d5a801 add books on Python and Unix to read later +4cc7c21 Instruct annex to add text files to Git +fca4a60 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-103-107 b/docs/basics/_examples/DL-101-103-107 index 69b27d27d..8a4b7fc7e 100644 --- a/docs/basics/_examples/DL-101-103-107 +++ b/docs/basics/_examples/DL-101-103-107 @@ -1,7 +1,7 @@ $ git log -p -n 2 -commit 84ce7c5a1a47f2ff9b27b7e9865c269d0d46eee8 +commit 1de8b79fb638a1510e7ebb0b3fcb2f2e51721456 Author: Elena Piscopia -Date: Mon Nov 18 10:51:15 2019 +0100 +Date: Tue Nov 26 12:53:01 2019 +0100 add note on datalad save @@ -18,9 +18,9 @@ index 3a7a1fe..bfa64d7 100644 +Always use informative, concise commit messages. + -commit d792dfb3711a31f407dad8705a3f3283372844bf +commit 0ed048a43dfe0fb64b7be9ac06a9074a154162f3 Author: Elena Piscopia -Date: Mon Nov 18 10:51:15 2019 +0100 +Date: Tue Nov 26 12:53:00 2019 +0100 Add notes on datalad create diff --git a/docs/basics/_examples/DL-101-106-101 b/docs/basics/_examples/DL-101-106-101 index 268e0d76a..6d52ce310 100644 --- a/docs/basics/_examples/DL-101-106-101 +++ b/docs/basics/_examples/DL-101-106-101 @@ -1,7 +1,7 @@ $ git log -p -n 2 -commit fd5cbf6987e377b5a38c4f4df304c22da8d6b597 +commit 67d32e901dd88b9ad093dce3e1884bcea85e478a Author: Elena Piscopia -Date: Mon Nov 18 10:52:29 2019 +0100 +Date: Tue Nov 26 12:53:15 2019 +0100 Add note on datalad install @@ -20,9 +20,9 @@ index bfa64d7..6d8cdce 100644 +root of the superdataset with the '-d' option. + -commit d4b91a53f95557adc4b78521af5b131c02fc35f6 +commit 776b0ac52acaaa20d3bd263d6c05d81e4a2e1029 Author: Elena Piscopia -Date: Mon Nov 18 10:51:18 2019 +0100 +Date: Tue Nov 26 12:53:03 2019 +0100 [DATALAD] Recorded changes @@ -44,9 +44,9 @@ index 0000000..dcc34fb @@ -0,0 +1 @@ +Subproject commit dcc34fbe669b06ced84ced381ba0db21cf5e665f -commit 84ce7c5a1a47f2ff9b27b7e9865c269d0d46eee8 +commit 1de8b79fb638a1510e7ebb0b3fcb2f2e51721456 Author: Elena Piscopia -Date: Mon Nov 18 10:51:15 2019 +0100 +Date: Tue Nov 26 12:53:01 2019 +0100 add note on datalad save @@ -63,9 +63,9 @@ index 3a7a1fe..bfa64d7 100644 +Always use informative, concise commit messages. + -commit d792dfb3711a31f407dad8705a3f3283372844bf +commit 0ed048a43dfe0fb64b7be9ac06a9074a154162f3 Author: Elena Piscopia -Date: Mon Nov 18 10:51:15 2019 +0100 +Date: Tue Nov 26 12:53:00 2019 +0100 Add notes on datalad create @@ -79,9 +79,9 @@ index 0000000..3a7a1fe +The dataset is created empty + -commit 8d6a681c2145aacaebf4855f675e892355560640 +commit 90ade2edb8566117c0d88cf478c654f97688c796 Author: Elena Piscopia -Date: Mon Nov 18 10:51:14 2019 +0100 +Date: Tue Nov 26 12:53:00 2019 +0100 add reference book about git @@ -94,9 +94,9 @@ index 0000000..c5d2ab2 +../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf \ No newline at end of file -commit 464b1e50a588101cff01de9d0a95ad1f7cfc0457 +commit 5d5a8014fa0f87fe11cc24223094a01e082c4b9b Author: Elena Piscopia -Date: Mon Nov 18 10:50:36 2019 +0100 +Date: Tue Nov 26 12:52:58 2019 +0100 add books on Python and Unix to read later @@ -117,9 +117,9 @@ index 0000000..9a812a0 +../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf \ No newline at end of file -commit 758f0af5a5d8865db0f8aac4717d8fa93771d8a4 +commit 4cc7c21a716368eda466b94630522d19abfef72f Author: Elena Piscopia -Date: Mon Nov 18 10:49:52 2019 +0100 +Date: Tue Nov 26 12:52:00 2019 +0100 Instruct annex to add text files to Git @@ -136,9 +136,9 @@ index c3aaefe..b223f34 100644 +* annex.largefiles=(not(mimetype=text/*)and(largerthan=0)) \ No newline at end of file -commit 715acb08af2fdab8d901180c3d9985a5335b0584 +commit fca4a608c6f8c749a2f3f83a3da8042b61e496ec Author: Elena Piscopia -Date: Mon Nov 18 10:49:52 2019 +0100 +Date: Tue Nov 26 12:52:00 2019 +0100 [DATALAD] new dataset @@ -155,12 +155,12 @@ index 0000000..b540820 \ No newline at end of file diff --git a/.datalad/config b/.datalad/config new file mode 100644 -index 0000000..ea9d4d6 +index 0000000..2f3e9f7 --- /dev/null +++ b/.datalad/config @@ -0,0 +1,2 @@ +[datalad "dataset"] -+ id = c4755e2a-09e8-11ea-98f1-e86a64c8054c ++ id = 27b04f08-1043-11ea-97f1-e86a64c8054c diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..c3aaefe diff --git a/docs/basics/_examples/DL-101-108-106 b/docs/basics/_examples/DL-101-108-106 index 60692351a..2bee778f5 100644 --- a/docs/basics/_examples/DL-101-108-106 +++ b/docs/basics/_examples/DL-101-108-106 @@ -1,7 +1,7 @@ $ git log -p -n 1 -commit c64173fe0a41bac4865c71f814bcbb84cd1c8146 +commit 69f9eccc548ef789bdfee266dbca045b099b3b36 Author: Elena Piscopia -Date: Mon Nov 18 10:52:30 2019 +0100 +Date: Tue Nov 26 12:53:16 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles @@ -9,7 +9,7 @@ Date: Mon Nov 18 10:52:30 2019 +0100 { "chain": [], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-108-108 b/docs/basics/_examples/DL-101-108-108 index 3a485e03d..8c8024ed4 100644 --- a/docs/basics/_examples/DL-101-108-108 +++ b/docs/basics/_examples/DL-101-108-108 @@ -1,11 +1,11 @@ $ git log --oneline -c64173f [DATALAD RUNCMD] create a list of podcast titles -9d578af Add simple script to write a list of podcast speakers and titles -fd5cbf6 Add note on datalad install -d4b91a5 [DATALAD] Recorded changes -84ce7c5 add note on datalad save -d792dfb Add notes on datalad create -8d6a681 add reference book about git -464b1e5 add books on Python and Unix to read later -758f0af Instruct annex to add text files to Git -715acb0 [DATALAD] new dataset +69f9ecc [DATALAD RUNCMD] create a list of podcast titles +12323f0 Add simple script to write a list of podcast speakers and titles +67d32e9 Add note on datalad install +776b0ac [DATALAD] Recorded changes +1de8b79 add note on datalad save +0ed048a Add notes on datalad create +90ade2e add reference book about git +5d5a801 add books on Python and Unix to read later +4cc7c21 Instruct annex to add text files to Git +fca4a60 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-108-113 b/docs/basics/_examples/DL-101-108-113 index b5e899e8e..6bbbd6a41 100644 --- a/docs/basics/_examples/DL-101-108-113 +++ b/docs/basics/_examples/DL-101-108-113 @@ -1,13 +1,13 @@ $ git log -n 2 -commit 854a655e0fb6e822739a8e83c2460fe58c107204 +commit 3923b9c26b11d8e4c22f89b49886ae6ccd70df21 Author: Elena Piscopia -Date: Mon Nov 18 10:52:32 2019 +0100 +Date: Tue Nov 26 12:53:18 2019 +0100 BF: list both directories content -commit c64173fe0a41bac4865c71f814bcbb84cd1c8146 +commit 69f9eccc548ef789bdfee266dbca045b099b3b36 Author: Elena Piscopia -Date: Mon Nov 18 10:52:30 2019 +0100 +Date: Tue Nov 26 12:53:16 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles @@ -15,7 +15,7 @@ Date: Mon Nov 18 10:52:30 2019 +0100 { "chain": [], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-108-114 b/docs/basics/_examples/DL-101-108-114 index 91a031085..a8a4ad0ac 100644 --- a/docs/basics/_examples/DL-101-108-114 +++ b/docs/basics/_examples/DL-101-108-114 @@ -1,5 +1,5 @@ -$ datalad rerun c64173fe0a41bac4865c71f814bcbb84cd1c8146 +$ datalad rerun 69f9eccc548ef789bdfee266dbca045b099b3b36 [INFO] == Command start (output follows) ===== [INFO] == Command exit (modification check follows) ===== add(ok): recordings/podcasts.tsv (file) diff --git a/docs/basics/_examples/DL-101-108-115 b/docs/basics/_examples/DL-101-108-115 index 2058ba2b4..c232490c5 100644 --- a/docs/basics/_examples/DL-101-108-115 +++ b/docs/basics/_examples/DL-101-108-115 @@ -1,17 +1,17 @@ $ git log -n 1 -commit 406b44c6474d167fc9e497cc807c415f3a810e14 +commit 140477ac946476718c89a152b1779d07ee0a5e6d Author: Elena Piscopia -Date: Mon Nov 18 10:52:33 2019 +0100 +Date: Tue Nov 26 12:53:19 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles === Do not change lines below === { "chain": [ - "c64173fe0a41bac4865c71f814bcbb84cd1c8146" + "69f9eccc548ef789bdfee266dbca045b099b3b36" ], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-108-120 b/docs/basics/_examples/DL-101-108-120 index 76ca906cf..da0f51476 100644 --- a/docs/basics/_examples/DL-101-108-120 +++ b/docs/basics/_examples/DL-101-108-120 @@ -1,17 +1,17 @@ $ git log -- recordings/podcasts.tsv -commit 406b44c6474d167fc9e497cc807c415f3a810e14 +commit 140477ac946476718c89a152b1779d07ee0a5e6d Author: Elena Piscopia -Date: Mon Nov 18 10:52:33 2019 +0100 +Date: Tue Nov 26 12:53:19 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles === Do not change lines below === { "chain": [ - "c64173fe0a41bac4865c71f814bcbb84cd1c8146" + "69f9eccc548ef789bdfee266dbca045b099b3b36" ], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", "exit": 0, "extra_inputs": [], "inputs": [], @@ -20,9 +20,9 @@ Date: Mon Nov 18 10:52:33 2019 +0100 } ^^^ Do not change lines above ^^^ -commit c64173fe0a41bac4865c71f814bcbb84cd1c8146 +commit 69f9eccc548ef789bdfee266dbca045b099b3b36 Author: Elena Piscopia -Date: Mon Nov 18 10:52:30 2019 +0100 +Date: Tue Nov 26 12:53:16 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles @@ -30,7 +30,7 @@ Date: Mon Nov 18 10:52:30 2019 +0100 { "chain": [], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-112-109 b/docs/basics/_examples/DL-101-112-109 index 012c58354..350ec937b 100644 --- a/docs/basics/_examples/DL-101-112-109 +++ b/docs/basics/_examples/DL-101-112-109 @@ -1,7 +1,7 @@ $ git log -p -n 2 -commit f609f69378303bfda1e6614d573ae59226dee8d5 +commit 66f97f3bf06fe4ed612f1b2a63a38a9108ff8aa1 Author: Elena Piscopia -Date: Mon Nov 18 10:52:43 2019 +0100 +Date: Tue Nov 26 12:53:26 2019 +0100 add note on clean datasets @@ -21,9 +21,9 @@ index d1ba8fc..8f0124f 100644 +to the files listed with --output flags. + -commit 009c66d0f2b2a8890674e5e455c43bb9232c9c48 +commit 17e39f474e033f46b798c9ea5a460cb9c7ed4f05 Author: Elena Piscopia -Date: Mon Nov 18 10:52:42 2019 +0100 +Date: Tue Nov 26 12:53:25 2019 +0100 [DATALAD RUNCMD] Resize logo for slides @@ -31,7 +31,7 @@ Date: Mon Nov 18 10:52:42 2019 +0100 { "chain": [], "cmd": "convert -resize 400x400 recordings/longnow/.datalad/feed_metadata/logo_salt.jpg recordings/salt_logo_small.jpg", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", "exit": 0, "extra_inputs": [], "inputs": [ diff --git a/docs/basics/_examples/DL-101-114-101 b/docs/basics/_examples/DL-101-114-101 index 9dcbdd9f3..7391f9a60 100644 --- a/docs/basics/_examples/DL-101-114-101 +++ b/docs/basics/_examples/DL-101-114-101 @@ -1,22 +1,22 @@ $ git log --reverse --oneline -715acb0 [DATALAD] new dataset -758f0af Instruct annex to add text files to Git -464b1e5 add books on Python and Unix to read later -8d6a681 add reference book about git -d792dfb Add notes on datalad create -84ce7c5 add note on datalad save -d4b91a5 [DATALAD] Recorded changes -fd5cbf6 Add note on datalad install -9d578af Add simple script to write a list of podcast speakers and titles -c64173f [DATALAD RUNCMD] create a list of podcast titles -854a655 BF: list both directories content -406b44c [DATALAD RUNCMD] create a list of podcast titles -e6e14a7 add note datalad and git diff -15e7810 add note on basic datalad run and datalad rerun -771909d [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -dc84ec4 resized picture by hand -054b821 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -3b21c7d add additional notes on run options -5e96512 [DATALAD RUNCMD] Resize logo for slides -009c66d [DATALAD RUNCMD] Resize logo for slides -f609f69 add note on clean datasets +fca4a60 [DATALAD] new dataset +4cc7c21 Instruct annex to add text files to Git +5d5a801 add books on Python and Unix to read later +90ade2e add reference book about git +0ed048a Add notes on datalad create +1de8b79 add note on datalad save +776b0ac [DATALAD] Recorded changes +67d32e9 Add note on datalad install +12323f0 Add simple script to write a list of podcast speakers and titles +69f9ecc [DATALAD RUNCMD] create a list of podcast titles +3923b9c BF: list both directories content +140477a [DATALAD RUNCMD] create a list of podcast titles +de40bb1 add note datalad and git diff +5ca7d13 add note on basic datalad run and datalad rerun +11dfb75 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +8cdc201 resized picture by hand +fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +048ae39 add additional notes on run options +68ade50 [DATALAD RUNCMD] Resize logo for slides +17e39f4 [DATALAD RUNCMD] Resize logo for slides +66f97f3 add note on clean datasets diff --git a/docs/basics/_examples/DL-101-115-103 b/docs/basics/_examples/DL-101-115-103 index 951b878a6..b7593e152 100644 --- a/docs/basics/_examples/DL-101-115-103 +++ b/docs/basics/_examples/DL-101-115-103 @@ -1,7 +1,7 @@ $ ls -lah total 20K -drwxr-xr-x 2 adina adina 4.0K Nov 18 10:51 . -drwxr-xr-x 7 adina adina 4.0K Nov 18 10:52 .. -lrwxrwxrwx 1 adina adina 131 Nov 18 10:50 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf +drwxr-xr-x 2 adina adina 4.0K Nov 26 12:53 . +drwxr-xr-x 7 adina adina 4.0K Nov 26 12:53 .. +lrwxrwxrwx 1 adina adina 131 Nov 26 12:52 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf lrwxrwxrwx 1 adina adina 133 Jun 29 17:39 progit.pdf -> ../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf lrwxrwxrwx 1 adina adina 131 Jan 28 2019 TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-116-105 b/docs/basics/_examples/DL-101-116-105 index cbbf57f11..ad2b53b61 100644 --- a/docs/basics/_examples/DL-101-116-105 +++ b/docs/basics/_examples/DL-101-116-105 @@ -1,4 +1,4 @@ $ git annex whereis books/TLCL.pdf whereis books/TLCL.pdf (1 copy) - a5e9c638-cc6b-4c27-99a5-14802c8a6898 -- course on DataLad-101 on my private Laptop [origin] + 9d17effd-b251-48ab-92b2-b629c15a9b3e -- course on DataLad-101 on my private Laptop [origin] ok diff --git a/docs/basics/_examples/DL-101-117-101 b/docs/basics/_examples/DL-101-117-101 index 2221cf29c..bbe896df9 100644 --- a/docs/basics/_examples/DL-101-117-101 +++ b/docs/basics/_examples/DL-101-117-101 @@ -6,7 +6,7 @@ $ cd recordings/longnow $ git annex whereis Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3 whereis Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3 (3 copies) 00000000-0000-0000-0000-000000000001 -- web - 1f1f78d5-c98d-4435-9f81-8bad94145ed5 -- me@muninn:~/dl-101/DataLad-101/recordings/longnow [origin] + 31ddeb94-12ef-4084-a439-8105568ec76c -- me@muninn:~/dl-101/DataLad-101/recordings/longnow [origin] da3bf937-5bd2-43ea-a07b-bcbe71f3b875 -- mih@medusa:/tmp/seminars-on-longterm-thinking web: http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 diff --git a/docs/basics/_examples/DL-101-118-102 b/docs/basics/_examples/DL-101-118-102 index e660877a2..dfc97630d 100644 --- a/docs/basics/_examples/DL-101-118-102 +++ b/docs/basics/_examples/DL-101-118-102 @@ -1,23 +1,23 @@ # lets view the history $ git log --oneline -f609f69 add note on clean datasets -009c66d [DATALAD RUNCMD] Resize logo for slides -5e96512 [DATALAD RUNCMD] Resize logo for slides -3b21c7d add additional notes on run options -054b821 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -dc84ec4 resized picture by hand -771909d [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -15e7810 add note on basic datalad run and datalad rerun -e6e14a7 add note datalad and git diff -406b44c [DATALAD RUNCMD] create a list of podcast titles -854a655 BF: list both directories content -c64173f [DATALAD RUNCMD] create a list of podcast titles -9d578af Add simple script to write a list of podcast speakers and titles -fd5cbf6 Add note on datalad install -d4b91a5 [DATALAD] Recorded changes -84ce7c5 add note on datalad save -d792dfb Add notes on datalad create -8d6a681 add reference book about git -464b1e5 add books on Python and Unix to read later -758f0af Instruct annex to add text files to Git -715acb0 [DATALAD] new dataset +66f97f3 add note on clean datasets +17e39f4 [DATALAD RUNCMD] Resize logo for slides +68ade50 [DATALAD RUNCMD] Resize logo for slides +048ae39 add additional notes on run options +fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +8cdc201 resized picture by hand +11dfb75 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +5ca7d13 add note on basic datalad run and datalad rerun +de40bb1 add note datalad and git diff +140477a [DATALAD RUNCMD] create a list of podcast titles +3923b9c BF: list both directories content +69f9ecc [DATALAD RUNCMD] create a list of podcast titles +12323f0 Add simple script to write a list of podcast speakers and titles +67d32e9 Add note on datalad install +776b0ac [DATALAD] Recorded changes +1de8b79 add note on datalad save +0ed048a Add notes on datalad create +90ade2e add reference book about git +5d5a801 add books on Python and Unix to read later +4cc7c21 Instruct annex to add text files to Git +fca4a60 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-118-103 b/docs/basics/_examples/DL-101-118-103 index fb296346f..53abb1ee0 100644 --- a/docs/basics/_examples/DL-101-118-103 +++ b/docs/basics/_examples/DL-101-118-103 @@ -1,5 +1,5 @@ -$ datalad rerun 009c66d0f2b2a8890674e5e455c43bb9232c9c48 +$ datalad rerun 17e39f474e033f46b798c9ea5a460cb9c7ed4f05 [INFO] Making sure inputs are available (this may take some time) [WARNING] no content present; cannot unlock [unlock(/home/me/dl-101/mock_user/DataLad-101/recordings/salt_logo_small.jpg)] [INFO] == Command start (output follows) ===== diff --git a/docs/basics/_examples/DL-101-121-103 b/docs/basics/_examples/DL-101-121-103 index 71f58d945..e5c0bc3ee 100644 --- a/docs/basics/_examples/DL-101-121-103 +++ b/docs/basics/_examples/DL-101-121-103 @@ -1,8 +1,7 @@ $ git log -n 1 -p - -commit 834615c670935bb5cb682ef301c6167d6786f02f +commit a2d01ff8ce412b74a9c5240e8a8985a917628d0d Author: Elena Piscopia -Date: Mon Nov 18 10:53:01 2019 +0100 +Date: Tue Nov 26 12:53:34 2019 +0100 Include nesting demo from datalad website diff --git a/docs/basics/_examples/DL-101-121-112 b/docs/basics/_examples/DL-101-121-112 index f2d6047ae..48f63d5a4 100644 --- a/docs/basics/_examples/DL-101-121-112 +++ b/docs/basics/_examples/DL-101-121-112 @@ -1,28 +1,27 @@ $ git log --oneline -db8aa3a Merge remote-tracking branch 'refs/remotes/roommate/master' -834615c Include nesting demo from datalad website -0e85c3e add note about datalad update -2e7a3ad add note on git annex whereis -73f44c4 add note about installing from paths and recursive installations -dd6fef5 add note on clean datasets -d9d0872 [DATALAD RUNCMD] Resize logo for slides -36a9cfd [DATALAD RUNCMD] Resize logo for slides -244c234 add additional notes on run options -e3277e9 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -257130b resized picture by hand -6e624f5 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -327fd83 add note on basic datalad run and datalad rerun -4e3fb6e add note datalad and git diff -d26a29b [DATALAD RUNCMD] create a list of podcast titles -d04c4af BF: list both directories content -ecde4e4 [DATALAD RUNCMD] create a list of podcast titles -ef8b726 Add simple script to write a list of podcast speakers and titles -e61c229 Add note on datalad install -49e85d1 [DATALAD] Recorded changes -5f98f2b add note on datalad save -b72b237 Add notes on datalad create -dac2fa7 add reference book about git -9a6fb63 add books on Python and Unix to read later -1a45fd4 Instruct annex to add text files to Git -e85b950 [DATALAD] new dataset - +26b599c Merge remote-tracking branch 'refs/remotes/roommate/master' +029add8 add note about datalad update +a2d01ff Include nesting demo from datalad website +2091c14 add note on git annex whereis +e2210e8 add note about installing from paths and recursive installations +66f97f3 add note on clean datasets +17e39f4 [DATALAD RUNCMD] Resize logo for slides +68ade50 [DATALAD RUNCMD] Resize logo for slides +048ae39 add additional notes on run options +fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +8cdc201 resized picture by hand +11dfb75 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +5ca7d13 add note on basic datalad run and datalad rerun +de40bb1 add note datalad and git diff +140477a [DATALAD RUNCMD] create a list of podcast titles +3923b9c BF: list both directories content +69f9ecc [DATALAD RUNCMD] create a list of podcast titles +12323f0 Add simple script to write a list of podcast speakers and titles +67d32e9 Add note on datalad install +776b0ac [DATALAD] Recorded changes +1de8b79 add note on datalad save +0ed048a Add notes on datalad create +90ade2e add reference book about git +5d5a801 add books on Python and Unix to read later +4cc7c21 Instruct annex to add text files to Git +fca4a60 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-122-101 b/docs/basics/_examples/DL-101-122-101 index cfa33df00..1038b2a65 100644 --- a/docs/basics/_examples/DL-101-122-101 +++ b/docs/basics/_examples/DL-101-122-101 @@ -5,7 +5,7 @@ $ cat .git/config bare = false logallrefupdates = true [annex] - uuid = a5e9c638-cc6b-4c27-99a5-14802c8a6898 + uuid = 9d17effd-b251-48ab-92b2-b629c15a9b3e version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -14,5 +14,5 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 838f9159-3b3d-4dd7-86b0-7dd7e51b05ed + annex-uuid = 7478bb68-77fb-466f-a3c6-a943d5b87569 annex-ignore = false diff --git a/docs/basics/_examples/DL-101-122-103 b/docs/basics/_examples/DL-101-122-103 index 366348874..fe4b62c8b 100644 --- a/docs/basics/_examples/DL-101-122-103 +++ b/docs/basics/_examples/DL-101-122-103 @@ -6,7 +6,7 @@ $ cat .git/config logallrefupdates = true editor = nano [annex] - uuid = a5e9c638-cc6b-4c27-99a5-14802c8a6898 + uuid = 9d17effd-b251-48ab-92b2-b629c15a9b3e version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -15,5 +15,5 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 838f9159-3b3d-4dd7-86b0-7dd7e51b05ed + annex-uuid = 7478bb68-77fb-466f-a3c6-a943d5b87569 annex-ignore = false diff --git a/docs/basics/_examples/DL-101-123-105 b/docs/basics/_examples/DL-101-123-105 index 59b332385..0bb2b5773 100644 --- a/docs/basics/_examples/DL-101-123-105 +++ b/docs/basics/_examples/DL-101-123-105 @@ -1,3 +1,3 @@ $ cat .datalad/config [datalad "dataset"] - id = 7256f1ae-000f-11ea-97b1-d0c637c523bc + id = 27b04f08-1043-11ea-97f1-e86a64c8054c diff --git a/docs/basics/_examples/DL-101-125-106 b/docs/basics/_examples/DL-101-125-106 deleted file mode 100644 index d2ef1c299..000000000 --- a/docs/basics/_examples/DL-101-125-106 +++ /dev/null @@ -1,3 +0,0 @@ -$ cat .datalad/config -[datalad "dataset"] - id = c4755e2a-09e8-11ea-98f1-e86a64c8054c diff --git a/docs/basics/_examples/DL-101-130-106 b/docs/basics/_examples/DL-101-130-106 index cb2b9fbee..8813ad4d1 100644 --- a/docs/basics/_examples/DL-101-130-106 +++ b/docs/basics/_examples/DL-101-130-106 @@ -1,5 +1,6 @@ $ cd ../ $ tree -d +$ cd midterm_project . ├── books ├── code diff --git a/docs/basics/_examples/DL-101-130-110 b/docs/basics/_examples/DL-101-130-110 index 73f5f3199..25d2ce53d 100644 --- a/docs/basics/_examples/DL-101-130-110 +++ b/docs/basics/_examples/DL-101-130-110 @@ -1,7 +1,7 @@ $ git show ready4analysis -commit 789e6b445eac4f2245a05420289bff3e1580e9f7 +commit 9bd5576290bcd43f9be6274b05ae31c00759cdb0 Author: Elena Piscopia -Date: Mon Nov 18 16:13:18 2019 +0100 +Date: Tue Nov 26 12:53:44 2019 +0100 add script for kNN classification and plotting diff --git a/docs/basics/_examples/DL-101-130-112 b/docs/basics/_examples/DL-101-130-112 index 42a513bcd..c0cb3846d 100644 --- a/docs/basics/_examples/DL-101-130-112 +++ b/docs/basics/_examples/DL-101-130-112 @@ -1,6 +1,6 @@ $ git log --oneline -c73efce [DATALAD RUNCMD] analyze iris data with classification analysis -789e6b4 add script for kNN classification and plotting -9466bd3 [DATALAD] Recorded changes -ab62c74 Apply YODA dataset setup -8d3c887 [DATALAD] new dataset +be29815 [DATALAD RUNCMD] analyze iris data with classification analysis +9bd5576 add script for kNN classification and plotting +ebb126e [DATALAD] Recorded changes +b9ebebd Apply YODA dataset setup +fcd1201 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-130-116 b/docs/basics/_examples/DL-101-130-116 index ca2423833..56f915583 100644 --- a/docs/basics/_examples/DL-101-130-116 +++ b/docs/basics/_examples/DL-101-130-116 @@ -1 +1,2 @@ -$ datalad create-sibling-github -d . -r midtermproject + +.: github(-) [https://github.com/adswa/midtermproject.git (git)] diff --git a/docs/basics/_examples/DL-101-132-101 b/docs/basics/_examples/DL-101-132-101 index eb04b041f..7c1d131a8 100644 --- a/docs/basics/_examples/DL-101-132-101 +++ b/docs/basics/_examples/DL-101-132-101 @@ -1,7 +1,7 @@ $ git log --oneline -7c4eb2c Provide project description -41bebc1 [DATALAD RUNCMD] analyze iris data with classification analysis -60250f0 add script for kNN classification and plotting -23be05e [DATALAD] Recorded changes -3d05185 Apply YODA dataset setup -8aceb01 [DATALAD] new dataset +b2c8f82 Provide project description +be29815 [DATALAD RUNCMD] analyze iris data with classification analysis +9bd5576 add script for kNN classification and plotting +ebb126e [DATALAD] Recorded changes +b9ebebd Apply YODA dataset setup +fcd1201 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-132-104 b/docs/basics/_examples/DL-101-132-104 index d752d1664..4c359c93f 100644 --- a/docs/basics/_examples/DL-101-132-104 +++ b/docs/basics/_examples/DL-101-132-104 @@ -1,14 +1,14 @@ $ git log -p -n 1 -commit 587ac34d63c93275649cb65cd932c6a052569fdd +commit 5e4c89a4e753b4f09866bf82e22eca1e3ae8d907 Author: Elena Piscopia -Date: Mon Nov 18 08:44:26 2019 +0100 +Date: Tue Nov 26 12:54:04 2019 +0100 finished my midterm project! diff --git a/midterm_project b/midterm_project -index 3d05185..7c4eb2c 160000 +index b9ebebd..b2c8f82 160000 --- a/midterm_project +++ b/midterm_project @@ -1 +1 @@ --Subproject commit 3d05185cd1b874835d56b232f44d4f3c35f6bbff -+Subproject commit 7c4eb2c13166c12853b3162778691304c4a93aa2 +-Subproject commit b9ebebd9a77f4335a5903ef56ed5e6c5e037b140 ++Subproject commit b2c8f82e096717a6d18c8138ad654ce513cbd81a diff --git a/docs/basics/_examples/DL-101-135-101 b/docs/basics/_examples/DL-101-135-101 index b5530c61c..00bf3e7ff 100644 --- a/docs/basics/_examples/DL-101-135-101 +++ b/docs/basics/_examples/DL-101-135-101 @@ -1,2 +1,2 @@ $ datalad --version -datalad 0.12.0rc6.dev91 +datalad 0.12.0rc6.dev97 diff --git a/docs/basics/_examples/DL-101-135-102 b/docs/basics/_examples/DL-101-135-102 index 27e99fc0d..a3662bd8c 100644 --- a/docs/basics/_examples/DL-101-135-102 +++ b/docs/basics/_examples/DL-101-135-102 @@ -2,8 +2,8 @@ $ datalad wtf # WTF ## configuration ## datalad - - version: 0.12.0rc6.dev91 - - full_version: 0.12.0rc6.dev91-gde46 + - version: 0.12.0rc6.dev97 + - full_version: 0.12.0rc6.dev97-g9250-dirty ## dataset - path: /home/me/dl-101/DataLad-101 - repo: AnnexRepo @@ -11,9 +11,9 @@ $ datalad wtf ## dependencies - tqdm: 4.32.2 - cmd:annex: 7.20190129 - - cmd:git: 2.20.1 - - cmd:bundled-git: 2.20.1 - - cmd:system-git: 2.20.1 + - cmd:git: 2.24.0 + - cmd:bundled-git: 2.24.0 + - cmd:system-git: 2.24.0 - cmd:system-ssh: 7.9p1 - appdirs: 1.4.3 - boto: 2.49.0 @@ -309,8 +309,8 @@ $ datalad wtf - type: posix - name: Linux - release: 4.19.0-6-amd64 - - version: #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) - - distribution: debian/10.1 + - version: #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) + - distribution: debian/10.2 - max_path_length: 283 - encoding: - default: utf-8 diff --git a/docs/basics/_examples/DL-101-136-101 b/docs/basics/_examples/DL-101-136-101 index 429ad061d..9568e1530 100644 --- a/docs/basics/_examples/DL-101-136-101 +++ b/docs/basics/_examples/DL-101-136-101 @@ -2,8 +2,8 @@ $ cd books/ $ mv TLCL.pdf The_Linux_Command_Line.pdf $ ls -lah total 20K -drwxr-xr-x 2 adina adina 4.0K Nov 18 10:53 . -drwxr-xr-x 7 adina adina 4.0K Nov 18 10:53 .. -lrwxrwxrwx 1 adina adina 131 Nov 18 10:50 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf +drwxr-xr-x 2 adina adina 4.0K Nov 26 12:54 . +drwxr-xr-x 8 adina adina 4.0K Nov 26 12:54 .. +lrwxrwxrwx 1 adina adina 131 Nov 26 12:52 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf lrwxrwxrwx 1 adina adina 133 Jun 29 17:39 progit.pdf -> ../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf lrwxrwxrwx 1 adina adina 131 Jan 28 2019 The_Linux_Command_Line.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-104 b/docs/basics/_examples/DL-101-136-104 index b11a97724..338fbed25 100644 --- a/docs/basics/_examples/DL-101-136-104 +++ b/docs/basics/_examples/DL-101-136-104 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit 4b793ed196fc072389963aa5d2d87e585a88e034 +commit c8f6b4847d05d7f3c4602f55b49e12880256b494 Author: Elena Piscopia -Date: Mon Nov 18 10:53:09 2019 +0100 +Date: Tue Nov 26 12:54:26 2019 +0100 rename the book diff --git a/docs/basics/_examples/DL-101-136-105 b/docs/basics/_examples/DL-101-136-105 index 0e1e4b6f4..6c18b8f26 100644 --- a/docs/basics/_examples/DL-101-136-105 +++ b/docs/basics/_examples/DL-101-136-105 @@ -1,4 +1,3 @@ $ git reset --hard HEAD~1 $ datalad status -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at 11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-108 b/docs/basics/_examples/DL-101-136-108 index 2729fc0ed..efa77dbcc 100644 --- a/docs/basics/_examples/DL-101-136-108 +++ b/docs/basics/_examples/DL-101-136-108 @@ -1,7 +1,6 @@ $ git status On branch master Changes to be committed: - (use "git reset HEAD ..." to unstage) - + (use "git restore --staged ..." to unstage) renamed: TLCL.pdf -> The_Linux_Command_Line.pdf diff --git a/docs/basics/_examples/DL-101-136-109 b/docs/basics/_examples/DL-101-136-109 index ed232aa80..8c34b1e2d 100644 --- a/docs/basics/_examples/DL-101-136-109 +++ b/docs/basics/_examples/DL-101-136-109 @@ -1,4 +1,4 @@ $ git commit -m "rename book" -[master 9892940] rename book +[master 6498fa3] rename book 1 file changed, 0 insertions(+), 0 deletions(-) rename books/{TLCL.pdf => The_Linux_Command_Line.pdf} (100%) diff --git a/docs/basics/_examples/DL-101-136-110 b/docs/basics/_examples/DL-101-136-110 index 0e1e4b6f4..6c18b8f26 100644 --- a/docs/basics/_examples/DL-101-136-110 +++ b/docs/basics/_examples/DL-101-136-110 @@ -1,4 +1,3 @@ $ git reset --hard HEAD~1 $ datalad status -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at 11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-121 b/docs/basics/_examples/DL-101-136-121 index de56dc3fa..bba4c21ee 100644 --- a/docs/basics/_examples/DL-101-136-121 +++ b/docs/basics/_examples/DL-101-136-121 @@ -1,3 +1,3 @@ $ cd ../ $ ls -l TLCL.pdf -lrwxrwxrwx 1 adina adina 131 Nov 18 10:53 TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 131 Nov 26 12:54 TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-122 b/docs/basics/_examples/DL-101-136-122 index 7fe3a9529..9c38196f2 100644 --- a/docs/basics/_examples/DL-101-136-122 +++ b/docs/basics/_examples/DL-101-136-122 @@ -7,4 +7,4 @@ action summary: add (ok: 1) delete (ok: 1) save (ok: 1) -lrwxrwxrwx 1 adina adina 128 Nov 18 10:53 TLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 128 Nov 26 12:54 TLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-123 b/docs/basics/_examples/DL-101-136-123 index 86b4e27f9..a46ffd8e4 100644 --- a/docs/basics/_examples/DL-101-136-123 +++ b/docs/basics/_examples/DL-101-136-123 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit 3189043ce3b7998446471aff4accb624898e21a0 +commit fc12e79c57deb44eb53231c516343acc6b40d8a0 Author: Elena Piscopia -Date: Mon Nov 18 10:53:11 2019 +0100 +Date: Tue Nov 26 12:54:27 2019 +0100 moved book into root diff --git a/docs/basics/_examples/DL-101-136-124 b/docs/basics/_examples/DL-101-136-124 index ee40b070e..195a7fc5f 100644 --- a/docs/basics/_examples/DL-101-136-124 +++ b/docs/basics/_examples/DL-101-136-124 @@ -1,3 +1,2 @@ $ git reset --hard HEAD~1 -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at 11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-132 b/docs/basics/_examples/DL-101-136-132 index 1a8b345f8..54ff75700 100644 --- a/docs/basics/_examples/DL-101-136-132 +++ b/docs/basics/_examples/DL-101-136-132 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit 1246e8c40678edf85874b5f72940500220d3a148 +commit 10b9fe8764cb2862199709d30c8c0bd10a2aab96 Author: Elena Piscopia -Date: Mon Nov 18 10:53:11 2019 +0100 +Date: Tue Nov 26 12:54:28 2019 +0100 add copy of TLCL.pdf diff --git a/docs/basics/_examples/DL-101-136-133 b/docs/basics/_examples/DL-101-136-133 index 70eeac6b7..30f9ba160 100644 --- a/docs/basics/_examples/DL-101-136-133 +++ b/docs/basics/_examples/DL-101-136-133 @@ -1,4 +1,4 @@ $ ls -l copyofTLCL.pdf $ ls -l books/TLCL.pdf -lrwxrwxrwx 1 adina adina 128 Nov 18 10:53 copyofTLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf -lrwxrwxrwx 1 adina adina 131 Nov 18 10:53 books/TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 128 Nov 26 12:54 copyofTLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 131 Nov 26 12:54 books/TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-134 b/docs/basics/_examples/DL-101-136-134 index 29615e42a..195a7fc5f 100644 --- a/docs/basics/_examples/DL-101-136-134 +++ b/docs/basics/_examples/DL-101-136-134 @@ -1,2 +1,2 @@ $ git reset --hard HEAD~1 -HEAD is now at 587ac34 finished my midterm project! +HEAD is now at 11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-140 b/docs/basics/_examples/DL-101-136-140 index 97754e5cc..b408ca36a 100644 --- a/docs/basics/_examples/DL-101-136-140 +++ b/docs/basics/_examples/DL-101-136-140 @@ -6,7 +6,7 @@ $ cat .git/config logallrefupdates = true editor = nano [annex] - uuid = a5e9c638-cc6b-4c27-99a5-14802c8a6898 + uuid = 9d17effd-b251-48ab-92b2-b629c15a9b3e version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -15,7 +15,7 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 050f8bfe-1ee0-4742-bcb4-b103347aaa05 + annex-uuid = 7478bb68-77fb-466f-a3c6-a943d5b87569 annex-ignore = false [submodule "midterm_project"] url = /home/me/dl-101/DataLad-101/midterm_project diff --git a/docs/basics/_examples/DL-101-136-144 b/docs/basics/_examples/DL-101-136-144 index 3a91c779c..b3c5b4e5a 100644 --- a/docs/basics/_examples/DL-101-136-144 +++ b/docs/basics/_examples/DL-101-136-144 @@ -6,7 +6,7 @@ $ cat .git/config logallrefupdates = true editor = nano [annex] - uuid = a5e9c638-cc6b-4c27-99a5-14802c8a6898 + uuid = 9d17effd-b251-48ab-92b2-b629c15a9b3e version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -15,7 +15,7 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/onemoredir/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 050f8bfe-1ee0-4742-bcb4-b103347aaa05 + annex-uuid = 7478bb68-77fb-466f-a3c6-a943d5b87569 annex-ignore = false [submodule "midterm_project"] url = /home/me/dl-101/DataLad-101/midterm_project diff --git a/docs/basics/_examples/DL-101-136-146 b/docs/basics/_examples/DL-101-136-146 index 152c158bd..4a60b0a04 100644 --- a/docs/basics/_examples/DL-101-136-146 +++ b/docs/basics/_examples/DL-101-136-146 @@ -1,5 +1,4 @@ $ cd ../mock_user && mv onemoredir/DataLad-101 . $ rm -r onemoredir $ cd ../DataLad-101 && git reset --hard master -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at 11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-152 b/docs/basics/_examples/DL-101-136-152 index bccb24948..195a7fc5f 100644 --- a/docs/basics/_examples/DL-101-136-152 +++ b/docs/basics/_examples/DL-101-136-152 @@ -1,4 +1,2 @@ $ git reset --hard HEAD~1 -<<<<<<< HEAD -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at 11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-156 b/docs/basics/_examples/DL-101-136-156 index c6f48fbce..2cfc8acb8 100644 --- a/docs/basics/_examples/DL-101-136-156 +++ b/docs/basics/_examples/DL-101-136-156 @@ -1,3 +1,3 @@ $ git reset --hard HEAD~1 warning: unable to rmdir 'longnow': Directory not empty -HEAD is now at 587ac34 finished my midterm project! +HEAD is now at 11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-164 b/docs/basics/_examples/DL-101-136-164 index d57881176..4e16be21e 100644 --- a/docs/basics/_examples/DL-101-136-164 +++ b/docs/basics/_examples/DL-101-136-164 @@ -1,3 +1,2 @@ $ git reset --hard HEAD~2 -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at 11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-174 b/docs/basics/_examples/DL-101-136-174 index 527366d66..fb90430ae 100644 --- a/docs/basics/_examples/DL-101-136-174 +++ b/docs/basics/_examples/DL-101-136-174 @@ -1,6 +1,6 @@ $ git reset --hard HEAD~1 $ ls -HEAD is now at da61732 Added flower mosaic from wikimedia +HEAD is now at bd87516 Added flower mosaic from wikimedia books code flowers.jpg diff --git a/docs/basics/_examples/DL-101-136-180 b/docs/basics/_examples/DL-101-136-180 index 4c6dd0fc6..4e16be21e 100644 --- a/docs/basics/_examples/DL-101-136-180 +++ b/docs/basics/_examples/DL-101-136-180 @@ -1,2 +1,2 @@ $ git reset --hard HEAD~2 -HEAD is now at 587ac34 finished my midterm project! +HEAD is now at 11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-101 b/docs/basics/_examples/DL-101-137-101 index fd51a8174..4f095b9e5 100644 --- a/docs/basics/_examples/DL-101-137-101 +++ b/docs/basics/_examples/DL-101-137-101 @@ -1,16 +1,16 @@ $ git log -15 --oneline -587ac34 finished my midterm project! -de7d909 [DATALAD] Recorded changes -37d74af add note on DataLads procedures -264d668 add note on configurations and git config -2277fe7 Add note on adding siblings -f015735 Merge remote-tracking branch 'refs/remotes/roommate/master' -0a01e8a Include nesting demo from datalad website -5e45b4e add note about datalad update -e613db1 add note on git annex whereis -118eed9 add note about installing from paths and recursive installations -f42379c add note on clean datasets -475d3de [DATALAD RUNCMD] Resize logo for slides -6856e4f [DATALAD RUNCMD] Resize logo for slides -72fa9f9 add additional notes on run options -b49af2f [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +11c2ecb add container and execute analysis within container +5e4c89a finished my midterm project! +ca88890 [DATALAD] Recorded changes +770e708 add note on DataLads procedures +ad89f03 add note on configurations and git config +0686627 Add note on adding siblings +26b599c Merge remote-tracking branch 'refs/remotes/roommate/master' +029add8 add note about datalad update +a2d01ff Include nesting demo from datalad website +2091c14 add note on git annex whereis +e2210e8 add note about installing from paths and recursive installations +66f97f3 add note on clean datasets +17e39f4 [DATALAD RUNCMD] Resize logo for slides +68ade50 [DATALAD RUNCMD] Resize logo for slides +048ae39 add additional notes on run options diff --git a/docs/basics/_examples/DL-101-137-105 b/docs/basics/_examples/DL-101-137-105 index 66160df0f..416f24aaa 100644 --- a/docs/basics/_examples/DL-101-137-105 +++ b/docs/basics/_examples/DL-101-137-105 @@ -1,7 +1,7 @@ $ git log -p -1 -commit 986ede3879a4a86a93b0b4d22ccee925c15c6230 +commit 8669cd7d57a6aeca76884a4f8935ff0c6a24fed4 Author: Elena Piscopia -Date: Mon Nov 18 10:53:25 2019 +0100 +Date: Tue Nov 26 12:54:39 2019 +0100 [DATALAD] Recorded changes diff --git a/docs/basics/_examples/DL-101-137-106 b/docs/basics/_examples/DL-101-137-106 index 3982396bd..e85316dfb 100644 --- a/docs/basics/_examples/DL-101-137-106 +++ b/docs/basics/_examples/DL-101-137-106 @@ -1,4 +1,4 @@ $ git log -n 3 --oneline -4604d99 [DATALAD] Recorded changes -587ac34 finished my midterm project! -de7d909 [DATALAD] Recorded changes +8669cd7 [DATALAD] Recorded changes +11c2ecb add container and execute analysis within container +5e4c89a finished my midterm project! diff --git a/docs/basics/_examples/DL-101-137-107 b/docs/basics/_examples/DL-101-137-107 index ab6678982..7e1add6a8 100644 --- a/docs/basics/_examples/DL-101-137-107 +++ b/docs/basics/_examples/DL-101-137-107 @@ -1 +1,2 @@ -$ git reset --mixed 587ac34d63c93275649cb65cd932c6a052569fdd + +$ git reset --mixed 11c2ecb85f22ad14f74809effc83b28346500e91 diff --git a/docs/basics/_examples/DL-101-137-108 b/docs/basics/_examples/DL-101-137-108 index 051d98c91..f9d8c86f1 100644 --- a/docs/basics/_examples/DL-101-137-108 +++ b/docs/basics/_examples/DL-101-137-108 @@ -1,3 +1,3 @@ $ git log -n 2 --oneline -587ac34 finished my midterm project! -de7d909 [DATALAD] Recorded changes \ No newline at end of file +11c2ecb add container and execute analysis within container +5e4c89a finished my midterm project! diff --git a/docs/basics/_examples/DL-101-137-112 b/docs/basics/_examples/DL-101-137-112 index 53e51f3e2..3d993d5db 100644 --- a/docs/basics/_examples/DL-101-137-112 +++ b/docs/basics/_examples/DL-101-137-112 @@ -1,12 +1,12 @@ -$ git log -n 2 -commit 51d763ed5883bc51f6eabc36248e4ef80b9f600f +$ git log -2 +commit 43af7fea4a87d59bcfa985f71dd401c041831eef Author: Elena Piscopia -Date: Mon Nov 18 08:44:46 2019 +0100 +Date: Tue Nov 26 12:54:39 2019 +0100 save my favorite Git joke -commit 587ac34d63c93275649cb65cd932c6a052569fdd +commit 11c2ecb85f22ad14f74809effc83b28346500e91 Author: Elena Piscopia -Date: Mon Nov 18 08:44:26 2019 +0100 +Date: Tue Nov 26 12:54:23 2019 +0100 - finished my midterm project! + add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-114 b/docs/basics/_examples/DL-101-137-114 index d3b1afd2b..a72f776e7 100644 --- a/docs/basics/_examples/DL-101-137-114 +++ b/docs/basics/_examples/DL-101-137-114 @@ -1,2 +1,2 @@ $ ls -l apdffile.pdf -lrwxrwxrwx 1 adina adina 122 Nov 18 10:53 apdffile.pdf -> .git/annex/objects/gV/gf/MD5E-s1842--40e4e7d327aa106fe64ef8b799a29d54.pdf/MD5E-s1842--40e4e7d327aa106fe64ef8b799a29d54.pdf +lrwxrwxrwx 1 adina adina 122 Nov 26 12:54 apdffile.pdf -> .git/annex/objects/vp/vm/MD5E-s1842--2377e02244882c5bf3d5d6ebc5930e68.pdf/MD5E-s1842--2377e02244882c5bf3d5d6ebc5930e68.pdf diff --git a/docs/basics/_examples/DL-101-137-116 b/docs/basics/_examples/DL-101-137-116 index f2dd736eb..ee8a00cc7 100644 --- a/docs/basics/_examples/DL-101-137-116 +++ b/docs/basics/_examples/DL-101-137-116 @@ -1,2 +1,2 @@ $ ls -l apdffile.pdf --rw-r--r-- 1 adina adina 1842 Nov 18 10:53 apdffile.pdf +-rw-r--r-- 1 adina adina 1842 Nov 26 12:54 apdffile.pdf diff --git a/docs/basics/_examples/DL-101-137-117 b/docs/basics/_examples/DL-101-137-117 index f873dcce8..14bb8aa6c 100644 --- a/docs/basics/_examples/DL-101-137-117 +++ b/docs/basics/_examples/DL-101-137-117 @@ -1,4 +1,4 @@ $ git log -n 3 --oneline -b602512 [DATALAD] Recorded changes -51d763e save my favourite Git joke -587ac34 finished my midterm project! +163f9e6 [DATALAD] Recorded changes +43af7fe save my favorite Git joke +11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-118 b/docs/basics/_examples/DL-101-137-118 index e327e1113..13927fcc7 100644 --- a/docs/basics/_examples/DL-101-137-118 +++ b/docs/basics/_examples/DL-101-137-118 @@ -1 +1,2 @@ -$ git reset --mixed 51d763ed5883bc51f6eabc36248e4ef80b9f600f + +$ git reset --mixed 43af7fea4a87d59bcfa985f71dd401c041831eef diff --git a/docs/basics/_examples/DL-101-137-119 b/docs/basics/_examples/DL-101-137-119 index 008555927..4389c5f79 100644 --- a/docs/basics/_examples/DL-101-137-119 +++ b/docs/basics/_examples/DL-101-137-119 @@ -1,3 +1,3 @@ $ git log -n 2 --oneline -51d763e save my favourite Git joke -587ac34 finished my midterm project! +43af7fe save my favorite Git joke +11c2ecb add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-121 b/docs/basics/_examples/DL-101-137-121 index 0be81d823..3950aa8cd 100644 --- a/docs/basics/_examples/DL-101-137-121 +++ b/docs/basics/_examples/DL-101-137-121 @@ -1,21 +1,21 @@ $ git log -n 20 --oneline -51d763e save my favourite Git joke -587ac34 finished my midterm project! -de7d909 [DATALAD] Recorded changes -37d74af add note on DataLads procedures -264d668 add note on configurations and git config -2277fe7 Add note on adding siblings -f015735 Merge remote-tracking branch 'refs/remotes/roommate/master' -0a01e8a Include nesting demo from datalad website -5e45b4e add note about datalad update -e613db1 add note on git annex whereis -118eed9 add note about installing from paths and recursive installations -f42379c add note on clean datasets -475d3de [DATALAD RUNCMD] Resize logo for slides -6856e4f [DATALAD RUNCMD] Resize logo for slides -72fa9f9 add additional notes on run options -b49af2f [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -f59a783 resized picture by hand -4698deb [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -cebdc20 add note on basic datalad run and datalad rerun -dd71a3c add note datalad and git diff +43af7fe save my favorite Git joke +11c2ecb add container and execute analysis within container +5e4c89a finished my midterm project! +ca88890 [DATALAD] Recorded changes +770e708 add note on DataLads procedures +ad89f03 add note on configurations and git config +0686627 Add note on adding siblings +26b599c Merge remote-tracking branch 'refs/remotes/roommate/master' +029add8 add note about datalad update +a2d01ff Include nesting demo from datalad website +2091c14 add note on git annex whereis +e2210e8 add note about installing from paths and recursive installations +66f97f3 add note on clean datasets +17e39f4 [DATALAD RUNCMD] Resize logo for slides +68ade50 [DATALAD RUNCMD] Resize logo for slides +048ae39 add additional notes on run options +fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +8cdc201 resized picture by hand +11dfb75 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +5ca7d13 add note on basic datalad run and datalad rerun diff --git a/docs/basics/_examples/DL-101-137-122 b/docs/basics/_examples/DL-101-137-122 index a9f08570c..c22ad18ab 100644 --- a/docs/basics/_examples/DL-101-137-122 +++ b/docs/basics/_examples/DL-101-137-122 @@ -1,14 +1,21 @@ -$ git checkout f59a78356d78525239347ecf763b6677130898af + +$ git checkout fa3fca41ffaafbfd1657203cd380273a70704c1e warning: unable to rmdir 'midterm_project': Directory not empty -Note: checking out 'f59a78356d78525239347ecf763b6677130898af'. +Note: switching to 'fa3fca41ffaafbfd1657203cd380273a70704c1e'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this -state without impacting any branches by performing another checkout. +state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may -do so (now or later) by using -b with the checkout command again. Example: +do so (now or later) by using -c with the switch command. Example: + + git switch -c + +Or undo this operation with: + + git switch - - git checkout -b +Turn off this advice by setting config variable advice.detachedHead to false -HEAD is now at f59a783 resized picture by hand +HEAD is now at fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... diff --git a/docs/basics/_examples/DL-101-137-124 b/docs/basics/_examples/DL-101-137-124 index cc8977241..034db2d13 100644 --- a/docs/basics/_examples/DL-101-137-124 +++ b/docs/basics/_examples/DL-101-137-124 @@ -1,3 +1,3 @@ $ git checkout master -Previous HEAD position was f59a783 resized picture by hand +Previous HEAD position was fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... Switched to branch 'master' diff --git a/docs/basics/_examples/DL-101-137-127 b/docs/basics/_examples/DL-101-137-127 index 932dbbb14..d85a6b40b 100644 --- a/docs/basics/_examples/DL-101-137-127 +++ b/docs/basics/_examples/DL-101-137-127 @@ -1,4 +1,5 @@ -$ git cat-file --textconv f59a78356d78525239347ecf763b6677130898af:notes.txt + +$ git cat-file --textconv fa3fca41ffaafbfd1657203cd380273a70704c1e:notes.txt One can create a new dataset with 'datalad create [--description] PATH'. The dataset is created empty diff --git a/docs/basics/_examples/DL-101-137-132 b/docs/basics/_examples/DL-101-137-132 index 100a3bbdf..7c9dee39a 100644 --- a/docs/basics/_examples/DL-101-137-132 +++ b/docs/basics/_examples/DL-101-137-132 @@ -1,3 +1,3 @@ $ git log -n 2 --oneline -b6dd7d1 add joke evaluation to joke -51d763e save my favourite Git joke +9ab836f add joke evaluation to joke +43af7fe save my favorite Git joke diff --git a/docs/basics/_examples/DL-101-137-133 b/docs/basics/_examples/DL-101-137-133 index 36b00e1f2..c2c5dd565 100644 --- a/docs/basics/_examples/DL-101-137-133 +++ b/docs/basics/_examples/DL-101-137-133 @@ -1,2 +1,3 @@ -$ git reset --hard 51d763ed5883bc51f6eabc36248e4ef80b9f600f -HEAD is now at 51d763e save my favourite Git joke + +$ git reset --hard 43af7fea4a87d59bcfa985f71dd401c041831eef +HEAD is now at 43af7fe save my favorite Git joke diff --git a/docs/basics/_examples/DL-101-137-144 b/docs/basics/_examples/DL-101-137-144 index e1fe1ca05..37aa182b8 100644 --- a/docs/basics/_examples/DL-101-137-144 +++ b/docs/basics/_examples/DL-101-137-144 @@ -1,5 +1,5 @@ -$ git revert aaa1988adfa83a703a756f911a0beffc4fefa08a -[master 37c73e7] Revert "did a bad modification" - Date: Mon Nov 18 08:44:48 2019 +0100 +$ git revert 7f888a04183c89c7e33e81cada7f89544b7b9c81 +[master 029320e] Revert "did a bad modification" + Date: Tue Nov 26 12:54:42 2019 +0100 1 file changed, 1 deletion(-) diff --git a/docs/basics/_examples/DL-101-137-146 b/docs/basics/_examples/DL-101-137-146 index 1d143d932..882550552 100644 --- a/docs/basics/_examples/DL-101-137-146 +++ b/docs/basics/_examples/DL-101-137-146 @@ -1,20 +1,20 @@ $ git log -n 3 -commit 37c73e7eff6df09ce3ea9a149e64ea5e82ed5d8b +commit 029320e95e2d7633aa447e07fd90d9a3ef8374ef Author: Elena Piscopia -Date: Mon Nov 18 08:44:48 2019 +0100 +Date: Tue Nov 26 12:54:42 2019 +0100 Revert "did a bad modification" - This reverts commit aaa1988adfa83a703a756f911a0beffc4fefa08a. + This reverts commit 7f888a04183c89c7e33e81cada7f89544b7b9c81. -commit 696c3878120b997eb3f6b4e552de5875664ab874 +commit ff291229226a01b85429ce6f708e80bd68fa1fbb Author: Elena Piscopia -Date: Mon Nov 18 08:44:48 2019 +0100 +Date: Tue Nov 26 12:54:42 2019 +0100 add note on helpful git resource -commit aaa1988adfa83a703a756f911a0beffc4fefa08a +commit 7f888a04183c89c7e33e81cada7f89544b7b9c81 Author: Elena Piscopia -Date: Mon Nov 18 08:44:48 2019 +0100 +Date: Tue Nov 26 12:54:41 2019 +0100 did a bad modification diff --git a/docs/basics/_examples/DL-101-145-101 b/docs/basics/_examples/DL-101-145-101 deleted file mode 100644 index d46fa9cc9..000000000 --- a/docs/basics/_examples/DL-101-145-101 +++ /dev/null @@ -1,9 +0,0 @@ -# we are in the midterm_project subdataset -$ datalad containers-add python --url shub://adswa/resources:1 -add(ok): .datalad/config (file) -save(ok): . (dataset) -containers_add(ok): /home/me/dl-101/DataLad-101/midterm_project/.datalad/environments/python/image (file) -action summary: - add (ok: 1) - containers_add (ok: 1) - save (ok: 1) diff --git a/docs/basics/_examples/DL-101-145-102 b/docs/basics/_examples/DL-101-145-102 deleted file mode 100644 index d24f71716..000000000 --- a/docs/basics/_examples/DL-101-145-102 +++ /dev/null @@ -1,7 +0,0 @@ -$ cat .datalad/config -[datalad "dataset"] - id = 942371e6-04a0-11ea-99ca-d0c637c523bc -[datalad "containers.python"] - updateurl = shub://adswa/resources:1 - image = .datalad/environments/python/image - cmdexec = singularity exec {img} {cmd} diff --git a/docs/basics/_examples/DL-101-145-103 b/docs/basics/_examples/DL-101-145-103 deleted file mode 100644 index f82332308..000000000 --- a/docs/basics/_examples/DL-101-145-103 +++ /dev/null @@ -1,26 +0,0 @@ -$ git log -n 1 -p -commit 3cdab542d2749d3812d0d71ece030acabfc9f6d9 -Author: Elena Piscopia -Date: Mon Nov 11 17:33:49 2019 +0100 - - [DATALAD] Configure containerized environment 'python' - -diff --git a/.datalad/config b/.datalad/config -index 71b5f13..0500889 100644 ---- a/.datalad/config -+++ b/.datalad/config -@@ -1,2 +1,6 @@ - [datalad "dataset"] - id = 942371e6-04a0-11ea-99ca-d0c637c523bc -+[datalad "containers.python"] -+ updateurl = shub://adswa/resources:1 -+ image = .datalad/environments/python/image -+ cmdexec = singularity exec {img} {cmd} -diff --git a/.datalad/environments/python/image b/.datalad/environments/python/image -new file mode 120000 -index 0000000..800282a ---- /dev/null -+++ b/.datalad/environments/python/image -@@ -0,0 +1 @@ -+../../../.git/annex/objects/zJ/8f/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10 -\ No newline at end of file diff --git a/docs/basics/_examples/DL-101-145-104 b/docs/basics/_examples/DL-101-145-104 deleted file mode 100644 index 457d879f3..000000000 --- a/docs/basics/_examples/DL-101-145-104 +++ /dev/null @@ -1,23 +0,0 @@ - -datalad containers-run -m rerun analysis in container --container-name python datalad rerun a39d7907214b1669612281a23a50656288c87735 -[INFO] Making sure inputs are available (this may take some time) -[INFO] == Command start (output follows) ===== -[INFO] Making sure inputs are available (this may take some time) -[INFO] == Command start (output follows) ===== -/usr/lib/python3/dist-packages/sklearn/externals/joblib.py:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses - import imp -[INFO] == Command exit (modification check follows) ===== -unlock(ok): output/prediction_report.csv (file) -unlock(ok): output/pairwise_relationships.png (file) -add(ok): output/pairwise_relationships.png (file) -add(ok): output/prediction_report.csv (file) -save(ok): /home/me/dl-101/DataLad-101/midterm_project (dataset) -action summary: - add (ok: 2) - get (notneeded: 2) - save (ok: 1) - unlock (ok: 2) -[INFO] == Command exit (modification check follows) ===== -action summary: - get (notneeded: 1) - save (notneeded: 2) diff --git a/docs/basics/_examples/DL-101-145-110 b/docs/basics/_examples/DL-101-145-110 deleted file mode 100644 index ddba701f3..000000000 --- a/docs/basics/_examples/DL-101-145-110 +++ /dev/null @@ -1,2 +0,0 @@ -$ datalad containers-list -python -> .datalad/environments/python/image diff --git a/docs/intro/_examples/how-to-1 b/docs/intro/_examples/how-to-1 index 59064f2de..22bd89f0b 100644 --- a/docs/intro/_examples/how-to-1 +++ b/docs/intro/_examples/how-to-1 @@ -1,5 +1,5 @@ $ ls -l output.txt 24+0 records in 24+0 records out -25165824 bytes (25 MB, 24 MiB) copied, 0.0195906 s, 1.3 GB/s --rw-r--r-- 1 adina adina 25165824 Nov 18 10:53 output.txt +25165824 bytes (25 MB, 24 MiB) copied, 0.013529 s, 1.9 GB/s +-rw-r--r-- 1 adina adina 25165824 Nov 26 12:54 output.txt diff --git a/docs/intro/_examples/how-to-2 b/docs/intro/_examples/how-to-2 index dcd6c3111..dff83e2d5 100644 --- a/docs/intro/_examples/how-to-2 +++ b/docs/intro/_examples/how-to-2 @@ -1,4 +1,4 @@ $ ls -lh output.txt # note that short options can be combined! # or alternatively $ ls -l --human-readable output.txt --rw-r--r-- 1 adina adina 24M Nov 18 10:53 output.txt +-rw-r--r-- 1 adina adina 24M Nov 26 12:54 output.txt From f21e65e8188c68251bc53a31a8afebd998929b76 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 12:58:52 +0100 Subject: [PATCH 16/33] add new examples --- docs/basics/_examples/DL-101-133-101 | 9 ++++++ docs/basics/_examples/DL-101-133-102 | 7 +++++ docs/basics/_examples/DL-101-133-103 | 26 ++++++++++++++++ docs/basics/_examples/DL-101-133-104 | 19 ++++++++++++ docs/basics/_examples/DL-101-133-110 | 2 ++ docs/basics/_examples/DL-101-133-111 | 45 ++++++++++++++++++++++++++++ docs/basics/_examples/DL-101-133-112 | 3 ++ docs/basics/_examples/DL-101-133-113 | 6 ++++ 8 files changed, 117 insertions(+) create mode 100644 docs/basics/_examples/DL-101-133-101 create mode 100644 docs/basics/_examples/DL-101-133-102 create mode 100644 docs/basics/_examples/DL-101-133-103 create mode 100644 docs/basics/_examples/DL-101-133-104 create mode 100644 docs/basics/_examples/DL-101-133-110 create mode 100644 docs/basics/_examples/DL-101-133-111 create mode 100644 docs/basics/_examples/DL-101-133-112 create mode 100644 docs/basics/_examples/DL-101-133-113 diff --git a/docs/basics/_examples/DL-101-133-101 b/docs/basics/_examples/DL-101-133-101 new file mode 100644 index 000000000..d46fa9cc9 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-101 @@ -0,0 +1,9 @@ +# we are in the midterm_project subdataset +$ datalad containers-add python --url shub://adswa/resources:1 +add(ok): .datalad/config (file) +save(ok): . (dataset) +containers_add(ok): /home/me/dl-101/DataLad-101/midterm_project/.datalad/environments/python/image (file) +action summary: + add (ok: 1) + containers_add (ok: 1) + save (ok: 1) diff --git a/docs/basics/_examples/DL-101-133-102 b/docs/basics/_examples/DL-101-133-102 new file mode 100644 index 000000000..1d1d75fc9 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-102 @@ -0,0 +1,7 @@ +$ cat .datalad/config +[datalad "dataset"] + id = 641bc4fe-1043-11ea-97f1-e86a64c8054c +[datalad "containers.python"] + updateurl = shub://adswa/resources:1 + image = .datalad/environments/python/image + cmdexec = singularity exec {img} {cmd} diff --git a/docs/basics/_examples/DL-101-133-103 b/docs/basics/_examples/DL-101-133-103 new file mode 100644 index 000000000..1c5073a5e --- /dev/null +++ b/docs/basics/_examples/DL-101-133-103 @@ -0,0 +1,26 @@ +$ git log -n 1 -p +commit e3b5fe16b5227b5e792b013fa20e433fa71121f5 +Author: Elena Piscopia +Date: Tue Nov 26 12:54:17 2019 +0100 + + [DATALAD] Configure containerized environment 'python' + +diff --git a/.datalad/config b/.datalad/config +index af6e089..6674d81 100644 +--- a/.datalad/config ++++ b/.datalad/config +@@ -1,2 +1,6 @@ + [datalad "dataset"] + id = 641bc4fe-1043-11ea-97f1-e86a64c8054c ++[datalad "containers.python"] ++ updateurl = shub://adswa/resources:1 ++ image = .datalad/environments/python/image ++ cmdexec = singularity exec {img} {cmd} +diff --git a/.datalad/environments/python/image b/.datalad/environments/python/image +new file mode 120000 +index 0000000..800282a +--- /dev/null ++++ b/.datalad/environments/python/image +@@ -0,0 +1 @@ ++../../../.git/annex/objects/zJ/8f/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10 +\ No newline at end of file diff --git a/docs/basics/_examples/DL-101-133-104 b/docs/basics/_examples/DL-101-133-104 new file mode 100644 index 000000000..d87efec6c --- /dev/null +++ b/docs/basics/_examples/DL-101-133-104 @@ -0,0 +1,19 @@ +$ datalad containers-run -m "rerun analysis in container" \ + --container-name python \ + --input "input/iris.csv" \ + --output "prediction_report.csv" \ + --output "pairwise_relationships.png" \ + "python3 code/script.py" +[INFO] Making sure inputs are available (this may take some time) +[INFO] == Command start (output follows) ===== +[INFO] == Command exit (modification check follows) ===== +unlock(ok): pairwise_relationships.png (file) +unlock(ok): prediction_report.csv (file) +add(ok): pairwise_relationships.png (file) +add(ok): prediction_report.csv (file) +save(ok): . (dataset) +action summary: + add (ok: 2) + get (notneeded: 2) + save (notneeded: 1, ok: 1) + unlock (ok: 2) diff --git a/docs/basics/_examples/DL-101-133-110 b/docs/basics/_examples/DL-101-133-110 new file mode 100644 index 000000000..ddba701f3 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-110 @@ -0,0 +1,2 @@ +$ datalad containers-list +python -> .datalad/environments/python/image diff --git a/docs/basics/_examples/DL-101-133-111 b/docs/basics/_examples/DL-101-133-111 new file mode 100644 index 000000000..77ac150eb --- /dev/null +++ b/docs/basics/_examples/DL-101-133-111 @@ -0,0 +1,45 @@ +$ git log -p -n 1 +commit a70122fd64dd7afb6173ea7e6efad0011e5a9c73 +Author: Elena Piscopia +Date: Tue Nov 26 12:54:22 2019 +0100 + + [DATALAD RUNCMD] rerun analysis in container + + === Do not change lines below === + { + "chain": [], + "cmd": "singularity exec .datalad/environments/python/image python3 code/script.py", + "dsid": "641bc4fe-1043-11ea-97f1-e86a64c8054c", + "exit": 0, + "extra_inputs": [ + ".datalad/environments/python/image" + ], + "inputs": [ + "input/iris.csv" + ], + "outputs": [ + "prediction_report.csv", + "pairwise_relationships.png" + ], + "pwd": "." + } + ^^^ Do not change lines above ^^^ + +diff --git a/pairwise_relationships.png b/pairwise_relationships.png +index 2f69f64..6d00014 120000 +--- a/pairwise_relationships.png ++++ b/pairwise_relationships.png +@@ -1 +1 @@ +-.git/annex/objects/Pz/Xm/MD5E-s175662--8a9a3e225267f327e1671cf6d01b1957.png/MD5E-s175662--8a9a3e225267f327e1671cf6d01b1957.png +\ No newline at end of file ++.git/annex/objects/z3/23/MD5E-s176597--87d8a72f5f7b1f4f191d0be1bfd15288.png/MD5E-s176597--87d8a72f5f7b1f4f191d0be1bfd15288.png +\ No newline at end of file +diff --git a/prediction_report.csv b/prediction_report.csv +index 42d194b..b46a2d5 120000 +--- a/prediction_report.csv ++++ b/prediction_report.csv +@@ -1 +1 @@ +-.git/annex/objects/8q/6M/MD5E-s345--a88cab39b1a5ec59ace322225cc88bc9.csv/MD5E-s345--a88cab39b1a5ec59ace322225cc88bc9.csv +\ No newline at end of file ++.git/annex/objects/VF/27/MD5E-s347--7d984f53676358222aa7aa55980f205b.csv/MD5E-s347--7d984f53676358222aa7aa55980f205b.csv +\ No newline at end of file diff --git a/docs/basics/_examples/DL-101-133-112 b/docs/basics/_examples/DL-101-133-112 new file mode 100644 index 000000000..875f3cfa2 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-112 @@ -0,0 +1,3 @@ +$ cd ../ +$ datalad status + modified: midterm_project (dataset) diff --git a/docs/basics/_examples/DL-101-133-113 b/docs/basics/_examples/DL-101-133-113 new file mode 100644 index 000000000..92d0e7121 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-113 @@ -0,0 +1,6 @@ +$ datalad save -m "add container and execute analysis within container" +add(ok): midterm_project (file) +save(ok): . (dataset) +action summary: + add (ok: 1) + save (ok: 1) From 18edcfcde00dbd5966a8bc7a4d0991b6283b240f Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 13:03:26 +0100 Subject: [PATCH 17/33] add missing word --- docs/basics/101-133-containersrun.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index 738cebdf8..3c0f34426 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -279,7 +279,7 @@ the most recent state of the subdataset to the superdataset ``DataLad-101``. $ datalad save -m "add container and execute analysis within container" -Software containers, the ``datalad-containers``, and DataLad thus work well together +Software containers, the ``datalad-containers`` extension, and DataLad thus work well together to make your analysis completely reproducible -- by not only linking code, data, and outputs, but also the software environment of an analysis. And this does not only benefit your future self, but also whomever you share your dataset with, as From ab9f697aca7e960472c9ebe722aacd9ff06c83f5 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Tue, 26 Nov 2019 13:07:53 +0100 Subject: [PATCH 18/33] add index entries --- docs/basics/101-133-containersrun.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index 3c0f34426..037b40863 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -107,6 +107,9 @@ Singularity (even without having Docker installed). Using ``datalad containers`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. index:: ! datalad command; containers-add +.. index:: ! datalad command; containers-run + One core feature of the ``datalad containers`` extension is that it registers computational containers to a dataset. This is done with the :command:`datalad containers-add` command. @@ -236,6 +239,9 @@ The complete command's structure looks like this:: $ datalad containers-run --name [-m ...] [--input ...] [--output ...] +.. index:: ! datalad command; containers-remove +.. index:: ! datalad command; containers-list + .. findoutmore:: How can I list available containers or remove them? The command :command:`datalad containers-list` will list all containers in From 11f9348dc1e56176601891932871d3e6a26cf9dd Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 27 Nov 2019 08:56:05 +0100 Subject: [PATCH 19/33] add short summary --- docs/basics/101-134-summary.rst | 57 +++++++++++++++++++++++++++++++++ docs/contents.rst.inc | 1 + 2 files changed, 58 insertions(+) create mode 100644 docs/basics/101-134-summary.rst diff --git a/docs/basics/101-134-summary.rst b/docs/basics/101-134-summary.rst new file mode 100644 index 000000000..f04d60958 --- /dev/null +++ b/docs/basics/101-134-summary.rst @@ -0,0 +1,57 @@ +.. _summary_containers: + +Summary +------- + +The last two sections have first of all extended your knowledge on dataset nesting: + +- When subdatasets are created or installed, they are registered to the superdataset + in their current version state (as identified by their most recent commit's hash). + For a freshly created subdatasets, the most recent commit is at the same time its + first commit. + +- Once the subdataset evolves, the superdataset recognizes this as a ``modification`` + of the subdatasets version state. If you want to record this, you need to + :command:`save` it in the superdataset:: + + $ datalad save -m "a short summary of changes in subds" + +But more than nesting concepts, they have also extended your knowledge on +reproducible analyses with :command:`datalad run` and you have experienced +for yourself why and how software containers can go hand-in-hand with DataLad: + +- A software container encapsulates a complete software environment, independent + from the environment of the computer it runs on. This allows you to create or + use secluded software and also share it together with your analysis to ensure + computational reproducibility. + +- The command :command:`datalad containers-add` registers an Image from a path or + url to your dataset. + +- If you use :command:`datalad containers-run` instead of :command:`datalad run`, + you can reproducibly execute a command of your choice *within* the software + environment. + +- A :command:`datalad rerun` of a commit produced with :command:`datalad containers-run` + will re-execute the command in the same software environment. + +Now what can I do with it? +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For one, you will not be surprised if you ever see a subdataset being shown as +``modified`` by :command:`datalad status`: You now know that if a subdataset +evolves, it's most recent state needs to be explicitly saved to the superdatasets +history. + +On a different matter, you are now able to capture and share analysis provenance that +includes the relevant software environment. This does not only make your analyses +projects automatically reproducible, but automatically *computationally* reproducible - +you can make sure that your analyses runs on any computer, regardless of the software +environment on this computer. Even if you are unsure how you can wrap up an +environment into a software container Image at this point, you could make use of +hundreds of publicly available Images on `Singularity-Hub `_ and +`Docker-Hub `_. + +With this, you have also gotten a first glimpse into an extension of DataLad: A +Python module you can install with Python package managers such as ``pip`` that +extends DataLad's functionality. diff --git a/docs/contents.rst.inc b/docs/contents.rst.inc index b9ee8f754..e5ceb88b8 100644 --- a/docs/contents.rst.inc +++ b/docs/contents.rst.inc @@ -107,6 +107,7 @@ One step further basics/101-132-advancednesting basics/101-133-containersrun + basics/101-134-summary basics/101-132-gitC From d38577b2bd54191ca0575feb874a358169f66caa Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 27 Nov 2019 08:56:21 +0100 Subject: [PATCH 20/33] link usecase, remove todos --- docs/basics/101-133-containersrun.rst | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index 037b40863..825ffaadd 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -41,12 +41,6 @@ This section will give a quick overview on what containers are and demonstrate how ``datalad-containers`` helps to capture full provenance of an analysis by linking containers to datasets and analyses. -.. todo:: - - - link to neuroimaging use case - - maybe mention the 10 year code challenge by ReScience - - Containers ^^^^^^^^^^ @@ -268,7 +262,14 @@ Here is how the history entry looks like: $ git log -p -n 1 -Because this modified the ``midterm_project`` subdirectory, we need to also save +If you would :command:`rerun` this commit, it would be re-executed in the +software container registered to the dataset. If you would share the dataset +with a friend and they would :command:`rerun` this commit, the Image would first +be obtained from its registered url, and thus your +friend can obtain the correct execution environment automatically. + +Note that because this new :command:`containers-run` command modified the +``midterm_project`` subdirectory, we need to also save the most recent state of the subdataset to the superdataset ``DataLad-101``. .. runrecord:: _examples/DL-101-133-112 @@ -292,6 +293,9 @@ only benefit your future self, but also whomever you share your dataset with, as the information about the container is shared together with the dataset. How cool is that? +If you are interested in more, you can read about another example of :command:`datalad containers-run` +in the usecase :ref:`usecase_reproduce_neuroimg`. + .. rubric:: Footnotes .. [#f1] To read more about DataLad's extensions, see section :ref:`extensions_intro`. From 1c2e57dbe3ace685ed6fd02d8b16d40e6e891581 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 27 Nov 2019 08:59:40 +0100 Subject: [PATCH 21/33] remove standalone git -C, instead integrate into exisiting findoutmore --- docs/basics/101-106-nesting.rst | 10 ++++++++-- docs/basics/101-132-gitC.rst | 15 --------------- docs/contents.rst.inc | 1 - 3 files changed, 8 insertions(+), 18 deletions(-) delete mode 100644 docs/basics/101-132-gitC.rst diff --git a/docs/basics/101-106-nesting.rst b/docs/basics/101-106-nesting.rst index ef3f22535..1793e4f26 100644 --- a/docs/basics/101-106-nesting.rst +++ b/docs/basics/101-106-nesting.rst @@ -84,8 +84,14 @@ we can set subdatasets to previous states, or *update* them. Previously, we used :command:`cd` to navigate into the subdataset, and subsequently opened the Git log. This is necessary, because a :command:`git log` - in the superdataset would only return the superdatasets history. There is one - trick, though: ``git -C`` lets you perform any Git command in a provided path. + in the superdataset would only return the superdatasets history. + While moving around with ``cd`` is straightforward, you also found it + slightly annoying from time to time to use the ``cd`` command so often and also + to remember in which directory you currently are in. There is one + trick, though: ``git -C`` (note that it is a capital C) lets you perform any + Git command in a provided path. Providing this option together with a path to + a Git command let's you run the command as if Git was started in this path + instead of the current working directory. Thus, from the root of ``DataLad-101``, this command would have given you the subdatasets history as well:: diff --git a/docs/basics/101-132-gitC.rst b/docs/basics/101-132-gitC.rst deleted file mode 100644 index e5b21c348..000000000 --- a/docs/basics/101-132-gitC.rst +++ /dev/null @@ -1,15 +0,0 @@ -Working in (sub-)datasets from afar -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Up until now, we made sure to diligently ``cd`` into the precise dataset or -subdataset we wanted to work with. This was straightforward and easy, but, to -be honest, you found it slightly annoying from time to time as well. - -Luckily, Git has an option to run commands in a path different from the current -working directory: ``-C`` (note that it is a capital C). Providing this -option together with a path to any Git command let's you run the command as -if Git was started in this path instead of the current working directory. - -.. todo:: - - this needs some examples and practice. \ No newline at end of file diff --git a/docs/contents.rst.inc b/docs/contents.rst.inc index e5ceb88b8..d6378e479 100644 --- a/docs/contents.rst.inc +++ b/docs/contents.rst.inc @@ -108,7 +108,6 @@ One step further basics/101-132-advancednesting basics/101-133-containersrun basics/101-134-summary - basics/101-132-gitC ############# From 6041795dca4b0d7568bb3ceff37d704c987bc072 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 27 Nov 2019 09:07:06 +0100 Subject: [PATCH 22/33] reference relevant sections in extension overview --- docs/basics/101-144-intro_extensions.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/basics/101-144-intro_extensions.rst b/docs/basics/101-144-intro_extensions.rst index 588f45943..05fa6afea 100644 --- a/docs/basics/101-144-intro_extensions.rst +++ b/docs/basics/101-144-intro_extensions.rst @@ -28,7 +28,8 @@ The following DataLad extensions are currently available: * - `DataLad Container `_ - Equips DataLad's :command:`run`/:command:`rerun` functionality with the ability to transparently execute commands in containerized - computational environments. + computational environments. The section :ref:`containersrun` demonstrates + how this extension can be used, as well as the usecase :ref:`usecase_reproduce_neuroimg`. * - `DataLad Neuroimaging `_ - Metadata extraction support for a range of standards common to neuroimaging data. From 5dff8b631dd14aefcda9d7c045c3a9e550af11a9 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Thu, 28 Nov 2019 08:41:27 +0100 Subject: [PATCH 23/33] add -d option to save when saving subds updates, also add hidden section to explain more --- docs/basics/101-132-advancednesting.rst | 33 +++++++++++++++++++++++-- docs/basics/101-133-containersrun.rst | 2 +- 2 files changed, 32 insertions(+), 3 deletions(-) diff --git a/docs/basics/101-132-advancednesting.rst b/docs/basics/101-132-advancednesting.rst index 33c005664..e7bc380f7 100644 --- a/docs/basics/101-132-advancednesting.rst +++ b/docs/basics/101-132-advancednesting.rst @@ -49,13 +49,42 @@ but it is worth repeating: If you modify a subdataset, you will need to save this *in the superdataset* in order to have a clean superdataset status. Let's save the modification of the subdataset into the history of the -superdataset: +superdataset. For this, to avoid confusion, you can specify explicitly to +which dataset you want to save a modification. ``-d .`` specifies the current +dataset, i.e., ``DataLad-101``, as the dataset to save to: .. runrecord:: _examples/DL-101-132-103 :language: console :workdir: dl-101/DataLad-101/ - $ datalad save -m "finished my midterm project!" + $ datalad save -d . -m "finished my midterm project!" midterm_project + +.. findoutmore:: More on how save can operate on nested datasets + + In a superdataset with subdatasets, :command:`datalad save` by default + tries to figure out on its own which dataset's history of all available + datasets a :command:`save` should be written to. However, it can reduce + confusion or allow specific operations to be very explicit in the command + call and tell DataLad where to save what kind of modifications to. + + If you want to save the current state of the subdataset into the superdataset + (as necessary here), start a ``save`` from the superdataset and have the + ``-d/--dataset`` option point to its root:: + + # in the root of the superds + $ datalad save -d . -m "update subdataset" + + If you are in the superdataset, and you want to save an unsaved modification + in a subdataset to the *subdatasets* history, let ``-d/--dataset`` point to + the subdataset:: + + # in the superds + $ datalad save -d path/to/subds -m "modified XY" + + The recursive option allows you to save any content underneath the specified + directory, and recurse into any potential subdatasets:: + + $ datalad save . --recursive Let's check which subproject commit is now recorded in the superdataset: diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index 825ffaadd..267afd718 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -283,7 +283,7 @@ the most recent state of the subdataset to the superdataset ``DataLad-101``. :language: console :workdir: dl-101/DataLad-101 - $ datalad save -m "add container and execute analysis within container" + $ datalad save -d . -m "add container and execute analysis within container" midterm_project Software containers, the ``datalad-containers`` extension, and DataLad thus work well together From d3b78e83aeea234a0712dc76db054ce0561312b3 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Thu, 28 Nov 2019 08:42:05 +0100 Subject: [PATCH 24/33] update save examples --- docs/basics/_examples/DL-101-132-103 | 2 +- docs/basics/_examples/DL-101-133-113 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/basics/_examples/DL-101-132-103 b/docs/basics/_examples/DL-101-132-103 index db6af24ce..0f7f472f7 100644 --- a/docs/basics/_examples/DL-101-132-103 +++ b/docs/basics/_examples/DL-101-132-103 @@ -1,4 +1,4 @@ -$ datalad save -m "finished my midterm project!" +$ datalad save -d . -m "finished my midterm project!" midterm_project add(ok): midterm_project (file) save(ok): . (dataset) action summary: diff --git a/docs/basics/_examples/DL-101-133-113 b/docs/basics/_examples/DL-101-133-113 index 92d0e7121..8bb646ba9 100644 --- a/docs/basics/_examples/DL-101-133-113 +++ b/docs/basics/_examples/DL-101-133-113 @@ -1,4 +1,4 @@ -$ datalad save -m "add container and execute analysis within container" +$ datalad save -d . -m "add container and execute analysis within container" midterm_project add(ok): midterm_project (file) save(ok): . (dataset) action summary: From 3285e6e0c68689838657f8e93df1050d61bc883f Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 4 Dec 2019 11:53:31 +0100 Subject: [PATCH 25/33] tweak extensions intro --- docs/basics/101-144-intro_extensions.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/basics/101-144-intro_extensions.rst b/docs/basics/101-144-intro_extensions.rst index 05fa6afea..7ffaba7fd 100644 --- a/docs/basics/101-144-intro_extensions.rst +++ b/docs/basics/101-144-intro_extensions.rst @@ -4,12 +4,15 @@ DataLad's extensions -------------------- The commands DataLad provides cover a broad range of domain-agnostic use cases. -However, there is a vast supply of extension packages that can add -(domain-specific) functionality and new commands. +However, there are extension packages that can add (domain-specific) +functionality and new commands. -Such extensions are shipped as proper Python packages, and are *not* included in +Such extensions are shipped as separate Python packages, and are *not* included in DataLad itself. Instead, users with the need for a particular extension can -install the extension package on top of DataLad. The installation is done with +install the extension package -- either on top of DataLad if DataLad is already +installed, or on its own (the extension will then pull in DataLad core +automatically, with no need to first or simultaneously install DataLad itself +explicitly). The installation is done with standard Python package managers, such as :term:`pip`, and beyond installation of the package, no additional setup is required. From 885ea9d4840d7aee62c39868de074584e1011e7c Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 4 Dec 2019 11:54:07 +0100 Subject: [PATCH 26/33] strip unmentioned extensions from overview, link PyPi search for overview on available extensions --- docs/basics/101-144-intro_extensions.rst | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/docs/basics/101-144-intro_extensions.rst b/docs/basics/101-144-intro_extensions.rst index 7ffaba7fd..e4c6a3422 100644 --- a/docs/basics/101-144-intro_extensions.rst +++ b/docs/basics/101-144-intro_extensions.rst @@ -16,11 +16,8 @@ explicitly). The installation is done with standard Python package managers, such as :term:`pip`, and beyond installation of the package, no additional setup is required. -The following DataLad extensions are currently available: - -.. todo:: - - Which extensions do you want to see mentioned (and which not)? +Among others (a full list can be found on `PyPi `_), +the following DataLad extensions are available: .. list-table:: :widths: 50 100 @@ -35,17 +32,15 @@ The following DataLad extensions are currently available: how this extension can be used, as well as the usecase :ref:`usecase_reproduce_neuroimg`. * - `DataLad Neuroimaging `_ - Metadata extraction support for a range of standards common to - neuroimaging data. + neuroimaging data. The usecase :ref:`usecase_reproduce_neuroimg` demonstrates + how this extension can be used. * - `DataLad Metalad `_ - - Equips DataLad with an alternative command suite for metadata handling - (extraction, aggregation, reporting). - * - `DataLad Webapp `_ - - DataLad extension for exposing a REST API of selected functionality. - * - `DataLad Crawler `_ - - Datalad for crawling web resources and automated data distributions. - * - `DataLad HTCondor `_ - - Remote code execution for DataLad via HTCondor. + - Equips DataLad with an alternative command suite and advanced tooling + for metadata handling (extraction, aggregation, reporting). + + .. todo:: + once section on metadata is done, link it here To install a DataLad extension, use From 5a321aac514b9c8a801b767d20050df8597f3a68 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 4 Dec 2019 12:26:25 +0100 Subject: [PATCH 27/33] first bunch of tweaks based on mihs comments --- docs/basics/101-133-containersrun.rst | 26 +++++++++++++++++--------- docs/basics/101-134-summary.rst | 4 ++-- 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index 267afd718..d9c17f30e 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -15,7 +15,7 @@ you mean", you hear in return. "On my machine, everything works fine!" -On its own, DataLad datasets can contain almost everything that is relevant to +On its own, DataLad datasets can contain almost anything that is relevant to ensure reproducibility: Data, code, human-readable analysis descriptions (e.g., ``README.md`` files), provenance on the origin of all files obtained from elsewhere, and machine-readable records that link generated @@ -47,7 +47,7 @@ Containers To put it simple, computational containers are cut-down virtual machines that allow you to package all software libraries and their dependencies (all in the precise version your analysis requires) into a bundle you can share with -others. On your own and others machines, the container constitutes a secluded +others. On your own and other's machines, the container constitutes a secluded software environment that - contains the exact software environment that you specified, ready to run @@ -62,11 +62,13 @@ others (and also yourself) have easy access to the correct software without the need to modify the software environment of the machine the container runs on. Thus, containers are ideal to encapsulate the software environment and share it together with the analysis code and data to ensure -computational reproducibility of your analyses. +computational reproducibility of your analyses, or to create a suitable +software environment on a computer that you do not have permissions to deploy +software on. There are a number of different tools to create and use containers, with `Docker `_ being one of the most well-known of them. -While being a powerful tool, it can not be used on high performance computing +While being a powerful tool, it is only rarely used on high performance computing (HPC) infrastructure [#f2]_. An alternative is `Singularity `_. Both of these tools share core terminology: @@ -85,11 +87,14 @@ Both of these tools share core terminology: **Hub** A storage resource to share and consume images. There is `Singularity-Hub `_ and - `Docker-Hub `_. + `Docker-Hub `_. Both are optional, additional services + not required to use software containers, but a convenient way to share recipes + and have imaged built from them by a service (instead of building them + manually and locally). Note that as of now, the ``datalad-containers`` extensions only supports Singularity images, but support for Docker is being actively developed. -Singularity is however very compatible with Docker -- you can, for example, use +Singularity is however compatible with Docker -- you can use Docker images as a basis for Singularity images, or run Docker images with Singularity (even without having Docker installed). @@ -141,9 +146,12 @@ section below has some pointers: sudo singularity build - will build a container (called ````) from the recipe. Alternatively, - `Singularity Hub `_ integrates with Github - and builds containers from Images pushed to repositories on Github. + will build a container (called ````) from the recipe. Note that this + command requires ``root`` privileges ("``sudo``"). You can build the container + on any machine, though, not necessarily the one that is later supposed to + actually run the analysis, e.g., your own laptop versus a compute cluster. + Alternatively, `Singularity Hub `_ integrates + with Github and builds containers from Images pushed to repositories on Github. `The docs `_ give you a set of instructions on how to do this. diff --git a/docs/basics/101-134-summary.rst b/docs/basics/101-134-summary.rst index f04d60958..026169118 100644 --- a/docs/basics/101-134-summary.rst +++ b/docs/basics/101-134-summary.rst @@ -46,8 +46,8 @@ history. On a different matter, you are now able to capture and share analysis provenance that includes the relevant software environment. This does not only make your analyses projects automatically reproducible, but automatically *computationally* reproducible - -you can make sure that your analyses runs on any computer, regardless of the software -environment on this computer. Even if you are unsure how you can wrap up an +you can make sure that your analyses runs on any computer with Singularity, +regardless of the software environment on this computer. Even if you are unsure how you can wrap up an environment into a software container Image at this point, you could make use of hundreds of publicly available Images on `Singularity-Hub `_ and `Docker-Hub `_. From 7a118c93e38f9e596c8eb017749435ff525fba22 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 4 Dec 2019 13:05:45 +0100 Subject: [PATCH 28/33] clarify/fix definitions --- docs/basics/101-133-containersrun.rst | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index d9c17f30e..d73419c0e 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -73,16 +73,19 @@ While being a powerful tool, it is only rarely used on high performance computin .io/docs/>`_. Both of these tools share core terminology: +**Recipe** + A text file template that lists all required components of the computational environment. + It is made by a human user. + **Image** - A template or "recipe" from which containers are build. It lists all - required components of the computational environment, and is made by a - human user. If you want to create your own container, you start by writing - an Image file, but you can also *pull* a publicly shared image from the - *Hub* of the tool you are using. Based on images, containers are *build*. + This is *build* from the recipe file. It is a static filesystem inside a file, + populated with the software specified in the recipe, and some initial configuration. **Container** - The instance that you can actually use for your computations, *build* from - an image. + A running instance of an Image that you can actually use for your computations. + If you want to create and run your own software container, you start by writing + a recipe file and build an Image from it. Alternatively, you can can also *pull* + an Image built from a publicly shared recipe from the *Hub* of the tool you are using. **Hub** A storage resource to share and consume images. There is From 91578171e08f6e5228def703e10a658ca36755dc Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 4 Dec 2019 13:57:27 +0100 Subject: [PATCH 29/33] change container name, disclose recipe source --- docs/basics/101-133-containersrun.rst | 15 +++++++++------ docs/basics/_examples/DL-101-133-101 | 4 ++-- docs/basics/_examples/DL-101-133-102 | 6 +++--- docs/basics/_examples/DL-101-133-103 | 18 +++++++++--------- docs/basics/_examples/DL-101-133-104 | 2 +- docs/basics/_examples/DL-101-133-110 | 2 +- docs/basics/_examples/DL-101-133-111 | 10 +++++----- 7 files changed, 30 insertions(+), 27 deletions(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index d73419c0e..5ee31bf4a 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -123,9 +123,12 @@ section :ref:`run` with the :command:`datalad containers-run` command. Let's see this in action for the ``midterm_analysis`` dataset by rerunning the analysis you did for the midterm project within a Singularity container. We start by registering a container to the dataset. -For this we will pull a container from Singularity hub. This container has -been build for the handbook, and it contains the relevant Python setup for -the analysis. If you're curious how to create a Singularity image, the hidden +For this, we will pull an Image from Singularity hub. This image was made +for the handbook, and it contains the relevant Python setup for +the analysis. Its recipe lives in the handbook's +`resources repository `_, and the +Image is built from the recipe via Singularity hub. +If you're curious how to create a Singularity image, the hidden section below has some pointers: .. findoutmore:: How to make a Singularity image @@ -166,12 +169,12 @@ name to give to the container, and a path or url to a container image: :workdir: dl-101/DataLad-101/midterm_project # we are in the midterm_project subdataset - $ datalad containers-add python --url shub://adswa/resources:1 + $ datalad containers-add midterm-software --url shub://adswa/resources:1 This command downloaded the container from Singularity Hub, added it to the ``midterm_project`` dataset, and recorded basic information on the -container under its name "python" in the dataset's configuration at +container under its name "midterm-software" in the dataset's configuration at ``.datalad/config``. .. findoutmore:: What has been added to .datalad/config? @@ -229,7 +232,7 @@ How would it look like as a ``containers-run`` command? :workdir: dl-101/DataLad-101/midterm_project $ datalad containers-run -m "rerun analysis in container" \ - --container-name python \ + --container-name midterm-software \ --input "input/iris.csv" \ --output "prediction_report.csv" \ --output "pairwise_relationships.png" \ diff --git a/docs/basics/_examples/DL-101-133-101 b/docs/basics/_examples/DL-101-133-101 index d46fa9cc9..2504453f3 100644 --- a/docs/basics/_examples/DL-101-133-101 +++ b/docs/basics/_examples/DL-101-133-101 @@ -1,8 +1,8 @@ # we are in the midterm_project subdataset -$ datalad containers-add python --url shub://adswa/resources:1 +$ datalad containers-add midterm-software --url shub://adswa/resources:1 add(ok): .datalad/config (file) save(ok): . (dataset) -containers_add(ok): /home/me/dl-101/DataLad-101/midterm_project/.datalad/environments/python/image (file) +containers_add(ok): /home/me/dl-101/DataLad-101/midterm_project/.datalad/environments/midterm-software/image (file) action summary: add (ok: 1) containers_add (ok: 1) diff --git a/docs/basics/_examples/DL-101-133-102 b/docs/basics/_examples/DL-101-133-102 index 1d1d75fc9..7a546acf9 100644 --- a/docs/basics/_examples/DL-101-133-102 +++ b/docs/basics/_examples/DL-101-133-102 @@ -1,7 +1,7 @@ $ cat .datalad/config [datalad "dataset"] - id = 641bc4fe-1043-11ea-97f1-e86a64c8054c -[datalad "containers.python"] + id = d62da69a-16a9-11ea-b652-d0c637c523bc +[datalad "containers.midterm-software"] updateurl = shub://adswa/resources:1 - image = .datalad/environments/python/image + image = .datalad/environments/midterm-software/image cmdexec = singularity exec {img} {cmd} diff --git a/docs/basics/_examples/DL-101-133-103 b/docs/basics/_examples/DL-101-133-103 index 1c5073a5e..4c226e915 100644 --- a/docs/basics/_examples/DL-101-133-103 +++ b/docs/basics/_examples/DL-101-133-103 @@ -1,26 +1,26 @@ $ git log -n 1 -p -commit e3b5fe16b5227b5e792b013fa20e433fa71121f5 +commit 2344fa6676afcc92b47c7a03bbd4090b7ddabe27 Author: Elena Piscopia -Date: Tue Nov 26 12:54:17 2019 +0100 +Date: Wed Dec 4 16:22:52 2019 +0100 - [DATALAD] Configure containerized environment 'python' + [DATALAD] Configure containerized environment 'midterm-software' diff --git a/.datalad/config b/.datalad/config -index af6e089..6674d81 100644 +index 61c5ac2..651422a 100644 --- a/.datalad/config +++ b/.datalad/config @@ -1,2 +1,6 @@ [datalad "dataset"] - id = 641bc4fe-1043-11ea-97f1-e86a64c8054c -+[datalad "containers.python"] + id = d62da69a-16a9-11ea-b652-d0c637c523bc ++[datalad "containers.midterm-software"] + updateurl = shub://adswa/resources:1 -+ image = .datalad/environments/python/image ++ image = .datalad/environments/midterm-software/image + cmdexec = singularity exec {img} {cmd} -diff --git a/.datalad/environments/python/image b/.datalad/environments/python/image +diff --git a/.datalad/environments/midterm-software/image b/.datalad/environments/midterm-software/image new file mode 120000 index 0000000..800282a --- /dev/null -+++ b/.datalad/environments/python/image ++++ b/.datalad/environments/midterm-software/image @@ -0,0 +1 @@ +../../../.git/annex/objects/zJ/8f/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10 \ No newline at end of file diff --git a/docs/basics/_examples/DL-101-133-104 b/docs/basics/_examples/DL-101-133-104 index d87efec6c..847fad385 100644 --- a/docs/basics/_examples/DL-101-133-104 +++ b/docs/basics/_examples/DL-101-133-104 @@ -1,5 +1,5 @@ $ datalad containers-run -m "rerun analysis in container" \ - --container-name python \ + --container-name midterm-software \ --input "input/iris.csv" \ --output "prediction_report.csv" \ --output "pairwise_relationships.png" \ diff --git a/docs/basics/_examples/DL-101-133-110 b/docs/basics/_examples/DL-101-133-110 index ddba701f3..15b6b53f9 100644 --- a/docs/basics/_examples/DL-101-133-110 +++ b/docs/basics/_examples/DL-101-133-110 @@ -1,2 +1,2 @@ $ datalad containers-list -python -> .datalad/environments/python/image +midterm-software -> .datalad/environments/midterm-software/image diff --git a/docs/basics/_examples/DL-101-133-111 b/docs/basics/_examples/DL-101-133-111 index 77ac150eb..6566c7a72 100644 --- a/docs/basics/_examples/DL-101-133-111 +++ b/docs/basics/_examples/DL-101-133-111 @@ -1,18 +1,18 @@ $ git log -p -n 1 -commit a70122fd64dd7afb6173ea7e6efad0011e5a9c73 +commit 326c92512ab422651717a3869ef9d01e61b5cb61 Author: Elena Piscopia -Date: Tue Nov 26 12:54:22 2019 +0100 +Date: Wed Dec 4 16:22:57 2019 +0100 [DATALAD RUNCMD] rerun analysis in container === Do not change lines below === { "chain": [], - "cmd": "singularity exec .datalad/environments/python/image python3 code/script.py", - "dsid": "641bc4fe-1043-11ea-97f1-e86a64c8054c", + "cmd": "singularity exec .datalad/environments/midterm-software/image python3 code/script.py", + "dsid": "d62da69a-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [ - ".datalad/environments/python/image" + ".datalad/environments/midterm-software/image" ], "inputs": [ "input/iris.csv" From 9cdb72be309677a4733ad6c62af45a7a9860959b Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 4 Dec 2019 16:31:10 +0100 Subject: [PATCH 30/33] mention Docker support --- docs/basics/101-133-containersrun.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index 5ee31bf4a..80962182c 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -95,10 +95,10 @@ Both of these tools share core terminology: and have imaged built from them by a service (instead of building them manually and locally). -Note that as of now, the ``datalad-containers`` extensions only supports -Singularity images, but support for Docker is being actively developed. -Singularity is however compatible with Docker -- you can use -Docker images as a basis for Singularity images, or run Docker images with +Note that as of now, the ``datalad-containers`` extension supports +Singularity and Docker images. +Singularity furthermore is compatible with Docker -- you can use +Docker Images as a basis for Singularity Images, or run Docker Images with Singularity (even without having Docker installed). .. note:: From 73238548827a87134581847f1598f326690d72c0 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 4 Dec 2019 16:31:39 +0100 Subject: [PATCH 31/33] image -> Image --- docs/basics/101-133-containersrun.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index 80962182c..5ba5afbe6 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -123,15 +123,15 @@ section :ref:`run` with the :command:`datalad containers-run` command. Let's see this in action for the ``midterm_analysis`` dataset by rerunning the analysis you did for the midterm project within a Singularity container. We start by registering a container to the dataset. -For this, we will pull an Image from Singularity hub. This image was made +For this, we will pull an Image from Singularity hub. This Image was made for the handbook, and it contains the relevant Python setup for the analysis. Its recipe lives in the handbook's `resources repository `_, and the Image is built from the recipe via Singularity hub. -If you're curious how to create a Singularity image, the hidden +If you're curious how to create a Singularity Image, the hidden section below has some pointers: -.. findoutmore:: How to make a Singularity image +.. findoutmore:: How to make a Singularity Image Singularity containers are build from Image files, often called "recipes", that hold a "definition" of the software container and its @@ -141,7 +141,7 @@ section below has some pointers: An alternative to writing the Image file by hand is to use `Neurodocker `_. This command-line program can help you generate custom Singularity recipes (and - also ``Dockerfiles``, from which Docker images are build). A wonderful tutorial + also ``Dockerfiles``, from which Docker Images are build). A wonderful tutorial on how to use Neurodocker is `this introduction `_ by Michael Notter. @@ -162,7 +162,7 @@ section below has some pointers: give you a set of instructions on how to do this. The :command:`datalad containers-add` command takes an arbitrary -name to give to the container, and a path or url to a container image: +name to give to the container, and a path or url to a container Image: .. runrecord:: _examples/DL-101-133-101 :language: console @@ -185,8 +185,8 @@ container under its name "midterm-software" in the dataset's configuration at $ cat .datalad/config - This recorded the image's origin on Singularity-Hub, the location of the - image in the dataset under ``.datalad/environments//image``, and it + This recorded the Image's origin on Singularity-Hub, the location of the + Image in the dataset under ``.datalad/environments//image``, and it specifies the way in which the container should be used: The line .. code-block:: bash @@ -198,7 +198,7 @@ container under its name "midterm-software" in the dataset's configuration at :command:`singularity exec` command. The mode of calling Singularity, namely ``exec``, means that the command will be executed inside of the container. - Note that the image is saved under ``.datalad/environments`` and the + Note that the Image is saved under ``.datalad/environments`` and the configuration is done in ``.datalad/config`` -- as these files are version controlled and shared with together with a dataset, your software container and the information where it can be re-obtained from are linked @@ -264,7 +264,7 @@ The complete command's structure looks like this:: The command :command:`datalad containers-remove` will remove a container from the dataset, if there exists a container with name given to the - command. Note that this will remove not only the image from the dataset, + command. Note that this will remove not only the Image from the dataset, but also the configuration for it in ``.datalad/config``. From 8955f235790bb736680edea9a3017bf6747dd435 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 4 Dec 2019 16:33:34 +0100 Subject: [PATCH 32/33] rebuild all examples --- docs/basics/_examples/DL-101-101-103 | 8 ++-- docs/basics/_examples/DL-101-102-103 | 4 +- docs/basics/_examples/DL-101-102-107 | 4 +- docs/basics/_examples/DL-101-102-108 | 2 +- docs/basics/_examples/DL-101-102-111 | 8 ++-- docs/basics/_examples/DL-101-103-107 | 8 ++-- docs/basics/_examples/DL-101-106-101 | 36 ++++++++--------- docs/basics/_examples/DL-101-108-104 | 2 +- docs/basics/_examples/DL-101-108-106 | 6 +-- docs/basics/_examples/DL-101-108-108 | 20 +++++----- docs/basics/_examples/DL-101-108-113 | 10 ++--- docs/basics/_examples/DL-101-108-114 | 2 +- docs/basics/_examples/DL-101-108-115 | 8 ++-- docs/basics/_examples/DL-101-108-120 | 14 +++---- docs/basics/_examples/DL-101-112-109 | 10 ++--- docs/basics/_examples/DL-101-114-101 | 42 ++++++++++---------- docs/basics/_examples/DL-101-115-103 | 6 +-- docs/basics/_examples/DL-101-116-105 | 2 +- docs/basics/_examples/DL-101-117-101 | 2 +- docs/basics/_examples/DL-101-118-102 | 42 ++++++++++---------- docs/basics/_examples/DL-101-118-103 | 2 +- docs/basics/_examples/DL-101-121-103 | 4 +- docs/basics/_examples/DL-101-121-112 | 52 ++++++++++++------------- docs/basics/_examples/DL-101-122-101 | 4 +- docs/basics/_examples/DL-101-122-103 | 4 +- docs/basics/_examples/DL-101-123-105 | 2 +- docs/basics/_examples/DL-101-124-101 | 1 + docs/basics/_examples/DL-101-130-107 | 2 +- docs/basics/_examples/DL-101-130-110 | 8 ++-- docs/basics/_examples/DL-101-130-112 | 10 ++--- docs/basics/_examples/DL-101-132-101 | 12 +++--- docs/basics/_examples/DL-101-132-104 | 1 - docs/basics/_examples/DL-101-132-110 | 2 +- docs/basics/_examples/DL-101-132-112 | 12 +++--- docs/basics/_examples/DL-101-135-102 | 58 ++++++++++++++++++++++++++-- docs/basics/_examples/DL-101-136-101 | 6 +-- docs/basics/_examples/DL-101-136-104 | 4 +- docs/basics/_examples/DL-101-136-105 | 2 +- docs/basics/_examples/DL-101-136-109 | 2 +- docs/basics/_examples/DL-101-136-110 | 2 +- docs/basics/_examples/DL-101-136-121 | 2 +- docs/basics/_examples/DL-101-136-122 | 2 +- docs/basics/_examples/DL-101-136-123 | 4 +- docs/basics/_examples/DL-101-136-124 | 2 +- docs/basics/_examples/DL-101-136-132 | 4 +- docs/basics/_examples/DL-101-136-133 | 4 +- docs/basics/_examples/DL-101-136-134 | 2 +- docs/basics/_examples/DL-101-136-140 | 4 +- docs/basics/_examples/DL-101-136-144 | 4 +- docs/basics/_examples/DL-101-136-146 | 2 +- docs/basics/_examples/DL-101-136-152 | 2 +- docs/basics/_examples/DL-101-136-156 | 2 +- docs/basics/_examples/DL-101-136-164 | 2 +- docs/basics/_examples/DL-101-136-174 | 2 +- docs/basics/_examples/DL-101-136-180 | 2 +- docs/basics/_examples/DL-101-137-101 | 30 +++++++------- docs/basics/_examples/DL-101-137-105 | 4 +- docs/basics/_examples/DL-101-137-106 | 6 +-- docs/basics/_examples/DL-101-137-107 | 2 +- docs/basics/_examples/DL-101-137-108 | 4 +- docs/basics/_examples/DL-101-137-112 | 8 ++-- docs/basics/_examples/DL-101-137-114 | 2 +- docs/basics/_examples/DL-101-137-116 | 2 +- docs/basics/_examples/DL-101-137-117 | 6 +-- docs/basics/_examples/DL-101-137-118 | 2 +- docs/basics/_examples/DL-101-137-119 | 4 +- docs/basics/_examples/DL-101-137-121 | 40 +++++++++---------- docs/basics/_examples/DL-101-137-122 | 6 +-- docs/basics/_examples/DL-101-137-124 | 2 +- docs/basics/_examples/DL-101-137-127 | 2 +- docs/basics/_examples/DL-101-137-132 | 4 +- docs/basics/_examples/DL-101-137-133 | 4 +- docs/basics/_examples/DL-101-137-144 | 6 +-- docs/basics/_examples/DL-101-137-146 | 14 +++---- 74 files changed, 333 insertions(+), 281 deletions(-) diff --git a/docs/basics/_examples/DL-101-101-103 b/docs/basics/_examples/DL-101-101-103 index 803ad9bd6..8ed885834 100644 --- a/docs/basics/_examples/DL-101-101-103 +++ b/docs/basics/_examples/DL-101-101-103 @@ -1,12 +1,12 @@ $ git log -commit 4cc7c21a716368eda466b94630522d19abfef72f +commit 6c2e4854217a5cbb06eb9b3d0c476573ee2b3ced Author: Elena Piscopia -Date: Tue Nov 26 12:52:00 2019 +0100 +Date: Wed Dec 4 16:21:12 2019 +0100 Instruct annex to add text files to Git -commit fca4a608c6f8c749a2f3f83a3da8042b61e496ec +commit 623d3f641b8c6e948a036a7be9ec4f056d1ae024 Author: Elena Piscopia -Date: Tue Nov 26 12:52:00 2019 +0100 +Date: Wed Dec 4 16:21:12 2019 +0100 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-102-103 b/docs/basics/_examples/DL-101-102-103 index 4a967d6ee..238150f7b 100644 --- a/docs/basics/_examples/DL-101-102-103 +++ b/docs/basics/_examples/DL-101-102-103 @@ -3,5 +3,5 @@ $ wget https://sourceforge.net/projects/linuxcommand/files/TLCL/19.01/TLCL-19.01 $ wget https://www.gitbook.com/download/pdf/book/swaroopch/byte-of-python -O byte-of-python.pdf # get back into the root of the dataset $ cd ../ -2019-11-26 12:52:56 URL:https://netcologne.dl.sourceforge.net/project/linuxcommand/TLCL/19.01/TLCL-19.01.pdf [2120211/2120211] -> "TLCL.pdf" [1] -2019-11-26 12:52:57 URL:https://legacy.gitbook.com/download/pdf/book/swaroopch/byte-of-python [4407669] -> "byte-of-python.pdf" [1] +2019-12-04 16:21:19 URL:https://netcologne.dl.sourceforge.net/project/linuxcommand/TLCL/19.01/TLCL-19.01.pdf [2120211/2120211] -> "TLCL.pdf" [1] +2019-12-04 16:21:21 URL:https://legacy.gitbook.com/download/pdf/book/swaroopch/byte-of-python [4407669] -> "byte-of-python.pdf" [1] diff --git a/docs/basics/_examples/DL-101-102-107 b/docs/basics/_examples/DL-101-102-107 index 41351976b..2e0db66ba 100644 --- a/docs/basics/_examples/DL-101-102-107 +++ b/docs/basics/_examples/DL-101-102-107 @@ -1,7 +1,7 @@ $ git log -p -n 1 -commit 5d5a8014fa0f87fe11cc24223094a01e082c4b9b +commit c2dcdaaf0107c3aceac2fc430f8e28da257434d1 Author: Elena Piscopia -Date: Tue Nov 26 12:52:58 2019 +0100 +Date: Wed Dec 4 16:21:22 2019 +0100 add books on Python and Unix to read later diff --git a/docs/basics/_examples/DL-101-102-108 b/docs/basics/_examples/DL-101-102-108 index bd7c064f4..1706ccf4e 100644 --- a/docs/basics/_examples/DL-101-102-108 +++ b/docs/basics/_examples/DL-101-102-108 @@ -1,4 +1,4 @@ $ cd books $ wget https://github.com/progit/progit2/releases/download/2.1.154/progit.pdf $ cd ../ -2019-11-26 12:53:00 URL:https://github-production-release-asset-2e65be.s3.amazonaws.com/15400220/57552a00-9a49-11e9-9144-d9607ed4c2db?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191126%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191126T115258Z&X-Amz-Expires=300&X-Amz-Signature=834f5e39852c20a317fae48510e80cb3f40172896a77497f97aeb1bb86e2b802&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprogit.pdf&response-content-type=application%2Foctet-stream [12465653/12465653] -> "progit.pdf" [1] +2019-12-04 16:21:24 URL:https://github-production-release-asset-2e65be.s3.amazonaws.com/15400220/57552a00-9a49-11e9-9144-d9607ed4c2db?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191204%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191204T152122Z&X-Amz-Expires=300&X-Amz-Signature=66df788d2f7db5a587cd679928904ddf3735912c6589aa767a65fdb322422ec5&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprogit.pdf&response-content-type=application%2Foctet-stream [12465653/12465653] -> "progit.pdf" [1] diff --git a/docs/basics/_examples/DL-101-102-111 b/docs/basics/_examples/DL-101-102-111 index d5dee43f7..405300987 100644 --- a/docs/basics/_examples/DL-101-102-111 +++ b/docs/basics/_examples/DL-101-102-111 @@ -1,6 +1,6 @@ # lets make the output a bit more concise with the --oneline option $ git log --oneline -90ade2e add reference book about git -5d5a801 add books on Python and Unix to read later -4cc7c21 Instruct annex to add text files to Git -fca4a60 [DATALAD] new dataset +3d3987b add reference book about git +c2dcdaa add books on Python and Unix to read later +6c2e485 Instruct annex to add text files to Git +623d3f6 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-103-107 b/docs/basics/_examples/DL-101-103-107 index 8a4b7fc7e..fa3c87deb 100644 --- a/docs/basics/_examples/DL-101-103-107 +++ b/docs/basics/_examples/DL-101-103-107 @@ -1,7 +1,7 @@ $ git log -p -n 2 -commit 1de8b79fb638a1510e7ebb0b3fcb2f2e51721456 +commit c56f7cb5c0637dee0ad15c5e0d99e68e1d7fd4f5 Author: Elena Piscopia -Date: Tue Nov 26 12:53:01 2019 +0100 +Date: Wed Dec 4 16:21:25 2019 +0100 add note on datalad save @@ -18,9 +18,9 @@ index 3a7a1fe..bfa64d7 100644 +Always use informative, concise commit messages. + -commit 0ed048a43dfe0fb64b7be9ac06a9074a154162f3 +commit 242271452e13ca4f06e578b82788099a1317c158 Author: Elena Piscopia -Date: Tue Nov 26 12:53:00 2019 +0100 +Date: Wed Dec 4 16:21:25 2019 +0100 Add notes on datalad create diff --git a/docs/basics/_examples/DL-101-106-101 b/docs/basics/_examples/DL-101-106-101 index 6d52ce310..25b6f27c9 100644 --- a/docs/basics/_examples/DL-101-106-101 +++ b/docs/basics/_examples/DL-101-106-101 @@ -1,7 +1,7 @@ $ git log -p -n 2 -commit 67d32e901dd88b9ad093dce3e1884bcea85e478a +commit 8db7558a987c679de83e63788611cf2ff34143e9 Author: Elena Piscopia -Date: Tue Nov 26 12:53:15 2019 +0100 +Date: Wed Dec 4 16:21:39 2019 +0100 Add note on datalad install @@ -20,9 +20,9 @@ index bfa64d7..6d8cdce 100644 +root of the superdataset with the '-d' option. + -commit 776b0ac52acaaa20d3bd263d6c05d81e4a2e1029 +commit ab0b11f4292c900b81f633cda09ee8bf749195e8 Author: Elena Piscopia -Date: Tue Nov 26 12:53:03 2019 +0100 +Date: Wed Dec 4 16:21:27 2019 +0100 [DATALAD] Recorded changes @@ -44,9 +44,9 @@ index 0000000..dcc34fb @@ -0,0 +1 @@ +Subproject commit dcc34fbe669b06ced84ced381ba0db21cf5e665f -commit 1de8b79fb638a1510e7ebb0b3fcb2f2e51721456 +commit c56f7cb5c0637dee0ad15c5e0d99e68e1d7fd4f5 Author: Elena Piscopia -Date: Tue Nov 26 12:53:01 2019 +0100 +Date: Wed Dec 4 16:21:25 2019 +0100 add note on datalad save @@ -63,9 +63,9 @@ index 3a7a1fe..bfa64d7 100644 +Always use informative, concise commit messages. + -commit 0ed048a43dfe0fb64b7be9ac06a9074a154162f3 +commit 242271452e13ca4f06e578b82788099a1317c158 Author: Elena Piscopia -Date: Tue Nov 26 12:53:00 2019 +0100 +Date: Wed Dec 4 16:21:25 2019 +0100 Add notes on datalad create @@ -79,9 +79,9 @@ index 0000000..3a7a1fe +The dataset is created empty + -commit 90ade2edb8566117c0d88cf478c654f97688c796 +commit 3d3987be908a115c4cd044705af0d51a25e2f47d Author: Elena Piscopia -Date: Tue Nov 26 12:53:00 2019 +0100 +Date: Wed Dec 4 16:21:24 2019 +0100 add reference book about git @@ -94,9 +94,9 @@ index 0000000..c5d2ab2 +../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf \ No newline at end of file -commit 5d5a8014fa0f87fe11cc24223094a01e082c4b9b +commit c2dcdaaf0107c3aceac2fc430f8e28da257434d1 Author: Elena Piscopia -Date: Tue Nov 26 12:52:58 2019 +0100 +Date: Wed Dec 4 16:21:22 2019 +0100 add books on Python and Unix to read later @@ -117,9 +117,9 @@ index 0000000..9a812a0 +../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf \ No newline at end of file -commit 4cc7c21a716368eda466b94630522d19abfef72f +commit 6c2e4854217a5cbb06eb9b3d0c476573ee2b3ced Author: Elena Piscopia -Date: Tue Nov 26 12:52:00 2019 +0100 +Date: Wed Dec 4 16:21:12 2019 +0100 Instruct annex to add text files to Git @@ -136,9 +136,9 @@ index c3aaefe..b223f34 100644 +* annex.largefiles=(not(mimetype=text/*)and(largerthan=0)) \ No newline at end of file -commit fca4a608c6f8c749a2f3f83a3da8042b61e496ec +commit 623d3f641b8c6e948a036a7be9ec4f056d1ae024 Author: Elena Piscopia -Date: Tue Nov 26 12:52:00 2019 +0100 +Date: Wed Dec 4 16:21:12 2019 +0100 [DATALAD] new dataset @@ -155,12 +155,12 @@ index 0000000..b540820 \ No newline at end of file diff --git a/.datalad/config b/.datalad/config new file mode 100644 -index 0000000..2f3e9f7 +index 0000000..95115e5 --- /dev/null +++ b/.datalad/config @@ -0,0 +1,2 @@ +[datalad "dataset"] -+ id = 27b04f08-1043-11ea-97f1-e86a64c8054c ++ id = b47220ee-16a9-11ea-b652-d0c637c523bc diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..c3aaefe diff --git a/docs/basics/_examples/DL-101-108-104 b/docs/basics/_examples/DL-101-108-104 index 62a99527f..f92f6b8c0 100644 --- a/docs/basics/_examples/DL-101-108-104 +++ b/docs/basics/_examples/DL-101-108-104 @@ -1,4 +1,4 @@ -$ datalad save -m "Add simple script to write a list of podcast speakers and titles" +$ datalad save -m "Add short script to write a list of podcast speakers and titles" add(ok): code/list_titles.sh (file) save(ok): . (dataset) action summary: diff --git a/docs/basics/_examples/DL-101-108-106 b/docs/basics/_examples/DL-101-108-106 index 2bee778f5..bb82894b9 100644 --- a/docs/basics/_examples/DL-101-108-106 +++ b/docs/basics/_examples/DL-101-108-106 @@ -1,7 +1,7 @@ $ git log -p -n 1 -commit 69f9eccc548ef789bdfee266dbca045b099b3b36 +commit 281a9a48a1451e85c5a9284b6b908c93a987654c Author: Elena Piscopia -Date: Tue Nov 26 12:53:16 2019 +0100 +Date: Wed Dec 4 16:21:41 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles @@ -9,7 +9,7 @@ Date: Tue Nov 26 12:53:16 2019 +0100 { "chain": [], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-108-108 b/docs/basics/_examples/DL-101-108-108 index 8c8024ed4..3480ce1a3 100644 --- a/docs/basics/_examples/DL-101-108-108 +++ b/docs/basics/_examples/DL-101-108-108 @@ -1,11 +1,11 @@ $ git log --oneline -69f9ecc [DATALAD RUNCMD] create a list of podcast titles -12323f0 Add simple script to write a list of podcast speakers and titles -67d32e9 Add note on datalad install -776b0ac [DATALAD] Recorded changes -1de8b79 add note on datalad save -0ed048a Add notes on datalad create -90ade2e add reference book about git -5d5a801 add books on Python and Unix to read later -4cc7c21 Instruct annex to add text files to Git -fca4a60 [DATALAD] new dataset +281a9a4 [DATALAD RUNCMD] create a list of podcast titles +61f3a4d Add short script to write a list of podcast speakers and titles +8db7558 Add note on datalad install +ab0b11f [DATALAD] Recorded changes +c56f7cb add note on datalad save +2422714 Add notes on datalad create +3d3987b add reference book about git +c2dcdaa add books on Python and Unix to read later +6c2e485 Instruct annex to add text files to Git +623d3f6 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-108-113 b/docs/basics/_examples/DL-101-108-113 index 6bbbd6a41..8edee1219 100644 --- a/docs/basics/_examples/DL-101-108-113 +++ b/docs/basics/_examples/DL-101-108-113 @@ -1,13 +1,13 @@ $ git log -n 2 -commit 3923b9c26b11d8e4c22f89b49886ae6ccd70df21 +commit 16787003e96f58235c73d4865f899fe2fdd85b2f Author: Elena Piscopia -Date: Tue Nov 26 12:53:18 2019 +0100 +Date: Wed Dec 4 16:21:42 2019 +0100 BF: list both directories content -commit 69f9eccc548ef789bdfee266dbca045b099b3b36 +commit 281a9a48a1451e85c5a9284b6b908c93a987654c Author: Elena Piscopia -Date: Tue Nov 26 12:53:16 2019 +0100 +Date: Wed Dec 4 16:21:41 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles @@ -15,7 +15,7 @@ Date: Tue Nov 26 12:53:16 2019 +0100 { "chain": [], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-108-114 b/docs/basics/_examples/DL-101-108-114 index a8a4ad0ac..adfed2e41 100644 --- a/docs/basics/_examples/DL-101-108-114 +++ b/docs/basics/_examples/DL-101-108-114 @@ -1,5 +1,5 @@ -$ datalad rerun 69f9eccc548ef789bdfee266dbca045b099b3b36 +$ datalad rerun 281a9a48a1451e85c5a9284b6b908c93a987654c [INFO] == Command start (output follows) ===== [INFO] == Command exit (modification check follows) ===== add(ok): recordings/podcasts.tsv (file) diff --git a/docs/basics/_examples/DL-101-108-115 b/docs/basics/_examples/DL-101-108-115 index c232490c5..86d0c84f1 100644 --- a/docs/basics/_examples/DL-101-108-115 +++ b/docs/basics/_examples/DL-101-108-115 @@ -1,17 +1,17 @@ $ git log -n 1 -commit 140477ac946476718c89a152b1779d07ee0a5e6d +commit b5497627adc2bac4cfba3de27cfea371f1447f6c Author: Elena Piscopia -Date: Tue Nov 26 12:53:19 2019 +0100 +Date: Wed Dec 4 16:21:43 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles === Do not change lines below === { "chain": [ - "69f9eccc548ef789bdfee266dbca045b099b3b36" + "281a9a48a1451e85c5a9284b6b908c93a987654c" ], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-108-120 b/docs/basics/_examples/DL-101-108-120 index da0f51476..6df34276a 100644 --- a/docs/basics/_examples/DL-101-108-120 +++ b/docs/basics/_examples/DL-101-108-120 @@ -1,17 +1,17 @@ $ git log -- recordings/podcasts.tsv -commit 140477ac946476718c89a152b1779d07ee0a5e6d +commit b5497627adc2bac4cfba3de27cfea371f1447f6c Author: Elena Piscopia -Date: Tue Nov 26 12:53:19 2019 +0100 +Date: Wed Dec 4 16:21:43 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles === Do not change lines below === { "chain": [ - "69f9eccc548ef789bdfee266dbca045b099b3b36" + "281a9a48a1451e85c5a9284b6b908c93a987654c" ], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], @@ -20,9 +20,9 @@ Date: Tue Nov 26 12:53:19 2019 +0100 } ^^^ Do not change lines above ^^^ -commit 69f9eccc548ef789bdfee266dbca045b099b3b36 +commit 281a9a48a1451e85c5a9284b6b908c93a987654c Author: Elena Piscopia -Date: Tue Nov 26 12:53:16 2019 +0100 +Date: Wed Dec 4 16:21:41 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles @@ -30,7 +30,7 @@ Date: Tue Nov 26 12:53:16 2019 +0100 { "chain": [], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-112-109 b/docs/basics/_examples/DL-101-112-109 index 350ec937b..b02d8922e 100644 --- a/docs/basics/_examples/DL-101-112-109 +++ b/docs/basics/_examples/DL-101-112-109 @@ -1,7 +1,7 @@ $ git log -p -n 2 -commit 66f97f3bf06fe4ed612f1b2a63a38a9108ff8aa1 +commit 199587fa10a7ea54885d3139b4d0ef042076da0d Author: Elena Piscopia -Date: Tue Nov 26 12:53:26 2019 +0100 +Date: Wed Dec 4 16:21:52 2019 +0100 add note on clean datasets @@ -21,9 +21,9 @@ index d1ba8fc..8f0124f 100644 +to the files listed with --output flags. + -commit 17e39f474e033f46b798c9ea5a460cb9c7ed4f05 +commit dd726a625ba18ac0a25b744adf5feab33d9d6866 Author: Elena Piscopia -Date: Tue Nov 26 12:53:25 2019 +0100 +Date: Wed Dec 4 16:21:51 2019 +0100 [DATALAD RUNCMD] Resize logo for slides @@ -31,7 +31,7 @@ Date: Tue Nov 26 12:53:25 2019 +0100 { "chain": [], "cmd": "convert -resize 400x400 recordings/longnow/.datalad/feed_metadata/logo_salt.jpg recordings/salt_logo_small.jpg", - "dsid": "27b04f08-1043-11ea-97f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [ diff --git a/docs/basics/_examples/DL-101-114-101 b/docs/basics/_examples/DL-101-114-101 index 7391f9a60..20e7d1e4d 100644 --- a/docs/basics/_examples/DL-101-114-101 +++ b/docs/basics/_examples/DL-101-114-101 @@ -1,22 +1,22 @@ $ git log --reverse --oneline -fca4a60 [DATALAD] new dataset -4cc7c21 Instruct annex to add text files to Git -5d5a801 add books on Python and Unix to read later -90ade2e add reference book about git -0ed048a Add notes on datalad create -1de8b79 add note on datalad save -776b0ac [DATALAD] Recorded changes -67d32e9 Add note on datalad install -12323f0 Add simple script to write a list of podcast speakers and titles -69f9ecc [DATALAD RUNCMD] create a list of podcast titles -3923b9c BF: list both directories content -140477a [DATALAD RUNCMD] create a list of podcast titles -de40bb1 add note datalad and git diff -5ca7d13 add note on basic datalad run and datalad rerun -11dfb75 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -8cdc201 resized picture by hand -fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -048ae39 add additional notes on run options -68ade50 [DATALAD RUNCMD] Resize logo for slides -17e39f4 [DATALAD RUNCMD] Resize logo for slides -66f97f3 add note on clean datasets +623d3f6 [DATALAD] new dataset +6c2e485 Instruct annex to add text files to Git +c2dcdaa add books on Python and Unix to read later +3d3987b add reference book about git +2422714 Add notes on datalad create +c56f7cb add note on datalad save +ab0b11f [DATALAD] Recorded changes +8db7558 Add note on datalad install +61f3a4d Add short script to write a list of podcast speakers and titles +281a9a4 [DATALAD RUNCMD] create a list of podcast titles +1678700 BF: list both directories content +b549762 [DATALAD RUNCMD] create a list of podcast titles +09c0513 add note datalad and git diff +aadf74e add note on basic datalad run and datalad rerun +86be52e [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +bd42809 resized picture by hand +3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +b08d657 add additional notes on run options +e4dd212 [DATALAD RUNCMD] Resize logo for slides +dd726a6 [DATALAD RUNCMD] Resize logo for slides +199587f add note on clean datasets diff --git a/docs/basics/_examples/DL-101-115-103 b/docs/basics/_examples/DL-101-115-103 index b7593e152..e2d8f8090 100644 --- a/docs/basics/_examples/DL-101-115-103 +++ b/docs/basics/_examples/DL-101-115-103 @@ -1,7 +1,7 @@ $ ls -lah total 20K -drwxr-xr-x 2 adina adina 4.0K Nov 26 12:53 . -drwxr-xr-x 7 adina adina 4.0K Nov 26 12:53 .. -lrwxrwxrwx 1 adina adina 131 Nov 26 12:52 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf +drwxr-xr-x 2 adina adina 4.0K Dec 4 16:21 . +drwxr-xr-x 7 adina adina 4.0K Dec 4 16:21 .. +lrwxrwxrwx 1 adina adina 131 Dec 4 16:21 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf lrwxrwxrwx 1 adina adina 133 Jun 29 17:39 progit.pdf -> ../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf lrwxrwxrwx 1 adina adina 131 Jan 28 2019 TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-116-105 b/docs/basics/_examples/DL-101-116-105 index ad2b53b61..d4ebc8d2d 100644 --- a/docs/basics/_examples/DL-101-116-105 +++ b/docs/basics/_examples/DL-101-116-105 @@ -1,4 +1,4 @@ $ git annex whereis books/TLCL.pdf whereis books/TLCL.pdf (1 copy) - 9d17effd-b251-48ab-92b2-b629c15a9b3e -- course on DataLad-101 on my private Laptop [origin] + fdbcfeff-5296-4e58-84dc-913343e50f4d -- course on DataLad-101 on my private Laptop [origin] ok diff --git a/docs/basics/_examples/DL-101-117-101 b/docs/basics/_examples/DL-101-117-101 index bbe896df9..bc2be8a2e 100644 --- a/docs/basics/_examples/DL-101-117-101 +++ b/docs/basics/_examples/DL-101-117-101 @@ -6,7 +6,7 @@ $ cd recordings/longnow $ git annex whereis Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3 whereis Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3 (3 copies) 00000000-0000-0000-0000-000000000001 -- web - 31ddeb94-12ef-4084-a439-8105568ec76c -- me@muninn:~/dl-101/DataLad-101/recordings/longnow [origin] + 740557bc-bafe-41ed-b4ae-09cd5e5cbb03 -- me@muninn:~/dl-101/DataLad-101/recordings/longnow [origin] da3bf937-5bd2-43ea-a07b-bcbe71f3b875 -- mih@medusa:/tmp/seminars-on-longterm-thinking web: http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 diff --git a/docs/basics/_examples/DL-101-118-102 b/docs/basics/_examples/DL-101-118-102 index dfc97630d..98954d804 100644 --- a/docs/basics/_examples/DL-101-118-102 +++ b/docs/basics/_examples/DL-101-118-102 @@ -1,23 +1,23 @@ # lets view the history $ git log --oneline -66f97f3 add note on clean datasets -17e39f4 [DATALAD RUNCMD] Resize logo for slides -68ade50 [DATALAD RUNCMD] Resize logo for slides -048ae39 add additional notes on run options -fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -8cdc201 resized picture by hand -11dfb75 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -5ca7d13 add note on basic datalad run and datalad rerun -de40bb1 add note datalad and git diff -140477a [DATALAD RUNCMD] create a list of podcast titles -3923b9c BF: list both directories content -69f9ecc [DATALAD RUNCMD] create a list of podcast titles -12323f0 Add simple script to write a list of podcast speakers and titles -67d32e9 Add note on datalad install -776b0ac [DATALAD] Recorded changes -1de8b79 add note on datalad save -0ed048a Add notes on datalad create -90ade2e add reference book about git -5d5a801 add books on Python and Unix to read later -4cc7c21 Instruct annex to add text files to Git -fca4a60 [DATALAD] new dataset +199587f add note on clean datasets +dd726a6 [DATALAD RUNCMD] Resize logo for slides +e4dd212 [DATALAD RUNCMD] Resize logo for slides +b08d657 add additional notes on run options +3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +bd42809 resized picture by hand +86be52e [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +aadf74e add note on basic datalad run and datalad rerun +09c0513 add note datalad and git diff +b549762 [DATALAD RUNCMD] create a list of podcast titles +1678700 BF: list both directories content +281a9a4 [DATALAD RUNCMD] create a list of podcast titles +61f3a4d Add short script to write a list of podcast speakers and titles +8db7558 Add note on datalad install +ab0b11f [DATALAD] Recorded changes +c56f7cb add note on datalad save +2422714 Add notes on datalad create +3d3987b add reference book about git +c2dcdaa add books on Python and Unix to read later +6c2e485 Instruct annex to add text files to Git +623d3f6 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-118-103 b/docs/basics/_examples/DL-101-118-103 index 53abb1ee0..a7f26b112 100644 --- a/docs/basics/_examples/DL-101-118-103 +++ b/docs/basics/_examples/DL-101-118-103 @@ -1,5 +1,5 @@ -$ datalad rerun 17e39f474e033f46b798c9ea5a460cb9c7ed4f05 +$ datalad rerun dd726a625ba18ac0a25b744adf5feab33d9d6866 [INFO] Making sure inputs are available (this may take some time) [WARNING] no content present; cannot unlock [unlock(/home/me/dl-101/mock_user/DataLad-101/recordings/salt_logo_small.jpg)] [INFO] == Command start (output follows) ===== diff --git a/docs/basics/_examples/DL-101-121-103 b/docs/basics/_examples/DL-101-121-103 index e5c0bc3ee..c591e7270 100644 --- a/docs/basics/_examples/DL-101-121-103 +++ b/docs/basics/_examples/DL-101-121-103 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit a2d01ff8ce412b74a9c5240e8a8985a917628d0d +commit aca517764b330a994c8523d74dd5320a7e54e3e8 Author: Elena Piscopia -Date: Tue Nov 26 12:53:34 2019 +0100 +Date: Wed Dec 4 16:22:01 2019 +0100 Include nesting demo from datalad website diff --git a/docs/basics/_examples/DL-101-121-112 b/docs/basics/_examples/DL-101-121-112 index 48f63d5a4..4021bab28 100644 --- a/docs/basics/_examples/DL-101-121-112 +++ b/docs/basics/_examples/DL-101-121-112 @@ -1,27 +1,27 @@ $ git log --oneline -26b599c Merge remote-tracking branch 'refs/remotes/roommate/master' -029add8 add note about datalad update -a2d01ff Include nesting demo from datalad website -2091c14 add note on git annex whereis -e2210e8 add note about installing from paths and recursive installations -66f97f3 add note on clean datasets -17e39f4 [DATALAD RUNCMD] Resize logo for slides -68ade50 [DATALAD RUNCMD] Resize logo for slides -048ae39 add additional notes on run options -fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -8cdc201 resized picture by hand -11dfb75 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -5ca7d13 add note on basic datalad run and datalad rerun -de40bb1 add note datalad and git diff -140477a [DATALAD RUNCMD] create a list of podcast titles -3923b9c BF: list both directories content -69f9ecc [DATALAD RUNCMD] create a list of podcast titles -12323f0 Add simple script to write a list of podcast speakers and titles -67d32e9 Add note on datalad install -776b0ac [DATALAD] Recorded changes -1de8b79 add note on datalad save -0ed048a Add notes on datalad create -90ade2e add reference book about git -5d5a801 add books on Python and Unix to read later -4cc7c21 Instruct annex to add text files to Git -fca4a60 [DATALAD] new dataset +c15ea3e Merge remote-tracking branch 'refs/remotes/roommate/master' +aca5177 Include nesting demo from datalad website +1291140 add note about datalad update +1b07d87 add note on git annex whereis +ad17112 add note about installing from paths and recursive installations +199587f add note on clean datasets +dd726a6 [DATALAD RUNCMD] Resize logo for slides +e4dd212 [DATALAD RUNCMD] Resize logo for slides +b08d657 add additional notes on run options +3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +bd42809 resized picture by hand +86be52e [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +aadf74e add note on basic datalad run and datalad rerun +09c0513 add note datalad and git diff +b549762 [DATALAD RUNCMD] create a list of podcast titles +1678700 BF: list both directories content +281a9a4 [DATALAD RUNCMD] create a list of podcast titles +61f3a4d Add short script to write a list of podcast speakers and titles +8db7558 Add note on datalad install +ab0b11f [DATALAD] Recorded changes +c56f7cb add note on datalad save +2422714 Add notes on datalad create +3d3987b add reference book about git +c2dcdaa add books on Python and Unix to read later +6c2e485 Instruct annex to add text files to Git +623d3f6 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-122-101 b/docs/basics/_examples/DL-101-122-101 index 1038b2a65..e01bc1bb9 100644 --- a/docs/basics/_examples/DL-101-122-101 +++ b/docs/basics/_examples/DL-101-122-101 @@ -5,7 +5,7 @@ $ cat .git/config bare = false logallrefupdates = true [annex] - uuid = 9d17effd-b251-48ab-92b2-b629c15a9b3e + uuid = fdbcfeff-5296-4e58-84dc-913343e50f4d version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -14,5 +14,5 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 7478bb68-77fb-466f-a3c6-a943d5b87569 + annex-uuid = 0cfd3714-d3a7-45ab-8e70-2f741f23e8ed annex-ignore = false diff --git a/docs/basics/_examples/DL-101-122-103 b/docs/basics/_examples/DL-101-122-103 index fe4b62c8b..c68b5abfc 100644 --- a/docs/basics/_examples/DL-101-122-103 +++ b/docs/basics/_examples/DL-101-122-103 @@ -6,7 +6,7 @@ $ cat .git/config logallrefupdates = true editor = nano [annex] - uuid = 9d17effd-b251-48ab-92b2-b629c15a9b3e + uuid = fdbcfeff-5296-4e58-84dc-913343e50f4d version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -15,5 +15,5 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 7478bb68-77fb-466f-a3c6-a943d5b87569 + annex-uuid = 0cfd3714-d3a7-45ab-8e70-2f741f23e8ed annex-ignore = false diff --git a/docs/basics/_examples/DL-101-123-105 b/docs/basics/_examples/DL-101-123-105 index 0bb2b5773..c9a3c5747 100644 --- a/docs/basics/_examples/DL-101-123-105 +++ b/docs/basics/_examples/DL-101-123-105 @@ -1,3 +1,3 @@ $ cat .datalad/config [datalad "dataset"] - id = 27b04f08-1043-11ea-97f1-e86a64c8054c + id = b47220ee-16a9-11ea-b652-d0c637c523bc diff --git a/docs/basics/_examples/DL-101-124-101 b/docs/basics/_examples/DL-101-124-101 index e0e3417d0..a05291e15 100644 --- a/docs/basics/_examples/DL-101-124-101 +++ b/docs/basics/_examples/DL-101-124-101 @@ -1,4 +1,5 @@ $ datalad run-procedure --discover +cfg_hirni (../../../adina/env/handbook/lib/python3.7/site-packages/datalad_hirni/resources/procedures/cfg_hirni.py) [python_script] cfg_bids (../../../adina/repos/datalad-neuroimaging/datalad_neuroimaging/resources/procedures/cfg_bids.py) [python_script] cfg_metadatatypes (../../../adina/repos/datalad/datalad/resources/procedures/cfg_metadatatypes.py) [python_script] cfg_text2git (../../../adina/repos/datalad/datalad/resources/procedures/cfg_text2git.py) [python_script] diff --git a/docs/basics/_examples/DL-101-130-107 b/docs/basics/_examples/DL-101-130-107 index b054945a4..e1813e993 100644 --- a/docs/basics/_examples/DL-101-130-107 +++ b/docs/basics/_examples/DL-101-130-107 @@ -9,7 +9,7 @@ from sklearn.metrics import classification_report data = "input/iris.csv" -# make sure that the data is obtained (get will also install linked sub-ds!): +# make sure that the data are obtained (get will also install linked sub-ds!): dl.get(data) # prepare the data as a pandas dataframe diff --git a/docs/basics/_examples/DL-101-130-110 b/docs/basics/_examples/DL-101-130-110 index 25d2ce53d..020da785e 100644 --- a/docs/basics/_examples/DL-101-130-110 +++ b/docs/basics/_examples/DL-101-130-110 @@ -1,13 +1,13 @@ $ git show ready4analysis -commit 9bd5576290bcd43f9be6274b05ae31c00759cdb0 +commit b0d252c3d692f42687b0fa9635f9cd188e4ecbb2 Author: Elena Piscopia -Date: Tue Nov 26 12:53:44 2019 +0100 +Date: Wed Dec 4 16:22:11 2019 +0100 add script for kNN classification and plotting diff --git a/code/script.py b/code/script.py new file mode 100644 -index 0000000..d82606c +index 0000000..e43d58e --- /dev/null +++ b/code/script.py @@ -0,0 +1,41 @@ @@ -21,7 +21,7 @@ index 0000000..d82606c + +data = "input/iris.csv" + -+# make sure that the data is obtained (get will also install linked sub-ds!): ++# make sure that the data are obtained (get will also install linked sub-ds!): +dl.get(data) + +# prepare the data as a pandas dataframe diff --git a/docs/basics/_examples/DL-101-130-112 b/docs/basics/_examples/DL-101-130-112 index c0cb3846d..9951d9398 100644 --- a/docs/basics/_examples/DL-101-130-112 +++ b/docs/basics/_examples/DL-101-130-112 @@ -1,6 +1,6 @@ $ git log --oneline -be29815 [DATALAD RUNCMD] analyze iris data with classification analysis -9bd5576 add script for kNN classification and plotting -ebb126e [DATALAD] Recorded changes -b9ebebd Apply YODA dataset setup -fcd1201 [DATALAD] new dataset +a65e1c3 [DATALAD RUNCMD] analyze iris data with classification analysis +b0d252c add script for kNN classification and plotting +a4bc6dd [DATALAD] Recorded changes +b57de50 Apply YODA dataset setup +f770195 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-132-101 b/docs/basics/_examples/DL-101-132-101 index 7c1d131a8..f4abc9582 100644 --- a/docs/basics/_examples/DL-101-132-101 +++ b/docs/basics/_examples/DL-101-132-101 @@ -1,7 +1,7 @@ $ git log --oneline -b2c8f82 Provide project description -be29815 [DATALAD RUNCMD] analyze iris data with classification analysis -9bd5576 add script for kNN classification and plotting -ebb126e [DATALAD] Recorded changes -b9ebebd Apply YODA dataset setup -fcd1201 [DATALAD] new dataset +b217c8b Provide project description +a65e1c3 [DATALAD RUNCMD] analyze iris data with classification analysis +b0d252c add script for kNN classification and plotting +a4bc6dd [DATALAD] Recorded changes +b57de50 Apply YODA dataset setup +f770195 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-132-104 b/docs/basics/_examples/DL-101-132-104 index 9968c48d7..be3db24ad 100644 --- a/docs/basics/_examples/DL-101-132-104 +++ b/docs/basics/_examples/DL-101-132-104 @@ -1,3 +1,2 @@ $ datalad status midterm_project untracked: midterm_project/an_empty_file (file) - diff --git a/docs/basics/_examples/DL-101-132-110 b/docs/basics/_examples/DL-101-132-110 index 71279a872..0f7f472f7 100644 --- a/docs/basics/_examples/DL-101-132-110 +++ b/docs/basics/_examples/DL-101-132-110 @@ -1,4 +1,4 @@ -$ datalad save -m "finished my midterm project" +$ datalad save -d . -m "finished my midterm project!" midterm_project add(ok): midterm_project (file) save(ok): . (dataset) action summary: diff --git a/docs/basics/_examples/DL-101-132-112 b/docs/basics/_examples/DL-101-132-112 index 8a889506e..79ae077ec 100644 --- a/docs/basics/_examples/DL-101-132-112 +++ b/docs/basics/_examples/DL-101-132-112 @@ -1,14 +1,14 @@ $ git log -p -n 1 -commit c260b371c1992bd2b9c8758c0e88ded01ef62d84 +commit c4aaeb2476206ef8e5dc6346bae179eb6875139a Author: Elena Piscopia -Date: Fri Nov 29 09:52:55 2019 +0100 +Date: Wed Dec 4 16:22:34 2019 +0100 - finished my midterm project + finished my midterm project! diff --git a/midterm_project b/midterm_project -index 855f0b7..e2b9975 160000 +index b57de50..b217c8b 160000 --- a/midterm_project +++ b/midterm_project @@ -1 +1 @@ --Subproject commit 855f0b757d8ce85f46eacbb24e77a0df3ba30ee0 -+Subproject commit e2b9975103bfcbe7ecec853b1d6771d8d19de8ae +-Subproject commit b57de50bb54b60d348e0e7af1aab3a01c75f50a3 ++Subproject commit b217c8beaa36c19459cbc4ca5b7abbdb4623b61b diff --git a/docs/basics/_examples/DL-101-135-102 b/docs/basics/_examples/DL-101-135-102 index a3662bd8c..893672be3 100644 --- a/docs/basics/_examples/DL-101-135-102 +++ b/docs/basics/_examples/DL-101-135-102 @@ -17,6 +17,7 @@ $ datalad wtf - cmd:system-ssh: 7.9p1 - appdirs: 1.4.3 - boto: 2.49.0 + - exifread: 2.1.2 - git: 2.1.11 - gitdb: 2.0.5 - humanize: 0.5.1 @@ -24,6 +25,7 @@ $ datalad wtf - keyring: 19.0.2 - keyrings.alt: 3.1.1 - msgpack: 0.6.1 + - mutagen: 1.43.0 - requests: 2.22.0 - scrapy: 1.7.4 - wrapt: 1.11.2 @@ -37,6 +39,53 @@ $ datalad wtf - PATH: /home/adina/env/handbook/bin:/home/adina/Documents/freesurfer/bin:/home/adina/Documents/freesurfer/fsfast/bin:/home/adina/Documents/freesurfer/tktools:/usr/share/fsl/5.0/bin:/usr/lib/fsl/5.0:/home/adina/Documents/freesurfer/mni/bin:/usr/share/fsl/5.0/5.0/bin:/home/adina/.local/bin:/home/adina/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/adina/.dotfiles/bin - GIT_PYTHON_GIT_EXECUTABLE: /usr/bin/git ## extensions + - webapp: + - load_error: None + - description: Generic web app support + - module: datalad_webapp + - version: 0.2 + - entrypoints: + - datalad_webapp.WebApp: + - module: datalad_webapp + - class: WebApp + - names: + - webapp + - webapp + - load_error: None + - hirni: + - load_error: None + - description: HIRNI workflows + - module: datalad_hirni + - version: 0.0.4 + - entrypoints: + - datalad_hirni.commands.import_dicoms.ImportDicoms: + - module: datalad_hirni.commands.import_dicoms + - class: ImportDicoms + - names: + - hirni-import-dcm + - hirni_import_dcm + - load_error: None + - datalad_hirni.commands.spec4anything.Spec4Anything: + - module: datalad_hirni.commands.spec4anything + - class: Spec4Anything + - names: + - hirni-spec4anything + - hirni_spec4anything + - load_error: None + - datalad_hirni.commands.dicom2spec.Dicom2Spec: + - module: datalad_hirni.commands.dicom2spec + - class: Dicom2Spec + - names: + - hirni-dicom2spec + - hirni_dicom2spec + - load_error: None + - datalad_hirni.commands.spec2bids.Spec2Bids: + - module: datalad_hirni.commands.spec2bids + - class: Spec2Bids + - names: + - hirni-spec2bids + - hirni_spec2bids + - load_error: None - crawler: - load_error: None - description: Crawl web resources @@ -275,7 +324,8 @@ $ datalad wtf - load_error: None - audio: - module: datalad.metadata.extractors.audio - - load_error: No module named 'mutagen' [audio.py::17] + - version: None + - load_error: None - datacite: - module: datalad.metadata.extractors.datacite - version: None @@ -290,7 +340,8 @@ $ datalad wtf - load_error: None - exif: - module: datalad.metadata.extractors.exif - - load_error: No module named 'exifread' [exif.py::16] + - version: None + - load_error: None - frictionless_datapackage: - module: datalad.metadata.extractors.frictionless_datapackage - version: None @@ -301,7 +352,8 @@ $ datalad wtf - load_error: None - xmp: - module: datalad.metadata.extractors.xmp - - load_error: No module named 'libxmp' [xmp.py::20] + - version: None + - load_error: None ## python - version: 3.7.3 - implementation: CPython diff --git a/docs/basics/_examples/DL-101-136-101 b/docs/basics/_examples/DL-101-136-101 index 9568e1530..80867a469 100644 --- a/docs/basics/_examples/DL-101-136-101 +++ b/docs/basics/_examples/DL-101-136-101 @@ -2,8 +2,8 @@ $ cd books/ $ mv TLCL.pdf The_Linux_Command_Line.pdf $ ls -lah total 20K -drwxr-xr-x 2 adina adina 4.0K Nov 26 12:54 . -drwxr-xr-x 8 adina adina 4.0K Nov 26 12:54 .. -lrwxrwxrwx 1 adina adina 131 Nov 26 12:52 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf +drwxr-xr-x 2 adina adina 4.0K Dec 4 16:23 . +drwxr-xr-x 8 adina adina 4.0K Dec 4 16:23 .. +lrwxrwxrwx 1 adina adina 131 Dec 4 16:21 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf lrwxrwxrwx 1 adina adina 133 Jun 29 17:39 progit.pdf -> ../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf lrwxrwxrwx 1 adina adina 131 Jan 28 2019 The_Linux_Command_Line.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-104 b/docs/basics/_examples/DL-101-136-104 index 338fbed25..083f793b0 100644 --- a/docs/basics/_examples/DL-101-136-104 +++ b/docs/basics/_examples/DL-101-136-104 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit c8f6b4847d05d7f3c4602f55b49e12880256b494 +commit d33b1e63e51e98e2c891205e39377d7fec615828 Author: Elena Piscopia -Date: Tue Nov 26 12:54:26 2019 +0100 +Date: Wed Dec 4 16:23:02 2019 +0100 rename the book diff --git a/docs/basics/_examples/DL-101-136-105 b/docs/basics/_examples/DL-101-136-105 index 6c18b8f26..e97726ad1 100644 --- a/docs/basics/_examples/DL-101-136-105 +++ b/docs/basics/_examples/DL-101-136-105 @@ -1,3 +1,3 @@ $ git reset --hard HEAD~1 $ datalad status -HEAD is now at 11c2ecb add container and execute analysis within container +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-109 b/docs/basics/_examples/DL-101-136-109 index 8c34b1e2d..d45ccab89 100644 --- a/docs/basics/_examples/DL-101-136-109 +++ b/docs/basics/_examples/DL-101-136-109 @@ -1,4 +1,4 @@ $ git commit -m "rename book" -[master 6498fa3] rename book +[master feb1fca] rename book 1 file changed, 0 insertions(+), 0 deletions(-) rename books/{TLCL.pdf => The_Linux_Command_Line.pdf} (100%) diff --git a/docs/basics/_examples/DL-101-136-110 b/docs/basics/_examples/DL-101-136-110 index 6c18b8f26..e97726ad1 100644 --- a/docs/basics/_examples/DL-101-136-110 +++ b/docs/basics/_examples/DL-101-136-110 @@ -1,3 +1,3 @@ $ git reset --hard HEAD~1 $ datalad status -HEAD is now at 11c2ecb add container and execute analysis within container +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-121 b/docs/basics/_examples/DL-101-136-121 index bba4c21ee..c2eb7fb2d 100644 --- a/docs/basics/_examples/DL-101-136-121 +++ b/docs/basics/_examples/DL-101-136-121 @@ -1,3 +1,3 @@ $ cd ../ $ ls -l TLCL.pdf -lrwxrwxrwx 1 adina adina 131 Nov 26 12:54 TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 131 Dec 4 16:23 TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-122 b/docs/basics/_examples/DL-101-136-122 index 9c38196f2..19fc4417a 100644 --- a/docs/basics/_examples/DL-101-136-122 +++ b/docs/basics/_examples/DL-101-136-122 @@ -7,4 +7,4 @@ action summary: add (ok: 1) delete (ok: 1) save (ok: 1) -lrwxrwxrwx 1 adina adina 128 Nov 26 12:54 TLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 128 Dec 4 16:23 TLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-123 b/docs/basics/_examples/DL-101-136-123 index a46ffd8e4..8b47e5350 100644 --- a/docs/basics/_examples/DL-101-136-123 +++ b/docs/basics/_examples/DL-101-136-123 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit fc12e79c57deb44eb53231c516343acc6b40d8a0 +commit 687fb1db82c3448073944690de9e17ab89696af1 Author: Elena Piscopia -Date: Tue Nov 26 12:54:27 2019 +0100 +Date: Wed Dec 4 16:23:03 2019 +0100 moved book into root diff --git a/docs/basics/_examples/DL-101-136-124 b/docs/basics/_examples/DL-101-136-124 index 195a7fc5f..2df35b84f 100644 --- a/docs/basics/_examples/DL-101-136-124 +++ b/docs/basics/_examples/DL-101-136-124 @@ -1,2 +1,2 @@ $ git reset --hard HEAD~1 -HEAD is now at 11c2ecb add container and execute analysis within container +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-132 b/docs/basics/_examples/DL-101-136-132 index 54ff75700..3ba831961 100644 --- a/docs/basics/_examples/DL-101-136-132 +++ b/docs/basics/_examples/DL-101-136-132 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit 10b9fe8764cb2862199709d30c8c0bd10a2aab96 +commit 8759da9a782ba607e8d9c8137176104fa988296e Author: Elena Piscopia -Date: Tue Nov 26 12:54:28 2019 +0100 +Date: Wed Dec 4 16:23:04 2019 +0100 add copy of TLCL.pdf diff --git a/docs/basics/_examples/DL-101-136-133 b/docs/basics/_examples/DL-101-136-133 index 30f9ba160..470f1f444 100644 --- a/docs/basics/_examples/DL-101-136-133 +++ b/docs/basics/_examples/DL-101-136-133 @@ -1,4 +1,4 @@ $ ls -l copyofTLCL.pdf $ ls -l books/TLCL.pdf -lrwxrwxrwx 1 adina adina 128 Nov 26 12:54 copyofTLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf -lrwxrwxrwx 1 adina adina 131 Nov 26 12:54 books/TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 128 Dec 4 16:23 copyofTLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 131 Dec 4 16:23 books/TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-134 b/docs/basics/_examples/DL-101-136-134 index 195a7fc5f..2df35b84f 100644 --- a/docs/basics/_examples/DL-101-136-134 +++ b/docs/basics/_examples/DL-101-136-134 @@ -1,2 +1,2 @@ $ git reset --hard HEAD~1 -HEAD is now at 11c2ecb add container and execute analysis within container +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-140 b/docs/basics/_examples/DL-101-136-140 index b408ca36a..e4c364f2f 100644 --- a/docs/basics/_examples/DL-101-136-140 +++ b/docs/basics/_examples/DL-101-136-140 @@ -6,7 +6,7 @@ $ cat .git/config logallrefupdates = true editor = nano [annex] - uuid = 9d17effd-b251-48ab-92b2-b629c15a9b3e + uuid = fdbcfeff-5296-4e58-84dc-913343e50f4d version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -15,7 +15,7 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 7478bb68-77fb-466f-a3c6-a943d5b87569 + annex-uuid = 0cfd3714-d3a7-45ab-8e70-2f741f23e8ed annex-ignore = false [submodule "midterm_project"] url = /home/me/dl-101/DataLad-101/midterm_project diff --git a/docs/basics/_examples/DL-101-136-144 b/docs/basics/_examples/DL-101-136-144 index b3c5b4e5a..beb798feb 100644 --- a/docs/basics/_examples/DL-101-136-144 +++ b/docs/basics/_examples/DL-101-136-144 @@ -6,7 +6,7 @@ $ cat .git/config logallrefupdates = true editor = nano [annex] - uuid = 9d17effd-b251-48ab-92b2-b629c15a9b3e + uuid = fdbcfeff-5296-4e58-84dc-913343e50f4d version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -15,7 +15,7 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/onemoredir/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 7478bb68-77fb-466f-a3c6-a943d5b87569 + annex-uuid = 0cfd3714-d3a7-45ab-8e70-2f741f23e8ed annex-ignore = false [submodule "midterm_project"] url = /home/me/dl-101/DataLad-101/midterm_project diff --git a/docs/basics/_examples/DL-101-136-146 b/docs/basics/_examples/DL-101-136-146 index 4a60b0a04..dc93147a5 100644 --- a/docs/basics/_examples/DL-101-136-146 +++ b/docs/basics/_examples/DL-101-136-146 @@ -1,4 +1,4 @@ $ cd ../mock_user && mv onemoredir/DataLad-101 . $ rm -r onemoredir $ cd ../DataLad-101 && git reset --hard master -HEAD is now at 11c2ecb add container and execute analysis within container +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-152 b/docs/basics/_examples/DL-101-136-152 index 195a7fc5f..2df35b84f 100644 --- a/docs/basics/_examples/DL-101-136-152 +++ b/docs/basics/_examples/DL-101-136-152 @@ -1,2 +1,2 @@ $ git reset --hard HEAD~1 -HEAD is now at 11c2ecb add container and execute analysis within container +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-156 b/docs/basics/_examples/DL-101-136-156 index 2cfc8acb8..4547934c2 100644 --- a/docs/basics/_examples/DL-101-136-156 +++ b/docs/basics/_examples/DL-101-136-156 @@ -1,3 +1,3 @@ $ git reset --hard HEAD~1 warning: unable to rmdir 'longnow': Directory not empty -HEAD is now at 11c2ecb add container and execute analysis within container +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-164 b/docs/basics/_examples/DL-101-136-164 index 4e16be21e..e3e402d65 100644 --- a/docs/basics/_examples/DL-101-136-164 +++ b/docs/basics/_examples/DL-101-136-164 @@ -1,2 +1,2 @@ $ git reset --hard HEAD~2 -HEAD is now at 11c2ecb add container and execute analysis within container +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-174 b/docs/basics/_examples/DL-101-136-174 index fb90430ae..f85879600 100644 --- a/docs/basics/_examples/DL-101-136-174 +++ b/docs/basics/_examples/DL-101-136-174 @@ -1,6 +1,6 @@ $ git reset --hard HEAD~1 $ ls -HEAD is now at bd87516 Added flower mosaic from wikimedia +HEAD is now at a88c158 Added flower mosaic from wikimedia books code flowers.jpg diff --git a/docs/basics/_examples/DL-101-136-180 b/docs/basics/_examples/DL-101-136-180 index 4e16be21e..e3e402d65 100644 --- a/docs/basics/_examples/DL-101-136-180 +++ b/docs/basics/_examples/DL-101-136-180 @@ -1,2 +1,2 @@ $ git reset --hard HEAD~2 -HEAD is now at 11c2ecb add container and execute analysis within container +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-101 b/docs/basics/_examples/DL-101-137-101 index 4f095b9e5..5e125069d 100644 --- a/docs/basics/_examples/DL-101-137-101 +++ b/docs/basics/_examples/DL-101-137-101 @@ -1,16 +1,16 @@ $ git log -15 --oneline -11c2ecb add container and execute analysis within container -5e4c89a finished my midterm project! -ca88890 [DATALAD] Recorded changes -770e708 add note on DataLads procedures -ad89f03 add note on configurations and git config -0686627 Add note on adding siblings -26b599c Merge remote-tracking branch 'refs/remotes/roommate/master' -029add8 add note about datalad update -a2d01ff Include nesting demo from datalad website -2091c14 add note on git annex whereis -e2210e8 add note about installing from paths and recursive installations -66f97f3 add note on clean datasets -17e39f4 [DATALAD RUNCMD] Resize logo for slides -68ade50 [DATALAD RUNCMD] Resize logo for slides -048ae39 add additional notes on run options +d3ee8e9 add container and execute analysis within container +c4aaeb2 finished my midterm project! +ca76f3b [DATALAD] Recorded changes +73f14ee add note on DataLads procedures +9a795e3 add note on configurations and git config +da9d193 Add note on adding siblings +c15ea3e Merge remote-tracking branch 'refs/remotes/roommate/master' +aca5177 Include nesting demo from datalad website +1291140 add note about datalad update +1b07d87 add note on git annex whereis +ad17112 add note about installing from paths and recursive installations +199587f add note on clean datasets +dd726a6 [DATALAD RUNCMD] Resize logo for slides +e4dd212 [DATALAD RUNCMD] Resize logo for slides +b08d657 add additional notes on run options diff --git a/docs/basics/_examples/DL-101-137-105 b/docs/basics/_examples/DL-101-137-105 index 416f24aaa..710ee2252 100644 --- a/docs/basics/_examples/DL-101-137-105 +++ b/docs/basics/_examples/DL-101-137-105 @@ -1,7 +1,7 @@ $ git log -p -1 -commit 8669cd7d57a6aeca76884a4f8935ff0c6a24fed4 +commit a84bc8b2d463077e33803a673d762133d8bb3e2e Author: Elena Piscopia -Date: Tue Nov 26 12:54:39 2019 +0100 +Date: Wed Dec 4 16:23:17 2019 +0100 [DATALAD] Recorded changes diff --git a/docs/basics/_examples/DL-101-137-106 b/docs/basics/_examples/DL-101-137-106 index e85316dfb..c3aebf4d4 100644 --- a/docs/basics/_examples/DL-101-137-106 +++ b/docs/basics/_examples/DL-101-137-106 @@ -1,4 +1,4 @@ $ git log -n 3 --oneline -8669cd7 [DATALAD] Recorded changes -11c2ecb add container and execute analysis within container -5e4c89a finished my midterm project! +a84bc8b [DATALAD] Recorded changes +d3ee8e9 add container and execute analysis within container +c4aaeb2 finished my midterm project! diff --git a/docs/basics/_examples/DL-101-137-107 b/docs/basics/_examples/DL-101-137-107 index 7e1add6a8..53c16e35b 100644 --- a/docs/basics/_examples/DL-101-137-107 +++ b/docs/basics/_examples/DL-101-137-107 @@ -1,2 +1,2 @@ -$ git reset --mixed 11c2ecb85f22ad14f74809effc83b28346500e91 +$ git reset --mixed d3ee8e9a53e8c321e97ccea5b66fa19e7d34056a diff --git a/docs/basics/_examples/DL-101-137-108 b/docs/basics/_examples/DL-101-137-108 index f9d8c86f1..fe10dd22b 100644 --- a/docs/basics/_examples/DL-101-137-108 +++ b/docs/basics/_examples/DL-101-137-108 @@ -1,3 +1,3 @@ $ git log -n 2 --oneline -11c2ecb add container and execute analysis within container -5e4c89a finished my midterm project! +d3ee8e9 add container and execute analysis within container +c4aaeb2 finished my midterm project! diff --git a/docs/basics/_examples/DL-101-137-112 b/docs/basics/_examples/DL-101-137-112 index 3d993d5db..187b4f529 100644 --- a/docs/basics/_examples/DL-101-137-112 +++ b/docs/basics/_examples/DL-101-137-112 @@ -1,12 +1,12 @@ $ git log -2 -commit 43af7fea4a87d59bcfa985f71dd401c041831eef +commit b8972e080e665903f10c32c3d174211c282324e1 Author: Elena Piscopia -Date: Tue Nov 26 12:54:39 2019 +0100 +Date: Wed Dec 4 16:23:17 2019 +0100 save my favorite Git joke -commit 11c2ecb85f22ad14f74809effc83b28346500e91 +commit d3ee8e9a53e8c321e97ccea5b66fa19e7d34056a Author: Elena Piscopia -Date: Tue Nov 26 12:54:23 2019 +0100 +Date: Wed Dec 4 16:22:59 2019 +0100 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-114 b/docs/basics/_examples/DL-101-137-114 index a72f776e7..03c7dc3be 100644 --- a/docs/basics/_examples/DL-101-137-114 +++ b/docs/basics/_examples/DL-101-137-114 @@ -1,2 +1,2 @@ $ ls -l apdffile.pdf -lrwxrwxrwx 1 adina adina 122 Nov 26 12:54 apdffile.pdf -> .git/annex/objects/vp/vm/MD5E-s1842--2377e02244882c5bf3d5d6ebc5930e68.pdf/MD5E-s1842--2377e02244882c5bf3d5d6ebc5930e68.pdf +lrwxrwxrwx 1 adina adina 122 Dec 4 16:23 apdffile.pdf -> .git/annex/objects/jg/Qk/MD5E-s1842--ae82655e482e95369a942ef787ffda24.pdf/MD5E-s1842--ae82655e482e95369a942ef787ffda24.pdf diff --git a/docs/basics/_examples/DL-101-137-116 b/docs/basics/_examples/DL-101-137-116 index ee8a00cc7..d1a67faa3 100644 --- a/docs/basics/_examples/DL-101-137-116 +++ b/docs/basics/_examples/DL-101-137-116 @@ -1,2 +1,2 @@ $ ls -l apdffile.pdf --rw-r--r-- 1 adina adina 1842 Nov 26 12:54 apdffile.pdf +-rw-r--r-- 1 adina adina 1842 Dec 4 16:23 apdffile.pdf diff --git a/docs/basics/_examples/DL-101-137-117 b/docs/basics/_examples/DL-101-137-117 index 14bb8aa6c..bf482fa38 100644 --- a/docs/basics/_examples/DL-101-137-117 +++ b/docs/basics/_examples/DL-101-137-117 @@ -1,4 +1,4 @@ $ git log -n 3 --oneline -163f9e6 [DATALAD] Recorded changes -43af7fe save my favorite Git joke -11c2ecb add container and execute analysis within container +76f65c9 [DATALAD] Recorded changes +b8972e0 save my favorite Git joke +d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-118 b/docs/basics/_examples/DL-101-137-118 index 13927fcc7..5e2b03128 100644 --- a/docs/basics/_examples/DL-101-137-118 +++ b/docs/basics/_examples/DL-101-137-118 @@ -1,2 +1,2 @@ -$ git reset --mixed 43af7fea4a87d59bcfa985f71dd401c041831eef +$ git reset --mixed b8972e080e665903f10c32c3d174211c282324e1 diff --git a/docs/basics/_examples/DL-101-137-119 b/docs/basics/_examples/DL-101-137-119 index 4389c5f79..931ee555d 100644 --- a/docs/basics/_examples/DL-101-137-119 +++ b/docs/basics/_examples/DL-101-137-119 @@ -1,3 +1,3 @@ $ git log -n 2 --oneline -43af7fe save my favorite Git joke -11c2ecb add container and execute analysis within container +b8972e0 save my favorite Git joke +d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-121 b/docs/basics/_examples/DL-101-137-121 index 3950aa8cd..f08d27a55 100644 --- a/docs/basics/_examples/DL-101-137-121 +++ b/docs/basics/_examples/DL-101-137-121 @@ -1,21 +1,21 @@ $ git log -n 20 --oneline -43af7fe save my favorite Git joke -11c2ecb add container and execute analysis within container -5e4c89a finished my midterm project! -ca88890 [DATALAD] Recorded changes -770e708 add note on DataLads procedures -ad89f03 add note on configurations and git config -0686627 Add note on adding siblings -26b599c Merge remote-tracking branch 'refs/remotes/roommate/master' -029add8 add note about datalad update -a2d01ff Include nesting demo from datalad website -2091c14 add note on git annex whereis -e2210e8 add note about installing from paths and recursive installations -66f97f3 add note on clean datasets -17e39f4 [DATALAD RUNCMD] Resize logo for slides -68ade50 [DATALAD RUNCMD] Resize logo for slides -048ae39 add additional notes on run options -fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -8cdc201 resized picture by hand -11dfb75 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -5ca7d13 add note on basic datalad run and datalad rerun +b8972e0 save my favorite Git joke +d3ee8e9 add container and execute analysis within container +c4aaeb2 finished my midterm project! +ca76f3b [DATALAD] Recorded changes +73f14ee add note on DataLads procedures +9a795e3 add note on configurations and git config +da9d193 Add note on adding siblings +c15ea3e Merge remote-tracking branch 'refs/remotes/roommate/master' +aca5177 Include nesting demo from datalad website +1291140 add note about datalad update +1b07d87 add note on git annex whereis +ad17112 add note about installing from paths and recursive installations +199587f add note on clean datasets +dd726a6 [DATALAD RUNCMD] Resize logo for slides +e4dd212 [DATALAD RUNCMD] Resize logo for slides +b08d657 add additional notes on run options +3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +bd42809 resized picture by hand +86be52e [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +aadf74e add note on basic datalad run and datalad rerun diff --git a/docs/basics/_examples/DL-101-137-122 b/docs/basics/_examples/DL-101-137-122 index c22ad18ab..1ea969f22 100644 --- a/docs/basics/_examples/DL-101-137-122 +++ b/docs/basics/_examples/DL-101-137-122 @@ -1,7 +1,7 @@ -$ git checkout fa3fca41ffaafbfd1657203cd380273a70704c1e +$ git checkout 3a4418dc8d74b9889da49b26e0f201e169902113 warning: unable to rmdir 'midterm_project': Directory not empty -Note: switching to 'fa3fca41ffaafbfd1657203cd380273a70704c1e'. +Note: switching to '3a4418dc8d74b9889da49b26e0f201e169902113'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this @@ -18,4 +18,4 @@ Or undo this operation with: Turn off this advice by setting config variable advice.detachedHead to false -HEAD is now at fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +HEAD is now at 3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... diff --git a/docs/basics/_examples/DL-101-137-124 b/docs/basics/_examples/DL-101-137-124 index 034db2d13..8c48e3fd4 100644 --- a/docs/basics/_examples/DL-101-137-124 +++ b/docs/basics/_examples/DL-101-137-124 @@ -1,3 +1,3 @@ $ git checkout master -Previous HEAD position was fa3fca4 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +Previous HEAD position was 3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... Switched to branch 'master' diff --git a/docs/basics/_examples/DL-101-137-127 b/docs/basics/_examples/DL-101-137-127 index d85a6b40b..c08003943 100644 --- a/docs/basics/_examples/DL-101-137-127 +++ b/docs/basics/_examples/DL-101-137-127 @@ -1,5 +1,5 @@ -$ git cat-file --textconv fa3fca41ffaafbfd1657203cd380273a70704c1e:notes.txt +$ git cat-file --textconv 3a4418dc8d74b9889da49b26e0f201e169902113:notes.txt One can create a new dataset with 'datalad create [--description] PATH'. The dataset is created empty diff --git a/docs/basics/_examples/DL-101-137-132 b/docs/basics/_examples/DL-101-137-132 index 7c9dee39a..6c855c56a 100644 --- a/docs/basics/_examples/DL-101-137-132 +++ b/docs/basics/_examples/DL-101-137-132 @@ -1,3 +1,3 @@ $ git log -n 2 --oneline -9ab836f add joke evaluation to joke -43af7fe save my favorite Git joke +26b14ab add joke evaluation to joke +b8972e0 save my favorite Git joke diff --git a/docs/basics/_examples/DL-101-137-133 b/docs/basics/_examples/DL-101-137-133 index c2c5dd565..97624c928 100644 --- a/docs/basics/_examples/DL-101-137-133 +++ b/docs/basics/_examples/DL-101-137-133 @@ -1,3 +1,3 @@ -$ git reset --hard 43af7fea4a87d59bcfa985f71dd401c041831eef -HEAD is now at 43af7fe save my favorite Git joke +$ git reset --hard b8972e080e665903f10c32c3d174211c282324e1 +HEAD is now at b8972e0 save my favorite Git joke diff --git a/docs/basics/_examples/DL-101-137-144 b/docs/basics/_examples/DL-101-137-144 index 37aa182b8..02c4a491b 100644 --- a/docs/basics/_examples/DL-101-137-144 +++ b/docs/basics/_examples/DL-101-137-144 @@ -1,5 +1,5 @@ -$ git revert 7f888a04183c89c7e33e81cada7f89544b7b9c81 -[master 029320e] Revert "did a bad modification" - Date: Tue Nov 26 12:54:42 2019 +0100 +$ git revert 795f94ab667188877575dc4db900f3d6aed0a4a2 +[master 7df383f] Revert "did a bad modification" + Date: Wed Dec 4 16:23:20 2019 +0100 1 file changed, 1 deletion(-) diff --git a/docs/basics/_examples/DL-101-137-146 b/docs/basics/_examples/DL-101-137-146 index 882550552..a82344df5 100644 --- a/docs/basics/_examples/DL-101-137-146 +++ b/docs/basics/_examples/DL-101-137-146 @@ -1,20 +1,20 @@ $ git log -n 3 -commit 029320e95e2d7633aa447e07fd90d9a3ef8374ef +commit 7df383fd912c9f9841c4cca0161930e14fbb3c8d Author: Elena Piscopia -Date: Tue Nov 26 12:54:42 2019 +0100 +Date: Wed Dec 4 16:23:20 2019 +0100 Revert "did a bad modification" - This reverts commit 7f888a04183c89c7e33e81cada7f89544b7b9c81. + This reverts commit 795f94ab667188877575dc4db900f3d6aed0a4a2. -commit ff291229226a01b85429ce6f708e80bd68fa1fbb +commit 35d77bee63472970f76668cb4566758044ff1f4b Author: Elena Piscopia -Date: Tue Nov 26 12:54:42 2019 +0100 +Date: Wed Dec 4 16:23:20 2019 +0100 add note on helpful git resource -commit 7f888a04183c89c7e33e81cada7f89544b7b9c81 +commit 795f94ab667188877575dc4db900f3d6aed0a4a2 Author: Elena Piscopia -Date: Tue Nov 26 12:54:41 2019 +0100 +Date: Wed Dec 4 16:23:19 2019 +0100 did a bad modification From df90742de4d9defff56a78d5be48783b93d05218 Mon Sep 17 00:00:00 2001 From: Adina Wagner Date: Wed, 4 Dec 2019 17:04:18 +0100 Subject: [PATCH 33/33] typo --- docs/basics/101-133-containersrun.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst index 5ba5afbe6..3e674bfc2 100644 --- a/docs/basics/101-133-containersrun.rst +++ b/docs/basics/101-133-containersrun.rst @@ -78,7 +78,7 @@ Both of these tools share core terminology: It is made by a human user. **Image** - This is *build* from the recipe file. It is a static filesystem inside a file, + This is *built* from the recipe file. It is a static filesystem inside a file, populated with the software specified in the recipe, and some initial configuration. **Container** @@ -317,4 +317,4 @@ in the usecase :ref:`usecase_reproduce_neuroimg`. it grants users "`superuser privileges `_". On multi-user systems such as HPC, users should not have those privileges, as it would enable them to temper with other's or shared - data and resources, posing a severe security threat. \ No newline at end of file + data and resources, posing a severe security threat.