diff --git a/docs/basics/101-106-nesting.rst b/docs/basics/101-106-nesting.rst index ef3f22535..1793e4f26 100644 --- a/docs/basics/101-106-nesting.rst +++ b/docs/basics/101-106-nesting.rst @@ -84,8 +84,14 @@ we can set subdatasets to previous states, or *update* them. Previously, we used :command:`cd` to navigate into the subdataset, and subsequently opened the Git log. This is necessary, because a :command:`git log` - in the superdataset would only return the superdatasets history. There is one - trick, though: ``git -C`` lets you perform any Git command in a provided path. + in the superdataset would only return the superdatasets history. + While moving around with ``cd`` is straightforward, you also found it + slightly annoying from time to time to use the ``cd`` command so often and also + to remember in which directory you currently are in. There is one + trick, though: ``git -C`` (note that it is a capital C) lets you perform any + Git command in a provided path. Providing this option together with a path to + a Git command let's you run the command as if Git was started in this path + instead of the current working directory. Thus, from the root of ``DataLad-101``, this command would have given you the subdatasets history as well:: diff --git a/docs/basics/101-130-yodaproject.rst b/docs/basics/101-130-yodaproject.rst index 17cff706b..99cdd156f 100644 --- a/docs/basics/101-130-yodaproject.rst +++ b/docs/basics/101-130-yodaproject.rst @@ -1,3 +1,5 @@ +.. _yoda_project: + YODA-compliant data analysis projects ------------------------------------- diff --git a/docs/basics/101-132-advancednesting.rst b/docs/basics/101-132-advancednesting.rst index e051bb278..614c015c9 100644 --- a/docs/basics/101-132-advancednesting.rst +++ b/docs/basics/101-132-advancednesting.rst @@ -39,8 +39,11 @@ evolved. Let's query the superdataset what it thinks about this. $ datalad status From the superdataset's perspective, the subdataset appears as being -"modified". Note that these modifications of the subdataset are not -automatically recorded to the superdataset! This makes sense -- after all it +"modified". Note how it is not individual files that show up as "modified", but +indeed the complete subdataset as a single entity. + +What this shows you is that the modifications of the subdataset you performed are not +automatically recorded to the superdataset. This makes sense -- after all it should be up to you to decide whether you want record something or not --, but it is worth repeating: If you modify a subdataset, you will need to save this *in the superdataset* in order to have a clean superdataset status. @@ -163,13 +166,42 @@ interested in this, checkout the hidden section below. $ datalad status --recursive Let's save the modification of the subdataset into the history of the -superdataset: +superdataset. For this, to avoid confusion, you can specify explicitly to +which dataset you want to save a modification. ``-d .`` specifies the current +dataset, i.e., ``DataLad-101``, as the dataset to save to: .. runrecord:: _examples/DL-101-132-110 :language: console :workdir: dl-101/DataLad-101/ - $ datalad save -m "finished my midterm project" + $ datalad save -d . -m "finished my midterm project!" midterm_project + +.. findoutmore:: More on how save can operate on nested datasets + + In a superdataset with subdatasets, :command:`datalad save` by default + tries to figure out on its own which dataset's history of all available + datasets a :command:`save` should be written to. However, it can reduce + confusion or allow specific operations to be very explicit in the command + call and tell DataLad where to save what kind of modifications to. + + If you want to save the current state of the subdataset into the superdataset + (as necessary here), start a ``save`` from the superdataset and have the + ``-d/--dataset`` option point to its root:: + + # in the root of the superds + $ datalad save -d . -m "update subdataset" + + If you are in the superdataset, and you want to save an unsaved modification + in a subdataset to the *subdatasets* history, let ``-d/--dataset`` point to + the subdataset:: + + # in the superds + $ datalad save -d path/to/subds -m "modified XY" + + The recursive option allows you to save any content underneath the specified + directory, and recurse into any potential subdatasets:: + + $ datalad save . --recursive Let's check which subproject commit is now recorded in the superdataset: diff --git a/docs/basics/101-132-gitC.rst b/docs/basics/101-132-gitC.rst deleted file mode 100644 index e5b21c348..000000000 --- a/docs/basics/101-132-gitC.rst +++ /dev/null @@ -1,15 +0,0 @@ -Working in (sub-)datasets from afar -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Up until now, we made sure to diligently ``cd`` into the precise dataset or -subdataset we wanted to work with. This was straightforward and easy, but, to -be honest, you found it slightly annoying from time to time as well. - -Luckily, Git has an option to run commands in a path different from the current -working directory: ``-C`` (note that it is a capital C). Providing this -option together with a path to any Git command let's you run the command as -if Git was started in this path instead of the current working directory. - -.. todo:: - - this needs some examples and practice. \ No newline at end of file diff --git a/docs/basics/101-133-containersrun.rst b/docs/basics/101-133-containersrun.rst new file mode 100644 index 000000000..3e674bfc2 --- /dev/null +++ b/docs/basics/101-133-containersrun.rst @@ -0,0 +1,320 @@ +.. _containersrun: + +Computational reproducibility with software containers +------------------------------------------------------ + +Just after submitting your midterm data analysis project, you get together +with your friends. "I'm curious: So what kind of analyses did y'all carry out?" +you ask. The variety of methods and datasets the others used is huge, and +one analysis interests you in particular. Later that day, you decide to +install this particular analysis dataset to learn more about the methods used +in there. However, when you :command:`re-run` your friends analysis script, +it throws an error. Hastily, you call her -- maybe she can quickly fix her +script and resubmit the project with only minor delays. "I don't know what +you mean", you hear in return. +"On my machine, everything works fine!" + + +On its own, DataLad datasets can contain almost anything that is relevant to +ensure reproducibility: Data, code, human-readable analysis descriptions +(e.g., ``README.md`` files), provenance on the origin of all files +obtained from elsewhere, and machine-readable records that link generated +outputs to the commands, scripts, and data they were created from. + +This however may not be sufficient to ensure that an analysis *reproduces* +(i.e., produces the same or highly similar results), let alone *works* on a +computer different than the one it was initially composed on. This is because +the analysis does not only depend on data and code, but also the +*software environment* that it is conducted in. + +A lack of information about the operating system of the computer, the precise +versions of installed software, or their configurations may +make it impossible to replicate your analysis on a different machine, or even +on your own machine once a new software update is installed. Therefore, it is +important to communicate all details about the computational environment for +an analysis as thoroughly as possible. Luckily, DataLad provides an extension +that can link computational environments to datasets, the +`datalad containers `_ +extension [#f1]_. + +This section will give a quick overview on what containers are and +demonstrate how ``datalad-containers`` helps to capture full provenance of an +analysis by linking containers to datasets and analyses. + +Containers +^^^^^^^^^^ + +To put it simple, computational containers are cut-down virtual machines that +allow you to package all software libraries and their dependencies (all in the +precise version your analysis requires) into a bundle you can share with +others. On your own and other's machines, the container constitutes a secluded +software environment that + +- contains the exact software environment that you specified, ready to run + analyses in +- does not effect any software outside of the container + +Unlike virtual machines, software containers do not have their own operating +system. Instead, they use basic services of the underlying operating system +of the computer they run on (in a read-only fashion). This makes them +lightweight and portable. By sharing software environments with containers, +others (and also yourself) have easy access to the correct software +without the need to modify the software environment of the machine the +container runs on. Thus, containers are ideal to encapsulate the software +environment and share it together with the analysis code and data to ensure +computational reproducibility of your analyses, or to create a suitable +software environment on a computer that you do not have permissions to deploy +software on. + +There are a number of different tools to create and use containers, with +`Docker `_ being one of the most well-known of them. +While being a powerful tool, it is only rarely used on high performance computing +(HPC) infrastructure [#f2]_. An alternative is `Singularity `_. +Both of these tools share core terminology: + +**Recipe** + A text file template that lists all required components of the computational environment. + It is made by a human user. + +**Image** + This is *built* from the recipe file. It is a static filesystem inside a file, + populated with the software specified in the recipe, and some initial configuration. + +**Container** + A running instance of an Image that you can actually use for your computations. + If you want to create and run your own software container, you start by writing + a recipe file and build an Image from it. Alternatively, you can can also *pull* + an Image built from a publicly shared recipe from the *Hub* of the tool you are using. + +**Hub** + A storage resource to share and consume images. There is + `Singularity-Hub `_ and + `Docker-Hub `_. Both are optional, additional services + not required to use software containers, but a convenient way to share recipes + and have imaged built from them by a service (instead of building them + manually and locally). + +Note that as of now, the ``datalad-containers`` extension supports +Singularity and Docker images. +Singularity furthermore is compatible with Docker -- you can use +Docker Images as a basis for Singularity Images, or run Docker Images with +Singularity (even without having Docker installed). + +.. note:: + + In order to use Singularity containers (and thus ``datalad containers``), you have to + `install `_ the software singularity. + +Using ``datalad containers`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. index:: ! datalad command; containers-add +.. index:: ! datalad command; containers-run + +One core feature of the ``datalad containers`` extension is that it registers +computational containers to a dataset. This is done with the +:command:`datalad containers-add` command. +Once a container is registered, arbitrary commands can be executed inside of +it, i.e., in the precise software environment the container encapsulates. All it +needs for this it to swap the :command:`datalad run` command introduced in +section :ref:`run` with the :command:`datalad containers-run` command. + +Let's see this in action for the ``midterm_analysis`` dataset by rerunning +the analysis you did for the midterm project within a Singularity container. +We start by registering a container to the dataset. +For this, we will pull an Image from Singularity hub. This Image was made +for the handbook, and it contains the relevant Python setup for +the analysis. Its recipe lives in the handbook's +`resources repository `_, and the +Image is built from the recipe via Singularity hub. +If you're curious how to create a Singularity Image, the hidden +section below has some pointers: + +.. findoutmore:: How to make a Singularity Image + + Singularity containers are build from Image files, often + called "recipes", that hold a "definition" of the software container and its + contents and components. The + `singularity documentation `_ + has its own tutorial on how to build such Images from scratch. + An alternative to writing the Image file by hand is to use + `Neurodocker `_. This + command-line program can help you generate custom Singularity recipes (and + also ``Dockerfiles``, from which Docker Images are build). A wonderful tutorial + on how to use Neurodocker is + `this introduction `_ + by Michael Notter. + + Once a recipe exists, the command + + .. code-block:: bash + + sudo singularity build + + will build a container (called ````) from the recipe. Note that this + command requires ``root`` privileges ("``sudo``"). You can build the container + on any machine, though, not necessarily the one that is later supposed to + actually run the analysis, e.g., your own laptop versus a compute cluster. + Alternatively, `Singularity Hub `_ integrates + with Github and builds containers from Images pushed to repositories on Github. + `The docs `_ + give you a set of instructions on how to do this. + +The :command:`datalad containers-add` command takes an arbitrary +name to give to the container, and a path or url to a container Image: + +.. runrecord:: _examples/DL-101-133-101 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + # we are in the midterm_project subdataset + $ datalad containers-add midterm-software --url shub://adswa/resources:1 + + +This command downloaded the container from Singularity Hub, added it to +the ``midterm_project`` dataset, and recorded basic information on the +container under its name "midterm-software" in the dataset's configuration at +``.datalad/config``. + +.. findoutmore:: What has been added to .datalad/config? + + .. runrecord:: _examples/DL-101-133-102 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ cat .datalad/config + + This recorded the Image's origin on Singularity-Hub, the location of the + Image in the dataset under ``.datalad/environments//image``, and it + specifies the way in which the container should be used: The line + + .. code-block:: bash + + cmdexec = singularity exec {img} {cmd} + + can be read as: "If this container is used, take the ``cmd`` (what you wrap in a + :command:`datalad containers-run` command) and plug it into a + :command:`singularity exec` command. The mode of calling Singularity, + namely ``exec``, means that the command will be executed inside of the container. + + Note that the Image is saved under ``.datalad/environments`` and the + configuration is done in ``.datalad/config`` -- as these files are version + controlled and shared with together with a dataset, your software + container and the information where it can be re-obtained from are linked + to your dataset. + + This is how the ``containers-add`` command is recorded in your history: + + .. runrecord:: _examples/DL-101-133-103 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ git log -n 1 -p + +Now that we have a complete computational environment linked to the ``midterm_project`` +dataset, we can execute commands in this environment. Let us for example try to repeat +the :command:`datalad run` command from the section :ref:`yoda_project` as a +:command:`datalad containers-run` command. + +The previous ``run`` command looked like this:: + + $ datalad run -m "analyze iris data with classification analysis" \ + --input "input/iris.csv" \ + --output "prediction_report.csv" \ + --output "pairwise_relationships.png" \ + "python3 code/script.py" + +How would it look like as a ``containers-run`` command? + +.. runrecord:: _examples/DL-101-133-104 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ datalad containers-run -m "rerun analysis in container" \ + --container-name midterm-software \ + --input "input/iris.csv" \ + --output "prediction_report.csv" \ + --output "pairwise_relationships.png" \ + "python3 code/script.py" + +Almost exactly like a :command:`datalad run` command! The only additional parameter +is ``container-name``. At this point, though, the ``--container-name`` +flag is even *optional* because there is only a single container registered to the dataset. +But if your dataset contains more than one container you will *need* to specify +the name of the container you want to use in your command. +The complete command's structure looks like this:: + + $ datalad containers-run --name [-m ...] [--input ...] [--output ...] + +.. index:: ! datalad command; containers-remove +.. index:: ! datalad command; containers-list + +.. findoutmore:: How can I list available containers or remove them? + + The command :command:`datalad containers-list` will list all containers in + the current dataset: + + .. runrecord:: _examples/DL-101-133-110 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + + $ datalad containers-list + + The command :command:`datalad containers-remove` will remove a container + from the dataset, if there exists a container with name given to the + command. Note that this will remove not only the Image from the dataset, + but also the configuration for it in ``.datalad/config``. + + +Here is how the history entry looks like: + +.. runrecord:: _examples/DL-101-133-111 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ git log -p -n 1 + +If you would :command:`rerun` this commit, it would be re-executed in the +software container registered to the dataset. If you would share the dataset +with a friend and they would :command:`rerun` this commit, the Image would first +be obtained from its registered url, and thus your +friend can obtain the correct execution environment automatically. + +Note that because this new :command:`containers-run` command modified the +``midterm_project`` subdirectory, we need to also save +the most recent state of the subdataset to the superdataset ``DataLad-101``. + +.. runrecord:: _examples/DL-101-133-112 + :language: console + :workdir: dl-101/DataLad-101/midterm_project + + $ cd ../ + $ datalad status + +.. runrecord:: _examples/DL-101-133-113 + :language: console + :workdir: dl-101/DataLad-101 + + $ datalad save -d . -m "add container and execute analysis within container" midterm_project + + +Software containers, the ``datalad-containers`` extension, and DataLad thus work well together +to make your analysis completely reproducible -- by not only linking code, data, +and outputs, but also the software environment of an analysis. And this does not +only benefit your future self, but also whomever you share your dataset with, as +the information about the container is shared together with the dataset. How cool +is that? + +If you are interested in more, you can read about another example of :command:`datalad containers-run` +in the usecase :ref:`usecase_reproduce_neuroimg`. + +.. rubric:: Footnotes + +.. [#f1] To read more about DataLad's extensions, see section :ref:`extensions_intro`. +.. [#f2] The main reason why Docker is not deployed on HPC systems is because + it grants users "`superuser privileges `_". + On multi-user systems such as HPC, users should not have those + privileges, as it would enable them to temper with other's or shared + data and resources, posing a severe security threat. diff --git a/docs/basics/101-134-summary.rst b/docs/basics/101-134-summary.rst new file mode 100644 index 000000000..026169118 --- /dev/null +++ b/docs/basics/101-134-summary.rst @@ -0,0 +1,57 @@ +.. _summary_containers: + +Summary +------- + +The last two sections have first of all extended your knowledge on dataset nesting: + +- When subdatasets are created or installed, they are registered to the superdataset + in their current version state (as identified by their most recent commit's hash). + For a freshly created subdatasets, the most recent commit is at the same time its + first commit. + +- Once the subdataset evolves, the superdataset recognizes this as a ``modification`` + of the subdatasets version state. If you want to record this, you need to + :command:`save` it in the superdataset:: + + $ datalad save -m "a short summary of changes in subds" + +But more than nesting concepts, they have also extended your knowledge on +reproducible analyses with :command:`datalad run` and you have experienced +for yourself why and how software containers can go hand-in-hand with DataLad: + +- A software container encapsulates a complete software environment, independent + from the environment of the computer it runs on. This allows you to create or + use secluded software and also share it together with your analysis to ensure + computational reproducibility. + +- The command :command:`datalad containers-add` registers an Image from a path or + url to your dataset. + +- If you use :command:`datalad containers-run` instead of :command:`datalad run`, + you can reproducibly execute a command of your choice *within* the software + environment. + +- A :command:`datalad rerun` of a commit produced with :command:`datalad containers-run` + will re-execute the command in the same software environment. + +Now what can I do with it? +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For one, you will not be surprised if you ever see a subdataset being shown as +``modified`` by :command:`datalad status`: You now know that if a subdataset +evolves, it's most recent state needs to be explicitly saved to the superdatasets +history. + +On a different matter, you are now able to capture and share analysis provenance that +includes the relevant software environment. This does not only make your analyses +projects automatically reproducible, but automatically *computationally* reproducible - +you can make sure that your analyses runs on any computer with Singularity, +regardless of the software environment on this computer. Even if you are unsure how you can wrap up an +environment into a software container Image at this point, you could make use of +hundreds of publicly available Images on `Singularity-Hub `_ and +`Docker-Hub `_. + +With this, you have also gotten a first glimpse into an extension of DataLad: A +Python module you can install with Python package managers such as ``pip`` that +extends DataLad's functionality. diff --git a/docs/basics/101-144-intro_extensions.rst b/docs/basics/101-144-intro_extensions.rst new file mode 100644 index 000000000..e4c6a3422 --- /dev/null +++ b/docs/basics/101-144-intro_extensions.rst @@ -0,0 +1,57 @@ +.. _extensions_intro: + +DataLad's extensions +-------------------- + +The commands DataLad provides cover a broad range of domain-agnostic use cases. +However, there are extension packages that can add (domain-specific) +functionality and new commands. + +Such extensions are shipped as separate Python packages, and are *not* included in +DataLad itself. Instead, users with the need for a particular extension can +install the extension package -- either on top of DataLad if DataLad is already +installed, or on its own (the extension will then pull in DataLad core +automatically, with no need to first or simultaneously install DataLad itself +explicitly). The installation is done with +standard Python package managers, such as :term:`pip`, and beyond installation +of the package, no additional setup is required. + +Among others (a full list can be found on `PyPi `_), +the following DataLad extensions are available: + +.. list-table:: + :widths: 50 100 + :header-rows: 1 + + * - Extension name + - Description + * - `DataLad Container `_ + - Equips DataLad's :command:`run`/:command:`rerun` functionality with + the ability to transparently execute commands in containerized + computational environments. The section :ref:`containersrun` demonstrates + how this extension can be used, as well as the usecase :ref:`usecase_reproduce_neuroimg`. + * - `DataLad Neuroimaging `_ + - Metadata extraction support for a range of standards common to + neuroimaging data. The usecase :ref:`usecase_reproduce_neuroimg` demonstrates + how this extension can be used. + * - `DataLad Metalad `_ + - Equips DataLad with an alternative command suite and advanced tooling + for metadata handling (extraction, aggregation, reporting). + + .. todo:: + + once section on metadata is done, link it here + +To install a DataLad extension, use + +.. code-block:: bash + + $ pip install + +such as in + +.. code-block:: bash + + $ pip install datalad-container + +Afterwards, the new DataLad functionality the extension provides is readily available. \ No newline at end of file diff --git a/docs/basics/_examples/DL-101-101-103 b/docs/basics/_examples/DL-101-101-103 index 1d6d17cd1..8ed885834 100644 --- a/docs/basics/_examples/DL-101-101-103 +++ b/docs/basics/_examples/DL-101-101-103 @@ -1,12 +1,12 @@ $ git log -commit 758f0af5a5d8865db0f8aac4717d8fa93771d8a4 +commit 6c2e4854217a5cbb06eb9b3d0c476573ee2b3ced Author: Elena Piscopia -Date: Mon Nov 18 10:49:52 2019 +0100 +Date: Wed Dec 4 16:21:12 2019 +0100 Instruct annex to add text files to Git -commit 715acb08af2fdab8d901180c3d9985a5335b0584 +commit 623d3f641b8c6e948a036a7be9ec4f056d1ae024 Author: Elena Piscopia -Date: Mon Nov 18 10:49:52 2019 +0100 +Date: Wed Dec 4 16:21:12 2019 +0100 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-102-103 b/docs/basics/_examples/DL-101-102-103 index 935d1f3ac..238150f7b 100644 --- a/docs/basics/_examples/DL-101-102-103 +++ b/docs/basics/_examples/DL-101-102-103 @@ -3,5 +3,5 @@ $ wget https://sourceforge.net/projects/linuxcommand/files/TLCL/19.01/TLCL-19.01 $ wget https://www.gitbook.com/download/pdf/book/swaroopch/byte-of-python -O byte-of-python.pdf # get back into the root of the dataset $ cd ../ -2019-11-18 10:50:31 URL:https://vorboss.dl.sourceforge.net/project/linuxcommand/TLCL/19.01/TLCL-19.01.pdf [2120211/2120211] -> "TLCL.pdf" [1] -2019-11-18 10:50:35 URL:https://legacy.gitbook.com/download/pdf/book/swaroopch/byte-of-python [4407669] -> "byte-of-python.pdf" [1] +2019-12-04 16:21:19 URL:https://netcologne.dl.sourceforge.net/project/linuxcommand/TLCL/19.01/TLCL-19.01.pdf [2120211/2120211] -> "TLCL.pdf" [1] +2019-12-04 16:21:21 URL:https://legacy.gitbook.com/download/pdf/book/swaroopch/byte-of-python [4407669] -> "byte-of-python.pdf" [1] diff --git a/docs/basics/_examples/DL-101-102-107 b/docs/basics/_examples/DL-101-102-107 index fe7591bac..2e0db66ba 100644 --- a/docs/basics/_examples/DL-101-102-107 +++ b/docs/basics/_examples/DL-101-102-107 @@ -1,7 +1,7 @@ $ git log -p -n 1 -commit 464b1e50a588101cff01de9d0a95ad1f7cfc0457 +commit c2dcdaaf0107c3aceac2fc430f8e28da257434d1 Author: Elena Piscopia -Date: Mon Nov 18 10:50:36 2019 +0100 +Date: Wed Dec 4 16:21:22 2019 +0100 add books on Python and Unix to read later diff --git a/docs/basics/_examples/DL-101-102-108 b/docs/basics/_examples/DL-101-102-108 index 92b366f4d..1706ccf4e 100644 --- a/docs/basics/_examples/DL-101-102-108 +++ b/docs/basics/_examples/DL-101-102-108 @@ -1,4 +1,4 @@ $ cd books $ wget https://github.com/progit/progit2/releases/download/2.1.154/progit.pdf $ cd ../ -2019-11-18 10:51:14 URL:https://github-production-release-asset-2e65be.s3.amazonaws.com/15400220/57552a00-9a49-11e9-9144-d9607ed4c2db?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191118T095037Z&X-Amz-Expires=300&X-Amz-Signature=ee3c1f736354cc22e799692c8f5ffe9a7501eac269b98a4f0d915e1d08264530&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprogit.pdf&response-content-type=application%2Foctet-stream [12465653/12465653] -> "progit.pdf" [1] +2019-12-04 16:21:24 URL:https://github-production-release-asset-2e65be.s3.amazonaws.com/15400220/57552a00-9a49-11e9-9144-d9607ed4c2db?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191204%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191204T152122Z&X-Amz-Expires=300&X-Amz-Signature=66df788d2f7db5a587cd679928904ddf3735912c6589aa767a65fdb322422ec5&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dprogit.pdf&response-content-type=application%2Foctet-stream [12465653/12465653] -> "progit.pdf" [1] diff --git a/docs/basics/_examples/DL-101-102-111 b/docs/basics/_examples/DL-101-102-111 index 2678b3f23..405300987 100644 --- a/docs/basics/_examples/DL-101-102-111 +++ b/docs/basics/_examples/DL-101-102-111 @@ -1,6 +1,6 @@ # lets make the output a bit more concise with the --oneline option $ git log --oneline -8d6a681 add reference book about git -464b1e5 add books on Python and Unix to read later -758f0af Instruct annex to add text files to Git -715acb0 [DATALAD] new dataset +3d3987b add reference book about git +c2dcdaa add books on Python and Unix to read later +6c2e485 Instruct annex to add text files to Git +623d3f6 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-103-107 b/docs/basics/_examples/DL-101-103-107 index 69b27d27d..fa3c87deb 100644 --- a/docs/basics/_examples/DL-101-103-107 +++ b/docs/basics/_examples/DL-101-103-107 @@ -1,7 +1,7 @@ $ git log -p -n 2 -commit 84ce7c5a1a47f2ff9b27b7e9865c269d0d46eee8 +commit c56f7cb5c0637dee0ad15c5e0d99e68e1d7fd4f5 Author: Elena Piscopia -Date: Mon Nov 18 10:51:15 2019 +0100 +Date: Wed Dec 4 16:21:25 2019 +0100 add note on datalad save @@ -18,9 +18,9 @@ index 3a7a1fe..bfa64d7 100644 +Always use informative, concise commit messages. + -commit d792dfb3711a31f407dad8705a3f3283372844bf +commit 242271452e13ca4f06e578b82788099a1317c158 Author: Elena Piscopia -Date: Mon Nov 18 10:51:15 2019 +0100 +Date: Wed Dec 4 16:21:25 2019 +0100 Add notes on datalad create diff --git a/docs/basics/_examples/DL-101-106-101 b/docs/basics/_examples/DL-101-106-101 index 268e0d76a..25b6f27c9 100644 --- a/docs/basics/_examples/DL-101-106-101 +++ b/docs/basics/_examples/DL-101-106-101 @@ -1,7 +1,7 @@ $ git log -p -n 2 -commit fd5cbf6987e377b5a38c4f4df304c22da8d6b597 +commit 8db7558a987c679de83e63788611cf2ff34143e9 Author: Elena Piscopia -Date: Mon Nov 18 10:52:29 2019 +0100 +Date: Wed Dec 4 16:21:39 2019 +0100 Add note on datalad install @@ -20,9 +20,9 @@ index bfa64d7..6d8cdce 100644 +root of the superdataset with the '-d' option. + -commit d4b91a53f95557adc4b78521af5b131c02fc35f6 +commit ab0b11f4292c900b81f633cda09ee8bf749195e8 Author: Elena Piscopia -Date: Mon Nov 18 10:51:18 2019 +0100 +Date: Wed Dec 4 16:21:27 2019 +0100 [DATALAD] Recorded changes @@ -44,9 +44,9 @@ index 0000000..dcc34fb @@ -0,0 +1 @@ +Subproject commit dcc34fbe669b06ced84ced381ba0db21cf5e665f -commit 84ce7c5a1a47f2ff9b27b7e9865c269d0d46eee8 +commit c56f7cb5c0637dee0ad15c5e0d99e68e1d7fd4f5 Author: Elena Piscopia -Date: Mon Nov 18 10:51:15 2019 +0100 +Date: Wed Dec 4 16:21:25 2019 +0100 add note on datalad save @@ -63,9 +63,9 @@ index 3a7a1fe..bfa64d7 100644 +Always use informative, concise commit messages. + -commit d792dfb3711a31f407dad8705a3f3283372844bf +commit 242271452e13ca4f06e578b82788099a1317c158 Author: Elena Piscopia -Date: Mon Nov 18 10:51:15 2019 +0100 +Date: Wed Dec 4 16:21:25 2019 +0100 Add notes on datalad create @@ -79,9 +79,9 @@ index 0000000..3a7a1fe +The dataset is created empty + -commit 8d6a681c2145aacaebf4855f675e892355560640 +commit 3d3987be908a115c4cd044705af0d51a25e2f47d Author: Elena Piscopia -Date: Mon Nov 18 10:51:14 2019 +0100 +Date: Wed Dec 4 16:21:24 2019 +0100 add reference book about git @@ -94,9 +94,9 @@ index 0000000..c5d2ab2 +../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf \ No newline at end of file -commit 464b1e50a588101cff01de9d0a95ad1f7cfc0457 +commit c2dcdaaf0107c3aceac2fc430f8e28da257434d1 Author: Elena Piscopia -Date: Mon Nov 18 10:50:36 2019 +0100 +Date: Wed Dec 4 16:21:22 2019 +0100 add books on Python and Unix to read later @@ -117,9 +117,9 @@ index 0000000..9a812a0 +../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf \ No newline at end of file -commit 758f0af5a5d8865db0f8aac4717d8fa93771d8a4 +commit 6c2e4854217a5cbb06eb9b3d0c476573ee2b3ced Author: Elena Piscopia -Date: Mon Nov 18 10:49:52 2019 +0100 +Date: Wed Dec 4 16:21:12 2019 +0100 Instruct annex to add text files to Git @@ -136,9 +136,9 @@ index c3aaefe..b223f34 100644 +* annex.largefiles=(not(mimetype=text/*)and(largerthan=0)) \ No newline at end of file -commit 715acb08af2fdab8d901180c3d9985a5335b0584 +commit 623d3f641b8c6e948a036a7be9ec4f056d1ae024 Author: Elena Piscopia -Date: Mon Nov 18 10:49:52 2019 +0100 +Date: Wed Dec 4 16:21:12 2019 +0100 [DATALAD] new dataset @@ -155,12 +155,12 @@ index 0000000..b540820 \ No newline at end of file diff --git a/.datalad/config b/.datalad/config new file mode 100644 -index 0000000..ea9d4d6 +index 0000000..95115e5 --- /dev/null +++ b/.datalad/config @@ -0,0 +1,2 @@ +[datalad "dataset"] -+ id = c4755e2a-09e8-11ea-98f1-e86a64c8054c ++ id = b47220ee-16a9-11ea-b652-d0c637c523bc diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..c3aaefe diff --git a/docs/basics/_examples/DL-101-108-104 b/docs/basics/_examples/DL-101-108-104 index 62a99527f..f92f6b8c0 100644 --- a/docs/basics/_examples/DL-101-108-104 +++ b/docs/basics/_examples/DL-101-108-104 @@ -1,4 +1,4 @@ -$ datalad save -m "Add simple script to write a list of podcast speakers and titles" +$ datalad save -m "Add short script to write a list of podcast speakers and titles" add(ok): code/list_titles.sh (file) save(ok): . (dataset) action summary: diff --git a/docs/basics/_examples/DL-101-108-106 b/docs/basics/_examples/DL-101-108-106 index 60692351a..bb82894b9 100644 --- a/docs/basics/_examples/DL-101-108-106 +++ b/docs/basics/_examples/DL-101-108-106 @@ -1,7 +1,7 @@ $ git log -p -n 1 -commit c64173fe0a41bac4865c71f814bcbb84cd1c8146 +commit 281a9a48a1451e85c5a9284b6b908c93a987654c Author: Elena Piscopia -Date: Mon Nov 18 10:52:30 2019 +0100 +Date: Wed Dec 4 16:21:41 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles @@ -9,7 +9,7 @@ Date: Mon Nov 18 10:52:30 2019 +0100 { "chain": [], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-108-108 b/docs/basics/_examples/DL-101-108-108 index 3a485e03d..3480ce1a3 100644 --- a/docs/basics/_examples/DL-101-108-108 +++ b/docs/basics/_examples/DL-101-108-108 @@ -1,11 +1,11 @@ $ git log --oneline -c64173f [DATALAD RUNCMD] create a list of podcast titles -9d578af Add simple script to write a list of podcast speakers and titles -fd5cbf6 Add note on datalad install -d4b91a5 [DATALAD] Recorded changes -84ce7c5 add note on datalad save -d792dfb Add notes on datalad create -8d6a681 add reference book about git -464b1e5 add books on Python and Unix to read later -758f0af Instruct annex to add text files to Git -715acb0 [DATALAD] new dataset +281a9a4 [DATALAD RUNCMD] create a list of podcast titles +61f3a4d Add short script to write a list of podcast speakers and titles +8db7558 Add note on datalad install +ab0b11f [DATALAD] Recorded changes +c56f7cb add note on datalad save +2422714 Add notes on datalad create +3d3987b add reference book about git +c2dcdaa add books on Python and Unix to read later +6c2e485 Instruct annex to add text files to Git +623d3f6 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-108-113 b/docs/basics/_examples/DL-101-108-113 index b5e899e8e..8edee1219 100644 --- a/docs/basics/_examples/DL-101-108-113 +++ b/docs/basics/_examples/DL-101-108-113 @@ -1,13 +1,13 @@ $ git log -n 2 -commit 854a655e0fb6e822739a8e83c2460fe58c107204 +commit 16787003e96f58235c73d4865f899fe2fdd85b2f Author: Elena Piscopia -Date: Mon Nov 18 10:52:32 2019 +0100 +Date: Wed Dec 4 16:21:42 2019 +0100 BF: list both directories content -commit c64173fe0a41bac4865c71f814bcbb84cd1c8146 +commit 281a9a48a1451e85c5a9284b6b908c93a987654c Author: Elena Piscopia -Date: Mon Nov 18 10:52:30 2019 +0100 +Date: Wed Dec 4 16:21:41 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles @@ -15,7 +15,7 @@ Date: Mon Nov 18 10:52:30 2019 +0100 { "chain": [], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-108-114 b/docs/basics/_examples/DL-101-108-114 index 91a031085..adfed2e41 100644 --- a/docs/basics/_examples/DL-101-108-114 +++ b/docs/basics/_examples/DL-101-108-114 @@ -1,5 +1,5 @@ -$ datalad rerun c64173fe0a41bac4865c71f814bcbb84cd1c8146 +$ datalad rerun 281a9a48a1451e85c5a9284b6b908c93a987654c [INFO] == Command start (output follows) ===== [INFO] == Command exit (modification check follows) ===== add(ok): recordings/podcasts.tsv (file) diff --git a/docs/basics/_examples/DL-101-108-115 b/docs/basics/_examples/DL-101-108-115 index 2058ba2b4..86d0c84f1 100644 --- a/docs/basics/_examples/DL-101-108-115 +++ b/docs/basics/_examples/DL-101-108-115 @@ -1,17 +1,17 @@ $ git log -n 1 -commit 406b44c6474d167fc9e497cc807c415f3a810e14 +commit b5497627adc2bac4cfba3de27cfea371f1447f6c Author: Elena Piscopia -Date: Mon Nov 18 10:52:33 2019 +0100 +Date: Wed Dec 4 16:21:43 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles === Do not change lines below === { "chain": [ - "c64173fe0a41bac4865c71f814bcbb84cd1c8146" + "281a9a48a1451e85c5a9284b6b908c93a987654c" ], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-108-120 b/docs/basics/_examples/DL-101-108-120 index 76ca906cf..6df34276a 100644 --- a/docs/basics/_examples/DL-101-108-120 +++ b/docs/basics/_examples/DL-101-108-120 @@ -1,17 +1,17 @@ $ git log -- recordings/podcasts.tsv -commit 406b44c6474d167fc9e497cc807c415f3a810e14 +commit b5497627adc2bac4cfba3de27cfea371f1447f6c Author: Elena Piscopia -Date: Mon Nov 18 10:52:33 2019 +0100 +Date: Wed Dec 4 16:21:43 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles === Do not change lines below === { "chain": [ - "c64173fe0a41bac4865c71f814bcbb84cd1c8146" + "281a9a48a1451e85c5a9284b6b908c93a987654c" ], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], @@ -20,9 +20,9 @@ Date: Mon Nov 18 10:52:33 2019 +0100 } ^^^ Do not change lines above ^^^ -commit c64173fe0a41bac4865c71f814bcbb84cd1c8146 +commit 281a9a48a1451e85c5a9284b6b908c93a987654c Author: Elena Piscopia -Date: Mon Nov 18 10:52:30 2019 +0100 +Date: Wed Dec 4 16:21:41 2019 +0100 [DATALAD RUNCMD] create a list of podcast titles @@ -30,7 +30,7 @@ Date: Mon Nov 18 10:52:30 2019 +0100 { "chain": [], "cmd": "bash code/list_titles.sh > recordings/podcasts.tsv", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [], diff --git a/docs/basics/_examples/DL-101-112-109 b/docs/basics/_examples/DL-101-112-109 index 012c58354..b02d8922e 100644 --- a/docs/basics/_examples/DL-101-112-109 +++ b/docs/basics/_examples/DL-101-112-109 @@ -1,7 +1,7 @@ $ git log -p -n 2 -commit f609f69378303bfda1e6614d573ae59226dee8d5 +commit 199587fa10a7ea54885d3139b4d0ef042076da0d Author: Elena Piscopia -Date: Mon Nov 18 10:52:43 2019 +0100 +Date: Wed Dec 4 16:21:52 2019 +0100 add note on clean datasets @@ -21,9 +21,9 @@ index d1ba8fc..8f0124f 100644 +to the files listed with --output flags. + -commit 009c66d0f2b2a8890674e5e455c43bb9232c9c48 +commit dd726a625ba18ac0a25b744adf5feab33d9d6866 Author: Elena Piscopia -Date: Mon Nov 18 10:52:42 2019 +0100 +Date: Wed Dec 4 16:21:51 2019 +0100 [DATALAD RUNCMD] Resize logo for slides @@ -31,7 +31,7 @@ Date: Mon Nov 18 10:52:42 2019 +0100 { "chain": [], "cmd": "convert -resize 400x400 recordings/longnow/.datalad/feed_metadata/logo_salt.jpg recordings/salt_logo_small.jpg", - "dsid": "c4755e2a-09e8-11ea-98f1-e86a64c8054c", + "dsid": "b47220ee-16a9-11ea-b652-d0c637c523bc", "exit": 0, "extra_inputs": [], "inputs": [ diff --git a/docs/basics/_examples/DL-101-114-101 b/docs/basics/_examples/DL-101-114-101 index 9dcbdd9f3..20e7d1e4d 100644 --- a/docs/basics/_examples/DL-101-114-101 +++ b/docs/basics/_examples/DL-101-114-101 @@ -1,22 +1,22 @@ $ git log --reverse --oneline -715acb0 [DATALAD] new dataset -758f0af Instruct annex to add text files to Git -464b1e5 add books on Python and Unix to read later -8d6a681 add reference book about git -d792dfb Add notes on datalad create -84ce7c5 add note on datalad save -d4b91a5 [DATALAD] Recorded changes -fd5cbf6 Add note on datalad install -9d578af Add simple script to write a list of podcast speakers and titles -c64173f [DATALAD RUNCMD] create a list of podcast titles -854a655 BF: list both directories content -406b44c [DATALAD RUNCMD] create a list of podcast titles -e6e14a7 add note datalad and git diff -15e7810 add note on basic datalad run and datalad rerun -771909d [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -dc84ec4 resized picture by hand -054b821 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -3b21c7d add additional notes on run options -5e96512 [DATALAD RUNCMD] Resize logo for slides -009c66d [DATALAD RUNCMD] Resize logo for slides -f609f69 add note on clean datasets +623d3f6 [DATALAD] new dataset +6c2e485 Instruct annex to add text files to Git +c2dcdaa add books on Python and Unix to read later +3d3987b add reference book about git +2422714 Add notes on datalad create +c56f7cb add note on datalad save +ab0b11f [DATALAD] Recorded changes +8db7558 Add note on datalad install +61f3a4d Add short script to write a list of podcast speakers and titles +281a9a4 [DATALAD RUNCMD] create a list of podcast titles +1678700 BF: list both directories content +b549762 [DATALAD RUNCMD] create a list of podcast titles +09c0513 add note datalad and git diff +aadf74e add note on basic datalad run and datalad rerun +86be52e [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +bd42809 resized picture by hand +3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +b08d657 add additional notes on run options +e4dd212 [DATALAD RUNCMD] Resize logo for slides +dd726a6 [DATALAD RUNCMD] Resize logo for slides +199587f add note on clean datasets diff --git a/docs/basics/_examples/DL-101-115-103 b/docs/basics/_examples/DL-101-115-103 index 951b878a6..e2d8f8090 100644 --- a/docs/basics/_examples/DL-101-115-103 +++ b/docs/basics/_examples/DL-101-115-103 @@ -1,7 +1,7 @@ $ ls -lah total 20K -drwxr-xr-x 2 adina adina 4.0K Nov 18 10:51 . -drwxr-xr-x 7 adina adina 4.0K Nov 18 10:52 .. -lrwxrwxrwx 1 adina adina 131 Nov 18 10:50 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf +drwxr-xr-x 2 adina adina 4.0K Dec 4 16:21 . +drwxr-xr-x 7 adina adina 4.0K Dec 4 16:21 .. +lrwxrwxrwx 1 adina adina 131 Dec 4 16:21 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf lrwxrwxrwx 1 adina adina 133 Jun 29 17:39 progit.pdf -> ../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf lrwxrwxrwx 1 adina adina 131 Jan 28 2019 TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-116-105 b/docs/basics/_examples/DL-101-116-105 index cbbf57f11..d4ebc8d2d 100644 --- a/docs/basics/_examples/DL-101-116-105 +++ b/docs/basics/_examples/DL-101-116-105 @@ -1,4 +1,4 @@ $ git annex whereis books/TLCL.pdf whereis books/TLCL.pdf (1 copy) - a5e9c638-cc6b-4c27-99a5-14802c8a6898 -- course on DataLad-101 on my private Laptop [origin] + fdbcfeff-5296-4e58-84dc-913343e50f4d -- course on DataLad-101 on my private Laptop [origin] ok diff --git a/docs/basics/_examples/DL-101-117-101 b/docs/basics/_examples/DL-101-117-101 index 2221cf29c..bc2be8a2e 100644 --- a/docs/basics/_examples/DL-101-117-101 +++ b/docs/basics/_examples/DL-101-117-101 @@ -6,7 +6,7 @@ $ cd recordings/longnow $ git annex whereis Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3 whereis Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3 (3 copies) 00000000-0000-0000-0000-000000000001 -- web - 1f1f78d5-c98d-4435-9f81-8bad94145ed5 -- me@muninn:~/dl-101/DataLad-101/recordings/longnow [origin] + 740557bc-bafe-41ed-b4ae-09cd5e5cbb03 -- me@muninn:~/dl-101/DataLad-101/recordings/longnow [origin] da3bf937-5bd2-43ea-a07b-bcbe71f3b875 -- mih@medusa:/tmp/seminars-on-longterm-thinking web: http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 diff --git a/docs/basics/_examples/DL-101-118-102 b/docs/basics/_examples/DL-101-118-102 index e660877a2..98954d804 100644 --- a/docs/basics/_examples/DL-101-118-102 +++ b/docs/basics/_examples/DL-101-118-102 @@ -1,23 +1,23 @@ # lets view the history $ git log --oneline -f609f69 add note on clean datasets -009c66d [DATALAD RUNCMD] Resize logo for slides -5e96512 [DATALAD RUNCMD] Resize logo for slides -3b21c7d add additional notes on run options -054b821 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -dc84ec4 resized picture by hand -771909d [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -15e7810 add note on basic datalad run and datalad rerun -e6e14a7 add note datalad and git diff -406b44c [DATALAD RUNCMD] create a list of podcast titles -854a655 BF: list both directories content -c64173f [DATALAD RUNCMD] create a list of podcast titles -9d578af Add simple script to write a list of podcast speakers and titles -fd5cbf6 Add note on datalad install -d4b91a5 [DATALAD] Recorded changes -84ce7c5 add note on datalad save -d792dfb Add notes on datalad create -8d6a681 add reference book about git -464b1e5 add books on Python and Unix to read later -758f0af Instruct annex to add text files to Git -715acb0 [DATALAD] new dataset +199587f add note on clean datasets +dd726a6 [DATALAD RUNCMD] Resize logo for slides +e4dd212 [DATALAD RUNCMD] Resize logo for slides +b08d657 add additional notes on run options +3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +bd42809 resized picture by hand +86be52e [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +aadf74e add note on basic datalad run and datalad rerun +09c0513 add note datalad and git diff +b549762 [DATALAD RUNCMD] create a list of podcast titles +1678700 BF: list both directories content +281a9a4 [DATALAD RUNCMD] create a list of podcast titles +61f3a4d Add short script to write a list of podcast speakers and titles +8db7558 Add note on datalad install +ab0b11f [DATALAD] Recorded changes +c56f7cb add note on datalad save +2422714 Add notes on datalad create +3d3987b add reference book about git +c2dcdaa add books on Python and Unix to read later +6c2e485 Instruct annex to add text files to Git +623d3f6 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-118-103 b/docs/basics/_examples/DL-101-118-103 index fb296346f..a7f26b112 100644 --- a/docs/basics/_examples/DL-101-118-103 +++ b/docs/basics/_examples/DL-101-118-103 @@ -1,5 +1,5 @@ -$ datalad rerun 009c66d0f2b2a8890674e5e455c43bb9232c9c48 +$ datalad rerun dd726a625ba18ac0a25b744adf5feab33d9d6866 [INFO] Making sure inputs are available (this may take some time) [WARNING] no content present; cannot unlock [unlock(/home/me/dl-101/mock_user/DataLad-101/recordings/salt_logo_small.jpg)] [INFO] == Command start (output follows) ===== diff --git a/docs/basics/_examples/DL-101-121-103 b/docs/basics/_examples/DL-101-121-103 index 71f58d945..c591e7270 100644 --- a/docs/basics/_examples/DL-101-121-103 +++ b/docs/basics/_examples/DL-101-121-103 @@ -1,8 +1,7 @@ $ git log -n 1 -p - -commit 834615c670935bb5cb682ef301c6167d6786f02f +commit aca517764b330a994c8523d74dd5320a7e54e3e8 Author: Elena Piscopia -Date: Mon Nov 18 10:53:01 2019 +0100 +Date: Wed Dec 4 16:22:01 2019 +0100 Include nesting demo from datalad website diff --git a/docs/basics/_examples/DL-101-121-112 b/docs/basics/_examples/DL-101-121-112 index f2d6047ae..4021bab28 100644 --- a/docs/basics/_examples/DL-101-121-112 +++ b/docs/basics/_examples/DL-101-121-112 @@ -1,28 +1,27 @@ $ git log --oneline -db8aa3a Merge remote-tracking branch 'refs/remotes/roommate/master' -834615c Include nesting demo from datalad website -0e85c3e add note about datalad update -2e7a3ad add note on git annex whereis -73f44c4 add note about installing from paths and recursive installations -dd6fef5 add note on clean datasets -d9d0872 [DATALAD RUNCMD] Resize logo for slides -36a9cfd [DATALAD RUNCMD] Resize logo for slides -244c234 add additional notes on run options -e3277e9 [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -257130b resized picture by hand -6e624f5 [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -327fd83 add note on basic datalad run and datalad rerun -4e3fb6e add note datalad and git diff -d26a29b [DATALAD RUNCMD] create a list of podcast titles -d04c4af BF: list both directories content -ecde4e4 [DATALAD RUNCMD] create a list of podcast titles -ef8b726 Add simple script to write a list of podcast speakers and titles -e61c229 Add note on datalad install -49e85d1 [DATALAD] Recorded changes -5f98f2b add note on datalad save -b72b237 Add notes on datalad create -dac2fa7 add reference book about git -9a6fb63 add books on Python and Unix to read later -1a45fd4 Instruct annex to add text files to Git -e85b950 [DATALAD] new dataset - +c15ea3e Merge remote-tracking branch 'refs/remotes/roommate/master' +aca5177 Include nesting demo from datalad website +1291140 add note about datalad update +1b07d87 add note on git annex whereis +ad17112 add note about installing from paths and recursive installations +199587f add note on clean datasets +dd726a6 [DATALAD RUNCMD] Resize logo for slides +e4dd212 [DATALAD RUNCMD] Resize logo for slides +b08d657 add additional notes on run options +3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +bd42809 resized picture by hand +86be52e [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +aadf74e add note on basic datalad run and datalad rerun +09c0513 add note datalad and git diff +b549762 [DATALAD RUNCMD] create a list of podcast titles +1678700 BF: list both directories content +281a9a4 [DATALAD RUNCMD] create a list of podcast titles +61f3a4d Add short script to write a list of podcast speakers and titles +8db7558 Add note on datalad install +ab0b11f [DATALAD] Recorded changes +c56f7cb add note on datalad save +2422714 Add notes on datalad create +3d3987b add reference book about git +c2dcdaa add books on Python and Unix to read later +6c2e485 Instruct annex to add text files to Git +623d3f6 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-122-101 b/docs/basics/_examples/DL-101-122-101 index cfa33df00..e01bc1bb9 100644 --- a/docs/basics/_examples/DL-101-122-101 +++ b/docs/basics/_examples/DL-101-122-101 @@ -5,7 +5,7 @@ $ cat .git/config bare = false logallrefupdates = true [annex] - uuid = a5e9c638-cc6b-4c27-99a5-14802c8a6898 + uuid = fdbcfeff-5296-4e58-84dc-913343e50f4d version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -14,5 +14,5 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 838f9159-3b3d-4dd7-86b0-7dd7e51b05ed + annex-uuid = 0cfd3714-d3a7-45ab-8e70-2f741f23e8ed annex-ignore = false diff --git a/docs/basics/_examples/DL-101-122-103 b/docs/basics/_examples/DL-101-122-103 index 366348874..c68b5abfc 100644 --- a/docs/basics/_examples/DL-101-122-103 +++ b/docs/basics/_examples/DL-101-122-103 @@ -6,7 +6,7 @@ $ cat .git/config logallrefupdates = true editor = nano [annex] - uuid = a5e9c638-cc6b-4c27-99a5-14802c8a6898 + uuid = fdbcfeff-5296-4e58-84dc-913343e50f4d version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -15,5 +15,5 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 838f9159-3b3d-4dd7-86b0-7dd7e51b05ed + annex-uuid = 0cfd3714-d3a7-45ab-8e70-2f741f23e8ed annex-ignore = false diff --git a/docs/basics/_examples/DL-101-123-105 b/docs/basics/_examples/DL-101-123-105 index 59b332385..c9a3c5747 100644 --- a/docs/basics/_examples/DL-101-123-105 +++ b/docs/basics/_examples/DL-101-123-105 @@ -1,3 +1,3 @@ $ cat .datalad/config [datalad "dataset"] - id = 7256f1ae-000f-11ea-97b1-d0c637c523bc + id = b47220ee-16a9-11ea-b652-d0c637c523bc diff --git a/docs/basics/_examples/DL-101-124-101 b/docs/basics/_examples/DL-101-124-101 index e0e3417d0..a05291e15 100644 --- a/docs/basics/_examples/DL-101-124-101 +++ b/docs/basics/_examples/DL-101-124-101 @@ -1,4 +1,5 @@ $ datalad run-procedure --discover +cfg_hirni (../../../adina/env/handbook/lib/python3.7/site-packages/datalad_hirni/resources/procedures/cfg_hirni.py) [python_script] cfg_bids (../../../adina/repos/datalad-neuroimaging/datalad_neuroimaging/resources/procedures/cfg_bids.py) [python_script] cfg_metadatatypes (../../../adina/repos/datalad/datalad/resources/procedures/cfg_metadatatypes.py) [python_script] cfg_text2git (../../../adina/repos/datalad/datalad/resources/procedures/cfg_text2git.py) [python_script] diff --git a/docs/basics/_examples/DL-101-125-106 b/docs/basics/_examples/DL-101-125-106 deleted file mode 100644 index d2ef1c299..000000000 --- a/docs/basics/_examples/DL-101-125-106 +++ /dev/null @@ -1,3 +0,0 @@ -$ cat .datalad/config -[datalad "dataset"] - id = c4755e2a-09e8-11ea-98f1-e86a64c8054c diff --git a/docs/basics/_examples/DL-101-130-106 b/docs/basics/_examples/DL-101-130-106 index cb2b9fbee..8813ad4d1 100644 --- a/docs/basics/_examples/DL-101-130-106 +++ b/docs/basics/_examples/DL-101-130-106 @@ -1,5 +1,6 @@ $ cd ../ $ tree -d +$ cd midterm_project . ├── books ├── code diff --git a/docs/basics/_examples/DL-101-130-107 b/docs/basics/_examples/DL-101-130-107 index b054945a4..e1813e993 100644 --- a/docs/basics/_examples/DL-101-130-107 +++ b/docs/basics/_examples/DL-101-130-107 @@ -9,7 +9,7 @@ from sklearn.metrics import classification_report data = "input/iris.csv" -# make sure that the data is obtained (get will also install linked sub-ds!): +# make sure that the data are obtained (get will also install linked sub-ds!): dl.get(data) # prepare the data as a pandas dataframe diff --git a/docs/basics/_examples/DL-101-130-110 b/docs/basics/_examples/DL-101-130-110 index 73f5f3199..020da785e 100644 --- a/docs/basics/_examples/DL-101-130-110 +++ b/docs/basics/_examples/DL-101-130-110 @@ -1,13 +1,13 @@ $ git show ready4analysis -commit 789e6b445eac4f2245a05420289bff3e1580e9f7 +commit b0d252c3d692f42687b0fa9635f9cd188e4ecbb2 Author: Elena Piscopia -Date: Mon Nov 18 16:13:18 2019 +0100 +Date: Wed Dec 4 16:22:11 2019 +0100 add script for kNN classification and plotting diff --git a/code/script.py b/code/script.py new file mode 100644 -index 0000000..d82606c +index 0000000..e43d58e --- /dev/null +++ b/code/script.py @@ -0,0 +1,41 @@ @@ -21,7 +21,7 @@ index 0000000..d82606c + +data = "input/iris.csv" + -+# make sure that the data is obtained (get will also install linked sub-ds!): ++# make sure that the data are obtained (get will also install linked sub-ds!): +dl.get(data) + +# prepare the data as a pandas dataframe diff --git a/docs/basics/_examples/DL-101-130-112 b/docs/basics/_examples/DL-101-130-112 index 42a513bcd..9951d9398 100644 --- a/docs/basics/_examples/DL-101-130-112 +++ b/docs/basics/_examples/DL-101-130-112 @@ -1,6 +1,6 @@ $ git log --oneline -c73efce [DATALAD RUNCMD] analyze iris data with classification analysis -789e6b4 add script for kNN classification and plotting -9466bd3 [DATALAD] Recorded changes -ab62c74 Apply YODA dataset setup -8d3c887 [DATALAD] new dataset +a65e1c3 [DATALAD RUNCMD] analyze iris data with classification analysis +b0d252c add script for kNN classification and plotting +a4bc6dd [DATALAD] Recorded changes +b57de50 Apply YODA dataset setup +f770195 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-130-116 b/docs/basics/_examples/DL-101-130-116 index ca2423833..56f915583 100644 --- a/docs/basics/_examples/DL-101-130-116 +++ b/docs/basics/_examples/DL-101-130-116 @@ -1 +1,2 @@ -$ datalad create-sibling-github -d . -r midtermproject + +.: github(-) [https://github.com/adswa/midtermproject.git (git)] diff --git a/docs/basics/_examples/DL-101-132-101 b/docs/basics/_examples/DL-101-132-101 index 3c204d45f..f4abc9582 100644 --- a/docs/basics/_examples/DL-101-132-101 +++ b/docs/basics/_examples/DL-101-132-101 @@ -1,7 +1,7 @@ $ git log --oneline -e2b9975 Provide project description -a44265a [DATALAD RUNCMD] analyze iris data with classification analysis -6f7ea3e add script for kNN classification and plotting -aeed9e0 [DATALAD] Recorded changes -855f0b7 Apply YODA dataset setup -63e1f68 [DATALAD] new dataset +b217c8b Provide project description +a65e1c3 [DATALAD RUNCMD] analyze iris data with classification analysis +b0d252c add script for kNN classification and plotting +a4bc6dd [DATALAD] Recorded changes +b57de50 Apply YODA dataset setup +f770195 [DATALAD] new dataset diff --git a/docs/basics/_examples/DL-101-132-110 b/docs/basics/_examples/DL-101-132-110 index 71279a872..0f7f472f7 100644 --- a/docs/basics/_examples/DL-101-132-110 +++ b/docs/basics/_examples/DL-101-132-110 @@ -1,4 +1,4 @@ -$ datalad save -m "finished my midterm project" +$ datalad save -d . -m "finished my midterm project!" midterm_project add(ok): midterm_project (file) save(ok): . (dataset) action summary: diff --git a/docs/basics/_examples/DL-101-132-112 b/docs/basics/_examples/DL-101-132-112 index 8a889506e..79ae077ec 100644 --- a/docs/basics/_examples/DL-101-132-112 +++ b/docs/basics/_examples/DL-101-132-112 @@ -1,14 +1,14 @@ $ git log -p -n 1 -commit c260b371c1992bd2b9c8758c0e88ded01ef62d84 +commit c4aaeb2476206ef8e5dc6346bae179eb6875139a Author: Elena Piscopia -Date: Fri Nov 29 09:52:55 2019 +0100 +Date: Wed Dec 4 16:22:34 2019 +0100 - finished my midterm project + finished my midterm project! diff --git a/midterm_project b/midterm_project -index 855f0b7..e2b9975 160000 +index b57de50..b217c8b 160000 --- a/midterm_project +++ b/midterm_project @@ -1 +1 @@ --Subproject commit 855f0b757d8ce85f46eacbb24e77a0df3ba30ee0 -+Subproject commit e2b9975103bfcbe7ecec853b1d6771d8d19de8ae +-Subproject commit b57de50bb54b60d348e0e7af1aab3a01c75f50a3 ++Subproject commit b217c8beaa36c19459cbc4ca5b7abbdb4623b61b diff --git a/docs/basics/_examples/DL-101-133-101 b/docs/basics/_examples/DL-101-133-101 new file mode 100644 index 000000000..2504453f3 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-101 @@ -0,0 +1,9 @@ +# we are in the midterm_project subdataset +$ datalad containers-add midterm-software --url shub://adswa/resources:1 +add(ok): .datalad/config (file) +save(ok): . (dataset) +containers_add(ok): /home/me/dl-101/DataLad-101/midterm_project/.datalad/environments/midterm-software/image (file) +action summary: + add (ok: 1) + containers_add (ok: 1) + save (ok: 1) diff --git a/docs/basics/_examples/DL-101-133-102 b/docs/basics/_examples/DL-101-133-102 new file mode 100644 index 000000000..7a546acf9 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-102 @@ -0,0 +1,7 @@ +$ cat .datalad/config +[datalad "dataset"] + id = d62da69a-16a9-11ea-b652-d0c637c523bc +[datalad "containers.midterm-software"] + updateurl = shub://adswa/resources:1 + image = .datalad/environments/midterm-software/image + cmdexec = singularity exec {img} {cmd} diff --git a/docs/basics/_examples/DL-101-133-103 b/docs/basics/_examples/DL-101-133-103 new file mode 100644 index 000000000..4c226e915 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-103 @@ -0,0 +1,26 @@ +$ git log -n 1 -p +commit 2344fa6676afcc92b47c7a03bbd4090b7ddabe27 +Author: Elena Piscopia +Date: Wed Dec 4 16:22:52 2019 +0100 + + [DATALAD] Configure containerized environment 'midterm-software' + +diff --git a/.datalad/config b/.datalad/config +index 61c5ac2..651422a 100644 +--- a/.datalad/config ++++ b/.datalad/config +@@ -1,2 +1,6 @@ + [datalad "dataset"] + id = d62da69a-16a9-11ea-b652-d0c637c523bc ++[datalad "containers.midterm-software"] ++ updateurl = shub://adswa/resources:1 ++ image = .datalad/environments/midterm-software/image ++ cmdexec = singularity exec {img} {cmd} +diff --git a/.datalad/environments/midterm-software/image b/.datalad/environments/midterm-software/image +new file mode 120000 +index 0000000..800282a +--- /dev/null ++++ b/.datalad/environments/midterm-software/image +@@ -0,0 +1 @@ ++../../../.git/annex/objects/zJ/8f/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10/MD5E-s232214559--49dcb6ac1a5787636c9897c4d4df7e10 +\ No newline at end of file diff --git a/docs/basics/_examples/DL-101-133-104 b/docs/basics/_examples/DL-101-133-104 new file mode 100644 index 000000000..847fad385 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-104 @@ -0,0 +1,19 @@ +$ datalad containers-run -m "rerun analysis in container" \ + --container-name midterm-software \ + --input "input/iris.csv" \ + --output "prediction_report.csv" \ + --output "pairwise_relationships.png" \ + "python3 code/script.py" +[INFO] Making sure inputs are available (this may take some time) +[INFO] == Command start (output follows) ===== +[INFO] == Command exit (modification check follows) ===== +unlock(ok): pairwise_relationships.png (file) +unlock(ok): prediction_report.csv (file) +add(ok): pairwise_relationships.png (file) +add(ok): prediction_report.csv (file) +save(ok): . (dataset) +action summary: + add (ok: 2) + get (notneeded: 2) + save (notneeded: 1, ok: 1) + unlock (ok: 2) diff --git a/docs/basics/_examples/DL-101-133-110 b/docs/basics/_examples/DL-101-133-110 new file mode 100644 index 000000000..15b6b53f9 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-110 @@ -0,0 +1,2 @@ +$ datalad containers-list +midterm-software -> .datalad/environments/midterm-software/image diff --git a/docs/basics/_examples/DL-101-133-111 b/docs/basics/_examples/DL-101-133-111 new file mode 100644 index 000000000..6566c7a72 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-111 @@ -0,0 +1,45 @@ +$ git log -p -n 1 +commit 326c92512ab422651717a3869ef9d01e61b5cb61 +Author: Elena Piscopia +Date: Wed Dec 4 16:22:57 2019 +0100 + + [DATALAD RUNCMD] rerun analysis in container + + === Do not change lines below === + { + "chain": [], + "cmd": "singularity exec .datalad/environments/midterm-software/image python3 code/script.py", + "dsid": "d62da69a-16a9-11ea-b652-d0c637c523bc", + "exit": 0, + "extra_inputs": [ + ".datalad/environments/midterm-software/image" + ], + "inputs": [ + "input/iris.csv" + ], + "outputs": [ + "prediction_report.csv", + "pairwise_relationships.png" + ], + "pwd": "." + } + ^^^ Do not change lines above ^^^ + +diff --git a/pairwise_relationships.png b/pairwise_relationships.png +index 2f69f64..6d00014 120000 +--- a/pairwise_relationships.png ++++ b/pairwise_relationships.png +@@ -1 +1 @@ +-.git/annex/objects/Pz/Xm/MD5E-s175662--8a9a3e225267f327e1671cf6d01b1957.png/MD5E-s175662--8a9a3e225267f327e1671cf6d01b1957.png +\ No newline at end of file ++.git/annex/objects/z3/23/MD5E-s176597--87d8a72f5f7b1f4f191d0be1bfd15288.png/MD5E-s176597--87d8a72f5f7b1f4f191d0be1bfd15288.png +\ No newline at end of file +diff --git a/prediction_report.csv b/prediction_report.csv +index 42d194b..b46a2d5 120000 +--- a/prediction_report.csv ++++ b/prediction_report.csv +@@ -1 +1 @@ +-.git/annex/objects/8q/6M/MD5E-s345--a88cab39b1a5ec59ace322225cc88bc9.csv/MD5E-s345--a88cab39b1a5ec59ace322225cc88bc9.csv +\ No newline at end of file ++.git/annex/objects/VF/27/MD5E-s347--7d984f53676358222aa7aa55980f205b.csv/MD5E-s347--7d984f53676358222aa7aa55980f205b.csv +\ No newline at end of file diff --git a/docs/basics/_examples/DL-101-133-112 b/docs/basics/_examples/DL-101-133-112 new file mode 100644 index 000000000..875f3cfa2 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-112 @@ -0,0 +1,3 @@ +$ cd ../ +$ datalad status + modified: midterm_project (dataset) diff --git a/docs/basics/_examples/DL-101-133-113 b/docs/basics/_examples/DL-101-133-113 new file mode 100644 index 000000000..8bb646ba9 --- /dev/null +++ b/docs/basics/_examples/DL-101-133-113 @@ -0,0 +1,6 @@ +$ datalad save -d . -m "add container and execute analysis within container" midterm_project +add(ok): midterm_project (file) +save(ok): . (dataset) +action summary: + add (ok: 1) + save (ok: 1) diff --git a/docs/basics/_examples/DL-101-135-101 b/docs/basics/_examples/DL-101-135-101 index b5530c61c..00bf3e7ff 100644 --- a/docs/basics/_examples/DL-101-135-101 +++ b/docs/basics/_examples/DL-101-135-101 @@ -1,2 +1,2 @@ $ datalad --version -datalad 0.12.0rc6.dev91 +datalad 0.12.0rc6.dev97 diff --git a/docs/basics/_examples/DL-101-135-102 b/docs/basics/_examples/DL-101-135-102 index 27e99fc0d..893672be3 100644 --- a/docs/basics/_examples/DL-101-135-102 +++ b/docs/basics/_examples/DL-101-135-102 @@ -2,8 +2,8 @@ $ datalad wtf # WTF ## configuration ## datalad - - version: 0.12.0rc6.dev91 - - full_version: 0.12.0rc6.dev91-gde46 + - version: 0.12.0rc6.dev97 + - full_version: 0.12.0rc6.dev97-g9250-dirty ## dataset - path: /home/me/dl-101/DataLad-101 - repo: AnnexRepo @@ -11,12 +11,13 @@ $ datalad wtf ## dependencies - tqdm: 4.32.2 - cmd:annex: 7.20190129 - - cmd:git: 2.20.1 - - cmd:bundled-git: 2.20.1 - - cmd:system-git: 2.20.1 + - cmd:git: 2.24.0 + - cmd:bundled-git: 2.24.0 + - cmd:system-git: 2.24.0 - cmd:system-ssh: 7.9p1 - appdirs: 1.4.3 - boto: 2.49.0 + - exifread: 2.1.2 - git: 2.1.11 - gitdb: 2.0.5 - humanize: 0.5.1 @@ -24,6 +25,7 @@ $ datalad wtf - keyring: 19.0.2 - keyrings.alt: 3.1.1 - msgpack: 0.6.1 + - mutagen: 1.43.0 - requests: 2.22.0 - scrapy: 1.7.4 - wrapt: 1.11.2 @@ -37,6 +39,53 @@ $ datalad wtf - PATH: /home/adina/env/handbook/bin:/home/adina/Documents/freesurfer/bin:/home/adina/Documents/freesurfer/fsfast/bin:/home/adina/Documents/freesurfer/tktools:/usr/share/fsl/5.0/bin:/usr/lib/fsl/5.0:/home/adina/Documents/freesurfer/mni/bin:/usr/share/fsl/5.0/5.0/bin:/home/adina/.local/bin:/home/adina/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/adina/.dotfiles/bin - GIT_PYTHON_GIT_EXECUTABLE: /usr/bin/git ## extensions + - webapp: + - load_error: None + - description: Generic web app support + - module: datalad_webapp + - version: 0.2 + - entrypoints: + - datalad_webapp.WebApp: + - module: datalad_webapp + - class: WebApp + - names: + - webapp + - webapp + - load_error: None + - hirni: + - load_error: None + - description: HIRNI workflows + - module: datalad_hirni + - version: 0.0.4 + - entrypoints: + - datalad_hirni.commands.import_dicoms.ImportDicoms: + - module: datalad_hirni.commands.import_dicoms + - class: ImportDicoms + - names: + - hirni-import-dcm + - hirni_import_dcm + - load_error: None + - datalad_hirni.commands.spec4anything.Spec4Anything: + - module: datalad_hirni.commands.spec4anything + - class: Spec4Anything + - names: + - hirni-spec4anything + - hirni_spec4anything + - load_error: None + - datalad_hirni.commands.dicom2spec.Dicom2Spec: + - module: datalad_hirni.commands.dicom2spec + - class: Dicom2Spec + - names: + - hirni-dicom2spec + - hirni_dicom2spec + - load_error: None + - datalad_hirni.commands.spec2bids.Spec2Bids: + - module: datalad_hirni.commands.spec2bids + - class: Spec2Bids + - names: + - hirni-spec2bids + - hirni_spec2bids + - load_error: None - crawler: - load_error: None - description: Crawl web resources @@ -275,7 +324,8 @@ $ datalad wtf - load_error: None - audio: - module: datalad.metadata.extractors.audio - - load_error: No module named 'mutagen' [audio.py::17] + - version: None + - load_error: None - datacite: - module: datalad.metadata.extractors.datacite - version: None @@ -290,7 +340,8 @@ $ datalad wtf - load_error: None - exif: - module: datalad.metadata.extractors.exif - - load_error: No module named 'exifread' [exif.py::16] + - version: None + - load_error: None - frictionless_datapackage: - module: datalad.metadata.extractors.frictionless_datapackage - version: None @@ -301,7 +352,8 @@ $ datalad wtf - load_error: None - xmp: - module: datalad.metadata.extractors.xmp - - load_error: No module named 'libxmp' [xmp.py::20] + - version: None + - load_error: None ## python - version: 3.7.3 - implementation: CPython @@ -309,8 +361,8 @@ $ datalad wtf - type: posix - name: Linux - release: 4.19.0-6-amd64 - - version: #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) - - distribution: debian/10.1 + - version: #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) + - distribution: debian/10.2 - max_path_length: 283 - encoding: - default: utf-8 diff --git a/docs/basics/_examples/DL-101-136-101 b/docs/basics/_examples/DL-101-136-101 index 429ad061d..80867a469 100644 --- a/docs/basics/_examples/DL-101-136-101 +++ b/docs/basics/_examples/DL-101-136-101 @@ -2,8 +2,8 @@ $ cd books/ $ mv TLCL.pdf The_Linux_Command_Line.pdf $ ls -lah total 20K -drwxr-xr-x 2 adina adina 4.0K Nov 18 10:53 . -drwxr-xr-x 7 adina adina 4.0K Nov 18 10:53 .. -lrwxrwxrwx 1 adina adina 131 Nov 18 10:50 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf +drwxr-xr-x 2 adina adina 4.0K Dec 4 16:23 . +drwxr-xr-x 8 adina adina 4.0K Dec 4 16:23 .. +lrwxrwxrwx 1 adina adina 131 Dec 4 16:21 byte-of-python.pdf -> ../.git/annex/objects/ZZ/f1/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf/MD5E-s4407669--32e6b03a08a6edda12ad42eb7bb06a5c.pdf lrwxrwxrwx 1 adina adina 133 Jun 29 17:39 progit.pdf -> ../.git/annex/objects/G6/Gj/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf/MD5E-s12465653--05cd7ed561d108c9bcf96022bc78a92c.pdf lrwxrwxrwx 1 adina adina 131 Jan 28 2019 The_Linux_Command_Line.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-104 b/docs/basics/_examples/DL-101-136-104 index b11a97724..083f793b0 100644 --- a/docs/basics/_examples/DL-101-136-104 +++ b/docs/basics/_examples/DL-101-136-104 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit 4b793ed196fc072389963aa5d2d87e585a88e034 +commit d33b1e63e51e98e2c891205e39377d7fec615828 Author: Elena Piscopia -Date: Mon Nov 18 10:53:09 2019 +0100 +Date: Wed Dec 4 16:23:02 2019 +0100 rename the book diff --git a/docs/basics/_examples/DL-101-136-105 b/docs/basics/_examples/DL-101-136-105 index 0e1e4b6f4..e97726ad1 100644 --- a/docs/basics/_examples/DL-101-136-105 +++ b/docs/basics/_examples/DL-101-136-105 @@ -1,4 +1,3 @@ $ git reset --hard HEAD~1 $ datalad status -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-108 b/docs/basics/_examples/DL-101-136-108 index 2729fc0ed..efa77dbcc 100644 --- a/docs/basics/_examples/DL-101-136-108 +++ b/docs/basics/_examples/DL-101-136-108 @@ -1,7 +1,6 @@ $ git status On branch master Changes to be committed: - (use "git reset HEAD ..." to unstage) - + (use "git restore --staged ..." to unstage) renamed: TLCL.pdf -> The_Linux_Command_Line.pdf diff --git a/docs/basics/_examples/DL-101-136-109 b/docs/basics/_examples/DL-101-136-109 index ed232aa80..d45ccab89 100644 --- a/docs/basics/_examples/DL-101-136-109 +++ b/docs/basics/_examples/DL-101-136-109 @@ -1,4 +1,4 @@ $ git commit -m "rename book" -[master 9892940] rename book +[master feb1fca] rename book 1 file changed, 0 insertions(+), 0 deletions(-) rename books/{TLCL.pdf => The_Linux_Command_Line.pdf} (100%) diff --git a/docs/basics/_examples/DL-101-136-110 b/docs/basics/_examples/DL-101-136-110 index 0e1e4b6f4..e97726ad1 100644 --- a/docs/basics/_examples/DL-101-136-110 +++ b/docs/basics/_examples/DL-101-136-110 @@ -1,4 +1,3 @@ $ git reset --hard HEAD~1 $ datalad status -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-121 b/docs/basics/_examples/DL-101-136-121 index de56dc3fa..c2eb7fb2d 100644 --- a/docs/basics/_examples/DL-101-136-121 +++ b/docs/basics/_examples/DL-101-136-121 @@ -1,3 +1,3 @@ $ cd ../ $ ls -l TLCL.pdf -lrwxrwxrwx 1 adina adina 131 Nov 18 10:53 TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 131 Dec 4 16:23 TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-122 b/docs/basics/_examples/DL-101-136-122 index 7fe3a9529..19fc4417a 100644 --- a/docs/basics/_examples/DL-101-136-122 +++ b/docs/basics/_examples/DL-101-136-122 @@ -7,4 +7,4 @@ action summary: add (ok: 1) delete (ok: 1) save (ok: 1) -lrwxrwxrwx 1 adina adina 128 Nov 18 10:53 TLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 128 Dec 4 16:23 TLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-123 b/docs/basics/_examples/DL-101-136-123 index 86b4e27f9..8b47e5350 100644 --- a/docs/basics/_examples/DL-101-136-123 +++ b/docs/basics/_examples/DL-101-136-123 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit 3189043ce3b7998446471aff4accb624898e21a0 +commit 687fb1db82c3448073944690de9e17ab89696af1 Author: Elena Piscopia -Date: Mon Nov 18 10:53:11 2019 +0100 +Date: Wed Dec 4 16:23:03 2019 +0100 moved book into root diff --git a/docs/basics/_examples/DL-101-136-124 b/docs/basics/_examples/DL-101-136-124 index ee40b070e..2df35b84f 100644 --- a/docs/basics/_examples/DL-101-136-124 +++ b/docs/basics/_examples/DL-101-136-124 @@ -1,3 +1,2 @@ $ git reset --hard HEAD~1 -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-132 b/docs/basics/_examples/DL-101-136-132 index 1a8b345f8..3ba831961 100644 --- a/docs/basics/_examples/DL-101-136-132 +++ b/docs/basics/_examples/DL-101-136-132 @@ -1,7 +1,7 @@ $ git log -n 1 -p -commit 1246e8c40678edf85874b5f72940500220d3a148 +commit 8759da9a782ba607e8d9c8137176104fa988296e Author: Elena Piscopia -Date: Mon Nov 18 10:53:11 2019 +0100 +Date: Wed Dec 4 16:23:04 2019 +0100 add copy of TLCL.pdf diff --git a/docs/basics/_examples/DL-101-136-133 b/docs/basics/_examples/DL-101-136-133 index 70eeac6b7..470f1f444 100644 --- a/docs/basics/_examples/DL-101-136-133 +++ b/docs/basics/_examples/DL-101-136-133 @@ -1,4 +1,4 @@ $ ls -l copyofTLCL.pdf $ ls -l books/TLCL.pdf -lrwxrwxrwx 1 adina adina 128 Nov 18 10:53 copyofTLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf -lrwxrwxrwx 1 adina adina 131 Nov 18 10:53 books/TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 128 Dec 4 16:23 copyofTLCL.pdf -> .git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf +lrwxrwxrwx 1 adina adina 131 Dec 4 16:23 books/TLCL.pdf -> ../.git/annex/objects/jf/3M/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf/MD5E-s2120211--06d1efcb05bb2c55cd039dab3fb28455.pdf diff --git a/docs/basics/_examples/DL-101-136-134 b/docs/basics/_examples/DL-101-136-134 index 29615e42a..2df35b84f 100644 --- a/docs/basics/_examples/DL-101-136-134 +++ b/docs/basics/_examples/DL-101-136-134 @@ -1,2 +1,2 @@ $ git reset --hard HEAD~1 -HEAD is now at 587ac34 finished my midterm project! +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-140 b/docs/basics/_examples/DL-101-136-140 index 97754e5cc..e4c364f2f 100644 --- a/docs/basics/_examples/DL-101-136-140 +++ b/docs/basics/_examples/DL-101-136-140 @@ -6,7 +6,7 @@ $ cat .git/config logallrefupdates = true editor = nano [annex] - uuid = a5e9c638-cc6b-4c27-99a5-14802c8a6898 + uuid = fdbcfeff-5296-4e58-84dc-913343e50f4d version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -15,7 +15,7 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 050f8bfe-1ee0-4742-bcb4-b103347aaa05 + annex-uuid = 0cfd3714-d3a7-45ab-8e70-2f741f23e8ed annex-ignore = false [submodule "midterm_project"] url = /home/me/dl-101/DataLad-101/midterm_project diff --git a/docs/basics/_examples/DL-101-136-144 b/docs/basics/_examples/DL-101-136-144 index 3a91c779c..beb798feb 100644 --- a/docs/basics/_examples/DL-101-136-144 +++ b/docs/basics/_examples/DL-101-136-144 @@ -6,7 +6,7 @@ $ cat .git/config logallrefupdates = true editor = nano [annex] - uuid = a5e9c638-cc6b-4c27-99a5-14802c8a6898 + uuid = fdbcfeff-5296-4e58-84dc-913343e50f4d version = 5 backends = MD5E [submodule "recordings/longnow"] @@ -15,7 +15,7 @@ $ cat .git/config [remote "roommate"] url = ../mock_user/onemoredir/DataLad-101 fetch = +refs/heads/*:refs/remotes/roommate/* - annex-uuid = 050f8bfe-1ee0-4742-bcb4-b103347aaa05 + annex-uuid = 0cfd3714-d3a7-45ab-8e70-2f741f23e8ed annex-ignore = false [submodule "midterm_project"] url = /home/me/dl-101/DataLad-101/midterm_project diff --git a/docs/basics/_examples/DL-101-136-146 b/docs/basics/_examples/DL-101-136-146 index 152c158bd..dc93147a5 100644 --- a/docs/basics/_examples/DL-101-136-146 +++ b/docs/basics/_examples/DL-101-136-146 @@ -1,5 +1,4 @@ $ cd ../mock_user && mv onemoredir/DataLad-101 . $ rm -r onemoredir $ cd ../DataLad-101 && git reset --hard master -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-152 b/docs/basics/_examples/DL-101-136-152 index bccb24948..2df35b84f 100644 --- a/docs/basics/_examples/DL-101-136-152 +++ b/docs/basics/_examples/DL-101-136-152 @@ -1,4 +1,2 @@ $ git reset --hard HEAD~1 -<<<<<<< HEAD -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-156 b/docs/basics/_examples/DL-101-136-156 index c6f48fbce..4547934c2 100644 --- a/docs/basics/_examples/DL-101-136-156 +++ b/docs/basics/_examples/DL-101-136-156 @@ -1,3 +1,3 @@ $ git reset --hard HEAD~1 warning: unable to rmdir 'longnow': Directory not empty -HEAD is now at 587ac34 finished my midterm project! +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-164 b/docs/basics/_examples/DL-101-136-164 index d57881176..e3e402d65 100644 --- a/docs/basics/_examples/DL-101-136-164 +++ b/docs/basics/_examples/DL-101-136-164 @@ -1,3 +1,2 @@ $ git reset --hard HEAD~2 -HEAD is now at 587ac34 finished my midterm project! - +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-136-174 b/docs/basics/_examples/DL-101-136-174 index 527366d66..f85879600 100644 --- a/docs/basics/_examples/DL-101-136-174 +++ b/docs/basics/_examples/DL-101-136-174 @@ -1,6 +1,6 @@ $ git reset --hard HEAD~1 $ ls -HEAD is now at da61732 Added flower mosaic from wikimedia +HEAD is now at a88c158 Added flower mosaic from wikimedia books code flowers.jpg diff --git a/docs/basics/_examples/DL-101-136-180 b/docs/basics/_examples/DL-101-136-180 index 4c6dd0fc6..e3e402d65 100644 --- a/docs/basics/_examples/DL-101-136-180 +++ b/docs/basics/_examples/DL-101-136-180 @@ -1,2 +1,2 @@ $ git reset --hard HEAD~2 -HEAD is now at 587ac34 finished my midterm project! +HEAD is now at d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-101 b/docs/basics/_examples/DL-101-137-101 index fd51a8174..5e125069d 100644 --- a/docs/basics/_examples/DL-101-137-101 +++ b/docs/basics/_examples/DL-101-137-101 @@ -1,16 +1,16 @@ $ git log -15 --oneline -587ac34 finished my midterm project! -de7d909 [DATALAD] Recorded changes -37d74af add note on DataLads procedures -264d668 add note on configurations and git config -2277fe7 Add note on adding siblings -f015735 Merge remote-tracking branch 'refs/remotes/roommate/master' -0a01e8a Include nesting demo from datalad website -5e45b4e add note about datalad update -e613db1 add note on git annex whereis -118eed9 add note about installing from paths and recursive installations -f42379c add note on clean datasets -475d3de [DATALAD RUNCMD] Resize logo for slides -6856e4f [DATALAD RUNCMD] Resize logo for slides -72fa9f9 add additional notes on run options -b49af2f [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +d3ee8e9 add container and execute analysis within container +c4aaeb2 finished my midterm project! +ca76f3b [DATALAD] Recorded changes +73f14ee add note on DataLads procedures +9a795e3 add note on configurations and git config +da9d193 Add note on adding siblings +c15ea3e Merge remote-tracking branch 'refs/remotes/roommate/master' +aca5177 Include nesting demo from datalad website +1291140 add note about datalad update +1b07d87 add note on git annex whereis +ad17112 add note about installing from paths and recursive installations +199587f add note on clean datasets +dd726a6 [DATALAD RUNCMD] Resize logo for slides +e4dd212 [DATALAD RUNCMD] Resize logo for slides +b08d657 add additional notes on run options diff --git a/docs/basics/_examples/DL-101-137-105 b/docs/basics/_examples/DL-101-137-105 index 66160df0f..710ee2252 100644 --- a/docs/basics/_examples/DL-101-137-105 +++ b/docs/basics/_examples/DL-101-137-105 @@ -1,7 +1,7 @@ $ git log -p -1 -commit 986ede3879a4a86a93b0b4d22ccee925c15c6230 +commit a84bc8b2d463077e33803a673d762133d8bb3e2e Author: Elena Piscopia -Date: Mon Nov 18 10:53:25 2019 +0100 +Date: Wed Dec 4 16:23:17 2019 +0100 [DATALAD] Recorded changes diff --git a/docs/basics/_examples/DL-101-137-106 b/docs/basics/_examples/DL-101-137-106 index 3982396bd..c3aebf4d4 100644 --- a/docs/basics/_examples/DL-101-137-106 +++ b/docs/basics/_examples/DL-101-137-106 @@ -1,4 +1,4 @@ $ git log -n 3 --oneline -4604d99 [DATALAD] Recorded changes -587ac34 finished my midterm project! -de7d909 [DATALAD] Recorded changes +a84bc8b [DATALAD] Recorded changes +d3ee8e9 add container and execute analysis within container +c4aaeb2 finished my midterm project! diff --git a/docs/basics/_examples/DL-101-137-107 b/docs/basics/_examples/DL-101-137-107 index ab6678982..53c16e35b 100644 --- a/docs/basics/_examples/DL-101-137-107 +++ b/docs/basics/_examples/DL-101-137-107 @@ -1 +1,2 @@ -$ git reset --mixed 587ac34d63c93275649cb65cd932c6a052569fdd + +$ git reset --mixed d3ee8e9a53e8c321e97ccea5b66fa19e7d34056a diff --git a/docs/basics/_examples/DL-101-137-108 b/docs/basics/_examples/DL-101-137-108 index 051d98c91..fe10dd22b 100644 --- a/docs/basics/_examples/DL-101-137-108 +++ b/docs/basics/_examples/DL-101-137-108 @@ -1,3 +1,3 @@ $ git log -n 2 --oneline -587ac34 finished my midterm project! -de7d909 [DATALAD] Recorded changes \ No newline at end of file +d3ee8e9 add container and execute analysis within container +c4aaeb2 finished my midterm project! diff --git a/docs/basics/_examples/DL-101-137-112 b/docs/basics/_examples/DL-101-137-112 index 53e51f3e2..187b4f529 100644 --- a/docs/basics/_examples/DL-101-137-112 +++ b/docs/basics/_examples/DL-101-137-112 @@ -1,12 +1,12 @@ -$ git log -n 2 -commit 51d763ed5883bc51f6eabc36248e4ef80b9f600f +$ git log -2 +commit b8972e080e665903f10c32c3d174211c282324e1 Author: Elena Piscopia -Date: Mon Nov 18 08:44:46 2019 +0100 +Date: Wed Dec 4 16:23:17 2019 +0100 save my favorite Git joke -commit 587ac34d63c93275649cb65cd932c6a052569fdd +commit d3ee8e9a53e8c321e97ccea5b66fa19e7d34056a Author: Elena Piscopia -Date: Mon Nov 18 08:44:26 2019 +0100 +Date: Wed Dec 4 16:22:59 2019 +0100 - finished my midterm project! + add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-114 b/docs/basics/_examples/DL-101-137-114 index d3b1afd2b..03c7dc3be 100644 --- a/docs/basics/_examples/DL-101-137-114 +++ b/docs/basics/_examples/DL-101-137-114 @@ -1,2 +1,2 @@ $ ls -l apdffile.pdf -lrwxrwxrwx 1 adina adina 122 Nov 18 10:53 apdffile.pdf -> .git/annex/objects/gV/gf/MD5E-s1842--40e4e7d327aa106fe64ef8b799a29d54.pdf/MD5E-s1842--40e4e7d327aa106fe64ef8b799a29d54.pdf +lrwxrwxrwx 1 adina adina 122 Dec 4 16:23 apdffile.pdf -> .git/annex/objects/jg/Qk/MD5E-s1842--ae82655e482e95369a942ef787ffda24.pdf/MD5E-s1842--ae82655e482e95369a942ef787ffda24.pdf diff --git a/docs/basics/_examples/DL-101-137-116 b/docs/basics/_examples/DL-101-137-116 index f2dd736eb..d1a67faa3 100644 --- a/docs/basics/_examples/DL-101-137-116 +++ b/docs/basics/_examples/DL-101-137-116 @@ -1,2 +1,2 @@ $ ls -l apdffile.pdf --rw-r--r-- 1 adina adina 1842 Nov 18 10:53 apdffile.pdf +-rw-r--r-- 1 adina adina 1842 Dec 4 16:23 apdffile.pdf diff --git a/docs/basics/_examples/DL-101-137-117 b/docs/basics/_examples/DL-101-137-117 index f873dcce8..bf482fa38 100644 --- a/docs/basics/_examples/DL-101-137-117 +++ b/docs/basics/_examples/DL-101-137-117 @@ -1,4 +1,4 @@ $ git log -n 3 --oneline -b602512 [DATALAD] Recorded changes -51d763e save my favourite Git joke -587ac34 finished my midterm project! +76f65c9 [DATALAD] Recorded changes +b8972e0 save my favorite Git joke +d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-118 b/docs/basics/_examples/DL-101-137-118 index e327e1113..5e2b03128 100644 --- a/docs/basics/_examples/DL-101-137-118 +++ b/docs/basics/_examples/DL-101-137-118 @@ -1 +1,2 @@ -$ git reset --mixed 51d763ed5883bc51f6eabc36248e4ef80b9f600f + +$ git reset --mixed b8972e080e665903f10c32c3d174211c282324e1 diff --git a/docs/basics/_examples/DL-101-137-119 b/docs/basics/_examples/DL-101-137-119 index 008555927..931ee555d 100644 --- a/docs/basics/_examples/DL-101-137-119 +++ b/docs/basics/_examples/DL-101-137-119 @@ -1,3 +1,3 @@ $ git log -n 2 --oneline -51d763e save my favourite Git joke -587ac34 finished my midterm project! +b8972e0 save my favorite Git joke +d3ee8e9 add container and execute analysis within container diff --git a/docs/basics/_examples/DL-101-137-121 b/docs/basics/_examples/DL-101-137-121 index 0be81d823..f08d27a55 100644 --- a/docs/basics/_examples/DL-101-137-121 +++ b/docs/basics/_examples/DL-101-137-121 @@ -1,21 +1,21 @@ $ git log -n 20 --oneline -51d763e save my favourite Git joke -587ac34 finished my midterm project! -de7d909 [DATALAD] Recorded changes -37d74af add note on DataLads procedures -264d668 add note on configurations and git config -2277fe7 Add note on adding siblings -f015735 Merge remote-tracking branch 'refs/remotes/roommate/master' -0a01e8a Include nesting demo from datalad website -5e45b4e add note about datalad update -e613db1 add note on git annex whereis -118eed9 add note about installing from paths and recursive installations -f42379c add note on clean datasets -475d3de [DATALAD RUNCMD] Resize logo for slides -6856e4f [DATALAD RUNCMD] Resize logo for slides -72fa9f9 add additional notes on run options -b49af2f [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... -f59a783 resized picture by hand -4698deb [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... -cebdc20 add note on basic datalad run and datalad rerun -dd71a3c add note datalad and git diff +b8972e0 save my favorite Git joke +d3ee8e9 add container and execute analysis within container +c4aaeb2 finished my midterm project! +ca76f3b [DATALAD] Recorded changes +73f14ee add note on DataLads procedures +9a795e3 add note on configurations and git config +da9d193 Add note on adding siblings +c15ea3e Merge remote-tracking branch 'refs/remotes/roommate/master' +aca5177 Include nesting demo from datalad website +1291140 add note about datalad update +1b07d87 add note on git annex whereis +ad17112 add note about installing from paths and recursive installations +199587f add note on clean datasets +dd726a6 [DATALAD RUNCMD] Resize logo for slides +e4dd212 [DATALAD RUNCMD] Resize logo for slides +b08d657 add additional notes on run options +3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... +bd42809 resized picture by hand +86be52e [DATALAD RUNCMD] convert -resize 400x400 recordings/longn... +aadf74e add note on basic datalad run and datalad rerun diff --git a/docs/basics/_examples/DL-101-137-122 b/docs/basics/_examples/DL-101-137-122 index a9f08570c..1ea969f22 100644 --- a/docs/basics/_examples/DL-101-137-122 +++ b/docs/basics/_examples/DL-101-137-122 @@ -1,14 +1,21 @@ -$ git checkout f59a78356d78525239347ecf763b6677130898af + +$ git checkout 3a4418dc8d74b9889da49b26e0f201e169902113 warning: unable to rmdir 'midterm_project': Directory not empty -Note: checking out 'f59a78356d78525239347ecf763b6677130898af'. +Note: switching to '3a4418dc8d74b9889da49b26e0f201e169902113'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this -state without impacting any branches by performing another checkout. +state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may -do so (now or later) by using -b with the checkout command again. Example: +do so (now or later) by using -c with the switch command. Example: + + git switch -c + +Or undo this operation with: + + git switch - - git checkout -b +Turn off this advice by setting config variable advice.detachedHead to false -HEAD is now at f59a783 resized picture by hand +HEAD is now at 3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... diff --git a/docs/basics/_examples/DL-101-137-124 b/docs/basics/_examples/DL-101-137-124 index cc8977241..8c48e3fd4 100644 --- a/docs/basics/_examples/DL-101-137-124 +++ b/docs/basics/_examples/DL-101-137-124 @@ -1,3 +1,3 @@ $ git checkout master -Previous HEAD position was f59a783 resized picture by hand +Previous HEAD position was 3a4418d [DATALAD RUNCMD] convert -resize 450x450 recordings/longn... Switched to branch 'master' diff --git a/docs/basics/_examples/DL-101-137-127 b/docs/basics/_examples/DL-101-137-127 index 932dbbb14..c08003943 100644 --- a/docs/basics/_examples/DL-101-137-127 +++ b/docs/basics/_examples/DL-101-137-127 @@ -1,4 +1,5 @@ -$ git cat-file --textconv f59a78356d78525239347ecf763b6677130898af:notes.txt + +$ git cat-file --textconv 3a4418dc8d74b9889da49b26e0f201e169902113:notes.txt One can create a new dataset with 'datalad create [--description] PATH'. The dataset is created empty diff --git a/docs/basics/_examples/DL-101-137-132 b/docs/basics/_examples/DL-101-137-132 index 100a3bbdf..6c855c56a 100644 --- a/docs/basics/_examples/DL-101-137-132 +++ b/docs/basics/_examples/DL-101-137-132 @@ -1,3 +1,3 @@ $ git log -n 2 --oneline -b6dd7d1 add joke evaluation to joke -51d763e save my favourite Git joke +26b14ab add joke evaluation to joke +b8972e0 save my favorite Git joke diff --git a/docs/basics/_examples/DL-101-137-133 b/docs/basics/_examples/DL-101-137-133 index 36b00e1f2..97624c928 100644 --- a/docs/basics/_examples/DL-101-137-133 +++ b/docs/basics/_examples/DL-101-137-133 @@ -1,2 +1,3 @@ -$ git reset --hard 51d763ed5883bc51f6eabc36248e4ef80b9f600f -HEAD is now at 51d763e save my favourite Git joke + +$ git reset --hard b8972e080e665903f10c32c3d174211c282324e1 +HEAD is now at b8972e0 save my favorite Git joke diff --git a/docs/basics/_examples/DL-101-137-144 b/docs/basics/_examples/DL-101-137-144 index e1fe1ca05..02c4a491b 100644 --- a/docs/basics/_examples/DL-101-137-144 +++ b/docs/basics/_examples/DL-101-137-144 @@ -1,5 +1,5 @@ -$ git revert aaa1988adfa83a703a756f911a0beffc4fefa08a -[master 37c73e7] Revert "did a bad modification" - Date: Mon Nov 18 08:44:48 2019 +0100 +$ git revert 795f94ab667188877575dc4db900f3d6aed0a4a2 +[master 7df383f] Revert "did a bad modification" + Date: Wed Dec 4 16:23:20 2019 +0100 1 file changed, 1 deletion(-) diff --git a/docs/basics/_examples/DL-101-137-146 b/docs/basics/_examples/DL-101-137-146 index 1d143d932..a82344df5 100644 --- a/docs/basics/_examples/DL-101-137-146 +++ b/docs/basics/_examples/DL-101-137-146 @@ -1,20 +1,20 @@ $ git log -n 3 -commit 37c73e7eff6df09ce3ea9a149e64ea5e82ed5d8b +commit 7df383fd912c9f9841c4cca0161930e14fbb3c8d Author: Elena Piscopia -Date: Mon Nov 18 08:44:48 2019 +0100 +Date: Wed Dec 4 16:23:20 2019 +0100 Revert "did a bad modification" - This reverts commit aaa1988adfa83a703a756f911a0beffc4fefa08a. + This reverts commit 795f94ab667188877575dc4db900f3d6aed0a4a2. -commit 696c3878120b997eb3f6b4e552de5875664ab874 +commit 35d77bee63472970f76668cb4566758044ff1f4b Author: Elena Piscopia -Date: Mon Nov 18 08:44:48 2019 +0100 +Date: Wed Dec 4 16:23:20 2019 +0100 add note on helpful git resource -commit aaa1988adfa83a703a756f911a0beffc4fefa08a +commit 795f94ab667188877575dc4db900f3d6aed0a4a2 Author: Elena Piscopia -Date: Mon Nov 18 08:44:48 2019 +0100 +Date: Wed Dec 4 16:23:19 2019 +0100 did a bad modification diff --git a/docs/contents.rst.inc b/docs/contents.rst.inc index 3be76d66b..d6378e479 100644 --- a/docs/contents.rst.inc +++ b/docs/contents.rst.inc @@ -89,7 +89,7 @@ Make the most out of datasets .. toctree:: :maxdepth: 1 - :caption: Organizational principles and best practices for data analyses + :caption: Organizational principles and best practices for reproducible data analyses basics/101-126-intro basics/101-127-yoda @@ -97,16 +97,17 @@ Make the most out of datasets basics/101-128-summary_yoda -########################### -Dataset nesting -- Advanced -########################### +################ +One step further +################ .. toctree:: :maxdepth: 1 - :caption: Manipulating subdatasets and its impact on the superdataset(s) + :caption: Advanced nesting and "extended" reproducibility basics/101-132-advancednesting - basics/101-132-gitC + basics/101-133-containersrun + basics/101-134-summary ############# @@ -133,6 +134,8 @@ Further options :caption: Small pieces of advice and helpful additional options basics/101-179-gitignore + basics/101-144-intro_extensions + ######### Use Cases diff --git a/docs/glossary.rst b/docs/glossary.rst index 6df8769b8..bdc6ee978 100644 --- a/docs/glossary.rst +++ b/docs/glossary.rst @@ -138,6 +138,11 @@ Glossary - ``execute permissions`` grant the ability to execute a file. Any script that should be an executable needs to get such permission. + pip + A Python package manager. Short for "Pip installs Python". ``pip install `` + searches the Python package index `PyPi `_ for a + package and installs it while resolving any potential dependencies. + provenance A record that describes entities and processes that were involved in producinng or influencing a digital resource. It provides a critical foundation for assessing authenticity, enables trust, diff --git a/docs/intro/_examples/how-to-1 b/docs/intro/_examples/how-to-1 index 59064f2de..22bd89f0b 100644 --- a/docs/intro/_examples/how-to-1 +++ b/docs/intro/_examples/how-to-1 @@ -1,5 +1,5 @@ $ ls -l output.txt 24+0 records in 24+0 records out -25165824 bytes (25 MB, 24 MiB) copied, 0.0195906 s, 1.3 GB/s --rw-r--r-- 1 adina adina 25165824 Nov 18 10:53 output.txt +25165824 bytes (25 MB, 24 MiB) copied, 0.013529 s, 1.9 GB/s +-rw-r--r-- 1 adina adina 25165824 Nov 26 12:54 output.txt diff --git a/docs/intro/_examples/how-to-2 b/docs/intro/_examples/how-to-2 index dcd6c3111..dff83e2d5 100644 --- a/docs/intro/_examples/how-to-2 +++ b/docs/intro/_examples/how-to-2 @@ -1,4 +1,4 @@ $ ls -lh output.txt # note that short options can be combined! # or alternatively $ ls -l --human-readable output.txt --rw-r--r-- 1 adina adina 24M Nov 18 10:53 output.txt +-rw-r--r-- 1 adina adina 24M Nov 26 12:54 output.txt