Skip to content
This repository has been archived by the owner on Aug 25, 2024. It is now read-only.

2nd Party #1401

Merged
merged 2,135 commits into from
Dec 5, 2023
Merged

2nd Party #1401

merged 2,135 commits into from
Dec 5, 2023

Conversation

johnandersen777
Copy link

@johnandersen777 johnandersen777 commented Jun 24, 2022 •
edited
Loading

Alice is Here! It’s the 2nd Party and everyone is invited 💃🥳. Alice is both the nickname for the Open Architecture, the methodology of description of any system architecture, as well as the entity defined using that description of architecture. She is the entity and the architecture.

@johnandersen777
Copy link
Author

johnandersen777 commented Jun 25, 2022 •
edited
Loading

Moved to: https://github.com/intel/dffml/blob/3ce85c8f48e702e3ed1e268769c8121377abe324/docs/tutorials/rolling_alice/0001_coach_alice/0000_introduction.md

Volume 1: Coach Alice: Introduction

To time travel, an entity must first accelerate. The entity we now
turn our attention to we know well. Her name is Alice, and she's
falling down the rabbit hole as we speak. We begin our series
somewhere between the tick and the tock. As she nears the bottom
of the rabbit hole time slows for a moment, for her, as she enters
Wonderland. The pattern which is Alice is a spark in the mind. She's
all in your head, after all, everything is all in your head. In a
way she is in your head, and you're in her head, because
conceptually, the architecture is the same, and the architecture
is one of concepts, because it's all, in fact, just in your head.

We will coach Alice and she will coach us. From our point of view
if you can't teach it you don't know it. So it's time to each Alice
how to be an open source contributor, by teaching her how to teach
it. In this volume, volume 1, we will build Coach Alice, our open
source developer coach. It's developer boot camp for Alice this
volume as her boots make contact with the ground at the bottom of
the rabbit hole.

Misc notes

  • If we can teach Alice how to operate based on intent,
    and how to have her intentions always be good. Where good is defined
    the communities strategic principles and values.
    and we validate the hell out
    of here. We will step through the looking glass into a community of
    the future where we can only trust ourselves. In that trust in
    ourselves we will find trust in others, in measured, yet meaningful
    ways.
  • Where we can work optimally in a hybrid workplace environment. Allowing us to
    reconnect with the physical world. To embrace the world that exists.
  • What this means in reality is that Alice will be communicating for us,
    we will begin to think of her as a messenger relaying a message
  • The most open, self reliant, confident humans. Ready to take on the world.

@johnandersen777
Copy link
Author

johnandersen777 commented Jun 27, 2022 •
edited
Loading

More Detailed TODOs

  • a thought is like a vector. The direction is dependent on context, herstory. As well as the thought itself (the system context, upstream, overlay, orchestrator)
  • How to add data from static file stored anywhere into flow by adding a new operation
  • Alice takes strategy to execution to feedback she’s the full analysis control feedback loop with strategic plans/principles at the helm. She sits at the intersection of CI/CD, Analytics, and Security. She enables optimal execution of agent flows via context aware overlays to best understand the specifics of their execution environment.
  • Sphinx builder to GitHub render able markdown, this way docs can just live in github without deploy phase.
    • We can build our new web5 git trees to be based on provenance, we can say the next link in the chain must be a successful docs build or the failure.
  • We are interested in exploiting the use of context switching for learning. We hypothesis in the docs/arch/alice/discussion files that there are aligned conscious states which alice will identify, cross entities, build training opetions over time for each agent
  • Helper script to use https://github.com/monomonedula/nvelope to generate JSON schemea
  • alice shell tutorial follow on where we use the -orchestrator option which we have flows which return orchestrators kind of like the dataflow/system context version of dataset_source(). We can then point to installed overlays -orchestrator dffml.overlays.alice.shell.in_local_container. We can also use the mechanisms by which we extend Entrypoint.load.
    • In yet another tutorial we will build on the first to do: alice open vscode which will launch a vscode instance locally or in a cloud development environment depending on context, if you are in a shell on a headless machine, it will pop you to the static vscode hosted site by Microsoft in your broswer and you will connect back to it with webrtc based comms (or websocket, or abitrary worm holes). This allows us to use the systems we already have most efficently, i.e. extra dev k8s cluster capacity.
    • Go around and have alice try to build the projects, generate template workspace template files/repos for people to fork which have the Dockerfile built for the dev env with kaniko build which on complete submits a PR to update the template to the latest container sha after validation passes. Run workflow to call coder templates update once completed
  • Distributed network of metric collectors. Of security information we should feed into cve bin tool. Maybe start by creating checker entrypoint for a checker which knows it’s running under a dataflow. Could execute using runpy with opimp self in globals. OpImp shared config property of object which is dataflow as class which listens for new vuln info for checker in background when instantiated. When a new vuln is detected we could trigger a scan on all previously scanned atrifacts for which we had scanned before by having a strategic plan overlayed on a long running flow which inspects historical contexts which executed scans against checker came up as exists within file scanned. Use this presence of existence within previous scans to query off chain data from historical system contexts. To build next contexts where results of scan opperatuon are removed so that running results in latest vuln info being incorporated into scan. This is analogous to dev pull model when new commit on branch released. Do scan rerun is same as redoing A/B feature testing of commits.
  • Address DORA metrics in Alice and the Health of the Ecosystem: https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance
  • Auto discovery via sourcegraph search github for entrypoint signatures, instantiate lazy loading OperationImplemtationNetworks within overlay apply add them to orchestrator as applicable.
  • Streaming development activity == Stream of consciousness == streaming consciousness == thought transport protocol == thought communication protocol == LMWC == Alice == Open Architecture == model of the human mind == integrated risk management (resilience, fault tolerance)
    • Implementations may have dynamic facilitators of communications for distributed data and compute (operations, agents, cells). They may also have synthesizers which take an Open Architecture and produce a static representation. In fact since the open architecture can be used to represent anything. If you take a binary and select the orchestrator equivalent of a NOP (we should just implement NOP orchestrator which does nothing, meaning the dataflow is not given to be executed in any environment and only to be used as a reference.). So you take input, you asynchronously produce output, the same binary, unmodified, thats what your program says to do. Well then we dont even need to run your program now do we? We already know based on analysis of its context stating that via the deployment orchestrator and the metadata embedded within the open architecture document which says it takes the files, does nothing with it (pass). The parent system context has brought your system context to equilibrium without needing to execute it. We get this for free because we understand context, because the parent system context acts as a loader which can dynamically execute sandboxed code within different trust boundaries (contexts)
  • Add arbitrary project name and metadata in policy.toml
    • policy.toml -> DataFlow -> Overlay
  • Synthesis of DataFlows to GitHub Actions workflows
    • Start by writing operation which just dump output of each templated workflow accessed with importlib.resources
      • Data Flow templating, with native async
    • Workflow ideas
      • Python build
      • Python test
      • Sphinx build
  • please contribute
    • recommended community standards
      • README.md
      • SECURITY.md
      • CODE_OF_CONDUCT.md
    • threats
      • THREATS.md
    • documentation
      • sphinx docs
        • Basic docs sourced from dffml.git/dffml/skel/common/docs
        • autodocs with python modules, can look at dffml.git/scripts/docs_api.py or whatever it's called
diff --git a/dffml/util/testing/consoletest/cli.py b/dffml/util/testing/consoletest/cli.py
index 0f8294155..dd9e057c8 100644
--- a/dffml/util/testing/consoletest/cli.py
+++ b/dffml/util/testing/consoletest/cli.py
@@ -44,7 +44,7 @@ async def main(argv: List[str]) -> None:
     nodes = []

     for node in parse_nodes(args.infile.read()):
-        if not node.options.get("test", False):
+        if args.filter is not None and not node.options.get(filter, False):
             continue
         if node.directive == "code-block":
             nodes.append(
diff --git a/entities/alice/README.rst b/entities/alice/README.rst
index aca0dbc87..53465db6f 100644
--- a/entities/alice/README.rst
+++ b/entities/alice/README.rst
@@ -4,6 +4,18 @@ Alice
 Install
 *******

+Install latest known working version
+
+.. code-block:: console
+
+    $ python -m pip install \
+        "https://github.com/intel/dffml/archive/42ed3da715f1c89b4c31d705cf7f7738f17c9306.zip#egg=dffml" \
+        "https://github.com/intel/dffml/archive/42ed3da715f1c89b4c31d705cf7f7738f17c9306.zip#egg=dffml-feature-git&subdirectory=feature/git" \
+        "https://github.com/intel/dffml/archive/42ed3da715f1c89b4c31d705cf7f7738f17c9306.zip#egg=shouldi&subdirectory=examples/shouldi" \
+        "https://github.com/intel/dffml/archive/42ed3da715f1c89b4c31d705cf7f7738f17c9306.zip#egg=dffml-config-yaml&subdirectory=configloader/yaml" \
+        "https://github.com/intel/dffml/archive/42ed3da715f1c89b4c31d705cf7f7738f17c9306.zip#egg=dffml-operations-innersource&subdirectory=operations/innersource" \
+        "https://github.com/intel/dffml/archive/42ed3da715f1c89b4c31d705cf7f7738f17c9306.zip#egg=alice&subdirectory=entities/alice"
+
 Install for development

 .. code-block:: console
diff --git a/setup.py b/setup.py
index 47c595547..8157381f4 100644
--- a/setup.py
+++ b/setup.py
@@ -75,6 +75,7 @@ setup(
     # Temporary until we split consoletest into it's own package
     install_requires=[],
     extras_require={
+        "consoletest-jsonpath-filter": ["jsonpath-python"],
         "dev": DEV_REQUIRES,
         **plugins.PACKAGE_NAMES_BY_PLUGIN_INSTALLABLE,
     },

withmyconfig

More detailed future work

  • Collaboratively define the first version of .tools/open_architecture/policy.yml with CVE Bin Tool community
    • Working with IETF/SCITT WG on this now as well, need to cross with CVE Bin Tool once policy method choice there is settled, if anyone wants to pursue this in the meantime go for it.
    • Based off their triage work, work within their codebase, write manifest there, link to from here.
    • Leverage Open Policy Agent
    • Arjan:
      • Allow for definition of per branch policies (this will be like the robots.txt of Alice)
      • Helps us handle decide how to handle vulnerability fixing pull requests.
        • Examples
          • If we are using openssl_client and CVEs keep coming in for OpenSSL, note in the policy file that openssl_server CVEs are on the next_regular_release cadence. Do not nag about them. openssl_client CVEs however nag us on X interval (daily, hourly, call me until I update).
  • alice how can I help? (alternate with issue assignment overlay: alice please assign me gh issues or alice please assign @team gh issues)
    • Life is not a miracle it’s a recipe with integrated dynamic risk management. The purpose of life is to sustain life, to create more life. Life is meaningless, we give it meaning. Meaning is intent. It’s what we do and intend to do from the tick to the tock.
      • upstream: What’s in our heads, our heads themselves. The brain, the open architecture, the methodology. “[It’s all in your head.] But so is everything” [Fiona Apple]
      • overlay: the prophecy, the plan we collaboratively build, the manifest destiny, the optimal security config for code, the optimal happiness of developers, those context aware policies that are the sprit of the law (which get translated into the letter on demand in a reproducible and auditable fashion)
        • All one needs to do to contribute to Alice is to think about how one can become the entity one wants to be. Refine that rolling plan, roll with it, roll with the self, roll with Alice, flow roll. We’re competing, but as a team, to make each other better. We are better when our training partners are better and Alice is a training partner, a learning partner, for us all. She is a shared reality, a shared view, a telescope without the lenses. She is the entity and the architecture.
          • We are exploiting the current lack of a shared reality to preform the consensus shift mind control attack (grep discussion for more info). Our objective in the execution of this technique is to patch the mind on an entity. We exploit so as to introduce a TCB upon which patches to mental security can be applied. The infrastructure assist levels up the entities to the restored state, the state at which a set is at critical velocity. The point at which they cannot be influenced by further mind control attacks, that entity is now fully in control of their own mind. Because the infra is controlled by no one and everyone all at once (grep: taking the lock) this means our flow roll for that set is optimal, we can switch in and out of intensive engagements, to support, to enabling, seamlessly in the aggregate. This policy also incorporates teamwork in a hostile workplace environment principles.
      • orchestrator: alice (implementation, ourselves)
        • The prophecy MUST be fulfilled (rfc2119)
          • Mutually Assured Victory
          • Take our reality, snapshot all of us, that is our current pointer to system context, whatever “now” is. For that system context to exist there are an infinite number of similar system contexts which do not exist. There are an infinite number of other options for “now”. Each one of those, while they do not exist within “now” still exist to the top level system context, the parent to our “now”, this generalization is recursive, turtles all the way up and down. There is only one valid system context within our universe at any given time. That valid system context is determined by time, the lock, the shared resource on which the single electron for our locality takes as it executes each operation in a cycle, that time in cycle and that election is also relative, because the architecture scales up and down, its the same architecture the whole way through, sometimes the electron (the agent) crosses locality boundaries, sometimes it is shared, these show up as trust boundaries for the team work in a hostile workplace environment tutorials, asset sharing across trust boundaries and risk management and response to hostile actions.
          • The moral of the story is. If I trust you I trust you. You’re getting root privs, and this is distributed compute. We will determine the optimal strategic principles using ethics, humanity, the giants whose shoulders we stand on, and each other as our guides in Volume 5
    • Go around and look at the profile of the developer based on the herstorical quality of their code (the feature extraction which we did off of the union of the repo collector flow caches overlayed with an output flow)
    • Estimate the coverage which the issue will touch based on herstorically similar issues based in part on similarity to coverage which was touched when the previous issue was fixed.
    • go around and each estimates how long the task would take (tS, tM), time estimate should be taking the feature from POC to prod, docs, tests, everything. always assume a scalled ammount of time required for comunication when working on issues and resaerch, have stages for issues where we have meta issues and then track exploration and reaserch as a timeblock itself.
    • Display time estimates for developer/agent to complete TODOs (or GitHub or Jira or other issues) as list.
    • Optional overlays for assignment of issues automatically or select top 10 matched with happiness (guiding metric, learning rate and balance with teaching rate, executed distributed remembering thoughts, and other things roll into this)
  • https://github.com/taesungp/contrastive-unpaired-translation
  • IoA
  • https://sites.google.com/pdx.edu/map-the-system/resources-to-make-a-systems-map
    • In relationship to our field mappijg. This goes somewhere in chapter 5 when Alice learns what dev activies she should engage in by looking for projects where knitting then togeter or bugfixes have large impact via field gap analysis. See also wardly maps comments
  • https://lists.spdx.org/g/Spdx-tech/message/4656
    • SPDX DAG

@johnandersen777
Copy link
Author

johnandersen777 commented Jun 29, 2022 •
edited
Loading

2022-06-29 14:00 UTC -7

  • For multiple styles of right hand overlay paths, override Entrypoint.load as Overlay.load, make it not do i.load() without first checking if alternate methods are available.

2022-06-30 13:00 UTC -7

Triggerable Workflow for Alice Please Contribute

Failure to Launch tbDEX Stack

  • Metadata
    • Date: 2022-07-07 08:00 UTC -7
  • web5 will be our vuln sharing data lake for cve-bin-tool scans uploaded cached when run via dataflow execution for collelation by us or Alice
  • https://frankhinek.com/getting-started-with-tbds-ssi-service/
    • Found via https://twitter.com/frankhinek/status/1541585711740092417
    • Needed docker-compose right off the bat
    • Switched to https://github.com/containers/podman-compose
    • Set docker.io as registry to pull FROM by default when no domain given in a container name via sudo vim /etc/containers/registries.conf
      • unqualified-search-registries = ["docker.io"]
    • Attempted podman-compose up
      • Blue screen of death on image pull, WSL seems to cause OOM on either the download or the write to disk, there were disk issues in WSL1, probably disk.
  • Misc.
    • Twitter is currently a good way to to monitor the state of the art in various domains / fields.

Cleaning Up the Docs Build

$ git log -p -- dffml/version.py \
                 | grep \+VERSION \
                 | grep -v rc \
                 | sed -e 's/.* = "//g' -e 's/"//g' \
                 | head -n 1
0.4.0
$ git log -p -- dffml/version.py                  | grep \+VERSION                  | grep -v rc
+VERSION = "0.4.0"
+VERSION = "0.3.7"
+VERSION = "0.3.6"
+VERSION = "0.3.5"
+VERSION = "0.3.4"
+VERSION = "0.3.3"
+VERSION = "0.3.2"
+VERSION = "0.3.1"
+VERSION = "0.3.0"
+VERSION = "0.2.1"
+VERSION = "0.2.0"
+VERSION = '0.2.0'
+VERSION = '0.1.2'
+VERSION = '0.1.1'
+VERSION = '0.1.0'
  • Will need update latest_release to look through git tags.

Refactor Meta Issue Creation to Accept Dynamic Inputs

  • Metadata
    • Date: 2022-07-07 12:00 UTC -7
  • meta_issue_body() in entities/alice/alice/please/contribute/recommended_community_standards/alice/operations/github/issue.py should be an OperationImplemenationContext and take inputs dynamically.

Debugging Meta Issues as Output Operations

  • Metadata
    • Date: 2022-07-07 14:30 UTC -7
  • Definition.spec type loading information is limited to builtin types currently
  • The unified config methodology (plugin, config, schema) is the manifest, is the upstream the overlay and the orchestrator is the Open Architecture.
  • How do we come up with alternate Open Architectures (threat model migrations based on intent)? We follow the pattern:
    • Enumerate everything we know
    • Determine which are assets and which are processes
    • Using tuned brute force, prune the trees based on strategic principles (top level strategic plans or conscious or conceptual layers)
  • All we're doing is providing a user/develop friendly experience to follow the pattern
  • architype
    • Is a mental construct to help us understand when data is aligned to a particular set (a system context, perhaps in part or completely overlaid) of strategic principles.

Verification of Successful Alice Should I Contribute Job

  • Metadata
    • Date: 2022-07-08 20:00 UTC -7
  • https://www.investopedia.com/terms/o/overlay.asp

    • What Is Overlay?
      Overlay refers to a management style that harmonizes an investor's separately managed accounts. Overlay management uses software to track an investor's combined position from separate accounts. The overlay system analyzes any portfolio adjustments to ensure the overall portfolio remains in balance and to prevent any inefficient transactions from occurring. Overlay portfolio management makes sure the investor’s strategies are implemented and coordinated successfully.

    • Remember we don't care about money (in fact remember we're going to get rid of money ;)
      • We care about risk management, this is applicable to our context, which is understood in part via overlays
  • Alice Should I Contribute? CI run:

We needed to get the artifact uploaded with the collector results to inspect
that the output of results.json made sense.

$ gh run list --workflow alice_shouldi_contribute.yml
completed success alice: ci: shouldi: contribute: Remove errant chdir to tempdir  Alice Should I Contribute?  alice workflow_dispatch 2639160823  1m48s 3h
completed success alice: ci: shouldi: contribute: Upload collector outputs as artifacts Alice Should I Contribute?  alice workflow_dispatch 2638950785  56m14s  4h
completed success alice: ci: shouldi: contribute: Basic job Alice Please Contribute Recommended Community Standards alice workflow_dispatch 2638890594  1m15s 4h
$ gh run list --workflow alice_shouldi_contribute.yml | awk '{print $(NF-2)}'
2639160823
2638950785
2638890594
$ gh run list --workflow alice_shouldi_contribute.yml | awk '{print $(NF-2)}' | head -n 1
2639160823

We figured out how to use the GitHub CLI's builtin jq style output
selection to build a JSON object where the keys are the artifact names
and the values are the URLs.

References:

$ gh api   -H "Accept: application/vnd.github+json"   /repos/intel/dffml/actions/runs/$(gh run list --workflow alice_shouldi_contribute.yml | awk '{print $(NF-2)}' | head -n 1)/artifacts --jq '.artifacts[] | {(.name): .archive_download_url}'
{"collector_output":"https://api.github.com/repos/intel/dffml/actions/artifacts/293370454/zip"}

Here we select only the URL of the archive. There is only one artifact,
so we only get one zip file in the output.

$ gh api   -H "Accept: application/vnd.github+json"   /repos/intel/dffml/actions/runs/$(gh run list --workflow alice_shouldi_contribute.yml | awk '{print $(NF-2)}' | head -n 1)/artifacts --jq '.artifacts[] | .archive_download_url'
https://api.github.com/repos/intel/dffml/actions/artifacts/293370454/zip

Confirm it is a zip file by looking at the bytes with xxd

$ gh api   -H "Accept: */*" $(gh api   -H "Accept: application/vnd.github+json"   /repos/intel/dffml/actions/runs/$(gh run list --workflow alice_shouldi_contribute.yml | awk '{print $(NF-2)}' | head -n 1)/artifacts --jq '.artifacts[] | .archive_download_url' | sed -e 's/https:\/\/api.github.com//g') | xxd
00000000: 504b 0304 1400 0800 0800 25be e854 0000  PK........%..T..
00000010: 0000 0000 0000 0000 0000 0a00 0000 7265  ..............re
00000020: 706f 732e 6a73 6f6e cd53 5dab db30 0cfd  pos.json.S]..0..
00000030: 2b25 cfed aad8 f247 fa76 5fef f39e 564a  +%.....G.v_...VJ
00000040: 906d a5c9 d626 2571 584b e97f 9f1d 36d8  .m...&%qXK....6.
00000050: 18bd dc3d 6ccc 200b eb1c 593e 46ba 1773  ...=l. ...Y>F..s
00000060: 1fe9 78e4 50ec 56f7 a28d f132 edb6 db63  ..x.P.V....2...c
00000070: 17db d97d f0c3 797b 09d7 cf43 dbf7 b76d  ...}..y{...C...m
00000080: 0623 4f71 617e e15b f2ef 4c58 af8a 8629  .#Oqa~.[..LX...)
00000090: ce23 4f4b f2c8 27a6 89eb af29 abeb eb0b  .#OK..'....)....
000000a0: 8fdd 901f b06f e834 f17a 15c7 39ed df0f  .....o.4.z..9...
000000b0: bfba 37a0 c51d 522d 9a63 3b8c f5a9 ebb9  ..7...R-.c;.....
000000c0: f643 1298 afbe 3fd6 a9f2 6b7a d9ea a50f  .C....?...kz....
000000d0: 3c4e dca7 b034 f824 9ec3 3fec 3758 953f  <N...4.$..?.7X.?
000000e0: c38b e5c2 49fe b98b f5d4 52d6 b92f 00ad  ....I.....R../..
000000f0: 2623 83a7 408d 7352 a317 dab0 6215 a0d4  &#..@.sR....b...
00000100: 2548 d056 a8aa ca1f f427 5caf 6440 cb8c  %H.V.....'\.d@..
00000110: 06ad 0343 257a cb3a 4803 440e 981c d920  ...C%z.:H.D....
00000120: 524a e62a 2d41 341e 146a acc8 2b6c 82f5  RJ.*-A4..j..+l..
00000130: 28bc 5589 a81b aca4 741a 1bf7 37b9 649d  (.U.....t...7.d.
00000140: 4228 856c 2482 d724 1481 f3d2 28e6 0a50  B(.l$..$....(..P
00000150: 1963 2450 c9fe bfe0 1e12 3974 7e69 9ae7  .c$P......9t~i..
00000160: 7df6 afdd 2135 5971 a229 d6f3 2550 5ce6  }...!5Yq.)..%P\.
00000170: b510 20c4 06cc 06ec 4721 7758 edca f253  .. .....G!wX...S
00000180: d6ca d738 529e b447 5adf 0050 4b07 08de  ...8R..GZ..PK...
00000190: 90d9 ba63 0100 00e3 0300 0050 4b01 0214  ...c.......PK...
000001a0: 0014 0008 0008 0025 bee8 54de 90d9 ba63  .......%..T....c
000001b0: 0100 00e3 0300 000a 0000 0000 0000 0000  ................
000001c0: 0000 0000 0000 0000 0072 6570 6f73 2e6a  .........repos.j
000001d0: 736f 6e50 4b05 0600 0000 0001 0001 0038  sonPK..........8
000001e0: 0000 009b 0100 0000 00                   .........

Make an authenticated query to the GitHub API asking for the resource. Pipe the
output to zip file to save it to disk.

TODO Operation to download all GitHub run artifacts.

$ gh api   -H "Accept: */*" $(gh api   -H "Accept: application/vnd.github+json"   /repos/intel/dffml/actions/runs/$(gh run list --workflow alice_shouldi_contribute.yml | awk '{print $(NF-2)}' | head -n 1)/artifacts --jq '.artifacts[] | .archive_download_url' | sed -e 's/https:\/\/api.github.com//g') > collector_output.zip

Extract the zipfile to a directory

$ python -m zipfile -e collector_output.zip collector_output/

Look at the contents of the extracted directory to confirm all the files we
think should be there are there.

TODO Verification via cartographic or other trust mechanisms

$ ls -lAF collector_output/
total 4
-rw-r--r-- 1 pdxjohnny pdxjohnny 995 Jul  8 20:09 repos.json

Then we ask Python to use the json.tool helper to pretty print the file.

$ python -m json.tool < collector_output/repos.json
{
    "untagged": {
        "https://github.com/pdxjohnny/httptest": {
            "key": "https://github.com/pdxjohnny/httptest",
            "features": {
                "release_within_period": [
                    false,
                    true,
                    false,
                    false,
                    false,
                    true,
                    false,
                    false,
                    false,
                    false
                ],
                "author_line_count": [
                    {},
                    {
                        "John Andersen": 374
                    },
                    {
                        "John Andersen": 37
                    },
                    {},
                    {},
                    {
                        "John Andersen": 51
                    },
                    {},
                    {},
                    {},
                    {}
                ],
                "commit_shas": [
                    "0486a73dcadafbb364c267e5e5d0161030682599",
                    "0486a73dcadafbb364c267e5e5d0161030682599",
                    "c53d48ee4748b07a14c8e6d370aab0eaba8d2103",
                    "56302fc054649ac54fd8c42c850ea6f4933b64fb",
                    "56302fc054649ac54fd8c42c850ea6f4933b64fb",
                    "56302fc054649ac54fd8c42c850ea6f4933b64fb",
                    "a8b540123f340c6a25a0bc375ee904577730a1ec",
                    "a8b540123f340c6a25a0bc375ee904577730a1ec",
                    "a8b540123f340c6a25a0bc375ee904577730a1ec",
                    "a8b540123f340c6a25a0bc375ee904577730a1ec"
                ],
                "dict": [
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false,
                    false
                ]
            },
            "last_updated": "2022-07-08T23:49:11Z",
            "extra": {}
        }
    }
}

Then we wrote this little mini tutorial by dumping our shell herstory and adding
some explination.

$ herstory | tail -n 50 > /tmp/recent-herstory
$ vim /tmp/recent-herstory

Then we copy paste and upload to somewhere our collogues and ourselves will have
access to in the future when they want to know more about our development
process for this patch. This is similar to and extended commit message. In fact,
we'll be playing with linkages to commits of data generated during development
for query later by Alice and others.

Manual Spin Up of Digital Ocean VM

nahdig is the term we use for blocklist, this means suspect TCB,
we are going to download a bunch of untrusted code on these machines,
we need to keep they keys separate to avoid confusion and help with
audit.

We will add a comment to our key whose email domain we currently
play with as concept where we are exploring scoping identities
to system contexts (alice.shouldi.contribute as the base flow /
system context in this example).

$ ssh-keygen -f ~/.ssh/nahdig -b 4096 -C 'pdxjohnny@contribute.shouldi.alice.nahdig.com' 
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/pdxjohnny/.ssh/nahdig
Your public key has been saved in /home/pdxjohnny/.ssh/nahdig.pub
The key fingerprint is:
SHA256:PsTjWi5ZTr3KCd2ZYTTT/Xmajkj9QutkFbysOogrWwg pdxjohnny@DESKTOP-3LLKECP
The key's randomart image is:
+---[RSA 4096]----+
|                 |
|           . o   |
|          + . +  |
|       . . o . +.|
|  E     S.o   +.o|
|   . . =o+.=.o o.|
|    . o**.=o=.o  |
|    ..+*o+o=o+   |
|    .ooo=.ooo.o  |
+----[SHA256]-----+
$ cat ~/.ssh/nahdig.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDX1xvaybJQLrUxytn+AR+F3dDeAxFDMr0dyDt6zGs45x8VsA3TDrneZZ7ggzN63Uzbk+CAuBRDPGms6FgPswliU6xgp8X073Pcn2va7JxbkIz0LQCxdzfAoOMKIIiI7SmYSD4IDFqrEHnN+I6j4el+IFLaGTibCAR0+zK4wKPX97NE27EPUL/DYkT2eBAF/onLZAQm3tRznTYUSyWaXlWHQUD6y/3QtvhH3WVIUKRV8b6POwoiVa6GeMjM5jVCBRB+nfrhjNBBp7Ro7MzuNn+z8D6puyV1GFxWtSm953UYFa5UcahhiwFRWXLfJmVjwEZDm0//hMnw1TcmapBR99dwrBftz+YFF5rpTyWvxbnl5G/wn2DQ/9RFR6SeD3GImYRhVSFkuNZkQCiaj2+bT+ngnFPEA5ed4nijFnIgvAzPz9kk7uojjW3SfEdhED0mhwwBlLNOr7pGu9+X2xZQIlFttuJaOjd+GYBWypchd7rWdURHoqR+07pXyyBAmNjy6CKwSWv9ydaIlWseCOTzKxjy3Wk81MoaH/RhBXdRFqS1mP12TuahMzTvvVuSfQQJKCO05sIrzSEykxg1u6HEZXDyeKoVwN9V1/tq3QGa4tE/WmMNaLukef9ws3Drt1D7HWTF7u/N/zjtfiyEXRAMkixqywHfCrrxXKGPR7uvueLUkQ== pdxjohnny@contribute.shouldi.alice.nahdig.com

TODO(security) Use VM startup script via cloud-init or otherwise to exfil
the SSH daemon's public key on sshd start (maybe via
systemd files, aka start after sshd is up) this way we
can set StrictHostKeyChecking=yes and provide a
UserKnownHostsFile with the server key in it.

We ssh into the new VM.

$ ssh -i ~/.ssh/nahdig -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no root@143.198.133.87

This is MUST happen, after spin up so that clients SSH'ing in
are sure they are not being Entity In The Middle'd (EITM).

POC Launch of tbDEX Stack

As root, add a non-root user with root/sudo privileges and
their own home directory (-m).

# useradd -m -s $(which bash) pdxjohnny
# usermod -aG sudo pdxjohnny

Allow the user to use sudo without a password.
Note that there is both a \t (tab) and a space in the line.

# sed -i 's/\%sudo\tALL=(ALL:ALL) ALL/\%sudo\tALL=(ALL:ALL) NOPASSWD:ALL/g' /etc/sudoers

Update the VM

# apt-get update && DEBIAN_FRONTEND=noninteractive apt-get upgrade -y

Install tools

# DEBIAN_FRONTEND=noninteractive apt-get install -y tmux bash-completion vim git python3 python3-pip

Install GitHub CLI for auth by adding custom GitHub package repo to OS.

References:

# curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
# echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null
# apt-get update
# apt-get install -y gh

Install Docker

References:

# apt-get update
# apt-get install -y \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
# mkdir -p /etc/apt/keyrings
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
# echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
# apt-get update
# apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

Verify docker is running by querying systemd.

# systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-07-09 05:45:29 UTC; 23s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 27732 (dockerd)
      Tasks: 13
     Memory: 34.3M
     CGroup: /system.slice/docker.service
             └─27732 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Jul 09 05:45:29 alice-shouldi-contribute-0 dockerd[27732]: time="2022-07-09T05:45:29.342805567Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jul 09 05:45:29 alice-shouldi-contribute-0 dockerd[27732]: time="2022-07-09T05:45:29.342882429Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Jul 09 05:45:29 alice-shouldi-contribute-0 dockerd[27732]: time="2022-07-09T05:45:29.342940895Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Jul 09 05:45:29 alice-shouldi-contribute-0 dockerd[27732]: time="2022-07-09T05:45:29.343190438Z" level=info msg="Loading containers: start."
Jul 09 05:45:29 alice-shouldi-contribute-0 dockerd[27732]: time="2022-07-09T05:45:29.459917630Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jul 09 05:45:29 alice-shouldi-contribute-0 dockerd[27732]: time="2022-07-09T05:45:29.518683858Z" level=info msg="Loading containers: done."
Jul 09 05:45:29 alice-shouldi-contribute-0 dockerd[27732]: time="2022-07-09T05:45:29.542989855Z" level=info msg="Docker daemon" commit=a89b842 graphdriver(s)=overlay2 version=20.10.17
Jul 09 05:45:29 alice-shouldi-contribute-0 dockerd[27732]: time="2022-07-09T05:45:29.543173681Z" level=info msg="Daemon has completed initialization"
Jul 09 05:45:29 alice-shouldi-contribute-0 systemd[1]: Started Docker Application Container Engine.
Jul 09 05:45:29 alice-shouldi-contribute-0 dockerd[27732]: time="2022-07-09T05:45:29.575936377Z" level=info msg="API listen on /run/docker.sock"

Add the non-root user to the docker group

# usermod -aG docker pdxjohnny

Now leave the root session.

Configure git by copying over creds to new user, run the following
on your local machine.

$ (cd ~ && tar -c .gitconfig .config/gh | ssh -i ~/.ssh/nahdig -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no pdxjohnny@143.198.133.87 tar -xv)

Then log into the VM via ssh as the new user.

$ ssh -i ~/.ssh/nahdig -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no pdxjohnny@143.198.133.87

Install dotfiles.

References:

$ git config --global user.name "John Andersen"
$ git config --global user.email johnandersenpdx@gmail.com
$ git clone https://github.com/pdxjohnny/dotfiles ~/.dotfiles
$ cd ~/.dotfiles
$ ./install.sh
$ echo 'source "${HOME}/.pdxjohnnyrc"' | tee -a ~/.bashrc
$ dotfiles_branch=$(hostname)-$(date "+%4Y-%m-%d-%H-%M")
$ git checkout -b $dotfiles_branch
$ sed -i "s/Dot Files/Dot Files: $dotfiles_branch/g" README.md
$ git commit -sam "Initial auto-tailor for $(hostname)"
$ git push --set-upstream origin $dotfiles_branch

Close out the SSH session, SSH back into the host recording
via asciinema's stored locally. Pass -t to ssh to allocate
a pty for full interactive functionally when running a command
on ssh (tmux after the hostname).

$ python -m asciinema rec --idle-time-limit 0.5 --title "$(date +%4Y-%m-%d-%H-%M-%ss)" --command "ssh -t -i ~/.ssh/nahdig -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no pdxjohnny@143.198.133.87 tmux" >(xz --stdout - > "$HOME/asciinema/rec-$(hostname)-$(date +%4Y-%m-%d-%H-%M-%ss).json.xz")

Update Python packaging core packages

$ python3 -m pip install -U pip setuptools wheel

Install docker-compose

$ python3 -m pip install docker-compose

Add the pip installed scripts to the PATH

References:

Find the location of the installed script directory and
replace the homedir literal with the HOME variable via sed.
Add the addition to the PATH to the .bashrc file or
.bash_profile on some distros. Restart the shell to pick up
changes and allow us to run docker-compose.

$ python_bin=$(python3 -c 'import os,sysconfig;print(sysconfig.get_path("scripts",f"{os.name}_user"))' | sed -e "s#${HOME}#\${HOME}#g")
$ echo "export PATH=\"\$PATH:$python_bin\"" | tee -a ~/.bashrc
$ exec bash

We should now see the directory containing docker-compose
at the end of our PATH.

$ echo $PATH
/home/pdxjohnny/.local/bin:/home/pdxjohnny/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/pdxjohnny/.bin:/home/pdxjohnny/.bin:/home/pdxjohnny/.local/bin:/home/pdxjohnny/.local/bin
  • Successful run of tutorial: https://frankhinek.com/getting-started-with-tbds-ssi-service/
  • https://github.com/TBD54566975/ssi-service

    • The Self Sovereign Identity Service (SSIS) facilitates all things relating to DIDs and Verifiable Credentials -- in a box! The service is a part of a larger Decentralized Web Platform architecture which you can learn more about in our collaboration repo. The SSI Service is a RESTful web service that wraps the ssi-sdk. The core functionality of the SSIS includes, but is not limited to: interacting with the standards around Verifiable Credentials, Credential Revocations, requesting Credentials, exchanging Credentials, data schemas for Credentials and other verifiable data, messaging using Decentralized Web Nodes, and usage of Decentralized Identifiers. Using these core standards, the SSIS enables robust functionality to facilitate all verifiable interactions such as creating, signing, issuing, curating, requesting, revoking, exchanging, validating, verifying credentials in varying degrees of complexity.

SSI Service high level architecture from their docs

Collector Result Storage

Export the token as a variable within the server tmux shell session.

$ export DIGITALOCEAN_ACCESS_TOKEN=asdjfojdf9j82efknm9dsfjsdf

Install Python library for interfacing with DigitalOcean

References:

$ pip install -U python-digitalocean

Install Python dependency to interact with Spaces API.

$ pip install boto3

Go to https://cloud.digitalocean.com/account/api/tokens.
Create the Space key with the name reflecting the scope
pdxjohnny.results.contribute.shouldi.alice.nahdig.com.
Export the key and secret as variables so we can use them
later when we write the source and run the collector.

DO NOT use . in DigitalOcean names for Spaces or
tokens / space key names. It seems fine at first but breaks
under the hood when you go to use it.

$ export SPACES_KEY=sdfjfjasdofj0iew
$ export SPACES_SECRET=3j41ioj239012j3k12j3k12jlkj2

Write a Python script to attempt to query the space contents.

upload_static_file_contents_to_space.py

import os
import boto3

session = boto3.session.Session()
client = session.client(
    "s3",
    region_name="sfo3",
    endpoint_url="https://sfo3.digitaloceanspaces.com",
    aws_access_key_id=os.getenv("SPACES_KEY"),
    aws_secret_access_key=os.getenv("SPACES_SECRET"),
)

response = client.list_buckets()
spaces = [space["Name"] for space in response["Buckets"]]
print("Spaces List: %s" % spaces)

# Step 3: Call the put_object command and specify the file to upload.
client.put_object(
    Bucket="results-alice-shouldi-contribute",  # The path to the directory you want to upload the object to, starting with your Space name.
    Key="collector.txt",  # Object key, referenced whenever you want to access this file later.
    Body=b"{SDflkasdj}",  # The object's contents.
    ACL="public-read",  # Defines Access-control List (ACL) permissions, such as private or public.
)

Run the upload.

$ python3 upload_static_file_contents_to_space.py
Spaces List: ['results-alice-shouldi-contribute']

Boto3 Source

upload_static_file_contents_to_space_asyncio.py

import os
import asyncio

import aioboto3


async def main():
    session = aioboto3.Session()
    async with session.client(
        "s3",
        region_name="sfo3",
        endpoint_url="https://sfo3.digitaloceanspaces.com",
        aws_access_key_id=os.getenv("SPACES_KEY"),
        aws_secret_access_key=os.getenv("SPACES_SECRET"),
    ) as client:
        # Grab the list of buckets
        response = await client.list_buckets()
        spaces = [space["Name"] for space in response["Buckets"]]
        print("Spaces List: %s" % spaces)
        # Call the put_object command and specify the file to upload.
        await client.put_object(
            Bucket="results-alice-shouldi-contribute",  # The path to the directory you want to upload the object to, starting with your Space name.
            Key="collector.txt",  # Object key, referenced whenever you want to access this file later.
            Body=b"{SDflkasdj}",  # The object's contents.
            ACL="public-read",  # Defines Access-control List (ACL) permissions, such as private or public.
        )

if __name__ == "__main__":
    asyncio.run(main())

See if it works, it does!

$ python3 upload_static_file_contents_to_space_asyncio.py
Spaces List: ['results-alice-shouldi-contribute']

We now begin creating a source based off MemorySource to save
record data as a JSON file.

aioboto3_dffml_source.py

"""
Source for storing and retrieving data from S3
"""
import os
import string
import asyncio
from typing import Dict, List, AsyncIterator

import aioboto3

from dffml import (
    config,
    field,
    Record,
    BaseSourceContext,
    BaseSource,
    entrypoint,
    export,
)


class Boto3SourceContext(BaseSourceContext):
    async def update(self, record):
        await self.parent.client.put_object(
            Bucket=self.parent.config.bucket,
            Key="".join(
                [
                    character
                    for character in record.key.lower()
                    if character in string.ascii_lowercase
                ]
            )
            + ".json",
            Body=json.dumps(export(record)),
            ACL=self.parent.config.acl,
        )

    async def records(self) -> AsyncIterator[Record]:
        pass

    async def record(self, key: str) -> Record:
        return Record(key)


@config
class Boto3SourceConfig:
    """
    References:
    - https://aioboto3.readthedocs.io/en/latest/usage.html
    """
    region_name: str
    endpoint_url: str
    aws_access_key_id: str
    aws_secret_access_key: str
    bucket: str
    acl: str = field(
        "Permissions level required for others to access. Options: private|public-read",
        default="private",
    )


@entrypoint("boto3")
class Boto3Source(BaseSource):
    """
    Uploads a record to S3 style storage
    """

    CONFIG = Boto3SourceConfig
    CONTEXT = Boto3SourceContext

    async def __aenter__(self) -> "Boto3Source":
        await super().__aenter__(config)
        self.session = aioboto3.Session()
        self.client = await self.session.client(
            "s3",
            region_name=self.config.region_name,
            endpoint_url=self.config.endpoint_url,
            aws_access_key_id=self.config.aws_access_key_id,
            aws_secret_access_key=self.config.aws_secret_access_key,
        ).__aenter__()
        return self

    async def __aexit__(self, _exc_type, _exc_value, _traceback) -> None:
        await self.client.__aexit__(None, None, None)
        self.client = None
        self.session = None


import dffml.noasync

dffml.noasync.save(
    Boto3Source(
        bucket="results-alice-shouldi-contribute",
        region_name="sfo3",
        endpoint_url="https://sfo3.digitaloceanspaces.com",
        aws_access_key_id=os.getenv("SPACES_KEY"),
        aws_secret_access_key=os.getenv("SPACES_SECRET"),
        acl="public-read",
    ),
    Record(
        key="https://github.com/pdxjohnny/httptest",
        features={
            "hello": "world",
        },
    ),
)

overlays/alice/shouldi/contribute/upload_collector_output_to_bucket.py

import os
import json
import string
import asyncio
import contextlib

import aioboto3
import aiobotocore.client

import dffml


AioBoto3Client = NewType("AioBoto3Client", aiobotocore.client.AioBaseClient)
AioBoto3RegionName = NewType("AioBoto3RegionName", str)
AioBoto3EndpointUrl = NewType("AioBoto3EndpointUrl", str)
AioBoto3AWSKeyId = NewType("AioBoto3AWSKeyId", str)
AioBoto3AWSAccessKey = NewType("AioBoto3AWSAccessKey", str)
AioBoto3AWSACL = NewType("AioBoto3AWSACL", str)
AioBoto3Bucket = NewType("AioBoto3Bucket", str)


MINIOServerShouldStart = NewType("MINIOServerShouldStart", bool)


@contextlib.asynccontextmanager
async def minio_server(
    should_start: MINIOServerShouldStart,
) -> AioBoto3EndpointUrl:
    # Bail out if not wanted, effectively auto start if wanted. Inclusion of this
    # operation within an overlay with the current overlay mechanisms at load
    # time happening in dffml_operations_innersource.cli and alice.cli for
    # shouldi and please contribute results in the operation getting combined
    # with the rest prior to first call to DataFlow.auto_flow.
    if not should_start:
        return
    with tempfile.TemporaryDirectory() as tempdir:
        # TODO Audit does this kill the container successfully aka clean it up
        # TODO We have no logger, can we pull from stack if we are in
        # MemoryOrchestrator?
        async for event, result in dffml.run_command_events(
            [
                "docker",
                "run",
                "quay.io/minio/minio",
                "server",
                "/data",
                "--console-address",
                ":9001",
            ],
            events=[
                dffml.Subprocess.STDOUT_READLINE,
                dffml.Subprocess.STDERR_READLINE,
            ],
        ):
            if (
                event is dffml.Subprocess.STDOUT_READLINE
                and result.startswith("API:")
            ):
                # API: http://172.17.0.2:9000  http://127.0.0.1:9000
                yield result.split()[1]


# **TODO** We have parsers for numpy style docstrings to config classes which
# can help us what was previously help field() argument.
@contextlib.asynccontextmanager
async def bucket_client_connect(
    endpoint_url: AioBoto3EndpointUrl,
    region_name: AioBoto3RegionName = None,
    aws_access_key_id: AioBoto3AWSKeyId = None,
    aws_secret_access_key: AioBoto3AWSAccessKey = None,
    acl: AioBoto3AWSACL = "private",
) -> AioBoto3Client:
    """
    Connect to an S3 bucket.

    References:
    - https://aioboto3.readthedocs.io/en/latest/usage.html

    This is the short description.

    This is the longer description.

    Parameters
    ----------
    acl : str
        Permissions level required for others to access. Options: private|public-read

    Returns
    -------
    str_super_cool_arg : AioBoto3Client
        The aiobotocore.client.AioBaseClient object

    Examples
    --------

    >>> async with connect_bucket_client(
    ...     region_name: str,
    ...     endpoint_url: str,
    ...     aws_access_key_id: str,
    ...     aws_secret_access_key: str,
    ...     acl: str = "private",
    ...     ,
    ... ) as client:
    ...
    """
    session = aioboto3.Session()
    async with session.client(
        "s3",
        region_name=config.region_name,
        endpoint_url=config.endpoint_url,
        aws_access_key_id=config.aws_access_key_id,
        aws_secret_access_key=config.aws_secret_access_key,
    ) as client:
        # Grab the list of buckets
        response = await client.list_buckets()
        buckets = [bucket["Name"] for bucket in response["Buckets"]]
        print("Buckets List: %s" % buckets)
        # Client initialization complete
        yield client


"""
# Old style runs into the issue where how do we provide the
# config server URL dynamically? So we experimented with this
# operation based approach with objects as inputs.
@dffml.op(
    inputs={"results": dffml.group_by_output},
    stage=dffml.Stage.OUTPUT,
    imp_enter={
        "client": (lambda self: aiohttp.ClientSession(trust_env=True))
    },
)
"""
async def upload_to_bucket(
    client: AioBoto3Client,
    bucket: AioBoto3Bucket,
    repo_url: dffml_feature_git.feature.defintions.URL,
    results: dffml.group_by_output,
) -> None:
    await client.put_object(
        Bucket=bucket,
        # TODO(security) Ensure we don't have collisions
        # with two different repo URLs generating the same
        # filename, pretty sure the bellow code has that
        # as an active issue!!!
        Key="".join(
            [
                character
                for character in repo_url.lower()
                if character in string.ascii_lowercase
            ]
        )
        + ".json",
        Body=json.dumps(export(results)),
        ACL=acl,
    )

Playing With Aysnc Context Managers as Data Flows

  • Metadata
    • Date: 2022-07-09 12:48 UTC -7
  • We will take our last example and attempt to modify @op
    so that when it sees something decorated with
    @contextlib.asynccontextmanager it creates an operation
    which calls the first operation to yield then resumes
    from the yield in a cleanup operation. We need to end up
    with two operations, one that's a cleanup and one that's
    the stage matching it's inputs within the auto flow.
    • This means that DataFlow also needs to be modified
      to unpack lists of operations if it receives a list
      of operations in place of where it would usually take
      a single instance of an operation. This should be
      *args in DataFlow.__init__ as well as operations
      and implementations within the keyword arguments.
      • May need to check DataFlow._fromdict() and
        DataFlow.export() as well.

@johnandersen777
Copy link
Author

johnandersen777 commented Jul 18, 2022 •
edited
Loading

Moved to #1406

@johnandersen777
Copy link
Author

Refactoring and Thinking About Locking of Repos for Contributions

  • Metadata
    • Date: 2022-07-27 13:00 UTC -7

@johnandersen777 johnandersen777 mentioned this pull request Nov 2, 2022
Open
20 tasks
johnandersen777 referenced this pull request Nov 8, 2022
@pdxjohnny
docs: tutorials: rolling alice: Offer the red pill
291cfbe
@johnandersen777 johnandersen777 mentioned this pull request Jan 28, 2023
Closed
@johnandersen777 johnandersen777 added the Alice Work related to our tutorial series "Rolling Alice" label Feb 9, 2023
johnandersen777 pushed a commit that referenced this pull request Feb 15, 2023
@pdxjohnny
schema: github: actions: result: container: Example of 1st party pull…
e900494 
… request validation manifest

Related: 1f347bc
Related: #1401
Related: #1207
Related: #1061
Alice Engineering Comms 2023-02-15 @pdxjohnny Engineering Logs: #1406
johnandersen777 pushed a commit that referenced this pull request May 4, 2023
@pdxjohnny
dffml: plugins: Create JSON with with only dffml-model-transformers
b084cc9 
Related: #1061
Related: #1401
johnandersen777 pushed a commit that referenced this pull request May 4, 2023
@pdxjohnny
dffml: plugins: dffml-model-transformers: Set branch to main
d3e0ea5 
Related: #1061
Related: #1401
Related: #1113
johnandersen777 pushed a commit that referenced this pull request May 4, 2023
@pdxjohnny
ci: pin: downstream: 2nd party: Attempt pinning
33a8531 
Related: #1061
Related: #1401
Related: #1113
@johnandersen777 johnandersen777 mentioned this pull request May 4, 2023
Merged
pdxjohnny and others added 28 commits December 4, 2023 18:05
@pdxjohnny
operations: innersource: npm-groovy-lint: Log cmd to be run
3d3ac02 
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
operations: innersource: npm-groovy-lint: Resolve path to install to …
89c81f0 
…run CodeNarc if not pathlib object

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
operations: innersource: has docs: Replace unicode errors with escape…
d030951 
…d characters on parse error

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
docs: tutorials: rolling alice: preface: strategic principles: reward…
5f1867c 
… trains of thought which align with strategic principles: Modify wording

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
feature: git: operations: cleanup git repo: If DFFML_FEATURE_GIT_SKIP…
144b0bd 
…_CLEANUP is set do not remove repo directory

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
operations: innersource: cli: local repo resolver: Enable skipping cl…
bf8048f 
…one and using local repo

.. code-block:: console

    $ alice shouldi contribute -log debug \
        -keys local \
        -inputs \
          https://github.com/intel/dffml=LocalRepoURL \
          $PWD=LocalRepoDirectory

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
df: base: Remove duplicate import of APPLY_INSTALLED_OVERLAYS
3caf15a 
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
operations: innersource: collector: Support lookup of repository clon…
d77a9c5 
…e URL from local directory with Git repo

.. code-block:: console

    $ DFFML_FEATURE_GIT_SKIP_CLEANUP=1 alice -log debug shouldi contribute -record-def LocalRepoDirectory -keys .

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
operations: innersource: cli: local repo resolver: Only replace .git …
7fe1297 
…with nothing if found at end of repo URL

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
operations: innersource: action yml files: Remove YAML files that are…
ccf36d5 
… not GitHub Actions by checking for runs: Keyword

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
service: activitypub-webhook-relay: Initial commit
7842b6f 
Related: #1315
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
service: activitypub-webhook-relay: Fix package install by fixing ver…
9264914 
…sion to not use setuptools_scm while in monorepo

Related: https://github.com/pypa/setuptools_scm/blob/e9cbb5a68b3ae6d5c549bda293ef60bb5ec8ec7e/src/setuptools_scm/_integration/pyproject_reading.py#L68-L73
Related: #1315
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
docs: tutorials: rolling alice: architecting alice: entering wonderla…
cae455c 
…nd: Isolated dynamic analysis
@pdxjohnny
operations: innersource: action.yml files: Refactor to match groovy f…
76a2aaf 
…ile identification

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny @agalvare
docs: tutorials: rolling alice: architecting alice: stream of conscio…
52ffc9b 
…usness: pseudo code: From Claude AI via Alfredo

Co-authored-by: Alfredo Alvarez <alfredo.g.alvarez@intel.com>
@pdxjohnny
alice: please: contribute: util: gh: Fix TODO to filter by logged in …
b765398 
…user when creating or updating issue by title

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
alice: please: contribute: util: gh: Remove duplicate issue creation …
35158e3 
…when searching only for issues created by logged in user

Introduced-in: 0ea349b
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
docs: discussions: Alice Engineering Comms: 2023-09-05T19:02:19+00:00
bbff686 
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
docs: tutorials: rolling alice: architecting alice: stream of conscio…
9f29538 
…usness: Remove references to Heartwood link to SCITT ActivityPub pull request

Related: scitt-community/scitt-api-emulator#37
@pdxjohnny
Update reply_0000.md
2ff4633
@pdxjohnny
docs: arch: A GitHub Public Key and TPM Based Supply Chain Security M…
640124a 
…itigation Option: Mention SCITT notary CWT issuer as ssh keys endpoint

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
docs: discussions: Alice Engineering Comms: 2023-11-13T18:50:46+00:00
3fbf9fd 
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
docs: tutorials: rolling alice: Update cordination link for pull requ…
84626ad 
…est to discussion thread and provide high level plan

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
docs: discussions: Alice Engineering Comms: 2023-11-21T22:52:21+00:00
5aeb0b2 
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
feature: git: aiohttp ls-remote
ae65c2f 
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
feature: git: aiohttp ls-remotes
ede8e70 
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
source: df: Ensure default_factory is used for mutable defaults (Pyth…
72f21c8 
…on 3.12 support)

Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@pdxjohnny
ci: tests: Remove schedule
ec7567f 
Signed-off-by: John Andersen <johnandersenpdx@gmail.com>
@johnandersen777 johnandersen777 merged commit 4e6ce6c into main Dec 5, 2023
3 of 4 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
Alice Work related to our tutorial series "Rolling Alice"
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@johnandersen777 @pdxjohnny @pbhutori