List of tools

[ShutdownRepo]

We need to document the tools installed in exegol. In my opinion, it should end up in the Exegol images section.

[ShutdownRepo]

Just got an idea. We could make install.sh create and update a tools list. And for all nightly and tag releases, the list would be fetched by a exegol-docs pipeline 🤷

I'd like to avoid having to maintain a list of hundreds of tools manually.

Why not also find a way to have the following values for each tool added to the list:

• tags (ad, web, osint, recon, kerberos, python, go, etc.)
• URL (link to the projects repo)
• comments, what the tool does

[thibaudrobin]

Main idea

add-to-list name "crackmapexec"
add-to-list tags "active directory, 2, 3"
add-to-list link "https://blablabla"
add-to-list desc "blablablabl blabla"

add-to-list function :

• Format each data tool to JSON or CSV
• Add all data in a file

[ShutdownRepo]

Data will be produced in the image and will then probably be extracted by a pipeline (either on exegol-images or exegol-docs) and be pushed to the exegol-docs repo. I'd like to publish a list of tools for each version of each image variant.

[ShutdownRepo]

Let's move forward with the following pseudo structure for now:

function install_whatevertool() {
  # install commands
  # aliases, history, test commands
  add-to-list "toolname,https://linkToTool/"
}

function add-to-list() {
  echo $1 >> "/.exegol/installed_tools"
}

The installed_tools will be in a CSV format. Exegol-docs will then build the tools list from it and will probably get some info dynamically from the links given.

[thibaudrobin]

Done : ThePorgs/Exegol-images#125

[ShutdownRepo]

We now need to code the parser on the doc side.
Parser will need to build a table of all installed tools for all Exegol images and versions, by extracting (and parsing ofc) the file populated with ThePorgs/Exegol-images#125

[QU35T-code]

It may be necessary that during the build of an image (Exegol-images) a step be added in order to recover the list "/.exegol/installed_tools" and from there, parse then push them on the docs repository with the right format.

[QU35T-code]

Also, the changes should be a pull request and not directly pushed to main from the docs repo to avoid issues.

[ShutdownRepo]

Let's do something like this.

In my opinion, the best way to do this is to edit the pipeline of exegol-images. We can add a step between the publish and clean_runners jobs, in which the runner extracts the list of tools at /.exegol/installed_tools, makes sure the csv is valid(1), and pushes changes(2) to exegol-doc.

(1) valid: in the first iteration of this iteration, we can simply check the number of columns in the file (number or ,)
(2) changes: move existing "current" content in "archive", and update the "content" table

[ShutdownRepo]

I got an initial working setup

https://github.com/ThePorgs/Exegol-images/actions/runs/5618467445/job/15224175922

It produces what follows

It's a start. I now need to do some testing, but I'm pretty happy with the result, we're close to a v1 !

[Dramelac]

Regarding the nightly tag, i don't thin we need archive for each commit, it will be too much, regarding nightly we only keep the latest nightly version and archive for the stable releases only imo.

[ShutdownRepo]

Agreed. I pushed the new pipelines steps on production. We'll have to keep an eye on it on the next iterations of the pipelines.

[ShutdownRepo]

We're good to go. Pipelines are in prod.
For each nightly build, the old nightly are removed from the list and replace with the new ones (the files are kept in the repo).
For each pre-release image, the list is added. In case we run pre-release twice (because other images are failing and we can't proceed with release), there shouldn't be duplicates, old pre-release with the same arch and version will be replaced.
Nightly and pre-release lists are pushed on the dev-images branch, meaning we'll need to merge dev-images -> main a bit more often.
We now need to do a full review of the tools list that was AI generated, meaning there are many mistakes.
Also, pipelines should be working fine but we'll need to keep an eye out for bugs.

[ShutdownRepo]

Re-opening, need to improve the pipeline
As of now, the export method is placed after the publish method. When publish is not run because one of the builds was not successful, the export is blocked. When re-running later on, the image that was successfully built got removed from the runner, and the export tools cannot do its job.
What we can do: move the export method into the build belt and use the PR_MODE entry argument to decide whether the method is run (nightly and releases) or not (external PRs) (because of risk of token leaks).

[ShutdownRepo]

Should be alright now, closing