stdenv/linux: document some tips in debugging stdenv bootstrap tower #208478

trofi · 2022-12-31T10:38:14Z

Just a few comments added:

added a few one-liners to explore which tools are rebuilt at each stdenv iteration during bootstrap
explicitly listed available toolchains and their sources for on each
bootstrap step: glibc, binutils, gcc, coreutils.

Should help slightly in explaining and untangling #208412

Description of changes

Things done

tpwrules · 2022-12-31T17:54:08Z

There are some other utilities present in the final stdenv like xz and things rebuilt multiple times during the bootstrap like perl. Should those be addressed here?

trofi · 2022-12-31T18:15:35Z

There are some other utilities present in the final stdenv like xz and things rebuilt multiple times during the bootstrap like perl. Should those be addressed here?

Good question. Some of the secondary dependencies are volatile in nature: they come and go with patches and release updates. It's quite a big list (dumped it below).

Some dependencies are very important, like libidn2->glibc->libidn2 loop. Or a hack to clobber ld's INTERP right after glibc upgrade. Some feel less relevant, like xz, bzip (and maybe perl?). Some are somewhat important, like static linkage of libmpc into final gcc, but not very special WRT which stage builds it.

We can pick more interesting cases and document them as well.

So far I mainly followed toolchain-like packages: those that produce code (gcc, binutils), or those that provide code for embedding (glibc).

The whole bootstrap list:

$ for stage in 0 1 2 3 4; do echo "stage${stage} used in:"; nix-store --query --graph $(nix-instantiate -A stdenv) | grep -P -v '[.]sh|[.]patch|bash|[.]tar' | grep -P ".*bootstrap-stage${stage}-stdenv.*->.*" | sed 's/"[0-9a-z]\{32\}-/"/g'; done


stage0 used in:
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
"bootstrap-stage0-stdenv-linux.drv" -> "bootstrap-stage2-gcc-wrapper-.drv" [color = "red"];
"bootstrap-stage0-stdenv-linux.drv" -> "bootstrap-stage0-glibc-bootstrap.drv" [color = "magenta"];
"bootstrap-stage0-stdenv-linux.drv" -> "bootstrap-stage1-gcc-wrapper-.drv" [color = "blue"];
"bootstrap-stage0-stdenv-linux.drv" -> "bootstrap-stage0-binutils-wrapper-.drv" [color = "red"];
"bootstrap-stage0-stdenv-linux.drv" -> "bootstrap-stage3-gcc-wrapper-.drv" [color = "black"];
"bootstrap-stage0-stdenv-linux.drv" -> "bootstrap-stage4-gcc-wrapper-13.0.0.drv" [color = "burlywood"];
stage1 used in:
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
"bootstrap-stage1-stdenv-linux.drv" -> "gnum4-1.4.19.drv" [color = "red"];
"bootstrap-stage1-stdenv-linux.drv" -> "bison-3.8.2.drv" [color = "black"];
"bootstrap-stage1-stdenv-linux.drv" -> "binutils-wrapper-2.39.drv" [color = "black"];
"bootstrap-stage1-stdenv-linux.drv" -> "expand-response-params.drv" [color = "red"];
"bootstrap-stage1-stdenv-linux.drv" -> "binutils-2.39.drv" [color = "red"];
"bootstrap-stage1-stdenv-linux.drv" -> "gettext-0.21.drv" [color = "red"];
"bootstrap-stage1-stdenv-linux.drv" -> "texinfo-6.8.drv" [color = "green"];
"bootstrap-stage1-stdenv-linux.drv" -> "xz-5.4.0.drv" [color = "red"];
"bootstrap-stage1-stdenv-linux.drv" -> "binutils-wrapper-2.39.drv" [color = "green"];
"bootstrap-stage1-stdenv-linux.drv" -> "zlib-1.2.13.drv" [color = "burlywood"];
"bootstrap-stage1-stdenv-linux.drv" -> "perl-5.36.0.drv" [color = "burlywood"];
stage2 used in:
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
"bootstrap-stage2-stdenv-linux.drv" -> "linux-headers-6.1.drv" [color = "black"];
"bootstrap-stage2-stdenv-linux.drv" -> "libidn2-2.3.2.drv" [color = "green"];
"bootstrap-stage2-stdenv-linux.drv" -> "libunistring-1.0.drv" [color = "red"];
"bootstrap-stage2-stdenv-linux.drv" -> "bootstrap-stage0-glibc-iconv-bootstrap.drv" [color = "blue"];
"bootstrap-stage2-stdenv-linux.drv" -> "binutils-2.39.drv" [color = "burlywood"];
"bootstrap-stage2-stdenv-linux.drv" -> "expand-response-params.drv" [color = "burlywood"];
"bootstrap-stage2-stdenv-linux.drv" -> "patchelf-0.15.0.drv" [color = "black"];
"bootstrap-stage2-stdenv-linux.drv" -> "glibc-2.35-224.drv" [color = "burlywood"];
"bootstrap-stage2-stdenv-linux.drv" -> "python3-minimal-3.10.8.drv" [color = "magenta"];
"bootstrap-stage2-stdenv-linux.drv" -> "autoconf-archive-2022.09.03.drv" [color = "burlywood"];
"bootstrap-stage2-stdenv-linux.drv" -> "xz-5.4.0.drv" [color = "green"];
"bootstrap-stage2-stdenv-linux.drv" -> "bzip2-1.0.8.drv" [color = "red"];
"bootstrap-stage2-stdenv-linux.drv" -> "hook.drv" [color = "green"];
"bootstrap-stage2-stdenv-linux.drv" -> "libtool-2.4.7.drv" [color = "burlywood"];
"bootstrap-stage2-stdenv-linux.drv" -> "autoconf-2.71.drv" [color = "blue"];
"bootstrap-stage2-stdenv-linux.drv" -> "automake-1.16.5.drv" [color = "black"];
"bootstrap-stage2-stdenv-linux.drv" -> "zlib-1.2.13.drv" [color = "magenta"];
"bootstrap-stage2-stdenv-linux.drv" -> "gettext-0.21.drv" [color = "black"];
"bootstrap-stage2-stdenv-linux.drv" -> "pkg-config-wrapper-0.29.2.drv" [color = "red"];
"bootstrap-stage2-stdenv-linux.drv" -> "expat-2.5.0.drv" [color = "red"];
"bootstrap-stage2-stdenv-linux.drv" -> "pkg-config-0.29.2.drv" [color = "black"];
"bootstrap-stage2-stdenv-linux.drv" -> "libxcrypt-4.4.33.drv" [color = "magenta"];
"bootstrap-stage2-stdenv-linux.drv" -> "texinfo-6.8.drv" [color = "black"];
"bootstrap-stage2-stdenv-linux.drv" -> "help2man-1.49.2.drv" [color = "blue"];
"bootstrap-stage2-stdenv-linux.drv" -> "perl5.36.0-gettext-1.07.drv" [color = "green"];
"bootstrap-stage2-stdenv-linux.drv" -> "perl-5.36.0.drv" [color = "red"];
"bootstrap-stage2-stdenv-linux.drv" -> "file-5.43.drv" [color = "black"];
"bootstrap-stage2-stdenv-linux.drv" -> "nuke-references.drv" [color = "red"];
"bootstrap-stage2-stdenv-linux.drv" -> "libffi-3.4.4.drv" [color = "black"];
stage3 used in:
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
"bootstrap-stage3-stdenv-linux.drv" -> "gettext-0.21.drv" [color = "burlywood"];
"bootstrap-stage3-stdenv-linux.drv" -> "texinfo-6.8.drv" [color = "black"];
"bootstrap-stage3-stdenv-linux.drv" -> "zlib-1.2.13.drv" [color = "green"];
"bootstrap-stage3-stdenv-linux.drv" -> "gcc-13.0.0.drv" [color = "red"];
"bootstrap-stage3-stdenv-linux.drv" -> "mpfr-stage3-4.1.1.drv" [color = "burlywood"];
"bootstrap-stage3-stdenv-linux.drv" -> "libmpc-stage3-1.2.1.drv" [color = "magenta"];
"bootstrap-stage3-stdenv-linux.drv" -> "mpfr-4.1.1.drv" [color = "blue"];
"bootstrap-stage3-stdenv-linux.drv" -> "expand-response-params.drv" [color = "green"];
"bootstrap-stage3-stdenv-linux.drv" -> "xz-5.4.0.drv" [color = "green"];
"bootstrap-stage3-stdenv-linux.drv" -> "flex-2.6.4.drv" [color = "blue"];
"bootstrap-stage3-stdenv-linux.drv" -> "help2man-1.49.2.drv" [color = "red"];
"bootstrap-stage3-stdenv-linux.drv" -> "perl5.36.0-gettext-1.07.drv" [color = "black"];
"bootstrap-stage3-stdenv-linux.drv" -> "perl-5.36.0.drv" [color = "red"];
"bootstrap-stage3-stdenv-linux.drv" -> "gmp-with-cxx-6.2.1.drv" [color = "burlywood"];
"bootstrap-stage3-stdenv-linux.drv" -> "which-2.21.drv" [color = "green"];
"bootstrap-stage3-stdenv-linux.drv" -> "libxcrypt-4.4.33.drv" [color = "green"];
"bootstrap-stage3-stdenv-linux.drv" -> "isl-stage3-0.20.drv" [color = "burlywood"];
"bootstrap-stage3-stdenv-linux.drv" -> "gmp-with-cxx-stage3-6.2.1.drv" [color = "black"];
"bootstrap-stage3-stdenv-linux.drv" -> "hook.drv" [color = "green"];
"bootstrap-stage3-stdenv-linux.drv" -> "libtool-2.4.7.drv" [color = "blue"];
"bootstrap-stage3-stdenv-linux.drv" -> "autoconf-2.71.drv" [color = "blue"];
"bootstrap-stage3-stdenv-linux.drv" -> "automake-1.16.5.drv" [color = "burlywood"];
"bootstrap-stage3-stdenv-linux.drv" -> "file-5.43.drv" [color = "green"];
stage4 used in:
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
"bootstrap-stage4-stdenv-linux.drv" -> "attr-2.5.1.drv" [color = "burlywood"];
"bootstrap-stage4-stdenv-linux.drv" -> "gmp-with-cxx-stage4-6.2.1.drv" [color = "green"];
"bootstrap-stage4-stdenv-linux.drv" -> "diffutils-3.8.drv" [color = "blue"];
"bootstrap-stage4-stdenv-linux.drv" -> "bzip2-1.0.8.drv" [color = "green"];
"bootstrap-stage4-stdenv-linux.drv" -> "hook.drv" [color = "magenta"];
"bootstrap-stage4-stdenv-linux.drv" -> "automake-1.16.5.drv" [color = "burlywood"];
"bootstrap-stage4-stdenv-linux.drv" -> "autoconf-2.71.drv" [color = "red"];
"bootstrap-stage4-stdenv-linux.drv" -> "libtool-2.4.7.drv" [color = "burlywood"];
"bootstrap-stage4-stdenv-linux.drv" -> "patchelf-0.15.0.drv" [color = "green"];
"bootstrap-stage4-stdenv-linux.drv" -> "gnused-4.9.drv" [color = "black"];
"bootstrap-stage4-stdenv-linux.drv" -> "gnugrep-3.7.drv" [color = "black"];
"bootstrap-stage4-stdenv-linux.drv" -> "gnutar-1.34.drv" [color = "red"];
"bootstrap-stage4-stdenv-linux.drv" -> "help2man-1.49.2.drv" [color = "blue"];
"bootstrap-stage4-stdenv-linux.drv" -> "gcc-wrapper-13.0.0.drv" [color = "red"];
"bootstrap-stage4-stdenv-linux.drv" -> "xz-5.4.0.drv" [color = "black"];
"bootstrap-stage4-stdenv-linux.drv" -> "coreutils-9.1.drv" [color = "blue"];
"bootstrap-stage4-stdenv-linux.drv" -> "perl5.36.0-gettext-1.07.drv" [color = "burlywood"];
"bootstrap-stage4-stdenv-linux.drv" -> "gawk-5.2.1.drv" [color = "burlywood"];
"bootstrap-stage4-stdenv-linux.drv" -> "perl-5.36.0.drv" [color = "blue"];
"bootstrap-stage4-stdenv-linux.drv" -> "patch-2.7.6.drv" [color = "burlywood"];
"bootstrap-stage4-stdenv-linux.drv" -> "binutils-wrapper-2.39.drv" [color = "burlywood"];
"bootstrap-stage4-stdenv-linux.drv" -> "gzip-1.12.drv" [color = "magenta"];
"bootstrap-stage4-stdenv-linux.drv" -> "gnumake-4.4.drv" [color = "blue"];
"bootstrap-stage4-stdenv-linux.drv" -> "file-5.43.drv" [color = "green"];
"bootstrap-stage4-stdenv-linux.drv" -> "setup-hook.drv" [color = "green"];
"bootstrap-stage4-stdenv-linux.drv" -> "ed-1.18.drv" [color = "black"];
"bootstrap-stage4-stdenv-linux.drv" -> "lzip-1.23.drv" [color = "burlywood"];
"bootstrap-stage4-stdenv-linux.drv" -> "glibc-iconv-2.35.drv" [color = "green"];
"bootstrap-stage4-stdenv-linux.drv" -> "acl-2.3.1.drv" [color = "blue"];
"bootstrap-stage4-stdenv-linux.drv" -> "binutils-2.39.drv" [color = "red"];
"bootstrap-stage4-stdenv-linux.drv" -> "findutils-4.9.0.drv" [color = "burlywood"];
"bootstrap-stage4-stdenv-linux.drv" -> "libxcrypt-4.4.33.drv" [color = "magenta"];
"bootstrap-stage4-stdenv-linux.drv" -> "pcre-8.45.drv" [color = "green"];

tpwrules · 2022-12-31T20:23:11Z

Looking at this again, I'd say you do actually address the other utilities implicitly in the new section on bootstrap goals. I just wanted to be sure that we knew where they should come from. The goals sound good to me. Maybe add some justification like "These goals ensure the final stdenv has been built exclusively by packages available in the final stdenv so that compiler versions and environments etc. are consistent". That at least is how I understand the goals.

Is there any mechanical way to check that those those goals are met? I think the allowedRequisites and disallowedRequisites will ensure point 1 is not violated by accident (though we currently violate it deliberately using cp; checking hashes should catch that). I'm not sure if there's a way to do point 2 except by manual inspection, and I'm not immediately sure what to look for.

Can you put the command to generate that list in the comments too? That looks very helpful.

trofi · 2022-12-31T23:27:37Z

Looking at this again, I'd say you do actually address the other utilities implicitly in the new section on bootstrap goals. I just wanted to be sure that we knew where they should come from. The goals sound good to me. Maybe add some justification like "These goals ensure the final stdenv has been built exclusively by packages available in the final stdenv so that compiler versions and environments etc. are consistent". That at least is how I understand the goals.

I think I get what you say. I added the following blurb:

# These goals ensure that final packages and final stdenv are built
# exclusively using nixpkgs package definitions and don't depend
# on bootstrapTools (via direct references or via inclusion
# of copied code of generated code by bootstrapTools).

I think we are on the same page, but just in case I'll try to explain myself a bit.

I did not include your definition as is because (AFAIU) there is a fundamental problem of bootstrap process that has to be solved by pulling package definitions from non-final stdenv:

most of pkgs depends on stdenv (by literally depending on things like gcc from stdenv)
stdenv depends on pkgs (version of gcc from previous stage of stdenv)

To untie this infinitely unfolding cycle final stdenv stage overrides some of normal package definitions to refer to package definitions of previous stdenv. This make latest stdenv to use non-vanilla package definitions.

The final result is morally equivalent to your phrasing: "final stdenv has been built exclusively by packages available in the final stdenv". But mechanically we do it the other way around: we declare that previous (stage4 instance) is good enough and untied enough from bootstrapTools to declare it final and copy some of it's reference into final stdenv as is. While in reality it's built from various stdenv iterations. We expect them to be the same if we add more rebuild iterations, but we can't easily ensure it's always true.

Thus the more vague wording around the goals.

I hope I did not make things worse with this brain dump.

Is there any mechanical way to check that those those goals are met? I think the allowedRequisites and disallowedRequisites will ensure point 1 is not violated by accident (though we currently violate it deliberately using cp; checking hashes should catch that). I'm not sure if there's a way to do point 2 except by manual inspection, and I'm not immediately sure what to look for.

I don't think there is a good way to detect even our current libgcc_s.so hack :( It does not have a hash within that refers to a /nix/store path. The only hints are stale RUNPATH entries to nuked references and old compiler ID in .notes sections. Both would be absent if we were a bit more eager removing these optional bits when we build bootstrapTools seed tarball.

cp is hard to detect reliably in general: for contentAddressed builds it's natural when produced binaries are identical to binaries produced on previous steps.

Can you put the command to generate that list in the comments too? That looks very helpful.

Sounds good. Added.

tpwrules · 2023-01-01T19:36:14Z

Rephrasing the blurb slightly and fixing the typo:

# These goals ensure that final packages and final stdenv are built
# exclusively using nixpkgs package definitions and don't depend
# on bootstrapTools (via direct references, inclusion
# of copied code, or code compiled directly by bootstrapTools).

I think this PR is good other than some other minor typos.

I don't think there is a good way to detect even our current libgcc_s.so hack :( It does not have a hash within that refers to a /nix/store path. The only hints are stale RUNPATH entries to nuked references and old compiler ID in .notes sections. Both would be absent if we were a bit more eager removing these optional bits when we build bootstrapTools seed tarball.

I did not actually check, but I guess it is not a given that the file, especially since it's a binary, would make it through without being changed. That is unfortunate but oh well.

trofi · 2023-01-01T20:51:33Z

Rephrasing the blurb slightly and fixing the typo:

# These goals ensure that final packages and final stdenv are built
# exclusively using nixpkgs package definitions and don't depend
# on bootstrapTools (via direct references, inclusion
# of copied code, or code compiled directly by bootstrapTools).

Applied this wording.

I think this PR is good other than some other minor typos.

Thank you!

ghost

Hey, thanks for documenting all of this!

Except for lib{mpfr,mpc,isl,gmp}.a all of my review comments are just typos and minor stuff. But please do take a look at the comments about how those are handled so we don't mislead people into thinking they are built by the final gcc['s stage1 bootstrap compiler] -- they aren't!

pkgs/stdenv/linux/default.nix

K900 · 2023-01-02T08:26:43Z

Can we also get rid of the libmpfr hack if we add a second gcc rebuild? Seems like it, so it might actually be worth it.

ghost · 2023-01-02T08:28:26Z

Can we also get rid of the libmpfr hack if we add a second gcc rebuild? Seems like it, so it might actually be worth it.

The ideal solution is to make gcc's stage1 and stage2 be separate nix derivations. Then we don't have to build gcc twice (or rather we don't have to build it four times). I think @Ericson2314 was the first to suggest this approach.

trofi · 2023-01-02T08:49:17Z

Hey, thanks for documenting all of this!

Except for lib{mpfr,mpc,isl,gmp}.a all of my review comments are just typos and minor stuff. But please do take a look at the comments about how those are handled so we don't mislead people into thinking they are built by the final gcc['s stage1 bootstrap compiler] -- they aren't!

Thank you, Adam! I applied (and squashed) all your suggestions as is. Great point on gcc's static libraries! The are very important to cal out.

tpwrules · 2023-01-02T21:31:54Z

Something got mangled with the goals, 1 and 2 are almost the same now. I think 2 should be the one about being compiled by the bootstrap compiler?

Just a few comments added: - added a few one-liners to explore which tools are rebuilt at each stdenv iteration during bootstrap - explicitly listed available toolchains and their sources for on each bootstrap step: glibc, binutils, gcc, coreutils. - added mention of static libraries linked into gcc Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>

trofi · 2023-01-02T21:57:52Z

Something got mangled with the goals, 1 and 2 are almost the same now. I think 2 should be the one about being compiled by the bootstrap compiler?

Good catch! Re-numbered to start from 1. and reworded last goal in terms of code generators:

...
# 2. final stdenv must not contain any of the bootstrap files
#    (the only current violation is libgcc_s.so in glibc).
# 3. final stdenv must not contain any of the files directly
#    generated by the bootstrap code generators (assembler, linker,
#    compiler).

Ericson2314 · 2023-01-02T22:29:30Z

Thanks for doing this! I do want to make it saner per things like building GCC separately just as @amjoseph-nixpkgs said, but yes let that not be a reason not to document how things exist currently. I learned stuff reading this too!

…ments express #208478 as assertions

PR NixOS#208478 added a lot of documentation about which packages were rebuilt in each stage of the stdenv bootstrap. However nothing checks that these comments agree with reality; they can bitrot over time. This PR rewrites those comments as assertions, so they cannot bitrot. This conversion did expose some ambiguity in our scheme for naming the stages. Suppose that `pkgs.stdenv.name=="stdenv-stage4", then which of these is "the stage4 coreutils"? ``` pkgs.coreutils pkgs.stdenv.__bootPackages.coreutils ``` The choice is arbitrary, and both choices have confusing corner cases. We should revisit this at some point.

github-actions bot added the 6.topic: stdenv Standard environment label Dec 31, 2022

trofi requested review from tpwrules, Ericson2314, lovesegfault and K900 December 31, 2022 10:39

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Dec 31, 2022

trofi force-pushed the comment-stdenv-bootstrap branch from a5f8cb8 to 7b1b4a0 Compare December 31, 2022 11:13

trofi force-pushed the comment-stdenv-bootstrap branch 3 times, most recently from e33dc67 to cf4c9b8 Compare December 31, 2022 23:14

trofi force-pushed the comment-stdenv-bootstrap branch from cf4c9b8 to 5f06d36 Compare January 1, 2023 20:50

ghost approved these changes Jan 2, 2023

View reviewed changes

trofi force-pushed the comment-stdenv-bootstrap branch from bb12159 to f5376df Compare January 2, 2023 08:47

trofi force-pushed the comment-stdenv-bootstrap branch from f5376df to 062e14b Compare January 2, 2023 21:55

Ericson2314 merged commit a6b1de7 into NixOS:master Jan 2, 2023

trofi deleted the comment-stdenv-bootstrap branch January 3, 2023 10:35

trofi mentioned this pull request Jan 3, 2023

Bootstrap Reform and aarch64-linux's GCC Upgrade #208412

Closed

This was referenced Jan 7, 2023

stdenv/linux/default.nix: add gcc rebuild during bootstrap #209063

Closed

express #208478 as assertions #210019

Merged

trofi added a commit that referenced this pull request Feb 21, 2023

Merge pull request #210019 from amjoseph-nixpkgs/pr/stdenv/verify-com…

3057968

…ments express #208478 as assertions

loicreynier mentioned this pull request Mar 2, 2023

python3Packages.myst parser 0.19.1 #219173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stdenv/linux: document some tips in debugging stdenv bootstrap tower #208478

stdenv/linux: document some tips in debugging stdenv bootstrap tower #208478

trofi commented Dec 31, 2022 •

edited

Loading

tpwrules commented Dec 31, 2022

trofi commented Dec 31, 2022 •

edited

Loading

tpwrules commented Dec 31, 2022 •

edited

Loading

trofi commented Dec 31, 2022

tpwrules commented Jan 1, 2023

trofi commented Jan 1, 2023

ghost left a comment •

edited by ghost

Loading

K900 commented Jan 2, 2023

ghost commented Jan 2, 2023 •

edited by ghost

Loading

trofi commented Jan 2, 2023

tpwrules commented Jan 2, 2023

trofi commented Jan 2, 2023

Ericson2314 commented Jan 2, 2023

stdenv/linux: document some tips in debugging stdenv bootstrap tower #208478

stdenv/linux: document some tips in debugging stdenv bootstrap tower #208478

Conversation

trofi commented Dec 31, 2022 • edited Loading

Description of changes

Things done

tpwrules commented Dec 31, 2022

trofi commented Dec 31, 2022 • edited Loading

tpwrules commented Dec 31, 2022 • edited Loading

trofi commented Dec 31, 2022

tpwrules commented Jan 1, 2023

trofi commented Jan 1, 2023

ghost left a comment • edited by ghost Loading

Choose a reason for hiding this comment

K900 commented Jan 2, 2023

ghost commented Jan 2, 2023 • edited by ghost Loading

trofi commented Jan 2, 2023

tpwrules commented Jan 2, 2023

trofi commented Jan 2, 2023

Ericson2314 commented Jan 2, 2023

trofi commented Dec 31, 2022 •

edited

Loading

trofi commented Dec 31, 2022 •

edited

Loading

tpwrules commented Dec 31, 2022 •

edited

Loading

ghost left a comment •

edited by ghost

Loading

ghost commented Jan 2, 2023 •

edited by ghost

Loading