Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[staging] linux.stdenv: gcc_10 → gcc_11 #165732

Merged
merged 6 commits into from
Mar 26, 2022

Conversation

fabianhjr
Copy link
Member

@fabianhjr fabianhjr commented Mar 25, 2022

Description of changes

Was able to build and boot into gnome 42 under kernel 5.17, seems like most packages already support gcc11 without changes; only 1 package gave issues and pinning it to a previous version was enough.

cat /proc/version                                                                                                                                                                                                                                                       Thu 24 Mar 2022 09:42:13 PM CST
Linux version 5.17.0 (nixbld@localhost) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.35.2) #1-NixOS SMP PREEMPT Sun Mar 20 20:14:17 UTC 2022

Currently running a home-manager switch but those are a ton more (~4500) over the system required packages (~2600)

EDIT:

Additionally built linux_4_9 (oldest supported LTS)

EDIT2:

Home environment also built, there are some packages that need updating but this works great so far! :3

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.05 Release Notes (or backporting 21.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@fabianhjr
Copy link
Member Author

/marvin opt-in
/status needs_reviewer

@marvin-mk2
Copy link

marvin-mk2 bot commented Mar 25, 2022

Hi! I'm an experimental bot. My goal is to guide this PR through its stages, hopefully ending with a merge. You can read up on the usage here.

@marvin-mk2 marvin-mk2 bot added marvin This PR was reviewed by Marvin, a discontinued bot: https://github.com/timokau/marvin-mk2 needs_reviewer labels Mar 25, 2022
@fabianhjr
Copy link
Member Author

@ofborg eval

@fabianhjr fabianhjr changed the title linux.stdenv: gcc_10 → gcc_11 [staging] linux.stdenv: gcc_10 → gcc_11 Mar 25, 2022
@ofborg ofborg bot added 10.rebuild-linux-stdenv This PR causes stdenv to rebuild 8.has: clean-up labels Mar 25, 2022
@ofborg ofborg bot requested review from edolstra, globin and kalbasit March 25, 2022 06:18
@ofborg ofborg bot requested a review from adevress March 25, 2022 06:18
@ofborg ofborg bot requested review from mmahut, np, lovek323 and WilliButz March 25, 2022 06:18
@trofi
Copy link
Contributor

trofi commented Apr 3, 2022

Worth moving gfortran to 11 version as well?

@fabianhjr
Copy link
Member Author

Worth moving gfortran to 11 version as well?

Probably, but didn't test; it is not necessary afaik.

@mweinelt
Copy link
Member

mweinelt commented Apr 4, 2022

I bisected grpc breakage on staging to 52f8cf5. Please take a look.

[ 98%] Building CXX object CMakeFiles/grpc++.dir/src/cpp/common/rpc_method.cc.o
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::numbers_internal::safe_strto64_base(absl::lts_20210324::string_view, long*, int)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::EndsWithIgnoreCase(absl::lts_20210324::string_view, absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::numbers_internal::safe_strtou64_base(absl::lts_20210324::string_view, unsigned long*, int)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgpr.so.22.0.0: undefined reference to `absl::lts_20210324::Cord::GetFlatAux(absl::lts_20210324::cord_internal::CordRep*, absl::lts_20210324::string_view*)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::string_view::find_first_of(absl::lts_20210324::string_view, unsigned long) const'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::StartsWithIgnoreCase(absl::lts_20210324::string_view, absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::optional_internal::throw_bad_optional_access()'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::numbers_internal::safe_strtou32_base(absl::lts_20210324::string_view, unsigned int*, int)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::ByString::ByString(absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::Status::SetPayload(absl::lts_20210324::string_view, absl::lts_20210324::Cord)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::StrReplaceAll[abi:cxx11](absl::lts_20210324::string_view, std::initializer_list<std::pair<absl::lts_20210324::string_view, absl::lts_20210324::string_view> >)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::string_view::find(absl::lts_20210324::string_view, unsigned long) const'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::Base64Escape[abi:cxx11](absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::BytesToHexString[abi:cxx11](absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::InternalError(absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::string_view::rfind(char, unsigned long) const'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `bool absl::lts_20210324::str_format_internal::FormatArgImpl::Dispatch<absl::lts_20210324::string_view>(absl::lts_20210324::str_format_internal::FormatArgImpl::Data, absl::lts_20210324::str_format_internal::FormatConversionSpecImpl, void*)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::ByChar::Find(absl::lts_20210324::string_view, unsigned long) const'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgpr.so.22.0.0: undefined reference to `absl::lts_20210324::CHexEscape[abi:cxx11](absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::string_view::rfind(absl::lts_20210324::string_view, unsigned long) const'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::FailedPreconditionError(absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::strings_internal::CatPieces[abi:cxx11](std::initializer_list<absl::lts_20210324::string_view>)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgpr.so.22.0.0: undefined reference to `absl::lts_20210324::Status::ForEachPayload(std::function<void (absl::lts_20210324::string_view, absl::lts_20210324::Cord const&)> const&) const'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgpr.so.22.0.0: undefined reference to `absl::lts_20210324::Cord::Cord(absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::ParseTime(absl::lts_20210324::string_view, absl::lts_20210324::string_view, absl::lts_20210324::Time*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::variant_internal::ThrowBadVariantAccess()'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::InvalidArgumentError(absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::ByString::Find(absl::lts_20210324::string_view, unsigned long) const'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::NotFoundError(absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::UnavailableError(absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::UnimplementedError(absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::FormatTime[abi:cxx11](absl::lts_20210324::string_view, absl::lts_20210324::Time, absl::lts_20210324::TimeZone)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::EqualsIgnoreCase(absl::lts_20210324::string_view, absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::Status::Status(absl::lts_20210324::StatusCode, absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::CUnescape(absl::lts_20210324::string_view, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::Status::GetPayload(absl::lts_20210324::string_view) const'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::UnknownError(absl::lts_20210324::string_view)'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::string_view::find(char, unsigned long) const'
/nix/store/q4fj5h3wfds7v0bk718ykbr3j4y8ap5p-binutils-2.35.2/bin/ld: libgrpc.so.22.0.0: undefined reference to `absl::lts_20210324::numbers_internal::safe_strto32_base(absl::lts_20210324::string_view, int*, int)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/gen_hpack_tables.dir/build.make:146: gen_hpack_tables] Error 1
make[1]: *** [CMakeFiles/Makefile2:641: CMakeFiles/gen_hpack_tables.dir/all] Error 2
[ 99%] Building CXX object CMakeFiles/grpc++.dir/src/cpp/common/resource_quota_cc.cc.o
[ 99%] Linking C shared library libgrpc_csharp_ext.so
[ 99%] Building CXX object CMakeFiles/grpc++.dir/src/cpp/common/core_codegen.cc.o
[ 99%] Building CXX object CMakeFiles/grpc++.dir/src/cpp/common/completion_queue_cc.cc.o
[ 99%] Building CXX object CMakeFiles/grpc++.dir/src/cpp/common/channel_filter.cc.o
[ 99%] Built target grpc_csharp_ext
[ 99%] Linking CXX shared library libgrpc++.so
[ 99%] Built target grpc++
make: *** [Makefile:136: all] Error 2

@fabianhjr
Copy link
Member Author

Taking a look, building lastest staging since binutils-2.38 got merged and seems like a linking/ld issue.

@fabianhjr
Copy link
Member Author

fabianhjr commented Apr 4, 2022

Was able to reproduce locally on current staging ( c9154e5 ), investigating the cause and possible fixes.

@fabianhjr
Copy link
Member Author

@mweinelt, seems like the change of c++14 to c++17 by default changed abseil but not grpc so there was a link time issue. The following patch fixes the build on nixpkgs but will need to take a look at packages that depend on abseil.

diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 16704f0ec64..f5fb9db5bae 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -17225,7 +17225,12 @@ with pkgs;
 
   grilo-plugins = callPackage ../development/libraries/grilo-plugins { };
 
-  grpc = callPackage ../development/libraries/grpc { };
+  grpc = callPackage ../development/libraries/grpc {
+    # grpc builds with c++14 so abseil must also be built that way
+    abseil-cpp = abseil-cpp.override {
+      cxxStandard = "14";
+    };
+  };
 
   gsettings-qt = libsForQt5.callPackage ../development/libraries/gsettings-qt { };

@samuela
Copy link
Member

samuela commented Apr 9, 2022

After this change we should also update

["11.4"]
version = "11.4.2"
url = "https://developer.download.nvidia.com/compute/cuda/11.4.2/local_installers/cuda_11.4.2_470.57.02_linux.run"
sha256 = "sha256-u9h8oOkT+DdFSnljZ0c1E83e9VUILk2G7Zo4ZZzIHwo="
gcc = "gcc10" # can bump to 11 along with stdenv.cc
["11.5"]
version = "11.5.0"
url = "https://developer.download.nvidia.com/compute/cuda/11.5.0/local_installers/cuda_11.5.0_495.29.05_linux.run"
sha256 = "sha256-rgoWk9lJfPPYHmlIlD43lGNpANtxyY1Y7v2sr38aHkw="
gcc = "gcc10" # can bump to 11 along with stdenv.cc
["11.6"]
version = "11.6.1"
url = "https://developer.download.nvidia.com/compute/cuda/11.6.1/local_installers/cuda_11.6.1_510.47.03_linux.run"
sha256 = "sha256-qyGa/OALdCABEyaYZvv/derQN7z8I1UagzjCaEyYTX4="
gcc = "gcc10" # can bump to 11 along with stdenv.cc
. cc @NixOS/cuda-maintainers

@samuela
Copy link
Member

samuela commented Apr 18, 2022

This change broke dm-tree, jax, optax, #169134, #169148, flax, tensorflow-datasets, and certainly many many others. What is the plan for doing this migration? How should packages respond when this broke their build?

I suspect the total number of breakages to be in the hundreds.

@fabianhjr
Copy link
Member Author

There are several paths; currently doing as many fixes as possible but at the rate of 10 or so per day; if a specific gcc version is needed then an override of the stdenv would do. (stdenv = if stdenv.cc.isGNU then gcc10Stdenv else stdenv)

@fabianhjr
Copy link
Member Author

google-cloud-cpp seems to be related to a mismatch between abseil -std and google-cloud-cpp -std a similar override to abseil-cpp as that done on grpc should fix it. Currently working on a merge request.

@fabianhjr
Copy link
Member Author

google-cloud-cpp fix: #169220

Can't reproduce python3Packages.jaxlib build issue.

@fabianhjr
Copy link
Member Author

I would recommend either attempting a fix or if not possible opening an issue and notifying the maintainers.

@samuela
Copy link
Member

samuela commented Apr 18, 2022

I would recommend either attempting a fix or if not possible opening an issue and notifying the maintainers.

I am the maintainer :P

@SomeoneSerge
Copy link
Contributor

I suspect the breakage mostly affects packages from import ./. { config.cudaSupport = true; config.allowUnfree = true; }.
What can we do to prevent such breakages in future?

@NixOS/cuda-maintainers are currently running their own infrastructure to build and test cuda-enabled packages, but we only do so post-factum, when the offending changes have already been merged. We also couldn't facilitate building the staging branch so far (we're building master, nixpkgs-unstable, nixos-unstable, and the last release)

I would really like that merging staging into master could be blocked until at least a very basic collection of cuda-enabled packages are guaranteed to build, but I don't know how to approach this?

@fabianhjr
Copy link
Member Author

Probably better to open a separate issue to discuss possible solutions since what you point out (testing in staging-next before merging and block merging in case of breakage) might be lost / not reach other stakeholders on this merge request.

@samuela
Copy link
Member

samuela commented Apr 19, 2022

Yes, we're seeing a significant number of breakages every time staging/staging-next get merged. It consumes an enormous amount of maintainer time and energy for us to play "catch up" with these massive changes that break wide swaths of our package landscape. What can we do to avoid this in the future?

@vcunat
Copy link
Member

vcunat commented Apr 19, 2022

That's "the same" everywhere. Only for packages built on Hydra usually someone else is fixing the regressions. (and mostly before merging to master)

@samuela
Copy link
Member

samuela commented Apr 19, 2022

This just doesn't seem like a sustainable long-term development model to me. As a maintainer it seems like I'm forced to either:

  • Constantly track all of the changes happening on master and recognize breakages before they make it into master. Maintainers have no infrastructure or support in doing this. My packages aren't in any hydra jobsets. How am I supposed to know when people break things before those changes end up in master?
  • Try to unbreak massive breakages after staging gets merged in and everything is messed up for reasons that maintainers don't understand or have context on. To make matters worse, there's 900 commits to sift through. So good luck sorting through the mess... This is a huge burden and leads to burnout.

Neither of these options are sustainable IMHO. We need a better system. Personally, I don't think people should be able to push PRs breaking other people's packages without a fallback option and a stated plan for completing a migration to the new version. Otherwise it's just chaos.

@vcunat
Copy link
Member

vcunat commented Apr 19, 2022

Quite many breakages do reach master all the time, even for packages built on Hydra, if we look at absolute numbers. It's one of the reasons why stable releases exist, also in many other distros. If you care about packages that don't get build on Hydra.nixos.org and/or OfBorg CI, I can't see how to significantly improve this for them without attacking the fact that they don't get build. Without that noone will even know that there is a breakage. (we're getting very off-topic on this thread)

EDIT: but maybe you or someone else will have another suggestion. Feel free to start a thread somewhere and mention me or cross-link from here.

In particular, this gcc bump had larger negative impact than I hoped, but that doesn't seem very related to the "system" questions.

@fabianhjr
Copy link
Member Author

In retrospective I was expecting a larger staging-next cycle of 2 weeks but it ended up being close to 3 days. (1 day when everything was building and only 2 days of fixes before it hit the main branch)

Even then I would have probably missed fixing breakages of packages / variants not built by hydra.

@samuela
Copy link
Member

samuela commented Apr 19, 2022

In particular, this gcc bump had larger negative impact than I hoped, but that doesn't seem very related to the "system" questions.

Agreed, we should take that discussion to a different thread somewhere. maybe discourse?

In retrospective I was expecting a larger staging-next cycle of 2 weeks but it ended up being close to 3 days. (1 day when everything was building and only 2 days of fixes before it hit the main branch)

Indeed and thank you for taking this on and addressing the failures so far, @fabianhjr! GCC bumps are not for the faint of heart, but they are important work. I don't fault you personally in this quagmire; I think the current development model, which I can only summarize as "branch aggressively and cross your fingers", is lacking for everyone involved. We need to think about designing a better system to give everyone more confidence in being able to make large changes more safely.

@vcunat
Copy link
Member

vcunat commented Apr 20, 2022

We do stabilization in separate branches and jobsets for some risky changes.

@samuela
Copy link
Member

samuela commented Apr 20, 2022

I started a discussion on Discourse for the broader conversation: https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741.

@trofi
Copy link
Contributor

trofi commented Apr 21, 2022

rili broke, proposed fix: #169540

@trofi
Copy link
Contributor

trofi commented Apr 21, 2022

zkfuse broke, proposed fix: #169543

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.has: clean-up 10.rebuild-darwin: 101-500 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ 10.rebuild-linux-stdenv This PR causes stdenv to rebuild awaiting_reviewer marvin This PR was reviewed by Marvin, a discontinued bot: https://github.com/timokau/marvin-mk2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants