Make libcore pass -Zvalidate-mir #73175

doctorn · 2020-06-09T18:44:51Z

Extends call-ability for a type T to pass whenever T: FnOnce.

Resolves #73109

doctorn · 2020-06-09T18:46:12Z

jonas-schievink · 2020-06-09T20:12:37Z

Yeah I think we should land this for now. But can you add a FIXME comment that points out that the shim shouldn't contain a call to Self in the first place?

doctorn · 2020-06-09T20:15:34Z

Yeah I think we should land this for now. But can you add a FIXME comment that points out that the shim shouldn't contain a call to Self in the first place?

I can definitely add a comment, but the problems are these and I can't imagine them ever changing:

rust/src/libcore/ops/function.rs

Line 264 in feb3536

(*self).call(args)
rust/src/libcore/ops/function.rs

Line 274 in feb3536

(*self).call_mut(args)
rust/src/libcore/ops/function.rs

Line 285 in feb3536

(*self).call_mut(args)

jonas-schievink · 2020-06-09T20:21:45Z

Are those functions really what generates the bad MIR? I'd expect them to use the FnDef of the respective call method, not Self as the callee.

doctorn · 2020-06-09T20:28:21Z

Are those functions really what generates the bad MIR? I'd expect them to use the FnDef of the respective call method, not Self as the callee.

Yeah that's where it's quoted as coming from in the ICE. Essentially because of the blanket impls you end up unable to normalize Self properly; however, checking Self: FnOnce doesn't require normalization. (Because of the super trait bounds.)

jonas-schievink · 2020-06-09T20:36:33Z

The ICE points to the trait definition:

rust/src/libcore/ops/function.rs

Lines 230 to 232 in feb3536

    
               /// Performs the call operation. 
        
               #[unstable(feature = "fn_traits", issue = "29625")] 
        
               extern "rust-call" fn call_once(self, args: Args) -> Self::Output;

If this was really caused by that call, I'd expect the span to point there.

MIR shims appear to use the def_id their instance is for to get the span of their body:

rust/src/librustc_mir/shim.rs

Line 688 in feb3536

let span = tcx.def_span(def_id);

I think that would end up producing what we see.

doctorn · 2020-06-09T20:40:18Z

Hmm... I could have sworn the ICE was sourced at those impls but you're right

I guess I'll add that FIXME then

RalfJung · 2020-06-09T20:52:11Z

During which MIR pass was that error happening? I am pretty sure that in later passes, we shouldn't see Self here any more. Miri/CTFE will ICE when that happens -- and Miri supports FnOnce just fine.

So, just allowing this is not a solution. We should understand why validation fails but Miri does not.

RalfJung · 2020-06-09T20:53:56Z

src/librustc_mir/transform/validate.rs

+            // We have a `FnPtr` or `FnDef` which is trivially safe to call.
+        } else if let Some(fn_once_trait) = self.tcx.lang_items().fn_once_trait() {
+            // FIXME(doctorn): call shims shouldn't reference unnormalized types, so
+            // this case should be removed.


Did you try calling normalize_erasing_regions with param_env.with_reveal_all()?

I tried normalize_erasing_regions (didn't help) - not with that param env though. What does param_env.with_reveal_all actually do?

I don't fully know. :/ It has something to do with how much stuff can be unfolded. For example, with_reveal_all will unfold impl Trait types to their underlying definition.

It lets you see through impl Trait and specialization. Not sure that would do anything here though.

The goal is to unfold Self.

Which TyKind variant even corresponds to Self?

If I am not completely mistaken, Self is TyKind::Param, which is substs[0] when using InternalSubsts::identity_for_item.

TyKind::Closure should also be trivially safe to call here.

I think we still need the below branch in case someone implements FnOnce manually though. e.g. Box<impl FnOnce> is also callable and is represented as TyKind::Adt afaik.

fn foo(b: Box<impl FnOnce()>) { b() }

TyKind::Closure should also be trivially safe to call here.

No. That is the type of the environment of a closure, which is not callable. It just implements a trait that permits some surface-level sugar. Miri will ICE when a Call terminator is a Closure.

What you are saying makes sense on the HIR, but not any more on the MIR, which is more lower-level and where closure conversion has already happened.

Ahhh, that's interesting 👍

So my comment mentioning Box<impl FnOnce> is also irrelevant, as at this stage, the call of b should be represented as TyKind::FnDef(def_id = FnOnce::call_once, substs = [Box<impl FnOnce()>, (,)]) I think. Thanks ❤️

~~edit: I would then expect normalize_erasing_regions with Reveal::All to work~~

as at this stage, the call of b should be represented as TyKind::FnDef(def_id = FnOnce::call_once, substs = [Box<impl FnOnce()>, (,)]) I think

Exactly. :)
(I used to have the exact same misconception.^^ The only reason I know this is that I found the relevant code in Miri at some point and realized it would actually ICE for TyKind::Closure... but closures work fine in Miri so something was clearly going on. This is actually great for Miri as it means we need to do no work at all to support closures; MIR construction has already done all the work for us.)

doctorn · 2020-06-09T20:58:14Z

During which MIR pass was that error happening? I am pretty sure that in later passes, we shouldn't see Self here any more. Miri/CTFE will ICE when that happens -- and Miri supports FnOnce just fine.

So, just allowing this is not a solution. We should understand why validation fails but Miri does not.

ICE occurs in 6 MIR passes - my understanding here is clearly fairly cloudy but it the Self is being introduced somewhere in the shim code and then hanging around through MIR passes

RalfJung · 2020-06-09T21:04:19Z

ICE occurs in 6 MIR passes - my understanding here is clearly fairly cloudy but it the Self is being introduced somewhere in the shim code and then hanging around through MIR passes

The question is more, is it getting removed again at some point?

RalfJung · 2020-06-09T21:05:09Z

Yeah I think we should land this for now.

I'm okay landing this if we keep the issue open. But this doesn't actually resolve anything, it just papers over a problem. That can be a reasonable thing to do, but warrants an issue to track fixing it properly at some point. :)

doctorn · 2020-06-09T21:06:05Z

Yeah I think we should land this for now.

I'm okay landing this if we keep the issue open. But this doesn't actually resolve anything, it just papers over a problem. That can be a reasonable thing to do, but warrants an issue to track fixing it properly at some point. :)

Ok - removed the resolves tag and I'll take a serious look at the shim code this week. I'll add a note in the issue.

doctorn · 2020-06-09T21:07:02Z

ICE occurs in 6 MIR passes - my understanding here is clearly fairly cloudy but it the Self is being introduced somewhere in the shim code and then hanging around through MIR passes

The question is more, is it getting removed again at some point?

The last ICE is this but I have no idea how deep this is in the pipeline:

error: internal compiler error: broken MIR in DefId(2:2112 ~ core[44b8]::ops[0]::function[0]::FnOnce[0]::call_once[0]) (after AddCallGuards in phase Const) at bb0[0]:
encountered non-callable type Self in `Call` terminator
   --> /home/nathan/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:232:5
    |
232 |     extern "rust-call" fn call_once(self, args: Args) -> Self::Output;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

thread 'rustc' panicked at 'no errors encountered even though `delay_span_bug` issued', src/librustc_errors/lib.rs:366:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

RalfJung · 2020-06-09T21:08:39Z

"after AddCallGuards in phase Const"

That's still pretty early. So maybe we can have a flag for the phase (const/optimizing... though these names are awful) and only allow these callers in early phases.

The final check that we do after mir_optimized should not allow these strange callers.

src/librustc_mir/transform/validate.rs

matthewjasper · 2020-06-09T21:50:10Z

That's still pretty early.

It's the last pass we do on shims. Shims only have one set of transforms applied to the after they're generated.

bors · 2020-06-10T04:28:53Z

☔ The latest upstream changes (presumably #73190) made this pull request unmergeable. Please resolve the merge conflicts.

RalfJung · 2020-06-10T06:56:14Z

It's the last pass we do on shims. Shims only have one set of transforms applied to the after they're generated.

Hm, fair. My point stands though: the validation we do from mir_optimized should not allow this.

doctorn · 2020-06-10T12:06:36Z

I think this should be ok to land now if you still think it's a good idea - I'm still going to investigate the shims either way so it's up to you guys

RalfJung · 2020-06-10T13:15:56Z

As I just said, my concern remains that this accepts Self callees in normal MIR, when we only observed them in shims (which are somewhat special), so IMO it makes sense to only allow them in shims.

doctorn · 2020-06-10T13:33:40Z

Ah sorry - I misunderstood that.

RalfJung · 2020-06-14T13:29:55Z

To be clear, I think this PR is fine if we add a parameter to validation that tells it whether this is a shim or not. Then all the extra code this adds should only run for shims. That's a reasonable band-aid until we decide if we want to adjust shim MIR (so that the "what is callable" invariant is globally uniform) or do something else.

doctorn · 2020-06-14T14:07:10Z

@RalfJung I've gated the paper to only be applied when validating shims. I think in this form, this PR does actually close #73109 and removing this paper is down to #69925.

RalfJung · 2020-06-14T15:06:04Z

src/librustc_mir/transform/validate.rs

@@ -26,12 +31,16 @@ impl<'tcx> MirPass<'tcx> for Validator {
    fn run_pass(&self, tcx: TyCtxt<'tcx>, source: MirSource<'tcx>, body: &mut Body<'tcx>) {
        let def_id = source.def_id();
        let param_env = tcx.param_env(def_id);
-        TypeChecker { when: &self.when, def_id, body, tcx, param_env }.visit_body(body);
+        let validating_shim =
+            if let ty::InstanceDef::Item(_) = source.instance { false } else { true };


Oh, that's interesting, I thought this would be passed in. But I guess this works, too? I am not very familiar with InstanceDef.

Have we found any instances where something that is not Self is being called? Because then you can also special-case this even further by checking that the callee is ty.is_param(0), instead of doing a trait query. And we could also only special-case the specific call shims we care about.

I don't think we have, but I think that this approach is a little more liberal in the sense that it allows shim implementations to change without changing the assumptions we require here - there's definitely a trade-off here though

Shims shouldn't grow more ways to violate the assumptions we make here, they should only change so they use an FnDef instead of Self as the callee eventually, fixing the underlying issue

I mean to say that another class of shim could be introduced or a previously monomorphic shim could be made polymorphic

I agree it's tidier to check ty.is_param(0), but we really would like Self to implement FnOnce right? It would almost certainly be malformed MIR if it didn't

All I can think at the moment is that we include both checks?

That would work too

Ah, you can just compare with tcx.types.self_param too

src/librustc_mir/transform/validate.rs

RalfJung · 2020-06-14T15:07:32Z

I've gated the paper to only be applied when validating shims.

Great. :)

I think in this form, this PR does actually close #73109 and removing this paper is down to #69925.

Hm, I'd still like to understand why Shims can "violate" the rules here and why that doesn't ICE in Miri. Does Self end up being substituted by a FnDef, or does Miri somehow not run these shims?

doctorn · 2020-06-14T15:14:24Z

Hm, I'd still like to understand why Shims can "violate" the rules here and why that doesn't ICE in Miri. Does Self end up being substituted by a FnDef, or does Miri somehow not run these shims?

I really don't know anything about MIRI internals so I couldn't begin to comment, have you got any ideas about how we could go about investigating that?

bjorn3 · 2020-06-14T15:44:34Z

I am really confused by why the validator thinks that the type Self is called. -Zmir-dump gives the following mir:

// MIR for `std::ops::FnOnce::call_once` before AddMovesForPackedDrops

fn std::ops::FnOnce::call_once(_1: Self, _2: Args) -> <Self as std::ops::FnOnce<Args>>::Output {
    let mut _0: <Self as std::ops::FnOnce<Args>>::Output;
    let _3: &mut Self;

    bb0: {
        _3 = &mut _1;
        _0 = const <Self as std::ops::FnMut<Args>>::call_mut(move _3, move _2) -> [return: bb1, unwind: bb3];
                                         // ty::Const
                                         // + ty: for<'r> extern "rust-call" fn(&'r mut Self, Args) -> <Self as std::ops::FnOnce<Args>>::Output {<Self as std::ops::FnMut<Args>>::call_mut}
                                         // + val: Value(Scalar(<ZST>))
                                         // mir::Constant
                                         // + span: ~/.rustup/toolchains/nightly-2020-06-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:232:5: 232:71
                                         // + literal: Const { ty: for<'r> extern "rust-call" fn(&'r mut Self, Args) -> <Self as std::ops::FnOnce<Args>>::Output {<Self as std::ops::FnMut<Args>>::call_mut}, val: Value(Scalar(<ZST>)) }
    }

    bb1: {
        drop(_1) -> bb2;
    }

    bb2: {
        return;
    }

    bb3 (cleanup): {
        drop(_1) -> bb4;
    }

    bb4 (cleanup): {
        resume;
    }
}

It clearly calls an FnDef.

jonas-schievink · 2020-06-14T18:26:53Z

@bjorn3 I'm not really sure, but you might be looking at an ClosureOnceShim, not an FnPtrShim

bjorn3 · 2020-06-14T18:59:11Z

I used

$ echo 'fn main() {
    let mut v: Vec<u8> = Vec::with_capacity(1); v.extend_from_slice(b"a"); std::mem::forget(v);
}' | rustc -Zdump-mir=all -Zvalidate-mir -Cpanic=abort -
error: internal compiler error: broken MIR in DefId(2:2112 ~ core[44b8]::ops[0]::function[0]::FnOnce[0]::call_once[0]) (input to phase Const) at bb0[0]:
encountered non-callable type Self in `Call` terminator
   --> /home/bjorn/.rustup/toolchains/nightly-2020-06-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:232:5
    |
232 |     extern "rust-call" fn call_once(self, args: Args) -> Self::Output;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[...]

The only files in mir_dump/ were:

rustc.main.001-000.CheckPackedRef.after.mir
rustc.main.001-000.CheckPackedRef.before.mir
rustc.main.001-001.SimplifyCfg-initial.after.mir
rustc.main.001-001.SimplifyCfg-initial.before.mir
rustc.main.001-002.SanityCheck.after.mir
rustc.main.001-002.SanityCheck.before.mir
rustc.main.002-000.PromoteTemps.after.mir
rustc.main.002-000.PromoteTemps.before.mir
rustc.main.002-001.SimplifyCfg-qualify-consts.after.mir
rustc.main.002-001.SimplifyCfg-qualify-consts.before.mir
rustc.main.003-000.NoLandingPads.after.mir
rustc.main.003-000.NoLandingPads.before.mir
rustc.main.003-001.SimplifyBranches-initial.after.mir
rustc.main.003-001.SimplifyBranches-initial.before.mir
rustc.main.003-002.RemoveNoopLandingPads.after.mir
rustc.main.003-002.RemoveNoopLandingPads.before.mir
rustc.main.003-003.CleanupNonCodegenStatements.after.mir
rustc.main.003-003.CleanupNonCodegenStatements.before.mir
rustc.main.003-004.SimplifyCfg-early-opt.after.mir
rustc.main.003-004.SimplifyCfg-early-opt.before.mir
rustc.main.003-005.AddCallGuards.after.mir
rustc.main.003-005.AddCallGuards.before.mir
rustc.main.003-006.ElaborateDrops.after.mir
rustc.main.003-006.ElaborateDrops.before.mir
rustc.main.003-007.NoLandingPads.after.mir
rustc.main.003-007.NoLandingPads.before.mir
rustc.main.003-008.AddMovesForPackedDrops.after.mir
rustc.main.003-008.AddMovesForPackedDrops.before.mir
rustc.main.003-009.AddRetag.after.mir
rustc.main.003-009.AddRetag.before.mir
rustc.main.003-010.SimplifyCfg-elaborate-drops.after.mir
rustc.main.003-010.SimplifyCfg-elaborate-drops.before.mir
rustc.main.003-011.UnreachablePropagation.after.mir
rustc.main.003-011.UnreachablePropagation.before.mir
rustc.main.003-012.UninhabitedEnumBranching.after.mir
rustc.main.003-012.UninhabitedEnumBranching.before.mir
rustc.main.003-013.SimplifyCfg-after-uninhabited-enum-branching.after.mir
rustc.main.003-013.SimplifyCfg-after-uninhabited-enum-branching.before.mir
rustc.main.003-014.Inline.after.mir
rustc.main.003-014.Inline.before.mir
rustc.main.003-015.StateTransform.after.mir
rustc.main.003-015.StateTransform.before.mir
rustc.main.003-016.InstCombine.after.mir
rustc.main.003-016.InstCombine.before.mir
rustc.main.003-017.ConstProp.after.mir
rustc.main.003-017.ConstProp.before.mir
rustc.main.003-018.SimplifyBranches-after-const-prop.after.mir
rustc.main.003-018.SimplifyBranches-after-const-prop.before.mir
rustc.main.003-019.Deaggregator.after.mir
rustc.main.003-019.Deaggregator.before.mir
rustc.main.003-020.SimplifyArmIdentity.after.mir
rustc.main.003-020.SimplifyArmIdentity.before.mir
rustc.main.003-021.SimplifyBranchSame.after.mir
rustc.main.003-021.SimplifyBranchSame.before.mir
rustc.main.003-022.CopyPropagation.after.mir
rustc.main.003-022.CopyPropagation.before.mir
rustc.main.003-023.SimplifyBranches-after-copy-prop.after.mir
rustc.main.003-023.SimplifyBranches-after-copy-prop.before.mir
rustc.main.003-024.RemoveNoopLandingPads.after.mir
rustc.main.003-024.RemoveNoopLandingPads.before.mir
rustc.main.003-025.SimplifyCfg-after-remove-noop-landing-pads.after.mir
rustc.main.003-025.SimplifyCfg-after-remove-noop-landing-pads.before.mir
rustc.main.003-026.SimplifyCfg-final.after.mir
rustc.main.003-026.SimplifyCfg-final.before.mir
rustc.main.003-027.RenameReturnPlace.after.mir
rustc.main.003-027.RenameReturnPlace.before.mir
rustc.main.003-028.SimplifyLocals.after.mir
rustc.main.003-028.SimplifyLocals.before.mir
rustc.main.003-029.AddCallGuards.after.mir
rustc.main.003-029.AddCallGuards.before.mir
rustc.main.003-030.PreCodegen.after.mir
rustc.main.003-030.PreCodegen.before.mir
rustc.main.-------.mir_map.0.mir
rustc.main.-------.nll.0.mir
rustc.main.-------.nll.0.regioncx.all.dot
rustc.main.-------.nll.0.regioncx.scc.dot
rustc.main.-------.renumber.0.mir
rustc.ops-function-FnOnce-call_once.001-000.AddMovesForPackedDrops.after.mir
rustc.ops-function-FnOnce-call_once.001-000.AddMovesForPackedDrops.before.mir
rustc.ops-function-FnOnce-call_once.001-001.NoLandingPads.after.mir
rustc.ops-function-FnOnce-call_once.001-001.NoLandingPads.before.mir
rustc.ops-function-FnOnce-call_once.001-002.RemoveNoopLandingPads.after.mir
rustc.ops-function-FnOnce-call_once.001-002.RemoveNoopLandingPads.before.mir
rustc.ops-function-FnOnce-call_once.001-003.SimplifyCfg-make_shim.after.mir
rustc.ops-function-FnOnce-call_once.001-003.SimplifyCfg-make_shim.before.mir
rustc.ops-function-FnOnce-call_once.001-004.AddCallGuards.after.mir
rustc.ops-function-FnOnce-call_once.001-004.AddCallGuards.before.mir
rustc.ptr-drop_in_place.001-000.AddMovesForPackedDrops.after.mir
rustc.ptr-drop_in_place.001-000.AddMovesForPackedDrops.before.mir
rustc.ptr-drop_in_place.001-001.NoLandingPads.after.mir
rustc.ptr-drop_in_place.001-001.NoLandingPads.before.mir
rustc.ptr-drop_in_place.001-002.RemoveNoopLandingPads.after.mir
rustc.ptr-drop_in_place.001-002.RemoveNoopLandingPads.before.mir
rustc.ptr-drop_in_place.001-003.SimplifyCfg-make_shim.after.mir
rustc.ptr-drop_in_place.001-003.SimplifyCfg-make_shim.before.mir
rustc.ptr-drop_in_place.001-004.AddCallGuards.after.mir
rustc.ptr-drop_in_place.001-004.AddCallGuards.before.mir

Excluding the rustc.main.*.mir and rustc.ptr-drop_in_place.*.mir that only leaves:

rustc.ops-function-FnOnce-call_once.001-000.AddMovesForPackedDrops.after.mir
rustc.ops-function-FnOnce-call_once.001-000.AddMovesForPackedDrops.before.mir
rustc.ops-function-FnOnce-call_once.001-001.NoLandingPads.after.mir
rustc.ops-function-FnOnce-call_once.001-001.NoLandingPads.before.mir
rustc.ops-function-FnOnce-call_once.001-002.RemoveNoopLandingPads.after.mir
rustc.ops-function-FnOnce-call_once.001-002.RemoveNoopLandingPads.before.mir
rustc.ops-function-FnOnce-call_once.001-003.SimplifyCfg-make_shim.after.mir
rustc.ops-function-FnOnce-call_once.001-003.SimplifyCfg-make_shim.before.mir
rustc.ops-function-FnOnce-call_once.001-004.AddCallGuards.after.mir
rustc.ops-function-FnOnce-call_once.001-004.AddCallGuards.before.mir

jonas-schievink · 2020-06-14T21:56:15Z

I've managed to fix the shim creation to avoid Self in #73359, so if that's accepted we won't have to add this workaround.

doctorn · 2020-06-15T07:59:31Z

I'm going to tidy this up following our conversations yesterday in case #73359 can't land for some reason but I'm really hoping we don't need this

rust-highfive assigned jonas-schievink Jun 9, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 9, 2020

RalfJung reviewed Jun 9, 2020

View reviewed changes

doctorn mentioned this pull request Jun 9, 2020

ICE: -Zvalidate-mir: broken mir in libcore #73109

Closed

jonas-schievink reviewed Jun 9, 2020

View reviewed changes

src/librustc_mir/transform/validate.rs Outdated Show resolved Hide resolved

doctorn added 3 commits June 14, 2020 14:49

Make libcore pass -Zvalidate-mir

5fe694d

Add FIXME

1cae892

Use require_lang_item

218eaee

Gate paper to only be applied when validating shims

9df2bc6

RalfJung reviewed Jun 14, 2020

View reviewed changes

src/librustc_mir/transform/validate.rs Show resolved Hide resolved

This was referenced Jun 14, 2020

-Zvalidate-mir: check that all referenced locals exist #73356

Closed

shim.rs: avoid creating Call terminators calling Self #73359

Merged

Update comments and add check for Self

82fa79a

bors closed this in fe4b485 Jun 20, 2020

Make libcore pass -Zvalidate-mir #73175

Make libcore pass -Zvalidate-mir #73175

Conversation

doctorn commented Jun 9, 2020 • edited Loading

doctorn commented Jun 9, 2020

jonas-schievink commented Jun 9, 2020

doctorn commented Jun 9, 2020

jonas-schievink commented Jun 9, 2020

doctorn commented Jun 9, 2020 • edited Loading

jonas-schievink commented Jun 9, 2020

doctorn commented Jun 9, 2020 • edited Loading

RalfJung commented Jun 9, 2020

Choose a reason for hiding this comment

doctorn Jun 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcnr Jun 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lcnr Jun 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doctorn commented Jun 9, 2020

RalfJung commented Jun 9, 2020

RalfJung commented Jun 9, 2020

doctorn commented Jun 9, 2020

doctorn commented Jun 9, 2020

RalfJung commented Jun 9, 2020

matthewjasper commented Jun 9, 2020

bors commented Jun 10, 2020

RalfJung commented Jun 10, 2020

doctorn commented Jun 10, 2020

RalfJung commented Jun 10, 2020 • edited Loading

doctorn commented Jun 10, 2020

RalfJung commented Jun 14, 2020 • edited Loading

doctorn commented Jun 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doctorn Jun 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RalfJung commented Jun 14, 2020

doctorn commented Jun 14, 2020

bjorn3 commented Jun 14, 2020

jonas-schievink commented Jun 14, 2020

bjorn3 commented Jun 14, 2020 • edited Loading

jonas-schievink commented Jun 14, 2020

doctorn commented Jun 15, 2020

doctorn commented Jun 9, 2020 •

edited

Loading

doctorn commented Jun 9, 2020 •

edited

Loading

doctorn commented Jun 9, 2020 •

edited

Loading

doctorn Jun 9, 2020 •

edited

Loading

lcnr Jun 14, 2020 •

edited

Loading

lcnr Jun 14, 2020 •

edited

Loading

RalfJung commented Jun 10, 2020 •

edited

Loading

RalfJung commented Jun 14, 2020 •

edited

Loading

doctorn Jun 14, 2020 •

edited

Loading

bjorn3 commented Jun 14, 2020 •

edited

Loading