Feature: Improved CW Calculation #1451

kwek20 · 2019-05-13T19:09:49Z

Description

Updated the CW calculation to be up to 3x faster.
Changing the Transforming map to a regular HashMap made another small improvement.
Could not find a clear increase in memory usage as no OOM occured during any testing, even with -Xmx512M

Fixes #1428

Type of change

Improvement to CW calc leading to faster GTTA.

How Has This Been Tested?

Tests ran, benchmarked, network tests with GTTA

** DEPTH 3**
1000*5 threads

hashId:

Cache:
summary = 5000 in 00:19:06 = 4.4/s Avg: 1013 Min: 379 Max: 24359 Err: 0 (0.00%)
summary = 5000 in 00:18:35 = 4.5/s Avg: 986 Min: 356 Max: 3006 Err: 0 (0.00%)

dev:
summary = 5000 in 00:35:36 = 2.3/s Avg: 1866 Min: 612 Max: 5439 Err: 0 (0.00%)

Just Hash:
summary = 5000 in 00:18:03 = 4.6/s Avg: 951 Min: 289 Max: 2891 Err: 0 (0.00%)
summary = 5000 in 00:18:04 = 4.6/s Avg: 953 Min: 271 Max: 2676 Err: 0 (0.00%)

-- All HashId Removed
** DEPTH 15**
100*10 threads, stoppped at 83
summary = 83 in 00:16:50 = 0.1/s Avg: 113670 Min: 9931 Max: 234565 Err: 0 (0.00%)

** DEPTH 15**
10*20 threads
summary = 200 in 00:28:11 = 0.1/s Avg: 154674 Min: 8683 Max: 339657 Err: 0 (0.00%)

Checklist:

My code follows the style guidelines for this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

…weight impl

GalRogozinski · 2019-05-27T12:30:32Z

I did some benchmark on the dev tip-selection with my laptop and a devnet db:

  │         Label         │    #    │ Averag │ Mi │ Ma │  Std.   │ Error  │ Throughp │   Received    │   Sent    │   Avg.   │
  │                       │ Samples │   e    │ n  │ x  │  Dev.   │   %    │    ut    │    KB/sec     │  KB/sec   │  Bytes   │
  ├───────────────────────┼─────────┼────────┼────┼────┼─────────┼────────┼──────────┼───────────────┼───────────┼──────────┤
  │ GetTransactionsToAppr │  5000   │  156   │ 38 │ 73 │  87.50  │ 0.000% │ 28.17568 │     11.60     │   7.10    │  421.6   │
  │          ove          │         │        │    │ 1  │         │        │          │               │           │          │
  ├───────────────────────┼─────────┼────────┼────┼────┼─────────┼────────┼──────────┼───────────────┼───────────┼──────────┤

This is on the PR:

  ┌───────────────────────┬─────────┬────────┬────┬────┬─────────┬────────┬──────────┬───────────────┬───────────┬──────────┐
  │         Label         │    #    │ Averag │ Mi │ Ma │  Std.   │ Error  │ Throughp │   Received    │   Sent    │   Avg.   │
  │                       │ Samples │   e    │ n  │ x  │  Dev.   │   %    │    ut    │    KB/sec     │  KB/sec   │  Bytes   │
  ├───────────────────────┼─────────┼────────┼────┼────┼─────────┼────────┼──────────┼───────────────┼───────────┼──────────┤
  │ GetTransactionsToAppr │  5000   │  152   │ 28 │ 46 │  82.16  │ 0.000% │ 28.88737 │     11.89     │   7.28    │  421.6   │
  │          ove          │         │        │    │ 4  │         │        │          │               │           │          │

It looks like the results are pretty similar. Also on some tests the PR results are a bit slower. We will soon have better reproducible tests.

I believe we can do better then the current code. More ideas in my upcoming review.

GalRogozinski

In order to make this algorithm run faster:

When you scan the entire subtangle save an in memory map of hash to approvers. Use the in memory map to traverse to approvers in getRating method. You save on DB calls and thus increase performance
Note that the size of the keyset of the map from step 1 is the CW of entrypoint. Using this will save you a large getRating calculation
Even though on my testdb there were no exceptions, using actual recursion can lead to stack overflow exceptions. Use ArrayDeque as stack to implement an iterative getRating().

GalRogozinski · 2019-05-27T07:38:46Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+
+/**
+ * Calculates the weight recursively/on the fly instead of building the tree and calculating after
+ */


Can you give a better description of the algorithm?
Try not to refer to the other implementation

Maybe delete the other implementation, then you can call this one Cumulative Weight Calculator. Then we won't have to maintain more code. Also it will make you improve the javadoc (you can define CW in the doc)

You should have copied the description but not the Link!!
The link explains the old algo which you deleted

Simply write exactly the following:
"Calculates the cumulative weight for each transaction above a certain entry point. Cumulative weight is an integer defined as the number of approving transactions plus one. We calculate it by doing a graph traversal from each transaction above a certain entry point and counting the number of different transactions we pass."

GalRogozinski · 2019-05-27T07:59:53Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+ */
+public class RecursiveWeightCalculator implements RatingCalculator {
+
+    public final Tangle tangle;


why public?

GalRogozinski · 2019-05-27T08:01:24Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+    @Override
+    public UnIterableMap<HashId, Integer> calculate(Hash entryPoint) throws Exception {
+        Set<Hash> toCheck = new HashSet<>(1);
+        toCheck.add(entryPoint);


Besides here, where do you use this set?

GalRogozinski · 2019-05-27T08:36:53Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+        toCheck.add(entryPoint);
+
+        // Initial capacity of 16, as default for java maps and lists
+        UnIterableMap<HashId, Integer> hashWeight = createTxHashToCumulativeWeightMap(16);


Instead of 16 let's try a more optimized number.
Let say we have 3 tps and normally we use depth 3. Let say a milestone comes every 5 minutes. So we 5 X 60 X 3 X 3 = 2700

Since you have the entry point you can know the exact depth btw

GalRogozinski · 2019-05-27T09:36:31Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+
+        // Initial capacity of 16, as default for java maps and lists
+        UnIterableMap<HashId, Integer> hashWeight = createTxHashToCumulativeWeightMap(16);
+        calculateRatingDfs(entryPoint, hashWeight);


I like the style of having the result as a return value rather than an input parameter that changes

GalRogozinski · 2019-07-18T09:24:42Z

I have performed another test on an isolated node that has a mainnetdb synced to ms #110932. Spamming with a GTTA of 3 with Jmeter I obtained the following results now:

Old CW:

Label	# Samples	Average	Min	Max	Std. Dev.	Error %	Throughput	Received KB/sec	Sent KB/sec	Avg. Bytes
GetTransactionsToApprove	340	4124	828	9147	1868.22	1.471%	1.21077	0.54	0.32	456.2
TOTAL	340	4124	828	9147	1868.22	1.471%	1.21077	0.54	0.32	456.2

New CW:

Label	# Samples	Average	Min	Max	Std. Dev.	Error %	Throughput	Received KB/sec	Sent KB/sec	Avg. Bytes
GetTransactionsToApprove	500	1472	317	4712	759.86	0.000%	3.12641	1.29	0.85	422.6
TOTAL	500	1472	317	4712	759.86	0.000%	3.12641	1.29	0.85	422.6

So the new implementation is approximately 3X faster

kwek20 · 2019-07-18T13:48:57Z

Whoa, did not expect that @GalRogozinski.
Guess we are going to use this in production code? Ill take another pass over it to check for some improvements

GalRogozinski

You can start by the change of removing the field...
I want to see how it affects the benchmark again

Then after we have the results we can decide on the other changes

GalRogozinski · 2019-07-18T16:58:41Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+    private final Tangle tangle;
+    private final SnapshotProvider snapshotProvider;
+
+    private Map<Hash, ArrayDeque<Hash>> txToDirectApprovers = new HashMap<>();


Hmm this field makes the calculator stateful
Let's remove it since we will have caching soon

GalRogozinski · 2019-07-18T16:59:56Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+        while (CollectionUtils.isNotEmpty(stack)) {
+            Hash txHash = stack.peek();
+            if (!hashWeight.containsKey(txHash)) {
+                Collection<Hash> appHashes = getTxDirectApproversHashes(txHash, txToDirectApprovers, txToDirectApprovers);


I suppose the fallkback is supposed to be the field...
Actually you are using the same parameter twice

GalRogozinski · 2019-07-18T17:00:58Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+     */
+    private ArrayDeque<Hash> getTxDirectApproversHashes(Hash txHash,  
+            Map<Hash, ArrayDeque<Hash>> txToDirectApprovers, 
+            Map<Hash, ArrayDeque<Hash>> fallback) 


no need for fallback

GalRogozinski · 2019-07-18T17:07:52Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+    private int getRating(Hash hash, Set<HashId> seenHashes) throws Exception {
+        int weight = 1;
+
+        ArrayDeque<Hash> approvers = getTxDirectApproversHashes(hash, txToDirectApprovers, null);


If instead of the field you will pass the parameter from calculatingRatingDfs you will also save on I/O calls

GalRogozinski · 2019-07-18T17:12:57Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+        for (Hash approver : approvers) {
+            if (!seenHashes.contains(approver)) {
+                seenHashes.add(approver);
+                weight += getRating(approver, seenHashes);


It is better to have something iterative rather than recursive calls. May prevent StackOverflows...
You are simply doing a dfs, so you can reuse the code of calculateRatingDfs

GalRogozinski · 2019-07-21T18:01:50Z

@kwek20
I just checked and even if you remove the field you get 3x better results

GalRogozinski · 2019-07-28T17:41:32Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+
+/**
+ * Calculates the weight recursively/on the fly instead of building the tree and calculating after
+ */


Maybe delete the other implementation, then you can call this one Cumulative Weight Calculator. Then we won't have to maintain more code. Also it will make you improve the javadoc (you can define CW in the doc)

GalRogozinski · 2019-07-28T17:50:11Z

src/test/java/com/iota/iri/service/tipselection/impl/CumulativeWeightCalculatorTest.java

@@ -269,6 +283,7 @@ public void testTangleWithCircle2() throws Exception {
    }

    @Test
+    @Ignore


I think we discussed this, but remind me again why are we ignoring a test that previously passed?

It tests a broken thing in our previous system, collisions between HashIds of hashes.
transactionHash2 is a similar hash to the other one, and is therefore not included.
No idea why we ever did that, as it is literally skipping on transactions this way.

Actually, id vote for removing the hashId entirely, having it might speed some things up, but if it blocks certain tx, its still not worth it. (in the test, tx1 and tx2 work because they reference to the same value in the end.)

An example on how to break it earlier is make tx3 and tx1 the same instead of 1 and 2. Then 3 should be value 1, and not 2, which will error

GalRogozinski · 2019-07-28T17:58:03Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

@@ -113,15 +107,14 @@ private Hash getAndRemoveApprover(Collection<Hash> appHashes) {
     * @return


why is this empty?

GalRogozinski · 2019-07-28T18:03:25Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

        stack.push(entryPoint);
+
        while (CollectionUtils.isNotEmpty(stack)) {


you can use regular !isEmpty() let's get rid of CollectionUtils

GalRogozinski · 2019-07-28T18:07:35Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

@@ -55,53 +53,49 @@ public RecursiveWeightCalculator(Tangle tangle, SnapshotProvider snapshotProvide

        // Estimated capacity per depth, assumes 5 minute gap in between milestones, at 3tps
        UnIterableMap<HashId, Integer> hashWeight = createTxHashToCumulativeWeightMap( 5 * 60 * 3 * depth);


Maybe call it hashToWeight or hashWeightMap.
This variable has an integer name

you changed it in one place, but you didn't change it in the others

GalRogozinski · 2019-07-28T18:18:57Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

-                weight += getRating(approver, seenHashes);
+    private int getRating(HashSet<Hash> nonDupes, Map<Hash, HashSet<Hash>> txToDirectApprovers) throws Exception {
+        Deque<Hash> stack = new ArrayDeque<>(nonDupes);
+        while (CollectionUtils.isNotEmpty(stack)) {


Get rid of CollectionUtils

GalRogozinski · 2019-07-30T08:56:00Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+
+/**
+ * Calculates the weight recursively/on the fly instead of building the tree and calculating after
+ */


You should have copied the description but not the Link!!
The link explains the old algo which you deleted

Simply write exactly the following:
"Calculates the cumulative weight for each transaction above a certain entry point. Cumulative weight is an integer defined as the number of approving transactions plus one. We calculate it by doing a graph traversal from each transaction above a certain entry point and counting the number of different transactions we pass."

GalRogozinski · 2019-07-30T08:56:24Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+ * Calculates the weight recursively/on the fly
+ * Used to create a weighted random walks.
+ *
+ * @see <a href="cumulative.md">https://github.com/alongalky/iota-docs/blob/master/cumulative.md</a>
 */
 public class RecursiveWeightCalculator implements RatingCalculator {


Change name to CumulativeWeightCalculator

GalRogozinski · 2019-07-30T08:58:02Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

@@ -55,53 +53,49 @@ public RecursiveWeightCalculator(Tangle tangle, SnapshotProvider snapshotProvide

        // Estimated capacity per depth, assumes 5 minute gap in between milestones, at 3tps
        UnIterableMap<HashId, Integer> hashWeight = createTxHashToCumulativeWeightMap( 5 * 60 * 3 * depth);


you changed it in one place, but you didn't change it in the others

GalRogozinski · 2019-07-30T09:00:41Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+     * @param startingSet
+     * @param txToDirectApproversCache 
+     * @return
+     * @throws Exception


Fill in the @param and @return
If you decided to write a javadoc, do it correctly

GalRogozinski · 2019-07-30T09:01:08Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+     * @param requester
+     * @param stack
+     * @return
+     */


empty javadoc?

GalRogozinski · 2019-07-30T09:18:28Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+                appHashes = Collections.emptySet();
+            } else {
+                appHashes = approvers.getHashes();
+            }


I think it is cleaner to write it this way:

Collection<Hash> appHashes = (approvers == null || approvers.getHashes() == null) ? Collections.emptySet() : approvers.getHashes();

Only change if you agree, you don't have to. I care less about that. It is just less lines.

GalRogozinski · 2019-07-30T09:29:37Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+     */
+    private int getRating(Set<Hash> startingSet, Map<Hash, Set<Hash>> txToDirectApproversCache) throws Exception {
+        Deque<Hash> stack = new ArrayDeque<>(startingSet);
+        while (stack.isEmpty()) {


After seeing this test I ran the unit tests locally

They failed...

GalRogozinski · 2019-07-30T12:12:51Z

src/main/java/com/iota/iri/service/tipselection/impl/RecursiveWeightCalculator.java

+     * @return
+     * @throws Exception
+     */
+    private int getRating(Set<Hash> startingSet, Map<Hash, Set<Hash>> txToDirectApproversCache) throws Exception {


Change the name to getWeight.

Optional:
I think it will be cleaner to add + 1 when you return at the end instead. Or conversely instead of startingSet you can have startingHash or entryHash. But then you will have that getApprovers call that you did before in calculateRatingDfs. It is presumably not so bad because you have the cache and the code will become more readable (you get the weight of the tx in the param).

GalRogozinski · 2019-07-30T12:16:43Z

src/test/java/com/iota/iri/service/tipselection/impl/CumulativeWeightCalculatorTest.java

@@ -283,7 +283,6 @@ public void testTangleWithCircle2() throws Exception {
    }

    @Test
-    @Ignore


Are we using the prefix again?
How come this is passing?

When I asked before I just didn't check what this unit test does.
Then you reminded and convinced me that this should be ignored and probably deleted later once we finalize tipsel.

…gorithm

GalRogozinski

Just delete the link please

GalRogozinski · 2019-07-30T15:58:36Z

src/main/java/com/iota/iri/service/tipselection/impl/CumulativeWeightCalculator.java

+ * Implementation of {@link RatingCalculator} that calculates the cumulative weight 
+ * Calculates the weight recursively/on the fly for each transaction referencing {@code entryPoint}. <br>
+ * Works using DFS search for new hashes and a BFS calculation. 
+ * Uses cached values to prevent double database lookup for approvers
 *
 * @see <a href="cumulative.md">https://github.com/alongalky/iota-docs/blob/master/cumulative.md</a>


Delete this link

GalRogozinski · 2019-07-30T16:01:47Z

src/main/java/com/iota/iri/service/tipselection/impl/CumulativeWeightCalculator.java

        }

-        return hashWeight;
+        // If we have a circular reference, its already added, otherwise we save a big calculation


circular -> self referencing

GalRogozinski · 2019-07-30T16:02:46Z

src/main/java/com/iota/iri/service/tipselection/impl/CumulativeWeightCalculator.java

-        return hashWeight;
+        // If we have a circular reference, its already added, otherwise we save a big calculation
+        if (!hashWeightMap.containsKey(entryPoint)) {
+            hashWeightMap.put(entryPoint, hashWeightMap.size() + 1);


GalRogozinski · 2019-07-30T16:06:25Z

src/test/java/com/iota/iri/service/tipselection/impl/CumulativeWeightCalculatorTest.java

@@ -18,7 +18,6 @@
 import org.junit.AfterClass;
 import org.junit.Assert;
 import org.junit.BeforeClass;
-import org.junit.Ignore;


So we're still using transforming map
You want to avoid doing more tests for now?

Fine with that, we can change it later

jakubcech · 2019-07-31T12:17:49Z

🎉

Brord van Wierst added 2 commits May 13, 2019 21:01

added recursive weight calc

a419063

Added ignore to failed test and changed cum-weight test to recursive …

855ec6a

…weight impl

kwek20 requested a review from GalRogozinski May 13, 2019 19:09

GalRogozinski reviewed May 27, 2019

View reviewed changes

Brord van Wierst added 2 commits June 2, 2019 15:12

Changed to queue

9951d39

Added some map optimizations and decreased db usage

676eef4

GalRogozinski self-assigned this Jul 1, 2019

Brord van Wierst added 2 commits July 17, 2019 17:32

Implement recursive in codebase

2966d99

Merge branch 'dev' into daf-cw

ccd5536

GalRogozinski suggested changes Jul 18, 2019

View reviewed changes

Merge branch 'dev' into daf-cw

fbaf71d

GalRogozinski mentioned this pull request Jul 23, 2019

Test the DaF CW calculation against the current one #1428

Closed

3 tasks

Brord van Wierst and others added 4 commits July 24, 2019 15:54

Removed backup cache

10e3f16

Merge branch 'daf-cw' of https://github.com/kwek20/iri into daf-cw

77fde23

Removed recursiveness

ed951f1

Merge branch 'dev' into daf-cw

34ee23b

iotaledger deleted a comment Jul 28, 2019

GalRogozinski suggested changes Jul 28, 2019

View reviewed changes

Brord van Wierst added 3 commits July 29, 2019 16:26

Review comments

4953955

Merge branch 'dev' into daf-cw

525d687

Merge branch 'daf-cw' of https://github.com/kwek20/iri into daf-cw

dba6828

iotaledger deleted a comment Jul 29, 2019

GalRogozinski suggested changes Jul 30, 2019

View reviewed changes

Brord van Wierst added 2 commits July 30, 2019 16:35

Renamed and used Cum weight calc, added some javadoc, improved the al…

1ae6a80

…gorithm

single depth search, removed hasAll

4d067a1

kwek20 changed the title ~~Test: CW Calculation~~ Feature: Improved CW Calculation Jul 30, 2019

Brord van Wierst added 3 commits July 30, 2019 17:38

Simplified check

1b0aaff

Fixed circular -> self referencing

29c40a0

Updated description

babc42f

GalRogozinski reviewed Jul 30, 2019

View reviewed changes

Removed link

d410622

iotaledger deleted a comment Jul 30, 2019

Brord van Wierst added 2 commits July 31, 2019 04:35

Removed HashId need

dfa4599

Fixed Transformingmap usage in tests

3f5d176

GalRogozinski approved these changes Jul 31, 2019

View reviewed changes

GalRogozinski merged commit b60541d into iotaledger:dev Jul 31, 2019

jakubcech mentioned this pull request Aug 1, 2019

Release 1.8.1 #1556

Closed

@@ @@ -269,6 +283,7 @@ public void testTangleWithCircle2() throws Exception { @@
                   }
                   @Test
+                  @Ignore

		@@ -113,15 +107,14 @@ private Hash getAndRemoveApprover(Collection<Hash> appHashes) {
		* @return

		stack.push(entryPoint);

		while (CollectionUtils.isNotEmpty(stack)) {

		@@ -55,53 +53,49 @@ public RecursiveWeightCalculator(Tangle tangle, SnapshotProvider snapshotProvide

		// Estimated capacity per depth, assumes 5 minute gap in between milestones, at 3tps
		UnIterableMap<HashId, Integer> hashWeight = createTxHashToCumulativeWeightMap( 5 * 60 * 3 * depth);

@@ @@ -283,7 +283,6 @@ public void testTangleWithCircle2() throws Exception { @@
                   }
                   @Test
-                  @Ignore

Feature: Improved CW Calculation #1451

Feature: Improved CW Calculation #1451

Conversation

kwek20 commented May 13, 2019 • edited Loading

Description

Type of change

How Has This Been Tested?

Checklist:

GalRogozinski commented May 27, 2019

GalRogozinski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GalRogozinski commented Jul 18, 2019

kwek20 commented Jul 18, 2019

GalRogozinski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GalRogozinski commented Jul 21, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GalRogozinski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakubcech commented Jul 31, 2019

kwek20 commented May 13, 2019 •

edited

Loading