Remove outliers when calculating BSQ rate #4706

chimp1984 · 2020-10-25T23:38:17Z

Remove outliers when calculating BSQ rate.
Add percentage for upper and lower threshold for outlier detection.
It filters outliers in the BSQ and in the USD market.

Settings:

As one can see below even a small % threshold removes the few outliers we had recently: 5 % is default value.

With 0%:

With 1%:

With 5%:

With 10%:

With 20%:

With 49% (max):

Add percentage for upper and lower threshold for outlier detection.

ghubstan · 2020-10-29T20:56:27Z

When I click the DAO menu, I get a jfoenix / reflection related NPE:

Oct-29 17:48:40.603 [JavaFX Application Thread] ERROR bisq.common.setup.CommonSetup: Uncaught Exception from thread JavaFX Application Thread 
Oct-29 17:48:40.603 [JavaFX Application Thread] ERROR bisq.common.setup.CommonSetup: throwableMessage= null 
Oct-29 17:48:40.603 [JavaFX Application Thread] ERROR bisq.common.setup.CommonSetup: throwableClass= class java.lang.NullPointerException 
Oct-29 17:48:40.604 [JavaFX Application Thread] ERROR bisq.common.setup.CommonSetup: Stack trace:
java.lang.NullPointerException
	at bisq.desktop.components.JFXTextFieldSkinBisqStyle.updateTextPos(JFXTextFieldSkinBisqStyle.java:95)
	at bisq.desktop.components.JFXTextFieldSkinBisqStyle.layoutChildren(JFXTextFieldSkinBisqStyle.java:79)
	at javafx.scene.control.Control.layoutChildren(Control.java:601)
	at javafx.scene.Parent.layout(Parent.java:1204)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Scene.doLayoutPass(Scene.java:576)
	at javafx.scene.Scene$ScenePulseListener.pulse(Scene.java:2482)
	at com.sun.javafx.tk.Toolkit.lambda$runPulse$2(Toolkit.java:412)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:389)
	at com.sun.javafx.tk.Toolkit.runPulse(Toolkit.java:411)
	at com.sun.javafx.tk.Toolkit.firePulse(Toolkit.java:438)
	at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:519)
	at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:499)
	at com.sun.javafx.tk.quantum.QuantumToolkit.pulseFromQueue(QuantumToolkit.java:492)
	at com.sun.javafx.tk.quantum.QuantumToolkit.lambda$runToolkit$11(QuantumToolkit.java:320)
	at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
	at com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
	at com.sun.glass.ui.gtk.GtkApplication.lambda$runLoop$11(GtkApplication.java:277)
	at java.base/java.lang.Thread.run(Thread.java:835)

It's happened after three startups -- two of them on a fresh data directory. I do think it is related to the current jfoenix-9.0.6.jar imcompatibility with JDK 11+, in combination with this change.

I am using OpenJDK 12 on Ubuntu 12.

Related PRs: #4443 #4463 #4242

chimp1984 · 2020-10-30T00:17:03Z

Hm... I am not aware of any change which would be related to jfoenix... Can you try if you get the exception also on master?
It can be that there is a nullpointer or any exception thrown and the UI unfortunatly does not provide useful stacktrace, thats common with javafx ;-(

ripcurlx · 2020-10-30T09:43:19Z

When I click the DAO menu, I get a jfoenix / reflection related NPE:

Oct-29 17:48:40.603 [JavaFX Application Thread] ERROR bisq.common.setup.CommonSetup: Uncaught Exception from thread JavaFX Application Thread 
Oct-29 17:48:40.603 [JavaFX Application Thread] ERROR bisq.common.setup.CommonSetup: throwableMessage= null 
Oct-29 17:48:40.603 [JavaFX Application Thread] ERROR bisq.common.setup.CommonSetup: throwableClass= class java.lang.NullPointerException 
Oct-29 17:48:40.604 [JavaFX Application Thread] ERROR bisq.common.setup.CommonSetup: Stack trace:
java.lang.NullPointerException
	at bisq.desktop.components.JFXTextFieldSkinBisqStyle.updateTextPos(JFXTextFieldSkinBisqStyle.java:95)
	at bisq.desktop.components.JFXTextFieldSkinBisqStyle.layoutChildren(JFXTextFieldSkinBisqStyle.java:79)
	at javafx.scene.control.Control.layoutChildren(Control.java:601)
	at javafx.scene.Parent.layout(Parent.java:1204)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Parent.layout(Parent.java:1211)
	at javafx.scene.Scene.doLayoutPass(Scene.java:576)
	at javafx.scene.Scene$ScenePulseListener.pulse(Scene.java:2482)
	at com.sun.javafx.tk.Toolkit.lambda$runPulse$2(Toolkit.java:412)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:389)
	at com.sun.javafx.tk.Toolkit.runPulse(Toolkit.java:411)
	at com.sun.javafx.tk.Toolkit.firePulse(Toolkit.java:438)
	at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:519)
	at com.sun.javafx.tk.quantum.QuantumToolkit.pulse(QuantumToolkit.java:499)
	at com.sun.javafx.tk.quantum.QuantumToolkit.pulseFromQueue(QuantumToolkit.java:492)
	at com.sun.javafx.tk.quantum.QuantumToolkit.lambda$runToolkit$11(QuantumToolkit.java:320)
	at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
	at com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
	at com.sun.glass.ui.gtk.GtkApplication.lambda$runLoop$11(GtkApplication.java:277)
	at java.base/java.lang.Thread.run(Thread.java:835)

It's happened after three startups -- two of them on a fresh data directory. I do think it is related to the current jfoenix-9.0.6.jar imcompatibility with JDK 11+, in combination with this change.

I am using OpenJDK 12 on Ubuntu 12.

Related PRs: #4443 #4463 #4242

This is a jFoenix issue with OpenJDK 12 quite sure. Should be gone when #4242 is merged and we use the newer jFoenix version.

ghubstan · 2020-10-30T13:34:47Z

I tried your PR's branch again, this time downgrading from JDK 12 to JDK 11, and there was no NPE. I got the same results in the master branch. Clicking the DAO menu item in:

Chimp's PR branch
JDK 11	12
    OK  JFOENIX NPE

Master Branch
JDK 11	12
    OK  JFOENIX NPE

sqrrm · 2020-10-31T11:16:39Z

@ghubstan Since it works on JDK 11, do you give this an ack?

ghubstan · 2020-10-31T15:12:37Z

@ghubstan Since it works on JDK 11, do you give this an ack?

ACK

cbeams · 2020-11-03T14:17:29Z

I'm not seeing a difference in BSQ rates displayed on DAO > Facts & Figures > Dashboard between v1.4.2 and latest master as of this writing (00725cf).

This is from v1.4.2:

This is from master @ 00725cf:

Note that the Market cap and Total available BSQ values do differ. Not sure if that's related to this change.

cbeams · 2020-11-03T14:40:22Z

I also noticed that when running latest master against a fresh data directory (i.e. when emulating a brand-new user's first run of Bisq), that the Facts & Figures Dashboard what appear to be incomplete data:

This is v1.4.2 running against a clean data dir:

And this is latest master against a clean data dir:

Notice how under latest master both BSQ/BTC and USD/BSQ the 30- and 90-day values are identical, and how few data points there are on the chart. Is this a known issue?

chimp1984 · 2020-11-03T16:21:22Z

Thanks for finding the issue. Its caused by #4628. Working on a bugfix....

chimp1984 · 2020-11-03T16:54:34Z

Fixed with #4745

cbeams · 2020-11-04T11:40:51Z

Fixed with #4745

Thanks, @chimp1984. As I just mentioned at #4745 (comment), I've tested this locally, and I can see that the fix works.

However, there is still a problem that goes back to my original comment on this PR above at #4706 (comment).

The following screenshots are of two different Bisq instances running against latest master at 63cae1c, i.e. with PR #4745 merged. Notice the discrepancy between the average trade price values:

This is master running against an fresh data directory:

And this is master running against my actual data directory, i.e. my personal Bisq client as I usually run it:

The former appears to have the outliers removed as intended, while the latter does not. The latter displays roughly the same values as it did against v1.4.2.

Note that DAO synchronization is complete on both of the above clients.

chimp1984 · 2020-11-04T15:42:48Z

@cbeams Can you check at settings/dao options what the outlier % is there? Maybe it got set to 0 at the update and if its 0 it does not do anything. 5% is default. I tested with fresh apps and it worked that 5% was initial value.

cbeams · 2020-11-04T16:25:46Z

Can you check at settings/dao options what the outlier % is there? Maybe it got set to 0 at the update and if its 0 it does not do anything. 5% is default.

Yes, this was the problem. It was set to 0% in my existing Bisq app after the upgrade. I manually set it to 5% and got the expected behavior.

chimp1984 · 2020-11-04T16:27:13Z

Hm... need to test again for the update scenario, it should take the 5% by default as well but might be a bug

chimp1984 · 2020-11-04T16:36:25Z

Yes was a bug, fixed in #4747

Remove outliers when calculating BSQ rate

8bfe9b8

Add percentage for upper and lower threshold for outlier detection.

This was referenced Oct 25, 2020

Change way how we determine the BSQ rate for a DAO cycle bisq-network/proposals#270

Closed

BSQ rates voting for cycle 19 bisq-network/proposals#271

Closed

For Cycle 18 bisq-network/compensation#687

Closed

ghubstan approved these changes Oct 31, 2020

View reviewed changes

sqrrm approved these changes Oct 31, 2020

View reviewed changes

sqrrm merged commit 89620a0 into bisq-network:master Oct 31, 2020

chimp1984 deleted the remove-outliers-at-bsq-rate branch October 31, 2020 23:23

ripcurlx added this to the v1.5.0 milestone Nov 3, 2020

chimp1984 mentioned this pull request Nov 3, 2020

Fix bug with reading historical data #4745

Merged

chimp1984 mentioned this pull request Nov 4, 2020

Set initial value for bsqAverageTrimThreshold at update to 5% #4747

Merged

ghubstan mentioned this pull request Nov 24, 2020

For Cycle 19 bisq-network/compensation#716

Closed

MwithM mentioned this pull request Nov 27, 2020

BSQ rates for Cycle 20 bisq-network/proposals#282

Closed

MwithM mentioned this pull request Dec 28, 2020

BSQ rates for cycle 21 bisq-network/proposals#289

Closed

MwithM mentioned this pull request Feb 28, 2021

BSQ rates for cycle 23 bisq-network/proposals#314

Closed

MwithM mentioned this pull request Jun 10, 2022

BSQ rate for Cycle 37 is 1.03 USD per BSQ bisq-network/compensation#1092

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove outliers when calculating BSQ rate #4706

Remove outliers when calculating BSQ rate #4706

chimp1984 commented Oct 25, 2020 •

edited

Loading

ghubstan commented Oct 29, 2020

chimp1984 commented Oct 30, 2020

ripcurlx commented Oct 30, 2020

ghubstan commented Oct 30, 2020

sqrrm commented Oct 31, 2020

ghubstan commented Oct 31, 2020

cbeams commented Nov 3, 2020

cbeams commented Nov 3, 2020

chimp1984 commented Nov 3, 2020

chimp1984 commented Nov 3, 2020

cbeams commented Nov 4, 2020

chimp1984 commented Nov 4, 2020

cbeams commented Nov 4, 2020

chimp1984 commented Nov 4, 2020

chimp1984 commented Nov 4, 2020

Remove outliers when calculating BSQ rate #4706

Remove outliers when calculating BSQ rate #4706

Conversation

chimp1984 commented Oct 25, 2020 • edited Loading

ghubstan commented Oct 29, 2020

chimp1984 commented Oct 30, 2020

ripcurlx commented Oct 30, 2020

ghubstan commented Oct 30, 2020

sqrrm commented Oct 31, 2020

ghubstan commented Oct 31, 2020

cbeams commented Nov 3, 2020

cbeams commented Nov 3, 2020

chimp1984 commented Nov 3, 2020

chimp1984 commented Nov 3, 2020

cbeams commented Nov 4, 2020

chimp1984 commented Nov 4, 2020

cbeams commented Nov 4, 2020

chimp1984 commented Nov 4, 2020

chimp1984 commented Nov 4, 2020

chimp1984 commented Oct 25, 2020 •

edited

Loading