-
Notifications
You must be signed in to change notification settings - Fork 466
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Partial replication failure on ClueWeb12-B13 for ax models #592
Comments
Somehow the AX results matched after indexing again with oracle JVM (1.8.0_201).... |
Interesting. Hypothesis is that pseudo-random number generation works differently in OpenJDK vs. Oracle JVM. This would explain the differences in ax results, which rely on sampling docs. @matthew-z Just to be clear, with Oracle JVM you are able to replicate our figures exactly, while with OpenJDK you come close? (If so, how close is "close" in the latter case?) |
Did you mean no-AX algorithms? They matched exactly. E.g., 201-250, Bm25, NDCG@20, it got 0.12862 and expected 0.1286 201-250, QL, NDCG@20, it got 0.11067 and expected 0.1107 |
Okay, thanks! I'll send a PR to capture this discussion and ask you to review. |
@matthew-z please look at #594 |
Resolved with #594, closing. |
I am not sure, but this is unlikely IMO. |
Whatever the issue, Oracle vs. openJDK isn't the issue. All regressions run fine on
|
According to #590 @matthew-z wasn't able to replicate results with ax models on ClueWeb12-B13. We should look into this...
The text was updated successfully, but these errors were encountered: