Partial replication failure on ClueWeb12-B13 for ax models

[lintool]

According to #590 @matthew-z wasn't able to replicate results with ax models on ClueWeb12-B13. We should look into this...

[matthew-z]

Somehow the AX results matched after indexing again with oracle JVM (1.8.0_201)....

[lintool]

Interesting.

Hypothesis is that pseudo-random number generation works differently in OpenJDK vs. Oracle JVM. This would explain the differences in ax results, which rely on sampling docs.

@matthew-z Just to be clear, with Oracle JVM you are able to replicate our figures exactly, while with OpenJDK you come close? (If so, how close is "close" in the latter case?)

[matthew-z]

Did you mean no-AX algorithms? They matched exactly.

E.g., 201-250, Bm25, NDCG@20, it got 0.12862 and expected 0.1286

201-250, QL, NDCG@20, it got 0.11067 and expected 0.1107

[lintool]

Okay, thanks! I'll send a PR to capture this discussion and ask you to review.

[lintool]

@matthew-z please look at #594

[lintool]

Resolved with #594, closing.

[Peilin-Yang]

Hypothesis is that pseudo-random number generation works differently in OpenJDK vs. Oracle JVM.

I am not sure, but this is unlikely IMO.
I am going to test it again by myself

[lintool]

Whatever the issue, Oracle vs. openJDK isn't the issue. All regressions run fine on damiano:

[jimmylin@damiano ~]$ java --version
openjdk 11.0.4 2019-07-16 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.4+11-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.4+11-LTS, mixed mode, sharing)