Questions about OGB-LSC

[YueFan1014]

Hello! I am interested in your OGB-LSC NeurIPS 2022, and I have several questions about the competition:

1. Will I get the result immediately if I submit my prediction? If not, how often is the leaderboard updated?

2. Is the public leaderboard sharing the same testing set with the competition?

3. Is the public leaderboard reflecting the up-to-date competition ranking?

Hoping for your reply, thanks!

[weihua916]

Hi! Thanks for your interest. These are great questions.

At high-level, the public leaderboard evaluates on the test-dev dataset, while NeurIPS competition 2022 will evaluate on the test-challenge dataset. They are separate things. Given this, below are the answers to your questions.

1. It will usually take a week for us to verify your submission to the public leaderboard. It involves making sure your technical report and Github repository contain enough information to reproduce your results.
2. No, the public leaderboard uses test-dev set.
3. The rank in the public leaderboard and the winner selection of the NeurIPS competition are completely independent. We do not have any leaderboards for the NeurIPS competition. For NeurIPS, we just evaluate the final submission on the test-challenge set once and announce the winners.

That being said, any NeurIPS competition participants are welcomed to submit their results on the public leaderboard, but the submission needs to satisfy leaderboard policies (e.g., public code, technical report required), see here for more details.

Hope this answers your questions!

[YueFan1014]

Thank you for your reply! It clarifies most of my concerns. But I have another question: how is the test-dev and the test-challenge sampled? For example, on the WikiKG90Mv2, Are the edges in test-dev and the test-challenge sampled by relation types or by the out-degrees of the heads? Are they both sampled from the same distribution? Thanks!

[weihua916]

Yes, test-dev and test-challenge follow the same distribution.

[YueFan1014]

OK, thanks!