XGBoost dist=gamma reports NA performance metrics

[exalate-issue-sync]

For the following example, the deviance grows with each tree until validation metrics report NA's. I was able to reproduce this on a single or multi-node cluster on 3.26.0.2.

{code:python}
df = h2o.import_file("hdfs://mr-0xd6.0xdata.loc:8020/datasets/airlines_all.05p.csv")

df["Year"] = (df["Year"] < 2000).ifelse(0, df["Year"])
train, valid = df.split_frame(seed=1234)

from h2o.estimators.xgboost import H2OXGBoostEstimator

xgb = H2OXGBoostEstimator(model_id="xgb1.hex",
col_sample_rate=0.7,
stopping_rounds=5,
stopping_metric="deviance",
stopping_tolerance=0.00001,
distribution="gamma",
ntrees=1000,
score_tree_interval=5,
)
xgb.train(training_frame=train, validation_frame=valid, y="Year")
{code}

[exalate-issue-sync]

Megan Kurka commented: Possibly related: [https://github.com/dmlc/xgboost/issues/5349|https://github.com/dmlc/xgboost/issues/5349|smart-link] ?

[exalate-issue-sync]

Megan Kurka commented: Note: H2O GBM with Gamma converges for me

[exalate-issue-sync]

Jan Sterba commented: reproduced with native xgboost, updated issue linked above with our finding, not much more we can do

[exalate-issue-sync]

Neema Mashayekhi commented: Reopening till native XGB is confirmed that this is the case. May be able to find workaround from dmlc

[exalate-issue-sync]

Neema Mashayekhi commented: [~accountid:5c2ff866e7af4d69171f81d9] - have you seen this before? Any ways to prevent it?

[exalate-issue-sync]

Jan Sterba commented: fixed upstream, will add to our builds [https://github.com/dmlc/xgboost/pull/6537|https://github.com/dmlc/xgboost/pull/6537|smart-link]

[exalate-issue-sync]

Jan Sterba commented: tried to verify the upstream fix, but it did not work

reported a new issue here: [https://github.com/dmlc/xgboost/issues/6723|https://github.com/dmlc/xgboost/issues/6723|smart-link]

[h2o-ops]

JIRA Issue Migration Info

Jira Issue: PUBDEV-7810
Assignee: Adam Valenta
Reporter: Megan Kurka
State: Open
Fix Version: N/A
Attachments: Available (Count: 1)
Development PRs: N/A

Attachments From Jira

Attachment Name: pyunit_gamma_zero_metrics.py
Attached By: Jan Sterba
File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-7810/pyunit_gamma_zero_metrics.py