[Bug] Lengthscale change with and without normalization

[Talheus]

🐛 Bug

I want to fit a Gaussian process over samples. After optimization, the lengthscale seems strange. I tried normalizing it and it appeared that normalization change lengthscale which is not expected.

To reproduce

** Code snippet to reproduce **

from gpytorch.mlls import ExactMarginalLogLikelihood

from botorch.models import SingleTaskGP
from botorch import fit_gpytorch_model

from torch import tensor

train_X = tensor([[0.3660, 0.7463],
                  [0.8714, 0.4299],
                  [0.5104, 0.0620],
                  [0.1276, 0.9511]], dtype=float)

train_y = tensor([10000, 11000, 12000, 20000], dtype=float).reshape(-1, 1)
train_yvar = tensor([100, 200, 150, 100], dtype=float).reshape(-1, 1)

gpr = SingleTaskGP(train_X=train_X,
                   train_Y=train_y,
                   train_Yvar=train_yvar
                   )

mll = ExactMarginalLogLikelihood(likelihood=gpr.likelihood,
                                 model=gpr
                                 )
print(gpr.covar_module.base_kernel.lengthscale)
fit_gpytorch_model(mll=mll)

print(gpr.covar_module.base_kernel.lengthscale)

mean = 1.000001#train_y.mean()
std = 1 #train_y.std()

gpr = SingleTaskGP(train_X=train_X,
                   train_Y=(train_y - mean) / std,
                   train_Yvar=train_yvar / std**2
                   )

mll = ExactMarginalLogLikelihood(likelihood=gpr.likelihood,
                                 model=gpr
                                 )
fit_gpytorch_model(mll=mll)

print(gpr.covar_module.base_kernel.lengthscale)

** Stack trace/error message **

tensor([[0.6931, 0.6931]], dtype=torch.float64, grad_fn=<SoftplusBackward0>)
tensor([[0.0622, 1.0933]], dtype=torch.float64, grad_fn=<SoftplusBackward0>)
tensor([[0.3289, 0.0342]], dtype=torch.float64, grad_fn=<SoftplusBackward0>)

Expected Behavior

We expect the lengthscale to be independent of the chosen normalization.

System information

Please complete the following information:

• BoTorch Version 0.10.0
• BoTorch Version 1.11
• PyTorch Version 2.3.0+cu118
• Computer OS : Windows / Linux

Additional context

Add any other context about the problem here.

[esantorella]

Standardizing data makes a difference because SingleTaskGP has a prior on the output scale. It's a Gamma(2, 0.15) prior, which you can access at gpr.covar_module.outputscale_prior. The output scale controls how much the function varies when you move from a bad x value to a good one. We generally recommend standardizing data by passing the Standardize transform, because this prior has worked well when the standard deviation of the outputs is around 1. This GPyTorch page on hyperparameters may be helpful.

Let's delve into your example. y varies by 10,000. This can't be explained by noise, because there is relatively little noise, but it also is hard to explain with variation in f because of the prior that f should not vary so much. Here's a slightly modified version of your code; model-fitting struggles, and with a different seed, it may fail entirely or give a different result. I also changed it to use double-precision data, which generally works better, and to use fit_gpytorch_mll, since fit_gpytorch_model no longer exists in the most recent version of BoTorch.

import warnings

from gpytorch.mlls import ExactMarginalLogLikelihood

from botorch.models import SingleTaskGP
from botorch import fit_gpytorch_mll
from botorch.models.transforms.outcome import Standardize

from torch import tensor
import torch
import numpy as np


train_X = tensor(
    [
        [0.3660, 0.7463],
        [0.8714, 0.4299],
        [0.5104, 0.0620],
        [0.1276, 0.9511]
    ],
    dtype=torch.float64,
)
train_y = tensor([10000, 11000, 12000, 20000], dtype=torch.float64).reshape(-1, 1)
train_yvar = tensor([100, 200, 150, 100], dtype=torch.float64).reshape(-1, 1)

gpr = SingleTaskGP(
    train_X=train_X,
    train_Y=train_y,
    train_Yvar=train_yvar
)

mll = ExactMarginalLogLikelihood(
    likelihood=gpr.likelihood,
    model=gpr
)
torch.manual_seed(0)  # model fitting fails entirely for some seeds
_ = fit_gpytorch_mll(mll=mll)

This prints out a bunch of warnings. One recommend standardizing the data:

.../botorch/models/utils/assorted.py:260: InputDataWarning: Data is not standardized (std = tensor([4573.4742], dtype=torch.float64), mean = tensor([13250.], dtype=torch.float64)). Please consider scaling the input to zero mean and unit variance.
  check_standardization(Y=train_Y, raise_on_fail=raise_on_fail)

Then there are several warnings that fitting the model is not going well:

OptimizationWarning: `scipy_minimize` terminated with status OptimizationStatus.FAILURE, displaying original message from `scipy.optimize.minimize`: ABNORMAL_TERMINATION_IN_LNSRCH

I think that inconsistency in model fit explains the variability you're seeing rather than the (relatively small) change in the data. I'm getting a variety of lengthscales when I run this with different seeds.

When I use a Standardize transform, there are no more errors and the results are the same regardless of the initial seed:

gpr = SingleTaskGP(
    train_X=train_X,
    train_Y=train_y,
    train_Yvar=train_yvar,
    outcome_transform=Standardize(m=1)
)

[Talheus]

Thank you for your explanation.