Fitting with data including y-error

[anir92]

My data involves y-error. How do I incorporate the error in y values in the fitting? Should I take --> weight = 1/y_error ?
The problem arises when my data set contains few y values and corresponding y_error values equal to zero. If y_error = 0, then corresponding weight is coming to be infinity. How to fit the data in this case ?

[cjekel]

How are you getting those errors where you have some data with and without error?

You could just do a very large weight to the values which have 0 error. You could play with a couple different values to see if it effects the results. I would say a couple orders of magnitude larger than the other errors

Linking weighted least squares for reference.

[anir92]

Thanks for the suggestion. I have looked deep into the data which actually comes from a measurement of a CCD detecting photons. I now understand that the detector was OFF during those instances (so both y (counts) =0 and y_err=0). So, I am either going to discard those data points or rebin the data set.

[anir92]

I have another query. Can you please inform how "Standard_error" is calculating the error on the breakpoints ? The description says "Calculate the standard errors for each beta parameter determined from the piecewise linear fit". Thanks in advance.

[cjekel]

https://jekel.me/piecewise_linear_fit_py/pwlf.html#pwlf.PiecewiseLinFit.standard_errors

One approximation that is commonly used for non-linear models is the 'delta method' or finite differences. I basically evaluate how sensitive the model is to the breakpoint locations. It is technically a first order taylor series expansion of the non-linear model.

This is the the paper that details the method https://mae.ufl.edu/nkim/Papers/paper54.pdf

Here is another reference on the delta method https://stats.idre.ucla.edu/r/faq/how-can-i-estimate-the-standard-error-of-transformed-regression-parameters-in-r-using-the-delta-method/

[cjekel]

I guess I should add that the delta method is only used when method='non-linear'