v3.0.0rc1

[guolinke]

@StrikerRUS @jameslamb @Laurae2 @henry0312 @btrotta
please list PRs or issues that you think should be fixed in this new release.

[StrikerRUS]

As 3.0 major release is a big step forward, I'd like to propose the following TODOs:

PRs:

• refactor LGBM_DatasetGetFeatureNames #3022
• adding sparse support to TreeSHAP in lightgbm #3000
• Feat/optimize single prediction #2992
• [R-package] added support for first_metric_only (fixes #2368) #2912
• Fix add features  #2754
• Add CTR conversion for categorical features (with bundling support) #2463
• Less restrictive monotone constraints #2305 (last remained part)

New Features

• Capture LGBM model's .fit messages and redirect them to python logger #1493
• GPU support on ppc64le #2937
• Interaction constraints #2884
• Unify out results of LGBM_BoosterDumpModel and LGBM_BoosterSaveModel #2604
• Add some features to R-package that are already in C API and Python package (Add support for saving weight values of a node in the R-package #2281, [R-package] exclude training data from being checked for early stopping #2472, Refit method for R-package #2369, [R-package] Add support for non-ASCII feature names #2983)
• CRAN release of course [R-Package] CRAN issues #629

Fixes:

• All known GPU bugs (lightgbm.basic.LightGBMError: Bug in GPU histogram! split 11937: 12, smaller_leaf: 10245, larger_leaf: 1704 #2793, [gpu] /root/repo/LightGBM/compute/include/boost/compute/utility/wait_list.hpp:166: void boost::compute::wait_list::wait() const: Assertion `clWaitForEvents(size(), get_event_ptr()) == 0' failed. Aborted (core dumped) #2648, OpenCL Bus error #2550, GPU build fails for lightgbm target #2126, Training on GPU fails (OSError: exception: access violation) #1717, BUG in GPU histogram #1003)
• [mmlspark] Memory Corruption #2392
• JVM crash caused by LGBM_DatasetCreateFromCSRSpark #2363
• [R-package] Factor out custom R interface to lib_lightgbm #3016

[guolinke]

@StrikerRUS
I agree with you, except for the GPU bugs.
most of them are hard to reproduce and to fix. also ping @huanzhang12

[guolinke]

BTW, I am busy with the 'company work' recently.
so the new features assigned to me may delay, maybe we can move them to the next release (3.0.0rc2 or 3.0.0).
I will have more time after June.

[StrikerRUS]

@guolinke

I agree with you, except for the GPU bugs.
most of them are hard to reproduce and to fix. also ping @huanzhang12

OK, maybe at least it worth to revisit them? A lot of them haven't got any updates for months.

BTW, I am busy with the 'company work' recently.
so the new features assigned to me may delay, maybe we can move them to the next release

Sure! Are you speaking mostly about #2884, right?
From new features section, I personally really would like to see #1493 in next release. I remember you said

I think it is easy to implement.
#1493 (comment)

[StrikerRUS]

Also, I'd like to propose a release checklist which we can copy-paste from old releases and it will help us to not forget any required action. Not sure that it will be actual for RC updates though.

• Update VERSION number.
• Update version in Appveyor config file.
• Update version and release date in DESCRIPTION file of R-package.
• Push commit with v* tag to trigger GitHubRelease action at Azure Pipelines.
• Convert automatic GitHub release from Draft to normal one.
• Update stable tag at GitHub.
• Upload release to PyPI.
• Upload release to CRAN (soon).
• Upload release to NuGet.
• Update version and hash in Homebrew formula.

[guolinke]

refer to the version normalizing problem: pypa/setuptools#308

[henry0312]

I think it's time to drop suuport of Python 2.7, since it has reached the end of life.
cf. https://www.python.org/doc/sunset-python-2/
If users that use Python 2.7, they can use LightGBM v2.x

[henry0312]

And if you want, I'll create a PR to drop the support.

[StrikerRUS]

@henry0312

I think it's time to drop suuport of Python 2.7, since it has reached the end of life.

Please take a look at the following comment: #3056 (comment).
I propose that we mark 3.0.0 release as the latest one which supports Python 2, and stop supporting it with our next release (3.0.1 or 3.1.0 whatever will be the next after 3.0.0). I'm looking forward to hear from you what do you think about it!

[henry0312]

@StrikerRUS
Returning to the previous discussion about Semantic Versioning (SemVer) in #2949, increment of major version can only breaks backward compatibility.
When we drop the support of Python 2.x, we will break backward compatibility because we maintain some (dirty) codes (like

LightGBM/python-package/lightgbm/compat.py

Lines 13 to 18 in b60294b

	if is_py3:
	zip_ = zip
	string_type = str
	numeric_types = (int, float, bool)
	integer_types = (int, )
	range_ = range

) to support it.
Therefore, I think it is a just time to only support Python 3.x or later.
And I think as long as popular libraries (including numpy, scipy and pandas) support Python 2. the number of the users to use Python 2 won't decrease, that is not the future Python community hope.

[henry0312]

So if we need support of Python 2.x (I don't need any more), I propose we will release both v2.4.x that will be the last version to support it and v3.x that will support Python 3.5 or later.

[jameslamb]

Sorry, closing was an accident.

[jameslamb]

I agree with @StrikerRUS 's suggestions in #3071 (comment).

#629 is my primary focus, and I do think we should try to finish it for 3.0.0. In my experience more than half of the issues we receive about the R package are related to installation challenges, so I want the 3.0.0 release to come with easier install for R users.

Improving installation

• [R-Package] CRAN issues #629
• [R-package] Factor out custom R interface to lib_lightgbm #3016
• [R-package] [ci] Add tests on R 4.0 compatibility #3024
• [R-package] MSVC compiler warnings #3025
• [R-package] Add support for R 4.0 (fixes #3064, fixes #3024) #3065
• fix MSVC warning about control paths (fixes #3067) #3068

Improving documentation:

• [R-package] Rewrite R demos, replace with vignettes #1944

Simplifying the usage of the R package

• [R-package] Remove lgb.prepare() and lgb.prepare_rules() #3075

I just created #3075. I think we should take the opportunity of a new major release to remove lgb.prepare() and lgb.prepare_rules() and move people only to lgb.prepare2() and lgb.prepare_rules2().

[StrikerRUS]

@henry0312

When we drop the support of Python 2.x, we will break backward compatibility because we maintain some (dirty) codes

That's true.

And I think as long as popular libraries (including numpy, scipy and pandas) support Python 2. the number of the users to use Python 2 won't decrease, that is not the future Python community hope.

All these libraries (our dependencies) are dropping or have already dropped Python 2 support.
pandas:

Officially Python 3.6.1 and above, 3.7, and 3.8.
https://pandas.pydata.org/pandas-docs/stable/getting_started/install.html#python-version-support

numpy: https://numpy.org/neps/nep-0014-dropping-python2.7-proposal.html
scipy : https://www.scipy.org/scipylib/faq.html#do-numpy-and-scipytill-support-python-2-7

Also, from the plot I provided earlier, it can be observed that number of our Python 2 users constantly decreasing starting from January 2020:

But still it is several thousand downloads per day for now (~2k). For example daily downloads for macOS are never greater 400, but it is not the reason to not support macOS and stop building wheels for it.

I propose we will release both v2.4.x that will be the last version to support it and v3.x that will support Python 3.5 or later.

What commits do you want to include in 2.4.x branch?

[guolinke]

due to the problems in R's version number, I think the release will delay for an additional day.

[guolinke]

BTW, this cause the CI error.

* checking if this is a source package ... WARNING
Subdirectory ‘src’ contains:
  VERSION.txt
These are unlikely file names for src files.

[guolinke]

@jameslamb I am trying to de-couple R with version.txt.
For this:

LightGBM/R-package/configure.ac

Line 7 in 53c64ba

AC_INIT([lightgbm], [m4_esyscmd_s([cat src/VERSION.txt])], [], [lightgbm], [])

, is that possible to read the version number from Description?

[jameslamb]

@jameslamb I am trying to de-couple R with version.txt.
For this:

LightGBM/R-package/configure.ac

Line 7 in 53c64ba

AC_INIT([lightgbm], [m4_esyscmd_s([cat src/VERSION.txt])], [], [lightgbm], [])

, is that possible to read the version number from Description?

I think reading from DESCRIPTION is too complicated. I don't want to do YAML processing inside that configure script.

The documentation on how to update configure is here: https://github.com/microsoft/LightGBM/blob/master/R-package/README.md#changing-the-cran-package

But I can update it and fix the other warnings you're seeing right now! Can I push to this branch?

[jameslamb]

@guolinke I just pushed 5d79e77 which should fix the other issues with the R package. That hard-coding is fine for right now to not delay the release. I will improve the automation in a follow-up PR.

[StrikerRUS]

@guolinke Will RCs be a part of our regular development process? I'm asking because 3.0 looks like an edge case as 3.0 will never be published on CRAN (R-package is not ready yet as per #629). So I believe we can set VERSION to 3.0, then manually postprocess names of files for PyPI (add rc suffix) and bump the version for development (as we do after each release) to 3.0.1.

[guolinke]

@StrikerRUS
Actually, I think the new solution for R version number is better.
In previous, it depends on both the number in DESCRIPTION and version.txt, and version.txt is mainly used to locate the generate package file in tests. So it is safe to decouple it.
And using 3.0.0-1 is more uniform with 3.0.0rc1. Refer to https://semver.org/#spec-item-9 .

@jameslamb in this case, Maybe we can generate the version number to DESCRIPTION, instead of manually update it?
I mean, auto-fill the version number (and date) to DESCRIPTION in recreate-configure.sh.
Not in this release, we can enhance this latter.

[guolinke]

@StrikerRUS @jameslamb
I will merge this in about 2 hours. Do we have any other problems to discuss?

[StrikerRUS]

@guolinke

I will merge this in about 2 hours.

OK!

Are you planning to go through all items from #3071 (comment) (except updating stable tag of course)? I think RC release to PyPI will be enough. Not sure about NuGet though.

[jameslamb]

I'm ok with a release any time. Trying to fix #3280 for it right now.

For R, after this release, we could do this:

1. add a placeholder in DESCRIPTION like ~~VERSION~~
2. in build_r.R, and build-cran-package.sh, replace that with the version in VERSION.txt (and similar approach for Date)
3. add a similar placeholder to configure.ac, and have recreate-configure.sh and build-cran-package.sh replace it. That way, we never have to rely on having configure.ac read a file
4. All this code that reads VERSION.txt can replace any rc with -

If we did all of that, it could be totally automated and the only thing that has to be changed is VERSION.txt.

And I think we could create a Makefile with a release recipe that handles all these things.

[guolinke]

@StrikerRUS very strange, it seems azure pipeline does not build for tag now.

updated:
now it is working after re-push.

[guolinke]

Thanks all contributors so much! The new version is released! 🍺🍺🍺

The python package is uploaded. you can install it via pip install lightgbm==3.0.0rc1.

For the NuGet package, I think we can release it in the 3.0.0 version.
@StrikerRUS how about the homebrew version? should we update it now or wait for 3.0.0?

[StrikerRUS]

@guolinke

how about the homebrew version?

I think PyPI is enough for and homebrew can wait for the stable version.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.