ray-packages v2.1.0

ray-patching.md

@@ -1,53 +1,116 @@
+# How to deal with patches in this feedstock/recipe/
+
+### Background
+This feedstock needs to permanently carry patches, because
+some of the things we need to adapt for the infrastructure of
+conda-forge are not necessarily suitable for integration in
+upstream ray
+
+### Default Workflow
+
+The main reason for touching these patches is of course when ray
+releases a new version and the patches (may) fail to apply cleanly.
+
+The default workflow is to keep a source-checkout of ray with a branch
+containing the necessary patches for conda-forge. Once ray releases a new
+version, it's (more or less) simply a matter or rebasing that branch and
+and fixing the conflicts.
+
+Initial setup:
+* `git clone https://github.com/ray-project/ray`
+* `cd ray`
+* `git tag --list`
+* `git checkout tags/ray-<old_version>`
+* `git checkout -b conda`
+* apply all the patches from the current feedstock (which are known to be good);
+  since they're sorted in the `patches/` folder, you can even use `xargs`, e.g.
+```sh
+ls -d ../path/to/feedstock/recipe/patches/ | xargs git am
+```
+
+Rebase to new version or ray:
+* `git rebase -i tags/ray-<new_version>` (the `-i` is optional, but you can
+  use it to drop patches that you know where upstreamed even if git doesn't auto-recognize them)
+* Fix eventual conflicts, taking care to keep the original patch attribution
+  (if you make substantial changes, add a `Co-Authored-By: ...`)
+  * I.e. if git tells you it couldn't apply commit `deadbeef`, then, after fixing the conflicts, do
+  * `git add .`
+  * `git show deadbeef`
+  * `git commit --date="<date_of_deadbeef>" --author="<author_of_deadbeef>"` (commit message is kept as-is automatically while rebasing)
+  * `git rebase --continue`
+* `git format-patch tags/ray-<new_version>`
+* remove all patches in the `patches`  folder
+* copy patches produced by `git format-patch` over to recipe and update `meta.yaml`
+
 ## How to prepare or adapt patches for third party Ray components (using Redis as the example)
 
+The above process is complicated by the fact that ray has patches for third party
+dependencies (like redis) in its source code - in other words, we need to patch
+the patches.
+
+The process we just described then repeats for each dependency we need to patch
+more or less as-is, with the exception that we now need to check in the patches
+to third-party code into the ray source-code, and include _that_ in the ray-patches
+we check into the feedstock. For added fun, it may happen that upstream ray changes
+their patches from version to version in ways that conflict with ours.
+
+Since patch application within the ray build is done by bazel (and may be subject
+to destructive options like `-pN`, which changes the file hierarchy level the
+patches apply to), it's best - for single patches at least - to extend the
+ray patch directly to suit our purposes. This makes it also much clearer (well...)
+when reviewing the diff(-of-the-diff-of-the-diff) on the feedstock.
+
 ### Prepare sources
-1. Get Ray repository (using Ray 1.1.0 for example)
+1. Get Ray repository (using Ray 2.1.0 for example)
 ```sh
 git clone https://github.com/ray-project/ray.git
 cd ray
-git checkout -b ray-1.1.0-original ray-1.1.0
+git checkout -b ray-2.1.0-original ray-2.1.0
 # apply recipe patches
 git am /ray-packages-feedstock/recipe/patches/*
 cd ..
 ```
-2. Find out repository and version needed (see `ray/bazel/ray_deps_setup.bzl`), clone it and checkout (taking version 6.0.9 for example):
+2. Find out repository and version needed (see `ray/bazel/ray_deps_setup.bzl`), clone it and checkout (taking version 7.0.5 for example):
 ```sh
 git clone  https://github.com/redis/redis
 cd redis
-git checkout -b 6.0.9-original 6.0.9
+git checkout -b 7.0.5-patched 7.0.5
 ```
-3. Apply relevant upstream patches first (1.1.0 has only one patch) and commit them:
+3. Apply relevant upstream patches first (2.1.0 has only one patch) and commit them:
 ```sh
-patch -p0 < patch -p0 < ../ray/thirdparty/patches/redis-quiet.patch
+patch -p1 < ../ray/thirdparty/patches/redis-quiet.patch
 git add -u
 git commit -m 'Upstream Ray patches'
 ```
 
 ### Make changes
 1. Make new branch and make the necessary changes (maybe adapting existing conda recipe patches, etc.)
 ```sh
-git checkout -b 6.0.9-patched
 # do the necessary changes - edit files, etc.
 git commit -m 'Fix the issue'
+# if ray carries a single commit, squash it (and our changes) into one by interactive rebasing
+git rebase -i 7.0.5
 ```
-2. Prepare the patch for Redis in Ray sources:
+2. Prepare the patch for Redis in Ray sources (note that if the patch application
+in bazel does not have a `-p1`, the call to `git format-patch` below also
+needs a `--no-prefix` option):
 ```
-git format-patch 6.0.9-original..6.0.9-patched --no-prefix --stdout > ../ray/thirdparty/patches/redis-deps-ar.patch
+git format-patch 7.0.5-patched..7.0.5 --stdout > ../ray/thirdparty/patches/redis-quiet.patch
 ```
 3. Commit the patch for Redis:
 ```sh
 cd ../ray
-git checkout -b ray-1.1.0-patched
-git add thirdparty/patches/redis-deps-ar.patch
-# edit ray/bazel/ray_deps_setup.bzl so it references this patch
+git checkout -b ray-2.1.0-patched
+# edit ray/bazel/ray_deps_setup.bzl so it includes your patch
+# if necessary, make sure it uses -p1 (or -p0 if patch was made with --no-prefix)
 git add -u
 git commit -m 'Fix the issue by Redis patch'
 ```
 
 ### Publish patch to recipe
 1. Take commits as patches to the recipe:
 ```sh
-git format-patch ray-1.1.0..ray-1.1.0-patched --output-directory /ray-packages-feedstock/recipe/patches/
+git format-patch ray-2.1.0..ray-2.1.0-patched --output-directory /ray-packages-feedstock/recipe/patches/
 ```
 2. Add the patches to the recipe:
 ```sh

recipe/meta.yaml

@@ -1,12 +1,12 @@
-{% set version = "2.0.1" %}
+{% set version = "2.1.0" %}
 
 package:
   name: ray-packages
   version: {{ version }}
 
 source:
   url: https://github.com/ray-project/ray/archive/ray-{{ version }}.tar.gz
-  sha256: b8b2f0a99d2ac4c001ff11c78b4521b217e2a02df95fb6270fd621412143f28b
+  sha256: 61d2ba82fe0d61d6d85394fff72766e5439c4e1ba25924ba8eadf38928fef463
   patches:
     - patches/0001-Redis-deps-now-build-but-do-not-link.patch
     - patches/0002-Disable-making-non-core-entry-scripts.patch
@@ -16,7 +16,7 @@ source:
 
 
 build:
-  number: 1
+  number: 0
   # skip on MacOS, needs macos 10.15
   skip: true  # [osx]
 
@@ -68,21 +68,21 @@ outputs:
         - python
         - aiosignal
         - attrs
-        - click >=7.0, <=8.0.4
+        - click >=7.0
         - colorama
         - filelock
         - frozenlist
-        # We override the 1.43.0 upper-bound in setup.py here,
-        # see https://github.com/ray-project/ray/issues/28354
-        - grpcio >=1.32.0,<1.48
+        # mattip/h-vetinari [2022/11]: unclear problems with conda-forge's 1.49
+        - grpcio >=1.42,<1.49
         - jsonschema
-        - msgpack-python >=1.0.0, <2.0.0
+        - msgpack-python >=1.0.0
         - numpy >=1.20
-        - protobuf >=3.15.3, <4.0.0
+        - packaging  # [py>=310]
+        - protobuf >=3.15.3,!=3.19.5
         - psutil
         - pyyaml
         - requests
-        - setproctitle =1.2.2
+        - setproctitle
         - virtualenv
 
     test:
@@ -94,7 +94,7 @@ outputs:
         - ray._private.state
         - ray._private.worker
       commands:
-        - python -c "import ray; ray.init()"
+        - python -c "import ray; ray.init(include_dashboard=False)"
 
   - name: ray-default
     requirements:
@@ -106,7 +106,7 @@ outputs:
         - aiohttp >=3.7
         - aiohttp-cors
         - colorful
-        - gpustat
+        - gpustat >=1.0.0
         - opencensus
         - prometheus_client >=0.7.1, <0.14.0
         - pydantic
@@ -212,7 +212,7 @@ outputs:
       run:
         - python
         - {{ pin_subpackage('ray-core', exact=True) }}
-        - uvicorn =0.16.0
+        - uvicorn
         - requests
         - starlette
         - fastapi
@@ -270,7 +270,9 @@ about:
   summary: Ray is a fast and simple framework for building and running distributed applications.
   description: |
     Ray is a fast and simple framework for building and running
-    distributed applications.
+    distributed applications. It is split into ray-core, ray-default,
+    ray-dashboard, ray-serve, ray-rllib, ray-k8s, ray-data, ray-tune, and
+    ray-all packages.
   doc_url: https://ray.readthedocs.io/
   dev_url: https://github.com/ray-project/ray