Merge remote-tracking branch 'origin/main' into migrate_flakes

Signed-off-by: Matt Lord <mattalord@gmail.com>

.github/workflows/static_checks_etc.yml

@@ -102,6 +102,7 @@ jobs:
             release_notes:
               - 'changelog/**'
               - './go/tools/releases/**'
+              - '.github/workflows/static_checks_etc.yml'
 
       - name: Set up Go
         if: steps.skip-workflow.outputs.skip-workflow == 'false' && (steps.changes.outputs.go_files == 'true' || steps.changes.outputs.parser_changes == 'true' || steps.changes.outputs.proto_changes == 'true')
@@ -218,3 +219,4 @@ jobs:
           echo 'Running `go run ./go/tools/releases/releases.go` on CI yields the following changes:'
           echo "$output"
           echo ""
+          exit 1

changelog/18.0/18.0.0/summary.md

@@ -18,6 +18,7 @@
     - [Deleted `vtgr`](#deleted-vtgr)
   - **[New stats](#new-stats)**
     - [VTGate Vindex unknown parameters](#vtgate-vindex-unknown-parameters)
+    - [VTBackup stat `PhaseStatus`](#vtbackup-stat-phase-status)
   - **[VTTablet](#vttablet)**
     - [VTTablet: New ResetSequences RPC](#vttablet-new-rpc-reset-sequences)
   - **[Docker](#docker)**
@@ -83,10 +84,20 @@ Throttler related `vttablet` flags:
 - `--throttle_check_as_check_self` is deprecated and will be removed in `v19.0`
 - `--throttler-config-via-topo` is deprecated after assumed `true` in `v17.0`. It will be removed in a future version.
 
+Cache related `vttablet` flags:
+
+- `--queryserver-config-query-cache-lfu` is deprecated and will be removed in `v19.0`. The query cache always uses a LFU implementation now.
+- `--queryserver-config-query-cache-size` is deprecated and will be removed in `v19.0`. This option only applied to LRU caches, which are now unsupported.
+
 Buffering related `vtgate` flags:
 
 - `--buffer_implementation` is deprecated and will be removed in `v19.0`
 
+Cache related `vtgate` flags:
+
+- `--gate_query_cache_lfu` is deprecated and will be removed in `v19.0`. The query cache always uses a LFU implementation now.
+- `--gate_query_cache_size` is deprecated and will be removed in `v19.0`. This option only applied to LRU caches, which are now unsupported.
+
 VTGate flag:
 
 - `--schema_change_signal_user` is deprecated and will be removed in `v19.0`
@@ -110,6 +121,14 @@ The `vtgr` has been deprecated in Vitess 17, also see https://github.com/vitessi
 
 The VTGate stat `VindexUnknownParameters` gauges unknown Vindex parameters found in the latest VSchema pulled from the topology.
 
+#### <a id="vtbackup-stat-phase-status"/>VTBackup `PhaseStatus` stat
+
+`PhaseStatus` reports a 1 (active) or a 0 (inactive) for each of the following phases and statuses:
+
+ * `CatchUpReplication` phase has statuses `Stalled` and `Stopped`.
+    * `Stalled` is set to `1` when replication stops advancing.
+    * `Stopped` is set to `1` when replication stops before `vtbackup` catches up with the primary.
+
 ### <a id="vttablet"/>VTTablet
 
 #### <a id="vttablet-new-rpc-reset-sequences"/>New ResetSequences rpc

changelog/18.0/README.md

@@ -1 +1,2 @@
 ## v18.0
+* **[18.0.0](18.0.0)**

doc/internal/README.md

@@ -1,7 +1,7 @@
 # Internal Documentation
 
-The documents in this category document internal processes which are taken care of by the Vitess Team e.g. re-publishing the website [vitess.io](https://vitess.io) or creating a new release.
+The documents in this category document internal processes which are taken care of by the Vitess Team e.g. creating a new release.
 
 We have put them here to increase transparency and make it easy for others to follow and improve processes.
 
-- [**Release Instructions**](./release/README.md)
+- [**Release**](./release/README.md)

doc/internal/release/README.md

@@ -4,10 +4,10 @@ This page describes the steps for cutting a new [open source release](https://gi
 
 ### Summary
 
+- [How to Release](./how-to-release.md)
 - [Versioning](./versioning.md)
 - [Release Branches](./release-branches.md)
 - [Release Tags](./release-tags.md)
 - [Docker Images](./docker-images.md)
 - [Java Packages](./java-packages.md)
-- [Release Cutover](./release-cutover.md)
 - [End Of Life Process](./eol-process.md)

doc/internal/release/release-cutover.md → doc/internal/release/how-to-release.md

@@ -3,7 +3,7 @@
 In this section we describe our current release process. Below is a summary of this document.
 
 - [**Pre-requisite for the release team**](#pre-requisites)
-- [**0verview**](#overview)
+- [**Overview**](#overview)
 - [**Pre-Release**](#pre-release)
 - [**Release**](#release)
 - [**Post-Release**](#post-release)
@@ -58,6 +58,7 @@ That includes:
 - **Making sure the people doing the release have access to all the tools and infrastructure needed to do the release.**
   > - This includes write access to the Vitess repository and to the Maven repository.
 - **Preparing and cleaning the release notes summary.**
+  > - If the release does not contain significant changes (i.e. a small patch release) then this step can be skipped
   > - One or more Pull Requests have to be submitted in advance to create and update the release summary.
   > - The summary files are located in: `./changelog/*.0/*.*.*/summary.md`.
   > - The summary file for a release candidate is the same as the one for the GA release.
@@ -77,19 +78,15 @@ That includes:
   > - While the Vitess Operator is located in a different repository, we also need to do a release for it.
   > - The Operator follows the same cycle: RC1 -> GA -> Patches.
   > - Documentation for the pre-release of the Vitess Operator is available [here](https://github.com/planetscale/vitess-operator/blob/main/docs/release-process.md#prepare-for-release).
-- **Update the release notes on `main`.**
-  > - One Pull Request against `main` must be created, it will contain the new release notes that we are adding in the Release Pull Request.
-  > - We open this Pull Request now to avoid waiting on the CI during release day.
-  > - All future changes to the release notes during the code freeze will need to be ported to both PRs: the one on `main` and the Release Pull Request.
 - **Update the website documentation.**
   > - We want to open a preparatory **draft** Pull Request to update the documentation.
   > - There are several pages we want to update:
-      >   - [The releases page](https://vitess.io/docs/releases/), we must add the new release to the list with all its information and link. The links can be broken (404 error) while we are preparing for the release, this is fine.
-  >   - [The local install page](https://vitess.io/docs/get-started/local/), we must use the proper version increment for this guide and the proper SHA. The SHA will have to be modified once the Release Pull Request and the release is tagged is merged.
+  >  - [The releases page](https://vitess.io/docs/releases/): we must add the new release to the list with all its information and link. The links can be broken (404 error) while we are preparing for the release, this is fine.
+  >  - [The local install page](https://vitess.io/docs/get-started/local/): we must use the proper version increment for this guide and the proper SHA. The SHA will have to be modified once the Release Pull Request and the release is tagged is merged.
   > - If we are doing a GA or RC release follow the instructions below:
-      >  - There are two scripts in the website repository in `./tools/{ga|rc}_release.sh`, use them to update the website documentation. The scripts automate:
-           >    - For an RC, we need to create a new version in the sidebar and mark the current version as RC.
-  >    - For a GA, we need to mark the version we are releasing as "Stable" and the next one as "Development".
+  >  - There are two scripts in the website repository in `./tools/{ga|rc}_release.sh`, use them to update the website documentation. The scripts automate:
+  >  - For an RC, we need to create a new entry in the sidebar which represents the next version on `main` and mark the version we are releasing as RC.
+  >  - For a GA, we need to mark the version we are releasing as "Stable" and the next one as "Development".
 
 -----
 
@@ -102,7 +99,7 @@ On the release day, there are several things to do:
 - **Tag the Vitess release.**
   > - A guide on how to tag a version is available in the [How To Release Vitess](#how-to-release-vitess) section.
 - **Update the release notes on `main`.**
-  > - During the code freeze, we created a Pull Request against `main` to update the release notes. It must be merged.
+  > - One Pull Request against `main` must be created, it will contain the new release notes that we are adding in the Release Pull Request.
 - **Create the corresponding Vitess operator release.**
   > - Applies only to versions greater or equal to `v14.0.0`.
   > - If we are doing an RC release, then we will need to create the Vitess Operator RC too. If we are doing a GA release, we're also doing a GA release in the Operator.
@@ -125,10 +122,10 @@ On the release day, there are several things to do:
   > - The benchmarks need to complete before announcing the blog posts or before they get cross-posted.
 - **Go back to dev mode on the release branch.**
   > - The version constants across the codebase must be updated to `SNAPSHOT`.
-- **Build k8s Docker images and publish them**
+- **Build k8s Docker images and publish them.**
   > - The docker image for `base`, `lite`, etc are built automatically by DockerHub. The k8s images however are dependent on these images and are required to be built manually.
   > - These images should be built after the `base` image has been built and available on DockerHub.
-  > - To build and publish these images, run `./release.sh` from the directory `vitess/docker`.
+  > - To build and publish these images, checkout the new release tag that was just created and run `./release.sh` from the directory `./docker`.
 
 -----
 
@@ -172,15 +169,12 @@ We need to verify that _arewefastyet_ has finished the benchmark too.
 
        The script will prompt you `Pausing so release notes can be added. Press enter to continue`. We are now going to generate the release notes, continue to the next sub-step.
 
-    2. Run the following command to generate the release notes:
-        1. Release Candidate:
-            ```shell
-            go run ./go/tools/release-notes --from "v14.0.3" --to "HEAD" --version "v15.0.0-rc1" --summary "./changelog/15.0/15.0.0/summary.md" [--threads=[0-9.]]
-            ```
-        2. General Availability:
-            ```shell
-            go run ./go/tools/release-notes --from "v14.0.3" --to "HEAD" --version "v15.0.0" --summary "./changelog/15.0/15.0.0/summary.md" [--threads=[0-9.]]
-            ```
+    2. Run the following command to generate the release notes. Note that you can omit the `--summary` flag if there are no summary.
+        ```shell
+        go run ./go/tools/release-notes --version "v15.0.0" --summary "./changelog/15.0/15.0.0/summary.md"
+        ```
+
+       > Make sure to also run `go run ./go/tools/releases/releases.go` to update the `./changelog` directory.
 
        > Important note: The release note generation fetches a lot of data from the GitHub API. You might reach the API request limit.
        In which case you should use the `--threads=` flag and set an integer value lower than 10 (the default).

examples/local/303_reshard.sh

@@ -19,4 +19,8 @@
 
 source ../common/env.sh
 
-vtctlclient Reshard -- --source_shards '0' --target_shards '-80,80-' Create customer.cust2cust
+vtctldclient Reshard --workflow cust2cust --target-keyspace customer create --source-shards '0' --target-shards '-80,80-'
+
+# Wait for the workflow to reach the running state.
+wait_for_workflow_running customer cust2cust
+

examples/local/304_switch_reads.sh

@@ -18,4 +18,4 @@
 
 source ../common/env.sh
 
-vtctlclient Reshard -- --tablet_types=rdonly,replica SwitchTraffic customer.cust2cust
+vtctldclient Reshard --workflow cust2cust --target-keyspace customer switchtraffic --tablet-types "rdonly,replica"

examples/local/305_switch_writes.sh

@@ -18,4 +18,5 @@
 
 source ../common/env.sh
 
-vtctlclient Reshard -- --tablet_types=primary SwitchTraffic customer.cust2cust
+vtctldclient Reshard --workflow cust2cust --target-keyspace customer switchtraffic --tablet-types "primary"
+

examples/local/306_down_shard_0.sh

@@ -17,7 +17,7 @@
 
 source ../common/env.sh
 
-vtctlclient Reshard Complete customer.cust2cust
+vtctldclient Reshard --workflow cust2cust --target-keyspace customer complete
 
 for i in 200 201 202; do
 	CELL=zone1 TABLET_UID=$i ../common/scripts/vttablet-down.sh

go.mod

@@ -9,7 +9,6 @@ require (
 	github.com/Azure/azure-storage-blob-go v0.15.0
 	github.com/DataDog/datadog-go v4.8.3+incompatible
 	github.com/HdrHistogram/hdrhistogram-go v0.9.0 // indirect
-	github.com/PuerkitoBio/goquery v1.5.1
 	github.com/aquarapid/vaultlib v0.5.1
 	github.com/armon/go-metrics v0.4.1 // indirect
 	github.com/aws/aws-sdk-go v1.44.258
@@ -76,7 +75,7 @@ require (
 	golang.org/x/mod v0.12.0 // indirect
 	golang.org/x/net v0.14.0
 	golang.org/x/oauth2 v0.7.0
-	golang.org/x/sys v0.11.0 // indirect
+	golang.org/x/sys v0.11.0
 	golang.org/x/term v0.11.0
 	golang.org/x/text v0.12.0
 	golang.org/x/time v0.3.0
@@ -97,6 +96,7 @@ require (
 require (
 	github.com/Shopify/toxiproxy/v2 v2.5.0
 	github.com/bndr/gotabulate v1.1.2
+	github.com/gammazero/deque v0.2.1
 	github.com/google/safehtml v0.1.0
 	github.com/hashicorp/go-version v1.6.0
 	github.com/kr/pretty v0.3.1
@@ -124,7 +124,6 @@ require (
 	github.com/DataDog/go-tuf v0.3.0--fix-localmeta-fork // indirect
 	github.com/DataDog/sketches-go v1.4.1 // indirect
 	github.com/Microsoft/go-winio v0.6.0 // indirect
-	github.com/andybalholm/cascadia v1.1.0 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/coreos/go-semver v0.3.1 // indirect
 	github.com/coreos/go-systemd/v22 v22.5.0 // indirect

go.sum

@@ -96,16 +96,12 @@ github.com/Microsoft/go-winio v0.5.0/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpz
 github.com/Microsoft/go-winio v0.5.1/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84=
 github.com/Microsoft/go-winio v0.6.0 h1:slsWYD/zyx7lCXoZVlvQrj0hPTM1HI4+v1sIda2yDvg=
 github.com/Microsoft/go-winio v0.6.0/go.mod h1:cTAf44im0RAYeL23bpB+fzCyDH2MJiz2BO69KH/soAE=
-github.com/PuerkitoBio/goquery v1.5.1 h1:PSPBGne8NIUWw+/7vFBV+kG2J/5MOjbzc7154OaKCSE=
-github.com/PuerkitoBio/goquery v1.5.1/go.mod h1:GsLWisAFVj4WgDibEWF4pvYnkVQBpKBKeU+7zCJoLcc=
 github.com/Shopify/toxiproxy/v2 v2.5.0 h1:i4LPT+qrSlKNtQf5QliVjdP08GyAH8+BUIc9gT0eahc=
 github.com/Shopify/toxiproxy/v2 v2.5.0/go.mod h1:yhM2epWtAmel9CB8r2+L+PCmhH6yH2pITaPAo7jxJl0=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
-github.com/andybalholm/cascadia v1.1.0 h1:BuuO6sSfQNFRu1LppgbD25Hr2vLYW25JvxHs5zzsLTo=
-github.com/andybalholm/cascadia v1.1.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y=
 github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
 github.com/aquarapid/vaultlib v0.5.1 h1:vuLWR6bZzLHybjJBSUYPgZlIp6KZ+SXeHLRRYTuk6d4=
 github.com/aquarapid/vaultlib v0.5.1/go.mod h1:yT7AlEXtuabkxylOc/+Ulyp18tff1+QjgNLTnFWTlOs=
@@ -195,6 +191,8 @@ github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMo
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
 github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
 github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
+github.com/gammazero/deque v0.2.1 h1:qSdsbG6pgp6nL7A0+K/B7s12mcCY/5l5SIUpMOl+dC0=
+github.com/gammazero/deque v0.2.1/go.mod h1:LFroj8x4cMYCukHJDbxFCkT+r9AndaJnFMuZDV34tuU=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
@@ -707,7 +705,6 @@ golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91
 golang.org/x/mod v0.7.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
 golang.org/x/mod v0.12.0 h1:rmsUpXtvNzj340zd98LZ4KntptpfRHwpFOHG188oHXc=
 golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
-golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
 golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=