chore(CI): Switch regression detector to new API and analysis service (…

…vectordotdev#17912)

This should be a transparent change. If you notice any issues, please
ping us in #single-machine-performance. We'll also be keeping a close
eye on things after this is merged.

## Single Machine Performance API (`curta`)

The Single Machine Performance team has built an API to interact with
the regression detector system. This will allow us to maintain
reliability while continuing to evolve the system. This is enabled with
the `--use-curta` flag, which will become the default in an upcoming
release.

The curta API uses a unique URI for each team. This isn't secret - all
endpoints have authorization - but we'd prefer it not be public. The URI
for Vector is in SMP-617. The `smp` binary can read this from the
`SINGLE_MACHINE_PERFORMANCE_API` environment variable or the
`--api-base` flag.

## Lambda Analysis Worker (`consignor`)

We have also changed how the analysis worker is deployed so it can run
on-demand for incoming requests. This improves analysis speed while
reducing our operational workload. This is set using the
`--use-consignor` flag, which will also become the default in a future
release.

---------

Co-authored-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>

.github/actions/spelling/allow.txt

@@ -246,6 +246,7 @@ compiletime
 coredns
 corejs
 coreutils
+curta
 daemonset
 databend
 datacenter

.github/workflows/regression.yml

@@ -37,6 +37,9 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.event.merge_group.head_sha || github.sha }}
   cancel-in-progress: true
 
+env:
+  SINGLE_MACHINE_PERFORMANCE_API: ${{ secrets.SINGLE_MACHINE_PERFORMANCE_API }}
+
 jobs:
 
   # Only run this workflow if files changed in areas that could possibly introduce a regression
@@ -229,7 +232,7 @@ jobs:
           export REPLICAS="10"
           export TOTAL_SAMPLES="600"
           export P_VALUE="0.1"
-          export SMP_CRATE_VERSION="0.7.3"
+          export SMP_CRATE_VERSION="0.9.0"
           export LADING_VERSION="0.12.0"
 
           echo "warmup seconds: ${WARMUP_SECONDS}"
@@ -496,6 +499,7 @@ jobs:
           chmod +x ${{ runner.temp }}/bin/smp
 
           ${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job submit \
+            --use-curta \
             --lading-version ${{ needs.compute-metadata.outputs.lading-version }} \
             --total-samples ${{ needs.compute-metadata.outputs.total-samples }} \
             --warmup-seconds ${{ needs.compute-metadata.outputs.warmup-seconds }} \
@@ -525,6 +529,7 @@ jobs:
           chmod +x ${{ runner.temp }}/bin/smp
 
           ${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job status \
+            --use-curta \
             --wait \
             --wait-delay-seconds 60 \
             --wait-timeout-minutes 90 \
@@ -537,6 +542,7 @@ jobs:
         run: |
           chmod +x ${{ runner.temp }}/bin/smp
           ${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job cancel \
+            --use-curta \
             --submission-metadata ${{ runner.temp }}/submission-metadata
 
       - name: Check status, cancelled
@@ -617,6 +623,8 @@ jobs:
           chmod +x ${{ runner.temp }}/bin/smp
 
           ${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job result \
+            --use-curta \
+            --use-consignor \
             --submission-metadata ${{ runner.temp }}/submission-metadata
 
       - name: Check status, cancelled
@@ -708,6 +716,8 @@ jobs:
           chmod +x ${{ runner.temp }}/bin/smp
 
           ${{ runner.temp }}/bin/smp --team-id ${{ secrets.SINGLE_MACHINE_PERFORMANCE_TEAM_ID }} job sync \
+            --use-curta \
+            --use-consignor \
             --submission-metadata ${{ runner.temp }}/submission-metadata \
             --output-path "${{ runner.temp }}/outputs"