GPU CI times out with debug=true

[samuelburnham]

Problem

The dev-ci Cargo profile implicitly enables debug info to troubleshoot CI failures on pull requests. However, when running GPU CI in e.g. https://github.com/lurk-lab/arecibo/actions/runs/7560396589/job/20592226334, we observe the following:

• GPU usage spikes to 100% and the program hangs when running tests::test_ivc_nontrivial_with_spark_compression
• Running with --release or adding debug = false to the dev-ci profile fixes the issue
• This also occurs when run with only the Pasta engines in https://github.com/lurk-lab/arecibo/blob/dev/src/lib.rs#L1485
• This likely occurs with the other hanging tests in the above GPU CI run: supernova::snark::test::test_compression_with_circuit_size_difference, supernova::snark::test::test_nivc_trivial_with_compression, and tests::test_ivc_nontrivial_with_compression.
• This issue does not occur on non-GPU servers or in lurk-rs.

Potential Diagnosis

Grumpkin-msm compiles a lot of code when GPU acceleration is enabled, as seen by its long compile times, so including debug info could be overloading the GPU resources. However, this wouldn't explain the lack of timeouts in lurk-rs.

Solutions

• Short term: switched GPU CI to release mode in ci: Disable debug info for GPU tests #271
• Long term: investigate the debug info generated by grumpkin-msm and try to repro test timeouts there. cc @winston-h-zhang