Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): Remove --nocapture from integration tests #13128

Merged
merged 1 commit into from
Jun 13, 2022

Conversation

jszwedko
Copy link
Member

This broke with the upgrade to cargo nextest 0.9.19 which wants you to
use an extra -- to pass along extra flags to the test binary, but it
occurred to me that I don't think we need this flag. It's useful for
debugging, but otherwise we don't really need to see the integration
test output unless it fails, in which case it will output it.

Signed-off-by: Jesse Szwedko jesse@szwedko.me

This broke with the upgrade to `cargo nextest` 0.9.19 which wants you to
use an extra `--` to pass along extra flags to the test binary, but it
occurred to me that I don't think we need this flag. It's useful for
debugging, but otherwise we don't really need to see the integration
test output unless it fails, in which case it will output it.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
@jszwedko jszwedko added the ci-condition: integration tests enable Run integration tests on this PR label Jun 13, 2022
@netlify
Copy link

netlify bot commented Jun 13, 2022

Deploy Preview for vector-project ready!

Name Link
🔨 Latest commit b820e4f
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/62a79fd5e264b00008f710d7
😎 Deploy Preview https://deploy-preview-13128--vector-project.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions github-actions bot added the domain: ci Anything related to Vector's CI environment label Jun 13, 2022
tobz added a commit that referenced this pull request Jun 13, 2022
Signed-off-by: Toby Lawrence <toby@nuclearfurnace.com>
@jszwedko jszwedko merged commit 8046713 into master Jun 13, 2022
@jszwedko jszwedko deleted the remove-nextest-no-capture branch June 13, 2022 21:04
@jszwedko
Copy link
Member Author

Merging since integration tests passed.

@sunshowers
Copy link

sunshowers commented Jun 13, 2022

Hey, apologies for breaking you!

Are you sure --nocapture was working? In your case the old --nocapture wouldn't actually have worked, --nocapture would have been read as a test filter i.e. only tests matching the string "--nocapture" (along with the other filters you passed in) would have been run. It's partly to catch situations like these that we added this detection (though we don't support --nocapture).

@jszwedko
Copy link
Member Author

Hey @sunshowers ! No worries, you might very well be right that it wasn't working correctly before. I know cargo nextest run also supports the flag natively with --no-capture, which we can use if we find the need.

@github-actions
Copy link

Soak Test Results

Baseline: f75dbb7
Comparison: b820e4f
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_log2metric_splunk_hec_metrics 138.32KiB 0.73 100.00% 18.38MiB 861.41KiB 17.52KiB 0 0.0457611 18.51MiB 628.41KiB 12.8KiB 0 0.0331398 False False
datadog_agent_remap_blackhole_acks 494.3KiB 0.71 100.00% 67.77MiB 3.73MiB 77.75KiB 0 0.0550121 68.25MiB 2.0MiB 41.76KiB 0 0.0292821 False False
splunk_transforms_splunk3 49.52KiB 0.38 55.37% 12.87MiB 2.19MiB 45.82KiB 0 0.170366 12.92MiB 2.21MiB 46.14KiB 0 0.170603 False False
http_to_http_noack 86.36KiB 0.35 99.99% 23.76MiB 1.0MiB 20.95KiB 0 0.0422602 23.84MiB 243.52KiB 4.97KiB 0 0.00997127 False False
http_pipelines_blackhole_acks 8.71KiB 0.19 39.51% 4.38MiB 581.0KiB 11.88KiB 0 0.129584 4.39MiB 583.68KiB 11.93KiB 0 0.129929 False False
socket_to_socket_blackhole 19.1KiB 0.14 75.85% 13.68MiB 580.12KiB 11.82KiB 0 0.0414126 13.7MiB 550.54KiB 11.23KiB 0 0.0392474 False False
syslog_log2metric_humio_metrics 7.99KiB 0.06 60.55% 14.09MiB 392.2KiB 8.0KiB 0 0.0271719 14.1MiB 239.99KiB 4.91KiB 0 0.0166179 False False
splunk_hec_indexer_ack_blackhole 9.71KiB 0.04 36.63% 23.77MiB 728.79KiB 14.83KiB 0 0.0299312 23.78MiB 685.45KiB 13.96KiB 0 0.0281402 False False
fluent_elasticsearch 12.59KiB 0.02 83.15% 79.46MiB 450.49KiB 9.08KiB 0 0.00553537 79.47MiB 53.13KiB 1.07KiB 0 0.000652777 False False
datadog_agent_remap_blackhole -162.41B -0 0.18% 64.51MiB 2.46MiB 51.58KiB 0 0.0381257 64.51MiB 2.27MiB 47.49KiB 0 0.0351443 False False
http_to_http_json 1.08KiB 0 8.64% 23.84MiB 343.2KiB 7.01KiB 0 0.0140542 23.84MiB 346.04KiB 7.06KiB 0 0.0141699 False False
splunk_hec_to_splunk_hec_logs_noack 130.86B 0 1.05% 23.84MiB 335.8KiB 6.87KiB 0 0.0137544 23.84MiB 336.87KiB 6.89KiB 0 0.0137982 False False
syslog_regex_logs2metric_ddmetrics -3.28KiB -0.03 12.68% 12.77MiB 711.77KiB 14.49KiB 0 0.0543992 12.77MiB 715.08KiB 14.55KiB 0 0.0546653 False False
splunk_hec_to_splunk_hec_logs_acks -9.7KiB -0.04 34.60% 23.77MiB 731.69KiB 14.89KiB 0 0.0300494 23.76MiB 771.17KiB 15.69KiB 0 0.0316833 False False
syslog_loki -5.44KiB -0.04 26.19% 15.15MiB 621.78KiB 12.68KiB 0 0.040067 15.15MiB 497.9KiB 10.18KiB 0 0.0320956 False False
syslog_humio_logs -8.32KiB -0.05 97.96% 16.61MiB 124.93KiB 2.56KiB 0 0.00734159 16.61MiB 122.78KiB 2.51KiB 0 0.00721851 False False
datadog_agent_remap_datadog_logs -41.95KiB -0.06 42.37% 63.5MiB 2.61MiB 54.42KiB 0 0.0410523 63.46MiB 2.47MiB 51.72KiB 0 0.0389168 False False
http_pipelines_no_grok_blackhole -17.29KiB -0.09 29.26% 18.21MiB 1.5MiB 31.29KiB 0 0.0821178 18.19MiB 1.62MiB 33.82KiB 0 0.0891386 False False
file_to_blackhole -151.71KiB -0.16 93.41% 95.31MiB 2.76MiB 57.42KiB 0 0.0289668 95.17MiB 2.87MiB 59.2KiB 0 0.0300995 False False
datadog_agent_remap_datadog_logs_acks -267.77KiB -0.41 100.00% 63.42MiB 592.03KiB 12.09KiB 0 0.00911471 63.16MiB 2.96MiB 61.93KiB 0 0.0468893 False False
http_pipelines_blackhole -28.06KiB -0.61 88.94% 4.49MiB 606.11KiB 12.42KiB 0 0.131829 4.46MiB 610.43KiB 12.44KiB 0 0.133584 False False
http_to_http_acks -162.71KiB -0.89 51.96% 17.79MiB 7.74MiB 161.8KiB 0 0.434737 17.63MiB 7.86MiB 164.29KiB 0 0.445689 True True
syslog_splunk_hec_logs -221.93KiB -1.29 100.00% 16.82MiB 740.54KiB 15.12KiB 0 0.0429742 16.61MiB 750.34KiB 15.35KiB 0 0.0441109 False False
splunk_hec_route_s3 -556.04KiB -2.76 100.00% 19.68MiB 2.24MiB 46.84KiB 0 0.113841 19.14MiB 2.44MiB 50.96KiB 0 0.127352 False False

jszwedko added a commit that referenced this pull request Jun 14, 2022
This broke with the upgrade to `cargo nextest` 0.9.19 which wants you to
use an extra `--` to pass along extra flags to the test binary, but it
occurred to me that I don't think we need this flag. It's useful for
debugging, but otherwise we don't really need to see the integration
test output unless it fails, in which case it will output it.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
jszwedko added a commit that referenced this pull request Jun 15, 2022
This broke with the upgrade to `cargo nextest` 0.9.19 which wants you to
use an extra `--` to pass along extra flags to the test binary, but it
occurred to me that I don't think we need this flag. It's useful for
debugging, but otherwise we don't really need to see the integration
test output unless it fails, in which case it will output it.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-condition: integration tests enable Run integration tests on this PR domain: ci Anything related to Vector's CI environment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants