Ensure that metrics are flushed before shutting down #1139

beniwohli · 2021-05-26T16:07:17Z

To ensure that the flushed metrics are actually sent, we need
to make sure that the transport thread is shut down last.

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Branch indexing
Start Time: 2021-05-28T16:27:15.631+0000
Duration: 32 min 17 sec
Commit: 41cec53

Test stats 🧪

Test	Results
Failed	2
Passed	9428
Skipped	8443
Total	17873

Trends 🧪

Test errors

Expand to view the tests failures

`Initializing / Test / Python-python-3.8-4 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests`

Expand to view the error details

 AssertionError: assert {'PUBLISH', '...sh_subscribe'} == {'PUBLISH', '...sh_subscribe'}
  Extra items in the right set:
  'SUBSCRIBE'
  Full diff:
  - {'test_publish_subscribe', 'PUBLISH', 'SUBSCRIBE'}
  ?                                     -------------
  + {'test_publish_subscribe', 'PUBLISH'}

Expand to view the stacktrace

 instrument = None
elasticapm_client = <tests.fixtures.TempStoreClient object at 0x7fbbfff96d60>
redis_conn = <Redis <ConnectionsPool [db:0, size:[1:10], free:0]>>

    @pytest.mark.integrationtest
    async def test_publish_subscribe(instrument, elasticapm_client, redis_conn):
        elasticapm_client.begin_transaction("transaction.test")
        with capture_span("test_publish_subscribe", "test"):
            # publish
            await redis_conn.publish("mykey", "a")
    
            #subscribe
            await redis_conn.subscribe("mykey")
    
        elasticapm_client.end_transaction("MyView")
    
        transactions = elasticapm_client.events[TRANSACTION]
        spans = elasticapm_client.spans_for_transaction(transactions[0])
    
        expected_signatures = {"test_publish_subscribe", "PUBLISH", "SUBSCRIBE"}
    
>       assert {t["name"] for t in spans} == expected_signatures
E       AssertionError: assert {'PUBLISH', '...sh_subscribe'} == {'PUBLISH', '...sh_subscribe'}
E         Extra items in the right set:
E         'SUBSCRIBE'
E         Full diff:
E         - {'test_publish_subscribe', 'PUBLISH', 'SUBSCRIBE'}
E         ?                                     -------------
E         + {'test_publish_subscribe', 'PUBLISH'}

tests/instrumentation/asyncio_tests/aioredis_tests.py:145: AssertionError

`Initializing / Test / Python-python-3.9-3 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests`

Expand to view the error details

 AssertionError: assert {'PUBLISH', '...sh_subscribe'} == {'PUBLISH', '...sh_subscribe'}
  Extra items in the right set:
  'SUBSCRIBE'
  Full diff:
  - {'PUBLISH', 'SUBSCRIBE', 'test_publish_subscribe'}
  ?            -------------
  + {'PUBLISH', 'test_publish_subscribe'}

Expand to view the stacktrace

 instrument = None
elasticapm_client = <tests.fixtures.TempStoreClient object at 0x7f05824e6550>
redis_conn = <Redis <ConnectionsPool [db:0, size:[1:10], free:0]>>

    @pytest.mark.integrationtest
    async def test_publish_subscribe(instrument, elasticapm_client, redis_conn):
        elasticapm_client.begin_transaction("transaction.test")
        with capture_span("test_publish_subscribe", "test"):
            # publish
            await redis_conn.publish("mykey", "a")
    
            #subscribe
            await redis_conn.subscribe("mykey")
    
        elasticapm_client.end_transaction("MyView")
    
        transactions = elasticapm_client.events[TRANSACTION]
        spans = elasticapm_client.spans_for_transaction(transactions[0])
    
        expected_signatures = {"test_publish_subscribe", "PUBLISH", "SUBSCRIBE"}
    
>       assert {t["name"] for t in spans} == expected_signatures
E       AssertionError: assert {'PUBLISH', '...sh_subscribe'} == {'PUBLISH', '...sh_subscribe'}
E         Extra items in the right set:
E         'SUBSCRIBE'
E         Full diff:
E         - {'PUBLISH', 'SUBSCRIBE', 'test_publish_subscribe'}
E         ?            -------------
E         + {'PUBLISH', 'test_publish_subscribe'}

tests/instrumentation/asyncio_tests/aioredis_tests.py:145: AssertionError

Steps errors

Expand to view the steps failures

Show only the first 10 steps failures

`Restore files previously stashed`

Took 0 min 0 sec . View more details on here
Description: coverage-python-3.8-elasticsearch-2

`Restore files previously stashed`

Took 0 min 0 sec . View more details on here
Description: coverage-python-3.8-gevent-newest

`Restore files previously stashed`

Took 0 min 0 sec . View more details on here
Description: coverage-python-3.8-aiopg-newest

`Restore files previously stashed`

Took 0 min 0 sec . View more details on here
Description: coverage-python-3.8-graphene-2

`Restore files previously stashed`

Took 0 min 0 sec . View more details on here
Description: coverage-python-3.9-elasticsearch-2

`Restore files previously stashed`

Took 0 min 0 sec . View more details on here
Description: coverage-python-3.9-gevent-newest

`Restore files previously stashed`

Took 0 min 0 sec . View more details on here
Description: coverage-python-3.9-aiopg-newest

`Restore files previously stashed`

Took 0 min 0 sec . View more details on here
Description: coverage-python-3.9-graphene-2

`Archive the artifacts`

Took 0 min 0 sec . View more details on here
Description: [2021-05-28T16:59:30.249Z] Archiving artifacts Python-python-3.8-aioredis-newest tests failed : hud

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Python-python-3.8-aioredis-newest tests failed : hudson.AbortException: script returned exit code 2

Log output

Expand to view the last 100 lines of log output

[2021-05-28T16:58:40.465Z] ERROR: No such saved stash ‘coverage-python-3.9-graphene-2’
[2021-05-28T16:58:49.106Z] + python3 -m coverage combine
[2021-05-28T16:59:27.914Z] + python3 -m coverage xml
[2021-05-28T16:59:29.345Z] [Cobertura] Skipping Cobertura coverage report as build was not SUCCESS or better ...
[2021-05-28T16:59:29.346Z] 
[2021-05-28T16:59:29.725Z] + curl -sLO https://code.jquery.com/jquery-3.3.1.slim.min.js
[2021-05-28T16:59:30.044Z] + curl -sLO https://cdn.jsdelivr.net/npm/htmlson.js@1.0.4/src/htmlson.js
[2021-05-28T16:59:30.249Z] Archiving artifacts
[2021-05-28T16:59:30.424Z] Stage "Building packages" skipped due to earlier failure(s)
[2021-05-28T16:59:30.474Z] Stage "Integration Tests" skipped due to earlier failure(s)
[2021-05-28T16:59:30.518Z] Stage "Benchmarks" skipped due to earlier failure(s)
[2021-05-28T16:59:30.621Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-05-28T16:59:30.643Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-05-28T16:59:30.689Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-05-28T16:59:30.737Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-05-28T16:59:31.956Z] Running on Jenkins in /var/lib/jenkins/workspace/hon_apm-agent-python-mbp_PR-1139
[2021-05-28T16:59:32.304Z] [INFO] getVaultSecret: Getting secrets
[2021-05-28T16:59:32.362Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-05-28T16:59:33.232Z] + chmod 755 generate-build-data.sh
[2021-05-28T16:59:33.232Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1139/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1139/runs/3 FAILURE 1937333
[2021-05-28T16:59:33.232Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1139/runs/3/steps/?limit=10000 -o steps-info.json
[2021-05-28T16:59:40.253Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1139/runs/3/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	2
Passed	9428
Skipped	8443
Total	17873

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Initializing / Test / Python-python-3.8-4 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests
Name: Initializing / Test / Python-python-3.9-3 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests

briancurtin

Looks good to me!

/cc @stj — this will help our use of custom metrics

elasticapm/transport/base.py

apmmachine · 2021-05-31T07:54:06Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #1139 updated
Start Time: 2021-05-31T07:27:07.121+0000
Duration: 26 min 42 sec
Commit: e6bb3b2

Test stats 🧪

Test	Results
Failed	0
Passed	8866
Skipped	8721
Total	17587

Trends 🧪

Steps errors

Expand to view the steps failures

`Archive the artifacts`

Took 0 min 0 sec . View more details on here
Description: [2021-05-31T07:52:45.682Z] Archiving artifacts org.jenkinsci.plugins.workflow.steps.FlowInterrupted

`Error signal`

Took 0 min 0 sec . View more details on here
Description: org.jenkinsci.plugins.workflow.steps.FlowInterruptedException

Log output

Expand to view the last 100 lines of log output

[2021-05-31T07:52:04.033Z] + python3 -m coverage combine
[2021-05-31T07:52:43.189Z] + python3 -m coverage xml
[2021-05-31T07:52:44.240Z] [Cobertura] Skipping Cobertura coverage report as build was not SUCCESS or better ...
[2021-05-31T07:52:44.240Z] 
[2021-05-31T07:52:44.827Z] + curl -sLO https://code.jquery.com/jquery-3.3.1.slim.min.js
[2021-05-31T07:52:45.244Z] + curl -sLO https://cdn.jsdelivr.net/npm/htmlson.js@1.0.4/src/htmlson.js
[2021-05-31T07:52:45.682Z] Archiving artifacts
[2021-05-31T07:52:46.023Z] Stage "Building packages" skipped due to earlier failure(s)
[2021-05-31T07:52:46.122Z] Stage "Integration Tests" skipped due to earlier failure(s)
[2021-05-31T07:52:46.223Z] Stage "Benchmarks" skipped due to earlier failure(s)
[2021-05-31T07:52:46.477Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-05-31T07:52:46.531Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-05-31T07:52:46.612Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-05-31T07:52:46.758Z] Stage "Prepare Release" skipped due to earlier failure(s)
[2021-05-31T07:52:47.828Z] Running on Jenkins in /var/lib/jenkins/workspace/hon_apm-agent-python-mbp_PR-1139
[2021-05-31T07:52:48.350Z] [INFO] getVaultSecret: Getting secrets
[2021-05-31T07:52:48.475Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-05-31T07:52:49.793Z] + chmod 755 generate-build-data.sh
[2021-05-31T07:52:49.793Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1139/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1139/runs/4 FAILURE 1542364
[2021-05-31T07:52:49.793Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1139/runs/4/steps/?limit=10000 -o steps-info.json
[2021-05-31T07:53:07.754Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1139/runs/4/tests/?status=FAILED -o tests-errors.json
[2021-05-31T07:53:08.005Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-python/apm-agent-python-mbp/PR-1139/runs/4/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	8866
Skipped	8721
Total	17587

* Ensure that metrics are flushed before shutting down To ensure that the flushed metrics are actually sent, we need to make sure that the transport thread is shut down last. * use a variable name that doesn't clash with the thread priority concept

Ensure that metrics are flushed before shutting down

a90bdac

To ensure that the flushed metrics are actually sent, we need to make sure that the transport thread is shut down last.

github-actions bot added the agent-python label May 26, 2021

briancurtin approved these changes May 26, 2021

View reviewed changes

felixbarny reviewed May 27, 2021

View reviewed changes

elasticapm/transport/base.py Outdated Show resolved Hide resolved

beniwohli added 2 commits May 28, 2021 10:48

use a variable name that doesn't clash with the thread priority concept

41cec53

Merge branch 'master' into flush-metrics

e6bb3b2

beniwohli merged commit 4943f76 into elastic:master May 31, 2021

beniwohli deleted the flush-metrics branch May 31, 2021 07:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure that metrics are flushed before shutting down #1139

Ensure that metrics are flushed before shutting down #1139

beniwohli commented May 26, 2021

apmmachine commented May 26, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

`Initializing / Test / Python-python-3.8-4 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests`

`Initializing / Test / Python-python-3.9-3 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Archive the artifacts`

`Error signal`

Test stats 🧪

Genuine test errors

briancurtin left a comment

apmmachine commented May 31, 2021

Build stats

Test stats 🧪

Trends 🧪

`Archive the artifacts`

`Error signal`

Test stats 🧪

Ensure that metrics are flushed before shutting down #1139

Ensure that metrics are flushed before shutting down #1139

Conversation

beniwohli commented May 26, 2021

apmmachine commented May 26, 2021 • edited Loading

💔 Tests Failed

Build stats

Test stats 🧪

Trends 🧪

Test errors

Initializing / Test / Python-python-3.8-4 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests

Initializing / Test / Python-python-3.9-3 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests

Steps errors

Restore files previously stashed

Restore files previously stashed

Restore files previously stashed

Restore files previously stashed

Restore files previously stashed

Restore files previously stashed

Restore files previously stashed

Restore files previously stashed

Archive the artifacts

Error signal

Log output

🐛 Flaky test report

Test stats 🧪

Genuine test errors

briancurtin left a comment

Choose a reason for hiding this comment

apmmachine commented May 31, 2021

💔 Build Failed

Build stats

Test stats 🧪

Trends 🧪

Steps errors

Archive the artifacts

Error signal

Log output

💚 Flaky test report

Test stats 🧪

apmmachine commented May 26, 2021 •

edited

Loading

`Initializing / Test / Python-python-3.8-4 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests`

`Initializing / Test / Python-python-3.9-3 / test_publish_subscribe – tests.instrumentation.asyncio_tests.aioredis_tests`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Restore files previously stashed`

`Archive the artifacts`

`Error signal`

`Archive the artifacts`

`Error signal`