ERROR: cannot find the current segment/subsegment when segment is open and uploading file to s3.

[RandyVuu]

Currently using python version 3.9, botocore version 1.29.50, aws_xray_sdk version 2.11.0

When I run the code snippet below I get an error saying there's no open segment for the calls the use upload file. The subsequent calls using put_object have no issues and show up in the segment data. Logging the trace_id was also successful.

    patch_all()
    xray_recorder.configure(context_missing='LOG_ERROR', daemon_address='localhost:2000')
    segment = xray_recorder.begin_segment('test segment')

    s3_resource = boto3.resource("s3")

    # These give the error
    s3_resource.Bucket('resultfiles-test' ).upload_file('hello_world.txt', 'hello_world.txt')
    logger.info(f'trace_id: {xray_recorder.current_segment().trace_id}')
    s3_resource.Object('resultfiles-test', 'hello_world.txt').upload_file(Filename='hello_world.txt')
    logger.info(f'trace_id: {xray_recorder.current_segment().trace_id}')


    # These work
    s3_resource.Bucket('resultfiles-test' ).put_object(Key='hello_world.txt', Body=b'hello world')
    s3_resource.Object('resultfiles-test', 'hello_world.txt').put(Body=b'hello_world.txt')
    with open('hello_world.txt', 'r') as f:
        s3_resource.Object('resultfiles-test', 'hello_world.txt').put(Body=f.read())

    xray_recorder.end_segment()

[carolabadeer]

Hello, I have a few questions that will hopefully help clarify where this issue is coming from:

1. Is there any currently active context that should be passed in when configuring the xray_recorder? Are you running in Lambda?
2. If you add another xray_recorder.current_segment().trace_id before the first call to upload_file, does that trace ID match that of "test segment"? Can you also try xray_recorder.get_trace_entity() to see what the currently active entity (segment or subsegment) is?
3. Is this code snippet all in one place/function? Is there any additional code that is being executed between the block that throws an error and the one that doesn't?

[RandyVuu]

Hello, to answer your questions

1. There is no active context to pass to the xray_recorder. I discovered this issue when using xray sdk on glue pyshell and then recreated it locally.
2. There is no additional code inbetween the lines. The code ran exactly as is.

Heres a snippet including more trace_id logs and the corresponding output

if __name__ == "__main__":
    patch_all()
    xray_recorder.configure(context_missing='LOG_ERROR', daemon_address='localhost:2000')
    segment = xray_recorder.begin_segment('test segment')
    logger.info(f'created segment trace_id:{segment.trace_id} ')
    s3_resource = boto3.resource("s3")
    logger.info(f'current trace entity trace_id: {xray_recorder.get_trace_entity().trace_id}')
    # this give the error
    logger.info(f'trace_id before error: {xray_recorder.current_segment().trace_id}')
    s3_resource.Bucket('resultfiles-test' ).upload_file('hello_world.txt', 'hello_world.txt')
    logger.info(f'trace_id after error: {xray_recorder.current_segment().trace_id}')
    # this works
    logger.info(f'trace_id before working: {xray_recorder.current_segment().trace_id}')
    s3_resource.Bucket('resultfiles-test' ).put_object(Key='hello_world.txt', Body=b'hello world')
    logger.info(f'trace_id after working: {xray_recorder.current_segment().trace_id}')
    xray_recorder.end_segment()

output

INFO:aws_xray_sdk.core.patcher:successfully patched module botocore
INFO:aws_xray_sdk.core.patcher:successfully patched module sqlite3
INFO:aws_xray_sdk.core.sampling.sampler:No effective centralized sampling rule match. Fallback to local rules.
INFO:root:created segment trace_id:1-63d85b07-0a470e1a68d647efad69bce2 
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:root:current trace entity trace_id: 1-63d85b07-0a470e1a68d647efad69bce2
INFO:root:trace_id before error: 1-63d85b07-0a470e1a68d647efad69bce2
ERROR:aws_xray_sdk.core.context:cannot find the current segment/subsegment, please make sure you have a segment open
WARNING:aws_xray_sdk.core.recorder:No segment found, cannot begin subsegment s3.
ERROR:aws_xray_sdk.core.context:cannot find the current segment/subsegment, please make sure you have a segment open
INFO:root:trace_id after error: 1-63d85b07-0a470e1a68d647efad69bce2
INFO:root:trace_id before working: 1-63d85b07-0a470e1a68d647efad69bce2
INFO:root:trace_id after working: 1-63d85b07-0a470e1a68d647efad69bce2

When I set the logging level to debug to see the segment sent, it includes the second working call to s3.

{
  "id": "a69a1a6eb4e12c86",
  "name": "test segment",
  "start_time": 1675123698.393169,
  "in_progress": false,
  "aws": {
    "xray": {
      "sdk": "X-Ray for Python",
      "sdk_version": "2.11.0"
    }
  },
  "subsegments": [
    {
      "id": "1130954ac4c9383e",
      "name": "s3",
      "start_time": 1675123699.1555898,
      "parent_id": "a69a1a6eb4e12c86",
      "in_progress": false,
      "http": {
        "response": {
          "status": 200
        }
      },
      "aws": {
        "operation": "PutObject",
        "region": "us-east-1",
        "request_id": "CTQFC52JXK7BED7F",
        "id_2": "4lazYkItN+LCNrSfBTnved/tEByY50Od+KArL3cX5uWrx8BPHDXNMCnxOi/hhT4Tv9wd+aGbGdw=",
        "key": "hello_world.txt",
        "bucket_name": "resultfiles-test"
      },
      "trace_id": "1-63d85bf2-67e791f683029bb451728c4a",
      "type": "subsegment",
      "namespace": "aws",
      "end_time": 1675123699.3337479
    }
  ],
  "trace_id": "1-63d85bf2-67e791f683029bb451728c4a",
  "service": {
    "runtime": "CPython",
    "runtime_version": "3.9.8"
  },
  "end_time": 1675123699.335922
}

[RandyVuu]

I discovered the issue can be resolved if I disable multi-threading on upload_file.