aws_xray_sdk.core.exceptions.exceptions.SegmentNotFoundException: cannot find the current segment/subsegment, please make sure you have a segment open

[krisb78]

With a basic integration, as described in #147 , I can't migrate my database.

When I'm trying to run ./manage.py migrate, I get

cannot find the current segment/subsegment, please make sure you have a segment open
Traceback (most recent call last):
 File "./manage.py", line 10, in <module>
 execute_from_command_line(sys.argv)
 File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
 utility.execute()
 File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
 self.fetch_command(subcommand).run_from_argv(self.argv)
 File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
 self.execute(*args, **cmd_options)
 File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 327, in execute
 self.check()
 File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 359, in check
 include_deployment_checks=include_deployment_checks,
 File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 62, in _run_checks
 issues.extend(super(Command, self)._run_checks(**kwargs))
 File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 346, in _run_checks
 return checks.run_checks(**kwargs)
 File "/usr/local/lib/python2.7/site-packages/django/core/checks/registry.py", line 81, in run_checks
 new_errors = check(app_configs=app_configs)
 File "/usr/local/lib/python2.7/site-packages/django/core/checks/urls.py", line 16, in check_url_config
 return check_resolver(resolver)
 File "/usr/local/lib/python2.7/site-packages/django/core/checks/urls.py", line 26, in check_resolver
 return check_method()
 File "/usr/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 256, in check
 for pattern in self.url_patterns:
 File "/usr/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
 res = instance.__dict__[self.name] = self.func(instance)
 File "/usr/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 407, in url_patterns
 patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
 File "/usr/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
 res = instance.__dict__[self.name] = self.func(instance)
 File "/usr/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 400, in urlconf_module
 return import_module(self.urlconf_name)
 File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
 __import__(name)
 File "/usr/src/app/icourt/urls.py", line 100, in <module>
 urls.include('cms.urls')
 File "/usr/local/lib/python2.7/site-packages/django/conf/urls/__init__.py", line 50, in include
 urlconf_module = import_module(urlconf_module)
 File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
 __import__(name)
 File "/usr/local/lib/python2.7/site-packages/cms/urls.py", line 19, in <module>
 urlpatterns = get_app_patterns()
 File "/usr/local/lib/python2.7/site-packages/cms/appresolver.py", line 189, in get_app_patterns
 site = get_current_site()
 File "/usr/local/lib/python2.7/site-packages/cms/utils/__init__.py", line 16, in get_current_site
 return Site.objects.get_current()
 File "/usr/local/lib/python2.7/site-packages/django/contrib/sites/models.py", line 63, in get_current
 return self._get_site_by_id(site_id)
 File "/usr/local/lib/python2.7/site-packages/django/contrib/sites/models.py", line 35, in _get_site_by_id
 site = self.get(pk=site_id)
 File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
 return getattr(self.get_queryset(), name)(*args, **kwargs)
 File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 374, in get
 num = len(clone)
 File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 232, in __len__
 self._fetch_all()
 File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1121, in _fetch_all
 self._result_cache = list(self._iterable_class(self))
 File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
 results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
 File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 887, in execute_sql
 cursor = self.connection.cursor()
 File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/ext/django/db.py", line 81, in cursor
 original_cursor = getattr(self, attr)(*args, **kwargs)
 File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
 return self._cursor()
 File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
 self.ensure_connection()
 File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
 self.connect()
 File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 191, in connect
 self.init_connection_state()
 File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 208, in init_connection_state
 timezone_changed = self.ensure_timezone()
 File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 201, in ensure_timezone
 cursor.execute(self.ops.set_time_zone_sql(), [timezone_name])
 File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/core/models/subsegment.py", line 58, in __call__
 meta_processor=None,
 File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/core/recorder.py", line 417, in record_subsegment
 subsegment = self.begin_subsegment(name, namespace)
 File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/core/recorder.py", line 284, in begin_subsegment
 segment = self.current_segment()
 File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/core/recorder.py", line 267, in current_segment
 entity = self.get_trace_entity()
 File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/core/recorder.py", line 369, in get_trace_entity
 return self.context.get_trace_entity()
 File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/core/context.py", line 93, in get_trace_entity
 return self.handle_context_missing()
 File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/core/context.py", line 118, in handle_context_missing
 raise SegmentNotFoundException(MISSING_SEGMENT_MSG)
aws_xray_sdk.core.exceptions.exceptions.SegmentNotFoundException: cannot find the current segment/subsegment, please make sure you have a segment open

[chanchiem]

I believe that this is a known limitation when running Django locally. Please read more about it here. Specifically, it looks like the migrator is attempting to create a subsegment when no parent segment exists.

The documentation talks about how runserver may fail because runserver performs migration checks which will attempt to begin a subsegment. In your case, it looks like you're performing the migration directly and that's what's causing this issue. In the documentation, it talks about how you may either set the context missing strategy to LOG_ERROR so that it emits the error on server startup, or you may define a ready() method that is launched at startup and manually create a segment by calling:

from aws_xray_sdk.core import xray_recorder
xray_recorder.begin_segment("Migration Segment")

Would you be able to try that and see if it solves your problem?

[krisb78]

I tried the LOG_ERROR setting, but now I'm getting another error (still trying to run ./manage.py migrate):

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 327, in execute
    self.check()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 359, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 62, in _run_checks
    issues.extend(super(Command, self)._run_checks(**kwargs))
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 346, in _run_checks
    return checks.run_checks(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/core/checks/registry.py", line 81, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/usr/local/lib/python2.7/site-packages/django/core/checks/urls.py", line 16, in check_url_config
    return check_resolver(resolver)
  File "/usr/local/lib/python2.7/site-packages/django/core/checks/urls.py", line 26, in check_resolver
    return check_method()
  File "/usr/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 256, in check
    for pattern in self.url_patterns:
  File "/usr/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 407, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 400, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/src/app/icourt/urls.py", line 100, in <module>
    urls.include('cms.urls')
  File "/usr/local/lib/python2.7/site-packages/django/conf/urls/__init__.py", line 50, in include
    urlconf_module = import_module(urlconf_module)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/local/lib/python2.7/site-packages/cms/urls.py", line 19, in <module>
    urlpatterns = get_app_patterns()
  File "/usr/local/lib/python2.7/site-packages/cms/appresolver.py", line 189, in get_app_patterns
    site = get_current_site()
  File "/usr/local/lib/python2.7/site-packages/cms/utils/__init__.py", line 16, in get_current_site
    return Site.objects.get_current()
  File "/usr/local/lib/python2.7/site-packages/django/contrib/sites/models.py", line 63, in get_current
    return self._get_site_by_id(site_id)
  File "/usr/local/lib/python2.7/site-packages/django/contrib/sites/models.py", line 35, in _get_site_by_id
    site = self.get(pk=site_id)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 374, in get
    num = len(clone)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 232, in __len__
    self._fetch_all()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1121, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 887, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python2.7/site-packages/aws_xray_sdk/ext/django/db.py", line 81, in cursor
    original_cursor = getattr(self, attr)(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
    return self._cursor()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
    self.connect()
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 192, in connect
    connection_created.send(sender=self.__class__, connection=self)
  File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 193, in send
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python2.7/site-packages/jsonfield/fields.py", line 166, in configure_database_connection
    loads=lambda x: x)
  File "/usr/local/lib/python2.7/site-packages/psycopg2/_json.py", line 178, in register_default_jsonb
    loads=loads, oid=JSONB_OID, array_oid=JSONBARRAY_OID, name='jsonb')
  File "/usr/local/lib/python2.7/site-packages/psycopg2/_json.py", line 147, in register_json
    register_type(JSON, not globally and conn_or_curs or None)
TypeError: argument 2 must be a connection, cursor or None

[krisb78]

Tried beginning a segment in ready(), same error.

Did some debugging there, it looks like

<XRayTracedConn at 0x7f6f1885ca70 for psycopg2.extensions.connection at 0x7f6effb0f188>, which is being passed into register_type as the second arg,
is not compatible with psycopg2.extensions.connection at 0x7f6effb0f188, at least from psycopg2's point of view...

[krisb78]

(this is with psycopg2==2.7.7)

[krisb78]

Yeah, if I do:

conn = not globally and conn_or_curs or None
register_type(JSON, conn.__wrapped__)

I don't get the error.

[krisb78]

The problem is really this:

ipdb> from aws_xray_sdk.ext.dbapi2 import XRayTracedConn
ipdb> from psycopg2.extensions import connection        
ipdb> issubclass(XRayTracedConn, connection)            
False

Since the XRayTracedConn is not a subclass of psycopg's connection, this has the potential of breaking more things imho...

[chanchiem]

Nice job finding the root cause of the issue :)! This is definitely something that will need some further investigation on my end. Does the solution you showed cause any other issues with your application? Is it a viable solution for now until we are able to find a more generalized solution with X-Ray?

[krisb78]

This is not really a solution, I was only able to do this in a ipdb debugging session. One option could be to have a separate settings file for migration (with no xray), but since the real issue is the incompatibility between the wrapped db connection and psycopg2, there could be other things at risk.

It looks to me like that this should ideally be fixed in wrapt itself, there seem to be some open issues relating to this (i haven't read through them in detail, though), e.g.: GrahamDumpleton/wrapt#130

I'll need to put the integration on hold until this is resolved. Happy to help if I can.

[chanchiem]

For now, I'll mark this as a bug until we can dig deeper to find a solution. As you had mentioned, it looks like the main issue is that wrapt is not properly subclassing the connection or cursor object, so it's not being marked as one as seen in the stacktrace you had sent.

[ezet]

I have added more details to this issue in #243 . I recommend merging this, #243 and #245 into a single issue.
I am currently testing a monkey-patched solution for this in Django, using .__wrapped__ if the instance is XRayTracedConn.

[srprash]

Hi @ezet Good suggestion. We can track this issue in #243 and reduce some clutter here.
@krisb78 Since the original error message was resolved, I'll close this one and tracking in #243

Thanks!

[dev-lusaja]

I have the same problem, and I am reading about lambda-powertools:
https://aws.amazon.com/es/blogs/developer/following-serverless-best-practices-with-aws-chalice-and-lambda-powertools/