Hide links from old versions in link checker

[MizukiTemma]

Describe the Bug

Internal server error will be caused during link replacement in the broken link list.

Steps to Reproduce

1. Go to https://admin.integreat-app.de/lk-emmendingen/analytics/linkcheck/invalid/?page=2
2. Find the link with linktext "Hauptstraße 15, 79341 Kenzingen"
3. Replace the URL with "https://www.google.com/maps/place/Hauptstra%C3%9Fe+15,+79341+Kenzingen/@48.1918235,7.7665451,17z/data=!3m1!4b1!4m6!3m5!1s0x47913cec05b5a17f:0xe65c8600197c8b17!8m2!3d48.19182!4d7.76912!16s%2Fg%2F11c5fw72pv?entry=ttu"
4. . See error

Expected Behavior

The link will be successfully updated.

Actual Behavior

Internal server error occurs

Additional Information

This issue is related to #2465

Traceback

Oct 16 09:24:55 ERROR   django.request - 500 Internal Server Error: /lk-emmendingen/analytics/linkcheck/invalid/746380/
Traceback (most recent call last):
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
  return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pagetranslation_unique_version"
DETAIL:  Key (page_id, language_id, version)=(14470, 74, 5) already exists.

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
  response = get_response(request)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
  response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view
  return self.dispatch(request, *args, **kwargs)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/utils/decorators.py", line 43, in _wrapper
  return bound_method(*args, **kwargs)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
  return view_func(request, *args, **kwargs)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/integreat_cms/cms/views/linkcheck/linkcheck_list_view.py", line 117, in dispatch
  return super().dispatch(request, *args, **kwargs)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
  return handler(request, *args, **kwargs)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/integreat_cms/cms/views/linkcheck/linkcheck_list_view.py", line 189, in post
  new_translation.save()
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/integreat_cms/cms/models/abstract_content_translation.py", line 591, in save
  super().save(*args, **kwargs)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
  self.save_base(using=using, force_insert=force_insert,
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
  updated = self._save_table(
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
  results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
  return manager._insert(
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
  return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
  return query.get_compiler(using=using).execute_sql(returning_fields)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
  cursor.execute(sql, params)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/cacheops/transaction.py", line 98, in execute
  result = self._no_monkey.execute(self, sql, params)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
  return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
  return executor(sql, params, many, context)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
  return self.cursor.execute(sql, params)
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
  raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
  return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "pagetranslation_unique_version"
DETAIL:  Key (page_id, language_id, version)=(14470, 74, 5) already exists.

[timobrembeck]

Hmm, the real source of the problem seems to be earlier (in the background job, not during link replacement):

Oct 16 09:06:57 ERROR   linkcheck.listeners - IntegrityError while running do_check_instance_links with args=(<class 'integreat_cms.cms.models.pages.page_translation.PageTranslation'>, <PageTranslation
(id: 5383498, page_id: 17315, language: de, slug: beratung)>, True) and kwargs={}: duplicate key value violates unique constraint "linkcheck_url_url_key"
DETAIL:  Key (url)=(https://www.hagen.de/irj/portal/FB-BRAR) already exists.
Traceback (most recent call last):
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "linkcheck_url_url_key"
DETAIL:  Key (url)=(https://www.hagen.de/irj/portal/FB-BRAR) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/linkcheck/listeners.py", line 33, in linkcheck_worker
    task['target'](*task['args'], **task['kwargs'])
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/linkcheck/listeners.py", line 107, in do_check_instance_links
    u.check_url()
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/linkcheck/models.py", line 282, in check_url
    return self.check_external(external_recheck_interval)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/linkcheck/models.py", line 469, in check_external
    self.save()
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/cacheops/transaction.py", line 98, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "linkcheck_url_url_key"
DETAIL:  Key (url)=(https://www.hagen.de/irj/portal/FB-BRAR) already exists.

So I'll blame our inconsistent database at the moment, I don't think there is a new bug in the link checker, just the existing one (with the root cause DjangoAdminHackers/django-linkcheck#168)

[timobrembeck]

Since this happened again after a few days, it seems to be relevant enough that we need a more informative error message here.
I think we should point out that the link is from an old version and that the users need to wait a week (to make sure the linkcheck scan can fix the inconsistency) or contact their administrator.

[PeterNerlich]

Do you have a method to reproduce the bug locally?

[timobrembeck]

Do you have a method to reproduce the bug locally?

No, it's not possible to reproduce the race condition deterministically.
However, you can just create a link object for an old translation version manually via the Django shell or via Django admin.