Rollover fails to start on es cluster which was using rollover before

[pavolloffay]

/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
Detected ElasticSearch Version 7
Creating index template jaeger-span
{"acknowledged":true}
Creating index jaeger-span-000001
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/curator/actions.py", line 559, in do_action
    self.client.indices.create(index=self.name, body=self.body)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/indices.py", line 105, in create
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 343, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 257, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'resource_already_exists_exception', 'index [jaeger-span-000001/bUHw2FzARlityt-oiTwsCA] already exists')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/es-rollover/esRollover.py", line 234, in <module>
    main()
  File "/es-rollover/esRollover.py", line 64, in main
    perform_action(action, client, write_alias, read_alias, prefix+'jaeger-span', 'jaeger-span')
  File "/es-rollover/esRollover.py", line 82, in perform_action
    create_index(client, index)
  File "/es-rollover/esRollover.py", line 116, in create_index
    raise e
  File "/es-rollover/esRollover.py", line 113, in create_index
    create.do_action()
  File "/usr/local/lib/python3.8/site-packages/curator/actions.py", line 566, in do_action
    raise exceptions.FailedExecution('Index %s already exists.' % self.name)
curator.exceptions.FailedExecution: Index jaeger-span-000001 already exists.

Example CR

# setup an elasticsearch with `make es`
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  query:
    image: "pavolloffay/jaeger-query:es-scripts-tls2"
  collector:
    image: "pavolloffay/jaeger-collector:es-scripts-tls2"
  storage:
    type: elasticsearch
    options:
      es:
        # Note: This assumes elasticsearch is running in the "default" namespace.
        server-urls: https://quickstart-es-http.default.svc:9200
        use-aliases: true
#        tls.ca: /es/secrets/ca.crt
        tls.skip-host-verify: true
#        username: elastic
#        password: ql7hbmqfzzkrtn6klcdsh8n5
    secretName: jaeger-secret
    esIndexCleaner:
      image: "pavolloffay/jaeger-es-index-cleaner:es-scripts-tls2"
      schedule: "*/1 * * * *"
    esRollover:
      image: "pavolloffay/jaeger-es-rollover:es-scripts-tls2"
      schedule: "*/2 * * * *"
  volumeMounts:
    - name: secrets
      mountPath: /es/secrets/
      readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: quickstart-es-http-certs-public

[pavolloffay]

make es

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        # Note: This assumes elasticsearch is running in the "default" namespace.
        server-urls: http://elasticsearch.default.svc:9200
        use-aliases: true
    esRollover:
      conditions: '{"max_age": "10s"}'
      schedule: "*/2 * * * *"
      image: pavolloffay/jaeger-es-rollover:rollover-idempotent

and delete the instance and create again.

[sergeykad]

@pavolloffay I am getting a similar error with Jaeger 1.53.0 deployed via Helm chart 3.2.0.
Is it a regression or a different issue?

Error: failed to create index: jaeger-span-000001, request failed, status code: 400, body: {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [jaeger-span-000001], already exists as alias","index_uuid":"_na_","index":"jaeger-span-000001"}],"type":"invalid_index_name_exception","reason":"Invalid index name [jaeger-span-000001], already exists as alias","index_uuid":"_na_","index":"jaeger-span-000001"},"status":400}

The failure occurs during helm upgrade execution.

esRollover:
  enabled: true
  cmdlineParams:
    es.use-ilm: "true"
    es.ilm-policy-name: "30-days-default" 
  initHook:
    extraEnv:
      - name: SHARDS
        value: "3"
      - name: REPLICAS
        value: "0"