upgrade from 6.0.0 to 7.0.0 fails with a lot of crashes

[innovia]

Issue Summary

A summary of the issue and the browser/OS environment in which it occurs.
Running on Docker (kubernetes)
got a successful upgrade from 5.0.1 to 6.0.0 (inclduing purging the queues)
now upgrade from 6.0.0 to 7.0.0 fails and crashes a lot

Steps to Reproduce

1. install v5.0.1
2. upgrade to v6.0.0 (manage.py db upgarde)
3. purge the queues
   celery purge --app=redash.worker --queues=queries,scheduled_queries,celery
4. upgrade to v.7.0.0 (manage.py db upgarde)

Any other info e.g. Why do you consider this to be a bug? What did you expect to happen instead?

Technical details:

• Redash Version: 7.0.0.b18042
• Browser/OS: Docker Kubernetes
• How did you install Redash: Docker on kubernetes

log from upgrades

6.0.0.b8537

./manage.py db upgrade
[2019-03-31 18:23:03,798][PID:34][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-03-31 18:23:03,880][PID:34][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2019-03-31 18:23:04,693][PID:34][INFO][root] Latest version: 7.0.0 (newer: True)
[2019-03-31 18:23:05,518][PID:34][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2019-03-31 18:23:05,518][PID:34][INFO][alembic.runtime.migration] Will assume transactional DDL.


celery purge --app=redash.worker --queues=queries,scheduled_queries,celery
WARNING: This will remove all tasks from queues: celery, queries, scheduled_queries.
         There is no undo for this operation!

(to skip this prompt use the -f option)

Are you sure you want to delete all tasks (yes/NO)? yes
Purged 14 messages from 3 known task queues.

v7.0.0
./manage.py db upgrade
[2019-03-31 20:28:53,178][PID:26][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-03-31 20:28:53,201][PID:26][INFO][root] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2019-03-31 20:28:54,695][PID:26][INFO][root] Latest version: 7.0.0 (newer: False)
[2019-03-31 20:28:55,305][PID:26][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2019-03-31 20:28:55,306][PID:26][INFO][alembic.runtime.migration] Will assume transactional DDL.
[2019-03-31 20:28:55,318][PID:26][INFO][alembic.runtime.migration] Running upgrade 71477dadd6ef -> 640888ce445d, Add new scheduling data.
[2019-03-31 20:29:06,486][PID:26][INFO][alembic.runtime.migration] Running upgrade 640888ce445d -> e7f8a917aa8e, Add user details JSON column.
[2019-03-31 20:29:06,511][PID:26][INFO][alembic.runtime.migration] Running upgrade e7f8a917aa8e -> 73beceabb948, bring_back_null_schedule
[2019-03-31 20:29:13,863][PID:26][INFO][alembic.runtime.migration] Running upgrade 73beceabb948 -> 98af61feea92, add_encrypted_options_to_data_sources
[2019-03-31 20:29:13,884][PID:26][INFO][alembic.runtime.migration] Running upgrade 98af61feea92 -> e5c7a4e2df4d, remove_query_tracker_keys

crash logs:

 values to unpack',)
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web Traceback (most recent call last):
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web   File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 382, in trace_task
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web     R = retval = fun(*args, **kwargs)
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web   File "/app/redash/worker.py", line 71, in __call__
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web     return TaskBase.__call__(self, *args, **kwargs)
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web   File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 641, in __protected_call__
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web     return self.run(*args, **kwargs)
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web   File "/app/redash/tasks/queries.py", line 274, in refresh_queries
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web     for query in models.Query.outdated_queries():
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web   File "/app/redash/models.py", line 1034, in outdated_queries
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web     if should_schedule_next(retrieved_at, now, query.schedule, query.schedule_failures):
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web   File "/app/redash/models.py", line 855, in should_schedule_next
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web     hour, minute = schedule.split(':')
redash-celery-worker-python-web-daemon-866f99848d-qt2zp web ValueError: too many values to unpack

[innovia]

more crashes

redash-server-python-web-daemon-7b884ddf44-hx9pw web [2019-03-31 21:14:11,516] ERROR in app: Exception on /api/admin/queries/tasks [GET]
redash-server-python-web-daemon-7b884ddf44-hx9pw web Traceback (most recent call last):
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1988, in wsgi_app
redash-server-python-web-daemon-7b884ddf44-hx9pw web     response = self.full_dispatch_request()
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1641, in full_dispatch_request
redash-server-python-web-daemon-7b884ddf44-hx9pw web     rv = self.handle_user_exception(e)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 271, in error_router
redash-server-python-web-daemon-7b884ddf44-hx9pw web     return original_handler(e)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1544, in handle_user_exception
redash-server-python-web-daemon-7b884ddf44-hx9pw web     reraise(exc_type, exc_value, tb)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1639, in full_dispatch_request
redash-server-python-web-daemon-7b884ddf44-hx9pw web     rv = self.dispatch_request()
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1625, in dispatch_request
redash-server-python-web-daemon-7b884ddf44-hx9pw web     return self.view_functions[rule.endpoint](**req.view_args)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/app/redash/permissions.py", line 48, in decorated
redash-server-python-web-daemon-7b884ddf44-hx9pw web     return fn(*args, **kwargs)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 228, in decorated_view
redash-server-python-web-daemon-7b884ddf44-hx9pw web     return func(*args, **kwargs)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/app/redash/handlers/admin.py", line 51, in queries_tasks
redash-server-python-web-daemon-7b884ddf44-hx9pw web     'tasks': celery_tasks(),
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/app/redash/monitor.py", line 132, in celery_tasks
redash-server-python-web-daemon-7b884ddf44-hx9pw web     tasks = parse_tasks(celery.control.inspect().active(), 'active')
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/app/redash/monitor.py", line 104, in parse_tasks
redash-server-python-web-daemon-7b884ddf44-hx9pw web     for task in itertools.chain(*task_lists.values()):
redash-server-python-web-daemon-7b884ddf44-hx9pw web AttributeError: 'NoneType' object has no attribute 'values'
redash-server-python-web-daemon-7b884ddf44-hx9pw web [2019-03-31 21:14:11,516][PID:108][ERROR][redash] Exception on /api/admin/queries/tasks [GET]
redash-server-python-web-daemon-7b884ddf44-hx9pw web Traceback (most recent call last):
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1988, in wsgi_app
redash-server-python-web-daemon-7b884ddf44-hx9pw web     response = self.full_dispatch_request()
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1641, in full_dispatch_request
redash-server-python-web-daemon-7b884ddf44-hx9pw web     rv = self.handle_user_exception(e)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 271, in error_router
redash-server-python-web-daemon-7b884ddf44-hx9pw web     return original_handler(e)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1544, in handle_user_exception
redash-server-python-web-daemon-7b884ddf44-hx9pw web     reraise(exc_type, exc_value, tb)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1639, in full_dispatch_request
redash-server-python-web-daemon-7b884ddf44-hx9pw web     rv = self.dispatch_request()
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1625, in dispatch_request
redash-server-python-web-daemon-7b884ddf44-hx9pw web     return self.view_functions[rule.endpoint](**req.view_args)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/app/redash/permissions.py", line 48, in decorated
redash-server-python-web-daemon-7b884ddf44-hx9pw web     return fn(*args, **kwargs)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 228, in decorated_view
redash-server-python-web-daemon-7b884ddf44-hx9pw web     return func(*args, **kwargs)
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/app/redash/handlers/admin.py", line 51, in queries_tasks
redash-server-python-web-daemon-7b884ddf44-hx9pw web     'tasks': celery_tasks(),
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/app/redash/monitor.py", line 132, in celery_tasks
redash-server-python-web-daemon-7b884ddf44-hx9pw web     tasks = parse_tasks(celery.control.inspect().active(), 'active')
redash-server-python-web-daemon-7b884ddf44-hx9pw web   File "/app/redash/monitor.py", line 104, in parse_tasks
redash-server-python-web-daemon-7b884ddf44-hx9pw web     for task in itertools.chain(*task_lists.values()):
redash-server-python-web-daemon-7b884ddf44-hx9pw web AttributeError: 'NoneType' object has no attribute 'values'
redash-server-python-web-daemon-7b884ddf44-hx9pw web [2019-03-31 21:14:11,520][PID:108][INFO][metrics] method=GET path=/api/admin/queries/tasks endpoint=redash_queries_tasks status=500 content_type=? content_length=-1 duration=1040.66 query_count=3 query_duration=3.84
redash-server-python-web-daemon-7b884ddf44-db2bx web [2019-03-31 21:14:16,452] ERROR in app: Exception on /api/admin/queries/tasks [GET]
redash-server-python-web-daemon-7b884ddf44-db2bx web Traceback (most recent call last):
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1988, in wsgi_app
redash-server-python-web-daemon-7b884ddf44-db2bx web     response = self.full_dispatch_request()
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1641, in full_dispatch_request
redash-server-python-web-daemon-7b884ddf44-db2bx web     rv = self.handle_user_exception(e)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 271, in error_router
redash-server-python-web-daemon-7b884ddf44-db2bx web     return original_handler(e)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1544, in handle_user_exception
redash-server-python-web-daemon-7b884ddf44-db2bx web     reraise(exc_type, exc_value, tb)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1639, in full_dispatch_request
redash-server-python-web-daemon-7b884ddf44-db2bx web     rv = self.dispatch_request()
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1625, in dispatch_request
redash-server-python-web-daemon-7b884ddf44-db2bx web     return self.view_functions[rule.endpoint](**req.view_args)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/app/redash/permissions.py", line 48, in decorated
redash-server-python-web-daemon-7b884ddf44-db2bx web     return fn(*args, **kwargs)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 228, in decorated_view
redash-server-python-web-daemon-7b884ddf44-db2bx web     return func(*args, **kwargs)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/app/redash/handlers/admin.py", line 51, in queries_tasks
redash-server-python-web-daemon-7b884ddf44-db2bx web     'tasks': celery_tasks(),
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/app/redash/monitor.py", line 132, in celery_tasks
redash-server-python-web-daemon-7b884ddf44-db2bx web     tasks = parse_tasks(celery.control.inspect().active(), 'active')
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/app/redash/monitor.py", line 104, in parse_tasks
redash-server-python-web-daemon-7b884ddf44-db2bx web     for task in itertools.chain(*task_lists.values()):
redash-server-python-web-daemon-7b884ddf44-db2bx web AttributeError: 'NoneType' object has no attribute 'values'
redash-server-python-web-daemon-7b884ddf44-db2bx web [2019-03-31 21:14:16,452][PID:111][ERROR][redash] Exception on /api/admin/queries/tasks [GET]
redash-server-python-web-daemon-7b884ddf44-db2bx web Traceback (most recent call last):
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1988, in wsgi_app
redash-server-python-web-daemon-7b884ddf44-db2bx web     response = self.full_dispatch_request()
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1641, in full_dispatch_request
redash-server-python-web-daemon-7b884ddf44-db2bx web     rv = self.handle_user_exception(e)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask_restful/__init__.py", line 271, in error_router
redash-server-python-web-daemon-7b884ddf44-db2bx web     return original_handler(e)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1544, in handle_user_exception
redash-server-python-web-daemon-7b884ddf44-db2bx web     reraise(exc_type, exc_value, tb)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1639, in full_dispatch_request
redash-server-python-web-daemon-7b884ddf44-db2bx web     rv = self.dispatch_request()
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1625, in dispatch_request
redash-server-python-web-daemon-7b884ddf44-db2bx web     return self.view_functions[rule.endpoint](**req.view_args)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/app/redash/permissions.py", line 48, in decorated
redash-server-python-web-daemon-7b884ddf44-db2bx web     return fn(*args, **kwargs)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/usr/local/lib/python2.7/dist-packages/flask_login/utils.py", line 228, in decorated_view
redash-server-python-web-daemon-7b884ddf44-db2bx web     return func(*args, **kwargs)
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/app/redash/handlers/admin.py", line 51, in queries_tasks
redash-server-python-web-daemon-7b884ddf44-db2bx web     'tasks': celery_tasks(),
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/app/redash/monitor.py", line 132, in celery_tasks
redash-server-python-web-daemon-7b884ddf44-db2bx web     tasks = parse_tasks(celery.control.inspect().active(), 'active')
redash-server-python-web-daemon-7b884ddf44-db2bx web   File "/app/redash/monitor.py", line 104, in parse_tasks
redash-server-python-web-daemon-7b884ddf44-db2bx web     for task in itertools.chain(*task_lists.values()):
redash-server-python-web-daemon-7b884ddf44-db2bx web AttributeError: 'NoneType' object has no attribute 'values'```

[innovia]

and another

redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web Traceback (most recent call last):
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 382, in trace_task
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     R = retval = fun(*args, **kwargs)
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/app/redash/worker.py", line 64, in __call__
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     return TaskBase.__call__(self, *args, **kwargs)
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 641, in __protected_call__
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     return self.run(*args, **kwargs)
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/app/redash/tasks/queries.py", line 190, in refresh_queries
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     metadata={'Query ID': query.id, 'Username': 'Scheduled'})
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/app/redash/tasks/queries.py", line 115, in enqueue_query
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     if job.ready():
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/app/redash/tasks/queries.py", line 91, in ready
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     return self._async_result.ready()
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 311, in ready
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     return self.state in self.backend.READY_STATES
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 471, in state
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     return self._get_task_meta()['status']
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 410, in _get_task_meta
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     return self._maybe_set_cache(self.backend.get_task_meta(self.id))
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 365, in get_task_meta
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     meta = self._get_task_meta_for(task_id)
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 680, in _get_task_meta_for
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     return self.decode_result(meta)
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 284, in decode_result
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     return self.meta_from_decoded(self.decode(payload))
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 280, in meta_from_decoded
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     meta['result'] = self.exception_to_python(meta['result'])
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web   File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 260, in exception_to_python
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web     exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
redash-celery-worker-python-web-daemon-5bcb4c8f65-fjg6l web TypeError: __init__() takes at least 4 arguments (2 given)```

[innovia]

1. Ok, the last one has been merged 4 days ago - the docker image points to latest so I can't use that tag...

2)the token for authentication on v7.0.0 is different from v6.0.0 and when doing a downgrade it tries to get that - i spent 2 hours fighting with it until i figure that this the token stored on a cookie of the browser

[ranbena]

Hey @innovia, as noted in the github issue template, we use GitHub only for bug reports
but for support questions plz use https://discuss.redash.io/c/support, so go ahead and copy this over to there.

Also, plz make sure to read the v7 upgrade notes here https://discuss.redash.io/t/redash-v7-released/3477

[arikfr]

@innovia before opening the thread in the forum, make sure all your containers use the correct image. Based on the logs it looks like you're running non v7 code.