Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Testing] Test the migration speed of Django from the EAV schema to JSONB #2786

Closed
szoupanos opened this issue Apr 22, 2019 · 2 comments · Fixed by #3090
Closed

[Testing] Test the migration speed of Django from the EAV schema to JSONB #2786

szoupanos opened this issue Apr 22, 2019 · 2 comments · Fixed by #3090
Assignees
Milestone

Comments

@szoupanos
Copy link
Contributor

Given the implementation found at aiida_core/django_jsonb branch, we should take and report the time that is needs for a complete migration of a database of a reasonable size (5m to 10M of nodes).

This is related to issue #2762 and can be inspired by that issue.

Based on the performance we can see if the migration needs improvement.

@szoupanos szoupanos self-assigned this Apr 22, 2019
@szoupanos szoupanos added this to the v1.0.1 milestone Apr 22, 2019
@szoupanos
Copy link
Contributor Author

I did a fast check on this using a small database of @giovannipizzi (Giovanni can comment on its contents) that had 63K nodes and 1.3M rows in the attributes table (extras almost empty).

The JSONB migration lasted a bit less than 4,5 mins.

For comparison, the migrations from 0008 to 0025 took 17+ mins (with one of them being particularly slower than the others)

If the migration scales linearly with the number of nodes, an 8M nodes database (with similar contents), will take 9 to 10 hours, which seems a reasonable time.

From a quick look, the migrated attributes seem OK (I saw attributes that make sense, numbers remained numbers and not strings etc)

@szoupanos
Copy link
Contributor Author

So the migration of the attributes and extras of a 7,3M node database (Davide's) finished in 20 hours.
(django migration: 0034_attributes_extras_settings_json)

More specifically, the timings are the following:
real 1208m12.169s
user 388m30.040s
sys 14m47.921s

And the database contained (rows):
Nodes: 7.351.846
Attributes: 282.505.715
Extras: 3.887.413

The database size decreased from 100GB to 26GB

A mechanical drive was used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants