Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YSQL] Support loading large data sets into YSQL using COPY #5241

Closed
rkarthik007 opened this issue Jul 28, 2020 · 0 comments
Closed

[YSQL] Support loading large data sets into YSQL using COPY #5241

rkarthik007 opened this issue Jul 28, 2020 · 0 comments
Assignees
Labels
area/docdb YugabyteDB core features area/ysql Yugabyte SQL (YSQL) kind/enhancement This is an enhancement of an existing feature roadmap-tracking-issue This issue tracks a major roadmap item, and usually appears in the roadmap list.

Comments

@rkarthik007
Copy link
Collaborator

rkarthik007 commented Jul 28, 2020

This is a master issue to track improving transactions which have a very large number of operations, as well as provide various options to batch load data more efficiently.

Status Feature GitHub Issue(s) Comments
Support for large transactions #1923 v2.3.0
Large transactional writes leave large memtables #1906 v2.3.0
Fix OOM when running large COPY FROM #5453 v2.3.1
Support txn batch size for COPY FROM to improve performance for large datasets #2855 v2.3.0
Fix OOM when running large \copy (or COPY FROM stdin) #5603 v2.3.3
Support txn batch size for \copy (and COPY FROM stdin) to improve performance for large datasets #6069 v2.3.3

As noted in the referenced issues above, these were tested (on a small local cluster) with data sets of at least up to 15gb in size (corresponding to ~375 million rows).

Analytics

@rkarthik007 rkarthik007 added kind/enhancement This is an enhancement of an existing feature area/ysql Yugabyte SQL (YSQL) area/docdb YugabyteDB core features labels Jul 28, 2020
@rkarthik007 rkarthik007 added the roadmap-tracking-issue This issue tracks a major roadmap item, and usually appears in the roadmap list. label Jul 28, 2020
@rkarthik007 rkarthik007 assigned emhna and unassigned ndeodhar Sep 10, 2020
@m-iancu m-iancu changed the title Support for large transaction batches and better batch loading [YSQL] Support loading large data sets into YSQL using COPY Oct 28, 2020
@m-iancu m-iancu closed this as completed Oct 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features area/ysql Yugabyte SQL (YSQL) kind/enhancement This is an enhancement of an existing feature roadmap-tracking-issue This issue tracks a major roadmap item, and usually appears in the roadmap list.
Projects
None yet
Development

No branches or pull requests

5 participants