Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use an accelerated numpy reader (np.genfromtxt) #446

Closed
kinverarity1 opened this issue Apr 16, 2021 · 3 comments
Closed

Use an accelerated numpy reader (np.genfromtxt) #446

kinverarity1 opened this issue Apr 16, 2021 · 3 comments
Assignees
@kinverarity1
Labels
data-section-parser A bug or enhancement relating to the data section parser enhancement las3 stuff relating to LAS 3.0

Comments

@kinverarity1
Copy link
Owner

Is your feature request related to a problem? Please describe.
lasio is quite slow - ref #444 - and we should use pandas' highly-optimised read functions, given that the LAS data section is often (not always) quite simply structured.

Describe the solution you'd like
Add an optional data section reader which uses pd.read_fwf and/or pd.read_csv internally could be a significant improvement. The use of it would be controlled by a keyword argument engine='pandas' in LASFile.read.

Additional context
There have previously been discussions around this issue in #1.
@kinverarity1 kinverarity1 added enhancement las3 stuff relating to LAS 3.0 data-section-parser A bug or enhancement relating to the data section parser labels Apr 16, 2021
@kinverarity1 kinverarity1 changed the title Use an accelerated pandas reader e.g. pd.read_csv/pd.read_fwf Use an accelerated pandas reader e.g. pd.read_csv/pd.read_fwf Apr 16, 2021
@kinverarity1 kinverarity1 mentioned this issue Apr 16, 2021
Open
9 tasks
@kinverarity1 kinverarity1 changed the title Use an accelerated pandas reader e.g. pd.read_csv/pd.read_fwf Use an accelerated numpy or pandas reader e.g. pd.read_csv/pd.read_fwf Apr 16, 2021
@kinverarity1
Copy link
Owner Author

See #450 for a draft PR of using pd.read_csv for this. Big performance improvements - more than twice as fast. But not all features or tests are passing.

@kinverarity1 kinverarity1 self-assigned this Apr 17, 2021
@dcslagel dcslagel mentioned this issue Apr 22, 2021
Merged
@kinverarity1
Copy link
Owner Author

Update: #452 will eventually resolve this issue. I'll update this issue title to reflect our preferred solution.

@kinverarity1 kinverarity1 changed the title Use an accelerated numpy or pandas reader e.g. pd.read_csv/pd.read_fwf Use an accelerated numpy reader (np.genfromtxt) Apr 24, 2021
@kinverarity1
Copy link
Owner Author

#452 was merged recently 🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kinverarity1 kinverarity1

Labels
data-section-parser A bug or enhancement relating to the data section parser enhancement las3 stuff relating to LAS 3.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kinverarity1