Issue: Large files continuously crash during data cleaning in QualCoder

[ChefDani]

Hello,

I’ve been encountering an issue where large files continuously cause QualCoder to crash during the data cleaning process. These files seem to exceed the program's capacity to handle memory effectively, especially with extensive datasets and multiple coding processes. Here are the key details:

QualCoder version: 3.5
Operating system: Windows

Steps to reproduce:

•     Load a large dataset into QualCoder ((original) file size over 3 MB).
  

•     Begin coding or cleaning the data.
  

•     After a few operations, the program crashes without saving the progress.
  

I’ve tried reducing the file size, but this is not feasible for my workflow. Is there a workaround for handling larger datasets or a solution that could help improve memory management?

Any assistance or guidance would be greatly appreciated.

Thanks!

[ccbogel]

Hmm, I don't have a large data file to test this. If it is possible to view yours - you can email me a zip of the project folder, I can test and see what could be done (if anything).
What do you mean by a large dataset ? - lots of files, or very large files. Specifically - very large text files.
Or also lots of attributes (e.g. case or file variables - are being loaded from a survey)?
When you state data cleaning - what are you doing for that exactly - is this editing a large text document before coding?

If it's a very large text file - can you edit it, i.e. clean it with another software before importing into QualCoder.
When opening the large file in Code Text: You can right-click on the file name and chunk text. So that smaller sections are loaded at a time.
From the wiki:
Large text files slow QualCoder. When opening a large text file (greater than 50,000 characters), right click on the file name. A menu option allows you to get the first approximately 50,000 characters, subsequent clicks will open the next approximately 50,000 and so on. Approximately, because QualCoder looks for a line-ending (such as a paragraph or sentence ending) to more nicely section the start and end of text chunks.

[ccbogel]

OK thanks. I have made one update so far, that will use the QPlainTextEdit which may be faster, and may not crash.

[kaixxx]

@ccbogel I'm also wondering if SQLite is able to handle such large blobs of text.

[kaixxx]

roughly 85 pages

That sounds much more reasonable!

[ccbogel]

@kaixx, a good question. I checked. Sqlite can handle text up to 1 billion bytes. So character encoding (depending on encoding) uses several bytes, UTF-8 uses up to 4 bytes. So a max length string would be 250 million characters. Displaying that in the Qt Widgets would be the issue with memory limits (crashing) and slowness.

[ccbogel]

I am backing up my data before I open the zip file. So I may or may not be able to view it tonight. 10.35pm here.

[ccbogel]

OK I did one file update in the latest code (3.6) which might help a little.

The main thing to make the editing faster is to do this:
Go to Mange Cases. Open that specific case in the Files Colum Then REMOVE the file from the case.
Then go to Manage Files. Then you can edit the file text with speed.
Then when finished editing (click OK), re-assign that file to the Case.

The problem is that whenever a character is changed (added or deleted), QualCoder is using the difflib module, which is slow for large files.

[ChefDani]

Will definitely check!

[ccbogel]

OK I have identified why it is so slow. There is a module called difflib that I use in DialogEditTextFile.update_positions()
difflib looks at differences in the previous to the changed text, when every character is added or deleted. This module is slow with large text files. There aremodules called diff_lib_patch and cdifflib which are faster. So I will look into this.

[ccbogel]

OK I have updated the code to use diff-lib-patch module instead of the difflib module. This makes editing text 20x faster and no visible slowness.
TODO need to update code_text edit text functionality to match.

[ChefDani]

How does one update the code_text edit text functionality, exactly on Qualcoder?

[ccbogel]

I will do that soon. but first there is a big branch merge to sort out.
Did you test out the edit speed change in Manage File > Edit text file from the latest code?