Nf chunking

[rachelelim]

Fixed chunking with no leftovers

[pep8speaks]

Hello @rachelelim! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file hexrd/grainmap/nfutil.py:

Line 1328:1: E302 expected 2 blank lines, found 1
Line 1328:42: E231 missing whitespace after ','
Line 1328:57: E231 missing whitespace after ','
Line 1328:71: E231 missing whitespace after ','
Line 1328:80: E501 line too long (87 > 79 characters)
Line 1331:18: E711 comparison to None should be 'if cond is None:'
Line 1332:23: E225 missing whitespace around operator
Line 1332:45: E231 missing whitespace after ','
Line 1333:18: E225 missing whitespace around operator
Line 1333:43: E231 missing whitespace after ','
Line 1335:23: E225 missing whitespace around operator
Line 1335:45: E231 missing whitespace after ','
Line 1336:18: E225 missing whitespace around operator
Line 1336:43: E231 missing whitespace after ','

• In the file scripts/process_nf.py:

Line 129:14: E261 at least two spaces before inline comment
Line 129:15: E262 inline comment should start with '# '
Line 211:1: E265 block comment should start with '# '
Line 213:1: E265 block comment should start with '# '
Line 236:1: E303 too many blank lines (5)
Line 236:64: E225 missing whitespace around operator
Line 236:78: E231 missing whitespace after ','
Line 236:80: E501 line too long (99 > 79 characters)
Line 238:1: E128 continuation line under-indented for visual indent
Line 238:1: E265 block comment should start with '# '
Line 239:1: E128 continuation line under-indented for visual indent
Line 239:1: E265 block comment should start with '# '
Line 240:1: E128 continuation line under-indented for visual indent
Line 240:1: E265 block comment should start with '# '
Line 240:44: W291 trailing whitespace
Line 264:1: E128 continuation line under-indented for visual indent
Line 264:1: E265 block comment should start with '# '
Line 267:1: E128 continuation line under-indented for visual indent
Line 268:5: E128 continuation line under-indented for visual indent
Line 268:19: E251 unexpected spaces around keyword / parameter equals
Line 268:21: E251 unexpected spaces around keyword / parameter equals
Line 269:80: E501 line too long (85 > 79 characters)
Line 271:5: E128 continuation line under-indented for visual indent
Line 273:5: E128 continuation line under-indented for visual indent
Line 273:24: E251 unexpected spaces around keyword / parameter equals
Line 273:26: E251 unexpected spaces around keyword / parameter equals
Line 276:5: E128 continuation line under-indented for visual indent
Line 277:40: E251 unexpected spaces around keyword / parameter equals
Line 277:42: E251 unexpected spaces around keyword / parameter equals
Line 280:1: E128 continuation line under-indented for visual indent
Line 282:5: E128 continuation line under-indented for visual indent
Line 282:19: E251 unexpected spaces around keyword / parameter equals
Line 282:21: E251 unexpected spaces around keyword / parameter equals
Line 283:5: E128 continuation line under-indented for visual indent
Line 283:24: E251 unexpected spaces around keyword / parameter equals
Line 283:26: E251 unexpected spaces around keyword / parameter equals
Line 285:5: E128 continuation line under-indented for visual indent
Line 286:5: E128 continuation line under-indented for visual indent
Line 287:23: E251 unexpected spaces around keyword / parameter equals
Line 287:25: E251 unexpected spaces around keyword / parameter equals
Line 288:80: E501 line too long (85 > 79 characters)
Line 290:23: E251 unexpected spaces around keyword / parameter equals
Line 290:25: E251 unexpected spaces around keyword / parameter equals
Line 291:80: E501 line too long (94 > 79 characters)
Line 293:56: E251 unexpected spaces around keyword / parameter equals
Line 293:58: E251 unexpected spaces around keyword / parameter equals
Line 293:80: E501 line too long (88 > 79 characters)
Line 297:80: E501 line too long (105 > 79 characters)
Line 297:83: E251 unexpected spaces around keyword / parameter equals
Line 297:85: E251 unexpected spaces around keyword / parameter equals
Line 300:80: E501 line too long (110 > 79 characters)
Line 300:83: E251 unexpected spaces around keyword / parameter equals
Line 300:85: E251 unexpected spaces around keyword / parameter equals
Line 302:5: E128 continuation line under-indented for visual indent
Line 302:5: E265 block comment should start with '# '
Line 303:5: E128 continuation line under-indented for visual indent
Line 304:23: E251 unexpected spaces around keyword / parameter equals
Line 304:25: E251 unexpected spaces around keyword / parameter equals
Line 306:23: E251 unexpected spaces around keyword / parameter equals
Line 306:25: E251 unexpected spaces around keyword / parameter equals
Line 307:80: E501 line too long (94 > 79 characters)
Line 308:56: E251 unexpected spaces around keyword / parameter equals
Line 308:58: E251 unexpected spaces around keyword / parameter equals
Line 308:80: E501 line too long (88 > 79 characters)
Line 310:1: W293 blank line contains whitespace
Line 312:48: E251 unexpected spaces around keyword / parameter equals
Line 312:50: E251 unexpected spaces around keyword / parameter equals
Line 313:1: W293 blank line contains whitespace
Line 315:48: E251 unexpected spaces around keyword / parameter equals
Line 315:50: E251 unexpected spaces around keyword / parameter equals
Line 316:1: W293 blank line contains whitespace
Line 317:5: E128 continuation line under-indented for visual indent
Line 317:5: E265 block comment should start with '# '
Line 318:5: E128 continuation line under-indented for visual indent
Line 318:14: E251 unexpected spaces around keyword / parameter equals
Line 318:16: E251 unexpected spaces around keyword / parameter equals
Line 319:5: E128 continuation line under-indented for visual indent
Line 319:19: E251 unexpected spaces around keyword / parameter equals
Line 319:21: E251 unexpected spaces around keyword / parameter equals
Line 319:59: W291 trailing whitespace

[joelvbernier]

is this a dupicate PR to #461 @rachelelim ?

[joelvbernier]

What is this hardcoded value? Don't like it, as it is generally bad practice. Python has utils for finding available memory:

import psutil

mem_info = psutil.virtual_memory()

# to get available memory in bytes, do:
# mem_info.available

[joelvbernier]

See above. Not good to hardcode things when you can ask the system for available memory. Available means not having to use disk.

[joelvbernier]

@rachelelim

your tasks to merge this:

• use psutil to set memory chunks (I'd use available_memory from the psutil.virtual_memory() object)
• fix the PEP8 issues. You can use Spyder (version >= 5) to auto-pep8 your code.

[joelvbernier]

Gonna make one comment: In places where you have placeholder strings, start thinking about a YAML config that specifies the required variables.