automatic copying within generators

[mileslucas]

A very common workflow for reducing data includes copying files after each operation to have a separate copy for each step of the reduction.

For example, I have 10 bias frames and I want to overscan subtract and trim them before saving to a new location. For each file, I want the new name to be <name>_oc_trim.<ext>. Or perhaps I want them to be saved in a new folder oc_trim/<name>.<ext> (or maybe both!).

We can automate this process using the generators from #27 with some new keyword arguments

• save - save the files using the options below.
• save_folder - new folder to save in
• save_prefix - add to name like <prefix>_<name>.<ext>
• save_suffix - add to end of name like <name>_<suffix>.<ext>
• save_delim="_" - the thing to put before/after the prefix or suffix

The general idea is that if any of save_folder, save_prefix, save_suffix is given then the files are copied and saved (ie save=true), otherwise they are not (ie save=false). The user can manually write save=false, for example, to get filenames using the mechanics provided without actually copying data.

Now, in terms of implementation I don't have a perfect idea yet, but I would generally want the users to be able to avoid manually calling any write functions. In other words, if I set these keywords appropriately, there should be nothing different in my for-loop compared to a version that doesn't save. To do this, you'll have to find a way using ResumableFunctions to get a "return value". We can discuss further ideas to accomplish this.