Avoid timeout on SRW frame request for large files

[moellep]

The simulation below creates large intermediary wavefront files during the simulation process.

When the sim_frame() request is made by the client, the wavefront is read into memory on the server, and then converted into an appropriate SRW output file based on the characteristic requested and written to disk. Then that datafile is read and scaled using scipy.ndimage and returned to the client. The request takes too long and can cause a time-out on the server, or potentially on the client.

Instead, and extra step should be take to make a copy of the wavefront during the simulation process, and reduce the scale at that point. Then the sim_frame() request will be working with a much reduced set of data, and should complete quickly.

https://www.sirepo.com/srw#/beamline/D093hGau

[mkeilman]

I'd like to check my approach with some pseudo-code before I get too far. In template/srw.py._generate_srw_main:

for n in beamline_info.names:
    ...
    content.append("v.ws_fnep = '{}'".format(prev_wavefront))
    content.append("op = set_optics(v, names, {})".format(is_last_watch))
    if not is_last_watch:
        content.append("srwl_bl.SRWLBeamline(_name=v.name).calc_all(v, op)")
        # vvv NEW vvv
        content.append("data = pickle load  prev_wavefront file")
        content.append("scipy scale data")
        content.append("pickle save scaled data")
        ...

where "scipy scale data" would emulate _resize_report (which would then be unnecessary?). Do I have the gist of it?

[moellep]

Yes, I think so. We want to keep the input pickle file which SRW uses unscaled, but once it is done with it, we should scale it if it is huge. The sim_frame is looking for the files to become available, so adding a suffix (like "-preprocessed") could keep them from being visible before they have been resized.

So if the generated code has many watchpoints could look like this

    v.ws = True
    v.ws_fne = 'initial-preprocessed.pkl'
    names = ['Aperture','Aperture_CM','CM','CM_M2','M2','M2_Before_Toroid','Before_Toroid']
    v.ws_fnep = 'wid-20-preprocessed.pkl'
    op = set_optics(v, names, False)
    srwl_bl.SRWLBeamline(_name=v.name).calc_all(v, op)

    # 'initial-preprocessed.pk' can be resized into 'initial.pkl'

    names = ['Toroid','After_Toroid']
    v.ws_fnei = 'wid-20-preprocessed.pkl'
    v.ws_fnep = 'wid-21-preprocessed.pkl'
    op = set_optics(v, names, False)
    srwl_bl.SRWLBeamline(_name=v.name).calc_all(v, op)

    # 'wid-20-preprocessed.pkl' can be resize now into 'wid-20.pkl'
    # and 'wid-20-preprocessed.pkl' can be deleted

    names = ['After_Toroid_Before_SSA','Before_SSA']
    v.ws_fnei = 'wid-21-preprocessed.pkl'
    v.ws_fnep = 'wid-17-preprocessed.pkl'
    op = set_optics(v, names, True)
    v.wm = False
    srwl_bl.SRWLBeamline(_name=v.name).calc_all(v, op)

    # 'wid-21-preprocessed.pkl' can be resize now into 'wid-21.pkl'
    # and 'wid-21-preprocessed.pkl' can be deleted
    # the same for the final wavefront 'wid-17'