Increase user-friendliness for new survey / instrument proposals. See also #30 for eBOSS specific.

[michaelJwilson]

Specsim is written to be flexible enough to consider new telescope / instrument proposals. While this is largely achieved, there are some teething problems after changes.

Baseline suggestion:
Most proposals under consideration are a tweak off DESI. PFS etc are not, due to the near-IR arm.
Consider including default near-IR that can be included / built upon.

Considerations:
-- For near-IR, need:
-- extended (dark) sky model. E.g. Sullivan and Simcoe.
-- extended moon model. E.g. Wehrli '85
-- extended atmospheric extinction. I have one for Mauna Kea.
-- All files need extensions to longer wavelengths, e.g. instrument throughput / resolution,
22AB source.
-- Well known observatory config files. E.g. NAOJ.

-- Specific problem that is hard to rectify:
L136-140 of camera.py after changing wavelength coverage in config and quickspectra.
Not clearly apparent what is desired - ccd_start? Should this be set by e.g. multiplex / petal
parameter.

-- Documentation on considerations when attempting to change instrument / telescope.

My use case:
-- PFS, hard but rough implementation ... near-IR arm etc.
-- DESI spectrographs on the Magellan (easy). Mostly 6.5m and a few tweaks. No changes to wavelength coverage.

See also #30 for eBOSS specific ticket.

Thanks @dkirkby!

[dkirkby]

What max wavelength do you think is necessary for a useful near-IR extension? PFS extends to 1.26um. Is that sufficient?

-- All files need extensions to longer wavelengths, e.g. instrument throughput / resolution,
22AB source.

The wavelength coverage is not hardcoded anywhere, but making changes from the DESI default is fragile and this needs to be improved.

[michaelJwilson]

PFS works for me and is likely representative of what's possible - MSE is also 1.3um. I don't know what interest there is in specsim from outside parties.

The wavelength coverage is not hardcoded anywhere, but making changes from the DESI default is fragile and this needs to be improved.

On a practical level, there are alot of files in which the wavelength is 'hard corded'. To the extent that less would have to be changed, it'd be easier if their default covered to e.g. 1.5um where possible. E.g. the 22AB mag. file is one example. Although some are instrument specific, it might be easier to e.g. have a file extend to 1.5um in which the throughput drops to zero, but that is built to mesh well with the optics / ccd constraints etc., as a suggestion.

[michaelJwilson]

Following up on this, what's the best way to proceed? A first list of preliminary list things I had to do for PFS, in config order.

wavelength_grid:
unit: Angstrom
min: 3796.0
max: 12600.0
step: 0.1

-- Change atmosphere to include near-IR with Sullivan and Simcoe
(https://arxiv.org/abs/1207.0817). I can provide the .txt file.

-- Near-IR transmission spectrum from here:
http://www.gemini.edu/sciops/telescopes-and-sites/observing-condition-constraints/ir-transmission-spectra. Again, I have the .txt with joint KPNO.

-- name: PFS
constants:
primary_mirror_diameter: 8.200 m # Effective light-affecting area.
obscuration_diameter: 0.0 m ## I assume I thought effective above included obscuration.
support_width: 0.025 m
fiber_diameter: 189.0 um # Diameter (Core/Cladding/Buffer)
# 127 / 165 / 189 um @
# Table 1 of https://pdfs.semanticscholar.org/507c/1b23c8e790a00de7\
# cb65eade8101b58eb8c9.pdf
field_radius: 224.0 mm # DESI: 414.0 mm; twice of distance from field centre to farthest
# positioner is 448 mm for PFS.

-- Platescale.txt. I have a PFS version that seems to stop at the field radius, desi is extended as above. I can't imagine I'd have changed this unless it threw an error without it, so presumably a necessary change.

-- fiberloss: eyeballing what I have for the ELG, I've linearly extended this from 1um to the 12.6um red limit. DESI seems to be linear leading up to 1um. Needs changed for all targets.

-- blur and offset are the 'same', e.g. throughput/DESI-0347_random_offset_1.fits. The specsim I've adapted is at least a year old.

These values are copied from Table 1 of https://arxiv.org/pdf/1206.0737.pdf

cameras:
b:
constants: # These values are copied from Table 1 of https://arxiv.org/pdf/1206.0737.pdf
read_noise: 3.0 electron/pixel2
dark_current: 1.4004 electron/(hour pixel2) # We treat "pixel" as a linear unit, so we need
# pixel**2 here.
gain: 1.0 electron/adu
num_sigmas_clip: 5 # Clip the resolution matrix at this number of sigmas.
output_pixel_size: 0.71 Angstrom

-- I've played around with the optics as following. I've seemingly taken the redwards upturn from DESI (colours), PFS in black out with a mean in the middle of the arm and wildly extrapolated.

-- Tophat-like throughputs can be extracted from the bottom of https://pfs.ipmu.jp/research/parameters.html.
-- Starting with spec-ABmag22.0.dat increase red end limit from 1um -> 1.5 to be safe?
-- observation I have as the same.

I've made a fork already, is the best way for me to start a fresh repo and commit + push there and you quickly review each one?

Thanks!

[dkirkby]

The core algorithms of specsim do not make any assumptions about the wavelength coverage of the instrument, but experience has shown that the steps required to adapt the DESI example to other instruments are not obvious and tricky to get right.

I see two levels to this issue:

• infrastructure improvements to make it easy to change the overall wavelength coverage in one place, with automatic padding of instrument throughputs, etc, so things work immediately and can then be iteratively refined.
• improved default dark sky emission and transmission curves with sufficient wavelength coverage (0.3 - 1.5um?) to be reasonable starting points for new instruments.

I've made a fork already, is the best way for me to start a fresh repo and commit + push there and
you quickly review each one?

Ideally, a new instrument just requires a new yaml file equivalent to data/config/desi.yaml that mostly refers to tabulated data files that are external to specsim, at least initially. Once that is working, then we can decide which files are sufficiently stable and of general interest to move into the repo.

[michaelJwilson]

Ok, understood. There's such a .yaml and external files here:
https://github.com/michaelJwilson/specsim/tree/master/aux_dat

Where I've started double checking and uploading what I've done previously. I've reached to fiberloss in terms of PFS version files and all runs fine (via quickspectra). In particular, the plots directory shows the main events.

The README shows a rough log, including questions (so far) that I'd be grateful if you could comment on.

Qs:
-- Is 2 angstrom sampling sufficient for the near-IR transmission curve; This has been downsampled to match the MKO optical transmission.

-- Should I be considering whether applying the extinction to the sky emission is necessary? Does this depend on the type of emission curve being called? ie. will sky emission curves be corrected for extinction typically?

-- PFS seems to have a Hexagonal focal plane. For field radius, you can then match R as the maximum separation on the plane, or equate the area to pi * R *R. Is there a better choice?

-- I've kept a similar curve to DESI for the plate scale. But adjusted to match the ~10% expected change in scale across the field for PFS (strictly this is 10% with a fiber microlens, not sure if that matter), and to match the 90.9 um/'' at the center. Is there a reason to think the shape would be different?

-- Is there anything smarter that could / should be done than linearly extrapolate the fiberloss to the near-IR? As I have currently.

-- I haven't touched the blur and offset files from desi.yaml. Not sure if this matters, and why an error isn't thrown that the wavelength coverage isn't sufficient.

Thanks! Happy New Year.

[michaelJwilson]

Further updates... as documented in the README and plots/ here:
https://github.com/michaelJwilson/specsim/tree/master/aux_dat

I think I'm almost there in updating the necessary files. The sole actual code change to date is adding a NAOJ entry to the observatories dict within specsim.transform.

For the remaining file, I still don't really understand what it is. For desi.yaml, it is the row_size, fwhm_resolution and neff_spatial entries of specpsf/psf_quicksim.fits

Any chance I can derive these from the dispersion Aa/pix and resolution A quoted in Table 1 of https://arxiv.org/pdf/1608.01075.pdf. Otherwise, any ideas are appreciated.

My previous quick hack was to simply extend the DESI values, but I have no idea if this is any where near legitimate. Colours are the DESI values, while black shows my PFS equivalent.

Thanks!

[michaelJwilson]

Potentially useful... the --- I/O FUNCTIONS --- for Chris Hirata's ETC (gsetc.c) read:

Read a spectrograph configuration file.
Format:
all lines starting with # are ignored.
all other lines start with a keyword.
OPTICS primary diam central obscuration EFL FOV rad/deg
SPOT rms/axis @ center - - - rms/axis @ edge
VIGNET center - - - - - [this line is optional; if excluded 1 is assumed]
FIBER entry radius/um
ARMS N arms
PARAM arm # lmin lmax npix width
CAMERA arm # f/ratio ccd thickness pix scale T rms spot fiber image diam dark read trace sep
THRPUT N thr
lambda_0 Thr(lambda_0) [1st] ... Thr(lambda_0) [5th]
lambda_N-1 Thr(lambda_N-1) [1st] ... Thr(lambda_N-1) [5th]

which are available for PFS here:
https://github.com/Subaru-PFS/spt_ExposureTimeCalculator/blob/master/python/pfsspecsim/config/PFS.dat

[michaelJwilson]

Ok, success (I believe).

With some help from KG on the camera specifics, the config is complete and runs (via quickspectra). Further testing likely required.

Link above is broken. New one is here:
https://github.com/michaelJwilson/specsim/tree/master/aux

[michaelJwilson]

CCD properties assumed:

ccd.pdf

[michaelJwilson]

Further improvements / unknowns:

-- support width.
-- gain assumed to be 1e- /ADU.
-- Blur/offset (turn these off?)
-- fiber positions.
-- fiberloss method. fiber microlens?