From 2d2d4af3c5e6c369446b1d7cc24157f439e10943 Mon Sep 17 00:00:00 2001 From: Bruno Sanchez Date: Tue, 5 Sep 2023 15:19:42 +0200 Subject: [PATCH] Update documentation for ip.diffim Remove documentation pages with outdated information. Include new tasks doc pages and fix docstrings. Add a new overview page for ip.diffim Adding the examples in the landing page Fix bug on docstring Fix review comments Update doc/lsst.ip.diffim/index.rst Co-authored-by: Ian Sullivan Update doc/lsst.ip.diffim/overview_ipdiffim.rst Co-authored-by: Ian Sullivan Fix review comments Fix review comments --- doc/lsst.ip.diffim/AL_implementation.rst | 8 +- doc/lsst.ip.diffim/code_notes.rst | 219 ------------------ doc/lsst.ip.diffim/index.rst | 64 ++++- doc/lsst.ip.diffim/overview_ipdiffim.rst | 78 +++++++ ...fim.AlardLuptonPreconvolveSubtractTask.rst | 59 +++++ ...lsst.ip.diffim.AlardLuptonSubtractTask.rst | 64 +++++ ...st.ip.diffim.DetectAndMeasureScoreTask.rst | 56 +++++ .../lsst.ip.diffim.DetectAndMeasureTask.rst | 60 +++++ .../tasks/lsst.ip.diffim.MakeKernelTask.rst | 57 ++--- 9 files changed, 396 insertions(+), 269 deletions(-) delete mode 100644 doc/lsst.ip.diffim/code_notes.rst create mode 100644 doc/lsst.ip.diffim/overview_ipdiffim.rst create mode 100644 doc/lsst.ip.diffim/tasks/lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask.rst create mode 100644 doc/lsst.ip.diffim/tasks/lsst.ip.diffim.AlardLuptonSubtractTask.rst create mode 100644 doc/lsst.ip.diffim/tasks/lsst.ip.diffim.DetectAndMeasureScoreTask.rst create mode 100644 doc/lsst.ip.diffim/tasks/lsst.ip.diffim.DetectAndMeasureTask.rst diff --git a/doc/lsst.ip.diffim/AL_implementation.rst b/doc/lsst.ip.diffim/AL_implementation.rst index 3a55e501..5c846a90 100644 --- a/doc/lsst.ip.diffim/AL_implementation.rst +++ b/doc/lsst.ip.diffim/AL_implementation.rst @@ -40,9 +40,11 @@ Basis functions Solving for the matching kernel starts with the generation of the basis functions. The number of basis functions and the properties of the Gaussian bases (e.g. widths in pixels) are determined in -`lsst.ip.diffim.makeKernelBasisList.generateAlardLuptonBasisList`. The -decision logic depends on configuration and psf fwhm values of the -images, some details is given in the function API documentation. +`lsst.ip.diffim.makeKernelBasisList.generateAlardLuptonBasisList`. +The full config for this is being set in the module +`lsst.ip.diffim.psfMatch` config classes. The decision logic depends +on this configuration and psf fwhm values of the images, some details +are given in the function API documentation. .. TODO More details about the basis function generation diff --git a/doc/lsst.ip.diffim/code_notes.rst b/doc/lsst.ip.diffim/code_notes.rst deleted file mode 100644 index 03753cdc..00000000 --- a/doc/lsst.ip.diffim/code_notes.rst +++ /dev/null @@ -1,219 +0,0 @@ -################################# -Package usage and technical notes -################################# - -This page is a collection of usage and code related notes about the -image differencing implementation. We do not summarise the Alard-Lupton -(AL) [AL_1998]_ and Zackay, Ofek, Gal-Yam (ZOGY) [ZOGY2016]_ papers -themselves here. - - -Usage notes -=========== - -Supported input data --------------------- - -ImageDifferenceTask supports two types of data products as templates, -*coadds* and *calexps*, which can be subtracted from calexp science -exposures *only*. Other combinations of inputs are not supported. The -input mode is selected by the ``getTemplate`` configurable field. The -``getTemplate`` subtask is a properly retargetable top level -``pexConfig.ConfigurableField`` with two supported target subtasks: -``GetCoaddAsTemplate`` and ``GetCalexpAsTemplate``. - -Enabling processing steps -------------------------- - -There is a sequence of pre-subtraction and post-subtraction processing -steps included in ``ImageDifferenceTask`` around the actual -subtraction operation of the images. Including the subtraction -operation itself, these steps can be enabled or disabled by top level -``do`` configuration options. These top-level configuration -options are summarised in :numref:`flowchart1` and -:numref:`flowchart2` (`flowchart source and standalone pdf version -`_ -). Some of these top level configuration options are also passed on to -invoked subtasks and influence their functionality. They may not be -specified for the subtasks directly. - -.. _flowchart1: - -.. figure:: figures/ImageDifference_flowchart.draw.io-Page-1.svg - :align: center - :alt: Subtasks page 1 - - Top level subtasks and top level configuration options in - ``ImageDifferenceTask``. - - Following the reading of the template and science images, the task - starts with the preprocessing of the science exposure on the top - and ends with post processing steps following the subtraction on - the bottom. Figure source - :download:`here `. - -.. _flowchart2: - -.. figure:: figures/ImageDifference_flowchart.draw.io-Page-2.svg - :align: center - :alt: Subtasks page 2 - - Top level configuration options in ``ZogyTask``. - -Specifying a coadd as template ------------------------------- - -Using the ``GetCoaddAsTemplate`` subtask, we specify one or more -science exposures by the ``--id`` general data specification -option. Tract or patch specification as part of the ``--id`` option -are silently ignored. The given (``coaddName``, defaults to "deep") -type coadd is looked up in the same input repository automatically. -As of Butler generation 2, if multiple coadd templates are reachable -through the repository parent references, one of them is picked -automatically. - -The subtask detemines the sky corners of a box that fully contains the -science exposure with a configurable boundary -(``templateBorderSize``). The subtask selects all patches of the coadd -skymap that overlap with the science exposure and cuts them -accordingly to assemble the template coadd image. The template coadd -image inherits the coadd WCS. It is an error if there is no -overlapping region. The ``--templateId`` option is silently -ignored. The following example specifies one science exposure and -implicitly the deep coadd residing in the same repository for image -differencing: - -.. code-block:: bash - - imageDifference.py repo/ingested/ --id visit=410915 filter=g - ccdnum=25 - -``GetCoaddAsTemplate`` also supports loading ``DcrModel`` coadds and -produce a color corrected template matching the ``filter`` of the -science exposure. - -Specifying a calexp as template -------------------------------- - -Using the ``GetCalexpAsTemplate`` subtask, we can select two calexp -data products for subtraction. The ``--id`` option specifies a data -reference of one or more science exposures (calexp). The -``--templateId`` specifies a data reference for the template -(calexp). Fields that are not specified for the template are inherited -from the science exposure data reference. The following example -subtract visit 410915 ccdnum 25 (template) from 411055 ccdnum 25 -(science exposure): - -.. code-block:: bash - - imageDifference.py repo/calibimgs --id visit=411055 - ccdnum=25 --templateId visit=410915 - -Implementation notes -==================== - -General -------- - -If the two images have the same origin and extent (same WCS) the -subtraction is performed pixel by pixel. Otherwise, the *template* -exposure is warped and resampled into the WCS frame of the science -exposure. - -The AL kernel fitting is entirely implemented in C++, an adaptation of -the `hotpants `_ C package -by Becker, while the ZOGY algorithm is entirely implemented in -Pyton based on mostly on the ``numpy`` FFT functionality. - -Beside the top level configuration options, the AL algorithm has its -own set of separate configuration parameters, while the ZOGY algorithm -does not have any algorithm specific options. - -Alard-Lupton algorithm ----------------------- - -The Alard-Lupton numerical fitting for the coefficients (of the -composing basis kernel functions) of the image convolution kernel is -performed independently on a number of stamp images around selected -sources. These independent solutions are called *kernel candidates* -throughout the code. The final kernel solution model for the whole -image is a smoothly varying spatial fit of the coefficients as a -function of the image pixel coordinates. - -If ``convolveTemplate==False`` the science exposure is convolved and -then the template is subtracted from the science image. - -The performance of the AL algorithm was studied in details in the -[Becker_LDM-227]_ report. This study forms the basis of the AL -algorithm default values; the degree of the polynomial -multiplicator of the Gaussian kernel basis functions (``degGauss``), -the degree of the polynomial that is fitted to the spatial variaton of -the solution coefficients accross the image (``spatialKernelOrder``) -and the default detection thresholds (5.5 sigma). As a legacy of this -study, the ``Winter2013ImageDifferenceTask`` is still available in -``imageDifference.py`` though it is unclear which test data repository -it requires. - -Due to noise in the template image, convolving the template introduces -correlation in the noise in the template image. The AL algorithm was -improved by an additional *afterburner* decorrelation to remove the -noise correlation in the image difference. The implemented -decorrelation method and its mathematical formulae of the -decorrelation kernel is summarised and studied in [Reiss_DMTN-021]_. - -Zackay-Ofek-Gal-Yam algorithm ------------------------------ - -[ZOGY2016]_ is free from the assumption that the template is noise -free or specially selected by any other means. We simply deal with two -images with different PSFs and noise characteristics (sigma). In the -basic version of the algorithm, the random noise in the pixels are -assumed to be background dominated i.e. uncorrelated between pixels -and independent of the pixel values. Also we assume that the noise has -zero expectation value i.e. the expectation value of the random noise -is already removed. ZOGY shows that if these assumptions hold, the -difference image noise is also independent and identically distributed -over its pixels (white) by construction, there is no need to -decorrelate the noise in the difference image. - -Following the variance addition rule of the difference of uncorrelated -random variables, exactly the same steps are repeated on the exposure -variance planes as on the data planes, only the subtraction step is -replaced by addition. - -The *nan* values are removed from the science and template images -before Fourier transformations and replaced by the image mean -values. On the immage difference, the mask plane ``UNMASKEDNAN`` is -set for pixels where originally any of the two inputs or the -difference result is *nan*. - -Pre-convolution is not implemented in the ZOGY algorithm. In case of -the ZOGY algorithm, the ``doPreConvolve==True`` config option selects -the *detection likelihood* image to be returned *instead of* the -difference image. Under the assumptions of the algorithm, this image -carries the likelihood ratio test statistic values similarly to the -usual match filter-convolved image and can be used for threshold -source detections. The ``S`` detection likelihood (or score) image -(eq. 12 in [ZOGY2016]_) and its *corrected variance* ``S_var`` (the -*denominator* of eq. 25 [ZOGY2016]_) are calculated and returned, -following the *simple* correction steps presented in the paper Section -3.3. This signal correction is introduced to account for the source -noise (bright sources) and also for other systematic noise -sources. The iterative approach of section 3.5 is not implemented. - - -References -========== - -.. [AL_1998] Alard, C.; Lupton, Robert H. A Method for Optimal Image - Subtraction - -.. [Reiss_DMTN-021] Reiss J. David, Lupton, Robert H. DMTN-021: - Implementation of Image Difference Decorrelation - -.. [ZOGY2016] Zackay B., Ofek E. O., Gal-Yam A., - Proper Image Subtraction—Optimal Transient Detection, - Photometry, and Hypothesis Testing, 2016, ApJ, 830, 27 - -.. [Becker_LDM-227] Becker A. et al. LDM-227 Report on Late Winter2013 - Production: Image Differencing diff --git a/doc/lsst.ip.diffim/index.rst b/doc/lsst.ip.diffim/index.rst index ce1f67db..5c0eae44 100644 --- a/doc/lsst.ip.diffim/index.rst +++ b/doc/lsst.ip.diffim/index.rst @@ -6,7 +6,26 @@ lsst.ip.diffim ############## -The ``lsst.ip.diffim`` module provides algorithms for astronomical image differencing. +.. Paragraph that describes what this Python module does and links to related modules and frameworks. + +The ``lsst.ip.diffim`` module provides algorithms and utilities for astronomical **image differencing** and **transient detection**. + +.. _lsst.ip.diffimg-overview: + +Overview of lsst.ip.diffim +========================== + +.. toctree linking to the basic overview of the package + +.. toctree:: + :maxdepth: 1 + +.. toctree:: + :maxdepth: 1 + + overview_ipdiffim + AL_implementation + .. _lsst.ip.diffim-using: @@ -15,14 +34,47 @@ Using lsst.ip.diffim .. toctree linking to topics related to using the module's APIs. -.. toctree:: +.. toctree :maxdepth: 1 -.. toctree:: +.. toctree :maxdepth: 1 - code_notes - AL_implementation +Production pipelines +-------------------- + +This package is used in production pipelines like ``lsst.ap.pipe`` and ``lsst.drp.pipe``, designed for processing large collections of survey data. + +For an example of how the two primary `~lsst.pipe.base.PipelineTasks` in this package (`~lsst.ip.diffim.AlardLuptonSubtractTask` and `~lsst.ip.diffim.DetectAndMeasureTask`) are used in a production pipeline, see `ap_pipe/pipelines/ApPipe.yaml`. + + +Single frame processing +----------------------- + +Single image frame usage can be carried out by importing the `~lsst.ip.diffim` as a library of tasks, as the following example shows: + +.. code-block:: python + + from lsst.ip.diffim import subtractImages + + ... + subtract_task = subtractImages.AlardLuptonSubtractTask() + + subtraction = subtract_task.run( + warped_template, science, science_source_catalog) + +For source detection a similar pattern works: + +.. code-block:: python + + from lsst.ip.diffim import DetectAndMeasureTask + + ... + detect_task = DetectAndMeasureTask() + + detect_and_measure = detect_and_measure_task.run( + science, subtraction.matchedTemplate, subtraction.difference) + .. _lsst.ip.diffim-contributing: @@ -61,7 +113,7 @@ Configurations .. lsst-configs:: :root: lsst.ip.diffim - :toctree: configs + :toctree: config .. _lsst.ip.diffim-pyapi: diff --git a/doc/lsst.ip.diffim/overview_ipdiffim.rst b/doc/lsst.ip.diffim/overview_ipdiffim.rst new file mode 100644 index 00000000..5382cb6a --- /dev/null +++ b/doc/lsst.ip.diffim/overview_ipdiffim.rst @@ -0,0 +1,78 @@ +################ +Package overview +################ + +``ip_diffim`` contains pipeline tasks to perform image subtraction and transient source detection on calibrated exposures, template images, and their image differences. +The various implementations of these tasks are described below. + +Image subtraction tasks +======================= + +The image differencing tasks in this package are implementations of the Alard & Lupton (1998) algorithm (AL) [AL_1998]_, and this can be summarised as applying a kernel determination algorithm in order to match observational properties between two images. +The method to match images varies according to which task is used, in combination with configuration parameters of such tasks. + + - The pipelines provide the `regular` AL implementation in the `~lsst.ip.diffim.AlardLuptonSubtractTask`, where a kernel is determined to transform the template image; + - the `auto-convolution` AL version where the kernel and transformation is applied at either the template or science image, according to its PSF properties --this is when the key parameter `mode` in the configuration of `~lsst.ip.diffim.AlardLuptonSubtractTask` is set to `auto`; + - the `pre-convolution` AL implementation of `~lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask` that uses an ad-hoc convolution kernel that is applied to the science image before calculating the optimal transformation kernel. + + +AlardLuptonSubtractTask +----------------------- + +This task is the ``ip_diffim`` implementation of AL algorithm, including further improvements such as de-correlation procedure (introduced in [Reiss_DMTN-021]_). +The task inputs are always a `template` image and a `science` image to subtract, and as well associated source catalogs and image properties information for the science exposure. +It allows to the user to configure whether to apply the de-correlation procedure, and some parameters that affect the catalog used for kernel determination, background subtraction, etc. + +This task is able as well of performing the mentioned `regular` and `auto-convolution` mode of image subtraction. In the former the `template` image is convolved with a kernel, and in the alternative the _science_ image is the one being transformed with the convolution kernel. +Finally, the `auto` setting delegates this decision to the task itself, and so when processing a batch of images some might be processed with `convolveTemplate` and the rest with the `convolveScience` version. + + +AlardLuptonPreconvolveSubtractTask +---------------------------------- + +The `pre-convolve` task runs a preparation stage before applying the AL algorithm itself, so the determination of the kernel for the `template` image is never a "deconvolution" kernel (negative wing profile, as shown in [Kovacs_DMTN-179]_). + +In this stage the `science` image is transformed with a convolution kernel :math:`H`, that is tipically the PSF of this image, becoming a "score image" for detection. +Once this transformation is applied, standard A&L algorithm derives a kernel for the `template` image so it matches the `science score` image properties, and the subtraction is obtained. \ +It is needed to clarify that this result is again a "score" image, and Maximum-Likelihood source detection can be run on it without the standard PSF cross-correlation step. + +If one wants to obtain the canonical difference image, then first a deconvolution with :math:`H` is needed. +Any correlations introduced in the noise properties are later taken into account by means of the standard final de-correlation [Reiss_DMTN-021]_. + + +Source detection and measurement tasks +====================================== + +Obtaining difference images is a means to the end of transient and variable source detection. +For LSST to find the variation in brightness of sources in the sky, it is needed to perform the task of finding the pixels with signal and perform a set of measurements on them. + +The main objective is to obtain a catalog of position, shapes and brightness of each identified source, and in addition, sets of flags indicating masked pixels, inaccuracies in the determination of measured parameters, and extra pieces of information about the pixel region related to each source. + + +DetectAndMeasureTask +-------------------- + +This task is able to perform maximum-likelihood source identification and determination of photometric parameters from difference images that were obtained with the A&L algorithm implementation in `~lsst.ip.diffim.AlardLuptonSubtractTask`. + +In this task, a pre-processing stage is needed where the resulting difference image is cross-correlated with the `science` image PSF profile, in order to obtain a "score image", suitable for maximum-likelihood source detection. + + +DetectAndMeasureScoreTask +------------------------- + +This task obtains sources and its properties from images resulting from the `~lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask`. + +In this case, the score image used comes straight from the subtraction task and no additional image processing is needed. + + +References +========== + +.. [AL_1998] Alard, C.; Lupton, Robert H. A Method for Optimal Image + Subtraction + +.. [Reiss_DMTN-021] Reiss J. David, Lupton, Robert H. DMTN-021: + Implementation of Image Difference Decorrelation + +.. [Kovacs_DMTN-179] Gabor Kovacs. DMTN-179 + The ZOGY image differencing matching kernel and PSF solutions and their practical implementation issues \ No newline at end of file diff --git a/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask.rst b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask.rst new file mode 100644 index 00000000..665471e1 --- /dev/null +++ b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask.rst @@ -0,0 +1,59 @@ +.. lsst-task-topic:: lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask + +################################## +AlardLuptonPreconvolveSubtractTask +################################## + +AlardLuptonPreconvolveSubtractTask is a modification of AlardLuptonSubtractTask behavior, and it performs image differencing between the `template` image and a custom kernel cross-correlated `science` image. + +.. _lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask-description: + +Description +=========== + +This task takes the pair of science and template exposures, together with the science image source catalog, and produces difference and matched template exposures as outputs. + +The process involves pre-convolving the `science` image with a predefined kernel (tipically its own PSF) before running the nominal set of steps for image subtraction, that is, selecting sources to obtain a matching convolution kernel, applying this to the `template` image, and performing the image differencing. + +In this case the subtraction produces a score image directly, and it can be used for maximum-likelihood source detection in a straightforward manner. This is done using `~lsst.ip.diffim.DetectAndMeasureScoreTask`. + + +.. _lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask-api: + +Python API summary +================== + +.. lsst-task-api-summary:: lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask + +.. _lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask-subtasks: + +Retargetable subtasks +===================== + +.. lsst-task-config-subtasks:: lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask + +.. _lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask-configs: + +Configuration fields +==================== + +.. lsst-task-config-fields:: lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask + +.. _lsst.ip.diffim.AlardLuptonPreconvolveSubtractTask-debug: + +.. Debugging +.. ========= + + +.. The ``pipetask`` command line interface supports a ``--debug`` flag to import +.. ``debug.py`` from your PYTHONPATH; see :ref:`lsstDebug` for more about ``debug.py`` +.. files. +.. The available variables in AlardLuptonPreconvolveSubtractTask include: + + +.. display : `bool` +.. Enable debug display output. +.. maskTransparency : `float` +.. Transparency of mask planes in the output display. +.. displayDiaSources : `bool` +.. Show exposure with dipole fitting results. diff --git a/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.AlardLuptonSubtractTask.rst b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.AlardLuptonSubtractTask.rst new file mode 100644 index 00000000..d35a2061 --- /dev/null +++ b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.AlardLuptonSubtractTask.rst @@ -0,0 +1,64 @@ +.. lsst-task-topic:: lsst.ip.diffim.AlardLuptonSubtractTask + +####################### +AlardLuptonSubtractTask +####################### + +AlardLuptonSubtractTask performs image differencing, including decorrelation. + + +.. _lsst.ip.diffim.AlardLuptonSubtractTask-description: + +Description +=========== + +This task takes the pair of science and template exposures, together with the science image source catalog, and produces difference and matched template exposures as outputs. + +The process involves selecting sources from the science exposure source catalog, obtaining the convolution kernel, applying this to the `target` image, and subtracting the images. + +As discussed in the general package overview the task has two possible modes: one convolving the `template` image and the alternative convolving the `science` image. +Both modes share the preparation sub-tasks, but the method that performs the convolution is not the same. +The parameter that controls this behavior is ``mode`` (check `~lsst.ip.diffim.AlardLuptonSubtractConfig`), and it allows ``convolveScience``, ``convolveTemplate`` and finally ``auto`` which means that the decision is made at runtime automatically. + +Finally, mask planes are set on the difference and matched template outputs based on the masks of the input science and template images. + + +.. _lsst.ip.diffim.AlardLuptonSubtractTask-api: + +Python API summary +================== + +.. lsst-task-api-summary:: lsst.ip.diffim.AlardLuptonSubtractTask + +.. _lsst.ip.diffim.AlardLuptonSubtractTask-subtasks: + +Retargetable subtasks +===================== + +.. lsst-task-config-subtasks:: lsst.ip.diffim.AlardLuptonSubtractTask + +.. _lsst.ip.diffim.AlardLuptonSubtractTask-configs: + +Configuration fields +==================== + +.. lsst-task-config-fields:: lsst.ip.diffim.AlardLuptonSubtractTask + +.. _lsst.ip.diffim.AlardLuptonSubtractTask-debug: + +.. Debugging +.. ========= + + +.. The ``pipetask`` command line interface supports a ``--debug`` flag to import +.. ``debug.py`` from your PYTHONPATH; see :ref:`lsstDebug` for more about ``debug.py`` +.. files. +.. The available variables in AlardLuptonSubtractTask include: + + +.. display : `bool` +.. Enable debug display output. +.. maskTransparency : `float` +.. Transparency of mask planes in the output display. +.. displayDiaSources : `bool` +.. Show exposure with dipole fitting results. diff --git a/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.DetectAndMeasureScoreTask.rst b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.DetectAndMeasureScoreTask.rst new file mode 100644 index 00000000..3340d4e6 --- /dev/null +++ b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.DetectAndMeasureScoreTask.rst @@ -0,0 +1,56 @@ +.. lsst-task-topic:: lsst.ip.diffim.DetectAndMeasureScoreTask + +######################### +DetectAndMeasureScoreTask +######################### + +DetectAndMeasureScoreTask performs source detection and execution of measurement plugins in difference images. + +.. _lsst.ip.diffim.DetectAndMeasureScoreTask-description: + +Description +=========== + +This task detects sources in score-like difference images, such as the ones obtained with pre-convolution image differencing, or ZOGY "S" images. +This means that cross-convolution is not needed, and the task obtains maximum-likelihood source candidates in a single step. +Once candidate sources are obtained this task performs the same operations as `~lsst.ip.diffim.DetectAndMeasureTask`, including measurements in the conventional difference image, not the score-like image; for further reference consult the documentation of `~lsst.ip.diffim.DetectAndMeasureTask`. + +.. _lsst.ip.diffim.DetectAndMeasureScoreTask-api: + +Python API summary +================== + +.. lsst-task-api-summary:: lsst.ip.diffim.DetectAndMeasureScoreTask + +.. _lsst.ip.diffim.DetectAndMeasureScoreTask-subtasks: + +Retargetable subtasks +===================== + +.. lsst-task-config-subtasks:: lsst.ip.diffim.DetectAndMeasureScoreTask + +.. _lsst.ip.diffim.DetectAndMeasureScoreTask-configs: + +Configuration fields +==================== + +.. lsst-task-config-fields:: lsst.ip.diffim.DetectAndMeasureScoreTask + +.. _lsst.ip.diffim.DetectAndMeasureScoreTask-debug: + +.. Debugging +.. ========= + + +.. The ``pipetask`` command line interface supports a ``--debug`` flag to import +.. ``debug.py`` from your PYTHONPATH; see :ref:`lsstDebug` for more about ``debug.py`` +.. files. +.. The available variables in DetectAndMeasureScoreTask include: + + +.. display : `bool` +.. Enable debug display output. +.. maskTransparency : `float` +.. Transparency of mask planes in the output display. +.. displayDiaSources : `bool` +.. Show exposure with dipole fitting results. diff --git a/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.DetectAndMeasureTask.rst b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.DetectAndMeasureTask.rst new file mode 100644 index 00000000..25fe6920 --- /dev/null +++ b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.DetectAndMeasureTask.rst @@ -0,0 +1,60 @@ +.. lsst-task-topic:: lsst.ip.diffim.DetectAndMeasureTask + +##################### +DetectAndMeasureTask +##################### + +DetectAndMeasureTask performs source detection and execution of measurement plugins in difference images. + +.. _lsst.ip.diffim.DetectAndMeasureTask-description: + +Description +=========== + +This task cross-convolves the difference image with a PSF approximation to obtain maximum-likelihood source candidates. +Measurements are run on all the detections, including positive and negative flux candidates. +Close sources will be combined, enabling trailed sources as well as dipole-like source detection. + +The plugins used by this task by default are meant to provide basic measurements for all types of transient sources: point sources, trailed sources and dipoles as well. +Additional forced measurements are provided through PSF flux photometry measurements and centroid determination plugins. +Finally, the plugins make use of the image mask planes to provide a wide range of source flags, including streak flagging and injected source flagging. + +.. _lsst.ip.diffim.DetectAndMeasureTask-api: + +Python API summary +================== + +.. lsst-task-api-summary:: lsst.ip.diffim.DetectAndMeasureTask + +.. _lsst.ip.diffim.DetectAndMeasureTask-subtasks: + +Retargetable subtasks +===================== + +.. lsst-task-config-subtasks:: lsst.ip.diffim.DetectAndMeasureTask + +.. _lsst.ip.diffim.DetectAndMeasureTask-configs: + +Configuration fields +==================== + +.. lsst-task-config-fields:: lsst.ip.diffim.DetectAndMeasureTask + +.. _lsst.ip.diffim.DetectAndMeasureTask-debug: + +.. Debugging +.. ========= + + +.. The ``pipetask`` command line interface supports a ``--debug`` flag to import +.. ``debug.py`` from your PYTHONPATH; see :ref:`lsstDebug` for more about ``debug.py`` +.. files. +.. The available variables in DetectAndMeasureTask include: + + +.. display : `bool` +.. Enable debug display output. +.. maskTransparency : `float` +.. Transparency of mask planes in the output display. +.. displayDiaSources : `bool` +.. Show exposure with dipole fitting results. diff --git a/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.MakeKernelTask.rst b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.MakeKernelTask.rst index 9c16485f..40443052 100644 --- a/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.MakeKernelTask.rst +++ b/doc/lsst.ip.diffim/tasks/lsst.ip.diffim.MakeKernelTask.rst @@ -11,26 +11,15 @@ MakeKernelTask creates a PSF-matching kernel for two images. Description =========== -Build a Psf-matching kernel using two input images, either as MaskedImages -(in which case they need to be astrometrically aligned) or Exposures (in which -case astrometric alignment will happen by default but may be turned off). This -requires a list of input Sources which may be provided by the calling Task; if -not, the Task will perform a coarse source detection and selection for this -purpose. Sources are vetted for signal-to-noise and masked pixels (in both the -template and science image), and substamps around each acceptable source are -extracted and used to create an instance of KernelCandidate. Each -KernelCandidate is then placed within a `~lsst.afw.math.SpatialCellSet`, which is -used by an ensemble of `~lsst.afw.math.CandidateVisitor` instances to build the -Psf-matching kernel. These visitors include, in the order that they are -called: ``BuildSingleKernelVisitor``, ``KernelSumVisitor``, ``BuildSpatialKernelVisitor``, -and ``AssessSpatialKernelVisitor``. - -Upon initialization, the kernel configuration is defined by -``self.config.kernel.active``. The task creates an `~lsst.afw.math.Warper` from the -subConfig ``self.config.kernel.active.warpingConfig``. A schema for the selection -and measurement of candidate `~lsst.ip.diffim.KernelCandidates` is defined, and -used to initize subTasks selectDetection (for candidate detection) and -selectMeasurement(for candidate measurement). +Build a Psf-matching kernel using two input images, either as MaskedImages or Exposures. +This requires a list of input Sources which may be provided by the calling Task; if not, the Task will perform a coarse source detection and selection for this purpose. +Sources are vetted for signal-to-noise and masked pixels (in both the template and science image), and substamps around each acceptable source are extracted and used to create an instance of KernelCandidate. +Each KernelCandidate is then placed within a `~lsst.afw.math.SpatialCellSet`, which is used by an ensemble of `~lsst.afw.math.CandidateVisitor` instances to build the Psf-matching kernel. +These visitors include, in the order that they are called: ``BuildSingleKernelVisitor``, ``KernelSumVisitor``, ``BuildSpatialKernelVisitor``, and ``AssessSpatialKernelVisitor``. + +Upon initialization, the kernel configuration is defined by ``self.config.kernel.active``. +The task creates an `~lsst.afw.math.Warper` from the subConfig ``self.config.kernel.active.warpingConfig``. +A schema for the selection and measurement of candidate `~lsst.ip.diffim.KernelCandidates` is defined, and used to initize subTasks selectDetection (for candidate detection) and selectMeasurement(for candidate measurement). Sigma clipping of KernelCandidates is performed as follows: @@ -38,24 +27,12 @@ Sigma clipping of KernelCandidates is performed as follows: * KernelSumVisitor, using the mean and standard deviation of the kernel sum from all candidates, if PsfMatchConfig.kernelSumClipping is True * AssessSpatialKernelVisitor, using the substamp diffim ressiduals from the spatial kernel fit, if PsfMatchConfig.spatialKernelClipping is True -The actual solving for the kernel (and differential background model) happens -in ``lsst.ip.diffim.PsfMatchTask._solve``. This involves a loop over the -SpatialCellSet that first builds the per-candidate matching kernel for the -requested number of KernelCandidates per cell -(``PsfMatchConfig.nStarPerCell``). The quality of this initial per-candidate -difference image is examined, using moments of the pixel residuals in the -difference image normalized by the square root of the variance(i.e. sigma); -ideally this should follow a normal (0, 1) distribution, but the rejection -thresholds are set by the config (``PsfMatchConfig.candidateResidualMeanMax`` and -``PsfMatchConfig.candidateResidualStdMax``). All candidates that pass this initial -build are then examined en masse to find the mean/stdev of the kernel sums -across all candidates. Objects that are significantly above or below the mean, -typically due to variability or sources that are saturated in one image but -not the other, are also rejected.This threshold is defined by -``PsfMatchConfig.maxKsumSigma``. Finally, a spatial model is built using all -currently-acceptable candidates, and the spatial model used to derive a second -set of (spatial) residuals which are again used to reject bad candidates, -using the same thresholds as above. +The actual solving for the kernel (and differential background model) happens in `~lsst.ip.diffim.PsfMatchTask._solve``. +This involves a loop over the SpatialCellSet that first builds the per-candidate matching kernel for the requested number of KernelCandidates per cell (``PsfMatchConfig.nStarPerCell``). +The quality of this initial per-candidate difference image is examined, using moments of the pixel residuals in the difference image normalized by the square root of the variance(i.e. sigma); ideally this should follow a normal (0, 1) distribution, but the rejection thresholds are set by the config (``PsfMatchConfig.candidateResidualMeanMax`` and ``PsfMatchConfig.candidateResidualStdMax``). +All candidates that pass this initial build are then examined en masse to find the mean/stdev of the kernel sums across all candidates. +Objects that are significantly above or below the mean, typically due to variability or sources that are saturated in one image but not the other, are also rejected.This threshold is defined by ``PsfMatchConfig.maxKsumSigma``. +Finally, a spatial model is built using all currently-acceptable candidates, and the spatial model used to derive a second set of (spatial) residuals which are again used to reject bad candidates, using the same thresholds as above. .. _lsst.ip.diffim.MakeKernelTask-api: @@ -83,9 +60,7 @@ Configuration fields Debugging ========= -The ``pipetask`` command line interface supports a ``--debug`` flag to import -``debug.py`` from your PYTHONPATH; see :ref:`lsstDebug` for more about ``debug.py`` -files. +The ``pipetask`` command line interface supports a ``--debug`` flag to import ``debug.py`` from your PYTHONPATH; see :ref:`lsstDebug` for more about ``debug.py`` files. The available variables in MakeKernelTask include: display : `bool`