|
LSSTApplications
10.0-2-g4f67435,11.0.rc2+1,11.0.rc2+12,11.0.rc2+3,11.0.rc2+4,11.0.rc2+5,11.0.rc2+6,11.0.rc2+7,11.0.rc2+8
LSSTDataManagementBasePackage
|
The meas_base package is the new home of the source measurement framework, which was formerly part of meas_algorithms.
The source measurement framework is a set of Python modules which allow measurements to be performed on calibrated exposures. The framework assumes that a detection catalog has been prepared to identify the objects to be measured. The detection catalog may have been produced with a detection pass on the exposure itself, but it might also be produced from other exposures, stacks, or even multiple exposures. Deblending may or may not have been performed during the creation of the detection catalog.
The framework steps through the detection catalog and performs a set of measurements, object by object, supplying each measurement plugin with the exposure and catalog information needed for its measurement. The measurement results (values, errors, and flags) are then placed in an output catalog.
The measurement framework includes the following features:
The single frame measurement framework is used when all the information about the sources to be measured comes from a single image, and hence those sources are detected (and possibly) deblended on that image before measurement. This image may be a coadd of other images, or even a difference of images - from the perspective of the measurement framework there is essentially no difference between these cases (though there may be important differences for particular measurement algorithms).
The high-level algorithm for single-frame measurement is:
The run() method is called on each image to be processed, with a SourceCatalog containing all the sources to be measured. These sources must have Footprints (generated by SourceDetectionTask), and a schema that matches that constructed by the previous step. The fields added to the schema during initialization will then be filled in by the measurement framework.
Before measuring any sources, the measurement framework replaces all sources in the catalog with noise (see NoiseReplacer), using the Footprints attached to the SourceCatalog to define their boundaries.
We then loop over all "parent" sources in the catalog - both those that were not blended, and those that represent the pre-deblend combined state of blends. For each parent, we loop again over all its children (if any), and for each of these, we re-insert the child source into the image (which, recall, currently contains only noise), call measure() on each of the plugins, and then replace the child source with noise again. We the insert the parent source, and call measure() on all of the plugins. Before replacing the parent with noise again, we then call measureN() twice for each plugin: once with the list of all children, and once with a single-element list containing just the parent. This ensures that each source (parent or child) is measured with both measure() and measureN(), with the former preceeding the latter.
Because measurement plugin algorithms are often dependent on each other (in particular, most measurements require a centroid as an input), they must be run in a particular order, and we need a mechanism for passing information between them. The order is defined by the 'executionOrder' config parameter, which is defined in the BasePluginConfig class, and hence present for every plugin. Generally, these will remain at their default values; it is the responsibility of a plugin implementor to ensure the default for that plugin is appropriate relative to any plugins it depends on. See BasePluginConfig.executionOrder for some guidelines (it may be easiest to read the Python docstring directly; Doxygen garbles config documentation).
The mechanism for passing information between plugins is SourceTable's slot system (see lsst::afw::table::SlotDefinition), in which particular measurements are given one of several predefined aliases (e.g. "slot_Centroid" -> "base_SdssCentroid"), which are used to implement getters on SourceRecord (e.g. getCentroid(). The measurement framework's configuration defines which measurements are assigned to each slot, and these slot measurements are available to other plugins as soon as the plugin whose outputs are assigned to the slot are run.
All this means that algorithms that need a centroid as input should simply call getCentroid() on the SourceRecord they're provided, and ensure that their executionOrder is higher than that of centroid algorithms. Similarly, algorithms that want a shape should simply call getShape(). Things are a bit trickier for centroid algorithms, which often need to be given an approximate centroid as an input; these should be prepared to look at the Peaks attached to the SourceRecord's Footprint as an initial value, as the slot centroid may not yet be valid. For wrapped C++ Algorithms (see measBaseAlgorithmConcept), this is handled automatically.
In forced photometry, an external "reference" catalog is used to constrain measurements on an image. While parts of the forced photometry framework could be with a reference catalog from virtually any source, a complete system for loading the reference catalogs that correspond to the region of sky being measured is only available when measurements from a coadd are used as the reference.
While essentially any measurement plugin can be run in forced mode, typically only photometric measurements are scientifically useful (though centroids and shapes may be useful for quality metrics). In fact, in forced mode we typically configure pseudo-measurements to provide the shape and centroid slots, and it is these — rather than anything special about the forced measurement framework — that constrains measurements. In particular, we generally use the ForcedTransformedCentroidPlugin and ForcedTransformedShapePlugin to provide the centroid and shape slots. Rather than measure the centroid and shape on the image, these simply transform the centroid and shape slots from the reference catalog to the appropriate coordinate system. This ensures that measurements that use these slots to obtain positions and ellipses use the same quantities used in generating the reference catalog.
The core of the forced measurement framework is ForcedMeasurementTask and ForcedPlugin, which broadly parallel SingleFrameMeasurementTask and SingleFramePlugin. The high-level algorithm is essentially the same, but with the SourceCatalog to be measured generated by ForcedMeasurementTask.generateSources() from the reference catalog, rather than provided by the user after running detection. The corresponding reference source and the Wcs objects that define the mapping between reference and measurement coordinate systems are also provided to each plugin.
The fact that the sources to be measured are generated from the reference catalog means that the Footprints attached to these sources must be transformed from the reference coordinate system to the measurement coordinate system, and at present that operation turns HeavyFootprints into regular Footprints. HeavyFootprints for child sources are necessary in order to correctly replace neighboring children of the same parent with noise prior to measurement (see NoiseReplacer), and the lack of these means that deblended measurement in forced photometry is essentially broken, except for plugins that implement measureN() and can hence correctly measure all children simultaneously without having to replace them with noise individually.
In addition to the ForcedMeasurementTask subtask and its plugins, the forced measurement framework also contains a pair of command-line driver tasks, ForcedPhotCcdTask and ForcedPhotCoaddTask. These run forced measurement on CCD-level images and coadd patch images, respectively, using the outputs of a previous single-frame measurement run on coadds as the reference catalog in both cases. These delegate the work of loading (and as necessary, filtering and merging) the appropriate reference catalog for the measurement image to a "references" subtask. The interface for the reference subtask is defined by BaseReferencesTask, with the concrete implementation that utilizes coadd processing outputs in CoaddSrcReferencesTask. In general, to use a reference catalog from another source, one should implement a new references subtask, and reuse ForcedPhotCcdTask and/or ForcedPhotCoaddTask. It should only be necessary to replace these and use ForcedMeasurementTask directly if you need to run forced photometry on data that isn't organized by the Butler or doesn't correspond to CCD- or patch-level images.
The "Plugin" interfaces used directly by the measurement tasks are defined completely in Python, and are rooted in the abstract base classes sfm.SingleFramePlugin and forcedImage.ForcedPlugin. There are also analogous C++ base classes, SingleFrameAlgorithm and ForcedAlgorithm, for plugins implemented in C++, as well as SimpleAlgorithm, a C++ base class for simple algorithms in which the same implementation can be used for both single-frame and forced measurement.
For a SingleFramePlugin/SingleFrameAlgorithm:
__init__ method with the same signature as the base class, in which fields saved by the the Plugin should be added to the schema passed to __init__, with keys saved as instance attributes for future use. In C++, implement a constructor with one of the signatures supported by wrappers.wrapSingleFrameAlgorithm.measure() to perform the actual measurement and save the result in the measRecord argument.fail() unless the Plugin cannot fail (except for environment errors).measureN() if the Plugin supports measuring multiple sources simultaneously.SingleFramePlugin.registry.register() at module scope (so the registration happens at import-time). Or, in C++, Swig the algorithm as you would any normal C++ class, and call wrappers.wrapSingleFrameAlgorithm to wrap and register the algorithm simultaneously.For a ForcedPlugin/ForcedAlgorithm
__init__ method with the same signature as the base class, in which fields saved by the the Plugin should be added to the outputSchema of the schemaMapper passed to __init__, with keys saved as instance attributes for future use. In C++, implement a constructor with one of the signatures supported by by wrappers.wrapForcedAlgorithm.measure() (in Python) or measureForced (C++) to perform the actual measurement and save the result in the measRecord argument. Note that the refRecord and refWcs are available during the measurement if needed.fail() unless the Plugin cannot fail (except for environment errors).measureN() (Python) or measureNForced() (C++) if the Plugin supports measuring multiple sources simultaneously.ForcedPlugin.registry.register() at module scope (so the registration happens at import-time). Or, in C++, Swig the algorithm as you would any normal C++ class, and call wrappers.wrapSingleFrameAlgorithm to wrap and register the algorithm simultaneously.In C++, one can also implement both interfaces at the same time using SimpleAlgorithm (see that class for more information).
When a Plugin (or the Algorithm it delegates to) raises any exception, the Task calling it will catch the exception, and call the fail() method of the Plugin, which should cause the plugin to set one or more flags in the output record. If the exception is MeasurementError, the Task will pass this exception back to the fail() method, as MeasurementError contains additional Plugin-specific information indicating the kind of failure. For most other exceptions, the Task will log the exception message as a warning, and pass None as the exception to fail(). In this case, the Plugin should just set the primary failure flag. This is handled automatically by the FlagHandler in Algorithm-based Plugins. Certain exceptions (in particular, out-of-memory errors) are considered fatal and will always be propagated up out of the Task.
Plugin/Algorithm code should endeavor to only throw MeasurementError for known failure modes, unless the problem is in the data and can always be fixed there before the measurement framework is invoked. In other words, we want warnings to appear in the logs only when there's a bug, whether that's in the processing before the measurement framework or in a particular Plugin/Algorithm not knowing and documenting its own failure modes. This means that Plugin/Algorithm implementations should generally have a global try/catch block that re-throwns lower-level exceptions corresponding to known failure modes as MeasurementErrors.
1.8.5