LSSTApplications  10.0-2-g4f67435,11.0.rc2+1,11.0.rc2+12,11.0.rc2+3,11.0.rc2+4,11.0.rc2+5,11.0.rc2+6,11.0.rc2+7,11.0.rc2+8
LSSTDataManagementBasePackage
forcedMeasurement.py
Go to the documentation of this file.
1 #!/usr/bin/env python
2 #
3 # LSST Data Management System
4 # Copyright 2008-2015 LSST Corporation.
5 #
6 # This product includes software developed by the
7 # LSST Project (http://www.lsst.org/).
8 #
9 # This program is free software: you can redistribute it and/or modify
10 # it under the terms of the GNU General Public License as published by
11 # the Free Software Foundation, either version 3 of the License, or
12 # (at your option) any later version.
13 #
14 # This program is distributed in the hope that it will be useful,
15 # but WITHOUT ANY WARRANTY; without even the implied warranty of
16 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
17 # GNU General Public License for more details.
18 #
19 # You should have received a copy of the LSST License Statement and
20 # the GNU General Public License along with this program. If not,
21 # see <http://www.lsstcorp.org/LegalNotices/>.
22 #
23 """Base classes for forced measurement plugins and the driver task for these.
24 
25 In forced measurement, a reference catalog is used to define restricted measurements (usually just fluxes)
26 on an image. As the reference catalog may be deeper than the detection limit of the measurement image, we
27 do not assume that we can use detection and deblend information from the measurement image. Instead, we
28 assume this information is present in the reference catalog and can be "transformed" in some sense to
29 the measurement frame. At the very least, this means that Footprints from the reference catalog should
30 be transformed and installed as Footprints in the output measurement catalog. If we have a procedure that
31 can transform HeavyFootprints, we can then proceed with measurement as usual, but using the reference
32 catalog's id and parent fields to define deblend families. If this transformation does not preserve
33 HeavyFootprints (this is currently the case, at least for CCD forced photometry), then we will only
34 be able to replace objects with noise one deblend family at a time, and hence measurements run in
35 single-object mode may be contaminated by neighbors when run on objects with parent != 0.
36 
37 Measurements are generally recorded in the coordinate system of the image being measured (and all
38 slot-eligible fields must be), but non-slot fields may be recorded in other coordinate systems if necessary
39 to avoid information loss (this should, of course, be indicated in the field documentation). Note that
40 the reference catalog may be in a different coordinate system; it is the responsibility of plugins
41 to transform the data they need themselves, using the reference WCS provided. However, for plugins
42 that only require a position or shape, they may simply use output SourceCatalog's centroid or shape slots,
43 which will generally be set to the transformed position of the reference object before any other plugins are
44 run, and hence avoid using the reference catalog at all.
45 
46 Command-line driver tasks for forced measurement can be found in forcedPhotImage.py, including
47 ForcedPhotImageTask, ForcedPhotCcdTask, and ForcedPhotCoaddTask.
48 """
49 
50 import lsst.pex.config
51 import lsst.pipe.base
52 
53 from .pluginRegistry import PluginRegistry
54 from .baseMeasurement import BasePluginConfig, BasePlugin, BaseMeasurementConfig, BaseMeasurementTask
55 from .noiseReplacer import NoiseReplacer, DummyNoiseReplacer
56 
57 __all__ = ("ForcedPluginConfig", "ForcedPlugin",
58  "ForcedMeasurementConfig", "ForcedMeasurementTask")
59 
60 class ForcedPluginConfig(BasePluginConfig):
61  """Base class for configs of forced measurement plugins."""
62  pass
63 
64 class ForcedPlugin(BasePlugin):
65 
66  # All subclasses of ForcedPlugin should be registered here
67  registry = PluginRegistry(ForcedPluginConfig)
68 
69  ConfigClass = ForcedPluginConfig
70 
71  def __init__(self, config, name, schemaMapper, metadata):
72  """Initialize the measurement object.
73 
74  @param[in] config An instance of this class's ConfigClass.
75  @param[in] name The string the plugin was registered with.
76  @param[in,out] schemaMapper A SchemaMapper that maps reference catalog fields to output
77  catalog fields. Output fields should be added to the
78  output schema. While most plugins will not need to map
79  fields from the reference schema, if they do so, those fields
80  will be transferred before any plugins are run.
81  @param[in] metadata Plugin metadata that will be attached to the output catalog
82  """
83  BasePlugin.__init__(self, config, name)
84 
85  def measure(self, measRecord, exposure, refRecord, refWcs):
86  """Measure the properties of a source on a single image, given data from a
87  reference record.
88 
89  @param[in] exposure lsst.afw.image.ExposureF, containing the pixel data to
90  be measured and the associated Psf, Wcs, etc. All
91  other sources in the image will have been replaced by
92  noise according to deblender outputs.
93  @param[in,out] measRecord lsst.afw.table.SourceRecord to be filled with outputs,
94  and from which previously-measured quantities can be
95  retreived.
96  @param[in] refRecord lsst.afw.table.SimpleRecord that contains additional
97  parameters to define the fit, as measured elsewhere.
98  @param[in] refWcs The coordinate system for the reference catalog values.
99  An afw.geom.Angle may be passed, indicating that a
100  local tangent Wcs should be created for each object
101  using afw.image.makeLocalWcs and the given angle as
102  a pixel scale.
103 
104  In the normal mode of operation, the source centroid will be set to the
105  WCS-transformed position of the reference object, so plugins that only
106  require a reference position should not have to access the reference object
107  at all.
108  """
109  raise NotImplementedError()
110 
111  def measureN(self, measCat, exposure, refCat, refWcs):
112  """Measure the properties of a group of blended sources on a single image,
113  given data from a reference record.
114 
115  @param[in] exposure lsst.afw.image.ExposureF, containing the pixel data to
116  be measured and the associated Psf, Wcs, etc. Sources
117  not in the blended hierarchy to be measured will have
118  been replaced with noise using deblender outputs.
119  @param[in,out] measCat lsst.afw.table.SourceCatalog to be filled with outputs,
120  and from which previously-measured quantities can be
121  retrieved, containing only the sources that should be
122  measured together in this call.
123  @param[in] refCat lsst.afw.table.SimpleCatalog that contains additional
124  parameters to define the fit, as measured elsewhere.
125  Ordered such that zip(sources, references) may be used.
126  @param[in] refWcs The coordinate system for the reference catalog values.
127  An afw.geom.Angle may be passed, indicating that a
128  local tangent Wcs should be created for each object
129  using afw.image.makeLocalWcs and the given Angle as
130  a pixel scale.
131 
132  In the normal mode of operation, the source centroids will be set to the
133  WCS-transformed position of the reference object, so plugins that only
134  require a reference position should not have to access the reference object
135  at all.
136  """
137  raise NotImplementedError()
138 
139 class ForcedMeasurementConfig(BaseMeasurementConfig):
140  """Config class for forced measurement driver task."""
141 
142  plugins = ForcedPlugin.registry.makeField(
143  multi=True,
144  default=["base_TransformedCentroid",
145  "base_TransformedShape",
146  "base_GaussianFlux",
147  "base_CircularApertureFlux",
148  "base_PsfFlux",
149  ],
150  doc="Plugins to be run and their configuration"
151  )
152  algorithms = property(lambda self: self.plugins, doc="backwards-compatibility alias for plugins")
153 
154  copyColumns = lsst.pex.config.DictField(
155  keytype=str, itemtype=str, doc="Mapping of reference columns to source columns",
156  default={"id": "objectId", "parent":"parentObjectId"}
157  )
158 
159  def setDefaults(self):
160  self.slots.centroid = "base_TransformedCentroid"
161  self.slots.shape = "base_TransformedShape"
162  self.slots.apFlux = None
163  self.slots.modelFlux = None
164  self.slots.psfFlux = None
165  self.slots.instFlux = None
166  self.slots.calibFlux = None
167 
168 ## \addtogroup LSST_task_documentation
169 ## \{
170 ## \page ForcedMeasurementTask
171 ## \ref ForcedMeasurementTask_ "ForcedMeasurementTask"
172 ## \copybrief ForcedMeasurementTask
173 ## \}
174 
175 class ForcedMeasurementTask(BaseMeasurementTask):
176  """!
177  \anchor ForcedMeasurementTask_
178 
179  \brief A subtask for measuring the properties of sources on a single
180  exposure, using an existing "reference" catalog to constrain some aspects
181  of the measurement.
182 
183  The task is configured with a list of "plugins": each plugin defines the values it
184  measures (i.e. the columns in a table it will fill) and conducts that measurement
185  on each detected source (see ForcedPlugin). The job of the
186  measurement task is to initialize the set of plugins (which includes setting up the
187  catalog schema) from their configuration, and then invoke each plugin on each
188  source.
189 
190  Most of the time, ForcedMeasurementTask will be used via one of the subclasses of
191  ForcedPhotImageTask, ForcedPhotCcdTask and ForcedPhotCoaddTask. These combine
192  this measurement subtask with a "references" subtask (see BaseReferencesTask and
193  CoaddSrcReferencesTask) to perform forced measurement using measurements performed on
194  another image as the references. There is generally little reason to use
195  ForcedMeasurementTask outside of one of these drivers, unless it is necessary to avoid
196  using the Butler for I/O.
197 
198  ForcedMeasurementTask has only three methods: __init__(), run(), and generateMeasCat().
199  For configuration options, see SingleFrameMeasurementConfig.
200  """
201 
202  ConfigClass = ForcedMeasurementConfig
203 
204  def __init__(self, refSchema, algMetadata=None, **kwds):
205  """!
206  Initialize the task. Set up the execution order of the plugins and initialize
207  the plugins, giving each plugin an opportunity to add its measurement fields to
208  the output schema and to record information in the task metadata.
209 
210  Note that while SingleFrameMeasurementTask is passed an initial Schema that is
211  appended to in order to create the output Schema, ForcedMeasurementTask is
212  initialized with the Schema of the reference catalog, from which a new Schema
213  for the output catalog is created. Fields to be copied directly from the
214  reference Schema are added before Plugin fields are added.
215 
216  @param[in] refSchema Schema of the reference catalog. Must match the catalog
217  later passed to generateMeasCat() and/or run().
218  @param[in,out] algMetadata lsst.daf.base.PropertyList used to record information about
219  each algorithm. An empty PropertyList will be created if None.
220  @param[in] **kwds Keyword arguments passed from lsst.pipe.base.Task.__init__
221  """
222  super(ForcedMeasurementTask, self).__init__(algMetadata=algMetadata, **kwds)
224  self.mapper.addMinimalSchema(lsst.afw.table.SourceTable.makeMinimalSchema())
225  self.config.slots.setupSchema(self.mapper.editOutputSchema())
226  for refName, targetName in self.config.copyColumns.items():
227  refItem = refSchema.find(refName)
228  self.mapper.addMapping(refItem.key, targetName)
229  self.config.slots.setupSchema(self.mapper.editOutputSchema())
230  self.initializePlugins(schemaMapper=self.mapper)
231  self.schema = self.mapper.getOutputSchema()
232  self.makeSubtask("applyApCorr", schema=self.schema)
233 
234  def run(self, measCat, exposure, refCat, refWcs, exposureId=None, beginOrder=None, endOrder=None,
235  allowApCorr=True):
236  """!
237  Perform forced measurement.
238 
239  @param[in] exposure lsst.afw.image.ExposureF to be measured; must have at least a Wcs attached.
240  @param[in] measCat Source catalog for measurement results; must be initialized with empty
241  records already corresponding to those in refCat (via e.g. generateMeasCat).
242  @param[in] refCat A sequence of SourceRecord objects that provide reference information
243  for the measurement. These will be passed to each Plugin in addition
244  to the output SourceRecord.
245  @param[in] refWcs Wcs that defines the X,Y coordinate system of refCat
246  @param[in] exposureId optional unique exposureId used to calculate random number
247  generator seed in the NoiseReplacer.
248  @param[in] beginOrder beginning execution order (inclusive): measurements with
249  executionOrder < beginOrder are not executed. None for no limit.
250  @param[in] endOrder ending execution order (exclusive): measurements with
251  executionOrder >= endOrder are not executed. None for no limit.
252  @param[in] allowApCorr allow application of aperture correction?
253 
254  Fills the initial empty SourceCatalog with forced measurement results. Two steps must occur
255  before run() can be called:
256  - generateMeasCat() must be called to create the output measCat argument.
257  - Footprints appropriate for the forced sources must be attached to the measCat records. The
258  attachTransformedFootprints() method can be used to do this, but this degrades HeavyFootprints
259  to regular Footprints, leading to non-deblended measurement, so most callers should provide
260  Footprints some other way. Typically, calling code will have access to information that will
261  allow them to provide HeavyFootprints - for instance, ForcedPhotCoaddTask uses the HeavyFootprints
262  from deblending run in the same band just before non-forced is run measurement in that band.
263  """
264  # First check that the reference catalog does not contain any children for which
265  # any member of their parent chain is not within the list. This can occur at
266  # boundaries when the parent is outside and one of the children is within.
267  # Currently, the parent chain is always only one deep, but just in case, this
268  # code checks for any case where when the parent chain to a child's topmost
269  # parent is broken and raises an exception if it occurs.
270  #
271  # I.e. this code checks that this precondition is satisfied by whatever reference
272  # catalog provider is being paired with it.
273  refCatIdDict = {ref.getId(): ref.getParent() for ref in refCat}
274  for ref in refCat:
275  refId = ref.getId()
276  topId = refId
277  while(topId > 0):
278  if not topId in refCatIdDict.keys():
279  raise RuntimeError("Reference catalog contains a child for which at least "
280  "one parent in its parent chain is not in the catalog.")
281  topId = refCatIdDict[topId]
282 
283  # Construct a footprints dict which looks like
284  # {ref.getId(): (ref.getParent(), source.getFootprint())}
285  # (i.e. getting the footprint from the transformed source footprint)
286  footprints = {ref.getId(): (ref.getParent(), measRecord.getFootprint())
287  for (ref, measRecord) in zip(refCat, measCat)}
288 
289  self.log.info("Performing forced measurement on %d sources" % len(refCat))
290 
291  if self.config.doReplaceWithNoise:
292  noiseReplacer = NoiseReplacer(self.config.noiseReplacer, exposure, footprints, log=self.log, exposureId=exposureId)
293  algMetadata = measCat.getTable().getMetadata()
294  if not algMetadata is None:
295  algMetadata.addInt("NOISE_SEED_MULTIPLIER", self.config.noiseReplacer.noiseSeedMultiplier)
296  algMetadata.addString("NOISE_SOURCE", self.config.noiseReplacer.noiseSource)
297  algMetadata.addDouble("NOISE_OFFSET", self.config.noiseReplacer.noiseOffset)
298  if not exposureId is None:
299  algMetadata.addLong("NOISE_EXPOSURE_ID", exposureId)
300  else:
301  noiseReplacer = DummyNoiseReplacer()
302 
303  # Create parent cat which slices both the refCat and measCat (sources)
304  # first, get the reference and source records which have no parent
305  refParentCat, measParentCat = refCat.getChildren(0, measCat)
306  for parentIdx, (refParentRecord, measParentRecord) in enumerate(zip(refParentCat,measParentCat)):
307 
308  # first process the records which have the current parent as children
309  refChildCat, measChildCat = refCat.getChildren(refParentRecord.getId(), measCat)
310  # TODO: skip this loop if there are no plugins configured for single-object mode
311  for refChildRecord, measChildRecord in zip(refChildCat, measChildCat):
312  noiseReplacer.insertSource(refChildRecord.getId())
313  self.callMeasure(measChildRecord, exposure, refChildRecord, refWcs,
314  beginOrder=beginOrder, endOrder=endOrder)
315  noiseReplacer.removeSource(refChildRecord.getId())
316 
317  # then process the parent record
318  noiseReplacer.insertSource(refParentRecord.getId())
319  self.callMeasure(measParentRecord, exposure, refParentRecord, refWcs,
320  beginOrder=beginOrder, endOrder=endOrder)
321  self.callMeasureN(measParentCat[parentIdx:parentIdx+1], exposure,
322  refParentCat[parentIdx:parentIdx+1],
323  beginOrder=beginOrder, endOrder=endOrder)
324  # measure all the children simultaneously
325  self.callMeasureN(measChildCat, exposure, refChildCat,
326  beginOrder=beginOrder, endOrder=endOrder)
327  noiseReplacer.removeSource(refParentRecord.getId())
328  noiseReplacer.end()
329 
330  if allowApCorr:
331  self._applyApCorrIfWanted(
332  sources = measCat,
333  apCorrMap = exposure.getInfo().getApCorrMap(),
334  endOrder = endOrder,
335  )
336 
337  def generateMeasCat(self, exposure, refCat, refWcs, idFactory=None):
338  """!Initialize an output SourceCatalog using information from the reference catalog.
339 
340  This generates a new blank SourceRecord for each record in refCat. Note that this
341  method does not attach any Footprints. Doing so is up to the caller (who may
342  call attachedTransformedFootprints or define their own method - see run() for more
343  information).
344 
345  @param[in] exposure Exposure to be measured
346  @param[in] refCat Sequence (not necessarily a SourceCatalog) of reference SourceRecords.
347  @param[in] refWcs Wcs that defines the X,Y coordinate system of refCat
348  @param[in] idFactory factory for creating IDs for sources
349 
350  @return Source catalog ready for measurement
351  """
352  if idFactory == None:
354  table = lsst.afw.table.SourceTable.make(self.schema, idFactory)
355  measCat = lsst.afw.table.SourceCatalog(table)
356  table = measCat.table
357  table.setMetadata(self.algMetadata)
358  table.preallocate(len(refCat))
359  for ref in refCat:
360  newSource = measCat.addNew()
361  newSource.assign(ref, self.mapper)
362  return measCat
363 
364  def attachTransformedFootprints(self, sources, refCat, exposure, refWcs):
365  """!Default implementation for attaching Footprints to blank sources prior to measurement
366 
367  Footprints for forced photometry must be in the pixel coordinate system of the image being
368  measured, while the actual detections may start out in a different coordinate system.
369  This default implementation transforms the Footprints from the reference catalog from the
370  refWcs to the exposure's Wcs, which downgrades HeavyFootprints into regular Footprints,
371  destroying deblend information.
372 
373  Note that ForcedPhotImageTask delegates to this method in its own attachFootprints method.
374  attachFootprints can then be overridden by its subclasses to define how their Footprints
375  should be generated.
376 
377  See the documentation for run() for information about the relationships between run(),
378  generateMeasCat(), and attachTransformedFootprints().
379  """
380  exposureWcs = exposure.getWcs()
381  region = exposure.getBBox(lsst.afw.image.PARENT)
382  for srcRecord, refRecord in zip(sources, refCat):
383  srcRecord.setFootprint(refRecord.getFootprint().transform(refWcs, exposureWcs, region))
A mapping between the keys of two Schemas, used to copy data between them.
Definition: SchemaMapper.h:19
A subtask for measuring the properties of sources on a single exposure, using an existing &quot;reference&quot;...
static boost::shared_ptr< SourceTable > make(Schema const &schema, boost::shared_ptr< IdFactory > const &idFactory)
Construct a new table.
static boost::shared_ptr< IdFactory > makeSimple()
Return a simple IdFactory that simply counts from 1.
static Schema makeMinimalSchema()
Return a minimal schema for Source tables and records.
Definition: Source.h:242
Custom catalog class for record/table subclasses that are guaranteed to have an ID, and should generally be sorted by that ID.
Definition: fwd.h:55
def attachTransformedFootprints
Default implementation for attaching Footprints to blank sources prior to measurement.
def generateMeasCat
Initialize an output SourceCatalog using information from the reference catalog.