LSSTApplications  17.0+10,17.0+52,17.0+91,18.0.0+11,18.0.0+16,18.0.0+38,18.0.0+4,18.0.0-2-ge43143a+8,18.1.0-1-g0001055+4,18.1.0-1-g1349e88+13,18.1.0-1-g2505f39+10,18.1.0-1-g380d4d4+13,18.1.0-1-g5315e5e,18.1.0-1-g5e4b7ea+4,18.1.0-1-g7e8fceb,18.1.0-1-g85f8cd4+10,18.1.0-1-g9a6769a+4,18.1.0-1-ga1a4c1a+9,18.1.0-1-gd55f500+5,18.1.0-1-ge10677a+10,18.1.0-11-gb2589d7b,18.1.0-13-g451e75588+2,18.1.0-13-gbfe7f7f+4,18.1.0-14-g2e73c10+1,18.1.0-2-g31c43f9+10,18.1.0-2-g919ecaf,18.1.0-2-g9c63283+13,18.1.0-2-gdf0b915+13,18.1.0-2-gfefb8b5+2,18.1.0-3-g52aa583+4,18.1.0-3-g8f4a2b1+4,18.1.0-3-g9cb968e+12,18.1.0-3-gab23065,18.1.0-4-g7bbbad0+4,18.1.0-5-g510c42a+12,18.1.0-5-gaeab27e+13,18.1.0-6-gc4bdb98+2,18.1.0-6-gdda7f3e+15,18.1.0-9-g9613d271+1,w.2019.34
LSSTDataManagementBasePackage
Public Member Functions | Public Attributes | Static Public Attributes | List of all members
lsst.pipe.tasks.mergeDetections.MergeDetectionsTask Class Reference

Merge coadd detections from multiple bands. More...

Inheritance diagram for lsst.pipe.tasks.mergeDetections.MergeDetectionsTask:

Public Member Functions

def getInputSchema (self, butler=None, schema=None)
 
def __init__ (self, butler=None, schema=None, initInputs=None, kwargs)
 Initialize the merge detections task. More...
 
def getInitOutputDatasets (self)
 
def runDataRef (self, patchRefList)
 
def adaptArgsAndRun (self, inputData, inputDataIds, outputDataIds, butler)
 
def run (self, catalogs, skyInfo, idFactory, skySeed)
 Merge multiple catalogs. More...
 
def cullPeaks (self, catalog)
 Attempt to remove garbage peaks (mostly on the outskirts of large blends). More...
 
def getSchemaCatalogs (self)
 Return a dict of empty catalogs for each catalog dataset produced by this task. More...
 
def getSkySourceFootprints (self, mergedList, skyInfo, seed)
 Return a list of Footprints of sky objects which don't overlap with anything in mergedList. More...
 
def write (self, patchRef, catalog)
 Write the output. More...
 
def writeMetadata (self, dataRefList)
 No metadata to write, and not sure how to write it for a list of dataRefs. More...
 

Public Attributes

 schema
 
 merged
 

Static Public Attributes

 ConfigClass = MergeDetectionsConfig
 
 RunnerClass = MergeSourcesRunner
 
string inputDataset = "det"
 
string outputDataset = "mergeDet"
 
 makeIdFactory = _makeMakeIdFactory("MergedCoaddId")
 

Detailed Description

Merge coadd detections from multiple bands.

Contents

Description

Command-line task that merges sources detected in coadds of exposures obtained with different filters.

To perform photometry consistently across coadds in multiple filter bands, we create a master catalog of sources from all bands by merging the sources (peaks & footprints) detected in each coadd, while keeping track of which band each source originates in.

The catalog merge is performed by getMergedSourceCatalog. Spurious peaks detected around bright objects are culled as described in CullPeaksConfig_.

Inputs:
deepCoadd_det{tract,patch,filter}: SourceCatalog (only parent Footprints)
Outputs:
deepCoadd_mergeDet{tract,patch}: SourceCatalog (only parent Footprints)
Data Unit:
tract, patch

Task initialisation

Initialize the merge detections task. A FootprintMergeList will be used to merge the source catalogs.

Parameters
[in]schemathe schema of the detection catalogs used as input to this one
[in]butlera butler used to read the input schema from disk, if schema is None
[in]initInputsThis a PipelineTask-only argument that holds all inputs passed in through the PipelineTask middleware
[in]**kwargskeyword arguments to be passed to CmdLineTask.__init__

The task will set its own self.schema attribute to the schema of the output merged catalog.

Invoking the Task

Merge multiple catalogs. After ordering the catalogs and filters in priority order, getMergedSourceCatalog of the FootprintMergeList created by __init__ is used to perform the actual merging. Finally, cullPeaks is used to remove garbage peaks detected around bright objects.

Parameters
[in]catalogs
[in]patchRef
[out]mergedList

Configuration parameters

See MergeDetectionsConfig_

Debug variables

The command line task interface supports a flag -d to import debug.py from your PYTHONPATH; see Using lsstDebug to control debugging output for more about debug.py files.

MergeDetectionsTask has no debug variables.

A complete example of using MergeDetectionsTask

MergeDetectionsTask is meant to be run after detecting sources in coadds generated for the chosen subset of the available bands. The purpose of the task is to merge sources (peaks & footprints) detected in the coadds generated from the chosen subset of filters. Subsequent tasks in the multi-band processing procedure will deblend the generated master list of sources and, eventually, perform forced photometry. Command-line usage of MergeDetectionsTask expects data references for all the coadds to be processed. A list of the available optional arguments can be obtained by calling mergeCoaddDetections.py with the --help command line argument:

mergeCoaddDetections.py --help

To demonstrate usage of the DetectCoaddSourcesTask in the larger context of multi-band processing, we will process HSC data in the ci_hsc package. Assuming one has finished step 5 at pipeTasks_multiBand, one may merge the catalogs of sources from each coadd as follows:

mergeCoaddDetections.py $CI_HSC_DIR/DATA --id patch=5,4 tract=0 filter=HSC-I^HSC-R

This will merge the HSC-I & -R band parent source catalogs and write the results to $CI_HSC_DIR/DATA/deepCoadd-results/merged/0/5,4/mergeDet-0-5,4.fits.

The next step in the multi-band processing procedure is MeasureMergedCoaddSourcesTask

Definition at line 114 of file mergeDetections.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.__init__ (   self,
  butler = None,
  schema = None,
  initInputs = None,
  kwargs 
)

Initialize the merge detections task.

A FootprintMergeList will be used to merge the source catalogs.

Parameters
[in]schemathe schema of the detection catalogs used as input to this one
[in]butlera butler used to read the input schema from disk, if schema is None
[in]initInputsThis a PipelineTask-only argument that holds all inputs passed in through the PipelineTask middleware
[in]**kwargskeyword arguments to be passed to CmdLineTask.__init__

The task will set its own self.schema attribute to the schema of the output merged catalog.

Definition at line 207 of file mergeDetections.py.

207  def __init__(self, butler=None, schema=None, initInputs=None, **kwargs):
208  # Make PipelineTask-only wording less transitional after cmdlineTask is removed
209  """!
210  @brief Initialize the merge detections task.
211 
212  A @ref FootprintMergeList_ "FootprintMergeList" will be used to
213  merge the source catalogs.
214 
215  @param[in] schema the schema of the detection catalogs used as input to this one
216  @param[in] butler a butler used to read the input schema from disk, if schema is None
217  @param[in] initInputs This a PipelineTask-only argument that holds all inputs passed in
218  through the PipelineTask middleware
219  @param[in] **kwargs keyword arguments to be passed to CmdLineTask.__init__
220 
221  The task will set its own self.schema attribute to the schema of the output merged catalog.
222  """
223  super().__init__(**kwargs)
224  if initInputs is not None:
225  schema = initInputs['schema'].schema
226 
227  self.makeSubtask("skyObjects")
228  self.schema = self.getInputSchema(butler=butler, schema=schema)
229 
230  filterNames = [getShortFilterName(name) for name in self.config.priorityList]
231  filterNames += [self.config.skyFilterName]
232  self.merged = afwDetect.FootprintMergeList(self.schema, filterNames)
233 
def __init__(self, minimum, dataRange, Q)

Member Function Documentation

◆ adaptArgsAndRun()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.adaptArgsAndRun (   self,
  inputData,
  inputDataIds,
  outputDataIds,
  butler 
)

Definition at line 246 of file mergeDetections.py.

246  def adaptArgsAndRun(self, inputData, inputDataIds, outputDataIds, butler):
247  packedId, maxBits = butler.registry.packDataId("tract_patch", outputDataIds['outputCatalog'],
248  returnMaxBits=True)
249  inputData["skySeed"] = packedId
250  inputData["idFactory"] = afwTable.IdFactory.makeSource(packedId, 64 - maxBits)
251  catalogDict = {dataId['abstract_filter']: cat for dataId, cat in zip(inputDataIds['catalogs'],
252  inputData['catalogs'])}
253  inputData['catalogs'] = catalogDict
254  skyMap = inputData.pop('skyMap')
255  # Can use the first dataId to find the tract and patch being worked on
256  tractNumber = inputDataIds['catalogs'][0]['tract']
257  tractInfo = skyMap[tractNumber]
258  patchInfo = tractInfo.getPatchInfo(inputDataIds['catalogs'][0]['patch'])
259  skyInfo = Struct(
260  skyMap=skyMap,
261  tractInfo=tractInfo,
262  patchInfo=patchInfo,
263  wcs=tractInfo.getWcs(),
264  bbox=patchInfo.getOuterBBox()
265  )
266  inputData['skyInfo'] = skyInfo
267  return self.run(**inputData)
268 

◆ cullPeaks()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.cullPeaks (   self,
  catalog 
)

Attempt to remove garbage peaks (mostly on the outskirts of large blends).

Parameters
[in]catalogSource catalog

Definition at line 316 of file mergeDetections.py.

316  def cullPeaks(self, catalog):
317  """!
318  @brief Attempt to remove garbage peaks (mostly on the outskirts of large blends).
319 
320  @param[in] catalog Source catalog
321  """
322  keys = [item.key for item in self.merged.getPeakSchema().extract("merge_peak_*").values()]
323  assert len(keys) > 0, "Error finding flags that associate peaks with their detection bands."
324  totalPeaks = 0
325  culledPeaks = 0
326  for parentSource in catalog:
327  # Make a list copy so we can clear the attached PeakCatalog and append the ones we're keeping
328  # to it (which is easier than deleting as we iterate).
329  keptPeaks = parentSource.getFootprint().getPeaks()
330  oldPeaks = list(keptPeaks)
331  keptPeaks.clear()
332  familySize = len(oldPeaks)
333  totalPeaks += familySize
334  for rank, peak in enumerate(oldPeaks):
335  if ((rank < self.config.cullPeaks.rankSufficient) or
336  (sum([peak.get(k) for k in keys]) >= self.config.cullPeaks.nBandsSufficient) or
337  (rank < self.config.cullPeaks.rankConsidered and
338  rank < self.config.cullPeaks.rankNormalizedConsidered * familySize)):
339  keptPeaks.append(peak)
340  else:
341  culledPeaks += 1
342  self.log.info("Culled %d of %d peaks" % (culledPeaks, totalPeaks))
343 
daf::base::PropertyList * list
Definition: fits.cc:885

◆ getInitOutputDatasets()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.getInitOutputDatasets (   self)

Definition at line 234 of file mergeDetections.py.

234  def getInitOutputDatasets(self):
235  return {"outputSchema": afwTable.SourceCatalog(self.schema),
236  "outputPeakSchema": afwDetect.PeakCatalog(self.merged.getPeakSchema())}
237 

◆ getInputSchema()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.getInputSchema (   self,
  butler = None,
  schema = None 
)

Definition at line 204 of file mergeDetections.py.

204  def getInputSchema(self, butler=None, schema=None):
205  return getInputSchema(self, butler, schema)
206 
def getInputSchema(task, butler=None, schema=None)
Obtain the input schema either directly or froma butler reference.

◆ getSchemaCatalogs()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.getSchemaCatalogs (   self)

Return a dict of empty catalogs for each catalog dataset produced by this task.

Parameters
[out]dictionaryof empty catalogs

Definition at line 344 of file mergeDetections.py.

344  def getSchemaCatalogs(self):
345  """!
346  Return a dict of empty catalogs for each catalog dataset produced by this task.
347 
348  @param[out] dictionary of empty catalogs
349  """
350  mergeDet = afwTable.SourceCatalog(self.schema)
351  peak = afwDetect.PeakCatalog(self.merged.getPeakSchema())
352  return {self.config.coaddName + "Coadd_mergeDet": mergeDet,
353  self.config.coaddName + "Coadd_peak": peak}
354 

◆ getSkySourceFootprints()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.getSkySourceFootprints (   self,
  mergedList,
  skyInfo,
  seed 
)

Return a list of Footprints of sky objects which don't overlap with anything in mergedList.

Parameters
mergedListThe merged Footprints from all the input bands
skyInfoA description of the patch
seedSeed for the random number generator

Definition at line 355 of file mergeDetections.py.

355  def getSkySourceFootprints(self, mergedList, skyInfo, seed):
356  """!
357  @brief Return a list of Footprints of sky objects which don't overlap with anything in mergedList
358 
359  @param mergedList The merged Footprints from all the input bands
360  @param skyInfo A description of the patch
361  @param seed Seed for the random number generator
362  """
363  mask = afwImage.Mask(skyInfo.patchInfo.getOuterBBox())
364  detected = mask.getPlaneBitMask("DETECTED")
365  for s in mergedList:
366  s.getFootprint().spans.setMask(mask, detected)
367 
368  footprints = self.skyObjects.run(mask, seed)
369  if not footprints:
370  return footprints
371 
372  # Need to convert the peak catalog's schema so we can set the "merge_peak_<skyFilterName>" flags
373  schema = self.merged.getPeakSchema()
374  mergeKey = schema.find("merge_peak_%s" % self.config.skyFilterName).key
375  converted = []
376  for oldFoot in footprints:
377  assert len(oldFoot.getPeaks()) == 1, "Should be a single peak only"
378  peak = oldFoot.getPeaks()[0]
379  newFoot = afwDetect.Footprint(oldFoot.spans, schema)
380  newFoot.addPeak(peak.getFx(), peak.getFy(), peak.getPeakValue())
381  newFoot.getPeaks()[0].set(mergeKey, True)
382  converted.append(newFoot)
383 
384  return converted
385 
daf::base::PropertySet * set
Definition: fits.cc:884
Represent a 2-dimensional array of bitmask pixels.
Definition: Mask.h:77
Class to describe the properties of a detected object from an image.
Definition: Footprint.h:62

◆ run()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.run (   self,
  catalogs,
  skyInfo,
  idFactory,
  skySeed 
)

Merge multiple catalogs.

After ordering the catalogs and filters in priority order, getMergedSourceCatalog of the FootprintMergeList created by __init__ is used to perform the actual merging. Finally, cullPeaks is used to remove garbage peaks detected around bright objects.

Parameters
[in]catalogs
[in]patchRef
[out]mergedList

Definition at line 269 of file mergeDetections.py.

269  def run(self, catalogs, skyInfo, idFactory, skySeed):
270  r"""!
271  @brief Merge multiple catalogs.
272 
273  After ordering the catalogs and filters in priority order,
274  @ref getMergedSourceCatalog of the @ref FootprintMergeList_ "FootprintMergeList" created by
275  @ref \_\_init\_\_ is used to perform the actual merging. Finally, @ref cullPeaks is used to remove
276  garbage peaks detected around bright objects.
277 
278  @param[in] catalogs
279  @param[in] patchRef
280  @param[out] mergedList
281  """
282 
283  # Convert distance to tract coordinate
284  tractWcs = skyInfo.wcs
285  peakDistance = self.config.minNewPeak / tractWcs.getPixelScale().asArcseconds()
286  samePeakDistance = self.config.maxSamePeak / tractWcs.getPixelScale().asArcseconds()
287 
288  # Put catalogs, filters in priority order
289  orderedCatalogs = [catalogs[band] for band in self.config.priorityList if band in catalogs.keys()]
290  orderedBands = [getShortFilterName(band) for band in self.config.priorityList
291  if band in catalogs.keys()]
292 
293  mergedList = self.merged.getMergedSourceCatalog(orderedCatalogs, orderedBands, peakDistance,
294  self.schema, idFactory,
295  samePeakDistance)
296 
297  #
298  # Add extra sources that correspond to blank sky
299  #
300  skySourceFootprints = self.getSkySourceFootprints(mergedList, skyInfo, skySeed)
301  if skySourceFootprints:
302  key = mergedList.schema.find("merge_footprint_%s" % self.config.skyFilterName).key
303  for foot in skySourceFootprints:
304  s = mergedList.addNew()
305  s.setFootprint(foot)
306  s.set(key, True)
307 
308  # Sort Peaks from brightest to faintest
309  for record in mergedList:
310  record.getFootprint().sortPeaks()
311  self.log.info("Merged to %d sources" % len(mergedList))
312  # Attempt to remove garbage peaks
313  self.cullPeaks(mergedList)
314  return Struct(outputCatalog=mergedList)
315 

◆ runDataRef()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.runDataRef (   self,
  patchRefList 
)

Definition at line 238 of file mergeDetections.py.

238  def runDataRef(self, patchRefList):
239  catalogs = dict(readCatalog(self, patchRef) for patchRef in patchRefList)
240  skyInfo = getSkyInfo(coaddName=self.config.coaddName, patchRef=patchRefList[0])
241  idFactory = self.makeIdFactory(patchRefList[0])
242  skySeed = patchRefList[0].get(self.config.coaddName + "MergedCoaddId")
243  mergeCatalogStruct = self.run(catalogs, skyInfo, idFactory, skySeed)
244  self.write(patchRefList[0], mergeCatalogStruct.outputCatalog)
245 
def readCatalog(task, patchRef)
Read input catalog.
def getSkyInfo(coaddName, patchRef)
Return the SkyMap, tract and patch information, wcs, and outer bbox of the patch to be coadded...
Definition: coaddBase.py:232

◆ write()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.write (   self,
  patchRef,
  catalog 
)

Write the output.

Parameters
[in]patchRefdata reference for patch
[in]catalogcatalog

We write as the dataset provided by the 'outputDataset' class variable.

Definition at line 386 of file mergeDetections.py.

386  def write(self, patchRef, catalog):
387  """!
388  @brief Write the output.
389 
390  @param[in] patchRef data reference for patch
391  @param[in] catalog catalog
392 
393  We write as the dataset provided by the 'outputDataset'
394  class variable.
395  """
396  patchRef.put(catalog, self.config.coaddName + "Coadd_" + self.outputDataset)
397  # since the filter isn't actually part of the data ID for the dataset we're saving,
398  # it's confusing to see it in the log message, even if the butler simply ignores it.
399  mergeDataId = patchRef.dataId.copy()
400  del mergeDataId["filter"]
401  self.log.info("Wrote merged catalog: %s" % (mergeDataId,))
402 

◆ writeMetadata()

def lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.writeMetadata (   self,
  dataRefList 
)

No metadata to write, and not sure how to write it for a list of dataRefs.

Definition at line 403 of file mergeDetections.py.

403  def writeMetadata(self, dataRefList):
404  """!
405  @brief No metadata to write, and not sure how to write it for a list of dataRefs.
406  """
407  pass
408 

Member Data Documentation

◆ ConfigClass

lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.ConfigClass = MergeDetectionsConfig
static

Definition at line 193 of file mergeDetections.py.

◆ inputDataset

string lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.inputDataset = "det"
static

Definition at line 196 of file mergeDetections.py.

◆ makeIdFactory

lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.makeIdFactory = _makeMakeIdFactory("MergedCoaddId")
static

Definition at line 198 of file mergeDetections.py.

◆ merged

lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.merged

Definition at line 232 of file mergeDetections.py.

◆ outputDataset

string lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.outputDataset = "mergeDet"
static

Definition at line 197 of file mergeDetections.py.

◆ RunnerClass

lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.RunnerClass = MergeSourcesRunner
static

Definition at line 194 of file mergeDetections.py.

◆ schema

lsst.pipe.tasks.mergeDetections.MergeDetectionsTask.schema

Definition at line 228 of file mergeDetections.py.


The documentation for this class was generated from the following file: