Merge measurements from multiple bands. More...

Inheritance diagram for lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask:

Public Member Functions
def	getInputSchema (self, butler=None, schema=None)

def	getInitOutputDatasets (self)

def	adaptArgsAndRun (self, inputData, inputDataIds, outputDataIds, butler)

def	__init__ (self, butler=None, schema=None, initInputs=None, kwargs)
	Initialize the task. More...

def	runDataRef (self, patchRefList)
	Merge coadd sources from multiple bands. More...

def	run (self, catalogs)
	Merge measurement catalogs to create a single reference catalog for forced photometry. More...

def	write (self, patchRef, catalog)
	Write the output. More...

def	writeMetadata (self, dataRefList)
	No metadata to write, and not sure how to write it for a list of dataRefs. More...

Public Attributes
	schemaMapper

	instFluxKey

	instFluxErrKey

	fluxFlagKey

	flagKeys

	schema

	pseudoFilterKeys

	badFlags

Static Public Attributes
	ConfigClass = MergeMeasurementsConfig

	RunnerClass = MergeSourcesRunner

string	inputDataset = "meas"

string	outputDataset = "ref"

	getSchemaCatalogs = _makeGetSchemaCatalogs("ref")

Detailed Description

Merge measurements from multiple bands.

Description

Command-line task that merges measurements from multiple bands.

Combines consistent (i.e. with the same peaks and footprints) catalogs of sources from multiple filter bands to construct a unified catalog that is suitable for driving forced photometry. Every source is required to have centroid, shape and flux measurements in each band.

Inputs:: deepCoadd_meas{tract,patch,filter}: SourceCatalog

Outputs:: deepCoadd_ref{tract,patch}: SourceCatalog

Data Unit:: tract, patch

MergeMeasurementsTask subclasses CmdLineTask.

Task initialization

Initialize the task.

Parameters

[in]	schema	the schema of the detection catalogs used as input to this one
[in]	butler	a butler used to read the input schema from disk, if schema is None

The task will set its own self.schema attribute to the schema of the output merged catalog.

Invoking the Task

Merge measurement catalogs to create a single reference catalog for forced photometry.

Parameters

[in] catalogs the catalogs to be merged

For parent sources, we choose the first band in config.priorityList for which the merge_footprint flag for that band is is True.

For child sources, the logic is the same, except that we use the merge_peak flags.

Configuration parameters

See MergeMeasurementsConfig_

Debug variables

The command line task interface supports a flag -d to import debug.py from your PYTHONPATH; see Using lsstDebug to control debugging output for more about debug.py files.

MergeMeasurementsTask has no debug variables.

A complete example

of using MergeMeasurementsTask

MergeMeasurementsTask is meant to be run after deblending & measuring sources in every band. The purpose of the task is to generate a catalog of sources suitable for driving forced photometry in coadds and individual exposures. Command-line usage of MergeMeasurementsTask expects a data reference to the coadds to be processed. A list of the available optional arguments can be obtained by calling mergeCoaddMeasurements.py with the --help command line argument:

mergeCoaddMeasurements.py --help

To demonstrate usage of the DetectCoaddSourcesTask in the larger context of multi-band processing, we will process HSC data in the ci_hsc package. Assuming one has finished step 7 at pipeTasks_multiBand, one may merge the catalogs generated after deblending and measuring as follows:

mergeCoaddMeasurements.py $CI_HSC_DIR/DATA --id patch=5,4 tract=0 filter=HSC-I^HSC-R

This will merge the HSC-I & HSC-R band catalogs. The results are written in $CI_HSC_DIR/DATA/deepCoadd-results/.

Definition at line 126 of file mergeMeasurements.py.

Constructor & Destructor Documentation

◆ init()

def lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.__init__	(	self,
		butler = `None`,
		schema = `None`,
		initInputs = `None`,
		kwargs
	)

Initialize the task.

Parameters

[in]	schema	the schema of the detection catalogs used as input to this one
[in]	butler	a butler used to read the input schema from disk, if schema is None

The task will set its own self.schema attribute to the schema of the output merged catalog.

Definition at line 225 of file mergeMeasurements.py.

     def __init__(self, butler=None, schema=None, initInputs=None, **kwargs):
         """!
         Initialize the task.
 
         @param[in] schema: the schema of the detection catalogs used as input to this one
         @param[in] butler: a butler used to read the input schema from disk, if schema is None
 
         The task will set its own self.schema attribute to the schema of the output merged catalog.
         """
         super().__init__(**kwargs)
 
         if initInputs is not None:
             inputSchema = initInputs['inputSchema'].schema
         else:
             inputSchema = self.getInputSchema(butler=butler, schema=schema)
         self.schemaMapper = afwTable.SchemaMapper(inputSchema, True)
         self.schemaMapper.addMinimalSchema(inputSchema, True)
         self.instFluxKey = inputSchema.find(self.config.snName + "_instFlux").getKey()
         self.instFluxErrKey = inputSchema.find(self.config.snName + "_instFluxErr").getKey()
         self.fluxFlagKey = inputSchema.find(self.config.snName + "_flag").getKey()
 
         self.flagKeys = {}
         for band in self.config.priorityList:
             short = getShortFilterName(band)
             outputKey = self.schemaMapper.editOutputSchema().addField(
                 "merge_measurement_%s" % short,
                 type="Flag",
                 doc="Flag field set if the measurements here are from the %s filter" % band
             )
             peakKey = inputSchema.find("merge_peak_%s" % short).key
             footprintKey = inputSchema.find("merge_footprint_%s" % short).key
             self.flagKeys[band] = pipeBase.Struct(peak=peakKey, footprint=footprintKey, output=outputKey)
         self.schema = self.schemaMapper.getOutputSchema()
 
         self.pseudoFilterKeys = []
         for filt in self.config.pseudoFilterList:
             try:
                 self.pseudoFilterKeys.append(self.schema.find("merge_peak_%s" % filt).getKey())
             except Exception as e:
                 self.log.warn("merge_peak is not set for pseudo-filter %s: %s" % (filt, e))
 
         self.badFlags = {}
         for flag in self.config.flags:
             try:
                 self.badFlags[flag] = self.schema.find(flag).getKey()
             except KeyError as exc:
                 self.log.warn("Can't find flag %s in schema: %s" % (flag, exc,))
 

Member Function Documentation

◆ adaptArgsAndRun()

def lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.adaptArgsAndRun	(	self,
		inputData,
		inputDataIds,
		outputDataIds,
		butler
	)

Definition at line 218 of file mergeMeasurements.py.

     def adaptArgsAndRun(self, inputData, inputDataIds, outputDataIds, butler):
         catalogDict = {dataId['abstract_filter']: cat for dataId, cat in zip(inputDataIds['catalogs'],
                                                                              inputData['catalogs'])}
         inputData['catalogs'] = catalogDict
 
         return super().adaptArgsAndRun(inputData, inputDataIds, outputDataIds, butler)
 

◆ getInitOutputDatasets()

def lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.getInitOutputDatasets ( self )

Definition at line 215 of file mergeMeasurements.py.

     def getInitOutputDatasets(self):
         return {"outputSchema": afwTable.SourceCatalog(self.schema), }
 

◆ getInputSchema()

def lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.getInputSchema	(	self,
		butler = `None`,
		schema = `None`
	)

Definition at line 212 of file mergeMeasurements.py.

     def getInputSchema(self, butler=None, schema=None):
         return getInputSchema(self, butler, schema)
 

◆ run()

def lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.run	(	self,
		catalogs
	)

Merge measurement catalogs to create a single reference catalog for forced photometry.

Parameters

[in] catalogs the catalogs to be merged

For parent sources, we choose the first band in config.priorityList for which the merge_footprint flag for that band is is True.

For child sources, the logic is the same, except that we use the merge_peak flags.

Definition at line 282 of file mergeMeasurements.py.

     def run(self, catalogs):
         """!
         Merge measurement catalogs to create a single reference catalog for forced photometry
 
         @param[in] catalogs: the catalogs to be merged
 
         For parent sources, we choose the first band in config.priorityList for which the
         merge_footprint flag for that band is is True.
 
         For child sources, the logic is the same, except that we use the merge_peak flags.
         """
         # Put catalogs, filters in priority order
         orderedCatalogs = [catalogs[band] for band in self.config.priorityList if band in catalogs.keys()]
         orderedKeys = [self.flagKeys[band] for band in self.config.priorityList if band in catalogs.keys()]
 
         mergedCatalog = afwTable.SourceCatalog(self.schema)
         mergedCatalog.reserve(len(orderedCatalogs[0]))
 
         idKey = orderedCatalogs[0].table.getIdKey()
         for catalog in orderedCatalogs[1:]:
             if numpy.any(orderedCatalogs[0].get(idKey) != catalog.get(idKey)):
                 raise ValueError("Error in inputs to MergeCoaddMeasurements: source IDs do not match")
 
         # This first zip iterates over all the catalogs simultaneously, yielding a sequence of one
         # record for each band, in priority order.
         for orderedRecords in zip(*orderedCatalogs):
 
             maxSNRecord = None
             maxSNFlagKeys = None
             maxSN = 0.
             priorityRecord = None
             priorityFlagKeys = None
             prioritySN = 0.
             hasPseudoFilter = False
 
             # Now we iterate over those record-band pairs, keeping track of the priority and the
             # largest S/N band.
             for inputRecord, flagKeys in zip(orderedRecords, orderedKeys):
                 parent = (inputRecord.getParent() == 0 and inputRecord.get(flagKeys.footprint))
                 child = (inputRecord.getParent() != 0 and inputRecord.get(flagKeys.peak))
 
                 if not (parent or child):
                     for pseudoFilterKey in self.pseudoFilterKeys:
                         if inputRecord.get(pseudoFilterKey):
                             hasPseudoFilter = True
                             priorityRecord = inputRecord
                             priorityFlagKeys = flagKeys
                             break
                     if hasPseudoFilter:
                         break
 
                 isBad = any(inputRecord.get(flag) for flag in self.badFlags)
                 if isBad or inputRecord.get(self.fluxFlagKey) or inputRecord.get(self.instFluxErrKey) == 0:
                     sn = 0.
                 else:
                     sn = inputRecord.get(self.instFluxKey)/inputRecord.get(self.instFluxErrKey)
                 if numpy.isnan(sn) or sn < 0.:
                     sn = 0.
                 if (parent or child) and priorityRecord is None:
                     priorityRecord = inputRecord
                     priorityFlagKeys = flagKeys
                     prioritySN = sn
                 if sn > maxSN:
                     maxSNRecord = inputRecord
                     maxSNFlagKeys = flagKeys
                     maxSN = sn
 
             # If the priority band has a low S/N we would like to choose the band with the highest S/N as
             # the reference band instead.  However, we only want to choose the highest S/N band if it is
             # significantly better than the priority band.  Therefore, to choose a band other than the
             # priority, we require that the priority S/N is below the minimum threshold and that the
             # difference between the priority and highest S/N is larger than the difference threshold.
             #
             # For pseudo code objects we always choose the first band in the priority list.
             bestRecord = None
             bestFlagKeys = None
             if hasPseudoFilter:
                 bestRecord = priorityRecord
                 bestFlagKeys = priorityFlagKeys
             elif (prioritySN < self.config.minSN and (maxSN - prioritySN) > self.config.minSNDiff and
                   maxSNRecord is not None):
                 bestRecord = maxSNRecord
                 bestFlagKeys = maxSNFlagKeys
             elif priorityRecord is not None:
                 bestRecord = priorityRecord
                 bestFlagKeys = priorityFlagKeys
 
             if bestRecord is not None and bestFlagKeys is not None:
                 outputRecord = mergedCatalog.addNew()
                 outputRecord.assign(bestRecord, self.schemaMapper)
                 outputRecord.set(bestFlagKeys.output, True)
             else:  # if we didn't find any records
                 raise ValueError("Error in inputs to MergeCoaddMeasurements: no valid reference for %s" %
                                  inputRecord.getId())
 
         # more checking for sane inputs, since zip silently iterates over the smallest sequence
         for inputCatalog in orderedCatalogs:
             if len(mergedCatalog) != len(inputCatalog):
                 raise ValueError("Mismatch between catalog sizes: %s != %s" %
                                  (len(mergedCatalog), len(orderedCatalogs)))
 
         return pipeBase.Struct(
             mergedCatalog=mergedCatalog
         )
 

◆ runDataRef()

def lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.runDataRef	(	self,
		patchRefList
	)

Merge coadd sources from multiple bands.

Calls run.

Parameters

[in] patchRefList list of data references for each filter

Definition at line 273 of file mergeMeasurements.py.

     def runDataRef(self, patchRefList):
         """!
         @brief Merge coadd sources from multiple bands. Calls @ref `run`.
         @param[in] patchRefList list of data references for each filter
         """
         catalogs = dict(readCatalog(self, patchRef) for patchRef in patchRefList)
         mergedCatalog = self.run(catalogs).mergedCatalog
         self.write(patchRefList[0], mergedCatalog)
 

◆ write()

def lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.write	(	self,
		patchRef,
		catalog
	)

Write the output.

Parameters

[in]	patchRef	data reference for patch
[in]	catalog	catalog

We write as the dataset provided by the 'outputDataset' class variable.

Definition at line 387 of file mergeMeasurements.py.

     def write(self, patchRef, catalog):
         """!
         @brief Write the output.
 
         @param[in]  patchRef   data reference for patch
         @param[in]  catalog    catalog
 
         We write as the dataset provided by the 'outputDataset'
         class variable.
         """
         patchRef.put(catalog, self.config.coaddName + "Coadd_" + self.outputDataset)
         # since the filter isn't actually part of the data ID for the dataset we're saving,
         # it's confusing to see it in the log message, even if the butler simply ignores it.
         mergeDataId = patchRef.dataId.copy()
         del mergeDataId["filter"]
         self.log.info("Wrote merged catalog: %s" % (mergeDataId,))
 

◆ writeMetadata()

def lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.writeMetadata	(	self,
		dataRefList
	)

No metadata to write, and not sure how to write it for a list of dataRefs.

Definition at line 404 of file mergeMeasurements.py.

     def writeMetadata(self, dataRefList):
         """!
         @brief No metadata to write, and not sure how to write it for a list of dataRefs.
         """
         pass
 

Member Data Documentation

◆ badFlags

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.badFlags

Definition at line 266 of file mergeMeasurements.py.

◆ ConfigClass

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.ConfigClass = MergeMeasurementsConfig

static

Definition at line 202 of file mergeMeasurements.py.

◆ flagKeys

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.flagKeys

Definition at line 246 of file mergeMeasurements.py.

◆ fluxFlagKey

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.fluxFlagKey

Definition at line 244 of file mergeMeasurements.py.

◆ getSchemaCatalogs

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.getSchemaCatalogs = _makeGetSchemaCatalogs("ref")

static

Definition at line 206 of file mergeMeasurements.py.

◆ inputDataset

string lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.inputDataset = "meas"

static

Definition at line 204 of file mergeMeasurements.py.

◆ instFluxErrKey

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.instFluxErrKey

Definition at line 243 of file mergeMeasurements.py.

◆ instFluxKey

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.instFluxKey

Definition at line 242 of file mergeMeasurements.py.

◆ outputDataset

string lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.outputDataset = "ref"

static

Definition at line 205 of file mergeMeasurements.py.

◆ pseudoFilterKeys

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.pseudoFilterKeys

Definition at line 259 of file mergeMeasurements.py.

◆ RunnerClass

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.RunnerClass = MergeSourcesRunner

static

Definition at line 203 of file mergeMeasurements.py.

◆ schema

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.schema

Definition at line 257 of file mergeMeasurements.py.

◆ schemaMapper

lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask.schemaMapper

Definition at line 240 of file mergeMeasurements.py.

The documentation for this class was generated from the following file:

/j/snowflake/release/lsstsw/stack/Linux64/pipe_tasks/18.1.0-11-gb2589d7b/python/lsst/pipe/tasks/mergeMeasurements.py

Public Member Functions

Public Attributes

Static Public Attributes

Detailed Description

Contents

Description

Task initialization

Invoking the Task

Configuration parameters

Debug variables

A complete example

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ adaptArgsAndRun()

◆ getInitOutputDatasets()

◆ getInputSchema()

◆ run()

◆ runDataRef()

◆ write()

◆ writeMetadata()

Member Data Documentation

◆ badFlags

◆ ConfigClass

◆ flagKeys

◆ fluxFlagKey

◆ getSchemaCatalogs

◆ inputDataset

◆ instFluxErrKey

◆ instFluxKey

◆ outputDataset

◆ pseudoFilterKeys

◆ RunnerClass

◆ schema

◆ schemaMapper

◆ init()