lsst.pipe.tasks.multiBand.MergeMeasurementsTask Class Reference

Merge measurements from multiple bands. More...

Inheritance diagram for lsst.pipe.tasks.multiBand.MergeMeasurementsTask:

Public Member Functions
def	__init__ (self, butler=None, schema=None, kwargs)
	Initialize the task. More...
def	run (self, catalogs, patchRef)
	Merge measurement catalogs to create a single reference catalog for forced photometry. More...
def	getInputSchema (self, butler=None, schema=None)
	Obtain the input schema either directly or froma butler reference. More...
def	runDataRef (self, patchRefList)
	Merge coadd sources from multiple bands. More...
def	readCatalog (self, patchRef)
	Read input catalog. More...
def	write (self, patchRef, catalog)
	Write the output. More...
def	writeMetadata (self, dataRefList)
	No metadata to write, and not sure how to write it for a list of dataRefs. More...

Public Attributes
	schemaMapper
	instFluxKey
	instFluxErrKey
	fluxFlagKey
	flagKeys
	schema
	pseudoFilterKeys
	badFlags

Static Public Attributes
	ConfigClass = MergeMeasurementsConfig
string	inputDataset = "meas"
string	outputDataset = "ref"
	getSchemaCatalogs = _makeGetSchemaCatalogs("ref")
	RunnerClass = MergeSourcesRunner

Detailed Description

Merge measurements from multiple bands.

Contents

• Description
• Task initialization
• Invoking the Task
• Configuration parameters
• Debug variables
• A complete example

Description

Command-line task that merges measurements from multiple bands.

Combines consistent (i.e. with the same peaks and footprints) catalogs of sources from multiple filter bands to construct a unified catalog that is suitable for driving forced photometry. Every source is required to have centroid, shape and flux measurements in each band.

Inputs:

deepCoadd_meas{tract,patch,filter}: SourceCatalog

Outputs:

deepCoadd_ref{tract,patch}: SourceCatalog

Data Unit:

tract, patch

MergeMeasurementsTask subclasses MergeSourcesTask.

Task initialization

Initialize the task. Additional keyword arguments (forwarded to MergeSourcesTask.__init__):

Parameters

[in]	schema	the schema of the detection catalogs used as input to this one
[in]	butler	a butler used to read the input schema from disk, if schema is None

The task will set its own self.schema attribute to the schema of the output merged catalog.

Invoking the Task

Merge measurement catalogs to create a single reference catalog for forced photometry.

Parameters

[in]	catalogs	the catalogs to be merged
[in]	patchRef	patch reference for data

For parent sources, we choose the first band in config.priorityList for which the merge_footprint flag for that band is is True.

For child sources, the logic is the same, except that we use the merge_peak flags.

Configuration parameters

See MergeMeasurementsConfig_

Debug variables

The command line task interface supports a flag -d to import debug.py from your PYTHONPATH; see Using lsstDebug to control debugging output for more about debug.py files.

MergeMeasurementsTask has no debug variables.

A complete example

of using MergeMeasurementsTask

MergeMeasurementsTask is meant to be run after deblending & measuring sources in every band. The purpose of the task is to generate a catalog of sources suitable for driving forced photometry in coadds and individual exposures. Command-line usage of MergeMeasurementsTask expects a data reference to the coadds to be processed. A list of the available optional arguments can be obtained by calling mergeCoaddMeasurements.py with the --help command line argument:

mergeCoaddMeasurements.py --help

To demonstrate usage of the DetectCoaddSourcesTask in the larger context of multi-band processing, we will process HSC data in the ci_hsc package. Assuming one has finished step 7 at pipeTasks_multiBand, one may merge the catalogs generated after deblending and measuring as follows:

mergeCoaddMeasurements.py $CI_HSC_DIR/DATA --id patch=5,4 tract=0 filter=HSC-I^HSC-R

This will merge the HSC-I & HSC-R band catalogs. The results are written in $CI_HSC_DIR/DATA/deepCoadd-results/.

Definition at line 1517 of file multiBand.py.

Constructor & Destructor Documentation

__init__()

def lsst.pipe.tasks.multiBand.MergeMeasurementsTask.__init__	(		self,
			butler = None,
			schema = None,
			kwargs
	)

Initialize the task.

Additional keyword arguments (forwarded to MergeSourcesTask.__init__):

Parameters

[in]	schema	the schema of the detection catalogs used as input to this one
[in]	butler	a butler used to read the input schema from disk, if schema is None

The task will set its own self.schema attribute to the schema of the output merged catalog.

Definition at line 1598 of file multiBand.py.

    def __init__(self, butler=None, schema=None, **kwargs):
        """!
        Initialize the task.

        Additional keyword arguments (forwarded to MergeSourcesTask.__init__):
        @param[in] schema: the schema of the detection catalogs used as input to this one
        @param[in] butler: a butler used to read the input schema from disk, if schema is None

        The task will set its own self.schema attribute to the schema of the output merged catalog.
        """
        MergeSourcesTask.__init__(self, butler=butler, schema=schema, **kwargs)
        inputSchema = self.getInputSchema(butler=butler, schema=schema)
        self.schemaMapper = afwTable.SchemaMapper(inputSchema, True)
        self.schemaMapper.addMinimalSchema(inputSchema, True)
        self.instFluxKey = inputSchema.find(self.config.snName + "_instFlux").getKey()
        self.instFluxErrKey = inputSchema.find(self.config.snName + "_instFluxErr").getKey()
        self.fluxFlagKey = inputSchema.find(self.config.snName + "_flag").getKey()

        self.flagKeys = {}
        for band in self.config.priorityList:
            short = getShortFilterName(band)
            outputKey = self.schemaMapper.editOutputSchema().addField(
                "merge_measurement_%s" % short,
                type="Flag",
                doc="Flag field set if the measurements here are from the %s filter" % band
            )
            peakKey = inputSchema.find("merge_peak_%s" % short).key
            footprintKey = inputSchema.find("merge_footprint_%s" % short).key
            self.flagKeys[band] = Struct(peak=peakKey, footprint=footprintKey, output=outputKey)
        self.schema = self.schemaMapper.getOutputSchema()

        self.pseudoFilterKeys = []
        for filt in self.config.pseudoFilterList:
            try:
                self.pseudoFilterKeys.append(self.schema.find("merge_peak_%s" % filt).getKey())
            except Exception as e:
                self.log.warn("merge_peak is not set for pseudo-filter %s: %s" % (filt, e))

        self.badFlags = {}
        for flag in self.config.flags:
            try:
                self.badFlags[flag] = self.schema.find(flag).getKey()
            except KeyError as exc:
                self.log.warn("Can't find flag %s in schema: %s" % (flag, exc,))

Member Function Documentation

getInputSchema()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.getInputSchema (   self,   butler = None,   schema = None  )

inherited

Obtain the input schema either directly or froma butler reference.

Parameters

[in]	butler	butler reference to obtain the input schema from
[in]	schema	the input schema

Definition at line 537 of file multiBand.py.

    def getInputSchema(self, butler=None, schema=None):
        """!
        @brief Obtain the input schema either directly or froma  butler reference.

        @param[in]  butler   butler reference to obtain the input schema from
        @param[in]  schema   the input schema
        """
        if schema is None:
            assert butler is not None, "Neither butler nor schema specified"
            schema = butler.get(self.config.coaddName + "Coadd_" + self.inputDataset + "_schema",
                                immediate=True).schema
        return schema

readCatalog()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.readCatalog (   self,   patchRef  )

inherited

Read input catalog.

We read the input dataset provided by the 'inputDataset' class variable.

Parameters

[in]

patchRef

data reference for patch

Returns

tuple consisting of the filter name and the catalog

Definition at line 574 of file multiBand.py.

    def readCatalog(self, patchRef):
        """!
        @brief Read input catalog.

        We read the input dataset provided by the 'inputDataset'
        class variable.

        @param[in]  patchRef   data reference for patch
        @return tuple consisting of the filter name and the catalog
        """
        filterName = patchRef.dataId["filter"]
        catalog = patchRef.get(self.config.coaddName + "Coadd_" + self.inputDataset, immediate=True)
        self.log.info("Read %d sources for filter %s: %s" % (len(catalog), filterName, patchRef.dataId))
        return filterName, catalog

run()

def lsst.pipe.tasks.multiBand.MergeMeasurementsTask.run	(		self,
			catalogs,
			patchRef
	)

Merge measurement catalogs to create a single reference catalog for forced photometry.

Parameters

[in]	catalogs	the catalogs to be merged
[in]	patchRef	patch reference for data

For parent sources, we choose the first band in config.priorityList for which the merge_footprint flag for that band is is True.

For child sources, the logic is the same, except that we use the merge_peak flags.

Definition at line 1643 of file multiBand.py.

    def run(self, catalogs, patchRef):
        """!
        Merge measurement catalogs to create a single reference catalog for forced photometry

        @param[in] catalogs: the catalogs to be merged
        @param[in] patchRef: patch reference for data

        For parent sources, we choose the first band in config.priorityList for which the
        merge_footprint flag for that band is is True.

        For child sources, the logic is the same, except that we use the merge_peak flags.
        """
        # Put catalogs, filters in priority order
        orderedCatalogs = [catalogs[band] for band in self.config.priorityList if band in catalogs.keys()]
        orderedKeys = [self.flagKeys[band] for band in self.config.priorityList if band in catalogs.keys()]

        mergedCatalog = afwTable.SourceCatalog(self.schema)
        mergedCatalog.reserve(len(orderedCatalogs[0]))

        idKey = orderedCatalogs[0].table.getIdKey()
        for catalog in orderedCatalogs[1:]:
            if numpy.any(orderedCatalogs[0].get(idKey) != catalog.get(idKey)):
                raise ValueError("Error in inputs to MergeCoaddMeasurements: source IDs do not match")

        # This first zip iterates over all the catalogs simultaneously, yielding a sequence of one
        # record for each band, in priority order.
        for orderedRecords in zip(*orderedCatalogs):

            maxSNRecord = None
            maxSNFlagKeys = None
            maxSN = 0.
            priorityRecord = None
            priorityFlagKeys = None
            prioritySN = 0.
            hasPseudoFilter = False

            # Now we iterate over those record-band pairs, keeping track of the priority and the
            # largest S/N band.
            for inputRecord, flagKeys in zip(orderedRecords, orderedKeys):
                parent = (inputRecord.getParent() == 0 and inputRecord.get(flagKeys.footprint))
                child = (inputRecord.getParent() != 0 and inputRecord.get(flagKeys.peak))

                if not (parent or child):
                    for pseudoFilterKey in self.pseudoFilterKeys:
                        if inputRecord.get(pseudoFilterKey):
                            hasPseudoFilter = True
                            priorityRecord = inputRecord
                            priorityFlagKeys = flagKeys
                            break
                    if hasPseudoFilter:
                        break

                isBad = any(inputRecord.get(flag) for flag in self.badFlags)
                if isBad or inputRecord.get(self.fluxFlagKey) or inputRecord.get(self.instFluxErrKey) == 0:
                    sn = 0.
                else:
                    sn = inputRecord.get(self.instFluxKey)/inputRecord.get(self.instFluxErrKey)
                if numpy.isnan(sn) or sn < 0.:
                    sn = 0.
                if (parent or child) and priorityRecord is None:
                    priorityRecord = inputRecord
                    priorityFlagKeys = flagKeys
                    prioritySN = sn
                if sn > maxSN:
                    maxSNRecord = inputRecord
                    maxSNFlagKeys = flagKeys
                    maxSN = sn

            # If the priority band has a low S/N we would like to choose the band with the highest S/N as
            # the reference band instead.  However, we only want to choose the highest S/N band if it is
            # significantly better than the priority band.  Therefore, to choose a band other than the
            # priority, we require that the priority S/N is below the minimum threshold and that the
            # difference between the priority and highest S/N is larger than the difference threshold.
            #
            # For pseudo code objects we always choose the first band in the priority list.
            bestRecord = None
            bestFlagKeys = None
            if hasPseudoFilter:
                bestRecord = priorityRecord
                bestFlagKeys = priorityFlagKeys
            elif (prioritySN < self.config.minSN and (maxSN - prioritySN) > self.config.minSNDiff and
                  maxSNRecord is not None):
                bestRecord = maxSNRecord
                bestFlagKeys = maxSNFlagKeys
            elif priorityRecord is not None:
                bestRecord = priorityRecord
                bestFlagKeys = priorityFlagKeys

            if bestRecord is not None and bestFlagKeys is not None:
                outputRecord = mergedCatalog.addNew()
                outputRecord.assign(bestRecord, self.schemaMapper)
                outputRecord.set(bestFlagKeys.output, True)
            else:  # if we didn't find any records
                raise ValueError("Error in inputs to MergeCoaddMeasurements: no valid reference for %s" %
                                 inputRecord.getId())

        # more checking for sane inputs, since zip silently iterates over the smallest sequence
        for inputCatalog in orderedCatalogs:
            if len(mergedCatalog) != len(inputCatalog):
                raise ValueError("Mismatch between catalog sizes: %s != %s" %
                                 (len(mergedCatalog), len(orderedCatalogs)))

        return mergedCatalog

runDataRef()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.runDataRef (   self,   patchRefList  )

inherited

Merge coadd sources from multiple bands.

Calls run which must be defined in subclasses that inherit from MergeSourcesTask.

Parameters

[in]

patchRefList

list of data references for each filter

Definition at line 563 of file multiBand.py.

    def runDataRef(self, patchRefList):
        """!
        @brief Merge coadd sources from multiple bands. Calls @ref `run` which must be defined in
        subclasses that inherit from MergeSourcesTask.

        @param[in] patchRefList list of data references for each filter
        """
        catalogs = dict(self.readCatalog(patchRef) for patchRef in patchRefList)
        mergedCatalog = self.run(catalogs, patchRefList[0])
        self.write(patchRefList[0], mergedCatalog)

write()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.write (   self,   patchRef,   catalog  )

inherited

Write the output.

Parameters

[in]	patchRef	data reference for patch
[in]	catalog	catalog

We write as the dataset provided by the 'outputDataset' class variable.

Definition at line 600 of file multiBand.py.

    def write(self, patchRef, catalog):
        """!
        @brief Write the output.

        @param[in]  patchRef   data reference for patch
        @param[in]  catalog    catalog

        We write as the dataset provided by the 'outputDataset'
        class variable.
        """
        patchRef.put(catalog, self.config.coaddName + "Coadd_" + self.outputDataset)
        # since the filter isn't actually part of the data ID for the dataset we're saving,
        # it's confusing to see it in the log message, even if the butler simply ignores it.
        mergeDataId = patchRef.dataId.copy()
        del mergeDataId["filter"]
        self.log.info("Wrote merged catalog: %s" % (mergeDataId,))

writeMetadata()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.writeMetadata (   self,   dataRefList  )

inherited

No metadata to write, and not sure how to write it for a list of dataRefs.

Definition at line 617 of file multiBand.py.

    def writeMetadata(self, dataRefList):
        """!
        @brief No metadata to write, and not sure how to write it for a list of dataRefs.
        """
        pass

Member Data Documentation

badFlags

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.badFlags

Definition at line 1636 of file multiBand.py.

ConfigClass

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.ConfigClass = MergeMeasurementsConfig

static

Definition at line 1593 of file multiBand.py.

flagKeys

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.flagKeys

Definition at line 1616 of file multiBand.py.

fluxFlagKey

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.fluxFlagKey

Definition at line 1614 of file multiBand.py.

getSchemaCatalogs

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.getSchemaCatalogs = _makeGetSchemaCatalogs("ref")

static

Definition at line 1596 of file multiBand.py.

inputDataset

string lsst.pipe.tasks.multiBand.MergeMeasurementsTask.inputDataset = "meas"

static

Definition at line 1594 of file multiBand.py.

instFluxErrKey

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.instFluxErrKey

Definition at line 1613 of file multiBand.py.

instFluxKey

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.instFluxKey

Definition at line 1612 of file multiBand.py.

outputDataset

string lsst.pipe.tasks.multiBand.MergeMeasurementsTask.outputDataset = "ref"

static

Definition at line 1595 of file multiBand.py.

pseudoFilterKeys

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.pseudoFilterKeys

Definition at line 1629 of file multiBand.py.

RunnerClass

lsst.pipe.tasks.multiBand.MergeSourcesTask.RunnerClass = MergeSourcesRunner

staticinherited

Definition at line 517 of file multiBand.py.

schema

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.schema

Definition at line 1627 of file multiBand.py.

schemaMapper

lsst.pipe.tasks.multiBand.MergeMeasurementsTask.schemaMapper

Definition at line 1610 of file multiBand.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/Linux64/pipe_tasks/16.0-27-g78173a71+3/python/lsst/pipe/tasks/multiBand.py

---

Generated on Wed Oct 31 2018 09:16:35 for LSSTApplications by   1.8.13