lsst.pipe.tasks.multiBand.MergeDetectionsTask Class Reference

Merge coadd detections from multiple bands. More...

Inheritance diagram for lsst.pipe.tasks.multiBand.MergeDetectionsTask:

Public Member Functions
def	__init__ (self, butler=None, schema=None, kwargs)
	Initialize the merge detections task. More...
def	run (self, catalogs, patchRef)
	Merge multiple catalogs. More...
def	cullPeaks (self, catalog)
	Attempt to remove garbage peaks (mostly on the outskirts of large blends). More...
def	getSchemaCatalogs (self)
	Return a dict of empty catalogs for each catalog dataset produced by this task. More...
def	getSkySourceFootprints (self, mergedList, skyInfo, seed)
	Return a list of Footprints of sky objects which don't overlap with anything in mergedList. More...
def	getInputSchema (self, butler=None, schema=None)
	Obtain the input schema either directly or froma butler reference. More...
def	runDataRef (self, patchRefList)
	Merge coadd sources from multiple bands. More...
def	readCatalog (self, patchRef)
	Read input catalog. More...
def	write (self, patchRef, catalog)
	Write the output. More...
def	writeMetadata (self, dataRefList)
	No metadata to write, and not sure how to write it for a list of dataRefs. More...

Public Attributes
	schema
	merged

Static Public Attributes
	ConfigClass = MergeDetectionsConfig
string	inputDataset = "det"
string	outputDataset = "mergeDet"
	makeIdFactory = _makeMakeIdFactory("MergedCoaddId")
	RunnerClass = MergeSourcesRunner
	getSchemaCatalogs = None

Detailed Description

Merge coadd detections from multiple bands.

Contents

• Description
• Task initialisation
• Invoking the Task
• Configuration parameters
• Debug variables
• A complete example of using MergeDetectionsTask

Description

Command-line task that merges sources detected in coadds of exposures obtained with different filters.

To perform photometry consistently across coadds in multiple filter bands, we create a master catalog of sources from all bands by merging the sources (peaks & footprints) detected in each coadd, while keeping track of which band each source originates in.

The catalog merge is performed by getMergedSourceCatalog. Spurious peaks detected around bright objects are culled as described in CullPeaksConfig_.

Inputs:

deepCoadd_det{tract,patch,filter}: SourceCatalog (only parent Footprints)

Outputs:

deepCoadd_mergeDet{tract,patch}: SourceCatalog (only parent Footprints)

Data Unit:

tract, patch

MergeDetectionsTask subclasses MergeSourcesTask.

Task initialisation

Initialize the merge detections task. A FootprintMergeList will be used to merge the source catalogs.

Additional keyword arguments (forwarded to MergeSourcesTask.__init__):

Parameters

[in]	schema	the schema of the detection catalogs used as input to this one
[in]	butler	a butler used to read the input schema from disk, if schema is None
[in]	**kwargs	keyword arguments to be passed to MergeSourcesTask.__init__

The task will set its own self.schema attribute to the schema of the output merged catalog.

Invoking the Task

Merge multiple catalogs. After ordering the catalogs and filters in priority order, getMergedSourceCatalog of the FootprintMergeList created by __init__ is used to perform the actual merging. Finally, cullPeaks is used to remove garbage peaks detected around bright objects.

Parameters

[in]	catalogs
[in]	patchRef
[out]	mergedList

Configuration parameters

See MergeDetectionsConfig_

Debug variables

The command line task interface supports a flag -d to import debug.py from your PYTHONPATH; see Using lsstDebug to control debugging output for more about debug.py files.

MergeDetectionsTask has no debug variables.

A complete example of using MergeDetectionsTask

MergeDetectionsTask is meant to be run after detecting sources in coadds generated for the chosen subset of the available bands. The purpose of the task is to merge sources (peaks & footprints) detected in the coadds generated from the chosen subset of filters. Subsequent tasks in the multi-band processing procedure will deblend the generated master list of sources and, eventually, perform forced photometry. Command-line usage of MergeDetectionsTask expects data references for all the coadds to be processed. A list of the available optional arguments can be obtained by calling mergeCoaddDetections.py with the --help command line argument:

mergeCoaddDetections.py --help

To demonstrate usage of the DetectCoaddSourcesTask in the larger context of multi-band processing, we will process HSC data in the ci_hsc package. Assuming one has finished step 5 at pipeTasks_multiBand, one may merge the catalogs of sources from each coadd as follows:

mergeCoaddDetections.py $CI_HSC_DIR/DATA --id patch=5,4 tract=0 filter=HSC-I^HSC-R

This will merge the HSC-I & -R band parent source catalogs and write the results to $CI_HSC_DIR/DATA/deepCoadd-results/merged/0/5,4/mergeDet-0-5,4.fits.

The next step in the multi-band processing procedure is MeasureMergedCoaddSourcesTask

Definition at line 691 of file multiBand.py.

Constructor & Destructor Documentation

__init__()

def lsst.pipe.tasks.multiBand.MergeDetectionsTask.__init__	(		self,
			butler = None,
			schema = None,
			kwargs
	)

Initialize the merge detections task.

A FootprintMergeList will be used to merge the source catalogs.

Additional keyword arguments (forwarded to MergeSourcesTask.__init__):

Parameters

[in]	schema	the schema of the detection catalogs used as input to this one
[in]	butler	a butler used to read the input schema from disk, if schema is None
[in]	**kwargs	keyword arguments to be passed to MergeSourcesTask.__init__

The task will set its own self.schema attribute to the schema of the output merged catalog.

Definition at line 778 of file multiBand.py.

    def __init__(self, butler=None, schema=None, **kwargs):
        """!
        @brief Initialize the merge detections task.

        A @ref FootprintMergeList_ "FootprintMergeList" will be used to
        merge the source catalogs.

        Additional keyword arguments (forwarded to MergeSourcesTask.__init__):
        @param[in] schema     the schema of the detection catalogs used as input to this one
        @param[in] butler     a butler used to read the input schema from disk, if schema is None
        @param[in] **kwargs   keyword arguments to be passed to MergeSourcesTask.__init__

        The task will set its own self.schema attribute to the schema of the output merged catalog.
        """
        MergeSourcesTask.__init__(self, butler=butler, schema=schema, **kwargs)
        self.makeSubtask("skyObjects")
        self.schema = self.getInputSchema(butler=butler, schema=schema)

        filterNames = [getShortFilterName(name) for name in self.config.priorityList]
        filterNames += [self.config.skyFilterName]
        self.merged = afwDetect.FootprintMergeList(self.schema, filterNames)

Member Function Documentation

cullPeaks()

def lsst.pipe.tasks.multiBand.MergeDetectionsTask.cullPeaks	(		self,
			catalog
	)

Attempt to remove garbage peaks (mostly on the outskirts of large blends).

Parameters

[in]

catalog

Source catalog

Definition at line 849 of file multiBand.py.

    def cullPeaks(self, catalog):
        """!
        @brief Attempt to remove garbage peaks (mostly on the outskirts of large blends).

        @param[in] catalog Source catalog
        """
        keys = [item.key for item in self.merged.getPeakSchema().extract("merge_peak_*").values()]
        assert len(keys) > 0, "Error finding flags that associate peaks with their detection bands."
        totalPeaks = 0
        culledPeaks = 0
        for parentSource in catalog:
            # Make a list copy so we can clear the attached PeakCatalog and append the ones we're keeping
            # to it (which is easier than deleting as we iterate).
            keptPeaks = parentSource.getFootprint().getPeaks()
            oldPeaks = list(keptPeaks)
            keptPeaks.clear()
            familySize = len(oldPeaks)
            totalPeaks += familySize
            for rank, peak in enumerate(oldPeaks):
                if ((rank < self.config.cullPeaks.rankSufficient) or
                    (sum([peak.get(k) for k in keys]) >= self.config.cullPeaks.nBandsSufficient) or
                    (rank < self.config.cullPeaks.rankConsidered and
                     rank < self.config.cullPeaks.rankNormalizedConsidered * familySize)):
                    keptPeaks.append(peak)
                else:
                    culledPeaks += 1
        self.log.info("Culled %d of %d peaks" % (culledPeaks, totalPeaks))

getInputSchema()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.getInputSchema (   self,   butler = None,   schema = None  )

inherited

Obtain the input schema either directly or froma butler reference.

Parameters

[in]	butler	butler reference to obtain the input schema from
[in]	schema	the input schema

Definition at line 537 of file multiBand.py.

    def getInputSchema(self, butler=None, schema=None):
        """!
        @brief Obtain the input schema either directly or froma  butler reference.

        @param[in]  butler   butler reference to obtain the input schema from
        @param[in]  schema   the input schema
        """
        if schema is None:
            assert butler is not None, "Neither butler nor schema specified"
            schema = butler.get(self.config.coaddName + "Coadd_" + self.inputDataset + "_schema",
                                immediate=True).schema
        return schema

getSchemaCatalogs()

def lsst.pipe.tasks.multiBand.MergeDetectionsTask.getSchemaCatalogs

(

self

)

Return a dict of empty catalogs for each catalog dataset produced by this task.

Parameters

[out]

dictionary

of empty catalogs

Definition at line 877 of file multiBand.py.

    def getSchemaCatalogs(self):
        """!
        Return a dict of empty catalogs for each catalog dataset produced by this task.

        @param[out] dictionary of empty catalogs
        """
        mergeDet = afwTable.SourceCatalog(self.schema)
        peak = afwDetect.PeakCatalog(self.merged.getPeakSchema())
        return {self.config.coaddName + "Coadd_mergeDet": mergeDet,
                self.config.coaddName + "Coadd_peak": peak}

getSkySourceFootprints()

def lsst.pipe.tasks.multiBand.MergeDetectionsTask.getSkySourceFootprints	(		self,
			mergedList,
			skyInfo,
			seed
	)

Return a list of Footprints of sky objects which don't overlap with anything in mergedList.

Parameters

mergedList	The merged Footprints from all the input bands
skyInfo	A description of the patch
seed	Seed for the random number generator

Definition at line 888 of file multiBand.py.

    def getSkySourceFootprints(self, mergedList, skyInfo, seed):
        """!
        @brief Return a list of Footprints of sky objects which don't overlap with anything in mergedList

        @param mergedList  The merged Footprints from all the input bands
        @param skyInfo     A description of the patch
        @param seed        Seed for the random number generator
        """
        mask = afwImage.Mask(skyInfo.patchInfo.getOuterBBox())
        detected = mask.getPlaneBitMask("DETECTED")
        for s in mergedList:
            s.getFootprint().spans.setMask(mask, detected)

        footprints = self.skyObjects.run(mask, seed)
        if not footprints:
            return footprints

        # Need to convert the peak catalog's schema so we can set the "merge_peak_<skyFilterName>" flags
        schema = self.merged.getPeakSchema()
        mergeKey = schema.find("merge_peak_%s" % self.config.skyFilterName).key
        converted = []
        for oldFoot in footprints:
            assert len(oldFoot.getPeaks()) == 1, "Should be a single peak only"
            peak = oldFoot.getPeaks()[0]
            newFoot = afwDetect.Footprint(oldFoot.spans, schema)
            newFoot.addPeak(peak.getFx(), peak.getFy(), peak.getPeakValue())
            newFoot.getPeaks()[0].set(mergeKey, True)
            converted.append(newFoot)

        return converted

readCatalog()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.readCatalog (   self,   patchRef  )

inherited

Read input catalog.

We read the input dataset provided by the 'inputDataset' class variable.

Parameters

[in]

patchRef

data reference for patch

Returns

tuple consisting of the filter name and the catalog

Definition at line 574 of file multiBand.py.

    def readCatalog(self, patchRef):
        """!
        @brief Read input catalog.

        We read the input dataset provided by the 'inputDataset'
        class variable.

        @param[in]  patchRef   data reference for patch
        @return tuple consisting of the filter name and the catalog
        """
        filterName = patchRef.dataId["filter"]
        catalog = patchRef.get(self.config.coaddName + "Coadd_" + self.inputDataset, immediate=True)
        self.log.info("Read %d sources for filter %s: %s" % (len(catalog), filterName, patchRef.dataId))
        return filterName, catalog

run()

def lsst.pipe.tasks.multiBand.MergeDetectionsTask.run	(		self,
			catalogs,
			patchRef
	)

Merge multiple catalogs.

After ordering the catalogs and filters in priority order, getMergedSourceCatalog of the FootprintMergeList created by __init__ is used to perform the actual merging. Finally, cullPeaks is used to remove garbage peaks detected around bright objects.

Parameters

[in]	catalogs
[in]	patchRef
[out]	mergedList

Definition at line 800 of file multiBand.py.

    def run(self, catalogs, patchRef):
        r"""!
        @brief Merge multiple catalogs.

        After ordering the catalogs and filters in priority order,
        @ref getMergedSourceCatalog of the @ref FootprintMergeList_ "FootprintMergeList" created by
        @ref \_\_init\_\_ is used to perform the actual merging. Finally, @ref cullPeaks is used to remove
        garbage peaks detected around bright objects.

        @param[in]  catalogs
        @param[in]  patchRef
        @param[out] mergedList
        """

        # Convert distance to tract coordinate
        skyInfo = getSkyInfo(coaddName=self.config.coaddName, patchRef=patchRef)
        tractWcs = skyInfo.wcs
        peakDistance = self.config.minNewPeak / tractWcs.getPixelScale().asArcseconds()
        samePeakDistance = self.config.maxSamePeak / tractWcs.getPixelScale().asArcseconds()

        # Put catalogs, filters in priority order
        orderedCatalogs = [catalogs[band] for band in self.config.priorityList if band in catalogs.keys()]
        orderedBands = [getShortFilterName(band) for band in self.config.priorityList
                        if band in catalogs.keys()]

        mergedList = self.merged.getMergedSourceCatalog(orderedCatalogs, orderedBands, peakDistance,
                                                        self.schema, self.makeIdFactory(patchRef),
                                                        samePeakDistance)

        #
        # Add extra sources that correspond to blank sky
        #
        skySeed = patchRef.get(self.config.coaddName + "MergedCoaddId")
        skySourceFootprints = self.getSkySourceFootprints(mergedList, skyInfo, skySeed)
        if skySourceFootprints:
            key = mergedList.schema.find("merge_footprint_%s" % self.config.skyFilterName).key
            for foot in skySourceFootprints:
                s = mergedList.addNew()
                s.setFootprint(foot)
                s.set(key, True)

        # Sort Peaks from brightest to faintest
        for record in mergedList:
            record.getFootprint().sortPeaks()
        self.log.info("Merged to %d sources" % len(mergedList))
        # Attempt to remove garbage peaks
        self.cullPeaks(mergedList)
        return mergedList

runDataRef()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.runDataRef (   self,   patchRefList  )

inherited

Merge coadd sources from multiple bands.

Calls run which must be defined in subclasses that inherit from MergeSourcesTask.

Parameters

[in]

patchRefList

list of data references for each filter

Definition at line 563 of file multiBand.py.

    def runDataRef(self, patchRefList):
        """!
        @brief Merge coadd sources from multiple bands. Calls @ref `run` which must be defined in
        subclasses that inherit from MergeSourcesTask.

        @param[in] patchRefList list of data references for each filter
        """
        catalogs = dict(self.readCatalog(patchRef) for patchRef in patchRefList)
        mergedCatalog = self.run(catalogs, patchRefList[0])
        self.write(patchRefList[0], mergedCatalog)

write()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.write (   self,   patchRef,   catalog  )

inherited

Write the output.

Parameters

[in]	patchRef	data reference for patch
[in]	catalog	catalog

We write as the dataset provided by the 'outputDataset' class variable.

Definition at line 600 of file multiBand.py.

    def write(self, patchRef, catalog):
        """!
        @brief Write the output.

        @param[in]  patchRef   data reference for patch
        @param[in]  catalog    catalog

        We write as the dataset provided by the 'outputDataset'
        class variable.
        """
        patchRef.put(catalog, self.config.coaddName + "Coadd_" + self.outputDataset)
        # since the filter isn't actually part of the data ID for the dataset we're saving,
        # it's confusing to see it in the log message, even if the butler simply ignores it.
        mergeDataId = patchRef.dataId.copy()
        del mergeDataId["filter"]
        self.log.info("Wrote merged catalog: %s" % (mergeDataId,))

writeMetadata()

def lsst.pipe.tasks.multiBand.MergeSourcesTask.writeMetadata (   self,   dataRefList  )

inherited

No metadata to write, and not sure how to write it for a list of dataRefs.

Definition at line 617 of file multiBand.py.

    def writeMetadata(self, dataRefList):
        """!
        @brief No metadata to write, and not sure how to write it for a list of dataRefs.
        """
        pass

Member Data Documentation

ConfigClass

lsst.pipe.tasks.multiBand.MergeDetectionsTask.ConfigClass = MergeDetectionsConfig

static

Definition at line 772 of file multiBand.py.

getSchemaCatalogs

lsst.pipe.tasks.multiBand.MergeSourcesTask.getSchemaCatalogs = None

staticinherited

Definition at line 520 of file multiBand.py.

inputDataset

string lsst.pipe.tasks.multiBand.MergeDetectionsTask.inputDataset = "det"

static

Definition at line 774 of file multiBand.py.

makeIdFactory

lsst.pipe.tasks.multiBand.MergeDetectionsTask.makeIdFactory = _makeMakeIdFactory("MergedCoaddId")

static

Definition at line 776 of file multiBand.py.

merged

lsst.pipe.tasks.multiBand.MergeDetectionsTask.merged

Definition at line 798 of file multiBand.py.

outputDataset

string lsst.pipe.tasks.multiBand.MergeDetectionsTask.outputDataset = "mergeDet"

static

Definition at line 775 of file multiBand.py.

RunnerClass

lsst.pipe.tasks.multiBand.MergeSourcesTask.RunnerClass = MergeSourcesRunner

staticinherited

Definition at line 517 of file multiBand.py.

schema

lsst.pipe.tasks.multiBand.MergeDetectionsTask.schema

Definition at line 794 of file multiBand.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/Linux64/pipe_tasks/16.0-27-g78173a71+3/python/lsst/pipe/tasks/multiBand.py