lsst.pipe.tasks.matchBackgrounds.MatchBackgroundsTask Class Reference

Inheritance diagram for lsst.pipe.tasks.matchBackgrounds.MatchBackgroundsTask:

Public Member Functions
def	__init__ (self, *args, **kwargs)
def	run (self, expRefList, expDatasetType, imageScalerList=None, refExpDataRef=None, refImageScaler=None)
def	selectRefExposure (self, expRefList, imageScalerList, expDatasetType)
def	matchBackgrounds (self, refExposure, sciExposure)

Public Attributes
	sctrl
	debugDataIdString

Static Public Attributes
	ConfigClass = MatchBackgroundsConfig

Detailed Description

Definition at line 139 of file matchBackgrounds.py.

Constructor & Destructor Documentation

__init__()

def lsst.pipe.tasks.matchBackgrounds.MatchBackgroundsTask.__init__	(		self,
		*	args,
		**	kwargs
	)

Definition at line 143 of file matchBackgrounds.py.

    def __init__(self, *args, **kwargs):
        pipeBase.Task.__init__(self, *args, **kwargs)
 
        self.sctrl = afwMath.StatisticsControl()
        self.sctrl.setAndMask(afwImage.Mask.getPlaneBitMask(self.config.badMaskPlanes))
        self.sctrl.setNanSafe(True)
 

Member Function Documentation

matchBackgrounds()

def lsst.pipe.tasks.matchBackgrounds.MatchBackgroundsTask.matchBackgrounds	(		self,
			refExposure,
			sciExposure
	)

Match science exposure's background level to that of reference exposure.

Process creates a difference image of the reference exposure minus the science exposure, and then
generates an afw.math.Background object. It assumes (but does not require/check) that the mask plane
already has detections set. If detections have not been set/masked, sources will bias the
background estimation.
The 'background' of the difference image is smoothed by spline interpolation (by the Background class)
or by polynomial interpolation by the Approximate class. This model of difference image
is added to the science exposure in memory.
Fit diagnostics are also calculated and returned.

@param[in] refExposure: reference exposure
@param[in,out] sciExposure: science exposure; modified by changing the background level
    to match that of the reference exposure
@returns a pipBase.Struct with fields:
    - backgroundModel: an afw.math.Approximate or an afw.math.Background.
    - fitRMS: rms of the fit. This is the sqrt(mean(residuals**2)).
    - matchedMSE: the MSE of the reference and matched images: mean((refImage - matchedSciImage)**2);
      should be comparable to difference image's mean variance.
    - diffImVar: the mean variance of the difference image.

Definition at line 332 of file matchBackgrounds.py.

    def matchBackgrounds(self, refExposure, sciExposure):
        """
        Match science exposure's background level to that of reference exposure.
 
        Process creates a difference image of the reference exposure minus the science exposure, and then
        generates an afw.math.Background object. It assumes (but does not require/check) that the mask plane
        already has detections set. If detections have not been set/masked, sources will bias the
        background estimation.
        The 'background' of the difference image is smoothed by spline interpolation (by the Background class)
        or by polynomial interpolation by the Approximate class. This model of difference image
        is added to the science exposure in memory.
        Fit diagnostics are also calculated and returned.
 
        @param[in] refExposure: reference exposure
        @param[in,out] sciExposure: science exposure; modified by changing the background level
            to match that of the reference exposure
        @returns a pipBase.Struct with fields:
            - backgroundModel: an afw.math.Approximate or an afw.math.Background.
            - fitRMS: rms of the fit. This is the sqrt(mean(residuals**2)).
            - matchedMSE: the MSE of the reference and matched images: mean((refImage - matchedSciImage)**2);
              should be comparable to difference image's mean variance.
            - diffImVar: the mean variance of the difference image.
        """
 
        if lsstDebug.Info(__name__).savefits:
            refExposure.writeFits(lsstDebug.Info(__name__).figpath + 'refExposure.fits')
            sciExposure.writeFits(lsstDebug.Info(__name__).figpath + 'sciExposure.fits')
 
        # Check Configs for polynomials:
        if self.config.usePolynomial:
            x, y = sciExposure.getDimensions()
            shortSideLength = min(x, y)
            if shortSideLength < self.config.binSize:
                raise ValueError("%d = config.binSize > shorter dimension = %d" % (self.config.binSize,
                                                                                   shortSideLength))
            npoints = shortSideLength // self.config.binSize
            if shortSideLength % self.config.binSize != 0:
                npoints += 1
 
            if self.config.order > npoints - 1:
                raise ValueError("%d = config.order > npoints - 1 = %d" % (self.config.order, npoints - 1))
 
        # Check that exposures are same shape
        if (sciExposure.getDimensions() != refExposure.getDimensions()):
            wSci, hSci = sciExposure.getDimensions()
            wRef, hRef = refExposure.getDimensions()
            raise RuntimeError(
                "Exposures are different dimensions. sci:(%i, %i) vs. ref:(%i, %i)" %
                (wSci, hSci, wRef, hRef))
 
        statsFlag = getattr(afwMath, self.config.gridStatistic)
        self.sctrl.setNumSigmaClip(self.config.numSigmaClip)
        self.sctrl.setNumIter(self.config.numIter)
 
        im = refExposure.getMaskedImage()
        diffMI = im.Factory(im, True)
        diffMI -= sciExposure.getMaskedImage()
 
        width = diffMI.getWidth()
        height = diffMI.getHeight()
        nx = width // self.config.binSize
        if width % self.config.binSize != 0:
            nx += 1
        ny = height // self.config.binSize
        if height % self.config.binSize != 0:
            ny += 1
 
        bctrl = afwMath.BackgroundControl(nx, ny, self.sctrl, statsFlag)
        bctrl.setUndersampleStyle(self.config.undersampleStyle)
 
        bkgd = afwMath.makeBackground(diffMI, bctrl)
 
        # Some config and input checks if config.usePolynomial:
        # 1) Check that order/bin size make sense:
        # 2) Change binsize or order if underconstrained.
        if self.config.usePolynomial:
            order = self.config.order
            bgX, bgY, bgZ, bgdZ = self._gridImage(diffMI, self.config.binSize, statsFlag)
            minNumberGridPoints = min(len(set(bgX)), len(set(bgY)))
            if len(bgZ) == 0:
                raise ValueError("No overlap with reference. Nothing to match")
            elif minNumberGridPoints <= self.config.order:
                # must either lower order or raise number of bins or throw exception
                if self.config.undersampleStyle == "THROW_EXCEPTION":
                    raise ValueError("Image does not cover enough of ref image for order and binsize")
                elif self.config.undersampleStyle == "REDUCE_INTERP_ORDER":
                    self.log.warning("Reducing order to %d", (minNumberGridPoints - 1))
                    order = minNumberGridPoints - 1
                elif self.config.undersampleStyle == "INCREASE_NXNYSAMPLE":
                    newBinSize = (minNumberGridPoints*self.config.binSize) // (self.config.order + 1)
                    bctrl.setNxSample(newBinSize)
                    bctrl.setNySample(newBinSize)
                    bkgd = afwMath.makeBackground(diffMI, bctrl)  # do over
                    self.log.warning("Decreasing binsize to %d", newBinSize)
 
            # If there is no variance in any image pixels, do not weight bins by inverse variance
            isUniformImageDiff = not numpy.any(bgdZ > self.config.gridStdevEpsilon)
            weightByInverseVariance = False if isUniformImageDiff else self.config.approxWeighting
 
        # Add offset to sciExposure
        try:
            if self.config.usePolynomial:
                actrl = afwMath.ApproximateControl(afwMath.ApproximateControl.CHEBYSHEV,
                                                   order, order, weightByInverseVariance)
                undersampleStyle = getattr(afwMath, self.config.undersampleStyle)
                approx = bkgd.getApproximate(actrl, undersampleStyle)
                bkgdImage = approx.getImage()
            else:
                bkgdImage = bkgd.getImageF(self.config.interpStyle, self.config.undersampleStyle)
        except Exception as e:
            raise RuntimeError("Background/Approximation failed to interp image %s: %s" % (
                self.debugDataIdString, e))
 
        sciMI = sciExposure.getMaskedImage()
        sciMI += bkgdImage
        del sciMI
 
        # Need RMS from fit: 2895 will replace this:
        rms = 0.0
        X, Y, Z, dZ = self._gridImage(diffMI, self.config.binSize, statsFlag)
        x0, y0 = diffMI.getXY0()
        modelValueArr = numpy.empty(len(Z))
        for i in range(len(X)):
            modelValueArr[i] = bkgdImage[int(X[i]-x0), int(Y[i]-y0), afwImage.LOCAL]
        resids = Z - modelValueArr
        rms = numpy.sqrt(numpy.mean(resids[~numpy.isnan(resids)]**2))
 
        if lsstDebug.Info(__name__).savefits:
            sciExposure.writeFits(lsstDebug.Info(__name__).figpath + 'sciMatchedExposure.fits')
 
        if lsstDebug.Info(__name__).savefig:
            bbox = geom.Box2D(refExposure.getMaskedImage().getBBox())
            try:
                self._debugPlot(X, Y, Z, dZ, bkgdImage, bbox, modelValueArr, resids)
            except Exception as e:
                self.log.warning('Debug plot not generated: %s', e)
 
        meanVar = afwMath.makeStatistics(diffMI.getVariance(), diffMI.getMask(),
                                         afwMath.MEANCLIP, self.sctrl).getValue()
 
        diffIm = diffMI.getImage()
        diffIm -= bkgdImage  # diffMI should now have a mean ~ 0
        del diffIm
        mse = afwMath.makeStatistics(diffMI, afwMath.MEANSQUARE, self.sctrl).getValue()
 
        outBkgd = approx if self.config.usePolynomial else bkgd
 
        return pipeBase.Struct(
            backgroundModel=outBkgd,
            fitRMS=rms,
            matchedMSE=mse,
            diffImVar=meanVar)
 

run()

def lsst.pipe.tasks.matchBackgrounds.MatchBackgroundsTask.run	(		self,
			expRefList,
			expDatasetType,
			imageScalerList = None,
			refExpDataRef = None,
			refImageScaler = None
	)

Match the backgrounds of a list of coadd temp exposures to a reference coadd temp exposure.

Choose a refExpDataRef automatically if none supplied.

@param[in] expRefList: list of data references to science exposures to be background-matched;
    all exposures must exist.
@param[in] expDatasetType: dataset type of exposures, e.g. 'goodSeeingCoadd_tempExp'
@param[in] imageScalerList: list of image scalers (coaddUtils.ImageScaler);
    if None then the images are not scaled
@param[in] refExpDataRef: data reference for the reference exposure.
    If None, then this task selects the best exposures from expRefList.
    if not None then must be one of the exposures in expRefList.
@param[in] refImageScaler: image scaler for reference image;
    ignored if refExpDataRef is None, else scaling is not performed if None

@return: a pipBase.Struct containing these fields:
- backgroundInfoList: a list of pipeBase.Struct, one per exposure in expRefList,
    each of which contains these fields:
    - isReference: this is the reference exposure (only one returned Struct will
        contain True for this value, unless the ref exposure is listed multiple times)
    - backgroundModel: differential background model (afw.Math.Background or afw.Math.Approximate).
        Add this to the science exposure to match the reference exposure.
    - fitRMS: rms of the fit. This is the sqrt(mean(residuals**2)).
    - matchedMSE: the MSE of the reference and matched images: mean((refImage - matchedSciImage)**2);
      should be comparable to difference image's mean variance.
    - diffImVar: the mean variance of the difference image.
    All fields except isReference will be None if isReference True or the fit failed.

@warning: all exposures must exist on disk

Definition at line 151 of file matchBackgrounds.py.

    def run(self, expRefList, expDatasetType, imageScalerList=None, refExpDataRef=None, refImageScaler=None):
        """Match the backgrounds of a list of coadd temp exposures to a reference coadd temp exposure.
 
        Choose a refExpDataRef automatically if none supplied.
 
        @param[in] expRefList: list of data references to science exposures to be background-matched;
            all exposures must exist.
        @param[in] expDatasetType: dataset type of exposures, e.g. 'goodSeeingCoadd_tempExp'
        @param[in] imageScalerList: list of image scalers (coaddUtils.ImageScaler);
            if None then the images are not scaled
        @param[in] refExpDataRef: data reference for the reference exposure.
            If None, then this task selects the best exposures from expRefList.
            if not None then must be one of the exposures in expRefList.
        @param[in] refImageScaler: image scaler for reference image;
            ignored if refExpDataRef is None, else scaling is not performed if None
 
        @return: a pipBase.Struct containing these fields:
        - backgroundInfoList: a list of pipeBase.Struct, one per exposure in expRefList,
            each of which contains these fields:
            - isReference: this is the reference exposure (only one returned Struct will
                contain True for this value, unless the ref exposure is listed multiple times)
            - backgroundModel: differential background model (afw.Math.Background or afw.Math.Approximate).
                Add this to the science exposure to match the reference exposure.
            - fitRMS: rms of the fit. This is the sqrt(mean(residuals**2)).
            - matchedMSE: the MSE of the reference and matched images: mean((refImage - matchedSciImage)**2);
              should be comparable to difference image's mean variance.
            - diffImVar: the mean variance of the difference image.
            All fields except isReference will be None if isReference True or the fit failed.
 
        @warning: all exposures must exist on disk
        """
 
        numExp = len(expRefList)
        if numExp < 1:
            raise pipeBase.TaskError("No exposures to match")
 
        if expDatasetType is None:
            raise pipeBase.TaskError("Must specify expDatasetType")
 
        if imageScalerList is None:
            self.log.info("imageScalerList is None; no scaling will be performed")
            imageScalerList = [None] * numExp
 
        if len(expRefList) != len(imageScalerList):
            raise RuntimeError("len(expRefList) = %s != %s = len(imageScalerList)" %
                               (len(expRefList), len(imageScalerList)))
 
        refInd = None
        if refExpDataRef is None:
            # select the best reference exposure from expRefList
            refInd = self.selectRefExposure(
                expRefList=expRefList,
                imageScalerList=imageScalerList,
                expDatasetType=expDatasetType,
            )
            refExpDataRef = expRefList[refInd]
            refImageScaler = imageScalerList[refInd]
 
        # refIndSet is the index of all exposures in expDataList that match the reference.
        # It is used to avoid background-matching an exposure to itself. It is a list
        # because it is possible (though unlikely) that expDataList will contain duplicates.
        expKeyList = refExpDataRef.butlerSubset.butler.getKeys(expDatasetType)
        refMatcher = DataRefMatcher(refExpDataRef.butlerSubset.butler, expDatasetType)
        refIndSet = set(refMatcher.matchList(ref0=refExpDataRef, refList=expRefList))
 
        if refInd is not None and refInd not in refIndSet:
            raise RuntimeError("Internal error: selected reference %s not found in expRefList")
 
        refExposure = refExpDataRef.get(expDatasetType, immediate=True)
        if refImageScaler is not None:
            refMI = refExposure.getMaskedImage()
            refImageScaler.scaleMaskedImage(refMI)
 
        debugIdKeyList = tuple(set(expKeyList) - set(['tract', 'patch']))
 
        self.log.info("Matching %d Exposures", numExp)
 
        backgroundInfoList = []
        for ind, (toMatchRef, imageScaler) in enumerate(zip(expRefList, imageScalerList)):
            if ind in refIndSet:
                backgroundInfoStruct = pipeBase.Struct(
                    isReference=True,
                    backgroundModel=None,
                    fitRMS=0.0,
                    matchedMSE=None,
                    diffImVar=None,
                )
            else:
                self.log.info("Matching background of %s to %s", toMatchRef.dataId, refExpDataRef.dataId)
                try:
                    toMatchExposure = toMatchRef.get(expDatasetType, immediate=True)
                    if imageScaler is not None:
                        toMatchMI = toMatchExposure.getMaskedImage()
                        imageScaler.scaleMaskedImage(toMatchMI)
                    # store a string specifying the visit to label debug plot
                    self.debugDataIdString = ''.join([str(toMatchRef.dataId[vk]) for vk in debugIdKeyList])
                    backgroundInfoStruct = self.matchBackgrounds(
                        refExposure=refExposure,
                        sciExposure=toMatchExposure,
                    )
                    backgroundInfoStruct.isReference = False
                except Exception as e:
                    self.log.warning("Failed to fit background %s: %s", toMatchRef.dataId, e)
                    backgroundInfoStruct = pipeBase.Struct(
                        isReference=False,
                        backgroundModel=None,
                        fitRMS=None,
                        matchedMSE=None,
                        diffImVar=None,
                    )
 
            backgroundInfoList.append(backgroundInfoStruct)
 
        return pipeBase.Struct(
            backgroundInfoList=backgroundInfoList)
 

selectRefExposure()

def lsst.pipe.tasks.matchBackgrounds.MatchBackgroundsTask.selectRefExposure	(		self,
			expRefList,
			imageScalerList,
			expDatasetType
	)

Find best exposure to use as the reference exposure

Calculate an appropriate reference exposure by minimizing a cost function that penalizes
high variance,  high background level, and low coverage. Use the following config parameters:
- bestRefWeightCoverage
- bestRefWeightVariance
- bestRefWeightLevel

@param[in] expRefList: list of data references to exposures.
    Retrieves dataset type specified by expDatasetType.
    If an exposure is not found, it is skipped with a warning.
@param[in] imageScalerList: list of image scalers (coaddUtils.ImageScaler);
    must be the same length as expRefList
@param[in] expDatasetType: dataset type of exposure: e.g. 'goodSeeingCoadd_tempExp'

@return: index of best exposure

@raise pipeBase.TaskError if none of the exposures in expRefList are found.

Definition at line 268 of file matchBackgrounds.py.

    def selectRefExposure(self, expRefList, imageScalerList, expDatasetType):
        """Find best exposure to use as the reference exposure
 
        Calculate an appropriate reference exposure by minimizing a cost function that penalizes
        high variance,  high background level, and low coverage. Use the following config parameters:
        - bestRefWeightCoverage
        - bestRefWeightVariance
        - bestRefWeightLevel
 
        @param[in] expRefList: list of data references to exposures.
            Retrieves dataset type specified by expDatasetType.
            If an exposure is not found, it is skipped with a warning.
        @param[in] imageScalerList: list of image scalers (coaddUtils.ImageScaler);
            must be the same length as expRefList
        @param[in] expDatasetType: dataset type of exposure: e.g. 'goodSeeingCoadd_tempExp'
 
        @return: index of best exposure
 
        @raise pipeBase.TaskError if none of the exposures in expRefList are found.
        """
        self.log.info("Calculating best reference visit")
        varList = []
        meanBkgdLevelList = []
        coverageList = []
 
        if len(expRefList) != len(imageScalerList):
            raise RuntimeError("len(expRefList) = %s != %s = len(imageScalerList)" %
                               (len(expRefList), len(imageScalerList)))
 
        for expRef, imageScaler in zip(expRefList, imageScalerList):
            exposure = expRef.get(expDatasetType, immediate=True)
            maskedImage = exposure.getMaskedImage()
            if imageScaler is not None:
                try:
                    imageScaler.scaleMaskedImage(maskedImage)
                except Exception:
                    # need to put a place holder in Arr
                    varList.append(numpy.nan)
                    meanBkgdLevelList.append(numpy.nan)
                    coverageList.append(numpy.nan)
                    continue
            statObjIm = afwMath.makeStatistics(maskedImage.getImage(), maskedImage.getMask(),
                                               afwMath.MEAN | afwMath.NPOINT | afwMath.VARIANCE, self.sctrl)
            meanVar, meanVarErr = statObjIm.getResult(afwMath.VARIANCE)
            meanBkgdLevel, meanBkgdLevelErr = statObjIm.getResult(afwMath.MEAN)
            npoints, npointsErr = statObjIm.getResult(afwMath.NPOINT)
            varList.append(meanVar)
            meanBkgdLevelList.append(meanBkgdLevel)
            coverageList.append(npoints)
        if not coverageList:
            raise pipeBase.TaskError(
                "None of the candidate %s exist; cannot select best reference exposure" % (expDatasetType,))
 
        # Normalize metrics to range from  0 to 1
        varArr = numpy.array(varList)/numpy.nanmax(varList)
        meanBkgdLevelArr = numpy.array(meanBkgdLevelList)/numpy.nanmax(meanBkgdLevelList)
        coverageArr = numpy.nanmin(coverageList)/numpy.array(coverageList)
 
        costFunctionArr = self.config.bestRefWeightVariance * varArr
        costFunctionArr += self.config.bestRefWeightLevel * meanBkgdLevelArr
        costFunctionArr += self.config.bestRefWeightCoverage * coverageArr
        return numpy.nanargmin(costFunctionArr)
 

Member Data Documentation

ConfigClass

lsst.pipe.tasks.matchBackgrounds.MatchBackgroundsTask.ConfigClass = MatchBackgroundsConfig

static

Definition at line 140 of file matchBackgrounds.py.

debugDataIdString

lsst.pipe.tasks.matchBackgrounds.MatchBackgroundsTask.debugDataIdString

Definition at line 246 of file matchBackgrounds.py.

sctrl

lsst.pipe.tasks.matchBackgrounds.MatchBackgroundsTask.sctrl

Definition at line 146 of file matchBackgrounds.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-0.7.0/Linux64/pipe_tasks/21.0.0-172-gfb10e10a+18fedfabac/python/lsst/pipe/tasks/matchBackgrounds.py