1272 """Perform instrument signature removal on an exposure.
1274 Steps included in the ISR processing, in order performed, are:
1275 - saturation and suspect pixel masking
1276 - overscan subtraction
1277 - CCD assembly of individual amplifiers
1279 - variance image construction
1280 - linearization of non-linear response
1282 - brighter-fatter correction
1285 - stray light subtraction
1287 - masking of known defects and camera specific features
1288 - vignette calculation
1289 - appending transmission curve and distortion model
1293 ccdExposure : `lsst.afw.image.Exposure`
1294 The raw exposure that is to be run through ISR. The
1295 exposure is modified by this method.
1296 camera : `lsst.afw.cameraGeom.Camera`, optional
1297 The camera geometry for this exposure. Required if
1298 one or more of ``ccdExposure``, ``bias``, ``dark``, or
1299 ``flat`` does not have an associated detector.
1300 bias : `lsst.afw.image.Exposure`, optional
1301 Bias calibration frame.
1302 linearizer : `lsst.ip.isr.linearize.LinearizeBase`, optional
1303 Functor for linearization.
1304 crosstalk : `lsst.ip.isr.crosstalk.CrosstalkCalib`, optional
1305 Calibration for crosstalk.
1306 crosstalkSources : `list`, optional
1307 List of possible crosstalk sources.
1308 dark : `lsst.afw.image.Exposure`, optional
1309 Dark calibration frame.
1310 flat : `lsst.afw.image.Exposure`, optional
1311 Flat calibration frame.
1312 ptc : `lsst.ip.isr.PhotonTransferCurveDataset`, optional
1313 Photon transfer curve dataset, with, e.g., gains
1315 bfKernel : `numpy.ndarray`, optional
1316 Brighter-fatter kernel.
1317 bfGains : `dict` of `float`, optional
1318 Gains used to override the detector's nominal gains for the
1319 brighter-fatter correction. A dict keyed by amplifier name for
1320 the detector in question.
1321 defects : `lsst.ip.isr.Defects`, optional
1323 fringes : `lsst.pipe.base.Struct`, optional
1324 Struct containing the fringe correction data, with
1326 - ``fringes``: fringe calibration frame (`afw.image.Exposure`)
1327 - ``seed``: random seed derived from the ccdExposureId for random
1328 number generator (`uint32`)
1329 opticsTransmission: `lsst.afw.image.TransmissionCurve`, optional
1330 A ``TransmissionCurve`` that represents the throughput of the,
1331 optics, to be evaluated in focal-plane coordinates.
1332 filterTransmission : `lsst.afw.image.TransmissionCurve`
1333 A ``TransmissionCurve`` that represents the throughput of the
1334 filter itself, to be evaluated in focal-plane coordinates.
1335 sensorTransmission : `lsst.afw.image.TransmissionCurve`
1336 A ``TransmissionCurve`` that represents the throughput of the
1337 sensor itself, to be evaluated in post-assembly trimmed detector
1339 atmosphereTransmission : `lsst.afw.image.TransmissionCurve`
1340 A ``TransmissionCurve`` that represents the throughput of the
1341 atmosphere, assumed to be spatially constant.
1342 detectorNum : `int`, optional
1343 The integer number for the detector to process.
1344 isGen3 : bool, optional
1345 Flag this call to run() as using the Gen3 butler environment.
1346 strayLightData : `object`, optional
1347 Opaque object containing calibration information for stray-light
1348 correction. If `None`, no correction will be performed.
1349 illumMaskedImage : `lsst.afw.image.MaskedImage`, optional
1350 Illumination correction image.
1354 result : `lsst.pipe.base.Struct`
1355 Result struct with component:
1356 - ``exposure`` : `afw.image.Exposure`
1357 The fully ISR corrected exposure.
1358 - ``outputExposure`` : `afw.image.Exposure`
1359 An alias for `exposure`
1360 - ``ossThumb`` : `numpy.ndarray`
1361 Thumbnail image of the exposure after overscan subtraction.
1362 - ``flattenedThumb`` : `numpy.ndarray`
1363 Thumbnail image of the exposure after flat-field correction.
1368 Raised if a configuration option is set to True, but the
1369 required calibration data has not been specified.
1373 The current processed exposure can be viewed by setting the
1374 appropriate lsstDebug entries in the `debug.display`
1375 dictionary. The names of these entries correspond to some of
1376 the IsrTaskConfig Boolean options, with the value denoting the
1377 frame to use. The exposure is shown inside the matching
1378 option check and after the processing of that step has
1379 finished. The steps with debug points are:
1390 In addition, setting the "postISRCCD" entry displays the
1391 exposure after all ISR processing has finished.
1400 ccdExposure = self.ensureExposure(ccdExposure, camera, detectorNum)
1401 bias = self.ensureExposure(bias, camera, detectorNum)
1402 dark = self.ensureExposure(dark, camera, detectorNum)
1403 flat = self.ensureExposure(flat, camera, detectorNum)
1405 if isinstance(ccdExposure, ButlerDataRef):
1406 return self.runDataRef(ccdExposure)
1408 ccd = ccdExposure.getDetector()
1409 filterLabel = ccdExposure.getFilterLabel()
1410 physicalFilter = isrFunctions.getPhysicalFilter(filterLabel, self.log)
1413 assert not self.config.doAssembleCcd,
"You need a Detector to run assembleCcd."
1414 ccd = [FakeAmp(ccdExposure, self.config)]
1417 if self.config.doBias
and bias
is None:
1418 raise RuntimeError(
"Must supply a bias exposure if config.doBias=True.")
1419 if self.doLinearize(ccd)
and linearizer
is None:
1420 raise RuntimeError(
"Must supply a linearizer if config.doLinearize=True for this detector.")
1421 if self.config.doBrighterFatter
and bfKernel
is None:
1422 raise RuntimeError(
"Must supply a kernel if config.doBrighterFatter=True.")
1423 if self.config.doDark
and dark
is None:
1424 raise RuntimeError(
"Must supply a dark exposure if config.doDark=True.")
1425 if self.config.doFlat
and flat
is None:
1426 raise RuntimeError(
"Must supply a flat exposure if config.doFlat=True.")
1427 if self.config.doDefect
and defects
is None:
1428 raise RuntimeError(
"Must supply defects if config.doDefect=True.")
1429 if (self.config.doFringe
and physicalFilter
in self.fringe.config.filters
1430 and fringes.fringes
is None):
1435 raise RuntimeError(
"Must supply fringe exposure as a pipeBase.Struct.")
1436 if (self.config.doIlluminationCorrection
and physicalFilter
in self.config.illumFilters
1437 and illumMaskedImage
is None):
1438 raise RuntimeError(
"Must supply an illumcor if config.doIlluminationCorrection=True.")
1441 if self.config.doConvertIntToFloat:
1442 self.log.
info(
"Converting exposure to floating point values.")
1443 ccdExposure = self.convertIntToFloat(ccdExposure)
1445 if self.config.doBias
and self.config.doBiasBeforeOverscan:
1446 self.log.
info(
"Applying bias correction.")
1447 isrFunctions.biasCorrection(ccdExposure.getMaskedImage(), bias.getMaskedImage(),
1448 trimToFit=self.config.doTrimToMatchCalib)
1449 self.debugView(ccdExposure,
"doBias")
1456 if ccdExposure.getBBox().
contains(amp.getBBox()):
1459 badAmp = self.maskAmplifier(ccdExposure, amp, defects)
1461 if self.config.doOverscan
and not badAmp:
1463 overscanResults = self.overscanCorrection(ccdExposure, amp)
1464 self.log.
debug(
"Corrected overscan for amplifier %s.", amp.getName())
1465 if overscanResults
is not None and \
1466 self.config.qa
is not None and self.config.qa.saveStats
is True:
1467 if isinstance(overscanResults.overscanFit, float):
1468 qaMedian = overscanResults.overscanFit
1469 qaStdev = float(
"NaN")
1472 afwMath.MEDIAN | afwMath.STDEVCLIP)
1473 qaMedian = qaStats.getValue(afwMath.MEDIAN)
1474 qaStdev = qaStats.getValue(afwMath.STDEVCLIP)
1476 self.metadata[f
"FIT MEDIAN {amp.getName()}"] = qaMedian
1477 self.metadata[f
"FIT STDEV {amp.getName()}"] = qaStdev
1478 self.log.
debug(
" Overscan stats for amplifer %s: %f +/- %f",
1479 amp.getName(), qaMedian, qaStdev)
1483 afwMath.MEDIAN | afwMath.STDEVCLIP)
1484 qaMedianAfter = qaStatsAfter.getValue(afwMath.MEDIAN)
1485 qaStdevAfter = qaStatsAfter.getValue(afwMath.STDEVCLIP)
1487 self.metadata[f
"RESIDUAL MEDIAN {amp.getName()}"] = qaMedianAfter
1488 self.metadata[f
"RESIDUAL STDEV {amp.getName()}"] = qaStdevAfter
1489 self.log.
debug(
" Overscan stats for amplifer %s after correction: %f +/- %f",
1490 amp.getName(), qaMedianAfter, qaStdevAfter)
1492 ccdExposure.getMetadata().
set(
'OVERSCAN',
"Overscan corrected")
1495 self.log.
warning(
"Amplifier %s is bad.", amp.getName())
1496 overscanResults =
None
1498 overscans.append(overscanResults
if overscanResults
is not None else None)
1500 self.log.
info(
"Skipped OSCAN for %s.", amp.getName())
1502 if self.config.doCrosstalk
and self.config.doCrosstalkBeforeAssemble:
1503 self.log.
info(
"Applying crosstalk correction.")
1504 self.crosstalk.
run(ccdExposure, crosstalk=crosstalk,
1505 crosstalkSources=crosstalkSources, camera=camera)
1506 self.debugView(ccdExposure,
"doCrosstalk")
1508 if self.config.doAssembleCcd:
1509 self.log.
info(
"Assembling CCD from amplifiers.")
1510 ccdExposure = self.assembleCcd.assembleCcd(ccdExposure)
1512 if self.config.expectWcs
and not ccdExposure.getWcs():
1513 self.log.
warning(
"No WCS found in input exposure.")
1514 self.debugView(ccdExposure,
"doAssembleCcd")
1517 if self.config.qa.doThumbnailOss:
1518 ossThumb = isrQa.makeThumbnail(ccdExposure, isrQaConfig=self.config.qa)
1520 if self.config.doBias
and not self.config.doBiasBeforeOverscan:
1521 self.log.
info(
"Applying bias correction.")
1522 isrFunctions.biasCorrection(ccdExposure.getMaskedImage(), bias.getMaskedImage(),
1523 trimToFit=self.config.doTrimToMatchCalib)
1524 self.debugView(ccdExposure,
"doBias")
1526 if self.config.doVariance:
1527 for amp, overscanResults
in zip(ccd, overscans):
1528 if ccdExposure.getBBox().
contains(amp.getBBox()):
1529 self.log.
debug(
"Constructing variance map for amplifer %s.", amp.getName())
1530 ampExposure = ccdExposure.Factory(ccdExposure, amp.getBBox())
1531 if overscanResults
is not None:
1532 self.updateVariance(ampExposure, amp,
1533 overscanImage=overscanResults.overscanImage,
1536 self.updateVariance(ampExposure, amp,
1539 if self.config.qa
is not None and self.config.qa.saveStats
is True:
1541 afwMath.MEDIAN | afwMath.STDEVCLIP)
1542 self.metadata[f
"ISR VARIANCE {amp.getName()} MEDIAN"] = \
1543 qaStats.getValue(afwMath.MEDIAN)
1544 self.metadata[f
"ISR VARIANCE {amp.getName()} STDEV"] = \
1545 qaStats.getValue(afwMath.STDEVCLIP)
1546 self.log.
debug(
" Variance stats for amplifer %s: %f +/- %f.",
1547 amp.getName(), qaStats.getValue(afwMath.MEDIAN),
1548 qaStats.getValue(afwMath.STDEVCLIP))
1549 if self.config.maskNegativeVariance:
1550 self.maskNegativeVariance(ccdExposure)
1552 if self.doLinearize(ccd):
1553 self.log.
info(
"Applying linearizer.")
1554 linearizer.applyLinearity(image=ccdExposure.getMaskedImage().getImage(),
1555 detector=ccd, log=self.log)
1557 if self.config.doCrosstalk
and not self.config.doCrosstalkBeforeAssemble:
1558 self.log.
info(
"Applying crosstalk correction.")
1559 self.crosstalk.
run(ccdExposure, crosstalk=crosstalk,
1560 crosstalkSources=crosstalkSources, isTrimmed=
True)
1561 self.debugView(ccdExposure,
"doCrosstalk")
1566 if self.config.doDefect:
1567 self.log.
info(
"Masking defects.")
1568 self.maskDefect(ccdExposure, defects)
1570 if self.config.numEdgeSuspect > 0:
1571 self.log.
info(
"Masking edges as SUSPECT.")
1572 self.maskEdges(ccdExposure, numEdgePixels=self.config.numEdgeSuspect,
1573 maskPlane=
"SUSPECT", level=self.config.edgeMaskLevel)
1575 if self.config.doNanMasking:
1576 self.log.
info(
"Masking non-finite (NAN, inf) value pixels.")
1577 self.maskNan(ccdExposure)
1579 if self.config.doWidenSaturationTrails:
1580 self.log.
info(
"Widening saturation trails.")
1581 isrFunctions.widenSaturationTrails(ccdExposure.getMaskedImage().getMask())
1583 if self.config.doCameraSpecificMasking:
1584 self.log.
info(
"Masking regions for camera specific reasons.")
1585 self.masking.
run(ccdExposure)
1587 if self.config.doBrighterFatter:
1597 interpExp = ccdExposure.clone()
1598 with self.flatContext(interpExp, flat, dark):
1599 isrFunctions.interpolateFromMask(
1600 maskedImage=interpExp.getMaskedImage(),
1601 fwhm=self.config.fwhm,
1602 growSaturatedFootprints=self.config.growSaturationFootprintSize,
1603 maskNameList=
list(self.config.brighterFatterMaskListToInterpolate)
1605 bfExp = interpExp.clone()
1607 self.log.
info(
"Applying brighter-fatter correction using kernel type %s / gains %s.",
1609 bfResults = isrFunctions.brighterFatterCorrection(bfExp, bfKernel,
1610 self.config.brighterFatterMaxIter,
1611 self.config.brighterFatterThreshold,
1612 self.config.brighterFatterApplyGain,
1614 if bfResults[1] == self.config.brighterFatterMaxIter:
1615 self.log.
warning(
"Brighter-fatter correction did not converge, final difference %f.",
1618 self.log.
info(
"Finished brighter-fatter correction in %d iterations.",
1620 image = ccdExposure.getMaskedImage().getImage()
1621 bfCorr = bfExp.getMaskedImage().getImage()
1622 bfCorr -= interpExp.getMaskedImage().getImage()
1631 self.log.
info(
"Ensuring image edges are masked as EDGE to the brighter-fatter kernel size.")
1632 self.maskEdges(ccdExposure, numEdgePixels=numpy.max(bfKernel.shape) // 2,
1635 if self.config.brighterFatterMaskGrowSize > 0:
1636 self.log.
info(
"Growing masks to account for brighter-fatter kernel convolution.")
1637 for maskPlane
in self.config.brighterFatterMaskListToInterpolate:
1638 isrFunctions.growMasks(ccdExposure.getMask(),
1639 radius=self.config.brighterFatterMaskGrowSize,
1640 maskNameList=maskPlane,
1641 maskValue=maskPlane)
1643 self.debugView(ccdExposure,
"doBrighterFatter")
1645 if self.config.doDark:
1646 self.log.
info(
"Applying dark correction.")
1647 self.darkCorrection(ccdExposure, dark)
1648 self.debugView(ccdExposure,
"doDark")
1650 if self.config.doFringe
and not self.config.fringeAfterFlat:
1651 self.log.
info(
"Applying fringe correction before flat.")
1652 self.fringe.
run(ccdExposure, **fringes.getDict())
1653 self.debugView(ccdExposure,
"doFringe")
1655 if self.config.doStrayLight
and self.strayLight.check(ccdExposure):
1656 self.log.
info(
"Checking strayLight correction.")
1657 self.strayLight.
run(ccdExposure, strayLightData)
1658 self.debugView(ccdExposure,
"doStrayLight")
1660 if self.config.doFlat:
1661 self.log.
info(
"Applying flat correction.")
1662 self.flatCorrection(ccdExposure, flat)
1663 self.debugView(ccdExposure,
"doFlat")
1665 if self.config.doApplyGains:
1666 self.log.
info(
"Applying gain correction instead of flat.")
1667 if self.config.usePtcGains:
1668 self.log.
info(
"Using gains from the Photon Transfer Curve.")
1669 isrFunctions.applyGains(ccdExposure, self.config.normalizeGains,
1672 isrFunctions.applyGains(ccdExposure, self.config.normalizeGains)
1674 if self.config.doFringe
and self.config.fringeAfterFlat:
1675 self.log.
info(
"Applying fringe correction after flat.")
1676 self.fringe.
run(ccdExposure, **fringes.getDict())
1678 if self.config.doVignette:
1679 self.log.
info(
"Constructing Vignette polygon.")
1680 self.vignettePolygon = self.vignette.
run(ccdExposure)
1682 if self.config.vignette.doWriteVignettePolygon:
1683 self.setValidPolygonIntersect(ccdExposure, self.vignettePolygon)
1685 if self.config.doAttachTransmissionCurve:
1686 self.log.
info(
"Adding transmission curves.")
1687 isrFunctions.attachTransmissionCurve(ccdExposure, opticsTransmission=opticsTransmission,
1688 filterTransmission=filterTransmission,
1689 sensorTransmission=sensorTransmission,
1690 atmosphereTransmission=atmosphereTransmission)
1692 flattenedThumb =
None
1693 if self.config.qa.doThumbnailFlattened:
1694 flattenedThumb = isrQa.makeThumbnail(ccdExposure, isrQaConfig=self.config.qa)
1696 if self.config.doIlluminationCorrection
and physicalFilter
in self.config.illumFilters:
1697 self.log.
info(
"Performing illumination correction.")
1698 isrFunctions.illuminationCorrection(ccdExposure.getMaskedImage(),
1699 illumMaskedImage, illumScale=self.config.illumScale,
1700 trimToFit=self.config.doTrimToMatchCalib)
1703 if self.config.doSaveInterpPixels:
1704 preInterpExp = ccdExposure.clone()
1719 if self.config.doSetBadRegions:
1720 badPixelCount, badPixelValue = isrFunctions.setBadRegions(ccdExposure)
1721 if badPixelCount > 0:
1722 self.log.
info(
"Set %d BAD pixels to %f.", badPixelCount, badPixelValue)
1724 if self.config.doInterpolate:
1725 self.log.
info(
"Interpolating masked pixels.")
1726 isrFunctions.interpolateFromMask(
1727 maskedImage=ccdExposure.getMaskedImage(),
1728 fwhm=self.config.fwhm,
1729 growSaturatedFootprints=self.config.growSaturationFootprintSize,
1730 maskNameList=
list(self.config.maskListToInterpolate)
1733 self.roughZeroPoint(ccdExposure)
1736 if self.config.doAmpOffset:
1737 self.log.
info(
"Correcting amp offsets.")
1738 self.ampOffset.
run(ccdExposure)
1740 if self.config.doMeasureBackground:
1741 self.log.
info(
"Measuring background level.")
1742 self.measureBackground(ccdExposure, self.config.qa)
1744 if self.config.qa
is not None and self.config.qa.saveStats
is True:
1746 ampExposure = ccdExposure.Factory(ccdExposure, amp.getBBox())
1748 afwMath.MEDIAN | afwMath.STDEVCLIP)
1749 self.metadata[f
"ISR BACKGROUND {amp.getName()} MEDIAN"] = qaStats.getValue(afwMath.MEDIAN)
1750 self.metadata[f
"ISR BACKGROUND {amp.getName()} STDEV"] = \
1751 qaStats.getValue(afwMath.STDEVCLIP)
1752 self.log.
debug(
" Background stats for amplifer %s: %f +/- %f",
1753 amp.getName(), qaStats.getValue(afwMath.MEDIAN),
1754 qaStats.getValue(afwMath.STDEVCLIP))
1756 self.debugView(ccdExposure,
"postISRCCD")
1758 return pipeBase.Struct(
1759 exposure=ccdExposure,
1761 flattenedThumb=flattenedThumb,
1763 preInterpExposure=preInterpExp,
1764 outputExposure=ccdExposure,
1765 outputOssThumbnail=ossThumb,
1766 outputFlattenedThumbnail=flattenedThumb,
daf::base::PropertyList * list
daf::base::PropertySet * set