1271 """Perform instrument signature removal on an exposure.
1273 Steps included in the ISR processing, in order performed, are:
1274 - saturation and suspect pixel masking
1275 - overscan subtraction
1276 - CCD assembly of individual amplifiers
1278 - variance image construction
1279 - linearization of non-linear response
1281 - brighter-fatter correction
1284 - stray light subtraction
1286 - masking of known defects and camera specific features
1287 - vignette calculation
1288 - appending transmission curve and distortion model
1292 ccdExposure : `lsst.afw.image.Exposure`
1293 The raw exposure that is to be run through ISR. The
1294 exposure is modified by this method.
1295 camera : `lsst.afw.cameraGeom.Camera`, optional
1296 The camera geometry for this exposure. Required if
1297 one or more of ``ccdExposure``, ``bias``, ``dark``, or
1298 ``flat`` does not have an associated detector.
1299 bias : `lsst.afw.image.Exposure`, optional
1300 Bias calibration frame.
1301 linearizer : `lsst.ip.isr.linearize.LinearizeBase`, optional
1302 Functor for linearization.
1303 crosstalk : `lsst.ip.isr.crosstalk.CrosstalkCalib`, optional
1304 Calibration for crosstalk.
1305 crosstalkSources : `list`, optional
1306 List of possible crosstalk sources.
1307 dark : `lsst.afw.image.Exposure`, optional
1308 Dark calibration frame.
1309 flat : `lsst.afw.image.Exposure`, optional
1310 Flat calibration frame.
1311 ptc : `lsst.ip.isr.PhotonTransferCurveDataset`, optional
1312 Photon transfer curve dataset, with, e.g., gains
1314 bfKernel : `numpy.ndarray`, optional
1315 Brighter-fatter kernel.
1316 bfGains : `dict` of `float`, optional
1317 Gains used to override the detector's nominal gains for the
1318 brighter-fatter correction. A dict keyed by amplifier name for
1319 the detector in question.
1320 defects : `lsst.ip.isr.Defects`, optional
1322 fringes : `lsst.pipe.base.Struct`, optional
1323 Struct containing the fringe correction data, with
1325 - ``fringes``: fringe calibration frame (`afw.image.Exposure`)
1326 - ``seed``: random seed derived from the ccdExposureId for random
1327 number generator (`uint32`)
1328 opticsTransmission: `lsst.afw.image.TransmissionCurve`, optional
1329 A ``TransmissionCurve`` that represents the throughput of the,
1330 optics, to be evaluated in focal-plane coordinates.
1331 filterTransmission : `lsst.afw.image.TransmissionCurve`
1332 A ``TransmissionCurve`` that represents the throughput of the
1333 filter itself, to be evaluated in focal-plane coordinates.
1334 sensorTransmission : `lsst.afw.image.TransmissionCurve`
1335 A ``TransmissionCurve`` that represents the throughput of the
1336 sensor itself, to be evaluated in post-assembly trimmed detector
1338 atmosphereTransmission : `lsst.afw.image.TransmissionCurve`
1339 A ``TransmissionCurve`` that represents the throughput of the
1340 atmosphere, assumed to be spatially constant.
1341 detectorNum : `int`, optional
1342 The integer number for the detector to process.
1343 isGen3 : bool, optional
1344 Flag this call to run() as using the Gen3 butler environment.
1345 strayLightData : `object`, optional
1346 Opaque object containing calibration information for stray-light
1347 correction. If `None`, no correction will be performed.
1348 illumMaskedImage : `lsst.afw.image.MaskedImage`, optional
1349 Illumination correction image.
1353 result : `lsst.pipe.base.Struct`
1354 Result struct with component:
1355 - ``exposure`` : `afw.image.Exposure`
1356 The fully ISR corrected exposure.
1357 - ``outputExposure`` : `afw.image.Exposure`
1358 An alias for `exposure`
1359 - ``ossThumb`` : `numpy.ndarray`
1360 Thumbnail image of the exposure after overscan subtraction.
1361 - ``flattenedThumb`` : `numpy.ndarray`
1362 Thumbnail image of the exposure after flat-field correction.
1367 Raised if a configuration option is set to True, but the
1368 required calibration data has not been specified.
1372 The current processed exposure can be viewed by setting the
1373 appropriate lsstDebug entries in the `debug.display`
1374 dictionary. The names of these entries correspond to some of
1375 the IsrTaskConfig Boolean options, with the value denoting the
1376 frame to use. The exposure is shown inside the matching
1377 option check and after the processing of that step has
1378 finished. The steps with debug points are:
1389 In addition, setting the "postISRCCD" entry displays the
1390 exposure after all ISR processing has finished.
1399 ccdExposure = self.ensureExposure(ccdExposure, camera, detectorNum)
1400 bias = self.ensureExposure(bias, camera, detectorNum)
1401 dark = self.ensureExposure(dark, camera, detectorNum)
1402 flat = self.ensureExposure(flat, camera, detectorNum)
1404 if isinstance(ccdExposure, ButlerDataRef):
1405 return self.runDataRef(ccdExposure)
1407 ccd = ccdExposure.getDetector()
1408 filterLabel = ccdExposure.getFilterLabel()
1409 physicalFilter = isrFunctions.getPhysicalFilter(filterLabel, self.log)
1412 assert not self.config.doAssembleCcd,
"You need a Detector to run assembleCcd."
1413 ccd = [FakeAmp(ccdExposure, self.config)]
1416 if self.config.doBias
and bias
is None:
1417 raise RuntimeError(
"Must supply a bias exposure if config.doBias=True.")
1418 if self.doLinearize(ccd)
and linearizer
is None:
1419 raise RuntimeError(
"Must supply a linearizer if config.doLinearize=True for this detector.")
1420 if self.config.doBrighterFatter
and bfKernel
is None:
1421 raise RuntimeError(
"Must supply a kernel if config.doBrighterFatter=True.")
1422 if self.config.doDark
and dark
is None:
1423 raise RuntimeError(
"Must supply a dark exposure if config.doDark=True.")
1424 if self.config.doFlat
and flat
is None:
1425 raise RuntimeError(
"Must supply a flat exposure if config.doFlat=True.")
1426 if self.config.doDefect
and defects
is None:
1427 raise RuntimeError(
"Must supply defects if config.doDefect=True.")
1428 if (self.config.doFringe
and physicalFilter
in self.fringe.config.filters
1429 and fringes.fringes
is None):
1434 raise RuntimeError(
"Must supply fringe exposure as a pipeBase.Struct.")
1435 if (self.config.doIlluminationCorrection
and physicalFilter
in self.config.illumFilters
1436 and illumMaskedImage
is None):
1437 raise RuntimeError(
"Must supply an illumcor if config.doIlluminationCorrection=True.")
1440 if self.config.doConvertIntToFloat:
1441 self.log.
info(
"Converting exposure to floating point values.")
1442 ccdExposure = self.convertIntToFloat(ccdExposure)
1444 if self.config.doBias
and self.config.doBiasBeforeOverscan:
1445 self.log.
info(
"Applying bias correction.")
1446 isrFunctions.biasCorrection(ccdExposure.getMaskedImage(), bias.getMaskedImage(),
1447 trimToFit=self.config.doTrimToMatchCalib)
1448 self.debugView(ccdExposure,
"doBias")
1455 if ccdExposure.getBBox().
contains(amp.getBBox()):
1458 badAmp = self.maskAmplifier(ccdExposure, amp, defects)
1460 if self.config.doOverscan
and not badAmp:
1462 overscanResults = self.overscanCorrection(ccdExposure, amp)
1463 self.log.
debug(
"Corrected overscan for amplifier %s.", amp.getName())
1464 if overscanResults
is not None and \
1465 self.config.qa
is not None and self.config.qa.saveStats
is True:
1466 if isinstance(overscanResults.overscanFit, float):
1467 qaMedian = overscanResults.overscanFit
1468 qaStdev = float(
"NaN")
1471 afwMath.MEDIAN | afwMath.STDEVCLIP)
1472 qaMedian = qaStats.getValue(afwMath.MEDIAN)
1473 qaStdev = qaStats.getValue(afwMath.STDEVCLIP)
1475 self.metadata.
set(f
"FIT MEDIAN {amp.getName()}", qaMedian)
1476 self.metadata.
set(f
"FIT STDEV {amp.getName()}", qaStdev)
1477 self.log.
debug(
" Overscan stats for amplifer %s: %f +/- %f",
1478 amp.getName(), qaMedian, qaStdev)
1482 afwMath.MEDIAN | afwMath.STDEVCLIP)
1483 qaMedianAfter = qaStatsAfter.getValue(afwMath.MEDIAN)
1484 qaStdevAfter = qaStatsAfter.getValue(afwMath.STDEVCLIP)
1486 self.metadata.
set(f
"RESIDUAL MEDIAN {amp.getName()}", qaMedianAfter)
1487 self.metadata.
set(f
"RESIDUAL STDEV {amp.getName()}", qaStdevAfter)
1488 self.log.
debug(
" Overscan stats for amplifer %s after correction: %f +/- %f",
1489 amp.getName(), qaMedianAfter, qaStdevAfter)
1491 ccdExposure.getMetadata().
set(
'OVERSCAN',
"Overscan corrected")
1494 self.log.
warning(
"Amplifier %s is bad.", amp.getName())
1495 overscanResults =
None
1497 overscans.append(overscanResults
if overscanResults
is not None else None)
1499 self.log.
info(
"Skipped OSCAN for %s.", amp.getName())
1501 if self.config.doCrosstalk
and self.config.doCrosstalkBeforeAssemble:
1502 self.log.
info(
"Applying crosstalk correction.")
1503 self.crosstalk.
run(ccdExposure, crosstalk=crosstalk,
1504 crosstalkSources=crosstalkSources, camera=camera)
1505 self.debugView(ccdExposure,
"doCrosstalk")
1507 if self.config.doAssembleCcd:
1508 self.log.
info(
"Assembling CCD from amplifiers.")
1509 ccdExposure = self.assembleCcd.assembleCcd(ccdExposure)
1511 if self.config.expectWcs
and not ccdExposure.getWcs():
1512 self.log.
warning(
"No WCS found in input exposure.")
1513 self.debugView(ccdExposure,
"doAssembleCcd")
1516 if self.config.qa.doThumbnailOss:
1517 ossThumb = isrQa.makeThumbnail(ccdExposure, isrQaConfig=self.config.qa)
1519 if self.config.doBias
and not self.config.doBiasBeforeOverscan:
1520 self.log.
info(
"Applying bias correction.")
1521 isrFunctions.biasCorrection(ccdExposure.getMaskedImage(), bias.getMaskedImage(),
1522 trimToFit=self.config.doTrimToMatchCalib)
1523 self.debugView(ccdExposure,
"doBias")
1525 if self.config.doVariance:
1526 for amp, overscanResults
in zip(ccd, overscans):
1527 if ccdExposure.getBBox().
contains(amp.getBBox()):
1528 self.log.
debug(
"Constructing variance map for amplifer %s.", amp.getName())
1529 ampExposure = ccdExposure.Factory(ccdExposure, amp.getBBox())
1530 if overscanResults
is not None:
1531 self.updateVariance(ampExposure, amp,
1532 overscanImage=overscanResults.overscanImage,
1535 self.updateVariance(ampExposure, amp,
1538 if self.config.qa
is not None and self.config.qa.saveStats
is True:
1540 afwMath.MEDIAN | afwMath.STDEVCLIP)
1541 self.metadata.
set(f
"ISR VARIANCE {amp.getName()} MEDIAN",
1542 qaStats.getValue(afwMath.MEDIAN))
1543 self.metadata.
set(f
"ISR VARIANCE {amp.getName()} STDEV",
1544 qaStats.getValue(afwMath.STDEVCLIP))
1545 self.log.
debug(
" Variance stats for amplifer %s: %f +/- %f.",
1546 amp.getName(), qaStats.getValue(afwMath.MEDIAN),
1547 qaStats.getValue(afwMath.STDEVCLIP))
1548 if self.config.maskNegativeVariance:
1549 self.maskNegativeVariance(ccdExposure)
1551 if self.doLinearize(ccd):
1552 self.log.
info(
"Applying linearizer.")
1553 linearizer.applyLinearity(image=ccdExposure.getMaskedImage().getImage(),
1554 detector=ccd, log=self.log)
1556 if self.config.doCrosstalk
and not self.config.doCrosstalkBeforeAssemble:
1557 self.log.
info(
"Applying crosstalk correction.")
1558 self.crosstalk.
run(ccdExposure, crosstalk=crosstalk,
1559 crosstalkSources=crosstalkSources, isTrimmed=
True)
1560 self.debugView(ccdExposure,
"doCrosstalk")
1565 if self.config.doDefect:
1566 self.log.
info(
"Masking defects.")
1567 self.maskDefect(ccdExposure, defects)
1569 if self.config.numEdgeSuspect > 0:
1570 self.log.
info(
"Masking edges as SUSPECT.")
1571 self.maskEdges(ccdExposure, numEdgePixels=self.config.numEdgeSuspect,
1572 maskPlane=
"SUSPECT", level=self.config.edgeMaskLevel)
1574 if self.config.doNanMasking:
1575 self.log.
info(
"Masking non-finite (NAN, inf) value pixels.")
1576 self.maskNan(ccdExposure)
1578 if self.config.doWidenSaturationTrails:
1579 self.log.
info(
"Widening saturation trails.")
1580 isrFunctions.widenSaturationTrails(ccdExposure.getMaskedImage().getMask())
1582 if self.config.doCameraSpecificMasking:
1583 self.log.
info(
"Masking regions for camera specific reasons.")
1584 self.masking.
run(ccdExposure)
1586 if self.config.doBrighterFatter:
1596 interpExp = ccdExposure.clone()
1597 with self.flatContext(interpExp, flat, dark):
1598 isrFunctions.interpolateFromMask(
1599 maskedImage=interpExp.getMaskedImage(),
1600 fwhm=self.config.fwhm,
1601 growSaturatedFootprints=self.config.growSaturationFootprintSize,
1602 maskNameList=
list(self.config.brighterFatterMaskListToInterpolate)
1604 bfExp = interpExp.clone()
1606 self.log.
info(
"Applying brighter-fatter correction using kernel type %s / gains %s.",
1608 bfResults = isrFunctions.brighterFatterCorrection(bfExp, bfKernel,
1609 self.config.brighterFatterMaxIter,
1610 self.config.brighterFatterThreshold,
1611 self.config.brighterFatterApplyGain,
1613 if bfResults[1] == self.config.brighterFatterMaxIter:
1614 self.log.
warning(
"Brighter-fatter correction did not converge, final difference %f.",
1617 self.log.
info(
"Finished brighter-fatter correction in %d iterations.",
1619 image = ccdExposure.getMaskedImage().getImage()
1620 bfCorr = bfExp.getMaskedImage().getImage()
1621 bfCorr -= interpExp.getMaskedImage().getImage()
1630 self.log.
info(
"Ensuring image edges are masked as EDGE to the brighter-fatter kernel size.")
1631 self.maskEdges(ccdExposure, numEdgePixels=numpy.max(bfKernel.shape) // 2,
1634 if self.config.brighterFatterMaskGrowSize > 0:
1635 self.log.
info(
"Growing masks to account for brighter-fatter kernel convolution.")
1636 for maskPlane
in self.config.brighterFatterMaskListToInterpolate:
1637 isrFunctions.growMasks(ccdExposure.getMask(),
1638 radius=self.config.brighterFatterMaskGrowSize,
1639 maskNameList=maskPlane,
1640 maskValue=maskPlane)
1642 self.debugView(ccdExposure,
"doBrighterFatter")
1644 if self.config.doDark:
1645 self.log.
info(
"Applying dark correction.")
1646 self.darkCorrection(ccdExposure, dark)
1647 self.debugView(ccdExposure,
"doDark")
1649 if self.config.doFringe
and not self.config.fringeAfterFlat:
1650 self.log.
info(
"Applying fringe correction before flat.")
1651 self.fringe.
run(ccdExposure, **fringes.getDict())
1652 self.debugView(ccdExposure,
"doFringe")
1654 if self.config.doStrayLight
and self.strayLight.check(ccdExposure):
1655 self.log.
info(
"Checking strayLight correction.")
1656 self.strayLight.
run(ccdExposure, strayLightData)
1657 self.debugView(ccdExposure,
"doStrayLight")
1659 if self.config.doFlat:
1660 self.log.
info(
"Applying flat correction.")
1661 self.flatCorrection(ccdExposure, flat)
1662 self.debugView(ccdExposure,
"doFlat")
1664 if self.config.doApplyGains:
1665 self.log.
info(
"Applying gain correction instead of flat.")
1666 if self.config.usePtcGains:
1667 self.log.
info(
"Using gains from the Photon Transfer Curve.")
1668 isrFunctions.applyGains(ccdExposure, self.config.normalizeGains,
1671 isrFunctions.applyGains(ccdExposure, self.config.normalizeGains)
1673 if self.config.doFringe
and self.config.fringeAfterFlat:
1674 self.log.
info(
"Applying fringe correction after flat.")
1675 self.fringe.
run(ccdExposure, **fringes.getDict())
1677 if self.config.doVignette:
1678 self.log.
info(
"Constructing Vignette polygon.")
1679 self.vignettePolygon = self.vignette.
run(ccdExposure)
1681 if self.config.vignette.doWriteVignettePolygon:
1682 self.setValidPolygonIntersect(ccdExposure, self.vignettePolygon)
1684 if self.config.doAttachTransmissionCurve:
1685 self.log.
info(
"Adding transmission curves.")
1686 isrFunctions.attachTransmissionCurve(ccdExposure, opticsTransmission=opticsTransmission,
1687 filterTransmission=filterTransmission,
1688 sensorTransmission=sensorTransmission,
1689 atmosphereTransmission=atmosphereTransmission)
1691 flattenedThumb =
None
1692 if self.config.qa.doThumbnailFlattened:
1693 flattenedThumb = isrQa.makeThumbnail(ccdExposure, isrQaConfig=self.config.qa)
1695 if self.config.doIlluminationCorrection
and physicalFilter
in self.config.illumFilters:
1696 self.log.
info(
"Performing illumination correction.")
1697 isrFunctions.illuminationCorrection(ccdExposure.getMaskedImage(),
1698 illumMaskedImage, illumScale=self.config.illumScale,
1699 trimToFit=self.config.doTrimToMatchCalib)
1702 if self.config.doSaveInterpPixels:
1703 preInterpExp = ccdExposure.clone()
1718 if self.config.doSetBadRegions:
1719 badPixelCount, badPixelValue = isrFunctions.setBadRegions(ccdExposure)
1720 if badPixelCount > 0:
1721 self.log.
info(
"Set %d BAD pixels to %f.", badPixelCount, badPixelValue)
1723 if self.config.doInterpolate:
1724 self.log.
info(
"Interpolating masked pixels.")
1725 isrFunctions.interpolateFromMask(
1726 maskedImage=ccdExposure.getMaskedImage(),
1727 fwhm=self.config.fwhm,
1728 growSaturatedFootprints=self.config.growSaturationFootprintSize,
1729 maskNameList=
list(self.config.maskListToInterpolate)
1732 self.roughZeroPoint(ccdExposure)
1735 if self.config.doAmpOffset:
1736 self.log.
info(
"Correcting amp offsets.")
1737 self.ampOffset.
run(ccdExposure)
1739 if self.config.doMeasureBackground:
1740 self.log.
info(
"Measuring background level.")
1741 self.measureBackground(ccdExposure, self.config.qa)
1743 if self.config.qa
is not None and self.config.qa.saveStats
is True:
1745 ampExposure = ccdExposure.Factory(ccdExposure, amp.getBBox())
1747 afwMath.MEDIAN | afwMath.STDEVCLIP)
1748 self.metadata.
set(
"ISR BACKGROUND {} MEDIAN".
format(amp.getName()),
1749 qaStats.getValue(afwMath.MEDIAN))
1750 self.metadata.
set(
"ISR BACKGROUND {} STDEV".
format(amp.getName()),
1751 qaStats.getValue(afwMath.STDEVCLIP))
1752 self.log.
debug(
" Background stats for amplifer %s: %f +/- %f",
1753 amp.getName(), qaStats.getValue(afwMath.MEDIAN),
1754 qaStats.getValue(afwMath.STDEVCLIP))
1756 self.debugView(ccdExposure,
"postISRCCD")
1758 return pipeBase.Struct(
1759 exposure=ccdExposure,
1761 flattenedThumb=flattenedThumb,
1763 preInterpExposure=preInterpExp,
1764 outputExposure=ccdExposure,
1765 outputOssThumbnail=ossThumb,
1766 outputFlattenedThumbnail=flattenedThumb,
daf::base::PropertyList * list
def format(config, name=None, writeSourceLine=True, prefix="", verbose=False)