Inheritance diagram for lsst.fgcmcal.fgcmCalibrateTractBase.FgcmCalibrateTractBaseTask:

Public Member Functions
def	__init__ (self, initInputs=None, butler=None, **kwargs)

def	runDataRef (self, butler, dataRefs)

def	run (self, dataRefDict, tract, buildStarsRefObjLoader=None, returnCatalogs=True, butler=None)

Detailed Description

Base class to calibrate a single tract using fgcmcal

Definition at line 174 of file fgcmCalibrateTractBase.py.

Constructor & Destructor Documentation

◆ init()

def lsst.fgcmcal.fgcmCalibrateTractBase.FgcmCalibrateTractBaseTask.__init__	(		self,
			initInputs = `None`,
			butler = `None`,
		**	kwargs
	)

Instantiate an `FgcmCalibrateTractTask`.

Parameters
----------
butler : `lsst.daf.persistence.Butler`, optional

Definition at line 177 of file fgcmCalibrateTractBase.py.

     def __init__(self, initInputs=None, butler=None, **kwargs):
         """
         Instantiate an `FgcmCalibrateTractTask`.
  
         Parameters
         ----------
         butler : `lsst.daf.persistence.Butler`, optional
         """
         super().__init__(**kwargs)
         self.makeSubtask("fgcmBuildStars", initInputs=initInputs, butler=butler)
         self.makeSubtask("fgcmOutputProducts", butler=butler)
  

Member Function Documentation

◆ run()

def lsst.fgcmcal.fgcmCalibrateTractBase.FgcmCalibrateTractBaseTask.run	(	self,
		dataRefDict,
		tract,
		buildStarsRefObjLoader = `None`,
		returnCatalogs = `True`,
		butler = `None`
	)

Run the calibrations for a single tract with fgcm.

Parameters
----------
dataRefDict : `dict`
    All dataRefs are `lsst.daf.persistence.ButlerDataRef` (gen2) or
    `lsst.daf.butler.DeferredDatasetHandle` (gen3)
    dataRef dictionary with the following keys.  Note that all
    keys need not be set based on config parameters.

    ``"camera"``
        Camera object (`lsst.afw.cameraGeom.Camera`)
    ``"source_catalogs"``
        `list` of dataRefs for input source catalogs.
    ``"sourceSchema"``
        Schema for the source catalogs.
    ``"fgcmLookUpTable"``
        dataRef for the FGCM look-up table.
    ``"calexps"``
        `list` of dataRefs for the input calexps (Gen3 only)
    ``"fgcmPhotoCalibs"``
        `dict` of output photoCalib dataRefs.  Key is
        (tract, visit, detector). (Gen3 only)
        Present if doZeropointOutput is True.
    ``"fgcmTransmissionAtmospheres"``
        `dict` of output atmosphere transmission dataRefs.
        Key is (tract, visit). (Gen3 only)
        Present if doAtmosphereOutput is True.
tract : `int`
    Tract number
buildStarsRefObjLoader : `lsst.meas.algorithms.ReferenceObjectLoader`, optional
    Reference object loader object for fgcmBuildStars.
returnCatalogs : `bool`, optional
    Return photoCalibs as per-visit exposure catalogs.
butler : `lsst.daf.persistence.Butler`, optional
    Gen2 butler used for reference star outputs

Returns
-------
outstruct : `lsst.pipe.base.Struct`
    Output structure with keys:

    offsets : `np.ndarray`
        Final reference offsets, per band.
    repeatability : `np.ndarray`
        Raw fgcm repeatability for bright stars, per band.
    atmospheres : `generator` [(`int`, `lsst.afw.image.TransmissionCurve`)]
        Generator that returns (visit, transmissionCurve) tuples.
    photoCalibs : `generator` [(`int`, `int`, `str`, `lsst.afw.image.PhotoCalib`)]
        Generator that returns (visit, ccd, filtername, photoCalib) tuples.
        (returned if returnCatalogs is False).
    photoCalibCatalogs : `generator` [(`int`, `lsst.afw.table.ExposureCatalog`)]
        Generator that returns (visit, exposureCatalog) tuples.
        (returned if returnCatalogs is True).

Definition at line 263 of file fgcmCalibrateTractBase.py.

             buildStarsRefObjLoader=None, returnCatalogs=True, butler=None):
         """Run the calibrations for a single tract with fgcm.
  
         Parameters
         ----------
         dataRefDict : `dict`
             All dataRefs are `lsst.daf.persistence.ButlerDataRef` (gen2) or
             `lsst.daf.butler.DeferredDatasetHandle` (gen3)
             dataRef dictionary with the following keys.  Note that all
             keys need not be set based on config parameters.
  
             ``"camera"``
                 Camera object (`lsst.afw.cameraGeom.Camera`)
             ``"source_catalogs"``
                 `list` of dataRefs for input source catalogs.
             ``"sourceSchema"``
                 Schema for the source catalogs.
             ``"fgcmLookUpTable"``
                 dataRef for the FGCM look-up table.
             ``"calexps"``
                 `list` of dataRefs for the input calexps (Gen3 only)
             ``"fgcmPhotoCalibs"``
                 `dict` of output photoCalib dataRefs.  Key is
                 (tract, visit, detector). (Gen3 only)
                 Present if doZeropointOutput is True.
             ``"fgcmTransmissionAtmospheres"``
                 `dict` of output atmosphere transmission dataRefs.
                 Key is (tract, visit). (Gen3 only)
                 Present if doAtmosphereOutput is True.
         tract : `int`
             Tract number
         buildStarsRefObjLoader : `lsst.meas.algorithms.ReferenceObjectLoader`, optional
             Reference object loader object for fgcmBuildStars.
         returnCatalogs : `bool`, optional
             Return photoCalibs as per-visit exposure catalogs.
         butler : `lsst.daf.persistence.Butler`, optional
             Gen2 butler used for reference star outputs
  
         Returns
         -------
         outstruct : `lsst.pipe.base.Struct`
             Output structure with keys:
  
             offsets : `np.ndarray`
                 Final reference offsets, per band.
             repeatability : `np.ndarray`
                 Raw fgcm repeatability for bright stars, per band.
             atmospheres : `generator` [(`int`, `lsst.afw.image.TransmissionCurve`)]
                 Generator that returns (visit, transmissionCurve) tuples.
             photoCalibs : `generator` [(`int`, `int`, `str`, `lsst.afw.image.PhotoCalib`)]
                 Generator that returns (visit, ccd, filtername, photoCalib) tuples.
                 (returned if returnCatalogs is False).
             photoCalibCatalogs : `generator` [(`int`, `lsst.afw.table.ExposureCatalog`)]
                 Generator that returns (visit, exposureCatalog) tuples.
                 (returned if returnCatalogs is True).
         """
         self.log.info("Running on tract %d", (tract))
  
         # Compute the aperture radius if necessary.  This is useful to do now before
         # any heavy lifting has happened (fail early).
         calibFluxApertureRadius = None
         if self.config.fgcmBuildStars.doSubtractLocalBackground:
             try:
                 field = self.config.fgcmBuildStars.instFluxField
                 calibFluxApertureRadius = computeApertureRadiusFromDataRef(dataRefDict['source_catalogs'][0],
                                                                            field)
             except RuntimeError:
                 raise RuntimeError("Could not determine aperture radius from %s. "
                                    "Cannot use doSubtractLocalBackground." %
                                    (field))
  
         # Run the build stars tasks
  
         # Note that we will need visitCat at the end of the procedure for the outputs
         if isinstance(butler, dafPersist.Butler):
             # Gen2
             groupedDataRefs = self.fgcmBuildStars._findAndGroupDataRefsGen2(butler, dataRefDict['camera'],
                                                                             dataRefDict['source_catalogs'])
         else:
             # Gen3
             groupedDataRefs = self.fgcmBuildStars._groupDataRefs(dataRefDict['sourceTableDataRefDict'],
                                                                  dataRefDict['visitSummaryDataRefDict'])
         visitCat = self.fgcmBuildStars.fgcmMakeVisitCatalog(dataRefDict['camera'], groupedDataRefs)
         rad = calibFluxApertureRadius
         fgcmStarObservationCat = self.fgcmBuildStars.fgcmMakeAllStarObservations(groupedDataRefs,
                                                                                  visitCat,
                                                                                  dataRefDict['sourceSchema'],
                                                                                  dataRefDict['camera'],
                                                                                  calibFluxApertureRadius=rad)
  
         if self.fgcmBuildStars.config.doReferenceMatches:
             lutDataRef = dataRefDict['fgcmLookUpTable']
             if buildStarsRefObjLoader is not None:
                 self.fgcmBuildStars.makeSubtask("fgcmLoadReferenceCatalog",
                                                 refObjLoader=buildStarsRefObjLoader)
             else:
                 self.fgcmBuildStars.makeSubtask("fgcmLoadReferenceCatalog", butler=butler)
         else:
             lutDataRef = None
  
         fgcmStarIdCat, fgcmStarIndicesCat, fgcmRefCat = \
             self.fgcmBuildStars.fgcmMatchStars(visitCat,
                                                fgcmStarObservationCat,
                                                lutDataRef=lutDataRef)
  
         # Load the LUT
         lutCat = dataRefDict['fgcmLookUpTable'].get()
         fgcmLut, lutIndexVals, lutStd = translateFgcmLut(lutCat,
                                                          dict(self.config.fgcmFitCycle.physicalFilterMap))
         del lutCat
  
         # Translate the visit catalog into fgcm format
         fgcmExpInfo = translateVisitCatalog(visitCat)
  
         configDict = makeConfigDict(self.config.fgcmFitCycle, self.log, dataRefDict['camera'],
                                     self.config.fgcmFitCycle.maxIterBeforeFinalCycle,
                                     True, False, lutIndexVals[0]['FILTERNAMES'],
                                     tract=tract)
  
         # Use the first orientation.
         # TODO: DM-21215 will generalize to arbitrary camera orientations
         ccdOffsets = computeCcdOffsets(dataRefDict['camera'], fgcmExpInfo['TELROT'][0])
  
         # Set up the fit cycle task
  
         noFitsDict = {'lutIndex': lutIndexVals,
                       'lutStd': lutStd,
                       'expInfo': fgcmExpInfo,
                       'ccdOffsets': ccdOffsets}
  
         fgcmFitCycle = fgcm.FgcmFitCycle(configDict, useFits=False,
                                          noFitsDict=noFitsDict, noOutput=True)
  
         # We determine the conversion from the native units (typically radians) to
         # degrees for the first star.  This allows us to treat coord_ra/coord_dec as
         # numpy arrays rather than Angles, which would we approximately 600x slower.
         conv = fgcmStarObservationCat[0]['ra'].asDegrees() / float(fgcmStarObservationCat[0]['ra'])
  
         # To load the stars, we need an initial parameter object
         fgcmPars = fgcm.FgcmParameters.newParsWithArrays(fgcmFitCycle.fgcmConfig,
                                                          fgcmLut,
                                                          fgcmExpInfo)
  
         # Match star observations to visits
         # Only those star observations that match visits from fgcmExpInfo['VISIT'] will
         # actually be transferred into fgcm using the indexing below.
  
         obsIndex = fgcmStarIndicesCat['obsIndex']
         visitIndex = np.searchsorted(fgcmExpInfo['VISIT'],
                                      fgcmStarObservationCat['visit'][obsIndex])
  
         refMag, refMagErr = extractReferenceMags(fgcmRefCat,
                                                  self.config.fgcmFitCycle.bands,
                                                  self.config.fgcmFitCycle.physicalFilterMap)
         refId = fgcmRefCat['fgcm_id'][:]
  
         fgcmStars = fgcm.FgcmStars(fgcmFitCycle.fgcmConfig)
         fgcmStars.loadStars(fgcmPars,
                             fgcmStarObservationCat['visit'][obsIndex],
                             fgcmStarObservationCat['ccd'][obsIndex],
                             fgcmStarObservationCat['ra'][obsIndex] * conv,
                             fgcmStarObservationCat['dec'][obsIndex] * conv,
                             fgcmStarObservationCat['instMag'][obsIndex],
                             fgcmStarObservationCat['instMagErr'][obsIndex],
                             fgcmExpInfo['FILTERNAME'][visitIndex],
                             fgcmStarIdCat['fgcm_id'][:],
                             fgcmStarIdCat['ra'][:],
                             fgcmStarIdCat['dec'][:],
                             fgcmStarIdCat['obsArrIndex'][:],
                             fgcmStarIdCat['nObs'][:],
                             obsX=fgcmStarObservationCat['x'][obsIndex],
                             obsY=fgcmStarObservationCat['y'][obsIndex],
                             obsDeltaMagBkg=fgcmStarObservationCat['deltaMagBkg'][obsIndex],
                             psfCandidate=fgcmStarObservationCat['psf_candidate'][obsIndex],
                             refID=refId,
                             refMag=refMag,
                             refMagErr=refMagErr,
                             flagID=None,
                             flagFlag=None,
                             computeNobs=True)
  
         # Clear out some memory
         del fgcmStarIdCat
         del fgcmStarIndicesCat
         del fgcmRefCat
  
         fgcmFitCycle.setLUT(fgcmLut)
         fgcmFitCycle.setStars(fgcmStars, fgcmPars)
  
         converged = False
         cycleNumber = 0
  
         previousReservedRawRepeatability = np.zeros(fgcmPars.nBands) + 1000.0
         previousParInfo = None
         previousParams = None
         previousSuperStar = None
  
         while (not converged and cycleNumber < self.config.maxFitCycles):
  
             fgcmFitCycle.fgcmConfig.updateCycleNumber(cycleNumber)
  
             if cycleNumber > 0:
                 # Use parameters from previous cycle
                 fgcmPars = fgcm.FgcmParameters.loadParsWithArrays(fgcmFitCycle.fgcmConfig,
                                                                   fgcmExpInfo,
                                                                   previousParInfo,
                                                                   previousParams,
                                                                   previousSuperStar)
                 # We need to reset the star magnitudes and errors for the next
                 # cycle
                 fgcmFitCycle.fgcmStars.reloadStarMagnitudes(fgcmStarObservationCat['instMag'][obsIndex],
                                                             fgcmStarObservationCat['instMagErr'][obsIndex])
                 fgcmFitCycle.initialCycle = False
  
             fgcmFitCycle.setPars(fgcmPars)
             fgcmFitCycle.finishSetup()
  
             fgcmFitCycle.run()
  
             # Grab the parameters for the next cycle
             previousParInfo, previousParams = fgcmFitCycle.fgcmPars.parsToArrays()
             previousSuperStar = fgcmFitCycle.fgcmPars.parSuperStarFlat.copy()
  
             self.log.info("Raw repeatability after cycle number %d is:" % (cycleNumber))
             for i, band in enumerate(fgcmFitCycle.fgcmPars.bands):
                 if not fgcmFitCycle.fgcmPars.hasExposuresInBand[i]:
                     continue
                 rep = fgcmFitCycle.fgcmPars.compReservedRawRepeatability[i] * 1000.0
                 self.log.info("  Band %s, repeatability: %.2f mmag" % (band, rep))
  
             # Check for convergence
             if np.alltrue((previousReservedRawRepeatability
                            - fgcmFitCycle.fgcmPars.compReservedRawRepeatability)
                           < self.config.convergenceTolerance):
                 self.log.info("Raw repeatability has converged after cycle number %d." % (cycleNumber))
                 converged = True
             else:
                 fgcmFitCycle.fgcmConfig.expGrayPhotometricCut[:] = fgcmFitCycle.updatedPhotometricCut
                 fgcmFitCycle.fgcmConfig.expGrayHighCut[:] = fgcmFitCycle.updatedHighCut
                 fgcmFitCycle.fgcmConfig.precomputeSuperStarInitialCycle = False
                 fgcmFitCycle.fgcmConfig.freezeStdAtmosphere = False
                 previousReservedRawRepeatability[:] = fgcmFitCycle.fgcmPars.compReservedRawRepeatability
                 self.log.info("Setting exposure gray photometricity cuts to:")
                 for i, band in enumerate(fgcmFitCycle.fgcmPars.bands):
                     if not fgcmFitCycle.fgcmPars.hasExposuresInBand[i]:
                         continue
                     cut = fgcmFitCycle.updatedPhotometricCut[i] * 1000.0
                     self.log.info("  Band %s, photometricity cut: %.2f mmag" % (band, cut))
  
             cycleNumber += 1
  
         # Log warning if not converged
         if not converged:
             self.log.warning("Maximum number of fit cycles exceeded (%d) without convergence.", cycleNumber)
  
         # Do final clean-up iteration
         fgcmFitCycle.fgcmConfig.freezeStdAtmosphere = False
         fgcmFitCycle.fgcmConfig.resetParameters = False
         fgcmFitCycle.fgcmConfig.maxIter = 0
         fgcmFitCycle.fgcmConfig.outputZeropoints = True
         fgcmFitCycle.fgcmConfig.outputStandards = True
         fgcmFitCycle.fgcmConfig.doPlots = self.config.doDebuggingPlots
         fgcmFitCycle.fgcmConfig.updateCycleNumber(cycleNumber)
         fgcmFitCycle.initialCycle = False
  
         fgcmPars = fgcm.FgcmParameters.loadParsWithArrays(fgcmFitCycle.fgcmConfig,
                                                           fgcmExpInfo,
                                                           previousParInfo,
                                                           previousParams,
                                                           previousSuperStar)
         fgcmFitCycle.fgcmStars.reloadStarMagnitudes(fgcmStarObservationCat['instMag'][obsIndex],
                                                     fgcmStarObservationCat['instMagErr'][obsIndex])
         fgcmFitCycle.setPars(fgcmPars)
         fgcmFitCycle.finishSetup()
  
         self.log.info("Running final clean-up fit cycle...")
         fgcmFitCycle.run()
  
         self.log.info("Raw repeatability after clean-up cycle is:")
         for i, band in enumerate(fgcmFitCycle.fgcmPars.bands):
             if not fgcmFitCycle.fgcmPars.hasExposuresInBand[i]:
                 continue
             rep = fgcmFitCycle.fgcmPars.compReservedRawRepeatability[i] * 1000.0
             self.log.info("  Band %s, repeatability: %.2f mmag" % (band, rep))
  
         # Do the outputs.  Need to keep track of tract.
  
         superStarChebSize = fgcmFitCycle.fgcmZpts.zpStruct['FGCM_FZPT_SSTAR_CHEB'].shape[1]
         zptChebSize = fgcmFitCycle.fgcmZpts.zpStruct['FGCM_FZPT_CHEB'].shape[1]
  
         zptSchema = makeZptSchema(superStarChebSize, zptChebSize)
         zptCat = makeZptCat(zptSchema, fgcmFitCycle.fgcmZpts.zpStruct)
  
         atmSchema = makeAtmSchema()
         atmCat = makeAtmCat(atmSchema, fgcmFitCycle.fgcmZpts.atmStruct)
  
         stdStruct, goodBands = fgcmFitCycle.fgcmStars.retrieveStdStarCatalog(fgcmFitCycle.fgcmPars)
         stdSchema = makeStdSchema(len(goodBands))
         stdCat = makeStdCat(stdSchema, stdStruct, goodBands)
  
         outStruct = self.fgcmOutputProducts.generateTractOutputProducts(dataRefDict,
                                                                         tract,
                                                                         visitCat,
                                                                         zptCat, atmCat, stdCat,
                                                                         self.config.fgcmBuildStars,
                                                                         returnCatalogs=returnCatalogs,
                                                                         butler=butler)
  
         outStruct.repeatability = fgcmFitCycle.fgcmPars.compReservedRawRepeatability
  
         fgcmFitCycle.freeSharedMemory()
  
         return outStruct

◆ runDataRef()

def lsst.fgcmcal.fgcmCalibrateTractBase.FgcmCalibrateTractBaseTask.runDataRef	(	self,
		butler,
		dataRefs
	)

Run full FGCM calibration on a single tract, including building star list,
fitting multiple cycles, and making outputs.

Parameters
----------
butler:  `lsst.daf.persistence.Butler`
dataRefs: `list` of `lsst.daf.persistence.ButlerDataRef`
   Data references for the input visits.
   These may be either per-ccd "src" or per-visit"sourceTable_visit"
   references.

Raises
------
RuntimeError: Raised if `config.fgcmBuildStars.doReferenceMatches` is
   not True, or if fgcmLookUpTable is not available, or if
   doSubtractLocalBackground is True and aperture radius cannot be
   determined.

Definition at line 194 of file fgcmCalibrateTractBase.py.

     def runDataRef(self, butler, dataRefs):
         """
         Run full FGCM calibration on a single tract, including building star list,
         fitting multiple cycles, and making outputs.
  
         Parameters
         ----------
         butler:  `lsst.daf.persistence.Butler`
         dataRefs: `list` of `lsst.daf.persistence.ButlerDataRef`
            Data references for the input visits.
            These may be either per-ccd "src" or per-visit"sourceTable_visit"
            references.
  
         Raises
         ------
         RuntimeError: Raised if `config.fgcmBuildStars.doReferenceMatches` is
            not True, or if fgcmLookUpTable is not available, or if
            doSubtractLocalBackground is True and aperture radius cannot be
            determined.
         """
         datasetType = dataRefs[0].butlerSubset.datasetType
         self.log.info("Running with %d %s dataRefs" % (len(dataRefs), datasetType))
  
         if not butler.datasetExists('fgcmLookUpTable'):
             raise RuntimeError("Must run FgcmCalibrateTract with an fgcmLookUpTable")
  
         if not self.config.fgcmBuildStars.doReferenceMatches:
             raise RuntimeError("Must run FgcmCalibrateTract with fgcmBuildStars.doReferenceMatches")
         if isinstance(self.config.fgcmBuildStars, FgcmBuildStarsConfig):
             if self.config.fgcmBuildStars.checkAllCcds:
                 raise RuntimeError("Cannot run FgcmCalibrateTract with "
                                    "fgcmBuildStars.checkAllCcds set to True")
  
         tract = int(dataRefs[0].dataId['tract'])
         camera = butler.get('camera')
  
         dataRefDict = {}
         dataRefDict['camera'] = camera
         dataRefDict['source_catalogs'] = dataRefs
         dataRefDict['sourceSchema'] = butler.get('src_schema', immediate=True).schema
         dataRefDict['fgcmLookUpTable'] = butler.dataRef('fgcmLookUpTable')
  
         struct = self.run(dataRefDict, tract, butler=butler, returnCatalogs=False)
  
         visitDataRefName = self.config.fgcmBuildStars.visitDataRefName
         ccdDataRefName = self.config.fgcmBuildStars.ccdDataRefName
  
         if struct.photoCalibs is not None:
             self.log.info("Outputting photoCalib files.")
  
             for visit, detector, physicalFilter, photoCalib in struct.photoCalibs:
                 butler.put(photoCalib, 'fgcm_tract_photoCalib',
                            dataId={visitDataRefName: visit,
                                    ccdDataRefName: detector,
                                    'filter': physicalFilter,
                                    'tract': tract})
  
             self.log.info("Done outputting photoCalib files.")
  
         if struct.atmospheres is not None:
             self.log.info("Outputting atmosphere files.")
             for visit, atm in struct.atmospheres:
                 butler.put(atm, "transmission_atmosphere_fgcm_tract",
                            dataId={visitDataRefName: visit,
                                    'tract': tract})
             self.log.info("Done outputting atmosphere transmissions.")
  
         return pipeBase.Struct(repeatability=struct.repeatability)
  

The documentation for this class was generated from the following file:

/j/snowflake/release/lsstsw/stack/lsst-scipipe-0.7.0/Linux64/fgcmcal/22.0.1-12-g9c3108e+df145f6f68/python/lsst/fgcmcal/fgcmCalibrateTractBase.py