lsst.fgcmcal.fgcmBuildStarsBase.FgcmBuildStarsBaseTask Class Reference

Inheritance diagram for lsst.fgcmcal.fgcmBuildStarsBase.FgcmBuildStarsBaseTask:

Public Member Functions
def	__init__ (self, initInputs=None, butler=None, **kwargs)
def	runDataRef (self, butler, dataRefs)
def	fgcmMakeAllStarObservations (self, groupedDataRefs, visitCat, sourceSchema, camera, calibFluxApertureRadius=None, visitCatDataRef=None, starObsDataRef=None, inStarObsCat=None)
def	fgcmMakeVisitCatalog (self, camera, groupedDataRefs, bkgDataRefDict=None, visitCatDataRef=None, inVisitCat=None)
def	fgcmMatchStars (self, visitCat, obsCat, lutDataRef=None)

Detailed Description

Base task to build stars for FGCM global calibration

Parameters
----------
butler : `lsst.daf.persistence.Butler`

Definition at line 275 of file fgcmBuildStarsBase.py.

Constructor & Destructor Documentation

__init__()

def lsst.fgcmcal.fgcmBuildStarsBase.FgcmBuildStarsBaseTask.__init__	(		self,
			initInputs = None,
			butler = None,
		**	kwargs
	)

Definition at line 283 of file fgcmBuildStarsBase.py.

    def __init__(self, initInputs=None, butler=None, **kwargs):
        super().__init__(**kwargs)
 
        self.makeSubtask("sourceSelector")
        # Only log warning and fatal errors from the sourceSelector
        self.sourceSelector.log.setLevel(self.sourceSelector.log.WARN)
 

Member Function Documentation

fgcmMakeAllStarObservations()

def lsst.fgcmcal.fgcmBuildStarsBase.FgcmBuildStarsBaseTask.fgcmMakeAllStarObservations	(		self,
			groupedDataRefs,
			visitCat,
			sourceSchema,
			camera,
			calibFluxApertureRadius = None,
			visitCatDataRef = None,
			starObsDataRef = None,
			inStarObsCat = None
	)

Compile all good star observations from visits in visitCat.  Checkpoint files
will be stored if both visitCatDataRef and starObsDataRef are not None.

Parameters
----------
groupedDataRefs : `dict` of `list`s
    Lists of `~lsst.daf.persistence.ButlerDataRef` or
    `~lsst.daf.butler.DeferredDatasetHandle`, grouped by visit.
visitCat : `~afw.table.BaseCatalog`
    Catalog with visit data for FGCM
sourceSchema : `~lsst.afw.table.Schema`
    Schema for the input src catalogs.
camera : `~lsst.afw.cameraGeom.Camera`
calibFluxApertureRadius : `float`, optional
    Aperture radius for calibration flux.
visitCatDataRef : `~lsst.daf.persistence.ButlerDataRef`, optional
    Dataref to write visitCat for checkpoints
starObsDataRef : `~lsst.daf.persistence.ButlerDataRef`, optional
    Dataref to write the star observation catalog for checkpoints.
inStarObsCat : `~afw.table.BaseCatalog`
    Input observation catalog.  If this is incomplete, observations
    will be appended from when it was cut off.

Returns
-------
fgcmStarObservations : `afw.table.BaseCatalog`
    Full catalog of good observations.

Raises
------
RuntimeError: Raised if doSubtractLocalBackground is True and
   calibFluxApertureRadius is not set.

Reimplemented in lsst.fgcmcal.fgcmBuildStarsTable.FgcmBuildStarsTableTask, and lsst.fgcmcal.fgcmBuildStars.FgcmBuildStarsTask.

Definition at line 416 of file fgcmBuildStarsBase.py.

                                    inStarObsCat=None):
        """
        Compile all good star observations from visits in visitCat.  Checkpoint files
        will be stored if both visitCatDataRef and starObsDataRef are not None.
 
        Parameters
        ----------
        groupedDataRefs : `dict` of `list`s
            Lists of `~lsst.daf.persistence.ButlerDataRef` or
            `~lsst.daf.butler.DeferredDatasetHandle`, grouped by visit.
        visitCat : `~afw.table.BaseCatalog`
            Catalog with visit data for FGCM
        sourceSchema : `~lsst.afw.table.Schema`
            Schema for the input src catalogs.
        camera : `~lsst.afw.cameraGeom.Camera`
        calibFluxApertureRadius : `float`, optional
            Aperture radius for calibration flux.
        visitCatDataRef : `~lsst.daf.persistence.ButlerDataRef`, optional
            Dataref to write visitCat for checkpoints
        starObsDataRef : `~lsst.daf.persistence.ButlerDataRef`, optional
            Dataref to write the star observation catalog for checkpoints.
        inStarObsCat : `~afw.table.BaseCatalog`
            Input observation catalog.  If this is incomplete, observations
            will be appended from when it was cut off.
 
        Returns
        -------
        fgcmStarObservations : `afw.table.BaseCatalog`
            Full catalog of good observations.
 
        Raises
        ------
        RuntimeError: Raised if doSubtractLocalBackground is True and
           calibFluxApertureRadius is not set.
        """
        raise NotImplementedError("fgcmMakeAllStarObservations not implemented.")
 

fgcmMakeVisitCatalog()

def lsst.fgcmcal.fgcmBuildStarsBase.FgcmBuildStarsBaseTask.fgcmMakeVisitCatalog	(		self,
			camera,
			groupedDataRefs,
			bkgDataRefDict = None,
			visitCatDataRef = None,
			inVisitCat = None
	)

Make a visit catalog with all the keys from each visit

Parameters
----------
camera: `lsst.afw.cameraGeom.Camera`
   Camera from the butler
groupedDataRefs: `dict`
   Dictionary with visit keys, and `list`s of
   `lsst.daf.persistence.ButlerDataRef`
bkgDataRefDict: `dict`, optional
   Dictionary of gen3 dataRefHandles for background info.
visitCatDataRef: `lsst.daf.persistence.ButlerDataRef`, optional
   Dataref to write visitCat for checkpoints
inVisitCat: `afw.table.BaseCatalog`, optional
   Input (possibly incomplete) visit catalog

Returns
-------
visitCat: `afw.table.BaseCatalog`

Definition at line 459 of file fgcmBuildStarsBase.py.

                             visitCatDataRef=None, inVisitCat=None):
        """
        Make a visit catalog with all the keys from each visit
 
        Parameters
        ----------
        camera: `lsst.afw.cameraGeom.Camera`
           Camera from the butler
        groupedDataRefs: `dict`
           Dictionary with visit keys, and `list`s of
           `lsst.daf.persistence.ButlerDataRef`
        bkgDataRefDict: `dict`, optional
           Dictionary of gen3 dataRefHandles for background info.
        visitCatDataRef: `lsst.daf.persistence.ButlerDataRef`, optional
           Dataref to write visitCat for checkpoints
        inVisitCat: `afw.table.BaseCatalog`, optional
           Input (possibly incomplete) visit catalog
 
        Returns
        -------
        visitCat: `afw.table.BaseCatalog`
        """
 
        self.log.info("Assembling visitCatalog from %d %ss" %
                      (len(groupedDataRefs), self.config.visitDataRefName))
 
        nCcd = len(camera)
 
        if inVisitCat is None:
            schema = self._makeFgcmVisitSchema(nCcd)
 
            visitCat = afwTable.BaseCatalog(schema)
            visitCat.reserve(len(groupedDataRefs))
            visitCat.resize(len(groupedDataRefs))
 
            visitCat['visit'] = list(groupedDataRefs.keys())
            visitCat['used'] = 0
            visitCat['sources_read'] = False
        else:
            visitCat = inVisitCat
 
        # No matter what, fill the catalog. This will check if it was
        # already read.
        self._fillVisitCatalog(visitCat, groupedDataRefs,
                               bkgDataRefDict=bkgDataRefDict,
                               visitCatDataRef=visitCatDataRef)
 
        return visitCat
 

fgcmMatchStars()

def lsst.fgcmcal.fgcmBuildStarsBase.FgcmBuildStarsBaseTask.fgcmMatchStars	(		self,
			visitCat,
			obsCat,
			lutDataRef = None
	)

Use FGCM code to match observations into unique stars.

Parameters
----------
visitCat: `afw.table.BaseCatalog`
   Catalog with visit data for fgcm
obsCat: `afw.table.BaseCatalog`
   Full catalog of star observations for fgcm
lutDataRef: `lsst.daf.persistence.ButlerDataRef` or
            `lsst.daf.butler.DeferredDatasetHandle`, optional
   Data reference to fgcm look-up table (used if matching reference stars).

Returns
-------
fgcmStarIdCat: `afw.table.BaseCatalog`
   Catalog of unique star identifiers and index keys
fgcmStarIndicesCat: `afwTable.BaseCatalog`
   Catalog of unique star indices
fgcmRefCat: `afw.table.BaseCatalog`
   Catalog of matched reference stars.
   Will be None if `config.doReferenceMatches` is False.

Definition at line 686 of file fgcmBuildStarsBase.py.

    def fgcmMatchStars(self, visitCat, obsCat, lutDataRef=None):
        """
        Use FGCM code to match observations into unique stars.
 
        Parameters
        ----------
        visitCat: `afw.table.BaseCatalog`
           Catalog with visit data for fgcm
        obsCat: `afw.table.BaseCatalog`
           Full catalog of star observations for fgcm
        lutDataRef: `lsst.daf.persistence.ButlerDataRef` or
                    `lsst.daf.butler.DeferredDatasetHandle`, optional
           Data reference to fgcm look-up table (used if matching reference stars).
 
        Returns
        -------
        fgcmStarIdCat: `afw.table.BaseCatalog`
           Catalog of unique star identifiers and index keys
        fgcmStarIndicesCat: `afwTable.BaseCatalog`
           Catalog of unique star indices
        fgcmRefCat: `afw.table.BaseCatalog`
           Catalog of matched reference stars.
           Will be None if `config.doReferenceMatches` is False.
        """
        # get filter names into a numpy array...
        # This is the type that is expected by the fgcm code
        visitFilterNames = np.zeros(len(visitCat), dtype='a30')
        for i in range(len(visitCat)):
            visitFilterNames[i] = visitCat[i]['physicalFilter']
 
        # match to put filterNames with observations
        visitIndex = np.searchsorted(visitCat['visit'],
                                     obsCat['visit'])
 
        obsFilterNames = visitFilterNames[visitIndex]
 
        if self.config.doReferenceMatches:
            # Get the reference filter names, using the LUT
            lutCat = lutDataRef.get()
 
            stdFilterDict = {filterName: stdFilter for (filterName, stdFilter) in
                             zip(lutCat[0]['physicalFilters'].split(','),
                                 lutCat[0]['stdPhysicalFilters'].split(','))}
            stdLambdaDict = {stdFilter: stdLambda for (stdFilter, stdLambda) in
                             zip(lutCat[0]['stdPhysicalFilters'].split(','),
                                 lutCat[0]['lambdaStdFilter'])}
 
            del lutCat
 
            referenceFilterNames = self._getReferenceFilterNames(visitCat,
                                                                 stdFilterDict,
                                                                 stdLambdaDict)
            self.log.info("Using the following reference filters: %s" %
                          (', '.join(referenceFilterNames)))
 
        else:
            # This should be an empty list
            referenceFilterNames = []
 
        # make the fgcm starConfig dict
        starConfig = {'logger': self.log,
                      'filterToBand': self.config.physicalFilterMap,
                      'requiredBands': self.config.requiredBands,
                      'minPerBand': self.config.minPerBand,
                      'matchRadius': self.config.matchRadius,
                      'isolationRadius': self.config.isolationRadius,
                      'matchNSide': self.config.matchNside,
                      'coarseNSide': self.config.coarseNside,
                      'densNSide': self.config.densityCutNside,
                      'densMaxPerPixel': self.config.densityCutMaxPerPixel,
                      'randomSeed': self.config.randomSeed,
                      'primaryBands': self.config.primaryBands,
                      'referenceFilterNames': referenceFilterNames}
 
        # initialize the FgcmMakeStars object
        fgcmMakeStars = fgcm.FgcmMakeStars(starConfig)
 
        # make the primary stars
        # note that the ra/dec native Angle format is radians
        # We determine the conversion from the native units (typically
        # radians) to degrees for the first observation.  This allows us
        # to treate ra/dec as numpy arrays rather than Angles, which would
        # be approximately 600x slower.
        conv = obsCat[0]['ra'].asDegrees() / float(obsCat[0]['ra'])
        fgcmMakeStars.makePrimaryStars(obsCat['ra'] * conv,
                                       obsCat['dec'] * conv,
                                       filterNameArray=obsFilterNames,
                                       bandSelected=False)
 
        # and match all the stars
        fgcmMakeStars.makeMatchedStars(obsCat['ra'] * conv,
                                       obsCat['dec'] * conv,
                                       obsFilterNames)
 
        if self.config.doReferenceMatches:
            fgcmMakeStars.makeReferenceMatches(self.fgcmLoadReferenceCatalog)
 
        # now persist
 
        objSchema = self._makeFgcmObjSchema()
 
        # make catalog and records
        fgcmStarIdCat = afwTable.BaseCatalog(objSchema)
        fgcmStarIdCat.reserve(fgcmMakeStars.objIndexCat.size)
        for i in range(fgcmMakeStars.objIndexCat.size):
            fgcmStarIdCat.addNew()
 
        # fill the catalog
        fgcmStarIdCat['fgcm_id'][:] = fgcmMakeStars.objIndexCat['fgcm_id']
        fgcmStarIdCat['ra'][:] = fgcmMakeStars.objIndexCat['ra']
        fgcmStarIdCat['dec'][:] = fgcmMakeStars.objIndexCat['dec']
        fgcmStarIdCat['obsArrIndex'][:] = fgcmMakeStars.objIndexCat['obsarrindex']
        fgcmStarIdCat['nObs'][:] = fgcmMakeStars.objIndexCat['nobs']
 
        obsSchema = self._makeFgcmObsSchema()
 
        fgcmStarIndicesCat = afwTable.BaseCatalog(obsSchema)
        fgcmStarIndicesCat.reserve(fgcmMakeStars.obsIndexCat.size)
        for i in range(fgcmMakeStars.obsIndexCat.size):
            fgcmStarIndicesCat.addNew()
 
        fgcmStarIndicesCat['obsIndex'][:] = fgcmMakeStars.obsIndexCat['obsindex']
 
        if self.config.doReferenceMatches:
            refSchema = self._makeFgcmRefSchema(len(referenceFilterNames))
 
            fgcmRefCat = afwTable.BaseCatalog(refSchema)
            fgcmRefCat.reserve(fgcmMakeStars.referenceCat.size)
 
            for i in range(fgcmMakeStars.referenceCat.size):
                fgcmRefCat.addNew()
 
            fgcmRefCat['fgcm_id'][:] = fgcmMakeStars.referenceCat['fgcm_id']
            fgcmRefCat['refMag'][:, :] = fgcmMakeStars.referenceCat['refMag']
            fgcmRefCat['refMagErr'][:, :] = fgcmMakeStars.referenceCat['refMagErr']
 
            md = PropertyList()
            md.set("REFSTARS_FORMAT_VERSION", REFSTARS_FORMAT_VERSION)
            md.set("FILTERNAMES", referenceFilterNames)
            fgcmRefCat.setMetadata(md)
 
        else:
            fgcmRefCat = None
 
        return fgcmStarIdCat, fgcmStarIndicesCat, fgcmRefCat
 

runDataRef()

def lsst.fgcmcal.fgcmBuildStarsBase.FgcmBuildStarsBaseTask.runDataRef	(		self,
			butler,
			dataRefs
	)

Cross-match and make star list for FGCM Input

Parameters
----------
butler:  `lsst.daf.persistence.Butler`
dataRefs: `list` of `lsst.daf.persistence.ButlerDataRef`
   Source data references for the input visits.

Raises
------
RuntimeErrror: Raised if `config.doReferenceMatches` is set and
   an fgcmLookUpTable is not available, or if computeFluxApertureRadius()
   fails if the calibFlux is not a CircularAperture flux.

Definition at line 295 of file fgcmBuildStarsBase.py.

    def runDataRef(self, butler, dataRefs):
        """
        Cross-match and make star list for FGCM Input
 
        Parameters
        ----------
        butler:  `lsst.daf.persistence.Butler`
        dataRefs: `list` of `lsst.daf.persistence.ButlerDataRef`
           Source data references for the input visits.
 
        Raises
        ------
        RuntimeErrror: Raised if `config.doReferenceMatches` is set and
           an fgcmLookUpTable is not available, or if computeFluxApertureRadius()
           fails if the calibFlux is not a CircularAperture flux.
        """
        datasetType = dataRefs[0].butlerSubset.datasetType
        self.log.info("Running with %d %s dataRefs", len(dataRefs), datasetType)
 
        if self.config.doReferenceMatches:
            self.makeSubtask("fgcmLoadReferenceCatalog", butler=butler)
            # Ensure that we have a LUT
            if not butler.datasetExists('fgcmLookUpTable'):
                raise RuntimeError("Must have fgcmLookUpTable if using config.doReferenceMatches")
        # Compute aperture radius if necessary.  This is useful to do now before
        # any heavy lifting has happened (fail early).
        calibFluxApertureRadius = None
        if self.config.doSubtractLocalBackground:
            try:
                calibFluxApertureRadius = computeApertureRadiusFromDataRef(dataRefs[0],
                                                                           self.config.instFluxField)
            except RuntimeError as e:
                raise RuntimeError("Could not determine aperture radius from %s. "
                                   "Cannot use doSubtractLocalBackground." %
                                   (self.config.instFluxField)) from e
 
        camera = butler.get('camera')
        groupedDataRefs = self._findAndGroupDataRefsGen2(butler, camera, dataRefs)
 
        # Make the visit catalog if necessary
        # First check if the visit catalog is in the _current_ path
        # We cannot use Gen2 datasetExists() because that checks all parent
        # directories as well, which would make recovering from faults
        # and fgcmcal reruns impossible.
        visitCatDataRef = butler.dataRef('fgcmVisitCatalog')
        filename = visitCatDataRef.get('fgcmVisitCatalog_filename')[0]
        if os.path.exists(filename):
            # This file exists and we should continue processing
            inVisitCat = visitCatDataRef.get()
            if len(inVisitCat) != len(groupedDataRefs):
                raise RuntimeError("Existing visitCatalog found, but has an inconsistent "
                                   "number of visits.  Cannot continue.")
        else:
            inVisitCat = None
 
        visitCat = self.fgcmMakeVisitCatalog(camera, groupedDataRefs,
                                             visitCatDataRef=visitCatDataRef,
                                             inVisitCat=inVisitCat)
 
        # Persist the visitCat as a checkpoint file.
        visitCatDataRef.put(visitCat)
 
        starObsDataRef = butler.dataRef('fgcmStarObservations')
        filename = starObsDataRef.get('fgcmStarObservations_filename')[0]
        if os.path.exists(filename):
            inStarObsCat = starObsDataRef.get()
        else:
            inStarObsCat = None
 
        rad = calibFluxApertureRadius
        sourceSchemaDataRef = butler.dataRef('src_schema')
        sourceSchema = sourceSchemaDataRef.get('src_schema', immediate=True).schema
        fgcmStarObservationCat = self.fgcmMakeAllStarObservations(groupedDataRefs,
                                                                  visitCat,
                                                                  sourceSchema,
                                                                  camera,
                                                                  calibFluxApertureRadius=rad,
                                                                  starObsDataRef=starObsDataRef,
                                                                  visitCatDataRef=visitCatDataRef,
                                                                  inStarObsCat=inStarObsCat)
        visitCatDataRef.put(visitCat)
        starObsDataRef.put(fgcmStarObservationCat)
 
        # Always do the matching.
        if self.config.doReferenceMatches:
            lutDataRef = butler.dataRef('fgcmLookUpTable')
        else:
            lutDataRef = None
        fgcmStarIdCat, fgcmStarIndicesCat, fgcmRefCat = self.fgcmMatchStars(visitCat,
                                                                            fgcmStarObservationCat,
                                                                            lutDataRef=lutDataRef)
 
        # Persist catalogs via the butler
        butler.put(fgcmStarIdCat, 'fgcmStarIds')
        butler.put(fgcmStarIndicesCat, 'fgcmStarIndices')
        if fgcmRefCat is not None:
            butler.put(fgcmRefCat, 'fgcmReferenceStars')
 

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-0.7.0/Linux64/fgcmcal/22.0.1-12-g9c3108e+df145f6f68/python/lsst/fgcmcal/fgcmBuildStarsBase.py