lsst.obs.base.gen2to3.repoWalker.handlers.MultiExtensionFileHandler Class Reference

Inheritance diagram for lsst.obs.base.gen2to3.repoWalker.handlers.MultiExtensionFileHandler:

Public Member Functions
def	handle (self, str path, nextDataId2, Mapping[DatasetType, Mapping[Optional[str], List[FileDataset]]] datasets, *Callable[[DataCoordinate], bool] predicate)
Tuple[Optional[DataCoordinate], Optional[str]]	translate (self, dict dataId2, *bool partial=False)
def	__str__ (self)
bool	isForFiles (self)
def	handle (self, str path, dict nextDataId2, Mapping[DatasetType, Mapping[Optional[str], List[FileDataset]]] datasets, *Callable[[DataCoordinate], bool] predicate)
bool	__call__ (self, str path, str name, Mapping[DatasetType, Mapping[Optional[str], List[FileDataset]]] datasets, *Callable[[DataCoordinate], bool] predicate)
int	rank (self)
def	__lt__ (self, PathElementHandler other)

Public Attributes
	lastDataId2

Detailed Description

Handler for FITS files that store image and metadata in multiple HDUs
per file, for example DECam raw and Community Pipeline calibrations.

Notes
-----
For now, this is only used by DECam, and may need to be made more generic
(e.g. making ``metadata['CCDNUM']`` use a configurable field) to be used
with other obs packages.

Definition at line 325 of file handlers.py.

Member Function Documentation

__call__()

bool lsst.obs.base.gen2to3.repoWalker.handlers.ParsedPathElementHandler.__call__ (   self, str  path, str  name, Mapping[DatasetType, Mapping[Optional[str], List[FileDataset]]]  datasets, *Callable[[DataCoordinate], bool]  predicate  )

inherited

Apply the handler to a file path.

Parameters
----------
path : `str`
    Full path of the file or directory.
name : `str`
    Local name of the file or directory within its parent directory.
datasets : `dict` [`DatasetType`, `dict` ]
    Dictionary that found datasets should be added to.  Nested dicts
    are keyed by either `None` (for most datasets) or a `str`
    "CALIBDATE" for calibration datasets.
predicate : `~collections.abc.Callable`
    A callable taking a single `DataCoordinate` argument and returning
    `bool`, indicating whether that (Gen3) data ID represents one
    that should be included in the scan.'

Returns
-------
matched : `bool`
    `True` if this handler was a match for the given path and no other
    handlers need to be tried on it, `False` otherwise.

Reimplemented from lsst.obs.base.gen2to3.repoWalker.scanner.PathElementHandler.

Definition at line 120 of file handlers.py.

                 predicate: Callable[[DataCoordinate], bool]) -> bool:
        # Docstring inherited from PathElementParser.
        nextDataId2 = self._parser.parse(name, self.lastDataId2)
        if nextDataId2 is None:
            return False
        self.handle(path, nextDataId2, datasets, predicate=predicate)
        return True
 

__lt__()

def lsst.obs.base.gen2to3.repoWalker.scanner.PathElementHandler.__lt__ (   self, PathElementHandler  other  )

inherited

Handlers are sorted by rank to reduce the possibility that more
flexible handlers will have a chance to match something they shouldn't.

Definition at line 140 of file scanner.py.

    def __lt__(self, other: PathElementHandler):
        """Handlers are sorted by rank to reduce the possibility that more
        flexible handlers will have a chance to match something they shouldn't.
        """
        return self.rank < other.rank
 

__str__()

def lsst.obs.base.gen2to3.repoWalker.handlers.TargetFileHandler.__str__ (   self )

inherited

Reimplemented from lsst.obs.base.gen2to3.repoWalker.handlers.ParsedPathElementHandler.

Definition at line 289 of file handlers.py.

    def __str__(self):
        return f"{type(self).__name__}({self._translator}, {self._datasetType})"
 

handle() [1/2]

def lsst.obs.base.gen2to3.repoWalker.handlers.ParsedPathElementHandler.handle (   self, str  path, dict  nextDataId2, Mapping[DatasetType, Mapping[Optional[str], List[FileDataset]]]  datasets, *Callable[[DataCoordinate], bool]  predicate  )

inherited

Customization hook for ``__call__``.

Subclasses must override this method, while external callers (i.e.
`DirectoryScanner` should instead invoke `__call__`.

Parameters
----------
path : `str`
    Full path of the file or directory.
nextDataId2 : `dict`
    Gen2 data ID (usually partial) extracted from the path so far.
datasets : `dict` [`DatasetType`, `list` [`FileDataset`] ]
    Dictionary that found datasets should be added to.
predicate : `~collections.abc.Callable`
    A callable taking a single `DataCoordinate` argument and returning
    `bool`, indicating whether that (Gen3) data ID represents one
    that should be included in the scan.
formatterMap : `dict`, optional
    Map dataset type to specialist formatter.

Reimplemented in lsst.obs.base.gen2to3.repoWalker.handlers.SkipHandler.

Definition at line 136 of file handlers.py.

               predicate: Callable[[DataCoordinate], bool]):
        """Customization hook for ``__call__``.
 
        Subclasses must override this method, while external callers (i.e.
        `DirectoryScanner` should instead invoke `__call__`.
 
        Parameters
        ----------
        path : `str`
            Full path of the file or directory.
        nextDataId2 : `dict`
            Gen2 data ID (usually partial) extracted from the path so far.
        datasets : `dict` [`DatasetType`, `list` [`FileDataset`] ]
            Dictionary that found datasets should be added to.
        predicate : `~collections.abc.Callable`
            A callable taking a single `DataCoordinate` argument and returning
            `bool`, indicating whether that (Gen3) data ID represents one
            that should be included in the scan.
        formatterMap : `dict`, optional
            Map dataset type to specialist formatter.
        """
        raise NotImplementedError()
 
 

handle() [2/2]

def lsst.obs.base.gen2to3.repoWalker.handlers.MultiExtensionFileHandler.handle	(		self,
		str	path,
			nextDataId2,
		Mapping[DatasetType, Mapping[Optional[str], List[FileDataset]]]	datasets,
		*Callable[[DataCoordinate], bool]	predicate
	)

Reimplemented from lsst.obs.base.gen2to3.repoWalker.handlers.TargetFileHandler.

Definition at line 335 of file handlers.py.

               predicate: Callable[[DataCoordinate], bool]):
        dataId3, calibDate = self.translate(nextDataId2, partial=True)
 
        def get_detectors(filename):
            fitsData = lsst.afw.fits.Fits(filename, 'r')
            # NOTE: The primary header (HDU=0) does not contain detector data.
            detectors = []
            for i in range(1, fitsData.countHdus()):
                fitsData.setHdu(i)
                metadata = fitsData.readMetadata()
                detectors.append(metadata['CCDNUM'])
            return detectors
 
        if predicate(dataId3):
            detectors = get_detectors(path)
            refs = []
            for detector in detectors:
                newDataId3 = DataCoordinate.standardize(dataId3,
                                                        graph=self._datasetType.dimensions,
                                                        detector=detector)
                refs.append(DatasetRef(self._datasetType, newDataId3))
 
            datasets[self._datasetType][calibDate].append(
                FileDataset(refs=refs, path=path, formatter=self._formatter)
            )
 

isForFiles()

bool lsst.obs.base.gen2to3.repoWalker.handlers.TargetFileHandler.isForFiles (   self )

inherited

Report what kind of path element this object handlers.

Returns
-------
Return `True` if this handler is for file entries, or `False` if it
is for directories.

Reimplemented from lsst.obs.base.gen2to3.repoWalker.scanner.PathElementHandler.

Definition at line 292 of file handlers.py.

    def isForFiles(self) -> bool:
        # Docstring inherited from PathElementHandler.
        return True
 

rank()

int lsst.obs.base.gen2to3.repoWalker.handlers.ParsedPathElementHandler.rank (   self )

inherited

Return a rough indication of how flexible this handler is in terms
of the path element names it can match.

Handlers that match a constant path element should always return zero.

Reimplemented from lsst.obs.base.gen2to3.repoWalker.scanner.PathElementHandler.

Definition at line 131 of file handlers.py.

    def rank(self) -> int:
        # Docstring inherited from PathElementParser.
        return len(self._parser.keys)
 

translate()

Tuple[Optional[DataCoordinate], Optional[str]] lsst.obs.base.gen2to3.repoWalker.handlers.MultiExtensionFileHandler.translate	(		self,
		dict	dataId2,
		*bool	partial = False
	)

Translate the given data ID from Gen2 to Gen3.

The default implementation returns `None`.  Subclasses that are able
to translate data IDs should override this method.

Parameters
----------
dataId2 : `dict`
    Gen2 data ID.
partial : `bool`, optional
    If `True` (`False` is default) this is a partial data ID for some
    dataset, and missing keys are expected.

Returns
-------
dataId3 : `lsst.daf.butler.DataCoordinate` or `None`
    A Gen3 data ID, or `None` if this handler cannot translate data
    IDs.
calibDate : `str` or `None`
    A Gen2 calibration "CALIBDATE" value, or `None` if there was no
    such value in the template.

Reimplemented from lsst.obs.base.gen2to3.repoWalker.handlers.TargetFileHandler.

Definition at line 363 of file handlers.py.

                  ) -> Tuple[Optional[DataCoordinate], Optional[str]]:
        assert partial is True, "We always require partial, to ignore 'ccdnum'"
        rawDataId3, calibDate = self._translator(dataId2, partial=partial)
        return (
            DataCoordinate.standardize(rawDataId3, universe=self._datasetType.dimensions.universe),
            calibDate,
        )

Member Data Documentation

lastDataId2

lsst.obs.base.gen2to3.repoWalker.scanner.PathElementHandler.lastDataId2

inherited

Definition at line 60 of file scanner.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/0.2.1/Linux64/obs_base/21.0.0-27-gbbd0d29+ae871e0f33/python/lsst/obs/base/gen2to3/repoWalker/handlers.py