lsst.obs.base.gen2to3.repoWalker.walker.RepoWalker Class Reference

Public Member Functions
def	__init__ (self, Iterable[Union[Target, Skip]] inputs, *Optional[re.Pattern] fileIgnoreRegEx=None, Optional[re.Pattern] dirIgnoreRegEx=None, Optional[Log] log=None)
Mapping[DatasetType, Mapping[Optional[str], List[FileDataset]]]	walk (self, str root, *Optional[Callable[[DataCoordinate], bool]] predicate)

Public Attributes
	log

Detailed Description

An object that recursively walks a Gen2 data repository tree, extracting
Gen3 `FileDataset` objects and warning about unrecognized or unconvertable
Gen2 datasets.

Parameters
----------
inputs : `~collections.abc.Iterable` of `Target` or `Skip`
    Structs that indicate dataset types to be extracted (`Target`) or
    explicitly skipped (`Skip`).  Skips may include a warning message to
    log when matching entries are encountered.
fileIgnoreRegEx : `re.Pattern`, optional
    A regular expression pattern that identifies non-dataset files that
    can be ignored, to be applied at all levels of the directory tree.
dirIgnoreRegEx : `re.Pattern`, optional
    A regular expression pattern that identifies non-dataset subdirectories
    that can be ignored, to be applied at all levels of the directory tree.
log : `Log`, optional
        Logger for warnings and diagnostic information.

Definition at line 51 of file walker.py.

Constructor & Destructor Documentation

__init__()

def lsst.obs.base.gen2to3.repoWalker.walker.RepoWalker.__init__	(		self,
		Iterable[Union[Target, Skip]]	inputs,
		*Optional[re.Pattern]	fileIgnoreRegEx = None,
		Optional[re.Pattern]	dirIgnoreRegEx = None,
		Optional[Log]	log = None
	)

Definition at line 71 of file walker.py.

                 log: Optional[Log] = None):
        super().__init__()
        if log is None:
            log = Log.getLogger("obs.base.gen2to3.TranslatorFactory")
        self.log = log
        tree = BuilderTree()
        allKeys: Dict[str, type] = {}
        for leaf in inputs:
            tree.insert(0, leaf)
            for key, dtype in leaf.keys.items():
                if allKeys.setdefault(key, dtype) != dtype:
                    raise ValueError(f"Multiple types for key '{key}': {dtype} "
                                     f"(from {leaf.template}) vs. {allKeys[key]}.")
        tree, messages, pruned = tree.prune()
        if not pruned:
            self._scanner = DirectoryScanner(log=self.log)
            tree.fill(self._scanner, allKeys, {}, fileIgnoreRegEx=fileIgnoreRegEx,
                      dirIgnoreRegEx=dirIgnoreRegEx)
        else:
            # Nothing to do; just remember this for later to avoid disturbing
            # higher-level code with the fact that walk() will be a no-op.
            self._scanner = None
 

Member Function Documentation

walk()

Mapping[DatasetType, Mapping[Optional[str], List[FileDataset]]] lsst.obs.base.gen2to3.repoWalker.walker.RepoWalker.walk	(		self,
		str	root,
		*Optional[Callable[[DataCoordinate], bool]]	predicate
	)

Walk a Gen2 repository root to extract Gen3 `FileDataset` instances
from it.

Parameters
----------
root : `str`
    Absolute path to the repository root.
predicate : `~collections.abc.Callable`, optional
    If not `None`, a callable that returns `True` if a `DataCoordinate`
    is consistent with what we want to extract.  If ``predicate``
    returns `False`, the file or directory that data ID was extracted
    from will not be processed, even if it includes target dataset
    types.

Returns
-------
datasets : `defaultdict` [`DatasetType`, `defaultdict` ]
    Extracted datasets, grouped by Gen3 `DatasetType`.  Nested dict
    keys are "CALIBDATE" strings (for calibration datasets) or `None`
    (otherwise).  Nested dict values are lists of `FileDataset`.

Definition at line 107 of file walker.py.

             ) -> Mapping[DatasetType, Mapping[Optional[str], List[FileDataset]]]:
        """Walk a Gen2 repository root to extract Gen3 `FileDataset` instances
        from it.
 
        Parameters
        ----------
        root : `str`
            Absolute path to the repository root.
        predicate : `~collections.abc.Callable`, optional
            If not `None`, a callable that returns `True` if a `DataCoordinate`
            is consistent with what we want to extract.  If ``predicate``
            returns `False`, the file or directory that data ID was extracted
            from will not be processed, even if it includes target dataset
            types.
 
        Returns
        -------
        datasets : `defaultdict` [`DatasetType`, `defaultdict` ]
            Extracted datasets, grouped by Gen3 `DatasetType`.  Nested dict
            keys are "CALIBDATE" strings (for calibration datasets) or `None`
            (otherwise).  Nested dict values are lists of `FileDataset`.
        """
        if predicate is None:
            def predicate(dataId: DataCoordinate) -> bool:
                return True
        datasets = defaultdict(lambda: defaultdict(list))
        if self._scanner is not None:
            self._scanner.scan(root, datasets, predicate=predicate)
        return datasets

Member Data Documentation

log

lsst.obs.base.gen2to3.repoWalker.walker.RepoWalker.log

Definition at line 78 of file walker.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/0.2.1/Linux64/obs_base/21.0.0-27-gbbd0d29+ae871e0f33/python/lsst/obs/base/gen2to3/repoWalker/walker.py