Inheritance diagram for lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter:

Public Member Functions
def	__init__ (self, **kwds)

bool	isDatasetTypeSpecial (self, str datasetTypeName)

List[str]	getSpecialDirectories (self)

Tuple[Optional[BaseSkyMap], Optional[str]]	findMatchingSkyMap (self, str datasetTypeName)

def	runRawIngest (self, pool=None)

def	runDefineVisits (self, pool=None)

def	prep (self)

Iterator[FileDataset]	iterDatasets (self)

str	getRun (self, str datasetTypeName, Optional[str] calibDate=None)

Iterator[Tuple[str, CameraMapperMapping]]	iterMappings (self)

RepoWalker.Target	makeRepoWalkerTarget (self, str datasetTypeName, str template, Dict[str, type] keys, StorageClass storageClass, FormatterParameter formatter=None, Optional[PathElementHandler] targetHandler=None)

List[str]	getCollectionChain (self)

def	findDatasets (self)

def	expandDataIds (self)

def	ingest (self)

None	finish (self)

Public Attributes
	butler2

	mapper

	task

	root

	instrument

	subset

Detailed Description

A specialization of `RepoConverter` for root data repositories.

`RootRepoConverter` adds support for raw images (mostly delegated to the
parent task's `RawIngestTask` subtask) and reference catalogs.

Parameters
----------
kwds
    Keyword arguments are forwarded to (and required by) `RepoConverter`.

Definition at line 63 of file rootRepoConverter.py.

Constructor & Destructor Documentation

◆ init()

def lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter.__init__	(		self,
		**	kwds
	)

Reimplemented from lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.

Definition at line 75 of file rootRepoConverter.py.

     def __init__(self, **kwds):
         super().__init__(run=None, **kwds)
         self._refCats: Dict[str, SkyPixDimension] = {}
         if self.task.config.rootSkyMapName is not None:
             self._rootSkyMap = self.task.config.skyMaps[self.task.config.rootSkyMapName].skyMap.apply()
         else:
             self._rootSkyMap = None  # All access to _rootSkyMap is guarded
         self._rawRefs = []
  

Member Function Documentation

◆ expandDataIds()

def lsst.obs.base.gen2to3.repoConverter.RepoConverter.expandDataIds ( self )

inherited

Expand the data IDs for all datasets to be inserted.

Subclasses may override this method, but must delegate to the base
class implementation if they do.

This involves queries to the registry, but not writes.  It is
guaranteed to be called between `findDatasets` and `ingest`.

Definition at line 441 of file repoConverter.py.

     def expandDataIds(self):
         """Expand the data IDs for all datasets to be inserted.
  
         Subclasses may override this method, but must delegate to the base
         class implementation if they do.
  
         This involves queries to the registry, but not writes.  It is
         guaranteed to be called between `findDatasets` and `ingest`.
         """
         import itertools
         for datasetType, datasetsByCalibDate in self._fileDatasets.items():
             for calibDate, datasetsForCalibDate in datasetsByCalibDate.items():
                 nDatasets = len(datasetsForCalibDate)
                 suffix = "" if nDatasets == 1 else "s"
                 if calibDate is not None:
                     self.task.log.info("Expanding data IDs for %s %s dataset%s at calibDate %s.",
                                        nDatasets,
                                        datasetType.name,
                                        suffix,
                                        calibDate)
                 else:
                     self.task.log.info("Expanding data IDs for %s %s non-calibration dataset%s.",
                                        nDatasets,
                                        datasetType.name,
                                        suffix)
                 expanded = []
                 for dataset in datasetsForCalibDate:
                     for i, ref in enumerate(dataset.refs):
                         self.task.log.debug("Expanding data ID %s.", ref.dataId)
                         try:
                             dataId = self.task.registry.expandDataId(ref.dataId)
                             dataset.refs[i] = ref.expanded(dataId)
                         except LookupError as err:
                             self.task.log.warn("Skipping ingestion for '%s': %s", dataset.path, err)
                             # Remove skipped datasets from multi-extension
                             # FileDatasets
                             dataset.refs[i] = None  # We will strip off the `None`s after the loop.
                     dataset.refs[:] = itertools.filterfalse(lambda x: x is None, dataset.refs)
                     if dataset.refs:
                         expanded.append(dataset)
                 datasetsForCalibDate[:] = expanded
  

◆ findDatasets()

def lsst.obs.base.gen2to3.repoConverter.RepoConverter.findDatasets ( self )

inherited

Definition at line 424 of file repoConverter.py.

     def findDatasets(self):
         assert self._repoWalker, "prep() must be called before findDatasets."
         self.task.log.info("Adding special datasets in repo %s.", self.root)
         for dataset in self.iterDatasets():
             assert len(dataset.refs) == 1
             # None index below is for calibDate, which is only relevant for
             # CalibRepoConverter.
             self._fileDatasets[dataset.refs[0].datasetType][None].append(dataset)
         self.task.log.info("Finding datasets from files in repo %s.", self.root)
         datasetsByTypeAndCalibDate = self._repoWalker.walk(
             self.root,
             predicate=(self.subset.isRelated if self.subset is not None else None)
         )
         for datasetType, datasetsByCalibDate in datasetsByTypeAndCalibDate.items():
             for calibDate, datasets in datasetsByCalibDate.items():
                 self._fileDatasets[datasetType][calibDate].extend(datasets)
  

◆ findMatchingSkyMap()

Tuple[Optional[BaseSkyMap], Optional[str]] lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter.findMatchingSkyMap	(		self,
		str	datasetTypeName
	)

Return the appropriate SkyMap for the given dataset type.

Parameters
----------
datasetTypeName : `str`
    Name of the dataset type for which a skymap is sought.

Returns
-------
skyMap : `BaseSkyMap` or `None`
    The `BaseSkyMap` instance, or `None` if there was no match.
skyMapName : `str` or `None`
    The Gen3 name for the SkyMap, or `None` if there was no match.

Reimplemented from lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.

Definition at line 97 of file rootRepoConverter.py.

     def findMatchingSkyMap(self, datasetTypeName: str) -> Tuple[Optional[BaseSkyMap], Optional[str]]:
         # Docstring inherited from StandardRepoConverter.findMatchingSkyMap.
         skyMap, name = super().findMatchingSkyMap(datasetTypeName)
         if skyMap is None and self.task.config.rootSkyMapName is not None:
             self.task.log.debug(
                 ("Assuming configured root skymap with name '%s' for dataset %s."),
                 self.task.config.rootSkyMapName, datasetTypeName
             )
             skyMap = self._rootSkyMap
             name = self.task.config.rootSkyMapName
         return skyMap, name
  

◆ finish()

None lsst.obs.base.gen2to3.repoConverter.RepoConverter.finish ( self )

inherited

Finish conversion of a repository.

This is run after ``ingest``, and delegates to `_finish`, which should
be overridden by derived classes instead of this method.

Definition at line 511 of file repoConverter.py.

     def finish(self) -> None:
         """Finish conversion of a repository.
  
         This is run after ``ingest``, and delegates to `_finish`, which should
         be overridden by derived classes instead of this method.
         """
         self._finish(self._fileDatasets)
  

◆ getCollectionChain()

List[str] lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.getCollectionChain ( self )

inherited

Return run names that can be used to construct a chained collection
that refers to the converted repository (`list` [ `str` ]).

Definition at line 207 of file standardRepoConverter.py.

     def getCollectionChain(self) -> List[str]:
         """Return run names that can be used to construct a chained collection
         that refers to the converted repository (`list` [ `str` ]).
         """
         return self._chain
  

◆ getRun()

str lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter.getRun	(		self,
		str	datasetTypeName,
		Optional[str]	calibDate = `None`
	)

Return the name of the run to insert instances of the given dataset
type into in this collection.

Parameters
----------
datasetTypeName : `str`
    Name of the dataset type.
calibDate : `str`, optional
    If not `None`, the "CALIBDATE" associated with this (calibration)
    dataset in the Gen2 data repository.

Returns
-------
run : `str`
    Name of the `~lsst.daf.butler.CollectionType.RUN` collection.

Reimplemented from lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.

Definition at line 186 of file rootRepoConverter.py.

     def getRun(self, datasetTypeName: str, calibDate: Optional[str] = None) -> str:
         # Docstring inherited from RepoConverter.
         if datasetTypeName in self._refCats:
             return self.instrument.makeRefCatCollectionName("gen2")
         return super().getRun(datasetTypeName, calibDate)
  

◆ getSpecialDirectories()

List[str] lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter.getSpecialDirectories ( self )

Return a list of directory paths that should not be searched for
files.

These may be directories that simply do not contain datasets (or
contain datasets in another repository), or directories whose datasets
are handled specially by a subclass.

Returns
-------
directories : `list` [`str`]
    The full paths of directories to skip, relative to the repository
    root.

Reimplemented from lsst.obs.base.gen2to3.repoConverter.RepoConverter.

Definition at line 93 of file rootRepoConverter.py.

     def getSpecialDirectories(self) -> List[str]:
         # Docstring inherited from RepoConverter.
         return super().getSpecialDirectories() + ["CALIB", "ref_cats", "rerun"]
  

◆ ingest()

def lsst.obs.base.gen2to3.repoConverter.RepoConverter.ingest ( self )

inherited

Insert converted datasets into the Gen3 repository.

Subclasses may override this method, but must delegate to the base
class implementation at some point in their own logic.

This method is guaranteed to be called after `expandDataIds`.

Definition at line 483 of file repoConverter.py.

     def ingest(self):
         """Insert converted datasets into the Gen3 repository.
  
         Subclasses may override this method, but must delegate to the base
         class implementation at some point in their own logic.
  
         This method is guaranteed to be called after `expandDataIds`.
         """
         for datasetType, datasetsByCalibDate in self._fileDatasets.items():
             self.task.registry.registerDatasetType(datasetType)
             for calibDate, datasetsForCalibDate in datasetsByCalibDate.items():
                 try:
                     run = self.getRun(datasetType.name, calibDate)
                 except LookupError:
                     self.task.log.warn(f"No run configured for dataset type {datasetType.name}.")
                     continue
                 nDatasets = len(datasetsForCalibDate)
                 self.task.log.info("Ingesting %s %s dataset%s into run %s.", nDatasets,
                                    datasetType.name, "" if nDatasets == 1 else "s", run)
                 try:
                     self.task.registry.registerRun(run)
                     self.task.butler3.ingest(*datasetsForCalibDate, transfer=self.task.config.transfer,
                                              run=run)
                 except LookupError as err:
                     raise LookupError(
                         f"Error expanding data ID for dataset type {datasetType.name}."
                     ) from err
  

◆ isDatasetTypeSpecial()

bool lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter.isDatasetTypeSpecial	(		self,
		str	datasetTypeName
	)

Test whether the given dataset is handled specially by this
converter and hence should be ignored by generic base-class logic that
searches for dataset types to convert.

Parameters
----------
datasetTypeName : `str`
    Name of the dataset type to test.

Returns
-------
special : `bool`
    `True` if the dataset type is special.

Reimplemented from lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.

Definition at line 84 of file rootRepoConverter.py.

     def isDatasetTypeSpecial(self, datasetTypeName: str) -> bool:
         # Docstring inherited from RepoConverter.
         return (
             super().isDatasetTypeSpecial(datasetTypeName)
             or datasetTypeName in ("raw", "ref_cat", "ref_cat_config")
             # in Gen2, some of these are in the root repo, not a calib repo
             or datasetTypeName in self.instrument.getCuratedCalibrationNames()
         )
  

◆ iterDatasets()

Iterator[FileDataset] lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter.iterDatasets ( self )

Iterate over datasets in the repository that should be ingested into
the Gen3 repository.

The base class implementation yields nothing; the datasets handled by
the `RepoConverter` base class itself are read directly in
`findDatasets`.

Subclasses should override this method if they support additional
datasets that are handled some other way.

Yields
------
dataset : `FileDataset`
    Structures representing datasets to be ingested.  Paths should be
    absolute.

Reimplemented from lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.

Definition at line 163 of file rootRepoConverter.py.

     def iterDatasets(self) -> Iterator[FileDataset]:
         # Docstring inherited from RepoConverter.
         # Iterate over reference catalog files.
         for refCat, dimension in self._refCats.items():
             datasetType = DatasetType(refCat, dimensions=[dimension], universe=self.task.universe,
                                       storageClass="SimpleCatalog")
             if self.subset is None:
                 regex = re.compile(r"(\d+)\.fits")
                 for fileName in os.listdir(os.path.join(self.root, "ref_cats", refCat)):
                     m = regex.match(fileName)
                     if m is not None:
                         htmId = int(m.group(1))
                         dataId = self.task.registry.expandDataId({dimension: htmId})
                         yield FileDataset(path=os.path.join(self.root, "ref_cats", refCat, fileName),
                                           refs=DatasetRef(datasetType, dataId))
             else:
                 for begin, end in self.subset.skypix[dimension]:
                     for htmId in range(begin, end):
                         dataId = self.task.registry.expandDataId({dimension: htmId})
                         yield FileDataset(path=os.path.join(self.root, "ref_cats", refCat, f"{htmId}.fits"),
                                           refs=DatasetRef(datasetType, dataId))
         yield from super().iterDatasets()
  

◆ iterMappings()

Iterator[Tuple[str, CameraMapperMapping]] lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.iterMappings ( self )

inherited

Iterate over all `CameraMapper` `Mapping` objects that should be
considered for conversion by this repository.

This this should include any datasets that may appear in the
repository, including those that are special (see
`isDatasetTypeSpecial`) and those that are being ignored (see
`ConvertRepoTask.isDatasetTypeIncluded`); this allows the converter
to identify and hence skip these datasets quietly instead of warning
about them as unrecognized.

Yields
------
datasetTypeName: `str`
    Name of the dataset type.
mapping : `lsst.obs.base.mapping.Mapping`
    Mapping object used by the Gen2 `CameraMapper` to describe the
    dataset type.

Reimplemented from lsst.obs.base.gen2to3.repoConverter.RepoConverter.

Definition at line 124 of file standardRepoConverter.py.

     def iterMappings(self) -> Iterator[Tuple[str, CameraMapperMapping]]:
         # Docstring inherited from RepoConverter.
         for datasetTypeName, mapping in self.mapper.mappings.items():
             if datasetTypeName not in self.mapper.calibrations:
                 yield datasetTypeName, mapping
  

◆ makeRepoWalkerTarget()

RepoWalker.Target lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.makeRepoWalkerTarget	(		self,
		str	datasetTypeName,
		str	template,
		Dict[str, type]	keys,
		StorageClass	storageClass,
		FormatterParameter	formatter = `None`,
		Optional[PathElementHandler]	targetHandler = `None`
	)

inherited

Make a struct that identifies a dataset type to be extracted by
walking the repo directory structure.

Parameters
----------
datasetTypeName : `str`
    Name of the dataset type (the same in both Gen2 and Gen3).
template : `str`
    The full Gen2 filename template.
keys : `dict` [`str`, `type`]
    A dictionary mapping Gen2 data ID key to the type of its value.
storageClass : `lsst.daf.butler.StorageClass`
    Gen3 storage class for this dataset type.
formatter : `lsst.daf.butler.Formatter` or `str`, optional
    A Gen 3 formatter class or fully-qualified name.
targetHandler : `PathElementHandler`, optional
    Specialist target handler to use for this dataset type.

Returns
-------
target : `RepoWalker.Target`
    A struct containing information about the target dataset (much of
    it simplify forwarded from the arguments).

Reimplemented from lsst.obs.base.gen2to3.repoConverter.RepoConverter.

Definition at line 167 of file standardRepoConverter.py.

                              ) -> RepoWalker.Target:
         # Docstring inherited from RepoConverter.
         skyMap, skyMapName = self.findMatchingSkyMap(datasetTypeName)
         return RepoWalker.Target(
             datasetTypeName=datasetTypeName,
             storageClass=storageClass,
             template=template,
             keys=keys,
             universe=self.task.registry.dimensions,
             instrument=self.task.instrument.getName(),
             skyMap=skyMap,
             skyMapName=skyMapName,
             formatter=formatter,
             targetHandler=targetHandler,
             translatorFactory=self.task.translatorFactory,
         )
  

◆ prep()

def lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter.prep ( self )

Perform preparatory work associated with the dataset types to be
converted from this repository (but not the datasets themselves).

Notes
-----
This should be a relatively fast operation that should not depend on
the size of the repository.

Subclasses may override this method, but must delegate to the base
class implementation at some point in their own logic.
More often, subclasses will specialize the behavior of `prep` by
overriding other methods to which the base class implementation
delegates.  These include:
 - `iterMappings`
 - `isDatasetTypeSpecial`
 - `getSpecialDirectories`
 - `makeRepoWalkerTarget`

This should not perform any write operations to the Gen3 repository.
It is guaranteed to be called before `ingest`.

Reimplemented from lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.

Definition at line 133 of file rootRepoConverter.py.

     def prep(self):
         # Docstring inherited from RepoConverter.
         # Gather information about reference catalogs.
         if self.task.isDatasetTypeIncluded("ref_cat") and len(self.task.config.refCats) != 0:
             from lsst.meas.algorithms import DatasetConfig as RefCatDatasetConfig
             for refCat in os.listdir(os.path.join(self.root, "ref_cats")):
                 path = os.path.join(self.root, "ref_cats", refCat)
                 configFile = os.path.join(path, "config.py")
                 if not os.path.exists(configFile):
                     continue
                 if refCat not in self.task.config.refCats:
                     continue
                 self.task.log.info(f"Preparing ref_cat {refCat} from root {self.root}.")
                 onDiskConfig = RefCatDatasetConfig()
                 onDiskConfig.load(configFile)
                 if onDiskConfig.indexer.name != "HTM":
                     raise ValueError(f"Reference catalog '{refCat}' uses unsupported "
                                      f"pixelization '{onDiskConfig.indexer.name}'.")
                 level = onDiskConfig.indexer["HTM"].depth
                 try:
                     dimension = self.task.universe[f"htm{level}"]
                 except KeyError as err:
                     raise ValueError(f"Reference catalog {refCat} uses HTM level {level}, but no htm{level} "
                                      f"skypix dimension is configured for this registry.") from err
                 self.task.useSkyPix(dimension)
                 self._refCats[refCat] = dimension
         if self.task.isDatasetTypeIncluded("brightObjectMask") and self.task.config.rootSkyMapName:
             self.task.useSkyMap(self._rootSkyMap, self.task.config.rootSkyMapName)
         super().prep()
  

◆ runDefineVisits()

def lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter.runDefineVisits	(	self,
		pool = `None`
	)

Definition at line 125 of file rootRepoConverter.py.

     def runDefineVisits(self, pool=None):
         if self.task.defineVisits is None:
             return
         dimensions = DimensionGraph(self.task.universe, names=["exposure"])
         exposureDataIds = set(ref.dataId.subset(dimensions) for ref in self._rawRefs)
         self.task.log.info("Defining visits from exposures.")
         self.task.defineVisits.run(exposureDataIds, pool=pool)
  

◆ runRawIngest()

def lsst.obs.base.gen2to3.rootRepoConverter.RootRepoConverter.runRawIngest	(	self,
		pool = `None`
	)

Definition at line 109 of file rootRepoConverter.py.

     def runRawIngest(self, pool=None):
         if self.task.raws is None:
             return
         self.task.log.info(f"Finding raws in root {self.root}.")
         if self.subset is not None:
             dataRefs = itertools.chain.from_iterable(
                 self.butler2.subset(self.task.config.rawDatasetType,
                                     visit=visit) for visit in self.subset.visits
             )
         else:
             dataRefs = self.butler2.subset(self.task.config.rawDatasetType)
         dataPaths = getDataPaths(dataRefs)
         self.task.log.info("Ingesting raws from root %s into run %s.", self.root, self.task.raws.butler.run)
         self._rawRefs.extend(self.task.raws.run(dataPaths, pool=pool))
         self._chain = [self.task.raws.butler.run]
  

Member Data Documentation

◆ butler2

lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.butler2

inherited

Definition at line 90 of file standardRepoConverter.py.

◆ instrument

lsst.obs.base.gen2to3.repoConverter.RepoConverter.instrument

inherited

Definition at line 213 of file repoConverter.py.

◆ mapper

lsst.obs.base.gen2to3.standardRepoConverter.StandardRepoConverter.mapper

inherited

Definition at line 91 of file standardRepoConverter.py.

◆ root

lsst.obs.base.gen2to3.repoConverter.RepoConverter.root

inherited

Definition at line 212 of file repoConverter.py.

◆ subset

lsst.obs.base.gen2to3.repoConverter.RepoConverter.subset

inherited

Definition at line 214 of file repoConverter.py.

◆ task

lsst.obs.base.gen2to3.repoConverter.RepoConverter.task

inherited

Definition at line 211 of file repoConverter.py.

The documentation for this class was generated from the following file:

/j/snowflake/release/lsstsw/stack/0.2.1/Linux64/obs_base/21.0.0-27-gbbd0d29+ae871e0f33/python/lsst/obs/base/gen2to3/rootRepoConverter.py

Public Member Functions

Public Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ expandDataIds()

◆ findDatasets()

◆ findMatchingSkyMap()

◆ finish()

◆ getCollectionChain()

◆ getRun()

◆ getSpecialDirectories()

◆ ingest()

◆ isDatasetTypeSpecial()

◆ iterDatasets()

◆ iterMappings()

◆ makeRepoWalkerTarget()

◆ prep()

◆ runDefineVisits()

◆ runRawIngest()

Member Data Documentation

◆ butler2

◆ instrument

◆ mapper

◆ root

◆ subset

◆ task

◆ init()