Inheritance diagram for lsst.obs.base.gen3.repoConverter.RepoConverter:

Public Member Functions
def	__init__ (self, root, universe, baseDataId, mapper=None, skyMap=None)

def	addDatasetType (self, datasetTypeName, storageClass)

def	extractDatasetRef (self, fileNameInRoot)

def	walkRepo (self, directory=None, skipDirs=())

def	convertRepo (self, butler, directory=None, transfer=None, formatter=None, skipDirs=())

Public Attributes
	root

	mapper

	universe

	baseDataId

	extractors

	skyMap

Static Public Attributes
tuple	COADD_NAMES = ("deep", "goodSeeing", "dcr")

tuple	REPO_ROOT_FILES

Detailed Description

A helper class that ingests (some of) the contents of a Gen2 data
repository into a Gen3 data repository.

Parameters
----------
root : `str`
    Root of the Gen2 data repository.
universe : `lsst.daf.butler.DimensionUniverse`
    Object containing all dimension definitions.
baseDataId : `dict`
    Key-value pairs that may need to appear in the Gen3 data ID, but can
    never be inferred from a Gen2 filename.  This should always include
    the instrument name (even Gen3 data IDs that don't involve the
    instrument dimension have instrument-dependent Gen2 filenames) and
    should also include the skymap name in order to process any data IDs
    that involve tracts or patches.
mapper : `lsst.obs.base.CameraMapper`, optional
    Object that defines Gen2 filename templates.  Will be identified,
    imported, and constructed from ``root`` if not provided.
skyMap : `lsst.skymap.BaseSkyMap`, optional
    SkyMap that defines tracts and patches.  Must be provided in order to
    provess datasets with a ``patch`` key in their data IDs.

Definition at line 142 of file repoConverter.py.

Constructor & Destructor Documentation

◆ init()

def lsst.obs.base.gen3.repoConverter.RepoConverter.__init__	(	self,
		root,
		universe,
		baseDataId,
		mapper = `None`,
		skyMap = `None`
	)

Definition at line 171 of file repoConverter.py.

     def __init__(self, root, *, universe, baseDataId, mapper=None, skyMap=None):
         self.root = root
         if mapper is None:
             # Shush spurious log messages from Gen2 Mapper classes.
             # These aren't spurious in other contexts - we're just playing fast
             # and loose with mapper initialization, because we don't care about
             # things like parent lookups (we just want the set of templates).
             with temporaryLogLevel("CameraMapper", Log.ERROR):
                 with temporaryLogLevel("HscMapper", Log.ERROR):
                     cls = findMapperClass(root)
                     mapper = cls(root=root)
         self.mapper = mapper
         self.universe = universe
         self.baseDataId = baseDataId
         self.extractors = OrderedDict()  # for move_to_end
         if "skymap" in baseDataId:
             if skyMap is None:
                 for name in self.COADD_NAMES:
                     mapping = self.mapper.mappings.get(f"{name}Coadd_skyMap", None)
                     if mapping is None:
                         continue
                     filename = os.path.join(self.root, mapping.template)
                     if os.path.exists(filename):
                         if skyMap is not None:
                             raise ValueError("Multiple SkyMaps found in repository; please use multiple "
                                              "RepoConverters with an explicit skyMap argument for each.")
                         with open(filename, "rb") as f:
                             skyMap = pickle.load(f, encoding="latin1")
         self.skyMap = skyMap
 

Member Function Documentation

◆ addDatasetType()

def lsst.obs.base.gen3.repoConverter.RepoConverter.addDatasetType	(	self,
		datasetTypeName,
		storageClass
	)

Add a dataset type to those recognized by the converter.

Parameters
----------
datasetTypeName : `str`
    String name of the dataset type.
storageClass : `str` or `lsst.daf.butler.StorageClass`
    Gen3 storage class of the dataset type.

Returns
-------
extractor : `DataIdExtractor`
    The object that will be used to extract data IDs for instances of
    this dataset type (also held internally, so the return value can
    usually be ignored).

Definition at line 201 of file repoConverter.py.

     def addDatasetType(self, datasetTypeName, storageClass):
         """Add a dataset type to those recognized by the converter.
 
         Parameters
         ----------
         datasetTypeName : `str`
             String name of the dataset type.
         storageClass : `str` or `lsst.daf.butler.StorageClass`
             Gen3 storage class of the dataset type.
 
         Returns
         -------
         extractor : `DataIdExtractor`
             The object that will be used to extract data IDs for instances of
             this dataset type (also held internally, so the return value can
             usually be ignored).
         """
         r = DataIdExtractor(datasetTypeName, storageClass, mapper=self.mapper,
                             universe=self.universe, baseDataId=self.baseDataId, skyMap=self.skyMap)
         self.extractors[datasetTypeName] = r
         return r
 

◆ convertRepo()

def lsst.obs.base.gen3.repoConverter.RepoConverter.convertRepo	(	self,
		butler,
		directory = `None`,
		transfer = `None`,
		formatter = `None`,
		skipDirs = `()`
	)

Ingest all recognized files into a Gen3 repository.

Parameters
----------
butler : `lsst.daf.butler.Butler`
    Gen3 butler that files should be ingested into.
directory : `str`, optional
    A subdirectory of the repository root to process, instead of
    processing the entire repository.
transfer : str, optional
    If not `None`, must be one of 'move', 'copy', 'hardlink', or
    'symlink' indicating how to transfer the file.
formatter : `lsst.daf.butler.Formatter`, optional
    Formatter that should be used to retreive the Dataset.  If not
    provided, the formatter will be constructed according to
    Datastore configuration.  This should only be used when converting
    only a single dataset type multiple dataset types of the same
    storage class.
skipDirs : sequence of `str`
    Subdirectories that should be skipped.

Definition at line 286 of file repoConverter.py.

     def convertRepo(self, butler, *, directory=None, transfer=None, formatter=None, skipDirs=()):
         """Ingest all recognized files into a Gen3 repository.
 
         Parameters
         ----------
         butler : `lsst.daf.butler.Butler`
             Gen3 butler that files should be ingested into.
         directory : `str`, optional
             A subdirectory of the repository root to process, instead of
             processing the entire repository.
         transfer : str, optional
             If not `None`, must be one of 'move', 'copy', 'hardlink', or
             'symlink' indicating how to transfer the file.
         formatter : `lsst.daf.butler.Formatter`, optional
             Formatter that should be used to retreive the Dataset.  If not
             provided, the formatter will be constructed according to
             Datastore configuration.  This should only be used when converting
             only a single dataset type multiple dataset types of the same
             storage class.
         skipDirs : sequence of `str`
             Subdirectories that should be skipped.
         """
         log = Log.getLogger("RepoConverter")
         for extractor in self.extractors.values():
             butler.registry.registerDatasetType(extractor.datasetType)
         skipped = {}
         for file in self.walkRepo(directory=directory, skipDirs=skipDirs):
             ref = self.extractDatasetRef(file)
             if ref is not None:
                 try:
                     butler.ingest(os.path.join(self.root, file), ref, transfer=transfer, formatter=formatter)
                 except Exception as err:
                     skipped.setdefault(type(err), []).append(str(err))
         if skipped:
             for cls, messages in skipped.items():
                 log.warn("Skipped %s files due to exceptions of type %s.", len(messages), cls.__name__)
                 if log.isDebugEnabled():
                     for message in messages:
                         log.debug(message)

◆ extractDatasetRef()

def lsst.obs.base.gen3.repoConverter.RepoConverter.extractDatasetRef	(	self,
		fileNameInRoot
	)

Extract a Gen3 `~lsst.daf.butler.DatasetRef` from a filename in a
Gen2 data repository.

Parameters
----------
fileNameInRoot : `str`
    Name of the file, relative to the root of its Gen2 repository.

Return
------
ref : `lsst.daf.butler.DatasetRef` or `None`
    Reference to the Gen3 dataset that would be created by converting
    this file, or `None` if the file is not recognized as an instance
    of a dataset type known to this converter.

Definition at line 223 of file repoConverter.py.

     def extractDatasetRef(self, fileNameInRoot):
         """Extract a Gen3 `~lsst.daf.butler.DatasetRef` from a filename in a
         Gen2 data repository.
 
         Parameters
         ----------
         fileNameInRoot : `str`
             Name of the file, relative to the root of its Gen2 repository.
 
         Return
         ------
         ref : `lsst.daf.butler.DatasetRef` or `None`
             Reference to the Gen3 dataset that would be created by converting
             this file, or `None` if the file is not recognized as an instance
             of a dataset type known to this converter.
         """
         for datasetTypeName, extractor in self.extractors.items():
             dataId = extractor.apply(fileNameInRoot)
             if dataId is not None:
                 # Move the extractor that matched to the front of the
                 # dictionary, as we're likely to see instances of the
                 # same DatasetType together.
                 self.extractors.move_to_end(datasetTypeName, last=False)
                 return DatasetRef(extractor.datasetType, dataId=dataId)
         return None
 

◆ walkRepo()

def lsst.obs.base.gen3.repoConverter.RepoConverter.walkRepo	(	self,
		directory = `None`,
		skipDirs = `()`
	)

Recursively a (subset of) a Gen2 data repository, yielding files
that may be convertible.

Parameters
----------
directory : `str`, optional
    A subdirectory of the repository root to process, instead of
    processing the entire repository.
skipDirs : sequence of `str`
    Subdirectories that should be skipped.

Yields
------
fileNameInRoot : `str`
    Name of a file in the repository, relative to the root of the
    repository.

Definition at line 249 of file repoConverter.py.

     def walkRepo(self, directory=None, skipDirs=()):
         """Recursively a (subset of) a Gen2 data repository, yielding files
         that may be convertible.
 
         Parameters
         ----------
         directory : `str`, optional
             A subdirectory of the repository root to process, instead of
             processing the entire repository.
         skipDirs : sequence of `str`
             Subdirectories that should be skipped.
 
         Yields
         ------
         fileNameInRoot : `str`
             Name of a file in the repository, relative to the root of the
             repository.
         """
         if directory is None:
             directory = self.root
         for dirPath, subdirNamesInDir, fileNamesInDir in os.walk(directory, followlinks=True):
             # Remove subdirectories that appear to be repositories themselves
             # from the walking
             def isRepoRoot(dirName):
                 return any(os.path.exists(os.path.join(dirPath, dirName, f))
                            for f in self.REPO_ROOT_FILES)
             subdirNamesInDir[:] = [d for d in subdirNamesInDir if not isRepoRoot(d) and d not in skipDirs]
             # Loop over files in this directory, and ask per-DatasetType
             # extractors if they recognize them and can extract a data ID;
             # if so, ingest.
             dirPathInRoot = dirPath[len(self.root) + len(os.path.sep):]
             for fileNameInDir in fileNamesInDir:
                 fileNameInRoot = os.path.join(dirPathInRoot, fileNameInDir)
                 if fileNameInRoot in self.REPO_ROOT_FILES:
                     continue
                 yield fileNameInRoot
 

Member Data Documentation

◆ baseDataId

lsst.obs.base.gen3.repoConverter.RepoConverter.baseDataId

Definition at line 184 of file repoConverter.py.

◆ COADD_NAMES

tuple lsst.obs.base.gen3.repoConverter.RepoConverter.COADD_NAMES = ("deep", "goodSeeing", "dcr")

static

Definition at line 167 of file repoConverter.py.

◆ extractors

lsst.obs.base.gen3.repoConverter.RepoConverter.extractors

Definition at line 185 of file repoConverter.py.

◆ mapper

lsst.obs.base.gen3.repoConverter.RepoConverter.mapper

Definition at line 182 of file repoConverter.py.

◆ REPO_ROOT_FILES

tuple lsst.obs.base.gen3.repoConverter.RepoConverter.REPO_ROOT_FILES

static

Initial value:

= ("registry.sqlite3", "_mapper", "repositoryCfg.yaml",

"calibRegistry.sqlite3", "_parent")

Definition at line 168 of file repoConverter.py.

◆ root

lsst.obs.base.gen3.repoConverter.RepoConverter.root

Definition at line 172 of file repoConverter.py.

◆ skyMap

lsst.obs.base.gen3.repoConverter.RepoConverter.skyMap

Definition at line 199 of file repoConverter.py.

◆ universe

lsst.obs.base.gen3.repoConverter.RepoConverter.universe

Definition at line 183 of file repoConverter.py.

The documentation for this class was generated from the following file:

/j/snowflake/release/lsstsw/stack/Linux64/obs_base/18.1.0-18-gb5d19ff+1/python/lsst/obs/base/gen3/repoConverter.py

Public Member Functions

Public Attributes

Static Public Attributes