Inheritance diagram for lsst.obs.base.gen3.calibRepoConverter.CalibRepoConverter:

Public Member Functions
def	__init__ (self, root, universe, baseDataId, mapper=None)

def	addDatasetType (self, datasetTypeName, storageClass)

def	convertRepo (self, butler, directory=None, transfer=None, formatter=None, skipDirs=())

def	extractDatasetRef (self, fileNameInRoot)

def	walkRepo (self, directory=None, skipDirs=())

Public Attributes
	db

	dimensionEntries

	baseDataId

	root

	mapper

	universe

	extractors

	skyMap

Static Public Attributes
tuple	COADD_NAMES = ("deep", "goodSeeing", "dcr")

tuple	REPO_ROOT_FILES

Detailed Description

A helper class that ingests (some of) the contents of a Gen2 calibration
data repository into a Gen3 data repository.

This class must be used instead of the base `RepoConverter` to process
dataset types that are associated with validity ranges.

Parameters
----------
root : `str`
    Root of the Gen2 data repository.
universe : `lsst.daf.butler.DimensionUniverse`
    Object containing all dimension definitions.
baseDataId : `dict`
    Key-value pairs that may need to appear in the Gen3 data ID, but can
    never be inferred from a Gen2 filename.  This should always include
    the instrument name (even Gen3 data IDs that don't involve the
    instrument dimension have instrument-dependent Gen2 filenames).
mapper : `lsst.obs.base.CameraMapper`, optional
    Object that defines Gen2 filename templates.  Will be identified,
    imported, and constructed from ``root`` if not provided.

Definition at line 32 of file calibRepoConverter.py.

Constructor & Destructor Documentation

◆ init()

def lsst.obs.base.gen3.calibRepoConverter.CalibRepoConverter.__init__	(	self,
		root,
		universe,
		baseDataId,
		mapper = `None`
	)

Definition at line 55 of file calibRepoConverter.py.

     def __init__(self, root, *, universe, baseDataId, mapper=None):
         super().__init__(root, universe=universe, baseDataId=baseDataId, mapper=mapper)
         self.db = sqlite3.connect(os.path.join(root, "calibRegistry.sqlite3"))
         self.db.row_factory = sqlite3.Row
         self.dimensionEntries = []
         self.baseDataId = baseDataId
 

Member Function Documentation

◆ addDatasetType()

def lsst.obs.base.gen3.calibRepoConverter.CalibRepoConverter.addDatasetType	(	self,
		datasetTypeName,
		storageClass
	)

Definition at line 62 of file calibRepoConverter.py.

     def addDatasetType(self, datasetTypeName, storageClass):
         # Docstring inherited from RepoConverter.addDatasetType
         extractor = super().addDatasetType(datasetTypeName, storageClass)
         if "calibration_label" not in extractor.datasetType.dimensions:
             return extractor
         fields = ["validStart", "validEnd", "calibDate"]
         if "detector" in extractor.datasetType.dimensions.names:
             fields.append("ccd")
         else:
             fields.append("NULL AS ccd")
         if "physical_filter" in extractor.datasetType.dimensions.names:
             fields.append("filter")
         else:
             fields.append("NULL AS filter")
         query = f"SELECT DISTINCT {', '.join(fields)} FROM {datasetTypeName};"
         for row in self.db.execute(query):
             label = makeCalibrationLabel(datasetTypeName, row["calibDate"],
                                          ccd=row["ccd"], filter=row["filter"])
             self.dimensionEntries.append({
                 "instrument": self.baseDataId["instrument"],
                 "calibration_label": label,
                 "valid_first": datetime.strptime(row["validStart"], "%Y-%m-%d"),
                 "valid_last": datetime.strptime(row["validEnd"], "%Y-%m-%d") + timedelta(days=1),
             })
         return extractor
 

◆ convertRepo()

def lsst.obs.base.gen3.calibRepoConverter.CalibRepoConverter.convertRepo	(	self,
		butler,
		directory = `None`,
		transfer = `None`,
		formatter = `None`,
		skipDirs = `()`
	)

Definition at line 88 of file calibRepoConverter.py.

     def convertRepo(self, butler, *, directory=None, transfer=None, formatter=None, skipDirs=()):
         # Docstring inherited from RepoConverter.convertRepo.
         butler.registry.addDimensionEntryList(butler.registry.dimensions["calibration_label"],
                                               self.dimensionEntries)
         super().convertRepo(butler, directory=directory, transfer=transfer, formatter=formatter,
                             skipDirs=tuple(skipDirs) + ("focalplane",))
 

◆ extractDatasetRef()

def lsst.obs.base.gen3.repoConverter.RepoConverter.extractDatasetRef	(	self,
		fileNameInRoot
	)

inherited

Extract a Gen3 `~lsst.daf.butler.DatasetRef` from a filename in a
Gen2 data repository.

Parameters
----------
fileNameInRoot : `str`
    Name of the file, relative to the root of its Gen2 repository.

Return
------
ref : `lsst.daf.butler.DatasetRef` or `None`
    Reference to the Gen3 dataset that would be created by converting
    this file, or `None` if the file is not recognized as an instance
    of a dataset type known to this converter.

Definition at line 223 of file repoConverter.py.

     def extractDatasetRef(self, fileNameInRoot):
         """Extract a Gen3 `~lsst.daf.butler.DatasetRef` from a filename in a
         Gen2 data repository.
 
         Parameters
         ----------
         fileNameInRoot : `str`
             Name of the file, relative to the root of its Gen2 repository.
 
         Return
         ------
         ref : `lsst.daf.butler.DatasetRef` or `None`
             Reference to the Gen3 dataset that would be created by converting
             this file, or `None` if the file is not recognized as an instance
             of a dataset type known to this converter.
         """
         for datasetTypeName, extractor in self.extractors.items():
             dataId = extractor.apply(fileNameInRoot)
             if dataId is not None:
                 # Move the extractor that matched to the front of the
                 # dictionary, as we're likely to see instances of the
                 # same DatasetType together.
                 self.extractors.move_to_end(datasetTypeName, last=False)
                 return DatasetRef(extractor.datasetType, dataId=dataId)
         return None
 

◆ walkRepo()

def lsst.obs.base.gen3.repoConverter.RepoConverter.walkRepo	(	self,
		directory = `None`,
		skipDirs = `()`
	)

inherited

Recursively a (subset of) a Gen2 data repository, yielding files
that may be convertible.

Parameters
----------
directory : `str`, optional
    A subdirectory of the repository root to process, instead of
    processing the entire repository.
skipDirs : sequence of `str`
    Subdirectories that should be skipped.

Yields
------
fileNameInRoot : `str`
    Name of a file in the repository, relative to the root of the
    repository.

Definition at line 249 of file repoConverter.py.

     def walkRepo(self, directory=None, skipDirs=()):
         """Recursively a (subset of) a Gen2 data repository, yielding files
         that may be convertible.
 
         Parameters
         ----------
         directory : `str`, optional
             A subdirectory of the repository root to process, instead of
             processing the entire repository.
         skipDirs : sequence of `str`
             Subdirectories that should be skipped.
 
         Yields
         ------
         fileNameInRoot : `str`
             Name of a file in the repository, relative to the root of the
             repository.
         """
         if directory is None:
             directory = self.root
         for dirPath, subdirNamesInDir, fileNamesInDir in os.walk(directory, followlinks=True):
             # Remove subdirectories that appear to be repositories themselves
             # from the walking
             def isRepoRoot(dirName):
                 return any(os.path.exists(os.path.join(dirPath, dirName, f))
                            for f in self.REPO_ROOT_FILES)
             subdirNamesInDir[:] = [d for d in subdirNamesInDir if not isRepoRoot(d) and d not in skipDirs]
             # Loop over files in this directory, and ask per-DatasetType
             # extractors if they recognize them and can extract a data ID;
             # if so, ingest.
             dirPathInRoot = dirPath[len(self.root) + len(os.path.sep):]
             for fileNameInDir in fileNamesInDir:
                 fileNameInRoot = os.path.join(dirPathInRoot, fileNameInDir)
                 if fileNameInRoot in self.REPO_ROOT_FILES:
                     continue
                 yield fileNameInRoot
 

Member Data Documentation

◆ baseDataId

lsst.obs.base.gen3.calibRepoConverter.CalibRepoConverter.baseDataId

Definition at line 60 of file calibRepoConverter.py.

◆ COADD_NAMES

tuple lsst.obs.base.gen3.repoConverter.RepoConverter.COADD_NAMES = ("deep", "goodSeeing", "dcr")

staticinherited

Definition at line 167 of file repoConverter.py.

◆ db

lsst.obs.base.gen3.calibRepoConverter.CalibRepoConverter.db

Definition at line 57 of file calibRepoConverter.py.

◆ dimensionEntries

lsst.obs.base.gen3.calibRepoConverter.CalibRepoConverter.dimensionEntries

Definition at line 59 of file calibRepoConverter.py.

◆ extractors

lsst.obs.base.gen3.repoConverter.RepoConverter.extractors

inherited

Definition at line 185 of file repoConverter.py.

◆ mapper

lsst.obs.base.gen3.repoConverter.RepoConverter.mapper

inherited

Definition at line 182 of file repoConverter.py.

◆ REPO_ROOT_FILES

tuple lsst.obs.base.gen3.repoConverter.RepoConverter.REPO_ROOT_FILES

staticinherited

Initial value:

= ("registry.sqlite3", "_mapper", "repositoryCfg.yaml",

"calibRegistry.sqlite3", "_parent")

Definition at line 168 of file repoConverter.py.

◆ root

lsst.obs.base.gen3.repoConverter.RepoConverter.root

inherited

Definition at line 172 of file repoConverter.py.

◆ skyMap

lsst.obs.base.gen3.repoConverter.RepoConverter.skyMap

inherited

Definition at line 199 of file repoConverter.py.

◆ universe

lsst.obs.base.gen3.repoConverter.RepoConverter.universe

inherited

Definition at line 183 of file repoConverter.py.

The documentation for this class was generated from the following file:

/j/snowflake/release/lsstsw/stack/Linux64/obs_base/18.1.0-18-gb5d19ff+1/python/lsst/obs/base/gen3/calibRepoConverter.py

Public Member Functions

Public Attributes

Static Public Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __init__()

Member Function Documentation

◆ addDatasetType()

◆ convertRepo()

◆ extractDatasetRef()

◆ walkRepo()

Member Data Documentation

◆ baseDataId

◆ COADD_NAMES

◆ db

◆ dimensionEntries

◆ extractors

◆ mapper

◆ REPO_ROOT_FILES

◆ root

◆ skyMap

◆ universe

◆ init()