LSSTApplications
10.0+286,10.0+36,10.0+46,10.0-2-g4f67435,10.1+152,10.1+37,11.0,11.0+1,11.0-1-g47edd16,11.0-1-g60db491,11.0-1-g7418c06,11.0-2-g04d2804,11.0-2-g68503cd,11.0-2-g818369d,11.0-2-gb8b8ce7
LSSTDataManagementBasePackage
|
Public Member Functions | |
def | __init__ |
def | defineAlias |
def | getKeys |
def | queryMetadata |
def | datasetExists |
def | get |
def | put |
def | subset |
def | dataRef |
def | __reduce__ |
Static Public Member Functions | |
def | getMapperClass |
Public Attributes | |
datasetTypeAliasDict | |
mapper | |
persistence | |
log | |
Private Member Functions | |
def | _combineDicts |
def | _read |
def | _resolveDatasetTypeAlias |
Butler provides a generic mechanism for persisting and retrieving data using mappers. A Butler manages a collection of datasets known as a repository. Each dataset has a type representing its intended usage and a location. Note that the dataset type is not the same as the C++ or Python type of the object containing the data. For example, an ExposureF object might be used to hold the data for a raw image, a post-ISR image, a calibrated science image, or a difference image. These would all be different dataset types. A Butler can produce a collection of possible values for a key (or tuples of values for multiple keys) if given a partial data identifier. It can check for the existence of a file containing a dataset given its type and data identifier. The Butler can then retrieve the dataset. Similarly, it can persist an object to an appropriate location when given its associated data identifier. Note that the Butler has two more advanced features when retrieving a data set. First, the retrieval is lazy. Input does not occur until the data set is actually accessed. This allows datasets to be retrieved and placed on a clipboard prospectively with little cost, even if the algorithm of a stage ends up not using them. Second, the Butler will call a standardization hook upon retrieval of the dataset. This function, contained in the input mapper object, must perform any necessary manipulations to force the retrieved object to conform to standards, including translating metadata. Public methods: __init__(self, root, mapper=None, **mapperArgs) getKeys(self, datasetType=None, level=None) queryMetadata(self, datasetType, keys, format=None, dataId={}, **rest) datasetExists(self, datasetType, dataId={}, **rest) get(self, datasetType, dataId={}, immediate=False, **rest) put(self, obj, datasetType, dataId={}, **rest) subset(self, datasetType, level=None, dataId={}, **rest))
def lsst.daf.persistence.butler.Butler.__init__ | ( | self, | |
root, | |||
mapper = None , |
|||
mapperArgs | |||
) |
Construct the Butler. If no mapper class is provided, then a file named "_mapper" is expected to be found in the repository, which must be a filesystem path. The first line in that file is read and must contain the fully-qualified name of a Mapper subclass, which is then imported and instantiated using the root and the mapperArgs. @param root (str) the repository to be managed (at least initially). May be None if a mapper is provided. @param mapper (Mapper) if present, the Mapper subclass instance to be used as the butler's mapper. @param **mapperArgs arguments to be passed to the mapper's __init__ method, in addition to the root.
Definition at line 112 of file butler.py.
def lsst.daf.persistence.butler.Butler.__reduce__ | ( | self | ) |
Definition at line 477 of file butler.py.
|
private |
|
private |
Definition at line 425 of file butler.py.
|
private |
Replaces all the known alias keywords in the given string with the alias value. @param (str)datasetType @return (str) the de-aliased string
Definition at line 480 of file butler.py.
def lsst.daf.persistence.butler.Butler.dataRef | ( | self, | |
datasetType, | |||
level = None , |
|||
dataId = {} , |
|||
rest | |||
) |
Returns a single ButlerDataRef. Given a complete dataId specified in dataId and **rest, find the unique dataset at the given level specified by a dataId key (e.g. visit or sensor or amp for a camera) and return a ButlerDataRef. @param datasetType (str) the type of dataset collection to reference @param level (str) the level of dataId at which to reference @param dataId (dict) the data id. @param **rest keyword arguments for the data id. @returns (ButlerDataRef) ButlerDataRef for dataset matching the data id
Definition at line 395 of file butler.py.
def lsst.daf.persistence.butler.Butler.datasetExists | ( | self, | |
datasetType, | |||
dataId = {} , |
|||
rest | |||
) |
Determines if a dataset file exists. @param datasetType (str) the type of dataset to inquire about. @param dataId (dict) the data id of the dataset. @param **rest keyword arguments for the data id. @returns (bool) True if the dataset exists or is non-file-based.
Definition at line 211 of file butler.py.
def lsst.daf.persistence.butler.Butler.defineAlias | ( | self, | |
alias, | |||
datasetType | |||
) |
Register an alias that will be substituted in datasetTypes. @param alias (str) the alias keyword. it may start with @ or not. It may not contain @ except as the first character. @param datasetType (str) the string that will be substituted when @alias is passed into datasetType. It may not contain '@'
Definition at line 142 of file butler.py.
def lsst.daf.persistence.butler.Butler.get | ( | self, | |
datasetType, | |||
dataId = {} , |
|||
immediate = False , |
|||
rest | |||
) |
Retrieves a dataset given an input collection data id. @param datasetType (str) the type of dataset to retrieve. @param dataId (dict) the data id. @param immediate (bool) don't use a proxy for delayed loading. @param **rest keyword arguments for the data id. @returns an object retrieved from the dataset (or a proxy for one).
Definition at line 243 of file butler.py.
def lsst.daf.persistence.butler.Butler.getKeys | ( | self, | |
datasetType = None , |
|||
level = None |
|||
) |
Returns a dict. The dict keys are the valid data id keys at or above the given level of hierarchy for the dataset type or the entire collection if None. The dict values are the basic Python types corresponding to the keys (int, float, str). @param datasetType (str) the type of dataset to get keys for, entire collection if None. @param level (str) the hierarchy level to descend to or None. @returns (dict) valid data id keys; values are corresponding types.
Definition at line 171 of file butler.py.
|
static |
def lsst.daf.persistence.butler.Butler.put | ( | self, | |
obj, | |||
datasetType, | |||
dataId = {} , |
|||
doBackup = False , |
|||
rest | |||
) |
Persists a dataset given an output collection data id. @param obj the object to persist. @param datasetType (str) the type of dataset to persist. @param dataId (dict) the data id. @param doBackup if True, rename existing instead of overwriting @param **rest keyword arguments for the data id. WARNING: Setting doBackup=True is not safe for parallel processing, as it may be subject to race conditions.
Definition at line 284 of file butler.py.
def lsst.daf.persistence.butler.Butler.queryMetadata | ( | self, | |
datasetType, | |||
key, | |||
format = None , |
|||
dataId = {} , |
|||
rest | |||
) |
Returns the valid values for one or more keys when given a partial input collection data id. @param datasetType (str) the type of dataset to inquire about. @param key (str) a key giving the level of granularity of the inquiry. @param format (str, tuple) an optional key or tuple of keys to be returned. @param dataId (dict) the partial data id. @param **rest keyword arguments for the partial data id. @returns (list) a list of valid values or tuples of valid values as specified by the format (defaulting to the same as the key) at the key's level of granularity.
Definition at line 186 of file butler.py.
def lsst.daf.persistence.butler.Butler.subset | ( | self, | |
datasetType, | |||
level = None , |
|||
dataId = {} , |
|||
rest | |||
) |
Extracts a subset of a dataset collection. Given a partial dataId specified in dataId and **rest, find all datasets at a given level specified by a dataId key (e.g. visit or sensor or amp for a camera) and return a collection of their dataIds as ButlerDataRefs. @param datasetType (str) the type of dataset collection to subset @param level (str) the level of dataId at which to subset @param dataId (dict) the data id. @param **rest keyword arguments for the data id. @returns (ButlerSubset) collection of ButlerDataRefs for datasets matching the data id.
Definition at line 373 of file butler.py.