Inheritance diagram for lsst.obs.base.mapping.Mapping:

Public Member Functions
def	__init__ (self, datasetType, policy, registry, rootStorage, provided=None)

def	template (self)

def	keys (self)

def	map (self, mapper, dataId, write=False)

def	lookup (self, properties, dataId)

def	have (self, properties, dataId)

def	need (self, properties, dataId)

Public Attributes
	datasetType

	registry

	rootStorage

	keyDict

	python

	persistable

	storage

	level

	tables

	range

	columns

	obsTimeName

	recipe

Detailed Description

Mapping is a base class for all mappings.  Mappings are used by
the Mapper to map (determine a path to some data given some
identifiers) and standardize (convert data into some standard
format or type) data, and to query the associated registry to see
what data is available.

Subclasses must specify self.storage or else override self.map().

Public methods: lookup, have, need, getKeys, map

Mappings are specified mainly by policy.  A Mapping policy should
consist of:

template (string): a Python string providing the filename for that
particular dataset type based on some data identifiers.  In the
case of redundancy in the path (e.g., file uniquely specified by
the exposure number, but filter in the path), the
redundant/dependent identifiers can be looked up in the registry.

python (string): the Python type for the retrieved data (e.g.
lsst.afw.image.ExposureF)

persistable (string): the Persistable registration for the on-disk data
(e.g. ImageU)

storage (string, optional): Storage type for this dataset type (e.g.
"FitsStorage")

level (string, optional): the level in the camera hierarchy at which the
data is stored (Amp, Ccd or skyTile), if relevant

tables (string, optional): a whitespace-delimited list of tables in the
registry that can be NATURAL JOIN-ed to look up additional
information.

Parameters
----------
datasetType : `str`
    Butler dataset type to be mapped.
policy : `daf_persistence.Policy`
    Mapping Policy.
registry : `lsst.obs.base.Registry`
    Registry for metadata lookups.
rootStorage : Storage subclass instance
    Interface to persisted repository data.
provided : `list` of `str`
    Keys provided by the mapper.

Definition at line 33 of file mapping.py.

Constructor & Destructor Documentation

◆ init()

def lsst.obs.base.mapping.Mapping.__init__	(	self,
		datasetType,
		policy,
		registry,
		rootStorage,
		provided = `None`
	)

Definition at line 84 of file mapping.py.

     def __init__(self, datasetType, policy, registry, rootStorage, provided=None):
  
         if policy is None:
             raise RuntimeError("No policy provided for mapping")
  
         self.datasetType = datasetType
         self.registry = registry
         self.rootStorage = rootStorage
  
         self._template = policy['template']  # Template path
         # in most cases, the template can not be used if it is empty, and is
         # accessed via a property that will raise if it is used while
         # `not self._template`. In this case we *do* allow it to be empty, for
         # the purpose of fetching the key dict so that the mapping can be
         # constructed, so that it can raise if it's invalid. I know it's a
         # little odd, but it allows this template check to be introduced
         # without a major refactor.
         if self._template:
             self.keyDict = dict([
                 (k, _formatMap(v, k, datasetType))
                 for k, v in
                 re.findall(r'\%\‍((\w+)\‍).*?([diouxXeEfFgGcrs])', self.template)
             ])
         else:
             self.keyDict = {}
         if provided is not None:
             for p in provided:
                 if p in self.keyDict:
                     del self.keyDict[p]
         self.python = policy['python']  # Python type
         self.persistable = policy['persistable']  # Persistable type
         self.storage = policy['storage']
         if 'level' in policy:
             self.level = policy['level']  # Level in camera hierarchy
         if 'tables' in policy:
             self.tables = policy.asArray('tables')
         else:
             self.tables = None
         self.range = None
         self.columns = None
         self.obsTimeName = policy['obsTimeName'] if 'obsTimeName' in policy else None
         self.recipe = policy['recipe'] if 'recipe' in policy else 'default'
  

Member Function Documentation

◆ have()

def lsst.obs.base.mapping.Mapping.have	(	self,
		properties,
		dataId
	)

Returns whether the provided data identifier has all
the properties in the provided list.

Parameters
----------
properties : `list of `str`
    Properties required.
dataId : `dict`
    Dataset identifier.

Returns
-------
bool
    True if all properties are present.

Definition at line 275 of file mapping.py.

     def have(self, properties, dataId):
         """Returns whether the provided data identifier has all
         the properties in the provided list.
  
         Parameters
         ----------
         properties : `list of `str`
             Properties required.
         dataId : `dict`
             Dataset identifier.
  
         Returns
         -------
         bool
             True if all properties are present.
         """
         for prop in properties:
             if prop not in dataId:
                 return False
         return True
  

◆ keys()

def lsst.obs.base.mapping.Mapping.keys ( self )

Return the dict of keys and value types required for this mapping.

Definition at line 135 of file mapping.py.

     def keys(self):
         """Return the dict of keys and value types required for this mapping.
         """
         return self.keyDict
  

◆ lookup()

def lsst.obs.base.mapping.Mapping.lookup	(	self,
		properties,
		dataId
	)

Look up properties for in a metadata registry given a partial
dataset identifier.

Parameters
----------
properties : `list` of `str`
    What to look up.
dataId : `dict`
    Dataset identifier

Returns
-------
`list` of `tuple`
    Values of properties.

Reimplemented in lsst.obs.base.mapping.CalibrationMapping.

Definition at line 192 of file mapping.py.

     def lookup(self, properties, dataId):
         """Look up properties for in a metadata registry given a partial
         dataset identifier.
  
         Parameters
         ----------
         properties : `list` of `str`
             What to look up.
         dataId : `dict`
             Dataset identifier
  
         Returns
         -------
         `list` of `tuple`
             Values of properties.
         """
         if self.registry is None:
             raise RuntimeError("No registry for lookup")
  
         skyMapKeys = ("tract", "patch")
  
         where = []
         values = []
  
         # Prepare to remove skymap entries from properties list.  These must
         # be in the data ID, so we store which ones we're removing and create
         # an OrderedDict that tells us where to re-insert them.  That maps the
         # name of the property to either its index in the properties list
         # *after* the skymap ones have been removed (for entries that aren't
         # skymap ones) or the value from the data ID (for those that are).
         removed = set()
         substitutions = OrderedDict()
         index = 0
         properties = list(properties)  # don't modify the original list
         for p in properties:
             if p in skyMapKeys:
                 try:
                     substitutions[p] = dataId[p]
                     removed.add(p)
                 except KeyError:
                     raise RuntimeError(
                         "Cannot look up skymap key '%s'; it must be explicitly included in the data ID" % p
                     )
             else:
                 substitutions[p] = index
                 index += 1
         # Can't actually remove while iterating above, so we do it here.
         for p in removed:
             properties.remove(p)
  
         fastPath = True
         for p in properties:
             if p not in ('filter', 'expTime', 'taiObs'):
                 fastPath = False
                 break
         if fastPath and 'visit' in dataId and "raw" in self.tables:
             lookupDataId = {'visit': dataId['visit']}
             result = self.registry.lookup(properties, 'raw_visit', lookupDataId, template=self.template)
         else:
             if dataId is not None:
                 for k, v in dataId.items():
                     if self.columns and k not in self.columns:
                         continue
                     if k == self.obsTimeName:
                         continue
                     if k in skyMapKeys:
                         continue
                     where.append((k, '?'))
                     values.append(v)
             lookupDataId = {k[0]: v for k, v in zip(where, values)}
             if self.range:
                 # format of self.range is
                 # ('?', isBetween-lowKey, isBetween-highKey)
                 # here we transform that to {(lowKey, highKey): value}
                 lookupDataId[(self.range[1], self.range[2])] = dataId[self.obsTimeName]
             result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
         if not removed:
             return result
         # Iterate over the query results, re-inserting the skymap entries.
         result = [tuple(v if k in removed else item[v] for k, v in substitutions.items())
                   for item in result]
         return result
  

◆ map()

def lsst.obs.base.mapping.Mapping.map	(	self,
		mapper,
		dataId,
		write = `False`
	)

Standard implementation of map function.

Parameters
----------
mapper: `lsst.daf.persistence.Mapper`
    Object to be mapped.
dataId: `dict`
    Dataset identifier.

Returns
-------
lsst.daf.persistence.ButlerLocation
    Location of object that was mapped.

Reimplemented in lsst.obs.base.mapping.CalibrationMapping.

Definition at line 140 of file mapping.py.

     def map(self, mapper, dataId, write=False):
         """Standard implementation of map function.
  
         Parameters
         ----------
         mapper: `lsst.daf.persistence.Mapper`
             Object to be mapped.
         dataId: `dict`
             Dataset identifier.
  
         Returns
         -------
         lsst.daf.persistence.ButlerLocation
             Location of object that was mapped.
         """
         actualId = self.need(iter(self.keyDict.keys()), dataId)
         usedDataId = {key: actualId[key] for key in self.keyDict.keys()}
         path = mapper._mapActualToPath(self.template, actualId)
         if os.path.isabs(path):
             raise RuntimeError("Mapped path should not be absolute.")
         if not write:
             # This allows mapped files to be compressed, ending in .gz or .fz,
             # without any indication from the policy that the file should be
             # compressed, easily allowing repositories to contain a combination
             # of comporessed and not-compressed files.
             # If needed we can add a policy flag to allow compressed files or
             # not, and perhaps a list of  allowed extensions that may exist
             # at the end of the template.
             for ext in (None, '.gz', '.fz'):
                 if ext and path.endswith(ext):
                     continue  # if the path already ends with the extension
                 extPath = path + ext if ext else path
                 newPath = self.rootStorage.instanceSearch(extPath)
                 if newPath:
                     path = newPath
                     break
         assert path, "Fully-qualified filename is empty."
  
         addFunc = "add_" + self.datasetType  # Name of method for additionalData
         if hasattr(mapper, addFunc):
             addFunc = getattr(mapper, addFunc)
             additionalData = addFunc(self.datasetType, actualId)
             assert isinstance(additionalData, PropertySet), \
                 "Bad type for returned data: %s" % (type(additionalData),)
         else:
             additionalData = None
  
         return ButlerLocation(pythonType=self.python, cppType=self.persistable, storageName=self.storage,
                               locationList=path, dataId=actualId.copy(), mapper=mapper,
                               storage=self.rootStorage, usedDataId=usedDataId, datasetType=self.datasetType,
                               additionalData=additionalData)
  

◆ need()

def lsst.obs.base.mapping.Mapping.need	(	self,
		properties,
		dataId
	)

Ensures all properties in the provided list are present in
the data identifier, looking them up as needed.  This is only
possible for the case where the data identifies a single
exposure.

Parameters
----------
properties : `list` of `str`
    Properties required.
dataId : `dict`
    Partial dataset identifier

Returns
-------
`dict`
    Copy of dataset identifier with enhanced values.

Definition at line 296 of file mapping.py.

     def need(self, properties, dataId):
         """Ensures all properties in the provided list are present in
         the data identifier, looking them up as needed.  This is only
         possible for the case where the data identifies a single
         exposure.
  
         Parameters
         ----------
         properties : `list` of `str`
             Properties required.
         dataId : `dict`
             Partial dataset identifier
  
         Returns
         -------
         `dict`
             Copy of dataset identifier with enhanced values.
         """
         newId = dataId.copy()
         newProps = []                    # Properties we don't already have
         for prop in properties:
             if prop not in newId:
                 newProps.append(prop)
         if len(newProps) == 0:
             return newId
  
         lookups = self.lookup(newProps, newId)
         if len(lookups) != 1:
             raise NoResults("No unique lookup for %s from %s: %d matches" %
                             (newProps, newId, len(lookups)),
                             self.datasetType, dataId)
         for i, prop in enumerate(newProps):
             newId[prop] = lookups[0][i]
         return newId
  
  

◆ template()

def lsst.obs.base.mapping.Mapping.template ( self )

Definition at line 128 of file mapping.py.

     def template(self):
         if self._template:  # template must not be an empty string or None
             return self._template
         else:
             raise RuntimeError(f"Template is not defined for the {self.datasetType} dataset type, "
                                "it must be set before it can be used.")
  

Member Data Documentation

◆ columns

lsst.obs.base.mapping.Mapping.columns

Definition at line 123 of file mapping.py.

◆ datasetType

lsst.obs.base.mapping.Mapping.datasetType

Definition at line 89 of file mapping.py.

◆ keyDict

lsst.obs.base.mapping.Mapping.keyDict

Definition at line 102 of file mapping.py.

◆ level

lsst.obs.base.mapping.Mapping.level

Definition at line 117 of file mapping.py.

◆ obsTimeName

lsst.obs.base.mapping.Mapping.obsTimeName

Definition at line 124 of file mapping.py.

◆ persistable

lsst.obs.base.mapping.Mapping.persistable

Definition at line 114 of file mapping.py.

◆ python

lsst.obs.base.mapping.Mapping.python

Definition at line 113 of file mapping.py.

◆ range

lsst.obs.base.mapping.Mapping.range

Definition at line 122 of file mapping.py.

◆ recipe

lsst.obs.base.mapping.Mapping.recipe

Definition at line 125 of file mapping.py.

◆ registry

lsst.obs.base.mapping.Mapping.registry

Definition at line 90 of file mapping.py.

◆ rootStorage

lsst.obs.base.mapping.Mapping.rootStorage

Definition at line 91 of file mapping.py.

◆ storage

lsst.obs.base.mapping.Mapping.storage

Definition at line 115 of file mapping.py.

◆ tables

lsst.obs.base.mapping.Mapping.tables

Definition at line 119 of file mapping.py.

The documentation for this class was generated from the following file:

/j/snowflake/release/lsstsw/stack/0.2.1/Linux64/obs_base/21.0.0-27-gbbd0d29+ae871e0f33/python/lsst/obs/base/mapping.py

Public Member Functions

Public Attributes