lsst.obs.base.mapping.ImageMapping Class Reference

Inheritance diagram for lsst.obs.base.mapping.ImageMapping:

Public Member Functions
def	__init__ (self, datasetType, policy, registry, root, **kwargs)
def	template (self)
def	keys (self)
def	map (self, mapper, dataId, write=False)
def	lookup (self, properties, dataId)
def	have (self, properties, dataId)
def	need (self, properties, dataId)

Public Attributes
	columns
	datasetType
	registry
	rootStorage
	keyDict
	python
	persistable
	storage
	level
	tables
	range
	obsTimeName
	recipe

Detailed Description

ImageMapping is a Mapping subclass for non-camera images.

Parameters
----------
datasetType : `str`
    Butler dataset type to be mapped.
policy : `daf_persistence.Policy`
    Mapping Policy.
registry : `lsst.obs.base.Registry`
    Registry for metadata lookups
root : `str`
    Path of root directory

Definition at line 346 of file mapping.py.

Constructor & Destructor Documentation

__init__()

def lsst.obs.base.mapping.ImageMapping.__init__	(		self,
			datasetType,
			policy,
			registry,
			root,
		**	kwargs
	)

Definition at line 361 of file mapping.py.

    def __init__(self, datasetType, policy, registry, root, **kwargs):
        Mapping.__init__(self, datasetType, policy, registry, root, **kwargs)
        self.columns = policy.asArray('columns') if 'columns' in policy else None
 
 

Member Function Documentation

have()

def lsst.obs.base.mapping.Mapping.have (   self,   properties,   dataId  )

inherited

Returns whether the provided data identifier has all
the properties in the provided list.

Parameters
----------
properties : `list of `str`
    Properties required.
dataId : `dict`
    Dataset identifier.

Returns
-------
bool
    True if all properties are present.

Definition at line 275 of file mapping.py.

    def have(self, properties, dataId):
        """Returns whether the provided data identifier has all
        the properties in the provided list.
 
        Parameters
        ----------
        properties : `list of `str`
            Properties required.
        dataId : `dict`
            Dataset identifier.
 
        Returns
        -------
        bool
            True if all properties are present.
        """
        for prop in properties:
            if prop not in dataId:
                return False
        return True
 

keys()

def lsst.obs.base.mapping.Mapping.keys (   self )

inherited

Return the dict of keys and value types required for this mapping.

Definition at line 135 of file mapping.py.

    def keys(self):
        """Return the dict of keys and value types required for this mapping.
        """
        return self.keyDict
 

lookup()

def lsst.obs.base.mapping.Mapping.lookup (   self,   properties,   dataId  )

inherited

Look up properties for in a metadata registry given a partial
dataset identifier.

Parameters
----------
properties : `list` of `str`
    What to look up.
dataId : `dict`
    Dataset identifier

Returns
-------
`list` of `tuple`
    Values of properties.

Reimplemented in lsst.obs.base.mapping.CalibrationMapping.

Definition at line 192 of file mapping.py.

    def lookup(self, properties, dataId):
        """Look up properties for in a metadata registry given a partial
        dataset identifier.
 
        Parameters
        ----------
        properties : `list` of `str`
            What to look up.
        dataId : `dict`
            Dataset identifier
 
        Returns
        -------
        `list` of `tuple`
            Values of properties.
        """
        if self.registry is None:
            raise RuntimeError("No registry for lookup")
 
        skyMapKeys = ("tract", "patch")
 
        where = []
        values = []
 
        # Prepare to remove skymap entries from properties list.  These must
        # be in the data ID, so we store which ones we're removing and create
        # an OrderedDict that tells us where to re-insert them.  That maps the
        # name of the property to either its index in the properties list
        # *after* the skymap ones have been removed (for entries that aren't
        # skymap ones) or the value from the data ID (for those that are).
        removed = set()
        substitutions = OrderedDict()
        index = 0
        properties = list(properties)  # don't modify the original list
        for p in properties:
            if p in skyMapKeys:
                try:
                    substitutions[p] = dataId[p]
                    removed.add(p)
                except KeyError:
                    raise RuntimeError(
                        "Cannot look up skymap key '%s'; it must be explicitly included in the data ID" % p
                    )
            else:
                substitutions[p] = index
                index += 1
        # Can't actually remove while iterating above, so we do it here.
        for p in removed:
            properties.remove(p)
 
        fastPath = True
        for p in properties:
            if p not in ('filter', 'expTime', 'taiObs'):
                fastPath = False
                break
        if fastPath and 'visit' in dataId and "raw" in self.tables:
            lookupDataId = {'visit': dataId['visit']}
            result = self.registry.lookup(properties, 'raw_visit', lookupDataId, template=self.template)
        else:
            if dataId is not None:
                for k, v in dataId.items():
                    if self.columns and k not in self.columns:
                        continue
                    if k == self.obsTimeName:
                        continue
                    if k in skyMapKeys:
                        continue
                    where.append((k, '?'))
                    values.append(v)
            lookupDataId = {k[0]: v for k, v in zip(where, values)}
            if self.range:
                # format of self.range is
                # ('?', isBetween-lowKey, isBetween-highKey)
                # here we transform that to {(lowKey, highKey): value}
                lookupDataId[(self.range[1], self.range[2])] = dataId[self.obsTimeName]
            result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
        if not removed:
            return result
        # Iterate over the query results, re-inserting the skymap entries.
        result = [tuple(v if k in removed else item[v] for k, v in substitutions.items())
                  for item in result]
        return result
 

map()

def lsst.obs.base.mapping.Mapping.map (   self,   mapper,   dataId,   write = False  )

inherited

Standard implementation of map function.

Parameters
----------
mapper: `lsst.daf.persistence.Mapper`
    Object to be mapped.
dataId: `dict`
    Dataset identifier.

Returns
-------
lsst.daf.persistence.ButlerLocation
    Location of object that was mapped.

Reimplemented in lsst.obs.base.mapping.CalibrationMapping.

Definition at line 140 of file mapping.py.

    def map(self, mapper, dataId, write=False):
        """Standard implementation of map function.
 
        Parameters
        ----------
        mapper: `lsst.daf.persistence.Mapper`
            Object to be mapped.
        dataId: `dict`
            Dataset identifier.
 
        Returns
        -------
        lsst.daf.persistence.ButlerLocation
            Location of object that was mapped.
        """
        actualId = self.need(iter(self.keyDict.keys()), dataId)
        usedDataId = {key: actualId[key] for key in self.keyDict.keys()}
        path = mapper._mapActualToPath(self.template, actualId)
        if os.path.isabs(path):
            raise RuntimeError("Mapped path should not be absolute.")
        if not write:
            # This allows mapped files to be compressed, ending in .gz or .fz,
            # without any indication from the policy that the file should be
            # compressed, easily allowing repositories to contain a combination
            # of comporessed and not-compressed files.
            # If needed we can add a policy flag to allow compressed files or
            # not, and perhaps a list of  allowed extensions that may exist
            # at the end of the template.
            for ext in (None, '.gz', '.fz'):
                if ext and path.endswith(ext):
                    continue  # if the path already ends with the extension
                extPath = path + ext if ext else path
                newPath = self.rootStorage.instanceSearch(extPath)
                if newPath:
                    path = newPath
                    break
        assert path, "Fully-qualified filename is empty."
 
        addFunc = "add_" + self.datasetType  # Name of method for additionalData
        if hasattr(mapper, addFunc):
            addFunc = getattr(mapper, addFunc)
            additionalData = addFunc(self.datasetType, actualId)
            assert isinstance(additionalData, PropertySet), \
                "Bad type for returned data: %s" % (type(additionalData),)
        else:
            additionalData = None
 
        return ButlerLocation(pythonType=self.python, cppType=self.persistable, storageName=self.storage,
                              locationList=path, dataId=actualId.copy(), mapper=mapper,
                              storage=self.rootStorage, usedDataId=usedDataId, datasetType=self.datasetType,
                              additionalData=additionalData)
 

need()

def lsst.obs.base.mapping.Mapping.need (   self,   properties,   dataId  )

inherited

Ensures all properties in the provided list are present in
the data identifier, looking them up as needed.  This is only
possible for the case where the data identifies a single
exposure.

Parameters
----------
properties : `list` of `str`
    Properties required.
dataId : `dict`
    Partial dataset identifier

Returns
-------
`dict`
    Copy of dataset identifier with enhanced values.

Definition at line 296 of file mapping.py.

    def need(self, properties, dataId):
        """Ensures all properties in the provided list are present in
        the data identifier, looking them up as needed.  This is only
        possible for the case where the data identifies a single
        exposure.
 
        Parameters
        ----------
        properties : `list` of `str`
            Properties required.
        dataId : `dict`
            Partial dataset identifier
 
        Returns
        -------
        `dict`
            Copy of dataset identifier with enhanced values.
        """
        newId = dataId.copy()
        newProps = []                    # Properties we don't already have
        for prop in properties:
            if prop not in newId:
                newProps.append(prop)
        if len(newProps) == 0:
            return newId
 
        lookups = self.lookup(newProps, newId)
        if len(lookups) != 1:
            raise NoResults("No unique lookup for %s from %s: %d matches" %
                            (newProps, newId, len(lookups)),
                            self.datasetType, dataId)
        for i, prop in enumerate(newProps):
            newId[prop] = lookups[0][i]
        return newId
 
 

template()

def lsst.obs.base.mapping.Mapping.template (   self )

inherited

Definition at line 128 of file mapping.py.

    def template(self):
        if self._template:  # template must not be an empty string or None
            return self._template
        else:
            raise RuntimeError(f"Template is not defined for the {self.datasetType} dataset type, "
                               "it must be set before it can be used.")
 

Member Data Documentation

columns

lsst.obs.base.mapping.ImageMapping.columns

Definition at line 363 of file mapping.py.

datasetType

lsst.obs.base.mapping.Mapping.datasetType

inherited

Definition at line 89 of file mapping.py.

keyDict

lsst.obs.base.mapping.Mapping.keyDict

inherited

Definition at line 102 of file mapping.py.

level

lsst.obs.base.mapping.Mapping.level

inherited

Definition at line 117 of file mapping.py.

obsTimeName

lsst.obs.base.mapping.Mapping.obsTimeName

inherited

Definition at line 124 of file mapping.py.

persistable

lsst.obs.base.mapping.Mapping.persistable

inherited

Definition at line 114 of file mapping.py.

python

lsst.obs.base.mapping.Mapping.python

inherited

Definition at line 113 of file mapping.py.

range

lsst.obs.base.mapping.Mapping.range

inherited

Definition at line 122 of file mapping.py.

recipe

lsst.obs.base.mapping.Mapping.recipe

inherited

Definition at line 125 of file mapping.py.

registry

lsst.obs.base.mapping.Mapping.registry

inherited

Definition at line 90 of file mapping.py.

rootStorage

lsst.obs.base.mapping.Mapping.rootStorage

inherited

Definition at line 91 of file mapping.py.

storage

lsst.obs.base.mapping.Mapping.storage

inherited

Definition at line 115 of file mapping.py.

tables

lsst.obs.base.mapping.Mapping.tables

inherited

Definition at line 119 of file mapping.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/0.2.1/Linux64/obs_base/21.0.0-27-gbbd0d29+ae871e0f33/python/lsst/obs/base/mapping.py