lsst.obs.base.mapping.ExposureMapping Class Reference

Inheritance diagram for lsst.obs.base.mapping.ExposureMapping:

Public Member Functions
def	__init__ (self, datasetType, policy, registry, root, kwargs)
def	standardize (self, mapper, item, dataId)
def	template (self)
def	keys (self)
def	map (self, mapper, dataId, write=False)
def	lookup (self, properties, dataId)
def	have (self, properties, dataId)
def	need (self, properties, dataId)

Public Attributes
	columns
	datasetType
	registry
	rootStorage
	keyDict
	python
	persistable
	storage
	level
	tables
	range
	obsTimeName
	recipe

Detailed Description

ExposureMapping is a Mapping subclass for normal exposures.

Parameters
----------
datasetType : `str`
    Butler dataset type to be mapped.
policy : `daf_persistence.Policy`
    Mapping Policy.
registry : `lsst.obs.base.Registry`
    Registry for metadata lookups
root : `str`
    Path of root directory

Definition at line 359 of file mapping.py.

Constructor & Destructor Documentation

__init__()

def lsst.obs.base.mapping.ExposureMapping.__init__	(		self,
			datasetType,
			policy,
			registry,
			root,
			kwargs
	)

Definition at line 374 of file mapping.py.

    def __init__(self, datasetType, policy, registry, root, **kwargs):
        Mapping.__init__(self, datasetType, policy, registry, root, **kwargs)
        self.columns = policy.asArray('columns') if 'columns' in policy else None

Member Function Documentation

have()

def lsst.obs.base.mapping.Mapping.have (   self,   properties,   dataId  )

inherited

Returns whether the provided data identifier has all
the properties in the provided list.

Parameters
----------
properties : `list of `str`
    Properties required.
dataId : `dict`
    Dataset identifier.

Returns
-------
bool
    True if all properties are present.

Definition at line 268 of file mapping.py.

    def have(self, properties, dataId):
        """Returns whether the provided data identifier has all
        the properties in the provided list.

        Parameters
        ----------
        properties : `list of `str`
            Properties required.
        dataId : `dict`
            Dataset identifier.

        Returns
        -------
        bool
            True if all properties are present.
        """
        for prop in properties:
            if prop not in dataId:
                return False
        return True

keys()

def lsst.obs.base.mapping.Mapping.keys (   self )

inherited

Return the dict of keys and value types required for this mapping.

Definition at line 132 of file mapping.py.

    def keys(self):
        """Return the dict of keys and value types required for this mapping."""
        return self.keyDict

lookup()

def lsst.obs.base.mapping.Mapping.lookup (   self,   properties,   dataId  )

inherited

Look up properties for in a metadata registry given a partial
dataset identifier.

Parameters
----------
properties : `list` of `str`
    What to look up.
dataId : `dict`
    Dataset identifier

Returns
-------
`list` of `tuple`
    Values of properties.

Definition at line 186 of file mapping.py.

    def lookup(self, properties, dataId):
        """Look up properties for in a metadata registry given a partial
        dataset identifier.

        Parameters
        ----------
        properties : `list` of `str`
            What to look up.
        dataId : `dict`
            Dataset identifier

        Returns
        -------
        `list` of `tuple`
            Values of properties.
        """
        if self.registry is None:
            raise RuntimeError("No registry for lookup")

        skyMapKeys = ("tract", "patch")

        where = []
        values = []

        # Prepare to remove skymap entries from properties list.  These must
        # be in the data ID, so we store which ones we're removing and create
        # an OrderedDict that tells us where to re-insert them.  That maps the
        # name of the property to either its index in the properties list
        # *after* the skymap ones have been removed (for entries that aren't
        # skymap ones) or the value from the data ID (for those that are).
        removed = set()
        substitutions = OrderedDict()
        index = 0
        properties = list(properties)  # don't modify the original list
        for p in properties:
            if p in skyMapKeys:
                try:
                    substitutions[p] = dataId[p]
                    removed.add(p)
                except KeyError:
                    raise RuntimeError(
                        "Cannot look up skymap key '%s'; it must be explicitly included in the data ID" % p
                    )
            else:
                substitutions[p] = index
                index += 1
        # Can't actually remove while iterating above, so we do it here.
        for p in removed:
            properties.remove(p)

        fastPath = True
        for p in properties:
            if p not in ('filter', 'expTime', 'taiObs'):
                fastPath = False
                break
        if fastPath and 'visit' in dataId and "raw" in self.tables:
            lookupDataId = {'visit': dataId['visit']}
            result = self.registry.lookup(properties, 'raw_visit', lookupDataId, template=self.template)
        else:
            if dataId is not None:
                for k, v in dataId.items():
                    if self.columns and k not in self.columns:
                        continue
                    if k == self.obsTimeName:
                        continue
                    if k in skyMapKeys:
                        continue
                    where.append((k, '?'))
                    values.append(v)
            lookupDataId = {k[0]: v for k, v in zip(where, values)}
            if self.range:
                # format of self.range is ('?', isBetween-lowKey, isBetween-highKey)
                # here we transform that to {(lowKey, highKey): value}
                lookupDataId[(self.range[1], self.range[2])] = dataId[self.obsTimeName]
            result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
        if not removed:
            return result
        # Iterate over the query results, re-inserting the skymap entries.
        result = [tuple(v if k in removed else item[v] for k, v in substitutions.items())
                  for item in result]
        return result

map()

def lsst.obs.base.mapping.Mapping.map (   self,   mapper,   dataId,   write = False  )

inherited

Standard implementation of map function.

Parameters
----------
mapper: `lsst.daf.persistence.Mapper`
    Object to be mapped.
dataId: `dict`
    Dataset identifier.

Returns
-------
lsst.daf.persistence.ButlerLocation
    Location of object that was mapped.

Definition at line 136 of file mapping.py.

    def map(self, mapper, dataId, write=False):
        """Standard implementation of map function.

        Parameters
        ----------
        mapper: `lsst.daf.persistence.Mapper`
            Object to be mapped.
        dataId: `dict`
            Dataset identifier.

        Returns
        -------
        lsst.daf.persistence.ButlerLocation
            Location of object that was mapped.
        """
        actualId = self.need(iter(self.keyDict.keys()), dataId)
        usedDataId = {key: actualId[key] for key in self.keyDict.keys()}
        path = mapper._mapActualToPath(self.template, actualId)
        if os.path.isabs(path):
            raise RuntimeError("Mapped path should not be absolute.")
        if not write:
            # This allows mapped files to be compressed, ending in .gz or .fz, without any indication from the
            # policy that the file should be compressed, easily allowing repositories to contain a combination
            # of comporessed and not-compressed files.
            # If needed we can add a policy flag to allow compressed files or not, and perhaps a list of
            # allowed extensions that may exist at the end of the template.
            for ext in (None, '.gz', '.fz'):
                if ext and path.endswith(ext):
                    continue  # if the path already ends with the extension
                extPath = path + ext if ext else path
                newPath = self.rootStorage.instanceSearch(extPath)
                if newPath:
                    path = newPath
                    break
        assert path, "Fully-qualified filename is empty."

        addFunc = "add_" + self.datasetType  # Name of method for additionalData
        if hasattr(mapper, addFunc):
            addFunc = getattr(mapper, addFunc)
            additionalData = addFunc(self.datasetType, actualId)
            assert isinstance(additionalData, PropertySet), \
                "Bad type for returned data: %s" (type(additionalData),)
        else:
            additionalData = None

        return ButlerLocation(pythonType=self.python, cppType=self.persistable, storageName=self.storage,
                              locationList=path, dataId=actualId.copy(), mapper=mapper,
                              storage=self.rootStorage, usedDataId=usedDataId, datasetType=self.datasetType,
                              additionalData=additionalData)

need()

def lsst.obs.base.mapping.Mapping.need (   self,   properties,   dataId  )

inherited

Ensures all properties in the provided list are present in
the data identifier, looking them up as needed.  This is only
possible for the case where the data identifies a single
exposure.

Parameters
----------
properties : `list` of `str`
    Properties required.
dataId : `dict`
    Partial dataset identifier

Returns
-------
`dict`
    Copy of dataset identifier with enhanced values.

Definition at line 289 of file mapping.py.

    def need(self, properties, dataId):
        """Ensures all properties in the provided list are present in
        the data identifier, looking them up as needed.  This is only
        possible for the case where the data identifies a single
        exposure.

        Parameters
        ----------
        properties : `list` of `str`
            Properties required.
        dataId : `dict`
            Partial dataset identifier

        Returns
        -------
        `dict`
            Copy of dataset identifier with enhanced values.
        """
        newId = dataId.copy()
        newProps = []                    # Properties we don't already have
        for prop in properties:
            if prop not in newId:
                newProps.append(prop)
        if len(newProps) == 0:
            return newId

        lookups = self.lookup(newProps, newId)
        if len(lookups) != 1:
            raise NoResults("No unique lookup for %s from %s: %d matches" %
                            (newProps, newId, len(lookups)),
                            self.datasetType, dataId)
        for i, prop in enumerate(newProps):
            newId[prop] = lookups[0][i]
        return newId

standardize()

def lsst.obs.base.mapping.ExposureMapping.standardize	(		self,
			mapper,
			item,
			dataId
	)

Definition at line 378 of file mapping.py.

    def standardize(self, mapper, item, dataId):
        return mapper._standardizeExposure(self, item, dataId)

template()

def lsst.obs.base.mapping.Mapping.template (   self )

inherited

Definition at line 125 of file mapping.py.

    def template(self):
        if self._template:  # template must not be an empty string or None
            return self._template
        else:
            raise RuntimeError("Template is not defined for the {} dataset type, ".format(self.datasetType) +
                               "it must be set before it can be used.")

Member Data Documentation

columns

lsst.obs.base.mapping.ExposureMapping.columns

Definition at line 376 of file mapping.py.

datasetType

lsst.obs.base.mapping.Mapping.datasetType

inherited

Definition at line 88 of file mapping.py.

keyDict

lsst.obs.base.mapping.Mapping.keyDict

inherited

Definition at line 99 of file mapping.py.

level

lsst.obs.base.mapping.Mapping.level

inherited

Definition at line 114 of file mapping.py.

obsTimeName

lsst.obs.base.mapping.Mapping.obsTimeName

inherited

Definition at line 121 of file mapping.py.

persistable

lsst.obs.base.mapping.Mapping.persistable

inherited

Definition at line 111 of file mapping.py.

python

lsst.obs.base.mapping.Mapping.python

inherited

Definition at line 110 of file mapping.py.

range

lsst.obs.base.mapping.Mapping.range

inherited

Definition at line 119 of file mapping.py.

recipe

lsst.obs.base.mapping.Mapping.recipe

inherited

Definition at line 122 of file mapping.py.

registry

lsst.obs.base.mapping.Mapping.registry

inherited

Definition at line 89 of file mapping.py.

rootStorage

lsst.obs.base.mapping.Mapping.rootStorage

inherited

Definition at line 90 of file mapping.py.

storage

lsst.obs.base.mapping.Mapping.storage

inherited

Definition at line 112 of file mapping.py.

tables

lsst.obs.base.mapping.Mapping.tables

inherited

Definition at line 116 of file mapping.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/Linux64/obs_base/18.1.0/python/lsst/obs/base/mapping.py