lsst.daf.butlerUtils.cameraMapper.CameraMapper Class Reference

Inheritance diagram for lsst.daf.butlerUtils.cameraMapper.CameraMapper:

Public Member Functions
def	__init__
def	backup
def	keys
def	getKeys
def	getDefaultLevel
def	getDefaultSubLevel
def	getCameraName
def	getPackageName
def	map_camera
def	bypass_camera
def	map_defects
def	bypass_defects
def	std_raw
def	map_skypolicy
def	std_skypolicy
Public Member Functions inherited from lsst.daf.persistence.mapper.Mapper
def	__new__
def	__init__
def	__getstate__
def	__setstate__
def	keys
def	queryMetadata
def	getDatasetTypes
def	map
def	canStandardize
def	standardize
def	validate
def	backup

Static Public Member Functions
def	getShortCcdName

Public Attributes
	log
	levels
	defaultLevel
	defaultSubLevels
	root
	registry
	keyDict
	mappings
	cameraDataLocation
	camera
	defectRegistry
	defectPath
	filters
	skypolicy

Static Public Attributes
	packageName = None

Private Member Functions
def	_parentSearch
def	_getCcdKeyVal
	Utility functions. More...
def	_setupRegistry
def	_transformId
def	_mapActualToPath
def	_extractDetectorName
def	_extractAmpId
def	_setAmpDetector
def	_setCcdDetector
def	_setFilter
def	_setTimes
def	_standardizeExposure
def	_defectLookup
def	_makeCamera

Detailed Description

CameraMapper is a base class for mappers that handle images from a
camera and products derived from them.  This provides an abstraction layer
between the data on disk and the code.

Public methods: keys, queryMetadata, getDatasetTypes, map,
canStandardize, standardize

Mappers for specific data sources (e.g., CFHT Megacam, LSST
simulations, etc.) should inherit this class.

The CameraMapper manages datasets within a "root" directory.  It can also
be given an "outputRoot".  If so, the input root is linked into the
outputRoot directory using a symlink named "_parent"; writes go into the
outputRoot while reads can come from either the root or outputRoot.  As
outputRoots are used as inputs for further processing, the chain of
_parent links allows any dataset to be retrieved.  Note that writing to a
dataset present in the input root will hide the existing dataset but not
overwrite it.  See #2160 for design discussion.

A camera is assumed to consist of one or more rafts, each composed of
multiple CCDs.  Each CCD is in turn composed of one or more amplifiers
(amps).  A camera is also assumed to have a camera geometry description
(CameraGeom object) as a policy file, a filter description (Filter class
static configuration) as another policy file, and an optional defects
description directory.

Information from the camera geometry and defects are inserted into all
Exposure objects returned.

The mapper uses one or two registries to retrieve metadata about the
images.  The first is a registry of all raw exposures.  This must contain
the time of the observation.  One or more tables (or the equivalent)
within the registry are used to look up data identifier components that
are not specified by the user (e.g. filter) and to return results for
metadata queries.  The second is an optional registry of all calibration
data.  This should contain validity start and end entries for each
calibration dataset in the same timescale as the observation time.

The following method must be provided by the subclass:

_extractDetectorName(self, dataId): returns the detector name for a CCD
(e.g., "CFHT 21", "R:1,2 S:3,4") as used in the AFW CameraGeom class given
a dataset identifier referring to that CCD or a subcomponent of it.

Other methods that the subclass may wish to override include:

_transformId(self, dataId): transformation of a data identifier
from colloquial usage (e.g., "ccdname") to proper/actual usage (e.g., "ccd"),
including making suitable for path expansion (e.g. removing commas).
The default implementation does nothing.  Note that this
method should not modify its input parameter.

getShortCcdName(self, ccdName): a static method that returns a shortened name
suitable for use as a filename. The default version converts spaces to underscores.

_getCcdKeyVal(self, dataId): return a CCD key and value
by which to look up defects in the defects registry.
The default value returns ("ccd", detector name)

_mapActualToPath(self, template, actualId): convert a template path to an
actual path, using the actual dataset identifier.

The mapper's behaviors are largely specified by the policy file.
See the MapperDictionary.paf for descriptions of the available items.

The 'exposures', 'calibrations', and 'datasets' subpolicies configure
mappings (see Mappings class).

Functions to map (provide a path to the data given a dataset
identifier dictionary) and standardize (convert data into some standard
format or type) may be provided in the subclass as "map_{dataset type}"
and "std_{dataset type}", respectively.

If non-Exposure datasets cannot be retrieved using standard
daf_persistence methods alone, a "bypass_{dataset type}" function may be
provided in the subclass to return the dataset instead of using the
"datasets" subpolicy.

Implementations of map_camera and bypass_camera that should typically be
sufficient are provided in this base class.

@todo
* Handle defects the same was as all other calibration products, using the calibration registry
* Instead of auto-loading the camera at construction time, load it from the calibration registry
* Rewrite defects as AFW tables so we don't need pyfits to unpersist them; then remove all mention
  of pyfits from this package.

Definition at line 40 of file cameraMapper.py.

Constructor & Destructor Documentation

def lsst.daf.butlerUtils.cameraMapper.CameraMapper.__init__	(		self,
			policy,
			repositoryDir,
			root = None,
			registry = None,
			calibRoot = None,
			calibRegistry = None,
			provided = None,
			outputRoot = None
	)

Initialize the CameraMapper.
@param policy        (pexPolicy.Policy) Policy with per-camera defaults
             already merged
@param repositoryDir (string) Policy repository for the subclassing
             module (obtained with getRepositoryPath() on the
             per-camera default dictionary)
@param root          (string) Root directory for data
@param registry      (string) Path to registry with data's metadata
@param calibRoot     (string) Root directory for calibrations
@param calibRegistry (string) Path to registry with calibrations'
             metadata
@param provided      (list of strings) Keys provided by the mapper
@param outputRoot    (string) Root directory for output data

Definition at line 133 of file cameraMapper.py.

                 provided=None, outputRoot=None):
        """Initialize the CameraMapper.
        @param policy        (pexPolicy.Policy) Policy with per-camera defaults
                             already merged
        @param repositoryDir (string) Policy repository for the subclassing
                             module (obtained with getRepositoryPath() on the
                             per-camera default dictionary)
        @param root          (string) Root directory for data
        @param registry      (string) Path to registry with data's metadata
        @param calibRoot     (string) Root directory for calibrations
        @param calibRegistry (string) Path to registry with calibrations'
                             metadata
        @param provided      (list of strings) Keys provided by the mapper
        @param outputRoot    (string) Root directory for output data
        """

        dafPersist.Mapper.__init__(self)

        self.log = pexLog.Log(pexLog.getDefaultLog(), "CameraMapper")

        # Dictionary
        dictFile = pexPolicy.DefaultPolicyFile("daf_butlerUtils",
                "MapperDictionary.paf", "policy")
        dictPolicy = pexPolicy.Policy.createPolicy(dictFile,
                dictFile.getRepositoryPath())
        policy.mergeDefaults(dictPolicy)

        # Levels
        self.levels = dict()
        if policy.exists("levels"):
            levelsPolicy = policy.getPolicy("levels")
            for key in levelsPolicy.names(True):
                self.levels[key] = set(levelsPolicy.getStringArray(key))
        self.defaultLevel = policy.getString("defaultLevel")
        self.defaultSubLevels = dict()
        if policy.exists("defaultSubLevels"):
            defaultSubLevelsPolicy = policy.getPolicy("defaultSubLevels")
            for key in defaultSubLevelsPolicy.names(True):
                self.defaultSubLevels[key] = defaultSubLevelsPolicy.getString(key)

        # Root directories
        if root is None:
            root = "."
        root = dafPersist.LogicalLocation(root).locString()

        if outputRoot is not None and os.path.abspath(outputRoot) != os.path.abspath(root):
            # Path manipulations are subject to race condition
            if not os.path.exists(outputRoot):
                try:
                    os.makedirs(outputRoot)
                except OSError, e:
                    if not e.errno == errno.EEXIST:
                        raise
                if not os.path.exists(outputRoot):
                    raise RuntimeError, "Unable to create output " \
                            "repository '%s'" % (outputRoot,)
            if os.path.exists(root):
                # Symlink existing input root to "_parent" in outputRoot.
                src = os.path.abspath(root)
                dst = os.path.join(outputRoot, "_parent")
                if not os.path.exists(dst):
                    try:
                        os.symlink(src, dst)
                    except OSError:
                        pass
                if os.path.exists(dst):
                    if os.path.realpath(dst) != os.path.realpath(src):
                        raise RuntimeError, "Output repository path " \
                                "'%s' already exists and differs from " \
                                "input repository path '%s'" % (dst, src)
                else:
                    raise RuntimeError, "Unable to symlink from input " \
                            "repository path '%s' to output repository " \
                            "path '%s'" % (src, dst)
            # We now use the outputRoot as the main root with access to the
            # input via "_parent".
            root = outputRoot

        if calibRoot is None:
            if policy.exists('calibRoot'):
                calibRoot = policy.getString('calibRoot')
                calibRoot = dafPersist.LogicalLocation(calibRoot).locString()
            else:
                calibRoot = root

        if not os.path.exists(root):
            self.log.log(pexLog.Log.WARN,
                    "Root directory not found: %s" % (root,))
        if not os.path.exists(calibRoot):
            self.log.log(pexLog.Log.WARN,
                    "Calibration root directory not found: %s" % (calibRoot,))
        self.root = root

        # Registries
        self.registry = self._setupRegistry(
                "registry", registry, policy, "registryPath", root)
        if policy.exists('needCalibRegistry') and \
                policy.getBool('needCalibRegistry'):
            calibRegistry = self._setupRegistry(
                    "calibRegistry", calibRegistry,
                    policy, "calibRegistryPath", calibRoot)
        else:
            calibRegistry = None

        # Sub-dictionaries (for exposure/calibration/dataset types)
        imgMappingFile = pexPolicy.DefaultPolicyFile("daf_butlerUtils",
                "ImageMappingDictionary.paf", "policy")
        imgMappingPolicy = pexPolicy.Policy.createPolicy(imgMappingFile,
                imgMappingFile.getRepositoryPath())
        expMappingFile = pexPolicy.DefaultPolicyFile("daf_butlerUtils",
                "ExposureMappingDictionary.paf", "policy")
        expMappingPolicy = pexPolicy.Policy.createPolicy(expMappingFile,
                expMappingFile.getRepositoryPath())
        calMappingFile = pexPolicy.DefaultPolicyFile("daf_butlerUtils",
                "CalibrationMappingDictionary.paf", "policy")
        calMappingPolicy = pexPolicy.Policy.createPolicy(calMappingFile,
                calMappingFile.getRepositoryPath())
        dsMappingFile = pexPolicy.DefaultPolicyFile("daf_butlerUtils",
                "DatasetMappingDictionary.paf", "policy")
        dsMappingPolicy = pexPolicy.Policy.createPolicy(dsMappingFile,
                dsMappingFile.getRepositoryPath())

        # Dict of valid keys and their value types
        self.keyDict = dict()

        # Mappings
        mappingList = (
                ("images", imgMappingPolicy, ImageMapping),
                ("exposures", expMappingPolicy, ExposureMapping),
                ("calibrations", calMappingPolicy, CalibrationMapping),
                ("datasets", dsMappingPolicy, DatasetMapping)
                )
        self.mappings = dict()
        for name, defPolicy, cls in mappingList:
            if policy.exists(name):
                datasets = policy.getPolicy(name)
                mappings = dict()
                setattr(self, name, mappings)
                for datasetType in datasets.names(True):
                    subPolicy = datasets.getPolicy(datasetType)
                    subPolicy.mergeDefaults(defPolicy)
                    if name == "calibrations":
                        mapping = cls(datasetType, subPolicy,
                                self.registry, calibRegistry, calibRoot, provided=provided)
                    else:
                        mapping = cls(datasetType, subPolicy,
                                self.registry, root, provided=provided)
                    self.keyDict.update(mapping.keys())
                    mappings[datasetType] = mapping
                    self.mappings[datasetType] = mapping
                    if not hasattr(self, "map_" + datasetType):
                        def mapClosure(dataId, write=False,
                                mapper=self, mapping=mapping):
                            return mapping.map(mapper, dataId, write)
                        setattr(self, "map_" + datasetType, mapClosure)
                    if not hasattr(self, "query_" + datasetType):
                        def queryClosure(key, format, dataId, mapping=mapping):
                            return mapping.lookup(format, dataId)
                        setattr(self, "query_" + datasetType, queryClosure)
                    if hasattr(mapping, "standardize") and \
                            not hasattr(self, "std_" + datasetType):
                        def stdClosure(item, dataId,
                                mapper=self, mapping=mapping):
                            return mapping.standardize(mapper, item, dataId)
                        setattr(self, "std_" + datasetType, stdClosure)

                    mapFunc = "map_" + datasetType + "_filename"
                    bypassFunc = "bypass_" + datasetType + "_filename"
                    if not hasattr(self, mapFunc):
                        setattr(self, mapFunc, getattr(self, "map_" + datasetType))
                    if not hasattr(self, bypassFunc):
                        setattr(self, bypassFunc,
                                lambda datasetType, pythonType, location, dataId: location.getLocations())

                    # Set up metadata versions
                    if name == "exposures" or name == "images":
                        expFunc = "map_" + datasetType # Function name to map exposure
                        mdFunc = expFunc + "_md"       # Function name to map metadata
                        bypassFunc = "bypass_" + datasetType + "_md" # Function name to bypass daf_persistence
                        if not hasattr(self, mdFunc):
                            setattr(self, mdFunc, getattr(self, expFunc))
                        if not hasattr(self, bypassFunc):
                            setattr(self, bypassFunc,
                                    lambda datasetType, pythonType, location, dataId:
                                    afwImage.readMetadata(location.getLocations()[0]))
                        if not hasattr(self, "query_" + datasetType + "_md"):
                            setattr(self, "query_" + datasetType + "_md",
                                    getattr(self, "query_" + datasetType))

                        subFunc = expFunc + "_sub" # Function name to map subimage
                        if not hasattr(self, subFunc):
                            def mapSubClosure(dataId, write=False, mapper=self, mapping=mapping):
                                subId = dataId.copy()
                                del subId['bbox']
                                loc = mapping.map(mapper, subId, write)
                                bbox = dataId['bbox']
                                llcX = bbox.getMinX()
                                llcY = bbox.getMinY()
                                width = bbox.getWidth()
                                height = bbox.getHeight()
                                loc.additionalData.set('llcX', llcX)
                                loc.additionalData.set('llcY', llcY)
                                loc.additionalData.set('width', width)
                                loc.additionalData.set('height', height)
                                if 'imageOrigin' in dataId:
                                    loc.additionalData.set('imageOrigin',
                                            dataId['imageOrigin'])
                                return loc
                            setattr(self, subFunc, mapSubClosure)
                        if not hasattr(self, "query_" + datasetType + "_sub"):
                            def querySubClosure(key, format, dataId, mapping=mapping):
                                subId = dataId.copy()
                                del subId['bbox']
                                return mapping.lookup(format, subId)
                            setattr(self, "query_" + datasetType + "_sub", querySubClosure)

        # Camera geometry
        self.cameraDataLocation = None # path to camera geometry config file
        self.camera = self._makeCamera(policy=policy, repositoryDir=repositoryDir)

        # Defect registry and root
        self.defectRegistry = None
        if policy.exists('defects'):
            self.defectPath = os.path.join(
                    repositoryDir, policy.getString('defects'))
            defectRegistryLocation = os.path.join(
                    self.defectPath, "defectRegistry.sqlite3")
            self.defectRegistry = \
                    Registry.create(defectRegistryLocation)

        # Filter translation table
        self.filters = None

        # Skytile policy
        self.skypolicy = policy.getPolicy("skytiles")

        # verify that the class variable packageName is set before attempting
        # to instantiate an instance
        if self.packageName is None:
            raise ValueError('class variable packageName must not be None')