LSSTApplications  10.0+286,10.0+36,10.0+46,10.0-2-g4f67435,10.1+152,10.1+37,11.0,11.0+1,11.0-1-g47edd16,11.0-1-g60db491,11.0-1-g7418c06,11.0-2-g04d2804,11.0-2-g68503cd,11.0-2-g818369d,11.0-2-gb8b8ce7
LSSTDataManagementBasePackage
Public Member Functions | Public Attributes | List of all members
lsst.daf.butlerUtils.mapping.Mapping Class Reference
Inheritance diagram for lsst.daf.butlerUtils.mapping.Mapping:
lsst.daf.butlerUtils.mapping.CalibrationMapping lsst.daf.butlerUtils.mapping.DatasetMapping lsst.daf.butlerUtils.mapping.ExposureMapping lsst.daf.butlerUtils.mapping.ImageMapping

Public Member Functions

def __init__
 
def keys
 
def map
 
def lookup
 
def have
 
def need
 

Public Attributes

 datasetType
 
 registry
 
 root
 
 template
 
 keyDict
 
 python
 
 persistable
 
 storage
 
 level
 
 tables
 
 range
 
 columns
 
 obsTimeName
 

Detailed Description

Mapping is a base class for all mappings.  Mappings are used by
the Mapper to map (determine a path to some data given some
identifiers) and standardize (convert data into some standard
format or type) data, and to query the associated registry to see
what data is available.

Subclasses must specify self.storage or else override self.map().

Public methods: lookup, have, need, getKeys, map

Mappings are specified mainly by policy.  A Mapping policy should
consist of:

template (string): a Python string providing the filename for that
particular dataset type based on some data identifiers.  In the
case of redundancy in the path (e.g., file uniquely specified by
the exposure number, but filter in the path), the
redundant/dependent identifiers can be looked up in the registry.

python (string): the Python type for the retrieved data (e.g.
lsst.afw.image.ExposureF)

persistable (string): the Persistable registration for the on-disk data
(e.g. ImageU)

storage (string, optional): Storage type for this dataset type (e.g.
"BoostStorage")

level (string, optional): the level in the camera hierarchy at which the
data is stored (Amp, Ccd or skyTile), if relevant

tables (string, optional): a whitespace-delimited list of tables in the
registry that can be NATURAL JOIN-ed to look up additional
information.  

Definition at line 31 of file mapping.py.

Constructor & Destructor Documentation

def lsst.daf.butlerUtils.mapping.Mapping.__init__ (   self,
  datasetType,
  policy,
  registry,
  root,
  provided = None 
)
Constructor for Mapping class.
@param datasetType    (string)
@param policy         (lsst.pex.policy.Policy) Mapping policy
@param registry       (lsst.daf.butlerUtils.Registry) Registry for metadata lookups
@param root           (string) Path of root directory
@param provided       (list of strings) Keys provided by the mapper

Definition at line 68 of file mapping.py.

68 
69  def __init__(self, datasetType, policy, registry, root, provided=None):
70  """Constructor for Mapping class.
71  @param datasetType (string)
72  @param policy (lsst.pex.policy.Policy) Mapping policy
73  @param registry (lsst.daf.butlerUtils.Registry) Registry for metadata lookups
74  @param root (string) Path of root directory
75  @param provided (list of strings) Keys provided by the mapper
76  """
77 
78  if policy is None:
79  raise RuntimeError, "No policy provided for mapping"
80 
81  self.datasetType = datasetType
82  self.registry = registry
83  self.root = root
84 
85  self.template = policy.getString("template") # Template path
86  self.keyDict = dict([
87  (k, _formatMap(v, k, datasetType))
88  for k, v in
89  re.findall(r'\%\((\w+)\).*?([diouxXeEfFgGcrs])', self.template)
90  ])
91  if provided is not None:
92  for p in provided:
93  if p in self.keyDict:
94  del self.keyDict[p]
95  self.python = policy.getString("python") # Python type
96  self.persistable = policy.getString("persistable") # Persistable type
97  self.storage = policy.getString("storage")
98  if policy.exists("level"):
99  self.level = policy.getString("level") # Level in camera hierarchy
100  if policy.exists("tables"):
101  self.tables = policy.getStringArray("tables")
102  else:
103  self.tables = None
104  self.range = None
105  self.columns = None
106  self.obsTimeName = policy.getString("obsTimeName") \
107  if policy.exists("obsTimeName") else None

Member Function Documentation

def lsst.daf.butlerUtils.mapping.Mapping.have (   self,
  properties,
  dataId 
)
Returns whether the provided data identifier has all
the properties in the provided list.
@param properties (list of strings) Properties required
@parm dataId      (dict) Dataset identifier
@return (bool) True if all properties are present

Definition at line 170 of file mapping.py.

171  def have(self, properties, dataId):
172  """Returns whether the provided data identifier has all
173  the properties in the provided list.
174  @param properties (list of strings) Properties required
175  @parm dataId (dict) Dataset identifier
176  @return (bool) True if all properties are present"""
177  for prop in properties:
178  if not dataId.has_key(prop):
179  return False
180  return True
def lsst.daf.butlerUtils.mapping.Mapping.keys (   self)
Return the dict of keys and value types required for this mapping.

Definition at line 108 of file mapping.py.

109  def keys(self):
110  """Return the dict of keys and value types required for this mapping."""
111  return self.keyDict
def lsst.daf.butlerUtils.mapping.Mapping.lookup (   self,
  properties,
  dataId 
)
Look up properties for in a metadata registry given a partial
dataset identifier.
@param properties (list of strings)
@param dataId     (dict) Dataset identifier
@return (list of tuples) values of properties

Definition at line 137 of file mapping.py.

138  def lookup(self, properties, dataId):
139  """Look up properties for in a metadata registry given a partial
140  dataset identifier.
141  @param properties (list of strings)
142  @param dataId (dict) Dataset identifier
143  @return (list of tuples) values of properties"""
144 
145  if self.registry is None:
146  raise RuntimeError, "No registry for lookup"
147 
148  where = []
149  values = []
150  fastPath = True
151  for p in properties:
152  if p not in ('filter', 'expTime', 'taiObs'):
153  fastPath = False
154  break
155  if fastPath and dataId.has_key('visit') and "raw" in self.tables:
156  return self.registry.executeQuery(properties, ('raw_visit',),
157  [('visit', '?')], None, (dataId['visit'],))
158  if dataId is not None:
159  for k, v in dataId.iteritems():
160  if self.columns and not k in self.columns:
161  continue
162  if k == self.obsTimeName:
163  continue
164  where.append((k, '?'))
165  values.append(v)
166  if self.range is not None:
167  values.append(dataId[self.obsTimeName])
168  return self.registry.executeQuery(properties, self.tables,
169  where, self.range, values)
def lsst.daf.butlerUtils.mapping.Mapping.map (   self,
  mapper,
  dataId,
  write = False 
)
Standard implementation of map function.
@param mapper (lsst.daf.persistence.Mapper)
@param dataId (dict) Dataset identifier
@return (lsst.daf.persistence.ButlerLocation)

Definition at line 112 of file mapping.py.

113  def map(self, mapper, dataId, write=False):
114  """Standard implementation of map function.
115  @param mapper (lsst.daf.persistence.Mapper)
116  @param dataId (dict) Dataset identifier
117  @return (lsst.daf.persistence.ButlerLocation)"""
118 
119  actualId = self.need(self.keyDict.iterkeys(), dataId)
120  path = mapper._mapActualToPath(self.template, actualId)
121  if not os.path.isabs(path):
122  path = os.path.join(self.root, path)
123  if not write:
124  newPath = mapper._parentSearch(path)
125  if newPath is not None:
126  path = newPath
127 
128  addFunc = "add_" + self.datasetType # Name of method for additionalData
129  if hasattr(mapper, addFunc):
130  addFunc = getattr(mapper, addFunc)
131  additionalData = addFunc(actualId)
132  assert isinstance(additionalData, dict), "Bad type for returned data"
133  else:
134  additionalData = actualId.copy()
135 
136  return ButlerLocation(self.python, self.persistable, self.storage, path, additionalData)
def lsst.daf.butlerUtils.mapping.Mapping.need (   self,
  properties,
  dataId 
)
Ensures all properties in the provided list are present in
the data identifier, looking them up as needed.  This is only
possible for the case where the data identifies a single
exposure.
@param properties (list of strings) Properties required
@param dataId     (dict) Partial dataset identifier
@return (dict) copy of dataset identifier with enhanced values

Definition at line 181 of file mapping.py.

182  def need(self, properties, dataId):
183  """Ensures all properties in the provided list are present in
184  the data identifier, looking them up as needed. This is only
185  possible for the case where the data identifies a single
186  exposure.
187  @param properties (list of strings) Properties required
188  @param dataId (dict) Partial dataset identifier
189  @return (dict) copy of dataset identifier with enhanced values
190  """
191 
192  newId = dataId.copy()
193  newProps = [] # Properties we don't already have
194  for prop in properties:
195  if not newId.has_key(prop):
196  newProps.append(prop)
197  if len(newProps) == 0:
198  return newId
199 
200  lookups = self.lookup(newProps, newId)
201  if len(lookups) != 1:
202  raise RuntimeError, "No unique lookup for %s from %s: %d matches" % (newProps, newId, len(lookups))
203  for i, prop in enumerate(newProps):
204  newId[prop] = lookups[0][i]
205  return newId

Member Data Documentation

lsst.daf.butlerUtils.mapping.Mapping.columns

Definition at line 104 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.datasetType

Definition at line 80 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.keyDict

Definition at line 85 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.level

Definition at line 98 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.obsTimeName

Definition at line 105 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.persistable

Definition at line 95 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.python

Definition at line 94 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.range

Definition at line 103 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.registry

Definition at line 81 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.root

Definition at line 82 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.storage

Definition at line 96 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.tables

Definition at line 100 of file mapping.py.

lsst.daf.butlerUtils.mapping.Mapping.template

Definition at line 84 of file mapping.py.


The documentation for this class was generated from the following file: