Public Member Functions
def	__init__ (self, datasetType, sourceKeyTuple)

def	addSourceMetrics (self, repoInfo, idKeyTuple, idValList, sourceTableList)

def	finalize (self)

Public Attributes
	datasetType

	repoInfoList

	sourceArr

	sourceIdDict

	repoArr

Detailed Description

Accumulate a set of measurements from a set of source tables

To use:
- specify the desired source measurements when constructing this object
- call addSourceMetrics for each repository you harvest data from
- call finalize to produce the final data

Data available after calling finalize:
- self.sourceArr: a numpy structured array of shape (num repositories, num sources)
    containing named columns for:
    - source ID
    - each data ID key
    - each item of data extracted from the source table
- self.sourceIdDict: a dict of (source ID: index of axis 1 of self.sourceArr)
- self.repoArr: a numpy structured array of shape (num repositories,)
    containing a named column for each repository key (see RepositoryIterator)

@note: sources that had non-finite data (e.g. NaN) for every value extracted are silently omitted

Definition at line 54 of file repositoryIterator.py.

Constructor & Destructor Documentation

◆ init()

def lsst.pipe.tasks.repositoryIterator.SourceData.__init__	(	self,
		datasetType,
		sourceKeyTuple
	)

@param[in] datasetType: dataset type for source
@param[in] sourceKeyTuple: list of keys of data items to extract from the source tables

@raise RuntimeError if sourceKeyTuple is empty

Definition at line 75 of file repositoryIterator.py.

     def __init__(self, datasetType, sourceKeyTuple):
         """
         @param[in] datasetType: dataset type for source
         @param[in] sourceKeyTuple: list of keys of data items to extract from the source tables
  
         @raise RuntimeError if sourceKeyTuple is empty
         """
         if len(sourceKeyTuple) < 1:
             raise RuntimeError("Must specify at least one key in sourceKeyTuple")
         self.datasetType = datasetType
         self._sourceKeyTuple = tuple(sourceKeyTuple)
  
         self._idKeyTuple = None  # tuple of data ID keys, in order; set by first call to _getSourceMetrics
         self._idKeyDTypeList = None  # numpy dtype for data ID tuple, as a list of (key, type);
         # set by first call to _getSourceMetrics
         self._sourceDTypeList = None  # numpy dtype for source data, as a list of (key, type);
         # set by first call to _getSourceMetrics
         self._repoKeyTuple = None  # tuple of repo ID keys, in order; set by first call to addSourceMetrics
         self._repoDTypeList = None  # numpy dtype for repoArr, as a list of (key, type);
         # set by first call to addSourceMetrics
  
         self._tempDataList = []  # list (one entry per repository)
         # of dict of source ID: tuple of data ID data concatenated with source metric data, where:
         # data ID data is in order self._idKeyTuple
         # source metric data is in order self._sourceKeyTuple
         self.repoInfoList = []  # list of repoInfo
  

Member Function Documentation

◆ addSourceMetrics()

def lsst.pipe.tasks.repositoryIterator.SourceData.addSourceMetrics	(	self,
		repoInfo,
		idKeyTuple,
		idValList,
		sourceTableList
	)

Accumulate source measurements from a list of source tables.

Once you have accumulated all source measurements, call finalize to process the data.

@param[in] repoInfo: a RepositoryInfo instance
@param[in] idKeyTuple: a tuple of data ID keys; must be the same for each call
@param[in] idValList: a list of data ID value tuples;
    each tuple contains values in the order in idKeyTuple
@param[in] sourceTableList: a list of source tables, one per entry in idValList

@raise RuntimeError if idKeyTuple is different than it was for the first call.

Accumulates the data in temporary cache self._tempDataList.

@return number of sources

Definition at line 153 of file repositoryIterator.py.

     def addSourceMetrics(self, repoInfo, idKeyTuple, idValList, sourceTableList):
         """Accumulate source measurements from a list of source tables.
  
         Once you have accumulated all source measurements, call finalize to process the data.
  
         @param[in] repoInfo: a RepositoryInfo instance
         @param[in] idKeyTuple: a tuple of data ID keys; must be the same for each call
         @param[in] idValList: a list of data ID value tuples;
             each tuple contains values in the order in idKeyTuple
         @param[in] sourceTableList: a list of source tables, one per entry in idValList
  
         @raise RuntimeError if idKeyTuple is different than it was for the first call.
  
         Accumulates the data in temporary cache self._tempDataList.
  
         @return number of sources
         """
         if self._repoKeyTuple is None:
             self._repoKeyTuple = repoInfo.keyTuple
             self._repoDTypeList = repoInfo.dtype
  
         dataDict = self._getSourceMetrics(idKeyTuple, idValList, sourceTableList)
  
         self._tempDataList.append(dataDict)
         self.repoInfoList.append(repoInfo)
         return len(dataDict)
  

◆ finalize()

def lsst.pipe.tasks.repositoryIterator.SourceData.finalize ( self )

Process the accumulated source measurements to create the final data products.

Only call this after you have added all source metrics using addSourceMetrics.

Reads temporary cache self._tempDataList and then deletes it.

Definition at line 180 of file repositoryIterator.py.

     def finalize(self):
         """Process the accumulated source measurements to create the final data products.
  
         Only call this after you have added all source metrics using addSourceMetrics.
  
         Reads temporary cache self._tempDataList and then deletes it.
         """
         if len(self._tempDataList) == 0:
             raise RuntimeError("No data found")
  
         fullSrcIdSet = set()
         for dataIdDict in self._tempDataList:
             fullSrcIdSet.update(iter(dataIdDict.keys()))
  
         # source data
         sourceArrDType = [("sourceId", int)] + self._idKeyDTypeList + self._sourceDTypeList
         # data for missing sources (only for the data in the source data dict, so excludes srcId)
         nullSourceTuple = tuple(numpy.zeros(1, dtype=self._idKeyDTypeList + self._sourceDTypeList)[0])
  
         sourceData = [[(srcId,) + srcDataDict.get(srcId, nullSourceTuple) for srcId in fullSrcIdSet]
                       for srcDataDict in self._tempDataList]
  
         self.sourceArr = numpy.array(sourceData, dtype=sourceArrDType)
         del sourceData
  
         self.sourceIdDict = dict((srcId, i) for i, srcId in enumerate(fullSrcIdSet))
  
         # repository data
         repoData = [repoInfo.valTuple for repoInfo in self.repoInfoList]
         self.repoArr = numpy.array(repoData, dtype=self._repoDTypeList)
  
         self._tempDataList = None
  
  

Member Data Documentation

◆ datasetType

lsst.pipe.tasks.repositoryIterator.SourceData.datasetType

Definition at line 84 of file repositoryIterator.py.

◆ repoArr

lsst.pipe.tasks.repositoryIterator.SourceData.repoArr

Definition at line 209 of file repositoryIterator.py.

◆ repoInfoList

lsst.pipe.tasks.repositoryIterator.SourceData.repoInfoList

Definition at line 100 of file repositoryIterator.py.

◆ sourceArr

lsst.pipe.tasks.repositoryIterator.SourceData.sourceArr

Definition at line 202 of file repositoryIterator.py.

◆ sourceIdDict

lsst.pipe.tasks.repositoryIterator.SourceData.sourceIdDict

Definition at line 205 of file repositoryIterator.py.

The documentation for this class was generated from the following file:

/j/snowflake/release/lsstsw/stack/lsst-scipipe-0.7.0/Linux64/pipe_tasks/21.0.0-172-gfb10e10a+18fedfabac/python/lsst/pipe/tasks/repositoryIterator.py

Public Member Functions

Public Attributes