Inheritance diagram for lsst.meas.base.diaCalculation.DiaObjectCalculationTask:
Public Member Functions | |
| __init__ (self, plugMetadata=None, **kwargs) | |
| initializePlugins (self) | |
| run (self, diaObjectCat, diaSourceCat, updatedDiaObjectIds, filterNames) | |
| callCompute (self, diaObjectCat, diaSourceCat, updatedDiaObjectIds, filterNames) | |
Public Attributes | |
| plugMetadata | |
| plugins | |
| outputCols | |
| executionDict | |
| log | |
Static Public Attributes | |
| ConfigClass = DiaObjectCalculationConfig | |
Protected Member Functions | |
| _validatePluginCols (self, plug) | |
| _initialize_dia_object (self, objId) | |
Static Protected Attributes | |
| str | _DefaultName = "diaObjectCalculation" |
Detailed Description
Run plugins which operate on a catalog of DIA sources.
This task facilitates running plugins which will operate on a source
catalog. These plugins may do things such as classifying an object based
on source record entries inserted during a measurement task.
This task differs from CatalogCaculationTask in the following ways:
-No multi mode is available for plugins. All plugins are assumed to run
in single mode.
-Input and output catalog types are assumed to be `pandas.DataFrames` with
columns following those used in the Apdb.
-No schema argument is passed to the plugins. Each plugin specifies
output columns and required inputs.
Parameters
----------
plugMetaData : `lsst.daf.base.PropertyList` or `None`
Will be modified in-place to contain metadata about the plugins being
run. If `None`, an empty `~lsst.daf.base.PropertyList` will be
created.
**kwargs
Additional arguments passed to the superclass constructor.
Notes
-----
Plugins may either take an entire catalog to work on at a time, or work on
individual records.
Definition at line 172 of file diaCalculation.py.
Constructor & Destructor Documentation
◆ __init__()
| lsst.meas.base.diaCalculation.DiaObjectCalculationTask.__init__ | ( | self, | |
| plugMetadata = None, | |||
| ** | kwargs ) |
Reimplemented from lsst.meas.base.catalogCalculation.CatalogCalculationTask.
Definition at line 207 of file diaCalculation.py.
207 def __init__(self, plugMetadata=None, **kwargs):
208 lsst.pipe.base.Task.__init__(self, **kwargs)
209 if plugMetadata is None:
210 plugMetadata = lsst.daf.base.PropertyList()
211 self.plugMetadata = plugMetadata
212 self.plugins = PluginMap()
213 self.outputCols = []
214
215 self.initializePlugins()
216
Class for storing ordered metadata with comments.
Definition PropertyList.h:68
Member Function Documentation
◆ _initialize_dia_object()
|
protected |
Create a new DiaObject with values required to be initialized by the
Apdb.
Parameters
----------
objid : `int`
``diaObjectId`` value for the of the new DiaObject.
Returns
-------
diaObject : `dict`
Newly created DiaObject with keys:
``diaObjectId``
Unique DiaObjectId (`int`).
``pmParallaxNdata``
Number of data points used for parallax calculation (`int`).
``nearbyObj1``
Id of the a nearbyObject in the Object table (`int`).
``nearbyObj2``
Id of the a nearbyObject in the Object table (`int`).
``nearbyObj3``
Id of the a nearbyObject in the Object table (`int`).
``?_psfFluxNdata``
Number of data points used to calculate point source flux
summary statistics in each bandpass (`int`).
Definition at line 476 of file diaCalculation.py.
476 def _initialize_dia_object(self, objId):
477 """Create a new DiaObject with values required to be initialized by the
478 Apdb.
479
480 Parameters
481 ----------
482 objid : `int`
483 ``diaObjectId`` value for the of the new DiaObject.
484
485 Returns
486 -------
487 diaObject : `dict`
488 Newly created DiaObject with keys:
489
490 ``diaObjectId``
491 Unique DiaObjectId (`int`).
492 ``pmParallaxNdata``
493 Number of data points used for parallax calculation (`int`).
494 ``nearbyObj1``
495 Id of the a nearbyObject in the Object table (`int`).
496 ``nearbyObj2``
497 Id of the a nearbyObject in the Object table (`int`).
498 ``nearbyObj3``
499 Id of the a nearbyObject in the Object table (`int`).
500 ``?_psfFluxNdata``
501 Number of data points used to calculate point source flux
502 summary statistics in each bandpass (`int`).
503 """
504 new_dia_object = {"diaObjectId": objId,
505 "pmParallaxNdata": 0,
506 "nearbyObj1": 0,
507 "nearbyObj2": 0,
508 "nearbyObj3": 0}
509 for f in ["u", "g", "r", "i", "z", "y"]:
510 new_dia_object["%s_psfFluxNdata" % f] = 0
511 return new_dia_object
◆ _validatePluginCols()
|
protected |
Assert that output columns are not duplicated and input columns
exist for dependent plugins.
Parameters
----------
plug : `lsst.ap.association.DiaCalculationPlugin`
Plugin to test for output collisions and input needs.
Definition at line 247 of file diaCalculation.py.
247 def _validatePluginCols(self, plug):
248 """Assert that output columns are not duplicated and input columns
249 exist for dependent plugins.
250
251 Parameters
252 ----------
253 plug : `lsst.ap.association.DiaCalculationPlugin`
254 Plugin to test for output collisions and input needs.
255 """
256 for inputName in plug.inputCols:
257 if inputName not in self.outputCols:
258 errorTuple = (plug.name, plug.getExecutionOrder(),
259 inputName)
260 raise ValueError(
261 "Plugin, {} with execution order {} requires DiaObject "
262 "column {} to exist. Check the execution order of the "
263 "plugin and make sure it runs after a plugin creating "
264 "the column is run.".format(*errorTuple))
265 for outputName in plug.outputCols:
266 if outputName in self.outputCols:
267 errorTuple = (plug.name, plug.getExecutionOrder(),
268 outputName)
269 raise ValueError(
270 "Plugin, {} with execution order {} is attempting to "
271 "output a column {}, however the column is already being "
272 "produced by another plugin. Check other plugins for "
273 "collisions with this one.".format(*errorTuple))
274 else:
275 self.outputCols.append(outputName)
276
◆ callCompute()
| lsst.meas.base.diaCalculation.DiaObjectCalculationTask.callCompute | ( | self, | |
| diaObjectCat, | |||
| diaSourceCat, | |||
| updatedDiaObjectIds, | |||
| filterNames ) |
Run each of the plugins on the catalog.
For catalog column names see the lsst.cat schema definitions for the
DiaObject and DiaSource tables (http://github.com/lsst/cat).
Parameters
----------
diaObjectCat : `pandas.DataFrame`
DiaObjects to update values of and append new objects to. DataFrame
should be indexed on "diaObjectId"
diaSourceCat : `pandas.DataFrame`
DiaSources associated with the DiaObjects in diaObjectCat.
DataFrame must be indexed on
["diaObjectId", "band", "diaSourceId"]`
updatedDiaObjectIds : `numpy.ndarray`
Integer ids of the DiaObjects to update and create.
filterNames : `list` of `str`
List of string names of filters to be being processed.
Returns
-------
returnStruct : `lsst.pipe.base.Struct`
Struct containing:
``diaObjectCat``
Full set of DiaObjects including both un-updated and
updated/new DiaObjects (`pandas.DataFrame`).
``updatedDiaObjects``
Catalog of DiaObjects that were updated or created by this
task (`pandas.DataFrame`).
Raises
------
KeyError
Raises if `pandas.DataFrame` indexing is not properly set.
Reimplemented from lsst.meas.base.catalogCalculation.CatalogCalculationTask.
Definition at line 346 of file diaCalculation.py.
350 filterNames):
351 """Run each of the plugins on the catalog.
352
353 For catalog column names see the lsst.cat schema definitions for the
354 DiaObject and DiaSource tables (http://github.com/lsst/cat).
355
356 Parameters
357 ----------
358 diaObjectCat : `pandas.DataFrame`
359 DiaObjects to update values of and append new objects to. DataFrame
360 should be indexed on "diaObjectId"
361 diaSourceCat : `pandas.DataFrame`
362 DiaSources associated with the DiaObjects in diaObjectCat.
363 DataFrame must be indexed on
364 ["diaObjectId", "band", "diaSourceId"]`
365 updatedDiaObjectIds : `numpy.ndarray`
366 Integer ids of the DiaObjects to update and create.
367 filterNames : `list` of `str`
368 List of string names of filters to be being processed.
369
370 Returns
371 -------
372 returnStruct : `lsst.pipe.base.Struct`
373 Struct containing:
374
375 ``diaObjectCat``
376 Full set of DiaObjects including both un-updated and
377 updated/new DiaObjects (`pandas.DataFrame`).
378 ``updatedDiaObjects``
379 Catalog of DiaObjects that were updated or created by this
380 task (`pandas.DataFrame`).
381
382 Raises
383 ------
384 KeyError
385 Raises if `pandas.DataFrame` indexing is not properly set.
386 """
387 # DiaObjects will be updated in place.
388 diaObjectsToUpdate = diaObjectCat.loc[updatedDiaObjectIds, :]
389 self.log.info("Calculating summary stats for %i DiaObjects",
390 len(diaObjectsToUpdate))
391
392 updatingDiaSources = diaSourceCat.loc[updatedDiaObjectIds, :]
393 diaSourcesGB = updatingDiaSources.groupby(level=0)
394 for runlevel in sorted(self.executionDict):
395 for plug in self.executionDict[runlevel].single:
396 if plug.needsFilter:
397 continue
398 for updatedDiaObjectId in updatedDiaObjectIds:
399
400 # Sub-select diaSources associated with this diaObject.
401 objDiaSources = updatingDiaSources.loc[updatedDiaObjectId]
402
403 # Sub-select on diaSources observed in the current filter.
404 with CCContext(plug, updatedDiaObjectId, self.log):
405 # We feed the catalog we need to update and the id
406 # so as to get a few into the catalog and not a copy.
407 # This updates the values in the catalog.
408 plug.calculate(diaObjects=diaObjectsToUpdate,
409 diaObjectId=updatedDiaObjectId,
410 diaSources=objDiaSources,
411 filterDiaSources=None,
412 band=None)
413 for plug in self.executionDict[runlevel].multi:
414 if plug.needsFilter:
415 continue
416 with CCContext(plug, diaObjectsToUpdate, self.log):
417 plug.calculate(diaObjects=diaObjectsToUpdate,
418 diaSources=diaSourcesGB,
419 filterDiaSources=None,
420 band=None)
421
422 for band in filterNames:
423 try:
424 updatingFilterDiaSources = updatingDiaSources.loc[
425 (slice(None), band), :
426 ]
427 except KeyError:
428 self.log.warning("No DiaSource data with fitler=%s. "
429 "Continuing...", band)
430 continue
431 # Level=0 here groups by diaObjectId.
432 filterDiaSourcesGB = updatingFilterDiaSources.groupby(level=0)
433
434 for runlevel in sorted(self.executionDict):
435 for plug in self.executionDict[runlevel].single:
436 if not plug.needsFilter:
437 continue
438 for updatedDiaObjectId in updatedDiaObjectIds:
439
440 # Sub-select diaSources associated with this diaObject.
441 objDiaSources = updatingDiaSources.loc[updatedDiaObjectId]
442
443 # Sub-select on diaSources observed in the current filter.
444 try:
445 filterObjDiaSources = objDiaSources.loc[band]
446 except KeyError:
447 self.log.warning(
448 "DiaObjectId={updatedDiaObjectId} has no "
449 "DiaSources for filter=%s. "
450 "Continuing...", band)
451 with CCContext(plug, updatedDiaObjectId, self.log):
452 # We feed the catalog we need to update and the id
453 # so as to get a few into the catalog and not a copy.
454 # This updates the values in the catalog.
455 plug.calculate(diaObjects=diaObjectsToUpdate,
456 diaObjectId=updatedDiaObjectId,
457 diaSources=objDiaSources,
458 filterDiaSources=filterObjDiaSources,
459 band=band)
460 for plug in self.executionDict[runlevel].multi:
461 if not plug.needsFilter:
462 continue
463 with CCContext(plug, diaObjectsToUpdate, self.log):
464 plug.calculate(diaObjects=diaObjectsToUpdate,
465 diaSources=diaSourcesGB,
466 filterDiaSources=filterDiaSourcesGB,
467 band=band)
468 # Need to store the newly updated diaObjects directly as the editing
469 # a view into diaObjectsToUpdate does not update the values of
470 # diaObjectCat.
471 diaObjectCat.loc[updatedDiaObjectIds, :] = diaObjectsToUpdate
472 return lsst.pipe.base.Struct(
473 diaObjectCat=diaObjectCat,
474 updatedDiaObjects=diaObjectsToUpdate)
475
◆ initializePlugins()
| lsst.meas.base.diaCalculation.DiaObjectCalculationTask.initializePlugins | ( | self | ) |
Initialize the plugins according to the configuration.
Reimplemented from lsst.meas.base.catalogCalculation.CatalogCalculationTask.
Definition at line 217 of file diaCalculation.py.
217 def initializePlugins(self):
218 """Initialize the plugins according to the configuration.
219 """
220
221 pluginType = namedtuple('pluginType', 'single multi')
222 self.executionDict = {}
223 # Read the properties for each plugin. Allocate a dictionary entry for
224 # each run level. Verify that the plugins are above the minimum run
225 # level for an catalogCalculation plugin. For each run level, the
226 # plugins are sorted into either single record, or multi record groups
227 # to later be run appropriately
228 for executionOrder, name, config, PluginClass in sorted(self.config.plugins.apply()):
229 if executionOrder not in self.executionDict:
230 self.executionDict[executionOrder] = pluginType(single=[], multi=[])
231 if PluginClass.getExecutionOrder() >= BasePlugin.DEFAULT_CATALOGCALCULATION:
232 plug = PluginClass(config, name, metadata=self.plugMetadata)
233
234 self._validatePluginCols(plug)
235
236 self.plugins[name] = plug
237 if plug.plugType == 'single':
238 self.executionDict[executionOrder].single.append(plug)
239 elif plug.plugType == 'multi':
240 self.executionDict[executionOrder].multi.append(plug)
241 else:
242 errorTuple = (PluginClass, PluginClass.getExecutionOrder(),
243 BasePlugin.DEFAULT_CATALOGCALCULATION)
244 raise ValueError("{} has an execution order less than the minimum for an catalogCalculation "
245 "plugin. Value {} : Minimum {}".format(*errorTuple))
246
◆ run()
| lsst.meas.base.diaCalculation.DiaObjectCalculationTask.run | ( | self, | |
| diaObjectCat, | |||
| diaSourceCat, | |||
| updatedDiaObjectIds, | |||
| filterNames ) |
The entry point for the DIA catalog calculation task.
Run method both updates the values in the diaObjectCat and appends
newly created DiaObjects to the catalog. For catalog column names
see the lsst.cat schema definitions for the DiaObject and DiaSource
tables (http://github.com/lsst/cat).
Parameters
----------
diaObjectCat : `pandas.DataFrame`
DiaObjects to update values of and append new objects to. DataFrame
should be indexed on "diaObjectId"
diaSourceCat : `pandas.DataFrame`
DiaSources associated with the DiaObjects in diaObjectCat.
DataFrame should be indexed on
`["diaObjectId", "band", "diaSourceId"]`
updatedDiaObjectIds : `numpy.ndarray`
Integer ids of the DiaObjects to update and create.
filterNames : `list` of `str`
List of string names of filters to be being processed.
Returns
-------
returnStruct : `lsst.pipe.base.Struct`
Struct containing:
``diaObjectCat``
Full set of DiaObjects including both un-updated and
updated/new DiaObjects (`pandas.DataFrame`).
``updatedDiaObjects``
Catalog of DiaObjects that were updated or created by this
task (`pandas.DataFrame`).
Reimplemented from lsst.meas.base.catalogCalculation.CatalogCalculationTask.
Definition at line 278 of file diaCalculation.py.
282 filterNames):
283 """The entry point for the DIA catalog calculation task.
284
285 Run method both updates the values in the diaObjectCat and appends
286 newly created DiaObjects to the catalog. For catalog column names
287 see the lsst.cat schema definitions for the DiaObject and DiaSource
288 tables (http://github.com/lsst/cat).
289
290 Parameters
291 ----------
292 diaObjectCat : `pandas.DataFrame`
293 DiaObjects to update values of and append new objects to. DataFrame
294 should be indexed on "diaObjectId"
295 diaSourceCat : `pandas.DataFrame`
296 DiaSources associated with the DiaObjects in diaObjectCat.
297 DataFrame should be indexed on
298 `["diaObjectId", "band", "diaSourceId"]`
299 updatedDiaObjectIds : `numpy.ndarray`
300 Integer ids of the DiaObjects to update and create.
301 filterNames : `list` of `str`
302 List of string names of filters to be being processed.
303
304 Returns
305 -------
306 returnStruct : `lsst.pipe.base.Struct`
307 Struct containing:
308
309 ``diaObjectCat``
310 Full set of DiaObjects including both un-updated and
311 updated/new DiaObjects (`pandas.DataFrame`).
312 ``updatedDiaObjects``
313 Catalog of DiaObjects that were updated or created by this
314 task (`pandas.DataFrame`).
315 """
316 if diaObjectCat.index.name is None:
317 diaObjectCat.set_index("diaObjectId", inplace=True, drop=False)
318 elif diaObjectCat.index.name != "diaObjectId":
319 self.log.warning(
320 "Input diaObjectCat is indexed on column(s) incompatible with "
321 "this task. Should be indexed on 'diaObjectId'. Trying to set "
322 "index regardless")
323 diaObjectCat.set_index("diaObjectId", inplace=True, drop=False)
324
325 # ``names`` by default is FrozenList([None]) hence we access the first
326 # element and test for None.
327 if diaSourceCat.index.names[0] is None:
328 diaSourceCat.set_index(
329 ["diaObjectId", "band", "diaSourceId"],
330 inplace=True,
331 drop=False)
332 elif (diaSourceCat.index.names
333 != ["diaObjectId", "band", "diaSourceId"]):
334 diaSourceCat.reset_index(inplace=True)
335 diaSourceCat.set_index(
336 ["diaObjectId", "band", "diaSourceId"],
337 inplace=True,
338 drop=False)
339
340 return self.callCompute(diaObjectCat,
341 diaSourceCat,
342 updatedDiaObjectIds,
343 filterNames)
344
Member Data Documentation
◆ _DefaultName
|
staticprotected |
Definition at line 205 of file diaCalculation.py.
◆ ConfigClass
|
static |
Definition at line 204 of file diaCalculation.py.
◆ executionDict
| lsst.meas.base.diaCalculation.DiaObjectCalculationTask.executionDict |
Definition at line 222 of file diaCalculation.py.
◆ log
| lsst.meas.base.diaCalculation.DiaObjectCalculationTask.log |
Definition at line 404 of file diaCalculation.py.
◆ outputCols
| lsst.meas.base.diaCalculation.DiaObjectCalculationTask.outputCols |
Definition at line 213 of file diaCalculation.py.
◆ plugins
| lsst.meas.base.diaCalculation.DiaObjectCalculationTask.plugins |
Definition at line 212 of file diaCalculation.py.
◆ plugMetadata
| lsst.meas.base.diaCalculation.DiaObjectCalculationTask.plugMetadata |
Definition at line 211 of file diaCalculation.py.
The documentation for this class was generated from the following file:
- /j/snowflake/release/lsstsw/stack/lsst-scipipe-8.0.0/Linux64/meas_base/gf18bd8381d+8d59551888/python/lsst/meas/base/diaCalculation.py
