LSSTApplications  17.0+11,17.0+35,17.0+60,17.0+61,17.0+63,17.0+7,17.0-1-g377950a+35,17.0.1-1-g114240f+2,17.0.1-1-g4d4fbc4+30,17.0.1-1-g55520dc+55,17.0.1-1-g5f4ed7e+59,17.0.1-1-g6dd7d69+22,17.0.1-1-g8de6c91+11,17.0.1-1-gb9095d2+7,17.0.1-1-ge9fec5e+5,17.0.1-1-gf4e0155+63,17.0.1-1-gfc65f5f+56,17.0.1-1-gfc6fb1f+20,17.0.1-10-g87f9f3f+9,17.0.1-12-g112a4bc+3,17.0.1-17-gab9750a3+5,17.0.1-17-gdae4c4a+16,17.0.1-19-g3a24bb2+2,17.0.1-2-g26618f5+35,17.0.1-2-g54f2ebc+9,17.0.1-2-gf403422+1,17.0.1-21-g52a398f+5,17.0.1-26-gd98a1d13,17.0.1-3-g7e86b59+45,17.0.1-3-gb5ca14a,17.0.1-3-gd08d533+46,17.0.1-31-gb0791f330,17.0.1-4-g59d126d+10,17.0.1-5-g3877d06+2,17.0.1-7-g35889ee+7,17.0.1-7-gc7c8782+20,17.0.1-7-gcb7da53+5,17.0.1-9-gc4bbfb2+10,w.2019.24
LSSTDataManagementBasePackage
Public Member Functions | Public Attributes | List of all members
lsst.pipe.base.task.Task Class Reference
Inheritance diagram for lsst.pipe.base.task.Task:
lsst.ip.isr.crosstalk.CrosstalkTask lsst.ip.isr.fringe.FringeTask lsst.ip.isr.masking.MaskingTask lsst.ip.isr.straylight.StrayLightTask lsst.ip.isr.vignette.VignetteTask lsst.meas.algorithms.measureApCorr.MeasureApCorrTask lsst.meas.algorithms.reserveSourcesTask.ReserveSourcesTask lsst.meas.algorithms.skyObjects.SkyObjectsTask lsst.meas.astrom.directMatch.DirectMatchTask lsst.meas.astrom.fitSipDistortion.FitSipDistortionTask lsst.meas.base.applyApCorr.ApplyApCorrTask lsst.meas.base.baseMeasurement.BaseMeasurementTask lsst.meas.base.catalogCalculation.CatalogCalculationTask lsst.meas.base.references.BaseReferencesTask lsst.obs.base.gen3.bootstrapRepo.BootstrapRepoTask lsst.obs.base.gen3.ingest.RawIngestTask lsst.pipe.base.cmdLineTask.CmdLineTask lsst.pipe.base.pipelineTask.PipelineTask lsst.pipe.drivers.background.MaskObjectsTask lsst.pipe.drivers.background.SkyMeasurementTask lsst.pipe.drivers.constructCalibs.CalibCombineTask lsst.pipe.drivers.constructCalibs.CalibStatsTask lsst.pipe.tasks.fakes.BaseFakeSourcesTask lsst.pipe.tasks.ingest.IngestTask lsst.pipe.tasks.ingest.ParseTask lsst.pipe.tasks.ingest.RegisterTask lsst.pipe.tasks.mocks.mockObject.MockObjectTask lsst.pipe.tasks.mocks.mockObservation.MockObservationTask lsst.pipe.tasks.mocks.mockSelect.MockSelectImagesTask lsst.pipe.tasks.propagateVisitFlags.PropagateVisitFlagsTask lsst.pipe.tasks.registerImage.RegisterTask lsst.pipe.tasks.scaleVariance.ScaleVarianceTask lsst.pipe.tasks.setPrimaryFlags.SetPrimaryFlagsTask

Public Member Functions

def __init__ (self, config=None, name=None, parentTask=None, log=None)
 
def emptyMetadata (self)
 
def getSchemaCatalogs (self)
 
def getAllSchemaCatalogs (self)
 
def getFullMetadata (self)
 
def getFullName (self)
 
def getName (self)
 
def getTaskDict (self)
 
def makeSubtask (self, name, keyArgs)
 
def timer (self, name, logLevel=Log.DEBUG)
 
def makeField (cls, doc)
 
def __reduce__ (self)
 

Public Attributes

 metadata
 
 log
 
 config
 

Detailed Description

Base class for data processing tasks.

See :ref:`task-framework-overview` to learn what tasks are, and :ref:`creating-a-task` for more
information about writing tasks.

Parameters
----------
config : `Task.ConfigClass` instance, optional
    Configuration for this task (an instance of Task.ConfigClass, which is a task-specific subclass of
    `lsst.pex.config.Config`, or `None`. If `None`:

    - If parentTask specified then defaults to parentTask.config.\<name>
    - If parentTask is None then defaults to self.ConfigClass()

name : `str`, optional
    Brief name of task, or `None`; if `None` then defaults to `Task._DefaultName`
parentTask : `Task`-type, optional
    The parent task of this subtask, if any.

    - If `None` (a top-level task) then you must specify config and name is ignored.
    - If not `None` (a subtask) then you must specify name.
log : `lsst.log.Log`, optional
    Log whose name is used as a log name prefix, or `None` for no prefix. Ignored if is parentTask
    specified, in which case ``parentTask.log``\ 's name is used as a prefix. The task's log name is
    ``prefix + "." + name`` if a prefix exists, else ``name``. The task's log is then a child logger of
    ``parentTask.log`` (if ``parentTask`` specified), or a child logger of the log from the argument
    (if ``log`` is not `None`).

Raises
------
RuntimeError
    Raised under these circumstances:

    - If ``parentTask`` is `None` and ``config`` is `None`.
    - If ``parentTask`` is not `None` and ``name`` is `None`.
    - If ``name`` is `None` and ``_DefaultName`` does not exist.

Notes
-----
Useful attributes include:

- ``log``: an lsst.log.Log
- ``config``: task-specific configuration; an instance of ``ConfigClass`` (see below).
- ``metadata``: an `lsst.daf.base.PropertyList` for collecting task-specific metadata,
    e.g. data quality and performance metrics. This is data that is only meant to be
    persisted, never to be used by the task.

Subclasses typically have a method named ``runDataRef`` to perform the main data processing. Details:

- ``runDataRef`` should process the minimum reasonable amount of data, typically a single CCD.
  Iteration, if desired, is performed by a caller of the method. This is good design and allows
  multiprocessing without the run method having to support it directly.
- If ``runDataRef`` can persist or unpersist data:

  - ``runDataRef`` should accept a butler data reference (or a collection of data references,
    if appropriate, e.g. coaddition).
  - There should be a way to run the task without persisting data. Typically the run method returns all
    data, even if it is persisted, and the task's config method offers a flag to disable persistence.

**Deprecated:** Tasks other than cmdLineTask.CmdLineTask%s should *not* accept a blob such as a butler
data reference.  How we will handle data references is still TBD, so don't make changes yet!
RHL 2014-06-27

Subclasses must also have an attribute ``ConfigClass`` that is a subclass of `lsst.pex.config.Config`
which configures the task. Subclasses should also have an attribute ``_DefaultName``:
the default name if there is no parent task. ``_DefaultName`` is required for subclasses of
`~lsst.pipe.base.CmdLineTask` and recommended for subclasses of Task because it simplifies construction
(e.g. for unit tests).

Tasks intended to be run from the command line should be subclasses of `~lsst.pipe.base.CmdLineTask`
not Task.

Definition at line 46 of file task.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.pipe.base.task.Task.__init__ (   self,
  config = None,
  name = None,
  parentTask = None,
  log = None 
)

Definition at line 120 of file task.py.

120  def __init__(self, config=None, name=None, parentTask=None, log=None):
121  self.metadata = dafBase.PropertyList()
122  self._parentTask = parentTask
123 
124  if parentTask is not None:
125  if name is None:
126  raise RuntimeError("name is required for a subtask")
127  self._name = name
128  self._fullName = parentTask._computeFullName(name)
129  if config is None:
130  config = getattr(parentTask.config, name)
131  self._taskDict = parentTask._taskDict
132  loggerName = parentTask.log.getName() + '.' + name
133  else:
134  if name is None:
135  name = getattr(self, "_DefaultName", None)
136  if name is None:
137  raise RuntimeError("name is required for a task unless it has attribute _DefaultName")
138  name = self._DefaultName
139  self._name = name
140  self._fullName = self._name
141  if config is None:
142  config = self.ConfigClass()
143  self._taskDict = dict()
144  loggerName = self._fullName
145  if log is not None and log.getName():
146  loggerName = log.getName() + '.' + loggerName
147 
148  self.log = Log.getLogger(loggerName)
149  self.config = config
150  self._display = lsstDebug.Info(self.__module__).display
151  self._taskDict[self._fullName] = self
152 
Class for storing ordered metadata with comments.
Definition: PropertyList.h:68
def __init__(self, minimum, dataRange, Q)

Member Function Documentation

◆ __reduce__()

def lsst.pipe.base.task.Task.__reduce__ (   self)
Pickler.

Definition at line 373 of file task.py.

373  def __reduce__(self):
374  """Pickler.
375  """
376  return self.__class__, (self.config, self._name, self._parentTask, None)
377 

◆ emptyMetadata()

def lsst.pipe.base.task.Task.emptyMetadata (   self)
Empty (clear) the metadata for this Task and all sub-Tasks.

Definition at line 153 of file task.py.

153  def emptyMetadata(self):
154  """Empty (clear) the metadata for this Task and all sub-Tasks.
155  """
156  for subtask in self._taskDict.values():
157  subtask.metadata = dafBase.PropertyList()
158 
Class for storing ordered metadata with comments.
Definition: PropertyList.h:68

◆ getAllSchemaCatalogs()

def lsst.pipe.base.task.Task.getAllSchemaCatalogs (   self)
Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.

Returns
-------
schemacatalogs : `dict`
    Keys are butler dataset type, values are a empty catalog (an instance of the appropriate
    lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down
    through all subtasks.

Notes
-----
This method may be called on any task in the hierarchy; it will return the same answer, regardless.

The default implementation should always suffice. If your subtask uses schemas the override
`Task.getSchemaCatalogs`, not this method.

Definition at line 188 of file task.py.

188  def getAllSchemaCatalogs(self):
189  """Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.
190 
191  Returns
192  -------
193  schemacatalogs : `dict`
194  Keys are butler dataset type, values are a empty catalog (an instance of the appropriate
195  lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down
196  through all subtasks.
197 
198  Notes
199  -----
200  This method may be called on any task in the hierarchy; it will return the same answer, regardless.
201 
202  The default implementation should always suffice. If your subtask uses schemas the override
203  `Task.getSchemaCatalogs`, not this method.
204  """
205  schemaDict = self.getSchemaCatalogs()
206  for subtask in self._taskDict.values():
207  schemaDict.update(subtask.getSchemaCatalogs())
208  return schemaDict
209 

◆ getFullMetadata()

def lsst.pipe.base.task.Task.getFullMetadata (   self)
Get metadata for all tasks.

Returns
-------
metadata : `lsst.daf.base.PropertySet`
    The `~lsst.daf.base.PropertySet` keys are the full task name. Values are metadata
    for the top-level task and all subtasks, sub-subtasks, etc..

Notes
-----
The returned metadata includes timing information (if ``@timer.timeMethod`` is used)
and any metadata set by the task. The name of each item consists of the full task name
with ``.`` replaced by ``:``, followed by ``.`` and the name of the item, e.g.::

    topLevelTaskName:subtaskName:subsubtaskName.itemName

using ``:`` in the full task name disambiguates the rare situation that a task has a subtask
and a metadata item with the same name.

Definition at line 210 of file task.py.

210  def getFullMetadata(self):
211  """Get metadata for all tasks.
212 
213  Returns
214  -------
215  metadata : `lsst.daf.base.PropertySet`
216  The `~lsst.daf.base.PropertySet` keys are the full task name. Values are metadata
217  for the top-level task and all subtasks, sub-subtasks, etc..
218 
219  Notes
220  -----
221  The returned metadata includes timing information (if ``@timer.timeMethod`` is used)
222  and any metadata set by the task. The name of each item consists of the full task name
223  with ``.`` replaced by ``:``, followed by ``.`` and the name of the item, e.g.::
224 
225  topLevelTaskName:subtaskName:subsubtaskName.itemName
226 
227  using ``:`` in the full task name disambiguates the rare situation that a task has a subtask
228  and a metadata item with the same name.
229  """
230  fullMetadata = dafBase.PropertySet()
231  for fullName, task in self.getTaskDict().items():
232  fullMetadata.set(fullName.replace(".", ":"), task.metadata)
233  return fullMetadata
234 
Class for storing generic metadata.
Definition: PropertySet.h:68
std::vector< SchemaItem< Flag > > * items

◆ getFullName()

def lsst.pipe.base.task.Task.getFullName (   self)
Get the task name as a hierarchical name including parent task names.

Returns
-------
fullName : `str`
    The full name consists of the name of the parent task and each subtask separated by periods.
    For example:

    - The full name of top-level task "top" is simply "top".
    - The full name of subtask "sub" of top-level task "top" is "top.sub".
    - The full name of subtask "sub2" of subtask "sub" of top-level task "top" is "top.sub.sub2".

Definition at line 235 of file task.py.

235  def getFullName(self):
236  """Get the task name as a hierarchical name including parent task names.
237 
238  Returns
239  -------
240  fullName : `str`
241  The full name consists of the name of the parent task and each subtask separated by periods.
242  For example:
243 
244  - The full name of top-level task "top" is simply "top".
245  - The full name of subtask "sub" of top-level task "top" is "top.sub".
246  - The full name of subtask "sub2" of subtask "sub" of top-level task "top" is "top.sub.sub2".
247  """
248  return self._fullName
249 

◆ getName()

def lsst.pipe.base.task.Task.getName (   self)
Get the name of the task.

Returns
-------
taskName : `str`
    Name of the task.

See also
--------
getFullName

Definition at line 250 of file task.py.

250  def getName(self):
251  """Get the name of the task.
252 
253  Returns
254  -------
255  taskName : `str`
256  Name of the task.
257 
258  See also
259  --------
260  getFullName
261  """
262  return self._name
263 

◆ getSchemaCatalogs()

def lsst.pipe.base.task.Task.getSchemaCatalogs (   self)
Get the schemas generated by this task.

Returns
-------
schemaCatalogs : `dict`
    Keys are butler dataset type, values are an empty catalog (an instance of the appropriate
    `lsst.afw.table` Catalog type) for this task.

Notes
-----

.. warning::

   Subclasses that use schemas must override this method. The default implemenation returns
   an empty dict.

This method may be called at any time after the Task is constructed, which means that all task
schemas should be computed at construction time, *not* when data is actually processed. This
reflects the philosophy that the schema should not depend on the data.

Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.

See also
--------
Task.getAllSchemaCatalogs

Definition at line 159 of file task.py.

159  def getSchemaCatalogs(self):
160  """Get the schemas generated by this task.
161 
162  Returns
163  -------
164  schemaCatalogs : `dict`
165  Keys are butler dataset type, values are an empty catalog (an instance of the appropriate
166  `lsst.afw.table` Catalog type) for this task.
167 
168  Notes
169  -----
170 
171  .. warning::
172 
173  Subclasses that use schemas must override this method. The default implemenation returns
174  an empty dict.
175 
176  This method may be called at any time after the Task is constructed, which means that all task
177  schemas should be computed at construction time, *not* when data is actually processed. This
178  reflects the philosophy that the schema should not depend on the data.
179 
180  Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.
181 
182  See also
183  --------
184  Task.getAllSchemaCatalogs
185  """
186  return {}
187 

◆ getTaskDict()

def lsst.pipe.base.task.Task.getTaskDict (   self)
Get a dictionary of all tasks as a shallow copy.

Returns
-------
taskDict : `dict`
    Dictionary containing full task name: task object for the top-level task and all subtasks,
    sub-subtasks, etc..

Definition at line 264 of file task.py.

264  def getTaskDict(self):
265  """Get a dictionary of all tasks as a shallow copy.
266 
267  Returns
268  -------
269  taskDict : `dict`
270  Dictionary containing full task name: task object for the top-level task and all subtasks,
271  sub-subtasks, etc..
272  """
273  return self._taskDict.copy()
274 
def getTaskDict(config, taskDict=None, baseName="")

◆ makeField()

def lsst.pipe.base.task.Task.makeField (   cls,
  doc 
)
Make a `lsst.pex.config.ConfigurableField` for this task.

Parameters
----------
doc : `str`
    Help text for the field.

Returns
-------
configurableField : `lsst.pex.config.ConfigurableField`
    A `~ConfigurableField` for this task.

Examples
--------
Provides a convenient way to specify this task is a subtask of another task.

Here is an example of use::

    class OtherTaskConfig(lsst.pex.config.Config)
aSubtask = ATaskClass.makeField("a brief description of what this task does")

Definition at line 329 of file task.py.

329  def makeField(cls, doc):
330  """Make a `lsst.pex.config.ConfigurableField` for this task.
331 
332  Parameters
333  ----------
334  doc : `str`
335  Help text for the field.
336 
337  Returns
338  -------
339  configurableField : `lsst.pex.config.ConfigurableField`
340  A `~ConfigurableField` for this task.
341 
342  Examples
343  --------
344  Provides a convenient way to specify this task is a subtask of another task.
345 
346  Here is an example of use::
347 
348  class OtherTaskConfig(lsst.pex.config.Config)
349  aSubtask = ATaskClass.makeField("a brief description of what this task does")
350  """
351  return ConfigurableField(doc=doc, target=cls)
352 

◆ makeSubtask()

def lsst.pipe.base.task.Task.makeSubtask (   self,
  name,
  keyArgs 
)
Create a subtask as a new instance as the ``name`` attribute of this task.

Parameters
----------
name : `str`
    Brief name of the subtask.
keyArgs
    Extra keyword arguments used to construct the task. The following arguments are automatically
    provided and cannot be overridden:

    - "config".
    - "parentTask".

Notes
-----
The subtask must be defined by ``Task.config.name``, an instance of pex_config ConfigurableField
or RegistryField.

Definition at line 275 of file task.py.

275  def makeSubtask(self, name, **keyArgs):
276  """Create a subtask as a new instance as the ``name`` attribute of this task.
277 
278  Parameters
279  ----------
280  name : `str`
281  Brief name of the subtask.
282  keyArgs
283  Extra keyword arguments used to construct the task. The following arguments are automatically
284  provided and cannot be overridden:
285 
286  - "config".
287  - "parentTask".
288 
289  Notes
290  -----
291  The subtask must be defined by ``Task.config.name``, an instance of pex_config ConfigurableField
292  or RegistryField.
293  """
294  taskField = getattr(self.config, name, None)
295  if taskField is None:
296  raise KeyError("%s's config does not have field %r" % (self.getFullName(), name))
297  subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
298  setattr(self, name, subtask)
299 

◆ timer()

def lsst.pipe.base.task.Task.timer (   self,
  name,
  logLevel = Log.DEBUG 
)
Context manager to log performance data for an arbitrary block of code.

Parameters
----------
name : `str`
    Name of code being timed; data will be logged using item name: ``Start`` and ``End``.
logLevel
    A `lsst.log` level constant.

Examples
--------
Creating a timer context::

    with self.timer("someCodeToTime"):
pass  # code to time

See also
--------
timer.logInfo

Definition at line 301 of file task.py.

301  def timer(self, name, logLevel=Log.DEBUG):
302  """Context manager to log performance data for an arbitrary block of code.
303 
304  Parameters
305  ----------
306  name : `str`
307  Name of code being timed; data will be logged using item name: ``Start`` and ``End``.
308  logLevel
309  A `lsst.log` level constant.
310 
311  Examples
312  --------
313  Creating a timer context::
314 
315  with self.timer("someCodeToTime"):
316  pass # code to time
317 
318  See also
319  --------
320  timer.logInfo
321  """
322  logInfo(obj=self, prefix=name + "Start", logLevel=logLevel)
323  try:
324  yield
325  finally:
326  logInfo(obj=self, prefix=name + "End", logLevel=logLevel)
327 
def logInfo(obj, prefix, logLevel=Log.DEBUG)
Definition: timer.py:62

Member Data Documentation

◆ config

lsst.pipe.base.task.Task.config

Definition at line 149 of file task.py.

◆ log

lsst.pipe.base.task.Task.log

Definition at line 148 of file task.py.

◆ metadata

lsst.pipe.base.task.Task.metadata

Definition at line 121 of file task.py.


The documentation for this class was generated from the following file: