LSST Applications  22.0.1,22.0.1+01bcf6a671,22.0.1+046ee49490,22.0.1+05c7de27da,22.0.1+0c6914dbf6,22.0.1+1220d50b50,22.0.1+12fd109e95,22.0.1+1a1dd69893,22.0.1+1c910dc348,22.0.1+1ef34551f5,22.0.1+30170c3d08,22.0.1+39153823fd,22.0.1+611137eacc,22.0.1+771eb1e3e8,22.0.1+94e66cc9ed,22.0.1+9a075d06e2,22.0.1+a5ff6e246e,22.0.1+a7db719c1a,22.0.1+ba0d97e778,22.0.1+bfe1ee9056,22.0.1+c4e1e0358a,22.0.1+cc34b8281e,22.0.1+d640e2c0fa,22.0.1+d72a2e677a,22.0.1+d9a6b571bd,22.0.1+e485e9761b,22.0.1+ebe8d3385e
LSST Data Management Base Package
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | List of all members
lsst.pipe.base.cmdLineTask.ButlerInitializedTaskRunner Class Reference
Inheritance diagram for lsst.pipe.base.cmdLineTask.ButlerInitializedTaskRunner:
lsst.pipe.base.cmdLineTask.TaskRunner lsst.pipe.drivers.singleFrameDriver.SingleFrameTaskRunner

Public Member Functions

def makeTask (self, parsedCmd=None, args=None)
 
def prepareForMultiProcessing (self)
 
def run (self, parsedCmd)
 
def precall (self, parsedCmd)
 
def __call__ (self, args)
 
def runTask (self, task, dataRef, kwargs)
 

Static Public Member Functions

def getTargetList (parsedCmd, **kwargs)
 

Public Attributes

 TaskClass
 
 doReturnResults
 
 config
 
 log
 
 doRaise
 
 clobberConfig
 
 doBackup
 
 numProcesses
 
 timeout
 

Static Public Attributes

int TIMEOUT = 3600*24*30
 

Detailed Description

A `TaskRunner` for `CmdLineTask`\ s that require a ``butler`` keyword
argument to be passed to their constructor.

Definition at line 510 of file cmdLineTask.py.

Member Function Documentation

◆ __call__()

def lsst.pipe.base.cmdLineTask.TaskRunner.__call__ (   self,
  args 
)
inherited
Run the Task on a single target.

Parameters
----------
args
    Arguments for Task.runDataRef()

Returns
-------
struct : `lsst.pipe.base.Struct`
    Contains these fields if ``doReturnResults`` is `True`:

    - ``dataRef``: the provided data reference.
    - ``metadata``: task metadata after execution of run.
    - ``result``: result returned by task run, or `None` if the task
      fails.
    - ``exitStatus``: 0 if the task completed successfully, 1
      otherwise.

    If ``doReturnResults`` is `False` the struct contains:

    - ``exitStatus``: 0 if the task completed successfully, 1
      otherwise.

Notes
-----
This default implementation assumes that the ``args`` is a tuple
containing a data reference and a dict of keyword arguments.

.. warning::

   If you override this method and wish to return something when
   ``doReturnResults`` is `False`, then it must be picklable to
   support multiprocessing and it should be small enough that pickling
   and unpickling do not add excessive overhead.

Reimplemented in lsst.pipe.drivers.constructCalibs.CalibTaskRunner.

Definition at line 380 of file cmdLineTask.py.

380  def __call__(self, args):
381  """Run the Task on a single target.
382 
383  Parameters
384  ----------
385  args
386  Arguments for Task.runDataRef()
387 
388  Returns
389  -------
390  struct : `lsst.pipe.base.Struct`
391  Contains these fields if ``doReturnResults`` is `True`:
392 
393  - ``dataRef``: the provided data reference.
394  - ``metadata``: task metadata after execution of run.
395  - ``result``: result returned by task run, or `None` if the task
396  fails.
397  - ``exitStatus``: 0 if the task completed successfully, 1
398  otherwise.
399 
400  If ``doReturnResults`` is `False` the struct contains:
401 
402  - ``exitStatus``: 0 if the task completed successfully, 1
403  otherwise.
404 
405  Notes
406  -----
407  This default implementation assumes that the ``args`` is a tuple
408  containing a data reference and a dict of keyword arguments.
409 
410  .. warning::
411 
412  If you override this method and wish to return something when
413  ``doReturnResults`` is `False`, then it must be picklable to
414  support multiprocessing and it should be small enough that pickling
415  and unpickling do not add excessive overhead.
416  """
417  dataRef, kwargs = args
418  if self.log is None:
419  self.log = Log.getDefaultLogger()
420  if hasattr(dataRef, "dataId"):
421  self.log.MDC("LABEL", str(dataRef.dataId))
422  elif isinstance(dataRef, (list, tuple)):
423  self.log.MDC("LABEL", str([ref.dataId for ref in dataRef if hasattr(ref, "dataId")]))
424  task = self.makeTask(args=args)
425  result = None # in case the task fails
426  exitStatus = 0 # exit status for the shell
427  if self.doRaise:
428  result = self.runTask(task, dataRef, kwargs)
429  else:
430  try:
431  result = self.runTask(task, dataRef, kwargs)
432  except Exception as e:
433  # The shell exit value will be the number of dataRefs returning
434  # non-zero, so the actual value used here is lost.
435  exitStatus = 1
436 
437  # don't use a try block as we need to preserve the original
438  # exception
439  eName = type(e).__name__
440  if hasattr(dataRef, "dataId"):
441  task.log.fatal("Failed on dataId=%s: %s: %s", dataRef.dataId, eName, e)
442  elif isinstance(dataRef, (list, tuple)):
443  task.log.fatal("Failed on dataIds=[%s]: %s: %s",
444  ", ".join(str(ref.dataId) for ref in dataRef), eName, e)
445  else:
446  task.log.fatal("Failed on dataRef=%s: %s: %s", dataRef, eName, e)
447 
448  if not isinstance(e, TaskError):
449  traceback.print_exc(file=sys.stderr)
450 
451  # Ensure all errors have been logged and aren't hanging around in a
452  # buffer
453  sys.stdout.flush()
454  sys.stderr.flush()
455 
456  task.writeMetadata(dataRef)
457 
458  # remove MDC so it does not show up outside of task context
459  self.log.MDCRemove("LABEL")
460 
461  if self.doReturnResults:
462  return Struct(
463  exitStatus=exitStatus,
464  dataRef=dataRef,
465  metadata=task.metadata,
466  result=result,
467  )
468  else:
469  return Struct(
470  exitStatus=exitStatus,
471  )
472 
table::Key< int > type
Definition: Detector.cc:163

◆ getTargetList()

def lsst.pipe.base.cmdLineTask.TaskRunner.getTargetList (   parsedCmd,
**  kwargs 
)
staticinherited
Get a list of (dataRef, kwargs) for `TaskRunner.__call__`.

Parameters
----------
parsedCmd : `argparse.Namespace`
    The parsed command object returned by
    `lsst.pipe.base.argumentParser.ArgumentParser.parse_args`.
kwargs
    Any additional keyword arguments. In the default `TaskRunner` this
    is an empty dict, but having it simplifies overriding `TaskRunner`
    for tasks whose runDataRef method takes additional arguments
    (see case (1) below).

Notes
-----
The default implementation of `TaskRunner.getTargetList` and
`TaskRunner.__call__` works for any command-line task whose
``runDataRef`` method takes exactly one argument: a data reference.
Otherwise you must provide a variant of TaskRunner that overrides
`TaskRunner.getTargetList` and possibly `TaskRunner.__call__`.
There are two cases.

**Case 1**

If your command-line task has a ``runDataRef`` method that takes one
data reference followed by additional arguments, then you need only
override `TaskRunner.getTargetList` to return the additional
arguments as an argument dict. To make this easier, your overridden
version of `~TaskRunner.getTargetList` may call
`TaskRunner.getTargetList` with the extra arguments as keyword
arguments. For example, the following adds an argument dict containing
a single key: "calExpList", whose value is the list of data IDs for
the calexp ID argument:

.. code-block:: python

    def getTargetList(parsedCmd):
        return TaskRunner.getTargetList(
            parsedCmd,
            calExpList=parsedCmd.calexp.idList
        )

It is equivalent to this slightly longer version:

.. code-block:: python

    @staticmethod
    def getTargetList(parsedCmd):
        argDict = dict(calExpList=parsedCmd.calexp.idList)
        return [(dataId, argDict) for dataId in parsedCmd.id.idList]

**Case 2**

If your task does not meet condition (1) then you must override both
TaskRunner.getTargetList and `TaskRunner.__call__`. You may do this
however you see fit, so long as `TaskRunner.getTargetList`
returns a list, each of whose elements is sent to
`TaskRunner.__call__`, which runs your task.

Reimplemented in lsst.pipe.tasks.multiBandUtils.MergeSourcesRunner, lsst.pipe.drivers.utils.ButlerTaskRunner, and lsst.pipe.drivers.constructCalibs.CalibTaskRunner.

Definition at line 253 of file cmdLineTask.py.

253  def getTargetList(parsedCmd, **kwargs):
254  """Get a list of (dataRef, kwargs) for `TaskRunner.__call__`.
255 
256  Parameters
257  ----------
258  parsedCmd : `argparse.Namespace`
259  The parsed command object returned by
260  `lsst.pipe.base.argumentParser.ArgumentParser.parse_args`.
261  kwargs
262  Any additional keyword arguments. In the default `TaskRunner` this
263  is an empty dict, but having it simplifies overriding `TaskRunner`
264  for tasks whose runDataRef method takes additional arguments
265  (see case (1) below).
266 
267  Notes
268  -----
269  The default implementation of `TaskRunner.getTargetList` and
270  `TaskRunner.__call__` works for any command-line task whose
271  ``runDataRef`` method takes exactly one argument: a data reference.
272  Otherwise you must provide a variant of TaskRunner that overrides
273  `TaskRunner.getTargetList` and possibly `TaskRunner.__call__`.
274  There are two cases.
275 
276  **Case 1**
277 
278  If your command-line task has a ``runDataRef`` method that takes one
279  data reference followed by additional arguments, then you need only
280  override `TaskRunner.getTargetList` to return the additional
281  arguments as an argument dict. To make this easier, your overridden
282  version of `~TaskRunner.getTargetList` may call
283  `TaskRunner.getTargetList` with the extra arguments as keyword
284  arguments. For example, the following adds an argument dict containing
285  a single key: "calExpList", whose value is the list of data IDs for
286  the calexp ID argument:
287 
288  .. code-block:: python
289 
290  def getTargetList(parsedCmd):
291  return TaskRunner.getTargetList(
292  parsedCmd,
293  calExpList=parsedCmd.calexp.idList
294  )
295 
296  It is equivalent to this slightly longer version:
297 
298  .. code-block:: python
299 
300  @staticmethod
301  def getTargetList(parsedCmd):
302  argDict = dict(calExpList=parsedCmd.calexp.idList)
303  return [(dataId, argDict) for dataId in parsedCmd.id.idList]
304 
305  **Case 2**
306 
307  If your task does not meet condition (1) then you must override both
308  TaskRunner.getTargetList and `TaskRunner.__call__`. You may do this
309  however you see fit, so long as `TaskRunner.getTargetList`
310  returns a list, each of whose elements is sent to
311  `TaskRunner.__call__`, which runs your task.
312  """
313  return [(ref, kwargs) for ref in parsedCmd.id.refList]
314 

◆ makeTask()

def lsst.pipe.base.cmdLineTask.ButlerInitializedTaskRunner.makeTask (   self,
  parsedCmd = None,
  args = None 
)
A variant of the base version that passes a butler argument to the
task's constructor.

Parameters
----------
parsedCmd : `argparse.Namespace`
    Parsed command-line options, as returned by the
    `~lsst.pipe.base.ArgumentParser`; if specified then args is
    ignored.
args
    Other arguments; if ``parsedCmd`` is `None` then this must be
    specified.

Raises
------
RuntimeError
    Raised if ``parsedCmd`` and ``args`` are both `None`.

Reimplemented from lsst.pipe.base.cmdLineTask.TaskRunner.

Definition at line 515 of file cmdLineTask.py.

515  def makeTask(self, parsedCmd=None, args=None):
516  """A variant of the base version that passes a butler argument to the
517  task's constructor.
518 
519  Parameters
520  ----------
521  parsedCmd : `argparse.Namespace`
522  Parsed command-line options, as returned by the
523  `~lsst.pipe.base.ArgumentParser`; if specified then args is
524  ignored.
525  args
526  Other arguments; if ``parsedCmd`` is `None` then this must be
527  specified.
528 
529  Raises
530  ------
531  RuntimeError
532  Raised if ``parsedCmd`` and ``args`` are both `None`.
533  """
534  if parsedCmd is not None:
535  butler = parsedCmd.butler
536  elif args is not None:
537  dataRef, kwargs = args
538  butler = dataRef.butlerSubset.butler
539  else:
540  raise RuntimeError("parsedCmd or args must be specified")
541  return self.TaskClass(config=self.config, log=self.log, butler=butler)
542 
543 

◆ precall()

def lsst.pipe.base.cmdLineTask.TaskRunner.precall (   self,
  parsedCmd 
)
inherited
Hook for code that should run exactly once, before multiprocessing.

Notes
-----
Must return True if `TaskRunner.__call__` should subsequently be
called.

.. warning::

   Implementations must take care to ensure that no unpicklable
   attributes are added to the TaskRunner itself, for compatibility
   with multiprocessing.

The default implementation writes package versions, schemas and
configs, or compares them to existing files on disk if present.

Definition at line 349 of file cmdLineTask.py.

349  def precall(self, parsedCmd):
350  """Hook for code that should run exactly once, before multiprocessing.
351 
352  Notes
353  -----
354  Must return True if `TaskRunner.__call__` should subsequently be
355  called.
356 
357  .. warning::
358 
359  Implementations must take care to ensure that no unpicklable
360  attributes are added to the TaskRunner itself, for compatibility
361  with multiprocessing.
362 
363  The default implementation writes package versions, schemas and
364  configs, or compares them to existing files on disk if present.
365  """
366  task = self.makeTask(parsedCmd=parsedCmd)
367 
368  if self.doRaise:
369  self._precallImpl(task, parsedCmd)
370  else:
371  try:
372  self._precallImpl(task, parsedCmd)
373  except Exception as e:
374  task.log.fatal("Failed in task initialization: %s", e)
375  if not isinstance(e, TaskError):
376  traceback.print_exc(file=sys.stderr)
377  return False
378  return True
379 

◆ prepareForMultiProcessing()

def lsst.pipe.base.cmdLineTask.TaskRunner.prepareForMultiProcessing (   self)
inherited
Prepare this instance for multiprocessing

Optional non-picklable elements are removed.

This is only called if the task is run under multiprocessing.

Definition at line 193 of file cmdLineTask.py.

193  def prepareForMultiProcessing(self):
194  """Prepare this instance for multiprocessing
195 
196  Optional non-picklable elements are removed.
197 
198  This is only called if the task is run under multiprocessing.
199  """
200  self.log = None
201 

◆ run()

def lsst.pipe.base.cmdLineTask.TaskRunner.run (   self,
  parsedCmd 
)
inherited
Run the task on all targets.

Parameters
----------
parsedCmd : `argparse.Namespace`
    Parsed command `argparse.Namespace`.

Returns
-------
resultList : `list`
    A list of results returned by `TaskRunner.__call__`, or an empty
    list if `TaskRunner.__call__` is not called (e.g. if
    `TaskRunner.precall` returns `False`). See `TaskRunner.__call__`
    for details.

Notes
-----
The task is run under multiprocessing if `TaskRunner.numProcesses`
is more than 1; otherwise processing is serial.

Reimplemented in lsst.ctrl.pool.parallel.BatchTaskRunner.

Definition at line 202 of file cmdLineTask.py.

202  def run(self, parsedCmd):
203  """Run the task on all targets.
204 
205  Parameters
206  ----------
207  parsedCmd : `argparse.Namespace`
208  Parsed command `argparse.Namespace`.
209 
210  Returns
211  -------
212  resultList : `list`
213  A list of results returned by `TaskRunner.__call__`, or an empty
214  list if `TaskRunner.__call__` is not called (e.g. if
215  `TaskRunner.precall` returns `False`). See `TaskRunner.__call__`
216  for details.
217 
218  Notes
219  -----
220  The task is run under multiprocessing if `TaskRunner.numProcesses`
221  is more than 1; otherwise processing is serial.
222  """
223  resultList = []
224  disableImplicitThreading() # To prevent thread contention
225  if self.numProcesses > 1:
226  import multiprocessing
227  self.prepareForMultiProcessing()
228  pool = multiprocessing.Pool(processes=self.numProcesses, maxtasksperchild=1)
229  mapFunc = functools.partial(_runPool, pool, self.timeout)
230  else:
231  pool = None
232  mapFunc = map
233 
234  if self.precall(parsedCmd):
235  profileName = parsedCmd.profile if hasattr(parsedCmd, "profile") else None
236  log = parsedCmd.log
237  targetList = self.getTargetList(parsedCmd)
238  if len(targetList) > 0:
239  with profile(profileName, log):
240  # Run the task using self.__call__
241  resultList = list(mapFunc(self, targetList))
242  else:
243  log.warn("Not running the task because there is no data to process; "
244  "you may preview data using \"--show data\"")
245 
246  if pool is not None:
247  pool.close()
248  pool.join()
249 
250  return resultList
251 
daf::base::PropertyList * list
Definition: fits.cc:913
bool disableImplicitThreading()
Disable threading that has not been set explicitly.
Definition: threads.cc:132
def profile(filename, log=None)
Definition: cmdLineTask.py:50
def run(self, skyInfo, tempExpRefList, imageScalerList, weightList, altMaskList=None, mask=None, supplementaryData=None)

◆ runTask()

def lsst.pipe.base.cmdLineTask.TaskRunner.runTask (   self,
  task,
  dataRef,
  kwargs 
)
inherited
Make the actual call to `runDataRef` for this task.

Parameters
----------
task : `lsst.pipe.base.CmdLineTask` class
    The class of the task to run.
dataRef
    Butler data reference that contains the data the task will process.
kwargs
    Any additional keyword arguments.  See `TaskRunner.getTargetList`
    above.

Notes
-----
The default implementation of `TaskRunner.runTask` works for any
command-line task which has a ``runDataRef`` method that takes a data
reference and an optional set of additional keyword arguments.
This method returns the results generated by the task's `runDataRef`
method.

Reimplemented in lsst.pipe.base.cmdLineTask.LegacyTaskRunner.

Definition at line 473 of file cmdLineTask.py.

473  def runTask(self, task, dataRef, kwargs):
474  """Make the actual call to `runDataRef` for this task.
475 
476  Parameters
477  ----------
478  task : `lsst.pipe.base.CmdLineTask` class
479  The class of the task to run.
480  dataRef
481  Butler data reference that contains the data the task will process.
482  kwargs
483  Any additional keyword arguments. See `TaskRunner.getTargetList`
484  above.
485 
486  Notes
487  -----
488  The default implementation of `TaskRunner.runTask` works for any
489  command-line task which has a ``runDataRef`` method that takes a data
490  reference and an optional set of additional keyword arguments.
491  This method returns the results generated by the task's `runDataRef`
492  method.
493 
494  """
495  return task.runDataRef(dataRef, **kwargs)
496 
497 

Member Data Documentation

◆ clobberConfig

lsst.pipe.base.cmdLineTask.TaskRunner.clobberConfig
inherited

Definition at line 180 of file cmdLineTask.py.

◆ config

lsst.pipe.base.cmdLineTask.TaskRunner.config
inherited

Definition at line 177 of file cmdLineTask.py.

◆ doBackup

lsst.pipe.base.cmdLineTask.TaskRunner.doBackup
inherited

Definition at line 181 of file cmdLineTask.py.

◆ doRaise

lsst.pipe.base.cmdLineTask.TaskRunner.doRaise
inherited

Definition at line 179 of file cmdLineTask.py.

◆ doReturnResults

lsst.pipe.base.cmdLineTask.TaskRunner.doReturnResults
inherited

Definition at line 176 of file cmdLineTask.py.

◆ log

lsst.pipe.base.cmdLineTask.TaskRunner.log
inherited

Definition at line 178 of file cmdLineTask.py.

◆ numProcesses

lsst.pipe.base.cmdLineTask.TaskRunner.numProcesses
inherited

Definition at line 182 of file cmdLineTask.py.

◆ TaskClass

lsst.pipe.base.cmdLineTask.TaskRunner.TaskClass
inherited

Definition at line 175 of file cmdLineTask.py.

◆ TIMEOUT

int lsst.pipe.base.cmdLineTask.TaskRunner.TIMEOUT = 3600*24*30
staticinherited

Definition at line 171 of file cmdLineTask.py.

◆ timeout

lsst.pipe.base.cmdLineTask.TaskRunner.timeout
inherited

Definition at line 184 of file cmdLineTask.py.


The documentation for this class was generated from the following file: