LSSTApplications
16.0-10-g0ee56ad+5,16.0-11-ga33d1f2+5,16.0-12-g3ef5c14+3,16.0-12-g71e5ef5+18,16.0-12-gbdf3636+3,16.0-13-g118c103+3,16.0-13-g8f68b0a+3,16.0-15-gbf5c1cb+4,16.0-16-gfd17674+3,16.0-17-g7c01f5c+3,16.0-18-g0a50484+1,16.0-20-ga20f992+8,16.0-21-g0e05fd4+6,16.0-21-g15e2d33+4,16.0-22-g62d8060+4,16.0-22-g847a80f+4,16.0-25-gf00d9b8+1,16.0-28-g3990c221+4,16.0-3-gf928089+3,16.0-32-g88a4f23+5,16.0-34-gd7987ad+3,16.0-37-gc7333cb+2,16.0-4-g10fc685+2,16.0-4-g18f3627+26,16.0-4-g5f3a788+26,16.0-5-gaf5c3d7+4,16.0-5-gcc1f4bb+1,16.0-6-g3b92700+4,16.0-6-g4412fcd+3,16.0-6-g7235603+4,16.0-69-g2562ce1b+2,16.0-8-g14ebd58+4,16.0-8-g2df868b+1,16.0-8-g4cec79c+6,16.0-8-gadf6c7a+1,16.0-8-gfc7ad86,16.0-82-g59ec2a54a+1,16.0-9-g5400cdc+2,16.0-9-ge6233d7+5,master-g2880f2d8cf+3,v17.0.rc1
LSSTDataManagementBasePackage
|
Public Member Functions | |
def | __init__ (self, TaskClass, parsedCmd, doReturnResults=False) |
def | prepareForMultiProcessing (self) |
def | run (self, parsedCmd) |
def | makeTask (self, parsedCmd=None, args=None) |
def | precall (self, parsedCmd) |
def | __call__ (self, args) |
def | runTask (self, task, dataRef, kwargs) |
Static Public Member Functions | |
def | getTargetList (parsedCmd, kwargs) |
Public Attributes | |
TaskClass | |
doReturnResults | |
config | |
log | |
doRaise | |
clobberConfig | |
doBackup | |
numProcesses | |
timeout | |
Static Public Attributes | |
int | TIMEOUT = 3600*24*30 |
Run a command-line task, using `multiprocessing` if requested. Parameters ---------- TaskClass : `lsst.pipe.base.Task` subclass The class of the task to run. parsedCmd : `argparse.Namespace` The parsed command-line arguments, as returned by the task's argument parser's `~lsst.pipe.base.ArgumentParser.parse_args` method. .. warning:: Do not store ``parsedCmd``, as this instance is pickled (if multiprocessing) and parsedCmd may contain non-picklable elements. It certainly contains more data than we need to send to each instance of the task. doReturnResults : `bool`, optional Should run return the collected result from each invocation of the task? This is only intended for unit tests and similar use. It can easily exhaust memory (if the task returns enough data and you call it enough times) and it will fail when using multiprocessing if the returned data cannot be pickled. Note that even if ``doReturnResults`` is False a struct with a single member "exitStatus" is returned, with value 0 or 1 to be returned to the unix shell. Raises ------ ImportError If multiprocessing is requested (and the task supports it) but the multiprocessing library cannot be imported. Notes ----- Each command-line task (subclass of `lsst.pipe.base.CmdLineTask`) has a task runner. By default it is this class, but some tasks require a subclass. See the manual :ref:`creating-a-command-line-task` for more information. See `CmdLineTask.parseAndRun` to see how a task runner is used. You may use this task runner for your command-line task if your task has a runDataRef method that takes exactly one argument: a butler data reference. Otherwise you must provide a task-specific subclass of this runner for your task's ``RunnerClass`` that overrides `TaskRunner.getTargetList` and possibly `TaskRunner.__call__`. See `TaskRunner.getTargetList` for details. This design matches the common pattern for command-line tasks: the runDataRef method takes a single data reference, of some suitable name. Additional arguments are rare, and if present, require a subclass of `TaskRunner` that calls these additional arguments by name. Instances of this class must be picklable in order to be compatible with multiprocessing. If multiprocessing is requested (``parsedCmd.numProcesses > 1``) then `runDataRef` calls `prepareForMultiProcessing` to jettison optional non-picklable elements. If your task runner is not compatible with multiprocessing then indicate this in your task by setting class variable ``canMultiprocess=False``. Due to a `python bug`__, handling a `KeyboardInterrupt` properly `requires specifying a timeout`__. This timeout (in sec) can be specified as the ``timeout`` element in the output from `~lsst.pipe.base.ArgumentParser` (the ``parsedCmd``), if available, otherwise we use `TaskRunner.TIMEOUT`. By default, we disable "implicit" threading -- ie, as provided by underlying numerical libraries such as MKL or BLAS. This is designed to avoid thread contention both when a single command line task spawns multiple processes and when multiple users are running on a shared system. Users can override this behaviour by setting the ``LSST_ALLOW_IMPLICIT_THREADS`` environment variable. .. __: http://bugs.python.org/issue8296 .. __: http://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool
Definition at line 87 of file cmdLineTask.py.
def lsst.pipe.base.cmdLineTask.TaskRunner.__init__ | ( | self, | |
TaskClass, | |||
parsedCmd, | |||
doReturnResults = False |
|||
) |
Definition at line 155 of file cmdLineTask.py.
def lsst.pipe.base.cmdLineTask.TaskRunner.__call__ | ( | self, | |
args | |||
) |
Run the Task on a single target. Parameters ---------- args Arguments for Task.runDataRef() Returns ------- struct : `lsst.pipe.base.Struct` Contains these fields if ``doReturnResults`` is `True`: - ``dataRef``: the provided data reference. - ``metadata``: task metadata after execution of run. - ``result``: result returned by task run, or `None` if the task fails. - ``exitStatus``: 0 if the task completed successfully, 1 otherwise. If ``doReturnResults`` is `False` the struct contains: - ``exitStatus``: 0 if the task completed successfully, 1 otherwise. Notes ----- This default implementation assumes that the ``args`` is a tuple containing a data reference and a dict of keyword arguments. .. warning:: If you override this method and wish to return something when ``doReturnResults`` is `False`, then it must be picklable to support multiprocessing and it should be small enough that pickling and unpickling do not add excessive overhead.
Definition at line 341 of file cmdLineTask.py.
|
static |
Get a list of (dataRef, kwargs) for `TaskRunner.__call__`. Parameters ---------- parsedCmd : `argparse.Namespace` The parsed command object returned by `lsst.pipe.base.argumentParser.ArgumentParser.parse_args`. kwargs Any additional keyword arguments. In the default `TaskRunner` this is an empty dict, but having it simplifies overriding `TaskRunner` for tasks whose runDataRef method takes additional arguments (see case (1) below). Notes ----- The default implementation of `TaskRunner.getTargetList` and `TaskRunner.__call__` works for any command-line task whose runDataRef method takes exactly one argument: a data reference. Otherwise you must provide a variant of TaskRunner that overrides `TaskRunner.getTargetList` and possibly `TaskRunner.__call__`. There are two cases. **Case 1** If your command-line task has a ``runDataRef`` method that takes one data reference followed by additional arguments, then you need only override `TaskRunner.getTargetList` to return the additional arguments as an argument dict. To make this easier, your overridden version of `~TaskRunner.getTargetList` may call `TaskRunner.getTargetList` with the extra arguments as keyword arguments. For example, the following adds an argument dict containing a single key: "calExpList", whose value is the list of data IDs for the calexp ID argument:: def getTargetList(parsedCmd): return TaskRunner.getTargetList( parsedCmd, calExpList=parsedCmd.calexp.idList ) It is equivalent to this slightly longer version:: @staticmethod def getTargetList(parsedCmd): argDict = dict(calExpList=parsedCmd.calexp.idList) return [(dataId, argDict) for dataId in parsedCmd.id.idList] **Case 2** If your task does not meet condition (1) then you must override both TaskRunner.getTargetList and `TaskRunner.__call__`. You may do this however you see fit, so long as `TaskRunner.getTargetList` returns a list, each of whose elements is sent to `TaskRunner.__call__`, which runs your task.
Definition at line 233 of file cmdLineTask.py.
def lsst.pipe.base.cmdLineTask.TaskRunner.makeTask | ( | self, | |
parsedCmd = None , |
|||
args = None |
|||
) |
Create a Task instance. Parameters ---------- parsedCmd Parsed command-line options (used for extra task args by some task runners). args Args tuple passed to `TaskRunner.__call__` (used for extra task arguments by some task runners). Notes ----- ``makeTask`` can be called with either the ``parsedCmd`` argument or ``args`` argument set to None, but it must construct identical Task instances in either case. Subclasses may ignore this method entirely if they reimplement both `TaskRunner.precall` and `TaskRunner.__call__`.
Definition at line 282 of file cmdLineTask.py.
def lsst.pipe.base.cmdLineTask.TaskRunner.precall | ( | self, | |
parsedCmd | |||
) |
Hook for code that should run exactly once, before multiprocessing. Notes ----- Must return True if `TaskRunner.__call__` should subsequently be called. .. warning:: Implementations must take care to ensure that no unpicklable attributes are added to the TaskRunner itself, for compatibility with multiprocessing. The default implementation writes package versions, schemas and configs, or compares them to existing files on disk if present.
Definition at line 312 of file cmdLineTask.py.
def lsst.pipe.base.cmdLineTask.TaskRunner.prepareForMultiProcessing | ( | self | ) |
Prepare this instance for multiprocessing Optional non-picklable elements are removed. This is only called if the task is run under multiprocessing.
Definition at line 174 of file cmdLineTask.py.
def lsst.pipe.base.cmdLineTask.TaskRunner.run | ( | self, | |
parsedCmd | |||
) |
Run the task on all targets. Parameters ---------- parsedCmd : `argparse.Namespace` Parsed command `argparse.Namespace`. Returns ------- resultList : `list` A list of results returned by `TaskRunner.__call__`, or an empty list if `TaskRunner.__call__` is not called (e.g. if `TaskRunner.precall` returns `False`). See `TaskRunner.__call__` for details. Notes ----- The task is run under multiprocessing if `TaskRunner.numProcesses` is more than 1; otherwise processing is serial.
Definition at line 183 of file cmdLineTask.py.
def lsst.pipe.base.cmdLineTask.TaskRunner.runTask | ( | self, | |
task, | |||
dataRef, | |||
kwargs | |||
) |
Make the actual call to `runDataRef` for this task. Parameters ---------- task : `lsst.pipe.base.CmdLineTask` class The class of the task to run. dataRef Butler data reference that contains the data the task will process. kwargs Any additional keyword arguments. See `TaskRunner.getTargetList` above. Notes ----- The default implementation of `TaskRunner.runTask` works for any command-line task which has a runDataRef method that takes a data reference and an optional set of additional keyword arguments. This method returns the results generated by the task's `runDataRef` method.
Definition at line 428 of file cmdLineTask.py.
lsst.pipe.base.cmdLineTask.TaskRunner.clobberConfig |
Definition at line 161 of file cmdLineTask.py.
lsst.pipe.base.cmdLineTask.TaskRunner.config |
Definition at line 158 of file cmdLineTask.py.
lsst.pipe.base.cmdLineTask.TaskRunner.doBackup |
Definition at line 162 of file cmdLineTask.py.
lsst.pipe.base.cmdLineTask.TaskRunner.doRaise |
Definition at line 160 of file cmdLineTask.py.
lsst.pipe.base.cmdLineTask.TaskRunner.doReturnResults |
Definition at line 157 of file cmdLineTask.py.
lsst.pipe.base.cmdLineTask.TaskRunner.log |
Definition at line 159 of file cmdLineTask.py.
lsst.pipe.base.cmdLineTask.TaskRunner.numProcesses |
Definition at line 163 of file cmdLineTask.py.
lsst.pipe.base.cmdLineTask.TaskRunner.TaskClass |
Definition at line 156 of file cmdLineTask.py.
|
static |
Definition at line 152 of file cmdLineTask.py.
lsst.pipe.base.cmdLineTask.TaskRunner.timeout |
Definition at line 165 of file cmdLineTask.py.