LSSTApplications  16.0-1-gce273f5+18,16.0-10-gc1446dd+17,16.0-12-g569485f+1,16.0-12-g6065a46,16.0-13-g5e87145+1,16.0-13-g80874fd,16.0-13-gb122224+8,16.0-13-gd9b1b71+17,16.0-14-g08f9460,16.0-15-g77ef378+3,16.0-16-gf910c00+1,16.0-17-g0bdc215+10,16.0-17-g6a7bfb3b+17,16.0-2-g0febb12+14,16.0-2-g839ba83+55,16.0-2-g9d5294e+43,16.0-2-gc6e0ed0+1,16.0-21-g8fa7d5a4,16.0-3-g404ea43+10,16.0-3-gbc759ec+15,16.0-3-gcfd6c53+41,16.0-4-g03cf288+32,16.0-4-g13a27c5+18,16.0-4-g5f3a788+13,16.0-4-g8a0f11a+39,16.0-4-ga3eb747+3,16.0-5-g1991253+17,16.0-5-g1e9226d+1,16.0-5-g865efd9+18,16.0-5-gb3f8a4b+49,16.0-5-gd0f1235+7,16.0-51-gb4b4a8566,16.0-6-gf9cb114+19,16.0-7-g6043bfc+5,16.0-7-ga8e1655+13,16.0-7-gde5bd64,16.0-8-g4dec96c+30,16.0-8-gfd407c0+3,16.0-9-g2f60796,master-g5768c874b9+2,w.2018.41
How to Write a Task

This document describes how to write a data processing task.

For an introduction to data processing tasks, please read pipe_base introduction. It also helps to have a basic understanding of data repositories and how to use the butler to read and write data (to be written; for now read existing tasks to see how it is done).

After reading this document you may wish to read How to Write a Command-Line Task for additional information needed to write a command-line task.



Tasks are subclasses of lsst.pipe.base.Task or (command-line tasks) lsst.pipe.base.CmdLineTask.

Tasks are constructed by calling __init__ with the task configuration. Occasionally additional arguments are required (see the task's documentation for details). lsst.pipe.base.Task.__init__ has a few other arguments that are usually only specified when a task is created as a subtask of another task; you will probably never have to specify them yourself. (Command-line tasks are often constructed and run by calling parseAndRun, as described in How to Write a Command-Line Task.)

Tasks typically have a run method that executes the task's main function. See Methods for more information.


Every task requires a configuration: a task-specific set of configuration parameters. The configuration is read-only; once you construct a task, the same configuration will be used to process all data with that task. This makes the data processing more predictable: it does not depend on the order in which items of data are processed.

The task's configuration is specified using the pex_config package, as a task-specific subclass of lsst.pex.config.Config. The task class specifies its configuration class using class variable ConfigClass. If the task has no configuration parameters then it may use lsst.pex.config.Config as its configuration class.

Some important details of configurations:

ExampleSigmaClippedStatsTask uses configuration class ExampleSigmaClippedStatsConfig:

The configuration class is specified as ExampleSigmaClippedStatsTask class variable ConfigClass, as described in Class Variables.

Here is the config for ExampleCmdLineTask, a task that calls one subtask named "stats"; notice the lsst.pex.config.ConfigurableField:

Class Variables

Tasks require several class variables to function:

Here are the class variables for exampleCmdLineTask.ExampleCmdLineTask:


Tasks have the following important methods:

These methods are described in more depth below:

The __init__ Method

Use the __init__ method (task constructor) to do the following:

Here is exampleCmdLineTask.ExampleCmdLineTask.__init__:

That task creates a subtask named stats to compute image statistics. Here is the __init__ method for the default version of the stats subtask: exampleTask.ExampleSigmaClippedStatsTask, which is slightly more interesting:

This creates a binary mask identifying bad pixels in the mask plane and an lsst.afw.math.StatisticsControl, specifying how statistics are computed. Both of these are constants, and thus are the same for each invocation of the run method; this is strongly recommended, as explained in the next section.

The run Method

Most tasks have a run method which perform's the task's data processing operation. In addition, command-line tasks require a runDataRef method that fetches the appropriate data products specified by Gen-2 Butler DataRefs. This is required for command-line tasks and strongly recommended for most other tasks. One exception is if your task needs different methods to handle different data types (C++ handles this using overloaded functions, but the standard technique is Python is to provide different methods for different call signatures).

If your task's processing can be divided into logical units, then we recommend that you provide methods for each unit. run can then call each method to do its work. This allows your task to be more easily adapted: a subclass can override just a few methods.

We strongly recommend that you make your task stateless, by not using instance variables as part of your data processing. Pass data between methods by calling and returning it. This makes the task much easier to reason about, since processing one item of data cannot affect future items of data.

The run and runDataRef methods should always return its results in an lsst.pipe.base.struct.Struct object, with a named field for each item of data. This is safer than returning a tuple of items, and allows adding fields without affecting existing code. Other methods should also return Structs if they return more than one or two items.

Any method that is likely to take significant time or memory should be preceded by this python decorator: @lsst.pipe.base.timeMethod. This automatically records the execution time and memory of the method in the task's metadata attribute.

The example exampleCmdLineTask.ExampleCmdLineTask is so simple that it needs no other methods; runDataRef does everything:

The statistics are actually computed by the "stats" subtask. Here is the run method for the default version of that task:

Debug Variables

Debug variables are variables the user may set while running your task, to enable additional debug output. To have your task support debug variables, have it import lsstDebug and call lsstDebug.Info(__name__).varname to get the debug variable varname specific to your task. If you look for a variable the user has not specified, it will have a value of False. For example, to look for a debug variable named "display":

import lsstDebug
display = lsstDebug.Info(__name__).display
if display:

See Using lsstDebug to control debugging output for more information about debug variables, including how to specify them while running a command-line task.


For others to use your task, it must be clearly documented. pipe/tasks/ and pipe/tasks/ provide useful examples and documentation templates.

Content should include:

Use """! instead of """ to start doc strings (i.e. include an exclamation mark). This causes Doxygen to parse Doxygen commands in the doc string, which is almost always what you want.

Include a section such as the following in your task's documentation. This will make it appear on the page Task Documentation, even if your task is not in the pipe_tasks package.

## \addtogroup LSST_task_documentation
## \{
## \page exampleCmdLineTask
## \ref ExampleCmdLineTask "ExampleCmdLineTask"
##      An example intended to show how to write a command-line task.
## \}


Each subtask is specified in the configuration as a field of type lsst.pex.config.ConfigurableField or (less commonly) lsst.pex.config.RegistryField. There are advantages to each:

Our recommendation: if you anticipate that users will often wish to override the subtask with a variant, then use an lsst.pex.config.RegistryField. Otherwise use an lsst.pex.config.ConfigurableField to keep config overrides simpler and easier to read.

For example PSF determiners and star selectors are perhaps best specified using lsst.pex.config.RegistryField because there are several variants users may wish to select from. However, calibration and instrument signature removal are best specified using lsst.pex.config.ConfigurableField because (for a given camera) there is likely to be only one logical variant, and that variant is specified in a camera-specific configuration override file, so the user need not specify it.