Introduction

lsst::pipe::base provides a data processing pipeline infrastructure. Data processing is performed by "tasks", which are instances of Task or CmdLineTask. Tasks perform a wide range of data processing operations, from basic operations such as assembling raw images into CCD images (trimming overscan), fitting a WCS or detecting sources on an image, to complex combinations of such operations.

Tasks are hierarchical. Each task may may call other tasks to perform some of its data processing; we say that a "parent task" calls a "subtask". The highest-level task is called the "top-level task". To call a subtask, the parent task constructs the subtask and then calls methods on it. Thus data transfer between tasks is simply a matter of passing the data as arguments to function calls.

Command-line tasks are tasks that can be run from the command line. You might think of them as the LSST equivalent of a data processing pipeline. Despite their extra capabilities, command-line tasks can also be used as ordinary tasks and called as subtasks by other tasks. Command-line tasks are subclasses of CmdLineTask.

Each task is configured using the pex_config package, using a task-specific subclass of pex.config.config.Config. The task's configuration includes all subtasks that the task may call. As a result, it is easy to replace (or "retarget") one subtask with another. A common use for this is to provide a camera-specific variant of a particular task, e.g. use one version for SDSS imager data and another version for Subaru Hyper Superime-Cam data).

Tasks may process multiple items of data in parallel, using Python's multiprocessing library. Support for this is built into the ArgumentParser and TaskRunner.

Most tasks have a run method that performs the primary data processing. Each task's run method should return a Struct. This allows named access to returned data, which provides safer evolution than relying on the order of returned values. All task methods that return more than one or two items of data should return the data in a Struct.

Many tasks are found in the pipe_tasks package, especially tasks that use many different packages and don't seem to belong in any one of them. Tasks that are associated with a particular package should be in that package; for example the instrument signature removal task ip.isr.isrTask.IsrTask is in the ip_isr package.

pipe_base is written purely in Python. The most important contents are:

CmdLineTask: base class for pipeline tasks that can be run from the command line.
Task: base class for subtasks that are not meant to be run from the command line.
Struct: object returned by the run method of a task.
ArgumentParser: command line parser for pipeline tasks.
timeMethod: decorator to log performance information for a Task method.
TaskRunner: a class that runs command-line tasks, using multiprocessing when requested. This will work "as is" for most command-line tasks, but will need to be be subclassed if, for instance, the task's run method needs something other than a single data reference.

Running Command-Line Tasks

Each command-line task typically has a short "task runner script" to run the task in the bin/ directory of whatever package the task is defined in. This section deals with the command-line options of these task runner scripts.

Specify --help to print help. When in doubt give this a try.

The first argument to a task must be the path to the input repository (or --help). For example:

myTask.py path/to/input --id... is valid: input path is the first argument
myTask.py --id ... path/to/input is invalid: an option comes before the input path

--output specifies the path to the output repository. Some tasks also support --calib: the path to input calibration data. To shorten input, output and calib paths see Environment Variables.

Data is specified usually specified by the --id argument with key=value pairs as the value, where the keys depend on the camera and type of data. If you run the task and specify both an input data repository and --help then the printed help will show you valid keys (the input repository tells the task what kind of camera data is being processed). See Specifying Data IDs for more information about data IDs. A few tasks take more than one kind of data ID, or have renamed the --id argument; run the task with --help or see the task's documentation for details.

You may show the config, subtasks and/or data using --show. By default --show quits after printing the information, but --show run allows the task to run. For example:

--show config data tasks shows the config, data and subtasks, and then quits.
--show tasks run show the subtasks and then runs the task.

For long or repetitive command lines you may wish to specify some arguments in separate text files. See Argument Files for details.

Specifying Data IDs

--id and other data identifier arguments are used to specify IDs for input and output data. The ID keys depend on the camera and on the kind of data being processed. For example, lsstSim calibrated exposures are identified by the following keys: visit, filter, raft and sensor (and a given visit has exactly one filter).

Omit a key to specify all values of that key. For example, for lsstSim calibrated exposures:

--id visit=54123 specifies all rafts and sensors for visit 54123 (and all filters, but there is just one filter per visit).
--id visit=54123 raft=1,0 specifies all sensors for visit raft 1,0 of visit 54123

To specify multiple data IDs you may separate values with ^ (a character that does not have special meaning to the unix command parser). The result is the outer product (all possible combinations). For example:

--id visit=54123^55523 raft=1,1^2,1 specifies four IDs: visits 54123 and 55523 of rafts 1,1 and 2,1

You may specify a data identifier argument as many times as you like. Each one is treated independently. Thus the following example specifies all sensors for four combinations of visit and raft, plus all sensors for one raft of two other visits for calibrated lsstSim data:

--id visit=54123^55523 raft=1,1^2,1 --id visit=623459^293423 raft=0,0

Argument Files

You may specify long or repetitive command-line arguments in text files and reference those files using @path.

The contents of the files are identical to the command line, except that long lines must not have a \ continuation character. For example if the file foo contains:

--id visit=54123^55523 raft=1,1^2,1

--config someParam=someValue --configfile configOverrideFilePath

you can then reference it with @@foo and mix that with other command-line arguments (including --id and --config):

myTask.py inputPath @@foo --config anotherParam=anotherValue --output outputPath

Overriding Configuration Parameters

The argument parser automatically loads specific configuration override files based on the camera name and its obs_ package. See Automatically Loaded Config Override Files. The format of a configuration override file matches the configuration shown using --show config (in particular, note that config in a configuration override file is the word that matches self.config in a task when the task uses its config).

In addition, you can specify configuration override files on the command line using --configfile and override some (but not all) configuration parameters directly on the command line using --config, as shown in these examples:

--config str1=foo str2="fancier string" int1=5 intList=2,4,-87 float1=1.53 floatList=3.14,-5.6e7
--configfile config.py, where file config.py contains:
config.strList = "first string", "second string"

Note: config in a configuration override file is equivalent to self.config in a task.

There are important limitations --config (use --configfile, instead, in these situations):

You cannot retarget subtasks
For items in registries, you can only specify values for the active (current) item
You cannot specify values for lists of strings
You cannot specify a subset of list; you must specify all values at once

Retargeting Subtasks

As a special case of overriding configuration parameters, users may replace one subtask with another; this is called "retargeting" the subtask. One common use case is to use a camera-specific variant of a subtask. Examples include:

lsst.obs.subaru.isr.SuprimeCamIsrTask: a version of instrument signature removal (ISR or detrending) for Suprime-Cam and Hyper Suprime-Cam
lsst.obs.sdss.selectSdssImages.SelectSdssImagesTask: an version of the task that selects images for co-addition of SDSS stripe 82 images

Here is an example of retargeting a subtask in a config override file; this retargets lsst.pipe.tasks.exampleCmdLineTask.ExampleCmdLineTask stats with a simpler version:

    from lsst.pipe.tasks.exampleStatsTasks import ExampleSimpleStatsTask
    config.stats.retarget(ExampleSimpleStatsTask)

Specifying Debug Variables

Some tasks support debug variables that can be set, while running from the command line, to display additional information. Each task documents which debug variables it supports. See Using lsstDebug to control debugging output for information about how to enable specific debug variables while running from the command line.

Automatically Loaded Config Override Files

When a pipeline task is run, two camera-specific configuration override files are loaded, if found; first one for the obs_ package then one for the camera. (There are two because some obs_ packages contain data for multiple cameras). These files may override configuration parameters or even retarget subtasks with camera-specific variants (e.g. for instrument signature removal). The configuration override files are, in order:

obs_path/config/task_name.py
obs_path/config/camera_name/task_name.py

where the path elements are:

task_name: the name of the pipeline task (the value of its _DefaultName class variable), e.g. "processCcd" for lsst.pipe.tasks.processCcd.ProcessCcdTask.
camera_name: the name of the camera, e.g. "lsstSim"
obs_path: the path to the obs_ package for the camera, e.g. the path to obs_lsstSim

Here are two examples:

obs_lsstSim/config/makeCoaddTempExp.py: specifies which version of the image selector task to use for co-adding LSST simulated images
obs_subaru/config/hsc/isr.py: provides overrides for the instrument signature removal (aka detrending) task for Hyper Suprime-Cam

Environment Variables

The command parser uses environment variables PIPE_INPUT_ROOT, PIPE_CALIB_ROOT, and PIPE_OUTPUT_ROOT, if available, to make it easier to specify the input, calib and output data repositories. Each environment variable is used as a root directory for relative paths and ignored for absolute paths. The default value for each of these environment variables is the current working directory. For example:

mytask foo # use $PIPE_INPUT_ROOT/foo as the input repository (or ./foo if $PIPE_INPUT_ROOT is undefined)
mytask . # use $PIPE_INPUT_ROOT (= $PIPE_INPUT_ROOT/.) as the input repository
mytask /a/b # use /a/b as the input repository ($PIPE_INPUT_ROOT is ignored for absolute paths)

Other Resources

pipe_tasks introduction includes links to:

A page listing documentation for many tasks
A manual on how to write a task
A manual on how to write a command-line task

Contents