LSSTApplications: lsst.pipe.base.cmdLineTask.CmdLineTask Class Reference

Base class for command-line tasks: tasks that may be executed from the command line. More...

Inheritance diagram for lsst.pipe.base.cmdLineTask.CmdLineTask:

Public Member Functions
def	applyOverrides
	A hook to allow a task to change the values of its config after the camera-specific overrides are loaded but before any command-line overrides are applied. More...

def	parseAndRun
	Parse an argument list and run the command. More...

def	writeConfig
	Write the configuration used for processing the data, or check that an existing one is equal to the new one if present. More...

def	writeSchemas
	Write the schemas returned by getAllSchemaCatalogs. More...

def	writeMetadata
	Write the metadata produced from processing the data. More...

Static Public Attributes
	RunnerClass = TaskRunner

	canMultiprocess = True

Private Member Functions
def	_makeArgumentParser
	Create and return an argument parser. More...

def	_getConfigName
	Return the name of the config dataset type, or None if config is not to be persisted. More...

def	_getMetadataName
	Return the name of the metadata dataset type, or None if metadata is not to be persisted. More...

Detailed Description

Base class for command-line tasks: tasks that may be executed from the command line.

See pipe_base introduction to learn what tasks are, and how to write a command-line task for more information about writing command-line tasks. If the second link is broken (as it will be before the documentation is cross-linked) then look at the main page of pipe_tasks documentation for a link.

Subclasses must specify the following class variables:

ConfigClass: configuration class for your task (a subclass of lsst.pex.config.Config, or if your task needs no configuration, then lsst.pex.config.Config itself)
_DefaultName: default name used for this task (a str)

Subclasses may also specify the following class variables:

RunnerClass: a task runner class. The default is TaskRunner, which works for any task with a run method that takes exactly one argument: a data reference. If your task does not meet this requirement then you must supply a variant of TaskRunner; see TaskRunner for more information.
canMultiprocess: the default is True; set False if your task does not support multiprocessing.

Subclasses must specify a method named "run":

By default run accepts a single butler data reference, but you can specify an alternate task runner (subclass of TaskRunner) as the value of class variable RunnerClass if your run method needs something else.
run is expected to return its data in a Struct. This provides safety for evolution of the task since new values may be added without harming existing code.
The data returned by run must be picklable if your task is to support multiprocessing.

Definition at line 356 of file cmdLineTask.py.

Member Function Documentation

def lsst.pipe.base.cmdLineTask.CmdLineTask._getConfigName ( self )

private

Return the name of the config dataset type, or None if config is not to be persisted.

Note: The name may depend on the config; that is why this is not a class method.

Definition at line 529 of file cmdLineTask.py.

 
     def _getConfigName(self):
         """!Return the name of the config dataset type, or None if config is not to be persisted
 
         @note The name may depend on the config; that is why this is not a class method.
         """
         return self._DefaultName + "_config"

lsst.pipe.base.cmdLineTask.CmdLineTask._getConfigName

def _getConfigName

Return the name of the config dataset type, or None if config is not to be persisted.

Definition: cmdLineTask.py:529

def lsst.pipe.base.cmdLineTask.CmdLineTask._getMetadataName ( self )

private

Return the name of the metadata dataset type, or None if metadata is not to be persisted.

Note: The name may depend on the config; that is why this is not a class method.

Definition at line 536 of file cmdLineTask.py.

 
     def _getMetadataName(self):
         """!Return the name of the metadata dataset type, or None if metadata is not to be persisted
 
         @note The name may depend on the config; that is why this is not a class method.
         """
         return self._DefaultName + "_metadata"

def lsst.pipe.base.cmdLineTask.CmdLineTask._makeArgumentParser ( cls )

private

Create and return an argument parser.

Parameters

[in] cls the class object

Returns: the argument parser for this task.

By default this returns an ArgumentParser with one ID argument named --id of dataset type "raw".

Your task subclass may need to override this method to change the dataset type or data ref level, or to add additional data ID arguments. If you add additional data ID arguments or your task's run method takes more than a single data reference then you will also have to provide a task-specific task runner (see TaskRunner for more information).

Definition at line 444 of file cmdLineTask.py.

 
     def _makeArgumentParser(cls):
         """!Create and return an argument parser
 
         @param[in] cls      the class object
         @return the argument parser for this task.
 
         By default this returns an ArgumentParser with one ID argument named `--id`  of dataset type "raw".
 
         Your task subclass may need to override this method to change the dataset type or data ref level,
         or to add additional data ID arguments. If you add additional data ID arguments or your task's
         run method takes more than a single data reference then you will also have to provide a task-specific
         task runner (see TaskRunner for more information).
         """
         parser = ArgumentParser(name=cls._DefaultName)
         parser.add_id_argument(name="--id", datasetType="raw", help="data ID, e.g. --id visit=12345 ccd=1,2")
         return parser

lsst.pipe.base.cmdLineTask.CmdLineTask._makeArgumentParser

def _makeArgumentParser

Create and return an argument parser.

Definition: cmdLineTask.py:444

def lsst.pipe.base.cmdLineTask.CmdLineTask.applyOverrides	(	cls,
		config
	)

A hook to allow a task to change the values of its config after the camera-specific overrides are loaded but before any command-line overrides are applied.

This is necessary in some cases because the camera-specific overrides may retarget subtasks, wiping out changes made in ConfigClass.setDefaults. See LSST Trac ticket #2282 for more discussion.

Warning: This is called by CmdLineTask.parseAndRun; other ways of constructing a config will not apply these overrides.

Parameters

[in]	cls	the class object
[in]	config	task configuration (an instance of cls.ConfigClass)

Definition at line 390 of file cmdLineTask.py.

 
     def applyOverrides(cls, config):
         """!A hook to allow a task to change the values of its config *after* the camera-specific
         overrides are loaded but before any command-line overrides are applied.
 
         This is necessary in some cases because the camera-specific overrides may retarget subtasks,
         wiping out changes made in ConfigClass.setDefaults. See LSST Trac ticket #2282 for more discussion.
 
         @warning This is called by CmdLineTask.parseAndRun; other ways of constructing a config
         will not apply these overrides.
 
         @param[in] cls      the class object
         @param[in] config   task configuration (an instance of cls.ConfigClass)
         """
         pass

lsst.pipe.base.cmdLineTask.CmdLineTask.applyOverrides

def applyOverrides

A hook to allow a task to change the values of its config after the camera-specific overrides are loa...

Definition: cmdLineTask.py:390

def lsst.pipe.base.cmdLineTask.CmdLineTask.parseAndRun	(	cls,
		args = `None`,
		config = `None`,
		log = `None`,
		doReturnResults = `False`
	)

Parse an argument list and run the command.

Calling this method with no arguments specified is the standard way to run a command-line task from the command line. For an example see pipe_tasks bin/makeSkyMap.py or almost any other file in that directory.

Parameters

cls	the class object
args	list of command-line arguments; if `None` use sys.argv
config	config for task (instance of pex_config Config); if `None` use cls.ConfigClass()
log	log (instance of lsst.pex.logging.Log); if `None` use the default log
doReturnResults	Return the collected results from each invocation of the task? This is only intended for unit tests and similar use. It can easily exhaust memory (if the task returns enough data and you call it enough times) and it will fail when using multiprocessing if the returned data cannot be pickled.

Returns

a Struct containing:

argumentParser: the argument parser
parsedCmd: the parsed command returned by the argument parser's parse_args method
taskRunner: the task runner used to run the task (an instance of cls.RunnerClass)
resultList: results returned by the task runner's run method, one entry per invocation. This will typically be a list of None unless doReturnResults is True; see cls.RunnerClass (TaskRunner by default) for more information.

Definition at line 406 of file cmdLineTask.py.

 
     def parseAndRun(cls, args=None, config=None, log=None, doReturnResults=False):
         """!Parse an argument list and run the command
 
         Calling this method with no arguments specified is the standard way to run a command-line task
         from the command line. For an example see pipe_tasks `bin/makeSkyMap.py` or almost any other
         file in that directory.
 
         @param cls      the class object
         @param args     list of command-line arguments; if `None` use sys.argv
         @param config   config for task (instance of pex_config Config); if `None` use cls.ConfigClass()
         @param log      log (instance of lsst.pex.logging.Log); if `None` use the default log
         @param doReturnResults  Return the collected results from each invocation of the task?
             This is only intended for unit tests and similar use.
             It can easily exhaust memory (if the task returns enough data and you call it enough times)
             and it will fail when using multiprocessing if the returned data cannot be pickled.
 
         @return a Struct containing:
         - argumentParser: the argument parser
         - parsedCmd: the parsed command returned by the argument parser's parse_args method
         - taskRunner: the task runner used to run the task (an instance of cls.RunnerClass)
         - resultList: results returned by the task runner's run method, one entry per invocation.
             This will typically be a list of `None` unless doReturnResults is `True`;
             see cls.RunnerClass (TaskRunner by default) for more information.
         """
         argumentParser = cls._makeArgumentParser()
         if config is None:
             config = cls.ConfigClass()
         parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
         taskRunner = cls.RunnerClass(TaskClass=cls, parsedCmd=parsedCmd, doReturnResults=doReturnResults)
         resultList = taskRunner.run(parsedCmd)
         return Struct(
             argumentParser = argumentParser,
             parsedCmd = parsedCmd,
             taskRunner = taskRunner,
             resultList = resultList,
         )

lsst.pipe.base.cmdLineTask.CmdLineTask.parseAndRun

def parseAndRun

Parse an argument list and run the command.

Definition: cmdLineTask.py:406

def lsst.pipe.base.cmdLineTask.CmdLineTask.writeConfig	(	self,
		butler,
		clobber = `False`
	)

Write the configuration used for processing the data, or check that an existing one is equal to the new one if present.

Parameters

[in]	butler	data butler used to write the config. The config is written to dataset type self._getConfigName()
[in]	clobber	a boolean flag that controls what happens if a config already has been saved: True: overwrite the existing config False: raise TaskError if this config does not match the existing config

Definition at line 461 of file cmdLineTask.py.

 
     def writeConfig(self, butler, clobber=False):
         """!Write the configuration used for processing the data, or check that an existing
         one is equal to the new one if present.
 
         @param[in] butler   data butler used to write the config.
             The config is written to dataset type self._getConfigName()
         @param[in] clobber  a boolean flag that controls what happens if a config already has been saved:
             - True: overwrite the existing config
             - False: raise TaskError if this config does not match the existing config
         """
         configName = self._getConfigName()
         if configName is None:
             return
         if clobber:
             butler.put(self.config, configName, doBackup=True)
         elif butler.datasetExists(configName):
             # this may be subject to a race condition; see #2789
             oldConfig = butler.get(configName, immediate=True)
             output = lambda msg: self.log.fatal("Comparing configuration: " + msg)
             if not self.config.compare(oldConfig, shortcut=False, output=output):
                 raise TaskError(
                     ("Config does not match existing task config %r on disk; tasks configurations " + \
                     "must be consistent within the same output repo (override with --clobber-config)") % \
                     (configName,))
         else:
             butler.put(self.config, configName)

lsst.pipe.base.cmdLineTask.CmdLineTask.writeConfig

def writeConfig

Write the configuration used for processing the data, or check that an existing one is equal to the n...

Definition: cmdLineTask.py:461

lsst.pipe.base.cmdLineTask.CmdLineTask._getConfigName

def _getConfigName

Return the name of the config dataset type, or None if config is not to be persisted.

Definition: cmdLineTask.py:529

def lsst.pipe.base.cmdLineTask.CmdLineTask.writeMetadata	(	self,
		dataRef
	)

Write the metadata produced from processing the data.

Parameters

[in] dataRef butler data reference used to write the metadata. The metadata is written to dataset type self._getMetadataName()

Definition at line 516 of file cmdLineTask.py.

 
     def writeMetadata(self, dataRef):
         """!Write the metadata produced from processing the data
 
         @param[in] dataRef  butler data reference used to write the metadata.
             The metadata is written to dataset type self._getMetadataName()
         """
         try:
             metadataName = self._getMetadataName()
             if metadataName is not None:
                 dataRef.put(self.getFullMetadata(), metadataName)
         except Exception, e:
             self.log.warn("Could not persist metadata for dataId=%s: %s" % (dataRef.dataId, e,))

lsst.pipe.base.cmdLineTask.CmdLineTask._getMetadataName

def _getMetadataName

Return the name of the metadata dataset type, or None if metadata is not to be persisted.

Definition: cmdLineTask.py:536

lsst.pipe.base.cmdLineTask.CmdLineTask.writeMetadata

def writeMetadata

Write the metadata produced from processing the data.

Definition: cmdLineTask.py:516

def lsst.pipe.base.cmdLineTask.CmdLineTask.writeSchemas	(	self,
		butler,
		clobber = `False`
	)

Write the schemas returned by getAllSchemaCatalogs.

Parameters

[in]	butler	data butler used to write the schema. Each schema is written to the dataset type specified as the key in the dict returned by getAllSchemaCatalogs.
[in]	clobber	a boolean flag that controls what happens if a schema already has been saved: True: overwrite the existing schema False: raise TaskError if this schema does not match the existing schema

Warning: if clobber is False and an existing schema does not match a current schema, then some schemas may have been saved successfully and others may not, and there is no easy way to tell which is which.

Definition at line 488 of file cmdLineTask.py.

 
     def writeSchemas(self, butler, clobber=False):
         """!Write the schemas returned by \ref task.Task.getAllSchemaCatalogs "getAllSchemaCatalogs"
 
         @param[in] butler   data butler used to write the schema.
             Each schema is written to the dataset type specified as the key in the dict returned by
             \ref task.Task.getAllSchemaCatalogs "getAllSchemaCatalogs".
         @param[in] clobber  a boolean flag that controls what happens if a schema already has been saved:
             - True: overwrite the existing schema
             - False: raise TaskError if this schema does not match the existing schema
 
         @warning if clobber is False and an existing schema does not match a current schema,
         then some schemas may have been saved successfully and others may not, and there is no easy way to
         tell which is which.
         """
         for dataset, catalog in self.getAllSchemaCatalogs().iteritems():
             schemaDataset = dataset + "_schema"
             if clobber:
                 butler.put(catalog, schemaDataset, doBackup=True)
             elif butler.datasetExists(schemaDataset):
                 oldSchema = butler.get(schemaDataset, immediate=True).getSchema()
                 if not oldSchema.compare(catalog.getSchema(), afwTable.Schema.IDENTICAL):
                     raise TaskError(
                         ("New schema does not match schema %r on disk; schemas must be " + \
                         " consistent within the same output repo (override with --clobber-config)") % \
                         (dataset,))
             else:
                 butler.put(catalog, schemaDataset)

lsst.pipe.base.cmdLineTask.CmdLineTask.writeSchemas

def writeSchemas

Write the schemas returned by getAllSchemaCatalogs.

Definition: cmdLineTask.py:488

Public Member Functions

Static Public Attributes

Private Member Functions

Detailed Description

Member Function Documentation

Member Data Documentation