LSSTApplications  10.0+286,10.0+36,10.0+46,10.0-2-g4f67435,10.1+152,10.1+37,11.0,11.0+1,11.0-1-g47edd16,11.0-1-g60db491,11.0-1-g7418c06,11.0-2-g04d2804,11.0-2-g68503cd,11.0-2-g818369d,11.0-2-gb8b8ce7
LSSTDataManagementBasePackage
Public Member Functions | Private Member Functions | Private Attributes | List of all members
lsst.pipe.base.argumentParser.ArgumentParser Class Reference

An argument parser for pipeline tasks that is based on argparse.ArgumentParser. More...

Inheritance diagram for lsst.pipe.base.argumentParser.ArgumentParser:

Public Member Functions

def __init__
 Construct an ArgumentParser. More...
 
def add_id_argument
 Add a data ID argument. More...
 
def parse_args
 Parse arguments for a pipeline task. More...
 
def handleCamera
 Perform camera-specific operations before parsing the command line. More...
 
def convert_arg_line_to_args
 Allow files of arguments referenced by @<path> to contain multiple values on each line. More...
 

Private Member Functions

def _processDataIds
 Process the parsed data for each data ID argument. More...
 
def _applyInitialOverrides
 Apply obs-package-specific and camera-specific config override files, if found. More...
 

Private Attributes

 _name
 
 _dataIdArgDict
 

Detailed Description

An argument parser for pipeline tasks that is based on argparse.ArgumentParser.

Users may wish to add additional arguments before calling parse_args.

Note
  • I would prefer to check data ID keys and values as they are parsed, but the required information comes from the butler, so I have to construct a butler before I do this checking. Constructing a butler is slow, so I only want do it once, after parsing the command line, so as to catch syntax errors quickly.

Definition at line 196 of file argumentParser.py.

Constructor & Destructor Documentation

def lsst.pipe.base.argumentParser.ArgumentParser.__init__ (   self,
  name,
  usage = "%(prog)s input [options]",
  kwargs 
)

Construct an ArgumentParser.

Parameters
[in]namename of top-level task; used to identify camera-specific override files
[in]usageusage string
[in]**kwargsadditional keyword arguments for argparse.ArgumentParser

Definition at line 207 of file argumentParser.py.

208  def __init__(self, name, usage = "%(prog)s input [options]", **kwargs):
209  """!Construct an ArgumentParser
210 
211  @param[in] name name of top-level task; used to identify camera-specific override files
212  @param[in] usage usage string
213  @param[in] **kwargs additional keyword arguments for argparse.ArgumentParser
214  """
215  self._name = name
216  self._dataIdArgDict = {} # Dict of data identifier specifications, by argument name
217  argparse.ArgumentParser.__init__(self,
218  usage = usage,
219  fromfile_prefix_chars = '@',
220  epilog = """Notes:
221 * --config, --configfile, --id, --loglevel and @file may appear multiple times;
222  all values are used, in order left to right
223 * @file reads command-line options from the specified file:
224  * data may be distributed among multiple lines (e.g. one option per line)
225  * data after # is treated as a comment and ignored
226  * blank lines and lines starting with # are ignored
227 * To specify multiple values for an option, do not use = after the option name:
228  * right: --configfile foo bar
229  * wrong: --configfile=foo bar
230 """,
231  formatter_class = argparse.RawDescriptionHelpFormatter,
232  **kwargs)
233  self.add_argument("input",
234  help="path to input data repository, relative to $%s" % (DEFAULT_INPUT_NAME,))
235  self.add_argument("--calib",
236  help="path to input calibration repository, relative to $%s" % (DEFAULT_CALIB_NAME,))
237  self.add_argument("--output",
238  help="path to output data repository (need not exist), relative to $%s" % (DEFAULT_OUTPUT_NAME,))
239  self.add_argument("-c", "--config", nargs="*", action=ConfigValueAction,
240  help="config override(s), e.g. -c foo=newfoo bar.baz=3", metavar="NAME=VALUE")
241  self.add_argument("-C", "--configfile", dest="configfile", nargs="*", action=ConfigFileAction,
242  help="config override file(s)")
243  self.add_argument("-L", "--loglevel", nargs="*", action=LogLevelAction,
244  help="logging level; supported levels are [debug|warn|info|fatal] or an integer; "
245  "trace level is negative log level, e.g. use level -3 for trace level 3",
246  metavar="LEVEL|COMPONENT=LEVEL")
247  self.add_argument("--debug", action="store_true", help="enable debugging output?")
248  self.add_argument("--doraise", action="store_true",
249  help="raise an exception on error (else log a message and continue)?")
250  self.add_argument("--profile", help="Dump cProfile statistics to filename")
251  self.add_argument("--logdest", help="logging destination")
252  self.add_argument("--show", nargs="+", default=(),
253  help="display the specified information to stdout and quit (unless run is specified).")
254  self.add_argument("-j", "--processes", type=int, default=1, help="Number of processes to use")
255  self.add_argument("-t", "--timeout", type=float,
256  help="Timeout for multiprocessing; maximum wall time (sec)")
257  self.add_argument("--clobber-output", action="store_true", dest="clobberOutput", default=False,
258  help=("remove and re-create the output directory if it already exists "
259  "(safe with -j, but not all other forms of parallel execution)"))
260  self.add_argument("--clobber-config", action="store_true", dest="clobberConfig", default=False,
261  help=("backup and then overwrite existing config files instead of checking them "
262  "(safe with -j, but not all other forms of parallel execution)"))
def __init__
Construct an ArgumentParser.

Member Function Documentation

def lsst.pipe.base.argumentParser.ArgumentParser._applyInitialOverrides (   self,
  namespace 
)
private

Apply obs-package-specific and camera-specific config override files, if found.

Parameters
[in]namespaceparsed namespace (an argparse.Namespace); reads these attributes:
  • obsPkg

Look in the package namespace.obsPkg for files:

Definition at line 455 of file argumentParser.py.

456  def _applyInitialOverrides(self, namespace):
457  """!Apply obs-package-specific and camera-specific config override files, if found
458 
459  @param[in] namespace parsed namespace (an argparse.Namespace);
460  reads these attributes:
461  - obsPkg
462 
463  Look in the package namespace.obsPkg for files:
464  - config/<task_name>.py
465  - config/<camera_name>/<task_name>.py
466  and load if found
467  """
468  obsPkgDir = lsst.utils.getPackageDir(namespace.obsPkg)
469  fileName = self._name + ".py"
470  for filePath in (
471  os.path.join(obsPkgDir, "config", fileName),
472  os.path.join(obsPkgDir, "config", namespace.camera, fileName),
473  ):
474  if os.path.exists(filePath):
475  namespace.log.info("Loading config overrride file %r" % (filePath,))
476  namespace.config.load(filePath)
477  else:
478  namespace.log.info("Config override file does not exist: %r" % (filePath,))
std::string getPackageDir(std::string const &packageName)
return the root directory of a setup package
Definition: Utils.cc:66
def _applyInitialOverrides
Apply obs-package-specific and camera-specific config override files, if found.
def lsst.pipe.base.argumentParser.ArgumentParser._processDataIds (   self,
  namespace 
)
private

Process the parsed data for each data ID argument.

Processing includes:

Parameters
[in,out]namespaceparsed namespace (an argparse.Namespace); reads these attributes:
  • butler
  • log
  • <name_dstype> for each data ID argument with a dynamic dataset type registered using add_id_argument and modifies these attributes:
  • <name> for each data ID argument registered using add_id_argument

Definition at line 426 of file argumentParser.py.

427  def _processDataIds(self, namespace):
428  """!Process the parsed data for each data ID argument
429 
430  Processing includes:
431  - Validate data ID keys
432  - Cast the data ID values to the correct type
433  - Compute data references from data IDs
434 
435  @param[in,out] namespace parsed namespace (an argparse.Namespace);
436  reads these attributes:
437  - butler
438  - log
439  - <name_dstype> for each data ID argument with a dynamic dataset type registered using
440  add_id_argument
441  and modifies these attributes:
442  - <name> for each data ID argument registered using add_id_argument
443  """
444  for dataIdArgument in self._dataIdArgDict.itervalues():
445  dataIdContainer = getattr(namespace, dataIdArgument.name)
446  dataIdContainer.setDatasetType(dataIdArgument.getDatasetType(namespace))
447  try:
448  dataIdContainer.castDataIds(butler = namespace.butler)
449  except (KeyError, TypeError) as e:
450  # failure of castDataIds indicates invalid command args
451  self.error(e)
452  # failure of makeDataRefList indicates a bug that wants a traceback
453  if dataIdArgument.doMakeDataRefList:
454  dataIdContainer.makeDataRefList(namespace)
def _processDataIds
Process the parsed data for each data ID argument.
def lsst.pipe.base.argumentParser.ArgumentParser.add_id_argument (   self,
  name,
  datasetType,
  help,
  level = None,
  doMakeDataRefList = True,
  ContainerClass = DataIdContainer 
)

Add a data ID argument.

Add an argument to specify data IDs. If datasetType is an instance of DatasetArgument, then add a second argument to specify the dataset type.

Parameters
[in]namename of name (including leading dashes, if wanted)
[in]datasetTypetype of dataset; supply a string for a fixed dataset type, or a DatasetArgument for a dynamically determined dataset type
[in]helphelp string for the argument
[in]levellevel of dataset, for butler
[in]doMakeDataRefListconstruct data references?
[in]ContainerClassdata ID container class to use to contain results; override the default if you need a special means of computing data references from data IDs

The associated data is put into namespace.<dataIdArgument.name> as an instance of ContainerClass; the container includes fields:

Definition at line 264 of file argumentParser.py.

265  ContainerClass=DataIdContainer):
266  """!Add a data ID argument
267 
268  Add an argument to specify data IDs. If datasetType is an instance of DatasetArgument,
269  then add a second argument to specify the dataset type.
270 
271  @param[in] name name of name (including leading dashes, if wanted)
272  @param[in] datasetType type of dataset; supply a string for a fixed dataset type,
273  or a DatasetArgument for a dynamically determined dataset type
274  @param[in] help help string for the argument
275  @param[in] level level of dataset, for butler
276  @param[in] doMakeDataRefList construct data references?
277  @param[in] ContainerClass data ID container class to use to contain results;
278  override the default if you need a special means of computing data references from data IDs
279 
280  The associated data is put into namespace.<dataIdArgument.name> as an instance of ContainerClass;
281  the container includes fields:
282  - idList: a list of data ID dicts
283  - refList: a list of butler data references (empty if doMakeDataRefList false)
284  """
285  argName = name.lstrip("-")
286 
287  if argName in self._dataIdArgDict:
288  raise RuntimeError("Data ID argument %s already exists" % (name,))
289  if argName in set(("camera", "config", "butler", "log", "obsPkg")):
290  raise RuntimeError("Data ID argument %s is a reserved name" % (name,))
291 
292  self.add_argument(name, nargs="*", action=IdValueAction, help=help,
293  metavar="KEY=VALUE1[^VALUE2[^VALUE3...]")
294 
295  dataIdArgument = DataIdArgument(
296  name = argName,
297  datasetType = datasetType,
298  level = level,
299  doMakeDataRefList = doMakeDataRefList,
300  ContainerClass = ContainerClass,
301  )
302 
303  if dataIdArgument.isDynamicDatasetType():
304  datasetType = dataIdArgument.datasetType
305  help = datasetType.help if datasetType.help else "dataset type for %s" % (name,)
306  self.add_argument(
307  "--" + dataIdArgument.datasetTypeName,
308  default = datasetType.default,
309  required = datasetType.required,
310  help = help,
311  )
312  self._dataIdArgDict[argName] = dataIdArgument
Glorified struct for data about id arguments, used by ArgumentParser.add_id_argument.
def lsst.pipe.base.argumentParser.ArgumentParser.convert_arg_line_to_args (   self,
  arg_line 
)

Allow files of arguments referenced by @<path> to contain multiple values on each line.

Parameters
[in]arg_lineline of text read from an argument file

Definition at line 492 of file argumentParser.py.

493  def convert_arg_line_to_args(self, arg_line):
494  """!Allow files of arguments referenced by `@<path>` to contain multiple values on each line
495 
496  @param[in] arg_line line of text read from an argument file
497  """
498  arg_line = arg_line.strip()
499  if not arg_line or arg_line.startswith("#"):
500  return
501  for arg in shlex.split(arg_line, comments=True, posix=True):
502  if not arg.strip():
503  continue
504  yield arg
def convert_arg_line_to_args
Allow files of arguments referenced by @&lt;path&gt; to contain multiple values on each line...
def lsst.pipe.base.argumentParser.ArgumentParser.handleCamera (   self,
  namespace 
)

Perform camera-specific operations before parsing the command line.

The default implementation does nothing.

Parameters
[in,out]namespacenamespace (an argparse.Namespace) with the following fields:
  • camera: the camera name
  • config: the config passed to parse_args, with no overrides applied
  • obsPkg: the obs_ package for this camera
  • log: a pex_logging log

Definition at line 479 of file argumentParser.py.

480  def handleCamera(self, namespace):
481  """!Perform camera-specific operations before parsing the command line.
482 
483  The default implementation does nothing.
484 
485  @param[in,out] namespace namespace (an argparse.Namespace) with the following fields:
486  - camera: the camera name
487  - config: the config passed to parse_args, with no overrides applied
488  - obsPkg: the obs_ package for this camera
489  - log: a pex_logging log
490  """
491  pass
def handleCamera
Perform camera-specific operations before parsing the command line.
def lsst.pipe.base.argumentParser.ArgumentParser.parse_args (   self,
  config,
  args = None,
  log = None,
  override = None 
)

Parse arguments for a pipeline task.

Parameters
[in,out]configconfig for the task being run
[in]argsargument list; if None use sys.argv[1:]
[in]loglog (instance pex_logging Log); if None use the default log
[in]overridea config override function; it must take the root config object as its only argument and must modify the config in place. This function is called after camera-specific overrides files are applied, and before command-line config overrides are applied (thus allowing the user the final word).
Returns
namespace: an argparse.Namespace containing many useful fields including:
  • camera: camera name
  • config: the supplied config with all overrides applied, validated and frozen
  • butler: a butler for the data
  • an entry for each of the data ID arguments registered by add_id_argument(), the value of which is a DataIdArgument that includes public elements 'idList' and 'refList'
  • log: a pex_logging log
  • an entry for each command-line argument, with the following exceptions:
    • config is the supplied config, suitably updated
    • configfile, id and loglevel are all missing
  • obsPkg: name of obs_ package for this camera

Definition at line 313 of file argumentParser.py.

314  def parse_args(self, config, args=None, log=None, override=None):
315  """!Parse arguments for a pipeline task
316 
317  @param[in,out] config config for the task being run
318  @param[in] args argument list; if None use sys.argv[1:]
319  @param[in] log log (instance pex_logging Log); if None use the default log
320  @param[in] override a config override function; it must take the root config object
321  as its only argument and must modify the config in place.
322  This function is called after camera-specific overrides files are applied, and before
323  command-line config overrides are applied (thus allowing the user the final word).
324 
325  @return namespace: an argparse.Namespace containing many useful fields including:
326  - camera: camera name
327  - config: the supplied config with all overrides applied, validated and frozen
328  - butler: a butler for the data
329  - an entry for each of the data ID arguments registered by add_id_argument(),
330  the value of which is a DataIdArgument that includes public elements 'idList' and 'refList'
331  - log: a pex_logging log
332  - an entry for each command-line argument, with the following exceptions:
333  - config is the supplied config, suitably updated
334  - configfile, id and loglevel are all missing
335  - obsPkg: name of obs_ package for this camera
336  """
337  if args == None:
338  args = sys.argv[1:]
339 
340  if len(args) < 1 or args[0].startswith("-") or args[0].startswith("@"):
341  self.print_help()
342  if len(args) == 1 and args[0] in ("-h", "--help"):
343  self.exit()
344  else:
345  self.exit("%s: error: Must specify input as first argument" % self.prog)
346 
347  # note: don't set namespace.input until after running parse_args, else it will get overwritten
348  inputRoot = _fixPath(DEFAULT_INPUT_NAME, args[0])
349  if not os.path.isdir(inputRoot):
350  self.error("Error: input=%r not found" % (inputRoot,))
351 
352  namespace = argparse.Namespace()
353  namespace.config = config
354  namespace.log = log if log is not None else pexLog.Log.getDefaultLog()
355  mapperClass = dafPersist.Butler.getMapperClass(inputRoot)
356  namespace.camera = mapperClass.getCameraName()
357  namespace.obsPkg = mapperClass.getPackageName()
358 
359  self.handleCamera(namespace)
360 
361  self._applyInitialOverrides(namespace)
362  if override is not None:
363  override(namespace.config)
364 
365  # Add data ID containers to namespace
366  for dataIdArgument in self._dataIdArgDict.itervalues():
367  setattr(namespace, dataIdArgument.name, dataIdArgument.ContainerClass(level=dataIdArgument.level))
368 
369  namespace = argparse.ArgumentParser.parse_args(self, args=args, namespace=namespace)
370  namespace.input = inputRoot
371  del namespace.configfile
372 
373  namespace.calib = _fixPath(DEFAULT_CALIB_NAME, namespace.calib)
374  namespace.output = _fixPath(DEFAULT_OUTPUT_NAME, namespace.output)
375 
376  if namespace.clobberOutput:
377  if namespace.output is None:
378  self.error("--clobber-output is only valid with --output")
379  elif namespace.output == namespace.input:
380  self.error("--clobber-output is not valid when the output and input repos are the same")
381  if os.path.exists(namespace.output):
382  namespace.log.info("Removing output repo %s for --clobber-output" % namespace.output)
383  shutil.rmtree(namespace.output)
384 
385  namespace.log.info("input=%s" % (namespace.input,))
386  namespace.log.info("calib=%s" % (namespace.calib,))
387  namespace.log.info("output=%s" % (namespace.output,))
388 
389  obeyShowArgument(namespace.show, namespace.config, exit=False)
390 
391  namespace.butler = dafPersist.Butler(
392  root = namespace.input,
393  calibRoot = namespace.calib,
394  outputRoot = namespace.output,
395  )
396 
397  # convert data in each of the identifier lists to proper types
398  # this is done after constructing the butler, hence after parsing the command line,
399  # because it takes a long time to construct a butler
400  self._processDataIds(namespace)
401  if "data" in namespace.show:
402  for dataIdName in self._dataIdArgDict.iterkeys():
403  for dataRef in getattr(namespace, dataIdName).refList:
404  print dataIdName + " dataRef.dataId =", dataRef.dataId
405 
406  if namespace.show and "run" not in namespace.show:
407  sys.exit(0)
408 
409  if namespace.debug:
410  try:
411  import debug
412  assert debug # silence pyflakes
413  except ImportError:
414  sys.stderr.write("Warning: no 'debug' module found\n")
415  namespace.debug = False
416 
417  if namespace.logdest:
418  namespace.log.addDestination(namespace.logdest)
419  del namespace.logdest
420  del namespace.loglevel
421 
422  namespace.config.validate()
423  namespace.config.freeze()
424 
425  return namespace
def _applyInitialOverrides
Apply obs-package-specific and camera-specific config override files, if found.
def handleCamera
Perform camera-specific operations before parsing the command line.
def parse_args
Parse arguments for a pipeline task.
def _processDataIds
Process the parsed data for each data ID argument.
def obeyShowArgument
Process arguments specified with –show (but ignores &quot;data&quot;)
def _fixPath
Apply environment variable as default root, if present, and abspath.

Member Data Documentation

lsst.pipe.base.argumentParser.ArgumentParser._dataIdArgDict
private

Definition at line 215 of file argumentParser.py.

lsst.pipe.base.argumentParser.ArgumentParser._name
private

Definition at line 214 of file argumentParser.py.


The documentation for this class was generated from the following file: