lsst.pipe.tasks.ingestDefects.IngestDefectsArgumentParser Class Reference

Inheritance diagram for lsst.pipe.tasks.ingestDefects.IngestDefectsArgumentParser:

Public Member Functions
def	__init__ (self, args, kwargs)
def	add_id_argument (self, name, datasetType, help, level=None, doMakeDataRefList=True, ContainerClass=DataIdContainer)
def	parse_args (self, config, args=None, log=None, override=None)
def	handleCamera (self, namespace)
def	convert_arg_line_to_args (self, arg_line)
def	addReuseOption (self, choices)

Static Public Attributes
bool	requireOutput = False

Detailed Description

Argument parser to support ingesting calibration images into the repository

Definition at line 10 of file ingestDefects.py.

Constructor & Destructor Documentation

__init__()

def lsst.pipe.tasks.ingestDefects.IngestDefectsArgumentParser.__init__	(		self,
			args,
			kwargs
	)

Definition at line 13 of file ingestDefects.py.

    def __init__(self, *args, **kwargs):
        InputOnlyArgumentParser.__init__(self, *args, **kwargs)
        self.add_argument("-n", "--dry-run", dest="dryrun", action="store_true",
                          default=False, help="Don't perform any action?")
        self.add_argument("--create", action="store_true", help="Create new registry?")
        self.add_argument("--ignore-ingested", dest="ignoreIngested", action="store_true",
                          help="Don't register files that have already been registered")
        self.add_argument("root", help="Root directory to scan for defects.")

Member Function Documentation

add_id_argument()

def lsst.pipe.base.argumentParser.ArgumentParser.add_id_argument (   self,   name,   datasetType,   help,   level = None,   doMakeDataRefList = True,   ContainerClass = DataIdContainer  )

inherited

Add a data ID argument.


Parameters
----------
name : `str`
    Data ID argument (including leading dashes, if wanted).
datasetType : `str` or `DynamicDatasetType`-type
    Type of dataset. Supply a string for a fixed dataset type.
    For a dynamically determined dataset type, supply
    a `DynamicDatasetType`, such a `DatasetArgument`.
help : `str`
    Help string for the argument.
level : `str`
    The lowest hierarchy level to descend to for this dataset type,
    for example `"amp"` for `"raw"` or `"ccd"` for `"calexp"`.
    Use `""` to use the mapper's default for the dataset type.
    Some container classes may also support `None`, which means
    the level should not be restricted; however the default class,
    `DataIdContainer`, does not support `None`.
doMakeDataRefList : bool, optional
    If `True` (default), construct data references.
ContainerClass : `class`, optional
Class to contain data IDs and data references; the default class
`DataIdContainer` will work for many, but not all, cases.
For example if the dataset type is specified on the command line
then use `DynamicDatasetType`.

Notes
-----
If ``datasetType`` is an instance of `DatasetArgument`,
then add a second argument to specify the dataset type.

The associated data is put into ``namespace.<dataIdArgument.name>``
as an instance of `ContainerClass`; the container includes fields:

- ``idList``: a list of data ID dicts.
- ``refList``: a list of `~lsst.daf.persistence.Butler`
    data references (empty if ``doMakeDataRefList`` is  `False`).

Definition at line 511 of file argumentParser.py.

                        ContainerClass=DataIdContainer):
        """Add a data ID argument.


        Parameters
        ----------
        name : `str`
            Data ID argument (including leading dashes, if wanted).
        datasetType : `str` or `DynamicDatasetType`-type
            Type of dataset. Supply a string for a fixed dataset type.
            For a dynamically determined dataset type, supply
            a `DynamicDatasetType`, such a `DatasetArgument`.
        help : `str`
            Help string for the argument.
        level : `str`
            The lowest hierarchy level to descend to for this dataset type,
            for example `"amp"` for `"raw"` or `"ccd"` for `"calexp"`.
            Use `""` to use the mapper's default for the dataset type.
            Some container classes may also support `None`, which means
            the level should not be restricted; however the default class,
            `DataIdContainer`, does not support `None`.
        doMakeDataRefList : bool, optional
            If `True` (default), construct data references.
        ContainerClass : `class`, optional
        Class to contain data IDs and data references; the default class
        `DataIdContainer` will work for many, but not all, cases.
        For example if the dataset type is specified on the command line
        then use `DynamicDatasetType`.

        Notes
        -----
        If ``datasetType`` is an instance of `DatasetArgument`,
        then add a second argument to specify the dataset type.

        The associated data is put into ``namespace.<dataIdArgument.name>``
        as an instance of `ContainerClass`; the container includes fields:

        - ``idList``: a list of data ID dicts.
        - ``refList``: a list of `~lsst.daf.persistence.Butler`
            data references (empty if ``doMakeDataRefList`` is  `False`).
        """
        argName = name.lstrip("-")

        if argName in self._dataIdArgDict:
            raise RuntimeError("Data ID argument %s already exists" % (name,))
        if argName in set(("camera", "config", "butler", "log", "obsPkg")):
            raise RuntimeError("Data ID argument %s is a reserved name" % (name,))

        self.add_argument(name, nargs="*", action=IdValueAction, help=help,
                          metavar="KEY=VALUE1[^VALUE2[^VALUE3...]")

        dataIdArgument = DataIdArgument(
            name=argName,
            datasetType=datasetType,
            level=level,
            doMakeDataRefList=doMakeDataRefList,
            ContainerClass=ContainerClass,
        )

        if dataIdArgument.isDynamicDatasetType:
            datasetType.addArgument(parser=self, idName=argName)

        self._dataIdArgDict[argName] = dataIdArgument

addReuseOption()

def lsst.pipe.base.argumentParser.ArgumentParser.addReuseOption (   self,   choices  )

inherited

Add a "--reuse-outputs-from SUBTASK" option to the argument
parser.

CmdLineTasks that can be restarted at an intermediate step using
outputs from earlier (but still internal) steps should use this
method to allow the user to control whether that happens when
outputs from earlier steps are present.

Parameters
----------
choices : sequence
    A sequence of string names (by convention, top-level subtasks)
    that identify the steps that could be skipped when their
    outputs are already present.  The list is ordered, so when the
    user specifies one step on the command line, all previous steps
    may be skipped as well.  In addition to the choices provided,
    users may pass "all" to indicate that all steps may be thus
    skipped.

When this method is called, the ``namespace`` object returned by
``parse_args`` will contain a ``reuse`` attribute containing
a list of all steps that should be skipped if their outputs
are already present.
If no steps should be skipped, the ``reuse`` will be an empty list.

Definition at line 880 of file argumentParser.py.

    def addReuseOption(self, choices):
        """Add a "--reuse-outputs-from SUBTASK" option to the argument
        parser.

        CmdLineTasks that can be restarted at an intermediate step using
        outputs from earlier (but still internal) steps should use this
        method to allow the user to control whether that happens when
        outputs from earlier steps are present.

        Parameters
        ----------
        choices : sequence
            A sequence of string names (by convention, top-level subtasks)
            that identify the steps that could be skipped when their
            outputs are already present.  The list is ordered, so when the
            user specifies one step on the command line, all previous steps
            may be skipped as well.  In addition to the choices provided,
            users may pass "all" to indicate that all steps may be thus
            skipped.

        When this method is called, the ``namespace`` object returned by
        ``parse_args`` will contain a ``reuse`` attribute containing
        a list of all steps that should be skipped if their outputs
        are already present.
        If no steps should be skipped, the ``reuse`` will be an empty list.
        """
        choices = list(choices)
        choices.append("all")
        self.add_argument("--reuse-outputs-from", dest="reuse", choices=choices,
                          default=[], action=ReuseAction,
                          help=("Skip the given subtask and its predecessors and reuse their outputs "
                                "if those outputs already exist.  Use 'all' to specify all subtasks."))

convert_arg_line_to_args()

def lsst.pipe.base.argumentParser.ArgumentParser.convert_arg_line_to_args (   self,   arg_line  )

inherited

Allow files of arguments referenced by ``@<path>`` to contain
multiple values on each line.

Parameters
----------
arg_line : `str`
    Line of text read from an argument file.

Definition at line 863 of file argumentParser.py.

    def convert_arg_line_to_args(self, arg_line):
        """Allow files of arguments referenced by ``@<path>`` to contain
        multiple values on each line.

        Parameters
        ----------
        arg_line : `str`
            Line of text read from an argument file.
        """
        arg_line = arg_line.strip()
        if not arg_line or arg_line.startswith("#"):
            return
        for arg in shlex.split(arg_line, comments=True, posix=True):
            if not arg.strip():
                continue
            yield arg

handleCamera()

def lsst.pipe.base.argumentParser.ArgumentParser.handleCamera (   self,   namespace  )

inherited

Perform camera-specific operations before parsing the command-line.

Parameters
----------
namespace : `argparse.Namespace`
    Namespace (an ) with the following fields:

    - ``camera``: the camera name.
    - ``config``: the config passed to parse_args, with no overrides applied.
    - ``obsPkg``: the ``obs_`` package for this camera.
    - ``log``: a `lsst.log` Log.

Notes
-----
The default implementation does nothing.

Definition at line 844 of file argumentParser.py.

    def handleCamera(self, namespace):
        """Perform camera-specific operations before parsing the command-line.

        Parameters
        ----------
        namespace : `argparse.Namespace`
            Namespace (an ) with the following fields:

            - ``camera``: the camera name.
            - ``config``: the config passed to parse_args, with no overrides applied.
            - ``obsPkg``: the ``obs_`` package for this camera.
            - ``log``: a `lsst.log` Log.

        Notes
        -----
        The default implementation does nothing.
        """
        pass

parse_args()

def lsst.pipe.base.argumentParser.ArgumentParser.parse_args (   self,   config,   args = None,   log = None,   override = None  )

inherited

Parse arguments for a command-line task.

Parameters
----------
config : `lsst.pex.config.Config`
    Config for the task being run.
args : `list`, optional
    Argument list; if `None` then ``sys.argv[1:]`` is used.
log : `lsst.log.Log`, optional
    `~lsst.log.Log` instance; if `None` use the default log.
override : callable, optional
    A config override function. It must take the root config object
    as its only argument and must modify the config in place.
    This function is called after camera-specific overrides files
    are applied, and before command-line config overrides
    are applied (thus allowing the user the final word).

Returns
-------
namespace : `argparse.Namespace`
    A `~argparse.Namespace` instance containing fields:

    - ``camera``: camera name.
    - ``config``: the supplied config with all overrides applied,
validated and frozen.
    - ``butler``: a `lsst.daf.persistence.Butler` for the data.
    - An entry for each of the data ID arguments registered by
`add_id_argument`, of the type passed to its ``ContainerClass``
keyword (`~lsst.pipe.base.DataIdContainer` by default). It
includes public elements ``idList`` and ``refList``.
    - ``log``: a `lsst.log` Log.
    - An entry for each command-line argument,
with the following exceptions:

      - config is the supplied config, suitably updated.
      - configfile, id and loglevel are all missing.
    - ``obsPkg``: name of the ``obs_`` package for this camera.

Definition at line 575 of file argumentParser.py.

    def parse_args(self, config, args=None, log=None, override=None):
        """Parse arguments for a command-line task.

        Parameters
        ----------
        config : `lsst.pex.config.Config`
            Config for the task being run.
        args : `list`, optional
            Argument list; if `None` then ``sys.argv[1:]`` is used.
        log : `lsst.log.Log`, optional
            `~lsst.log.Log` instance; if `None` use the default log.
        override : callable, optional
            A config override function. It must take the root config object
            as its only argument and must modify the config in place.
            This function is called after camera-specific overrides files
            are applied, and before command-line config overrides
            are applied (thus allowing the user the final word).

        Returns
        -------
        namespace : `argparse.Namespace`
            A `~argparse.Namespace` instance containing fields:

            - ``camera``: camera name.
            - ``config``: the supplied config with all overrides applied,
                validated and frozen.
            - ``butler``: a `lsst.daf.persistence.Butler` for the data.
            - An entry for each of the data ID arguments registered by
                `add_id_argument`, of the type passed to its ``ContainerClass``
                keyword (`~lsst.pipe.base.DataIdContainer` by default). It
                includes public elements ``idList`` and ``refList``.
            - ``log``: a `lsst.log` Log.
            - An entry for each command-line argument,
                with the following exceptions:

              - config is the supplied config, suitably updated.
              - configfile, id and loglevel are all missing.
            - ``obsPkg``: name of the ``obs_`` package for this camera.
        """
        if args is None:
            args = sys.argv[1:]

        if len(args) < 1 or args[0].startswith("-") or args[0].startswith("@"):
            self.print_help()
            if len(args) == 1 and args[0] in ("-h", "--help"):
                self.exit()
            else:
                self.exit("%s: error: Must specify input as first argument" % self.prog)

        # Note that --rerun may change namespace.input, but if it does
        # we verify that the new input has the same mapper class.
        namespace = argparse.Namespace()
        namespace.input = _fixPath(DEFAULT_INPUT_NAME, args[0])
        if not os.path.isdir(namespace.input):
            self.error("Error: input=%r not found" % (namespace.input,))

        namespace.config = config
        namespace.log = log if log is not None else lsstLog.Log.getDefaultLogger()
        mapperClass = dafPersist.Butler.getMapperClass(namespace.input)
        if mapperClass is None:
            self.error("Error: no mapper specified for input repo %r" % (namespace.input,))

        namespace.camera = mapperClass.getCameraName()
        namespace.obsPkg = mapperClass.getPackageName()

        self.handleCamera(namespace)

        self._applyInitialOverrides(namespace)
        if override is not None:
            override(namespace.config)

        # Add data ID containers to namespace
        for dataIdArgument in self._dataIdArgDict.values():
            setattr(namespace, dataIdArgument.name, dataIdArgument.ContainerClass(level=dataIdArgument.level))

        namespace = argparse.ArgumentParser.parse_args(self, args=args, namespace=namespace)
        del namespace.configfile

        self._parseDirectories(namespace)

        if namespace.clobberOutput:
            if namespace.output is None:
                self.error("--clobber-output is only valid with --output or --rerun")
            elif namespace.output == namespace.input:
                self.error("--clobber-output is not valid when the output and input repos are the same")
            if os.path.exists(namespace.output):
                namespace.log.info("Removing output repo %s for --clobber-output", namespace.output)
                shutil.rmtree(namespace.output)

        namespace.log.debug("input=%s", namespace.input)
        namespace.log.debug("calib=%s", namespace.calib)
        namespace.log.debug("output=%s", namespace.output)

        obeyShowArgument(namespace.show, namespace.config, exit=False)

        # No environment variable or --output or --rerun specified.
        if self.requireOutput and namespace.output is None and namespace.rerun is None:
            self.error("no output directory specified.\n"
                       "An output directory must be specified with the --output or --rerun\n"
                       "command-line arguments.\n")

        butlerArgs = {}  # common arguments for butler elements
        if namespace.calib:
            butlerArgs = {'mapperArgs': {'calibRoot': namespace.calib}}
        if namespace.output:
            outputs = {'root': namespace.output, 'mode': 'rw'}
            inputs = {'root': namespace.input}
            inputs.update(butlerArgs)
            outputs.update(butlerArgs)
            namespace.butler = dafPersist.Butler(inputs=inputs, outputs=outputs)
        else:
            outputs = {'root': namespace.input, 'mode': 'rw'}
            outputs.update(butlerArgs)
            namespace.butler = dafPersist.Butler(outputs=outputs)

        # convert data in each of the identifier lists to proper types
        # this is done after constructing the butler,
        # hence after parsing the command line,
        # because it takes a long time to construct a butler
        self._processDataIds(namespace)
        if "data" in namespace.show:
            for dataIdName in self._dataIdArgDict.keys():
                for dataRef in getattr(namespace, dataIdName).refList:
                    print("%s dataRef.dataId = %s" % (dataIdName, dataRef.dataId))

        if namespace.show and "run" not in namespace.show:
            sys.exit(0)

        if namespace.debug:
            try:
                import debug
                assert debug  # silence pyflakes
            except ImportError:
                sys.stderr.write("Warning: no 'debug' module found\n")
                namespace.debug = False

        del namespace.loglevel

        if namespace.longlog:
            lsstLog.configure_prop("""
log4j.rootLogger=INFO, A1
log4j.appender.A1=ConsoleAppender
log4j.appender.A1.Target=System.out
log4j.appender.A1.layout=PatternLayout
log4j.appender.A1.layout.ConversionPattern=%-5p %d{yyyy-MM-ddTHH:mm:ss.SSSZ} %c (%X{LABEL})(%F:%L)- %m%n
""")
        del namespace.longlog

        namespace.config.validate()
        namespace.config.freeze()

        return namespace

Member Data Documentation

requireOutput

bool lsst.pipe.base.argumentParser.InputOnlyArgumentParser.requireOutput = False

staticinherited

Definition at line 918 of file argumentParser.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/Linux64/pipe_tasks/19.0.0+3/python/lsst/pipe/tasks/ingestDefects.py