lsst.pipe.tasks.multiBand.DeblendCoaddSourcesRunner Class Reference

Inheritance diagram for lsst.pipe.tasks.multiBand.DeblendCoaddSourcesRunner:

Public Member Functions
def	makeTask (self, parsedCmd=None, args=None)
def	prepareForMultiProcessing (self)
def	run (self, parsedCmd)
def	precall (self, parsedCmd)
def	__call__ (self, args)
def	runTask (self, task, dataRef, kwargs)

Static Public Member Functions
def	getTargetList (parsedCmd, kwargs)
def	buildRefDict (parsedCmd)

Public Attributes
	TaskClass
	doReturnResults
	config
	log
	doRaise
	clobberConfig
	doBackup
	numProcesses
	timeout

Static Public Attributes
int	TIMEOUT = 3600*24*30

Detailed Description

Task runner for the `MergeSourcesTask`

Required because the run method requires a list of
dataRefs rather than a single dataRef.

Definition at line 369 of file multiBand.py.

Member Function Documentation

__call__()

def lsst.pipe.base.cmdLineTask.TaskRunner.__call__ (   self,   args  )

inherited

Run the Task on a single target.

Parameters
----------
args
    Arguments for Task.runDataRef()

Returns
-------
struct : `lsst.pipe.base.Struct`
    Contains these fields if ``doReturnResults`` is `True`:

    - ``dataRef``: the provided data reference.
    - ``metadata``: task metadata after execution of run.
    - ``result``: result returned by task run, or `None` if the task fails.
    - ``exitStatus``: 0 if the task completed successfully, 1 otherwise.

    If ``doReturnResults`` is `False` the struct contains:

    - ``exitStatus``: 0 if the task completed successfully, 1 otherwise.

Notes
-----
This default implementation assumes that the ``args`` is a tuple containing a data reference and a
dict of keyword arguments.

.. warning::

   If you override this method and wish to return something when ``doReturnResults`` is `False`,
   then it must be picklable to support multiprocessing and it should be small enough that pickling
   and unpickling do not add excessive overhead.

Definition at line 341 of file cmdLineTask.py.

    def __call__(self, args):
        """Run the Task on a single target.

        Parameters
        ----------
        args
            Arguments for Task.runDataRef()

        Returns
        -------
        struct : `lsst.pipe.base.Struct`
            Contains these fields if ``doReturnResults`` is `True`:

            - ``dataRef``: the provided data reference.
            - ``metadata``: task metadata after execution of run.
            - ``result``: result returned by task run, or `None` if the task fails.
            - ``exitStatus``: 0 if the task completed successfully, 1 otherwise.

            If ``doReturnResults`` is `False` the struct contains:

            - ``exitStatus``: 0 if the task completed successfully, 1 otherwise.

        Notes
        -----
        This default implementation assumes that the ``args`` is a tuple containing a data reference and a
        dict of keyword arguments.

        .. warning::

           If you override this method and wish to return something when ``doReturnResults`` is `False`,
           then it must be picklable to support multiprocessing and it should be small enough that pickling
           and unpickling do not add excessive overhead.
        """
        dataRef, kwargs = args
        if self.log is None:
            self.log = Log.getDefaultLogger()
        if hasattr(dataRef, "dataId"):
            self.log.MDC("LABEL", str(dataRef.dataId))
        elif isinstance(dataRef, (list, tuple)):
            self.log.MDC("LABEL", str([ref.dataId for ref in dataRef if hasattr(ref, "dataId")]))
        task = self.makeTask(args=args)
        result = None                   # in case the task fails
        exitStatus = 0                  # exit status for the shell
        if self.doRaise:
            result = self.runTask(task, dataRef, kwargs)
        else:
            try:
                result = self.runTask(task, dataRef, kwargs)
            except Exception as e:
                # The shell exit value will be the number of dataRefs returning
                # non-zero, so the actual value used here is lost.
                exitStatus = 1

                # don't use a try block as we need to preserve the original exception
                eName = type(e).__name__
                if hasattr(dataRef, "dataId"):
                    task.log.fatal("Failed on dataId=%s: %s: %s", dataRef.dataId, eName, e)
                elif isinstance(dataRef, (list, tuple)):
                    task.log.fatal("Failed on dataIds=[%s]: %s: %s",
                                   ", ".join(str(ref.dataId) for ref in dataRef), eName, e)
                else:
                    task.log.fatal("Failed on dataRef=%s: %s: %s", dataRef, eName, e)

                if not isinstance(e, TaskError):
                    traceback.print_exc(file=sys.stderr)

        # Ensure all errors have been logged and aren't hanging around in a buffer
        sys.stdout.flush()
        sys.stderr.flush()

        task.writeMetadata(dataRef)

        # remove MDC so it does not show up outside of task context
        self.log.MDCRemove("LABEL")

        if self.doReturnResults:
            return Struct(
                exitStatus=exitStatus,
                dataRef=dataRef,
                metadata=task.metadata,
                result=result,
            )
        else:
            return Struct(
                exitStatus=exitStatus,
            )

buildRefDict()

def lsst.pipe.tasks.multiBandUtils.MergeSourcesRunner.buildRefDict (   parsedCmd )

staticinherited

Build a hierarchical dictionary of patch references

Parameters
----------
parsedCmd:
    The parsed command

Returns
-------
refDict: dict
    A reference dictionary of the form {patch: {tract: {filter: dataRef}}}

Raises
------
RuntimeError
    Thrown when multiple references are provided for the same
    combination of tract, patch and filter

Definition at line 41 of file multiBandUtils.py.

    def buildRefDict(parsedCmd):
        """Build a hierarchical dictionary of patch references

        Parameters
        ----------
        parsedCmd:
            The parsed command

        Returns
        -------
        refDict: dict
            A reference dictionary of the form {patch: {tract: {filter: dataRef}}}

        Raises
        ------
        RuntimeError
            Thrown when multiple references are provided for the same
            combination of tract, patch and filter
        """
        refDict = {}  # Will index this as refDict[tract][patch][filter] = ref
        for ref in parsedCmd.id.refList:
            tract = ref.dataId["tract"]
            patch = ref.dataId["patch"]
            filter = ref.dataId["filter"]
            if tract not in refDict:
                refDict[tract] = {}
            if patch not in refDict[tract]:
                refDict[tract][patch] = {}
            if filter in refDict[tract][patch]:
                raise RuntimeError("Multiple versions of %s" % (ref.dataId,))
            refDict[tract][patch][filter] = ref
        return refDict

getTargetList()

def lsst.pipe.tasks.multiBand.DeblendCoaddSourcesRunner.getTargetList (   parsedCmd,   kwargs  )

static

Provide a list of patch references for each patch, tract, filter combo.

Parameters
----------
parsedCmd:
    The parsed command
kwargs:
    Keyword arguments passed to the task

Returns
-------
targetList: list
    List of tuples, where each tuple is a (dataRef, kwargs) pair.

Definition at line 376 of file multiBand.py.

    def getTargetList(parsedCmd, **kwargs):
        """Provide a list of patch references for each patch, tract, filter combo.

        Parameters
        ----------
        parsedCmd:
            The parsed command
        kwargs:
            Keyword arguments passed to the task

        Returns
        -------
        targetList: list
            List of tuples, where each tuple is a (dataRef, kwargs) pair.
        """
        refDict = MergeSourcesRunner.buildRefDict(parsedCmd)
        kwargs["psfCache"] = parsedCmd.psfCache
        return [(list(p.values()), kwargs) for t in refDict.values() for p in t.values()]

makeTask()

def lsst.pipe.tasks.multiBandUtils.MergeSourcesRunner.makeTask (   self,   parsedCmd = None,   args = None  )

inherited

Provide a butler to the Task constructor.

Parameters
----------
parsedCmd:
    The parsed command
args: tuple
    Tuple of a list of data references and kwargs (un-used)

Raises
------
RuntimeError
    Thrown if both `parsedCmd` & `args` are `None`

Definition at line 16 of file multiBandUtils.py.

    def makeTask(self, parsedCmd=None, args=None):
        """Provide a butler to the Task constructor.

        Parameters
        ----------
        parsedCmd:
            The parsed command
        args: tuple
            Tuple of a list of data references and kwargs (un-used)

        Raises
        ------
        RuntimeError
            Thrown if both `parsedCmd` & `args` are `None`
        """
        if parsedCmd is not None:
            butler = parsedCmd.butler
        elif args is not None:
            dataRefList, kwargs = args
            butler = dataRefList[0].getButler()
        else:
            raise RuntimeError("Neither parsedCmd or args specified")
        return self.TaskClass(config=self.config, log=self.log, butler=butler)

precall()

def lsst.pipe.base.cmdLineTask.TaskRunner.precall (   self,   parsedCmd  )

inherited

Hook for code that should run exactly once, before multiprocessing.

Notes
-----
Must return True if `TaskRunner.__call__` should subsequently be called.

.. warning::

   Implementations must take care to ensure that no unpicklable attributes are added to the
   TaskRunner itself, for compatibility with multiprocessing.

The default implementation writes package versions, schemas and configs, or compares them to existing
files on disk if present.

Definition at line 312 of file cmdLineTask.py.

    def precall(self, parsedCmd):
        """Hook for code that should run exactly once, before multiprocessing.

        Notes
        -----
        Must return True if `TaskRunner.__call__` should subsequently be called.

        .. warning::

           Implementations must take care to ensure that no unpicklable attributes are added to the
           TaskRunner itself, for compatibility with multiprocessing.

        The default implementation writes package versions, schemas and configs, or compares them to existing
        files on disk if present.
        """
        task = self.makeTask(parsedCmd=parsedCmd)

        if self.doRaise:
            self._precallImpl(task, parsedCmd)
        else:
            try:
                self._precallImpl(task, parsedCmd)
            except Exception as e:
                task.log.fatal("Failed in task initialization: %s", e)
                if not isinstance(e, TaskError):
                    traceback.print_exc(file=sys.stderr)
                return False
        return True

prepareForMultiProcessing()

def lsst.pipe.base.cmdLineTask.TaskRunner.prepareForMultiProcessing (   self )

inherited

Prepare this instance for multiprocessing

Optional non-picklable elements are removed.

This is only called if the task is run under multiprocessing.

Definition at line 174 of file cmdLineTask.py.

    def prepareForMultiProcessing(self):
        """Prepare this instance for multiprocessing

        Optional non-picklable elements are removed.

        This is only called if the task is run under multiprocessing.
        """
        self.log = None

run()

def lsst.pipe.base.cmdLineTask.TaskRunner.run (   self,   parsedCmd  )

inherited

Run the task on all targets.

Parameters
----------
parsedCmd : `argparse.Namespace`
    Parsed command `argparse.Namespace`.

Returns
-------
resultList : `list`
    A list of results returned by `TaskRunner.__call__`, or an empty list if `TaskRunner.__call__`
    is not called (e.g. if `TaskRunner.precall` returns `False`). See `TaskRunner.__call__`
    for details.

Notes
-----
The task is run under multiprocessing if `TaskRunner.numProcesses` is more than 1; otherwise
processing is serial.

Definition at line 183 of file cmdLineTask.py.

    def run(self, parsedCmd):
        """Run the task on all targets.

        Parameters
        ----------
        parsedCmd : `argparse.Namespace`
            Parsed command `argparse.Namespace`.

        Returns
        -------
        resultList : `list`
            A list of results returned by `TaskRunner.__call__`, or an empty list if `TaskRunner.__call__`
            is not called (e.g. if `TaskRunner.precall` returns `False`). See `TaskRunner.__call__`
            for details.

        Notes
        -----
        The task is run under multiprocessing if `TaskRunner.numProcesses` is more than 1; otherwise
        processing is serial.
        """
        resultList = []
        disableImplicitThreading()  # To prevent thread contention
        if self.numProcesses > 1:
            import multiprocessing
            self.prepareForMultiProcessing()
            pool = multiprocessing.Pool(processes=self.numProcesses, maxtasksperchild=1)
            mapFunc = functools.partial(_runPool, pool, self.timeout)
        else:
            pool = None
            mapFunc = map

        if self.precall(parsedCmd):
            profileName = parsedCmd.profile if hasattr(parsedCmd, "profile") else None
            log = parsedCmd.log
            targetList = self.getTargetList(parsedCmd)
            if len(targetList) > 0:
                with profile(profileName, log):
                    # Run the task using self.__call__
                    resultList = list(mapFunc(self, targetList))
            else:
                log.warn("Not running the task because there is no data to process; "
                         "you may preview data using \"--show data\"")

        if pool is not None:
            pool.close()
            pool.join()

        return resultList

runTask()

def lsst.pipe.base.cmdLineTask.TaskRunner.runTask (   self,   task,   dataRef,   kwargs  )

inherited

Make the actual call to `runDataRef` for this task.

Parameters
----------
task : `lsst.pipe.base.CmdLineTask` class
    The class of the task to run.
dataRef
    Butler data reference that contains the data the task will process.
kwargs
    Any additional keyword arguments.  See `TaskRunner.getTargetList` above.

Notes
-----
The default implementation of `TaskRunner.runTask` works for any command-line task which has a
runDataRef method that takes a data reference and an optional set of additional keyword arguments.
This method returns the results generated by the task's `runDataRef` method.

Definition at line 428 of file cmdLineTask.py.

    def runTask(self, task, dataRef, kwargs):
        """Make the actual call to `runDataRef` for this task.

        Parameters
        ----------
        task : `lsst.pipe.base.CmdLineTask` class
            The class of the task to run.
        dataRef
            Butler data reference that contains the data the task will process.
        kwargs
            Any additional keyword arguments.  See `TaskRunner.getTargetList` above.

        Notes
        -----
        The default implementation of `TaskRunner.runTask` works for any command-line task which has a
        runDataRef method that takes a data reference and an optional set of additional keyword arguments.
        This method returns the results generated by the task's `runDataRef` method.

        """
        return task.runDataRef(dataRef, **kwargs)

Member Data Documentation

clobberConfig

lsst.pipe.base.cmdLineTask.TaskRunner.clobberConfig

inherited

Definition at line 161 of file cmdLineTask.py.

config

lsst.pipe.base.cmdLineTask.TaskRunner.config

inherited

Definition at line 158 of file cmdLineTask.py.

doBackup

lsst.pipe.base.cmdLineTask.TaskRunner.doBackup

inherited

Definition at line 162 of file cmdLineTask.py.

doRaise

lsst.pipe.base.cmdLineTask.TaskRunner.doRaise

inherited

Definition at line 160 of file cmdLineTask.py.

doReturnResults

lsst.pipe.base.cmdLineTask.TaskRunner.doReturnResults

inherited

Definition at line 157 of file cmdLineTask.py.

log

lsst.pipe.base.cmdLineTask.TaskRunner.log

inherited

Definition at line 159 of file cmdLineTask.py.

numProcesses

lsst.pipe.base.cmdLineTask.TaskRunner.numProcesses

inherited

Definition at line 163 of file cmdLineTask.py.

TaskClass

lsst.pipe.base.cmdLineTask.TaskRunner.TaskClass

inherited

Definition at line 156 of file cmdLineTask.py.

TIMEOUT

int lsst.pipe.base.cmdLineTask.TaskRunner.TIMEOUT = 3600*24*30

staticinherited

Definition at line 152 of file cmdLineTask.py.

timeout

lsst.pipe.base.cmdLineTask.TaskRunner.timeout

inherited

Definition at line 165 of file cmdLineTask.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/Linux64/pipe_tasks/18.1.0-11-gb2589d7b/python/lsst/pipe/tasks/multiBand.py