lsst.pipe.drivers.singleFrameDriver.SingleFrameDriverTask Class Reference

Inheritance diagram for lsst.pipe.drivers.singleFrameDriver.SingleFrameDriverTask:

Public Member Functions
def	__init__ (self, butler=None, psfRefObjLoader=None, astromRefObjLoader=None, photoRefObjLoader=None, *args, **kwargs)
	Constructor. More...
def	runDataRef (self, sensorRef)
def	parseAndRun (cls, *args, **kwargs)
def	parseAndSubmit (cls, args=None, **kwargs)
def	batchWallTime (cls, time, parsedCmd, numCores)
	Return walltime request for batch job. More...
def	batchCommand (cls, args)
	Return command to run CmdLineTask. More...
def	logOperation (self, operation, catch=False, trace=True)
	Provide a context manager for logging an operation. More...

Public Attributes
	ignoreCcds

Static Public Attributes
	ConfigClass = SingleFrameDriverConfig
	RunnerClass = SingleFrameTaskRunner

Detailed Description

Process CCDs in parallel

Definition at line 31 of file singleFrameDriver.py.

Constructor & Destructor Documentation

__init__()

def lsst.pipe.drivers.singleFrameDriver.SingleFrameDriverTask.__init__	(		self,
			butler = None,
			psfRefObjLoader = None,
			astromRefObjLoader = None,
			photoRefObjLoader = None,
		*	args,
		**	kwargs
	)

Constructor.

The psfRefObjLoader, astromRefObjLoader, photoRefObjLoader should be an instance of LoadReferenceObjectsTasks that supplies an external reference catalog. They may be None if the butler argument is provided or the particular reference catalog is not required.

Parameters

[in]	butler	The butler is passed to the refObjLoader constructor in case it is needed. Ignored if the refObjLoader argument provides a loader directly.
[in]	psfRefObjLoader	Reference catalog loader for PSF determination.
[in]	astromRefObjLoader	Reference catalog loader for astrometric calibration.
[in]	photoRefObjLoader	Reference catalog loader for photometric calibration.
[in,out]	kwargs	other keyword arguments for lsst.ctrl.pool.BatchParallelTask

Definition at line 38 of file singleFrameDriver.py.

                 *args, **kwargs):
        """!
        Constructor
 
        The psfRefObjLoader, astromRefObjLoader, photoRefObjLoader should
        be an instance of LoadReferenceObjectsTasks that supplies an external
        reference catalog. They may be None if the butler argument is
        provided or the particular reference catalog is not required.
 
        @param[in] butler  The butler is passed to the refObjLoader constructor in case it is
            needed.  Ignored if the refObjLoader argument provides a loader directly.
        @param[in] psfRefObjLoader  Reference catalog loader for PSF determination.
        @param[in] astromRefObjLoader  Reference catalog loader for astrometric calibration.
        @param[in] photoRefObjLoader Reference catalog loader for photometric calibration.
        @param[in,out] kwargs  other keyword arguments for lsst.ctrl.pool.BatchParallelTask
        """
        BatchParallelTask.__init__(self, *args, **kwargs)
        self.ignoreCcds = set(self.config.ignoreCcdList)
        self.makeSubtask("processCcd", butler=butler, psfRefObjLoader=psfRefObjLoader,
                         astromRefObjLoader=astromRefObjLoader, photoRefObjLoader=photoRefObjLoader)
        if self.config.doMakeSourceTable:
            self.makeSubtask("writeSourceTable")
            self.makeSubtask("transformSourceTable")
 

Member Function Documentation

batchCommand()

def lsst.ctrl.pool.parallel.BatchCmdLineTask.batchCommand (   cls,   args  )

inherited

Return command to run CmdLineTask.

    @param cls: Class
    @param args: Parsed batch job arguments (from BatchArgumentParser)

Definition at line 476 of file parallel.py.

    def batchCommand(cls, args):
        """!Return command to run CmdLineTask
 
        @param cls: Class
        @param args: Parsed batch job arguments (from BatchArgumentParser)
        """
        job = args.job if args.job is not None else "job"
        module = cls.__module__
        script = ("import os; os.umask(%#05o); " +
                  "import lsst.base; lsst.base.disableImplicitThreading(); " +
                  "import lsst.ctrl.pool.log; lsst.ctrl.pool.log.jobLog(\"%s\"); ") % (UMASK, job)
 
        if args.batchStats:
            script += ("import lsst.ctrl.pool.parallel; import atexit; " +
                       "atexit.register(lsst.ctrl.pool.parallel.printProcessStats); ")
 
        script += "import %s; %s.%s.parseAndRun();" % (module, module, cls.__name__)
 
        profilePre = "import cProfile; import os; cProfile.run(\"\"\""
        profilePost = "\"\"\", filename=\"profile-" + job + "-%s-%d.dat\" % (os.uname()[1], os.getpid()))"
 
        return ("python -c '" + (profilePre if args.batchProfile else "") + script +
                (profilePost if args.batchProfile else "") + "' " + shCommandFromArgs(args.leftover) +
                " --noExit")
 

batchWallTime()

def lsst.ctrl.pool.parallel.BatchCmdLineTask.batchWallTime (   cls,   time,   parsedCmd,   numCores  )

inherited

Return walltime request for batch job.

    Subclasses should override if the walltime should be calculated
    differently (e.g., addition of some serial time).

    @param cls: Class
    @param time: Requested time per iteration
    @param parsedCmd: Results of argument parsing
    @param numCores: Number of cores

Reimplemented in lsst.pipe.drivers.multiBandDriver.MultiBandDriverTask, lsst.pipe.drivers.visualizeVisit.VisualizeVisitTask, lsst.pipe.drivers.skyCorrection.SkyCorrectionTask, lsst.pipe.drivers.ingestDriver.PoolIngestTask, lsst.pipe.drivers.constructCalibs.CalibTask, lsst.pipe.drivers.coaddDriver.CoaddDriverTask, and lsst.ctrl.pool.test.demoTask.DemoTask.

Definition at line 461 of file parallel.py.

    def batchWallTime(cls, time, parsedCmd, numCores):
        """!Return walltime request for batch job
 
        Subclasses should override if the walltime should be calculated
        differently (e.g., addition of some serial time).
 
        @param cls: Class
        @param time: Requested time per iteration
        @param parsedCmd: Results of argument parsing
        @param numCores: Number of cores
        """
        numTargets = len(cls.RunnerClass.getTargetList(parsedCmd))
        return time*numTargets/float(numCores)
 

logOperation()

def lsst.ctrl.pool.parallel.BatchCmdLineTask.logOperation (   self,   operation,   catch = False,   trace = True  )

inherited

Provide a context manager for logging an operation.

    @param operation: description of operation (string)
    @param catch: Catch all exceptions?
    @param trace: Log a traceback of caught exception?

    Note that if 'catch' is True, all exceptions are swallowed, but there may
    be other side-effects such as undefined variables.

Definition at line 502 of file parallel.py.

    def logOperation(self, operation, catch=False, trace=True):
        """!Provide a context manager for logging an operation
 
        @param operation: description of operation (string)
        @param catch: Catch all exceptions?
        @param trace: Log a traceback of caught exception?
 
        Note that if 'catch' is True, all exceptions are swallowed, but there may
        be other side-effects such as undefined variables.
        """
        self.log.info("%s: Start %s" % (NODE, operation))
        try:
            yield
        except Exception:
            if catch:
                cls, e, _ = sys.exc_info()
                self.log.warn("%s: Caught %s while %s: %s" % (NODE, cls.__name__, operation, e))
                if trace:
                    self.log.info("%s: Traceback:\n%s" % (NODE, traceback.format_exc()))
                return
            raise
        finally:
            self.log.info("%s: Finished %s" % (NODE, operation))
 
 

parseAndRun()

def lsst.ctrl.pool.parallel.BatchParallelTask.parseAndRun (   cls, *  args, **  kwargs  )

inherited

Parse an argument list and run the command

This is the entry point when we run in earnest, so start the process pool
so that the worker nodes don't go any further.

Definition at line 615 of file parallel.py.

    def parseAndRun(cls, *args, **kwargs):
        """Parse an argument list and run the command
 
        This is the entry point when we run in earnest, so start the process pool
        so that the worker nodes don't go any further.
        """
        pool = startPool()
        results = super(BatchParallelTask, cls).parseAndRun(*args, **kwargs)
        pool.exit()
        return results

parseAndSubmit()

def lsst.ctrl.pool.parallel.BatchCmdLineTask.parseAndSubmit (   cls,   args = None, **  kwargs  )

inherited

Definition at line 435 of file parallel.py.

    def parseAndSubmit(cls, args=None, **kwargs):
        taskParser = cls._makeArgumentParser(doBatch=True, add_help=False)
        batchParser = BatchArgumentParser(parent=taskParser)
        batchArgs = batchParser.parse_args(config=cls.ConfigClass(), args=args, override=cls.applyOverrides,
                                           **kwargs)
 
        if not cls.RunnerClass(cls, batchArgs.parent).precall(batchArgs.parent):  # Write config, schema
            taskParser.error("Error in task preparation")
 
        setBatchType(batchArgs.batch)
 
        if batchArgs.batch is None:     # don't use a batch system
            sys.argv = [sys.argv[0]] + batchArgs.leftover  # Remove all batch arguments
 
            return cls.parseAndRun()
        else:
            if batchArgs.walltime > 0:
                walltime = batchArgs.walltime
            else:
                numCores = batchArgs.cores if batchArgs.cores > 0 else batchArgs.nodes*batchArgs.procs
                walltime = cls.batchWallTime(batchArgs.time, batchArgs.parent, numCores)
 
            command = cls.batchCommand(batchArgs)
            batchArgs.batch.run(command, walltime=walltime)
 

runDataRef()

def lsst.pipe.drivers.singleFrameDriver.SingleFrameDriverTask.runDataRef	(		self,
			sensorRef
	)

Process a single CCD, with scatter-gather-scatter using MPI.

Definition at line 74 of file singleFrameDriver.py.

    def runDataRef(self, sensorRef):
        """Process a single CCD, with scatter-gather-scatter using MPI.
        """
        if sensorRef.dataId[self.config.ccdKey] in self.ignoreCcds:
            self.log.warn("Ignoring %s: CCD in ignoreCcdList" %
                          (sensorRef.dataId))
            return None
 
        with self.logOperation("processing %s" % (sensorRef.dataId,)):
            result = self.processCcd.runDataRef(sensorRef)
            if self.config.doMakeSourceTable:
                parquet = self.writeSourceTable.run(result.calibRes.sourceCat,
                                                    ccdVisitId=sensorRef.get('ccdExposureId'))
                if self.config.doSaveWideSourceTable:
                    sensorRef.put(parquet.table, 'source')
 
                df = self.transformSourceTable.run(parquet.table,
                                                   funcs=self.transformSourceTable.getFunctors(),
                                                   dataId=sensorRef.dataId)
                self.transformSourceTable.write(df, sensorRef)
 
        return result

Member Data Documentation

ConfigClass

lsst.pipe.drivers.singleFrameDriver.SingleFrameDriverTask.ConfigClass = SingleFrameDriverConfig

static

Definition at line 34 of file singleFrameDriver.py.

ignoreCcds

lsst.pipe.drivers.singleFrameDriver.SingleFrameDriverTask.ignoreCcds

Definition at line 56 of file singleFrameDriver.py.

RunnerClass

lsst.pipe.drivers.singleFrameDriver.SingleFrameDriverTask.RunnerClass = SingleFrameTaskRunner

static

Definition at line 36 of file singleFrameDriver.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-0.7.0/Linux64/pipe_drivers/22.0.0+a28f4c53b1/python/lsst/pipe/drivers/singleFrameDriver.py