LSST Applications
21.0.0-147-g0e635eb1+1acddb5be5,22.0.0+052faf71bd,22.0.0+1ea9a8b2b2,22.0.0+6312710a6c,22.0.0+729191ecac,22.0.0+7589c3a021,22.0.0+9f079a9461,22.0.1-1-g7d6de66+b8044ec9de,22.0.1-1-g87000a6+536b1ee016,22.0.1-1-g8e32f31+6312710a6c,22.0.1-10-gd060f87+016f7cdc03,22.0.1-12-g9c3108e+df145f6f68,22.0.1-16-g314fa6d+c825727ab8,22.0.1-19-g93a5c75+d23f2fb6d8,22.0.1-19-gb93eaa13+aab3ef7709,22.0.1-2-g8ef0a89+b8044ec9de,22.0.1-2-g92698f7+9f079a9461,22.0.1-2-ga9b0f51+052faf71bd,22.0.1-2-gac51dbf+052faf71bd,22.0.1-2-gb66926d+6312710a6c,22.0.1-2-gcb770ba+09e3807989,22.0.1-20-g32debb5+b8044ec9de,22.0.1-23-gc2439a9a+fb0756638e,22.0.1-3-g496fd5d+09117f784f,22.0.1-3-g59f966b+1e6ba2c031,22.0.1-3-g849a1b8+f8b568069f,22.0.1-3-gaaec9c0+c5c846a8b1,22.0.1-32-g5ddfab5d3+60ce4897b0,22.0.1-4-g037fbe1+64e601228d,22.0.1-4-g8623105+b8044ec9de,22.0.1-5-g096abc9+d18c45d440,22.0.1-5-g15c806e+57f5c03693,22.0.1-7-gba73697+57f5c03693,master-g6e05de7fdc+c1283a92b8,master-g72cdda8301+729191ecac,w.2021.39
LSST Data Management Base Package
|
Public Member Functions | |
def | batchWallTime (cls, time, parsedCmd, numCores) |
Return walltime request for batch job. More... | |
def | parseAndRun (cls, *args, **kwargs) |
def | runFileWrapper (self, struct, args) |
def | run (self, args) |
def | writeConfig (self, *args, **kwargs) |
def | writeMetadata (self, *args, **kwargs) |
def | parseAndSubmit (cls, args=None, **kwargs) |
def | batchCommand (cls, args) |
Return command to run CmdLineTask. More... | |
def | logOperation (self, operation, catch=False, trace=True) |
Provide a context manager for logging an operation. More... | |
def | parseAndRun (cls) |
def | prepareTask (cls, root=None, dryrun=False, mode="move", create=False, ignoreIngested=False) |
def | ingest (self, infile, outfile, mode="move", dryrun=False) |
def | isBadFile (self, filename, badFileList) |
def | isBadId (self, info, badIdList) |
def | expandFiles (self, fileNameList) |
Expand a set of filenames and globs, returning a list of filenames. More... | |
def | runFile (self, infile, registry, args, pos) |
Examine and ingest a single file. More... | |
def | ingestFiles (self, fileList) |
Static Public Attributes | |
ConfigClass = IngestConfig | |
ArgumentParser = IngestArgumentParser | |
Parallel version of IngestTask
Definition at line 7 of file ingestDriver.py.
|
inherited |
Return command to run CmdLineTask.
@param cls: Class @param args: Parsed batch job arguments (from BatchArgumentParser)
Definition at line 476 of file parallel.py.
def lsst.pipe.drivers.ingestDriver.PoolIngestTask.batchWallTime | ( | cls, | |
time, | |||
parsedCmd, | |||
numCores | |||
) |
Return walltime request for batch job.
Subclasses should override if the walltime should be calculated differently (e.g., addition of some serial time). @param cls: Class @param time: Requested time per iteration @param parsedCmd: Results of argument parsing @param numCores: Number of cores
Reimplemented from lsst.ctrl.pool.parallel.BatchCmdLineTask.
Definition at line 10 of file ingestDriver.py.
|
inherited |
|
inherited |
|
inherited |
|
inherited |
|
inherited |
Return whether the file information qualifies as bad We match against the list of bad data identifiers.
Definition at line 545 of file ingest.py.
|
inherited |
Provide a context manager for logging an operation.
@param operation: description of operation (string) @param catch: Catch all exceptions? @param trace: Log a traceback of caught exception? Note that if 'catch' is True, all exceptions are swallowed, but there may be other side-effects such as undefined variables.
Definition at line 502 of file parallel.py.
|
inherited |
def lsst.pipe.drivers.ingestDriver.PoolIngestTask.parseAndRun | ( | cls, | |
* | args, | ||
** | kwargs | ||
) |
Run with a MPI process pool
Definition at line 24 of file ingestDriver.py.
|
inherited |
Definition at line 435 of file parallel.py.
|
inherited |
Prepare for running the task repeatedly with `ingestFiles`. Saves the parsed arguments, including the Butler and log, as a private instance variable. Parameters ---------- root : `str`, optional Repository root pathname. If None, run the Task using the command line arguments, ignoring all other arguments below. dryrun : `bool`, optional If True, don't perform any action; log what would have happened. mode : `str`, optional How files are delivered to their destination. Default is "move", unlike the command-line default of "link". create : `bool`, optional If True, create a new registry, clobbering any old one present. ignoreIngested : `bool`, optional If True, do not complain if the file is already present in the registry (and do nothing else). Returns ------- task : `IngestTask` If `root` was provided, the IngestTask instance
Definition at line 434 of file ingest.py.
def lsst.pipe.drivers.ingestDriver.PoolIngestTask.run | ( | self, | |
args | |||
) |
Run ingest We read and ingest the files in parallel, and then stuff the registry database in serial.
Reimplemented from lsst.pipe.tasks.ingest.IngestTask.
Definition at line 60 of file ingestDriver.py.
|
inherited |
Examine and ingest a single file.
@param infile: File to process @param registry: Registry into which to insert Butler metadata, or None @param args: Parsed command-line arguments @param pos: Position number of this file in the input list @return parsed information from FITS HDUs if registry is None; or None
Definition at line 576 of file ingest.py.
def lsst.pipe.drivers.ingestDriver.PoolIngestTask.runFileWrapper | ( | self, | |
struct, | |||
args | |||
) |
Run ingest on one file This is a wrapper method for calling ``runFile``. Parameters ---------- struct : `lsst.pipe.base.Struct` Structure containing ``filename`` (`str`) and ``position`` (`int`). args : `argparse.Namespace` Parsed command-line arguments. Returns ------- hduInfoList : `list` of `dict` Parsed information from FITS HDUs, or ``None``.
Definition at line 34 of file ingestDriver.py.
def lsst.pipe.drivers.ingestDriver.PoolIngestTask.writeConfig | ( | self, | |
* | args, | ||
** | kwargs | ||
) |
Definition at line 82 of file ingestDriver.py.
def lsst.pipe.drivers.ingestDriver.PoolIngestTask.writeMetadata | ( | self, | |
* | args, | ||
** | kwargs | ||
) |
Definition at line 85 of file ingestDriver.py.
|
staticinherited |
|
staticinherited |