LSSTApplications
18.0.0+106,18.0.0+50,19.0.0,19.0.0+1,19.0.0+10,19.0.0+11,19.0.0+13,19.0.0+17,19.0.0+2,19.0.0-1-g20d9b18+6,19.0.0-1-g425ff20,19.0.0-1-g5549ca4,19.0.0-1-g580fafe+6,19.0.0-1-g6fe20d0+1,19.0.0-1-g7011481+9,19.0.0-1-g8c57eb9+6,19.0.0-1-gb5175dc+11,19.0.0-1-gdc0e4a7+9,19.0.0-1-ge272bc4+6,19.0.0-1-ge3aa853,19.0.0-10-g448f008b,19.0.0-12-g6990b2c,19.0.0-2-g0d9f9cd+11,19.0.0-2-g3d9e4fb2+11,19.0.0-2-g5037de4,19.0.0-2-gb96a1c4+3,19.0.0-2-gd955cfd+15,19.0.0-3-g2d13df8,19.0.0-3-g6f3c7dc,19.0.0-4-g725f80e+11,19.0.0-4-ga671dab3b+1,19.0.0-4-gad373c5+3,19.0.0-5-ga2acb9c+2,19.0.0-5-gfe96e6c+2,w.2020.01
LSSTDataManagementBasePackage
|
Base class for constructing calibs. More...
Public Member Functions | |
def | __init__ (self, args, kwargs) |
def | batchWallTime (cls, time, parsedCmd, numCores) |
def | runDataRef (self, expRefList, butler, calibId) |
Construct a calib from a list of exposure references. More... | |
def | getOutputId (self, expRefList, calibId) |
Generate the data identifier for the output calib. More... | |
def | getMjd (self, butler, dataId, timescale=dafBase.DateTime.UTC) |
def | getFilter (self, butler, dataId) |
def | addMissingKeys (self, dataId, butler, missingKeys=None, calibName=None) |
def | updateMetadata (self, calibImage, exposureTime, darkTime=None, kwargs) |
Update the metadata from the VisitInfo. More... | |
def | scatterProcess (self, pool, ccdIdLists) |
Scatter the processing among the nodes. More... | |
def | process (self, cache, ccdId, outputName="postISRCCD", kwargs) |
Process a CCD, specified by a data identifier. More... | |
def | processSingle (self, dataRef) |
def | processWrite (self, dataRef, exposure, outputName="postISRCCD") |
Write the processed CCD. More... | |
def | processResult (self, exposure) |
def | scale (self, ccdIdLists, data) |
Determine scaling across CCDs and exposures. More... | |
def | scatterCombine (self, pool, outputId, ccdIdLists, scales) |
Scatter the combination of exposures across multiple nodes. More... | |
def | getFullyQualifiedOutputId (self, ccdName, butler, outputId) |
def | combine (self, cache, struct, outputId) |
Combine multiple exposures of a particular CCD and write the output. More... | |
def | calculateOutputHeaderFromRaws (self, butler, calib, dataIdList, outputId) |
Calculate the output header from the raw headers. More... | |
def | recordCalibInputs (self, butler, calib, dataIdList, outputId) |
Record metadata including the inputs and creation details. More... | |
def | interpolateNans (self, image) |
def | write (self, butler, exposure, dataId) |
Write the final combined calib. More... | |
def | makeCameraImage (self, camera, dataId, calibs) |
Create and write an image of the entire camera. More... | |
def | checkCcdIdLists (self, ccdIdLists) |
def | parseAndRun (cls, args, kwargs) |
def | parseAndRun (cls, args=None, config=None, log=None, doReturnResults=False) |
def | parseAndSubmit (cls, args=None, kwargs) |
def | batchCommand (cls, args) |
Return command to run CmdLineTask. More... | |
def | logOperation (self, operation, catch=False, trace=True) |
Provide a context manager for logging an operation. More... | |
def | applyOverrides (cls, config) |
def | writeConfig (self, butler, clobber=False, doBackup=True) |
def | writeSchemas (self, butler, clobber=False, doBackup=True) |
def | writeMetadata (self, dataRef) |
def | writePackageVersions (self, butler, clobber=False, doBackup=True, dataset="packages") |
def | emptyMetadata (self) |
def | getSchemaCatalogs (self) |
def | getAllSchemaCatalogs (self) |
def | getFullMetadata (self) |
def | getFullName (self) |
def | getName (self) |
def | getTaskDict (self) |
def | makeSubtask (self, name, keyArgs) |
def | timer (self, name, logLevel=Log.DEBUG) |
def | makeField (cls, doc) |
def | __reduce__ (self) |
Public Attributes | |
metadata | |
log | |
config | |
Static Public Attributes | |
ConfigClass = CalibConfig | |
RunnerClass = CalibTaskRunner | |
filterName = None | |
calibName = None | |
float | exposureTime = 1.0 |
bool | canMultiprocess = True |
Base class for constructing calibs.
This should be subclassed for each of the required calib types. The subclass should be sure to define the following class variables:
Definition at line 387 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.__init__ | ( | self, | |
args, | |||
kwargs | |||
) |
Constructor
Definition at line 403 of file constructCalibs.py.
|
inherited |
Pickler.
def lsst.pipe.drivers.constructCalibs.CalibTask.addMissingKeys | ( | self, | |
dataId, | |||
butler, | |||
missingKeys = None , |
|||
calibName = None |
|||
) |
Definition at line 549 of file constructCalibs.py.
|
inherited |
A hook to allow a task to change the values of its config *after* the camera-specific overrides are loaded but before any command-line overrides are applied. Parameters ---------- config : instance of task's ``ConfigClass`` Task configuration. Notes ----- This is necessary in some cases because the camera-specific overrides may retarget subtasks, wiping out changes made in ConfigClass.setDefaults. See LSST Trac ticket #2282 for more discussion. .. warning:: This is called by CmdLineTask.parseAndRun; other ways of constructing a config will not apply these overrides.
Definition at line 527 of file cmdLineTask.py.
|
inherited |
Return command to run CmdLineTask.
cls | Class |
args | Parsed batch job arguments (from BatchArgumentParser) |
Definition at line 471 of file parallel.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.batchWallTime | ( | cls, | |
time, | |||
parsedCmd, | |||
numCores | |||
) |
Definition at line 410 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.calculateOutputHeaderFromRaws | ( | self, | |
butler, | |||
calib, | |||
dataIdList, | |||
outputId | |||
) |
Calculate the output header from the raw headers.
This metadata will go into the output FITS header. It will include all headers that are identical in all inputs.
butler | Data butler |
calib | Combined calib exposure. |
dataIdList | List of data identifiers for calibration inputs |
outputId | Data identifier for output |
Definition at line 766 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.checkCcdIdLists | ( | self, | |
ccdIdLists | |||
) |
Check that the list of CCD dataIds is consistent @param ccdIdLists Dict of data identifier lists for each CCD name @return Number of exposures, number of CCDs
Definition at line 886 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.combine | ( | self, | |
cache, | |||
struct, | |||
outputId | |||
) |
Combine multiple exposures of a particular CCD and write the output.
Only the slave nodes execute this method.
cache | Process pool cache |
struct | Parameters for the combination, which has the following components:
|
outputId | Data identifier for combined image (exposure part only) |
Definition at line 726 of file constructCalibs.py.
|
inherited |
Empty (clear) the metadata for this Task and all sub-Tasks.
Definition at line 153 of file task.py.
|
inherited |
Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict. Returns ------- schemacatalogs : `dict` Keys are butler dataset type, values are a empty catalog (an instance of the appropriate lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down through all subtasks. Notes ----- This method may be called on any task in the hierarchy; it will return the same answer, regardless. The default implementation should always suffice. If your subtask uses schemas the override `Task.getSchemaCatalogs`, not this method.
def lsst.pipe.drivers.constructCalibs.CalibTask.getFilter | ( | self, | |
butler, | |||
dataId | |||
) |
Determine the filter from a data identifier
Definition at line 544 of file constructCalibs.py.
|
inherited |
Get metadata for all tasks. Returns ------- metadata : `lsst.daf.base.PropertySet` The `~lsst.daf.base.PropertySet` keys are the full task name. Values are metadata for the top-level task and all subtasks, sub-subtasks, etc.. Notes ----- The returned metadata includes timing information (if ``@timer.timeMethod`` is used) and any metadata set by the task. The name of each item consists of the full task name with ``.`` replaced by ``:``, followed by ``.`` and the name of the item, e.g.:: topLevelTaskName:subtaskName:subsubtaskName.itemName using ``:`` in the full task name disambiguates the rare situation that a task has a subtask and a metadata item with the same name.
Definition at line 210 of file task.py.
|
inherited |
Get the task name as a hierarchical name including parent task names. Returns ------- fullName : `str` The full name consists of the name of the parent task and each subtask separated by periods. For example: - The full name of top-level task "top" is simply "top". - The full name of subtask "sub" of top-level task "top" is "top.sub". - The full name of subtask "sub2" of subtask "sub" of top-level task "top" is "top.sub.sub2".
Definition at line 235 of file task.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.getFullyQualifiedOutputId | ( | self, | |
ccdName, | |||
butler, | |||
outputId | |||
) |
Get fully-qualified output data identifier We may need to look up keys that aren't in the output dataId. @param ccdName Name tuple for CCD @param butler Data butler @param outputId Data identifier for combined image (exposure part only) @return fully-qualified output dataId
Definition at line 710 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.getMjd | ( | self, | |
butler, | |||
dataId, | |||
timescale = dafBase.DateTime.UTC |
|||
) |
Determine the Modified Julian Date (MJD; in TAI) from a data identifier
Definition at line 531 of file constructCalibs.py.
|
inherited |
Get the name of the task. Returns ------- taskName : `str` Name of the task. See also -------- getFullName
def lsst.pipe.drivers.constructCalibs.CalibTask.getOutputId | ( | self, | |
expRefList, | |||
calibId | |||
) |
Generate the data identifier for the output calib.
The mean date and the common filter are included, using keywords from the configuration. The CCD-specific part is not included in the data identifier.
expRefList | List of data references at exposure level |
calibId | Data identifier elements for the calib provided by the user |
Definition at line 496 of file constructCalibs.py.
|
inherited |
Get the schemas generated by this task. Returns ------- schemaCatalogs : `dict` Keys are butler dataset type, values are an empty catalog (an instance of the appropriate `lsst.afw.table` Catalog type) for this task. Notes ----- .. warning:: Subclasses that use schemas must override this method. The default implemenation returns an empty dict. This method may be called at any time after the Task is constructed, which means that all task schemas should be computed at construction time, *not* when data is actually processed. This reflects the philosophy that the schema should not depend on the data. Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well. See also -------- Task.getAllSchemaCatalogs
|
inherited |
Get a dictionary of all tasks as a shallow copy. Returns ------- taskDict : `dict` Dictionary containing full task name: task object for the top-level task and all subtasks, sub-subtasks, etc..
Definition at line 264 of file task.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.interpolateNans | ( | self, | |
image | |||
) |
Interpolate over NANs in the combined image NANs can result from masked areas on the CCD. We don't want them getting into our science images, so we replace them with the median of the image.
Definition at line 847 of file constructCalibs.py.
|
inherited |
Provide a context manager for logging an operation.
operation | description of operation (string) |
catch | Catch all exceptions? |
trace | Log a traceback of caught exception? |
Note that if 'catch' is True, all exceptions are swallowed, but there may be other side-effects such as undefined variables.
Definition at line 497 of file parallel.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.makeCameraImage | ( | self, | |
camera, | |||
dataId, | |||
calibs | |||
) |
Create and write an image of the entire camera.
This is useful for judging the quality or getting an overview of the features of the calib.
camera | Camera object |
dataId | Data identifier for output |
calibs | Dict mapping CCD detector ID to calib image |
Definition at line 874 of file constructCalibs.py.
|
inherited |
Make a `lsst.pex.config.ConfigurableField` for this task. Parameters ---------- doc : `str` Help text for the field. Returns ------- configurableField : `lsst.pex.config.ConfigurableField` A `~ConfigurableField` for this task. Examples -------- Provides a convenient way to specify this task is a subtask of another task. Here is an example of use:: class OtherTaskConfig(lsst.pex.config.Config) aSubtask = ATaskClass.makeField("a brief description of what this task does")
Definition at line 329 of file task.py.
|
inherited |
Create a subtask as a new instance as the ``name`` attribute of this task. Parameters ---------- name : `str` Brief name of the subtask. keyArgs Extra keyword arguments used to construct the task. The following arguments are automatically provided and cannot be overridden: - "config". - "parentTask". Notes ----- The subtask must be defined by ``Task.config.name``, an instance of pex_config ConfigurableField or RegistryField.
|
inherited |
Run with a MPI process pool
Definition at line 529 of file parallel.py.
|
inherited |
Parse an argument list and run the command. Parameters ---------- args : `list`, optional List of command-line arguments; if `None` use `sys.argv`. config : `lsst.pex.config.Config`-type, optional Config for task. If `None` use `Task.ConfigClass`. log : `lsst.log.Log`-type, optional Log. If `None` use the default log. doReturnResults : `bool`, optional If `True`, return the results of this task. Default is `False`. This is only intended for unit tests and similar use. It can easily exhaust memory (if the task returns enough data and you call it enough times) and it will fail when using multiprocessing if the returned data cannot be pickled. Returns ------- struct : `lsst.pipe.base.Struct` Fields are: ``argumentParser`` the argument parser (`lsst.pipe.base.ArgumentParser`). ``parsedCmd`` the parsed command returned by the argument parser's `~lsst.pipe.base.ArgumentParser.parse_args` method (`argparse.Namespace`). ``taskRunner`` the task runner used to run the task (an instance of `Task.RunnerClass`). ``resultList`` results returned by the task runner's ``run`` method, one entry per invocation (`list`). This will typically be a list of `Struct`, each containing at least an ``exitStatus`` integer (0 or 1); see `Task.RunnerClass` (`TaskRunner` by default) for more details. Notes ----- Calling this method with no arguments specified is the standard way to run a command-line task from the command-line. For an example see ``pipe_tasks`` ``bin/makeSkyMap.py`` or almost any other file in that directory. If one or more of the dataIds fails then this routine will exit (with a status giving the number of failed dataIds) rather than returning this struct; this behaviour can be overridden by specifying the ``--noExit`` command-line option.
Definition at line 549 of file cmdLineTask.py.
|
inherited |
Definition at line 433 of file parallel.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.process | ( | self, | |
cache, | |||
ccdId, | |||
outputName = "postISRCCD" , |
|||
kwargs | |||
) |
Process a CCD, specified by a data identifier.
After processing, optionally returns a result (produced by the 'processResult' method) calculated from the processed exposure. These results will be gathered by the master node, and is a means for coordinated scaling of all CCDs for flats, etc.
Only slave nodes execute this method.
cache | Process pool cache |
ccdId | Data identifier for CCD |
outputName | Output dataset name for butler |
Definition at line 602 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.processResult | ( | self, | |
exposure | |||
) |
Extract processing results from a processed exposure This method generates what is gathered by the master node. This can be a background measurement or similar for scaling flat-fields. It must be picklable! Only slave nodes execute this method.
Definition at line 660 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.processSingle | ( | self, | |
dataRef | |||
) |
Process a single CCD, specified by a data reference Generally, this simply means doing ISR. Only slave nodes execute this method.
Definition at line 637 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.processWrite | ( | self, | |
dataRef, | |||
exposure, | |||
outputName = "postISRCCD" |
|||
) |
Write the processed CCD.
We need to write these out because we can't hold them all in memory at once.
Only slave nodes execute this method.
dataRef | Data reference |
exposure | CCD exposure to write |
outputName | Output dataset name for butler. |
Definition at line 646 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.recordCalibInputs | ( | self, | |
butler, | |||
calib, | |||
dataIdList, | |||
outputId | |||
) |
Record metadata including the inputs and creation details.
This metadata will go into the FITS header.
butler | Data butler |
calib | Combined calib exposure. |
dataIdList | List of data identifiers for calibration inputs |
outputId | Data identifier for output |
Definition at line 819 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.runDataRef | ( | self, | |
expRefList, | |||
butler, | |||
calibId | |||
) |
Construct a calib from a list of exposure references.
This is the entry point, called by the TaskRunner.__call__
Only the master node executes this method.
expRefList | List of data references at the exposure level |
butler | Data butler |
calibId | Identifier dict for calib |
Definition at line 422 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.scale | ( | self, | |
ccdIdLists, | |||
data | |||
) |
Determine scaling across CCDs and exposures.
This is necessary mainly for flats, so as to determine a consistent scaling across the entire focal plane. This implementation is simply a placeholder.
Only the master node executes this method.
ccdIdLists | Dict of data identifier lists for each CCD tuple |
data | Dict of lists of returned data for each CCD tuple |
Definition at line 671 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.scatterCombine | ( | self, | |
pool, | |||
outputId, | |||
ccdIdLists, | |||
scales | |||
) |
Scatter the combination of exposures across multiple nodes.
In this case, we can only scatter across as many nodes as there are CCDs.
Only the master node executes this method.
pool | Process pool |
outputId | Output identifier (exposure part only) |
ccdIdLists | Dict of data identifier lists for each CCD name |
scales | Dict of structs with scales, for each CCD name |
dict | of binned images |
Definition at line 690 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.scatterProcess | ( | self, | |
pool, | |||
ccdIdLists | |||
) |
Scatter the processing among the nodes.
We scatter each CCD independently (exposures aren't grouped together), to make full use of all available processors. This necessitates piecing everything back together in the same format as ccdIdLists afterwards.
Only the master node executes this method.
pool | Process pool |
ccdIdLists | Dict of data identifier lists for each CCD name |
Definition at line 586 of file constructCalibs.py.
|
inherited |
Context manager to log performance data for an arbitrary block of code. Parameters ---------- name : `str` Name of code being timed; data will be logged using item name: ``Start`` and ``End``. logLevel A `lsst.log` level constant. Examples -------- Creating a timer context:: with self.timer("someCodeToTime"): pass # code to time See also -------- timer.logInfo
Definition at line 301 of file task.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.updateMetadata | ( | self, | |
calibImage, | |||
exposureTime, | |||
darkTime = None , |
|||
kwargs | |||
) |
Update the metadata from the VisitInfo.
calibImage | The image whose metadata is to be set |
exposureTime | The exposure time for the image |
darkTime | The time since the last read (default: exposureTime) |
Definition at line 570 of file constructCalibs.py.
def lsst.pipe.drivers.constructCalibs.CalibTask.write | ( | self, | |
butler, | |||
exposure, | |||
dataId | |||
) |
Write the final combined calib.
Only the slave nodes execute this method
butler | Data butler |
exposure | CCD exposure to write |
dataId | Data identifier for output |
Definition at line 862 of file constructCalibs.py.
|
inherited |
Write the configuration used for processing the data, or check that an existing one is equal to the new one if present. Parameters ---------- butler : `lsst.daf.persistence.Butler` Data butler used to write the config. The config is written to dataset type `CmdLineTask._getConfigName`. clobber : `bool`, optional A boolean flag that controls what happens if a config already has been saved: - `True`: overwrite or rename the existing config, depending on ``doBackup``. - `False`: raise `TaskError` if this config does not match the existing config. doBackup : bool, optional Set to `True` to backup the config files if clobbering.
Definition at line 656 of file cmdLineTask.py.
|
inherited |
Write the metadata produced from processing the data. Parameters ---------- dataRef Butler data reference used to write the metadata. The metadata is written to dataset type `CmdLineTask._getMetadataName`.
Definition at line 731 of file cmdLineTask.py.
|
inherited |
Compare and write package versions. Parameters ---------- butler : `lsst.daf.persistence.Butler` Data butler used to read/write the package versions. clobber : `bool`, optional A boolean flag that controls what happens if versions already have been saved: - `True`: overwrite or rename the existing version info, depending on ``doBackup``. - `False`: raise `TaskError` if this version info does not match the existing. doBackup : `bool`, optional If `True` and clobbering, old package version files are backed up. dataset : `str`, optional Name of dataset to read/write. Raises ------ TaskError Raised if there is a version mismatch with current and persisted lists of package versions. Notes ----- Note that this operation is subject to a race condition.
Definition at line 747 of file cmdLineTask.py.
|
inherited |
Write the schemas returned by `lsst.pipe.base.Task.getAllSchemaCatalogs`. Parameters ---------- butler : `lsst.daf.persistence.Butler` Data butler used to write the schema. Each schema is written to the dataset type specified as the key in the dict returned by `~lsst.pipe.base.Task.getAllSchemaCatalogs`. clobber : `bool`, optional A boolean flag that controls what happens if a schema already has been saved: - `True`: overwrite or rename the existing schema, depending on ``doBackup``. - `False`: raise `TaskError` if this schema does not match the existing schema. doBackup : `bool`, optional Set to `True` to backup the schema files if clobbering. Notes ----- If ``clobber`` is `False` and an existing schema does not match a current schema, then some schemas may have been saved successfully and others may not, and there is no easy way to tell which is which.
Definition at line 696 of file cmdLineTask.py.
|
static |
Definition at line 400 of file constructCalibs.py.
|
staticinherited |
Definition at line 524 of file cmdLineTask.py.
|
static |
Definition at line 397 of file constructCalibs.py.
|
static |
Definition at line 401 of file constructCalibs.py.
|
static |
Definition at line 399 of file constructCalibs.py.
|
static |
Definition at line 398 of file constructCalibs.py.