LSST Applications  21.0.0+75b29a8a7f,21.0.0+e70536a077,21.0.0-1-ga51b5d4+62c747d40b,21.0.0-10-gbfb87ad6+3307648ee3,21.0.0-15-gedb9d5423+47cba9fc36,21.0.0-2-g103fe59+fdf0863a2a,21.0.0-2-g1367e85+d38a93257c,21.0.0-2-g45278ab+e70536a077,21.0.0-2-g5242d73+d38a93257c,21.0.0-2-g7f82c8f+e682ffb718,21.0.0-2-g8dde007+d179fbfa6a,21.0.0-2-g8f08a60+9402881886,21.0.0-2-ga326454+e682ffb718,21.0.0-2-ga63a54e+08647d4b1b,21.0.0-2-gde069b7+26c92b3210,21.0.0-2-gecfae73+0445ed2f95,21.0.0-2-gfc62afb+d38a93257c,21.0.0-27-gbbd0d29+ae871e0f33,21.0.0-28-g5fc5e037+feb0e9397b,21.0.0-3-g21c7a62+f4b9c0ff5c,21.0.0-3-g357aad2+57b0bddf0b,21.0.0-3-g4be5c26+d38a93257c,21.0.0-3-g65f322c+3f454acf5d,21.0.0-3-g7d9da8d+75b29a8a7f,21.0.0-3-gaa929c8+9e4ef6332c,21.0.0-3-ge02ed75+4b120a55c4,21.0.0-4-g3300ddd+e70536a077,21.0.0-4-g591bb35+4b120a55c4,21.0.0-4-gc004bbf+4911b9cd27,21.0.0-4-gccdca77+f94adcd104,21.0.0-4-ge8fba5a+2b3a696ff9,21.0.0-5-gb155db7+2c5429117a,21.0.0-5-gdf36809+637e4641ee,21.0.0-6-g00874e7+c9fd7f7160,21.0.0-6-g4e60332+4b120a55c4,21.0.0-7-gc8ca178+40eb9cf840,21.0.0-8-gfbe0b4b+9e4ef6332c,21.0.0-9-g2fd488a+d83b7cd606,w.2021.05
LSST Data Management Base Package
Public Member Functions | Public Attributes | Static Public Attributes | List of all members
lsst.obs.base.defineVisits.DefineVisitsTask Class Reference
Inheritance diagram for lsst.obs.base.defineVisits.DefineVisitsTask:
lsst.pipe.base.task.Task

Public Member Functions

def __init__ (self, Optional[DefineVisitsConfig] config=None, *Butler butler, **Any kwargs)
 
def run (self, Iterable[DataId] dataIds, *Optional[Pool] pool=None, int processes=1, Optional[str] collections=None)
 
def emptyMetadata (self)
 
def getSchemaCatalogs (self)
 
def getAllSchemaCatalogs (self)
 
def getFullMetadata (self)
 
def getFullName (self)
 
def getName (self)
 
def getTaskDict (self)
 
def makeSubtask (self, name, **keyArgs)
 
def timer (self, name, logLevel=Log.DEBUG)
 
def makeField (cls, doc)
 
def __reduce__ (self)
 

Public Attributes

 butler
 
 universe
 
 metadata
 
 log
 
 config
 

Static Public Attributes

 ConfigClass = DefineVisitsConfig
 

Detailed Description

Driver Task for defining visits (and their spatial regions) in Gen3
Butler repositories.

Parameters
----------
config : `DefineVisitsConfig`
    Configuration for the task.
butler : `~lsst.daf.butler.Butler`
    Writeable butler instance.  Will be used to read `raw.wcs` and `camera`
    datasets and insert/sync dimension data.
**kwargs
    Additional keyword arguments are forwarded to the `lsst.pipe.base.Task`
    constructor.

Notes
-----
Each instance of `DefineVisitsTask` reads from / writes to the same Butler.
Each invocation of `DefineVisitsTask.run` processes an independent group of
exposures into one or more new vists, all belonging to the same visit
system and instrument.

The actual work of grouping exposures and computing regions is delegated
to pluggable subtasks (`GroupExposuresTask` and `ComputeVisitRegionsTask`),
respectively.  The defaults are to create one visit for every exposure,
and to use exactly one (arbitrary) detector-level raw dataset's WCS along
with camera geometry to compute regions for all detectors.  Other
implementations can be created and configured for instruments for which
these choices are unsuitable (e.g. because visits and exposures are not
one-to-one, or because ``raw.wcs`` datasets for different detectors may not
be consistent with camera geomery).

It is not necessary in general to ingest all raws for an exposure before
defining a visit that includes the exposure; this depends entirely on the
`ComputeVisitRegionTask` subclass used.  For the default configuration,
a single raw for each exposure is sufficient.

Defining the same visit the same way multiple times (e.g. via multiple
invocations of this task on the same exposures, with the same
configuration) is safe, but it may be inefficient, as most of the work must
be done before new visits can be compared to existing visits.

Definition at line 281 of file defineVisits.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.obs.base.defineVisits.DefineVisitsTask.__init__ (   self,
Optional[DefineVisitsConfig]   config = None,
*Butler  butler,
**Any  kwargs 
)

Definition at line 323 of file defineVisits.py.

323  def __init__(self, config: Optional[DefineVisitsConfig] = None, *, butler: Butler, **kwargs: Any):
324  config.validate() # Not a CmdlineTask nor PipelineTask, so have to validate the config here.
325  super().__init__(config, **kwargs)
326  self.butler = butler
327  self.universe = self.butler.registry.dimensions
328  self.makeSubtask("groupExposures")
329  self.makeSubtask("computeVisitRegions", butler=self.butler)
330 

Member Function Documentation

◆ __reduce__()

def lsst.pipe.base.task.Task.__reduce__ (   self)
inherited
Pickler.

Reimplemented in lsst.pipe.drivers.multiBandDriver.MultiBandDriverTask, and lsst.pipe.drivers.coaddDriver.CoaddDriverTask.

Definition at line 432 of file task.py.

432  def __reduce__(self):
433  """Pickler.
434  """
435  return self._unpickle_via_factory, (self.__class__, [], self._reduce_kwargs())

◆ emptyMetadata()

def lsst.pipe.base.task.Task.emptyMetadata (   self)
inherited
Empty (clear) the metadata for this Task and all sub-Tasks.

Definition at line 166 of file task.py.

166  def emptyMetadata(self):
167  """Empty (clear) the metadata for this Task and all sub-Tasks.
168  """
169  for subtask in self._taskDict.values():
170  subtask.metadata = dafBase.PropertyList()
171 
Class for storing ordered metadata with comments.
Definition: PropertyList.h:68

◆ getAllSchemaCatalogs()

def lsst.pipe.base.task.Task.getAllSchemaCatalogs (   self)
inherited
Get schema catalogs for all tasks in the hierarchy, combining the
results into a single dict.

Returns
-------
schemacatalogs : `dict`
    Keys are butler dataset type, values are a empty catalog (an
    instance of the appropriate `lsst.afw.table` Catalog type) for all
    tasks in the hierarchy, from the top-level task down
    through all subtasks.

Notes
-----
This method may be called on any task in the hierarchy; it will return
the same answer, regardless.

The default implementation should always suffice. If your subtask uses
schemas the override `Task.getSchemaCatalogs`, not this method.

Definition at line 204 of file task.py.

204  def getAllSchemaCatalogs(self):
205  """Get schema catalogs for all tasks in the hierarchy, combining the
206  results into a single dict.
207 
208  Returns
209  -------
210  schemacatalogs : `dict`
211  Keys are butler dataset type, values are a empty catalog (an
212  instance of the appropriate `lsst.afw.table` Catalog type) for all
213  tasks in the hierarchy, from the top-level task down
214  through all subtasks.
215 
216  Notes
217  -----
218  This method may be called on any task in the hierarchy; it will return
219  the same answer, regardless.
220 
221  The default implementation should always suffice. If your subtask uses
222  schemas the override `Task.getSchemaCatalogs`, not this method.
223  """
224  schemaDict = self.getSchemaCatalogs()
225  for subtask in self._taskDict.values():
226  schemaDict.update(subtask.getSchemaCatalogs())
227  return schemaDict
228 

◆ getFullMetadata()

def lsst.pipe.base.task.Task.getFullMetadata (   self)
inherited
Get metadata for all tasks.

Returns
-------
metadata : `lsst.daf.base.PropertySet`
    The `~lsst.daf.base.PropertySet` keys are the full task name.
    Values are metadata for the top-level task and all subtasks,
    sub-subtasks, etc.

Notes
-----
The returned metadata includes timing information (if
``@timer.timeMethod`` is used) and any metadata set by the task. The
name of each item consists of the full task name with ``.`` replaced
by ``:``, followed by ``.`` and the name of the item, e.g.::

    topLevelTaskName:subtaskName:subsubtaskName.itemName

using ``:`` in the full task name disambiguates the rare situation
that a task has a subtask and a metadata item with the same name.

Definition at line 229 of file task.py.

229  def getFullMetadata(self):
230  """Get metadata for all tasks.
231 
232  Returns
233  -------
234  metadata : `lsst.daf.base.PropertySet`
235  The `~lsst.daf.base.PropertySet` keys are the full task name.
236  Values are metadata for the top-level task and all subtasks,
237  sub-subtasks, etc.
238 
239  Notes
240  -----
241  The returned metadata includes timing information (if
242  ``@timer.timeMethod`` is used) and any metadata set by the task. The
243  name of each item consists of the full task name with ``.`` replaced
244  by ``:``, followed by ``.`` and the name of the item, e.g.::
245 
246  topLevelTaskName:subtaskName:subsubtaskName.itemName
247 
248  using ``:`` in the full task name disambiguates the rare situation
249  that a task has a subtask and a metadata item with the same name.
250  """
251  fullMetadata = dafBase.PropertySet()
252  for fullName, task in self.getTaskDict().items():
253  fullMetadata.set(fullName.replace(".", ":"), task.metadata)
254  return fullMetadata
255 
std::vector< SchemaItem< Flag > > * items
Class for storing generic metadata.
Definition: PropertySet.h:67

◆ getFullName()

def lsst.pipe.base.task.Task.getFullName (   self)
inherited
Get the task name as a hierarchical name including parent task
names.

Returns
-------
fullName : `str`
    The full name consists of the name of the parent task and each
    subtask separated by periods. For example:

    - The full name of top-level task "top" is simply "top".
    - The full name of subtask "sub" of top-level task "top" is
      "top.sub".
    - The full name of subtask "sub2" of subtask "sub" of top-level
      task "top" is "top.sub.sub2".

Definition at line 256 of file task.py.

256  def getFullName(self):
257  """Get the task name as a hierarchical name including parent task
258  names.
259 
260  Returns
261  -------
262  fullName : `str`
263  The full name consists of the name of the parent task and each
264  subtask separated by periods. For example:
265 
266  - The full name of top-level task "top" is simply "top".
267  - The full name of subtask "sub" of top-level task "top" is
268  "top.sub".
269  - The full name of subtask "sub2" of subtask "sub" of top-level
270  task "top" is "top.sub.sub2".
271  """
272  return self._fullName
273 

◆ getName()

def lsst.pipe.base.task.Task.getName (   self)
inherited
Get the name of the task.

Returns
-------
taskName : `str`
    Name of the task.

See also
--------
getFullName

Definition at line 274 of file task.py.

274  def getName(self):
275  """Get the name of the task.
276 
277  Returns
278  -------
279  taskName : `str`
280  Name of the task.
281 
282  See also
283  --------
284  getFullName
285  """
286  return self._name
287 
std::string const & getName() const noexcept
Return a filter's name.
Definition: Filter.h:78

◆ getSchemaCatalogs()

def lsst.pipe.base.task.Task.getSchemaCatalogs (   self)
inherited
Get the schemas generated by this task.

Returns
-------
schemaCatalogs : `dict`
    Keys are butler dataset type, values are an empty catalog (an
    instance of the appropriate `lsst.afw.table` Catalog type) for
    this task.

Notes
-----

.. warning::

   Subclasses that use schemas must override this method. The default
   implementation returns an empty dict.

This method may be called at any time after the Task is constructed,
which means that all task schemas should be computed at construction
time, *not* when data is actually processed. This reflects the
philosophy that the schema should not depend on the data.

Returning catalogs rather than just schemas allows us to save e.g.
slots for SourceCatalog as well.

See also
--------
Task.getAllSchemaCatalogs

Definition at line 172 of file task.py.

172  def getSchemaCatalogs(self):
173  """Get the schemas generated by this task.
174 
175  Returns
176  -------
177  schemaCatalogs : `dict`
178  Keys are butler dataset type, values are an empty catalog (an
179  instance of the appropriate `lsst.afw.table` Catalog type) for
180  this task.
181 
182  Notes
183  -----
184 
185  .. warning::
186 
187  Subclasses that use schemas must override this method. The default
188  implementation returns an empty dict.
189 
190  This method may be called at any time after the Task is constructed,
191  which means that all task schemas should be computed at construction
192  time, *not* when data is actually processed. This reflects the
193  philosophy that the schema should not depend on the data.
194 
195  Returning catalogs rather than just schemas allows us to save e.g.
196  slots for SourceCatalog as well.
197 
198  See also
199  --------
200  Task.getAllSchemaCatalogs
201  """
202  return {}
203 

◆ getTaskDict()

def lsst.pipe.base.task.Task.getTaskDict (   self)
inherited
Get a dictionary of all tasks as a shallow copy.

Returns
-------
taskDict : `dict`
    Dictionary containing full task name: task object for the top-level
    task and all subtasks, sub-subtasks, etc.

Definition at line 288 of file task.py.

288  def getTaskDict(self):
289  """Get a dictionary of all tasks as a shallow copy.
290 
291  Returns
292  -------
293  taskDict : `dict`
294  Dictionary containing full task name: task object for the top-level
295  task and all subtasks, sub-subtasks, etc.
296  """
297  return self._taskDict.copy()
298 
def getTaskDict(config, taskDict=None, baseName="")

◆ makeField()

def lsst.pipe.base.task.Task.makeField (   cls,
  doc 
)
inherited
Make a `lsst.pex.config.ConfigurableField` for this task.

Parameters
----------
doc : `str`
    Help text for the field.

Returns
-------
configurableField : `lsst.pex.config.ConfigurableField`
    A `~ConfigurableField` for this task.

Examples
--------
Provides a convenient way to specify this task is a subtask of another
task.

Here is an example of use:

.. code-block:: python

    class OtherTaskConfig(lsst.pex.config.Config):
        aSubtask = ATaskClass.makeField("brief description of task")

Definition at line 359 of file task.py.

359  def makeField(cls, doc):
360  """Make a `lsst.pex.config.ConfigurableField` for this task.
361 
362  Parameters
363  ----------
364  doc : `str`
365  Help text for the field.
366 
367  Returns
368  -------
369  configurableField : `lsst.pex.config.ConfigurableField`
370  A `~ConfigurableField` for this task.
371 
372  Examples
373  --------
374  Provides a convenient way to specify this task is a subtask of another
375  task.
376 
377  Here is an example of use:
378 
379  .. code-block:: python
380 
381  class OtherTaskConfig(lsst.pex.config.Config):
382  aSubtask = ATaskClass.makeField("brief description of task")
383  """
384  return ConfigurableField(doc=doc, target=cls)
385 

◆ makeSubtask()

def lsst.pipe.base.task.Task.makeSubtask (   self,
  name,
**  keyArgs 
)
inherited
Create a subtask as a new instance as the ``name`` attribute of this
task.

Parameters
----------
name : `str`
    Brief name of the subtask.
keyArgs
    Extra keyword arguments used to construct the task. The following
    arguments are automatically provided and cannot be overridden:

    - "config".
    - "parentTask".

Notes
-----
The subtask must be defined by ``Task.config.name``, an instance of
`~lsst.pex.config.ConfigurableField` or
`~lsst.pex.config.RegistryField`.

Definition at line 299 of file task.py.

299  def makeSubtask(self, name, **keyArgs):
300  """Create a subtask as a new instance as the ``name`` attribute of this
301  task.
302 
303  Parameters
304  ----------
305  name : `str`
306  Brief name of the subtask.
307  keyArgs
308  Extra keyword arguments used to construct the task. The following
309  arguments are automatically provided and cannot be overridden:
310 
311  - "config".
312  - "parentTask".
313 
314  Notes
315  -----
316  The subtask must be defined by ``Task.config.name``, an instance of
317  `~lsst.pex.config.ConfigurableField` or
318  `~lsst.pex.config.RegistryField`.
319  """
320  taskField = getattr(self.config, name, None)
321  if taskField is None:
322  raise KeyError(f"{self.getFullName()}'s config does not have field {name!r}")
323  subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
324  setattr(self, name, subtask)
325 

◆ run()

def lsst.obs.base.defineVisits.DefineVisitsTask.run (   self,
Iterable[DataId]  dataIds,
*Optional[Pool]   pool = None,
int   processes = 1,
Optional[str]   collections = None 
)
Add visit definitions to the registry for the given exposures.

Parameters
----------
dataIds : `Iterable` [ `dict` or `DataCoordinate` ]
    Exposure-level data IDs.  These must all correspond to the same
    instrument, and are expected to be on-sky science exposures.
pool : `multiprocessing.Pool`, optional
    If not `None`, a process pool with which to parallelize some
    operations.
processes : `int`, optional
    The number of processes to use.  Ignored if ``pool`` is not `None`.
collections : Any, optional
    Collections to be searched for raws and camera geometry, overriding
    ``self.butler.collections``.
    Can be any of the types supported by the ``collections`` argument
    to butler construction.

Raises
------
lsst.daf.butler.registry.ConflictingDefinitionError
    Raised if a visit ID conflict is detected and the existing visit
    differs from the new one.

Definition at line 469 of file defineVisits.py.

472  collections: Optional[str] = None):
473  """Add visit definitions to the registry for the given exposures.
474 
475  Parameters
476  ----------
477  dataIds : `Iterable` [ `dict` or `DataCoordinate` ]
478  Exposure-level data IDs. These must all correspond to the same
479  instrument, and are expected to be on-sky science exposures.
480  pool : `multiprocessing.Pool`, optional
481  If not `None`, a process pool with which to parallelize some
482  operations.
483  processes : `int`, optional
484  The number of processes to use. Ignored if ``pool`` is not `None`.
485  collections : Any, optional
486  Collections to be searched for raws and camera geometry, overriding
487  ``self.butler.collections``.
488  Can be any of the types supported by the ``collections`` argument
489  to butler construction.
490 
491  Raises
492  ------
493  lsst.daf.butler.registry.ConflictingDefinitionError
494  Raised if a visit ID conflict is detected and the existing visit
495  differs from the new one.
496  """
497  # Set up multiprocessing, if desired.
498  if pool is None and processes > 1:
499  pool = Pool(processes)
500  mapFunc = map if pool is None else pool.imap_unordered
501  # Normalize, expand, and deduplicate data IDs.
502  self.log.info("Preprocessing data IDs.")
503  dataIds = set(mapFunc(self._expandExposureId, dataIds))
504  if not dataIds:
505  raise RuntimeError("No exposures given.")
506  # Extract exposure DimensionRecords, check that there's only one
507  # instrument in play, and check for non-science exposures.
508  exposures = []
509  instruments = set()
510  for dataId in dataIds:
511  record = dataId.records["exposure"]
512  if record.observation_type != "science":
513  if self.config.ignoreNonScienceExposures:
514  continue
515  else:
516  raise RuntimeError(f"Input exposure {dataId} has observation_type "
517  f"{record.observation_type}, not 'science'.")
518  instruments.add(dataId["instrument"])
519  exposures.append(record)
520  if not exposures:
521  self.log.info("No science exposures found after filtering.")
522  return
523  if len(instruments) > 1:
524  raise RuntimeError(
525  f"All data IDs passed to DefineVisitsTask.run must be "
526  f"from the same instrument; got {instruments}."
527  )
528  instrument, = instruments
529  # Ensure the visit_system our grouping algorithm uses is in the
530  # registry, if it wasn't already.
531  visitSystemId, visitSystemName = self.groupExposures.getVisitSystem()
532  self.log.info("Registering visit_system %d: %s.", visitSystemId, visitSystemName)
533  self.butler.registry.syncDimensionData(
534  "visit_system",
535  {"instrument": instrument, "id": visitSystemId, "name": visitSystemName}
536  )
537  # Group exposures into visits, delegating to subtask.
538  self.log.info("Grouping %d exposure(s) into visits.", len(exposures))
539  definitions = list(self.groupExposures.group(exposures))
540  # Compute regions and build DimensionRecords for each visit.
541  # This is the only parallel step, but it _should_ be the most expensive
542  # one (unless DB operations are slow).
543  self.log.info("Computing regions and other metadata for %d visit(s).", len(definitions))
544  allRecords = mapFunc(self._buildVisitRecordsSingle,
545  zip(definitions, itertools.repeat(collections)))
546  # Iterate over visits and insert dimension data, one transaction per
547  # visit. If a visit already exists, we skip all other inserts.
548  for visitRecords in allRecords:
549  with self.butler.registry.transaction():
550  if self.butler.registry.syncDimensionData("visit", visitRecords.visit):
551  self.butler.registry.insertDimensionData("visit_definition",
552  *visitRecords.visit_definition)
553  self.butler.registry.insertDimensionData("visit_detector_region",
554  *visitRecords.visit_detector_region)
555 
556 
daf::base::PropertyList * list
Definition: fits.cc:913
daf::base::PropertySet * set
Definition: fits.cc:912
table::Key< table::Array< int > > group
Definition: PsfexPsf.cc:359

◆ timer()

def lsst.pipe.base.task.Task.timer (   self,
  name,
  logLevel = Log.DEBUG 
)
inherited
Context manager to log performance data for an arbitrary block of
code.

Parameters
----------
name : `str`
    Name of code being timed; data will be logged using item name:
    ``Start`` and ``End``.
logLevel
    A `lsst.log` level constant.

Examples
--------
Creating a timer context:

.. code-block:: python

    with self.timer("someCodeToTime"):
        pass  # code to time

See also
--------
timer.logInfo

Definition at line 327 of file task.py.

327  def timer(self, name, logLevel=Log.DEBUG):
328  """Context manager to log performance data for an arbitrary block of
329  code.
330 
331  Parameters
332  ----------
333  name : `str`
334  Name of code being timed; data will be logged using item name:
335  ``Start`` and ``End``.
336  logLevel
337  A `lsst.log` level constant.
338 
339  Examples
340  --------
341  Creating a timer context:
342 
343  .. code-block:: python
344 
345  with self.timer("someCodeToTime"):
346  pass # code to time
347 
348  See also
349  --------
350  timer.logInfo
351  """
352  logInfo(obj=self, prefix=name + "Start", logLevel=logLevel)
353  try:
354  yield
355  finally:
356  logInfo(obj=self, prefix=name + "End", logLevel=logLevel)
357 
def logInfo(obj, prefix, logLevel=Log.DEBUG)
Definition: timer.py:63

Member Data Documentation

◆ butler

lsst.obs.base.defineVisits.DefineVisitsTask.butler

Definition at line 326 of file defineVisits.py.

◆ config

lsst.pipe.base.task.Task.config
inherited

Definition at line 162 of file task.py.

◆ ConfigClass

lsst.obs.base.defineVisits.DefineVisitsTask.ConfigClass = DefineVisitsConfig
static

Definition at line 335 of file defineVisits.py.

◆ log

lsst.pipe.base.task.Task.log
inherited

Definition at line 161 of file task.py.

◆ metadata

lsst.pipe.base.task.Task.metadata
inherited

Definition at line 134 of file task.py.

◆ universe

lsst.obs.base.defineVisits.DefineVisitsTask.universe

Definition at line 327 of file defineVisits.py.


The documentation for this class was generated from the following file: