lsst.obs.base.gen3.ingest.RawIngestTask Class Reference

Inheritance diagram for lsst.obs.base.gen3.ingest.RawIngestTask:

Public Member Functions
def	getDatasetType (self)
def	__init__ (self, config=None, butler, kwds)
def	run (self, files)
def	readHeaders (self, file)
def	buildRegion (self, headers)
def	ensureDimensions (self, file)
def	ingestFile (self, file, headers, dataId, run=None)
def	processFile (self, file)
def	extractDataId (self, file, headers, obsInfo)
def	getFormatter (self, file, headers, dataId)
def	emptyMetadata (self)
def	getSchemaCatalogs (self)
def	getAllSchemaCatalogs (self)
def	getFullMetadata (self)
def	getFullName (self)
def	getName (self)
def	getTaskDict (self)
def	makeSubtask (self, name, keyArgs)
def	timer (self, name, logLevel=Log.DEBUG)
def	makeField (cls, doc)
def	__reduce__ (self)

Public Attributes
	butler
	datasetType
	dimensions
	dimensionEntriesDone
	instrumentCache
	stashRun
	visitRegions
	metadata
	log
	config

Static Public Attributes
	ConfigClass = RawIngestConfig

Detailed Description

Driver Task for ingesting raw data into Gen3 Butler repositories.

This Task is intended to be runnable from the command-line, but it doesn't
meet the other requirements of CmdLineTask or PipelineTask, and wouldn't
gain much from being one.  It also wouldn't really be appropriate as a
subtask of a CmdLineTask or PipelineTask; it's a Task essentially just to
leverage the logging and configurability functionality that provides.

Each instance of `RawIngestTask` writes to the same Butler and maintains a
cache of Dimension entries that have already been added to or extracted
from its Registry.  Each invocation of `RawIngestTask.run` ingests a list
of files (possibly semi-atomically; see `RawIngestConfig.onError`).

RawIngestTask may be subclassed to specialize ingest for the actual
structure of raw data files produced by a particular instrument, but this
is usually unnecessary because the instrument-specific header-extraction
provided by the ``astro_metadata_translator`` is usually enough.

Parameters
----------
config : `RawIngestConfig`
    Configuration for whether/how to transfer files and how to handle
    conflicts and errors.
butler : `~lsst.daf.butler.Butler`
    Butler instance.  Ingested Datasets will be created as part of
    ``butler.run`` and associated with its Collection.

Other keyword arguments are forwarded to the Task base class constructor.

Definition at line 106 of file ingest.py.

Constructor & Destructor Documentation

__init__()

def lsst.obs.base.gen3.ingest.RawIngestTask.__init__	(		self,
			config = None,
			butler,
			kwds
	)

Definition at line 147 of file ingest.py.

    def __init__(self, config=None, *, butler, **kwds):
        super().__init__(config, **kwds)
        self.butler = butler
        self.datasetType = self.getDatasetType()
        self.dimensions = butler.registry.dimensions.extract(["instrument", "detector", "physical_filter",
                                                              "visit", "exposure"])
        # Dictionary of {Dimension: set(DataId)} indicating Dimension entries
        # we know are in the Registry.
        self.dimensionEntriesDone = {k: set() for k in self.dimensions}
        # Cache of instrument instances retrieved from Registry; needed to look
        # up formatters.
        self.instrumentCache = {}
        # (Possibly) create a Run object for the "stash": where we put datasets
        # that lose conflicts.  Note that this doesn't actually add this Run
        # to the Registry; we only do that on first use.
        self.stashRun = Run(self.config.stash) if self.config.stash is not None else None
        self.visitRegions = {}

Member Function Documentation

__reduce__()

def lsst.pipe.base.task.Task.__reduce__ (   self )

inherited

Pickler.

Definition at line 373 of file task.py.

    def __reduce__(self):
        """Pickler.
        """
        return self.__class__, (self.config, self._name, self._parentTask, None)

buildRegion()

def lsst.obs.base.gen3.ingest.RawIngestTask.buildRegion	(		self,
			headers
	)

Builds a region from information contained in a header

Parameters
----------
headers : `lsst.daf.base.PropertyList`
    Property list containing the information from the header of
    one file.

Returns
-------
region : `lsst.sphgeom.ConvexPolygon`

Raises
------
ValueError :
    If required header keys can not be found to construct region

Definition at line 242 of file ingest.py.

    def buildRegion(self, headers):
        """Builds a region from information contained in a header

        Parameters
        ----------
        headers : `lsst.daf.base.PropertyList`
            Property list containing the information from the header of
            one file.

        Returns
        -------
        region : `lsst.sphgeom.ConvexPolygon`

        Raises
        ------
        ValueError :
            If required header keys can not be found to construct region
        """
        # Default implementation is for headers to be a one element list
        header = headers[0]
        wcs = SkyWcs(header)
        bbox = Box2D(bboxFromMetadata(header))
        if self.config.padRegionAmount > 0:
            bbox.grow(self.config.padRegionAmount)
        corners = bbox.getCorners()
        sphCorners = [wcs.pixelToSky(point).getVector() for point in corners]
        return ConvexPolygon(sphCorners)

emptyMetadata()

def lsst.pipe.base.task.Task.emptyMetadata (   self )

inherited

Empty (clear) the metadata for this Task and all sub-Tasks.

Definition at line 153 of file task.py.

    def emptyMetadata(self):
        """Empty (clear) the metadata for this Task and all sub-Tasks.
        """
        for subtask in self._taskDict.values():
            subtask.metadata = dafBase.PropertyList()

ensureDimensions()

def lsst.obs.base.gen3.ingest.RawIngestTask.ensureDimensions	(		self,
			file
	)

Extract metadata from a raw file and add exposure and visit
Dimension entries.

Any needed instrument, detector, and physical_filter Dimension entries must
exist in the Registry before `run` is called.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file to be ingested.

Returns
-------
headers : `list` of `~lsst.daf.base.PropertyList`
    Result of calling `readHeaders`.
dataId : `DataId`
    Data ID dictionary, as returned by `extractDataId`.

Definition at line 270 of file ingest.py.

    def ensureDimensions(self, file):
        """Extract metadata from a raw file and add exposure and visit
        Dimension entries.

        Any needed instrument, detector, and physical_filter Dimension entries must
        exist in the Registry before `run` is called.

        Parameters
        ----------
        file : `str` or path-like object
            Absolute path to the file to be ingested.

        Returns
        -------
        headers : `list` of `~lsst.daf.base.PropertyList`
            Result of calling `readHeaders`.
        dataId : `DataId`
            Data ID dictionary, as returned by `extractDataId`.
        """
        headers = self.readHeaders(file)
        obsInfo = ObservationInfo(headers[0])

        # Extract a DataId that covers all of self.dimensions.
        fullDataId = self.extractDataId(file, headers, obsInfo=obsInfo)

        for dimension in self.dimensions:
            if fullDataId.get(dimension.name) is None:
                continue
            dimensionDataId = DataId(fullDataId, dimension=dimension)
            if dimensionDataId not in self.dimensionEntriesDone[dimension]:
                # Next look in the Registry
                dimensionEntryDict = self.butler.registry.findDimensionEntry(dimension, dimensionDataId)
                if dimensionEntryDict is None:
                    if dimension.name in ("visit", "exposure"):
                        # Add the entry into the Registry.
                        self.butler.registry.addDimensionEntry(dimension, dimensionDataId)
                    else:
                        raise LookupError(
                            f"Entry for {dimension.name} with ID {dimensionDataId} not found; must be "
                            f"present in Registry prior to ingest."
                        )
                # Record that we've handled this entry.
                self.dimensionEntriesDone[dimension].add(dimensionDataId)
        # Do this after the loop to ensure all the dimensions are added
        if self.config.doAddRegions and obsInfo.tracking_radec is not None:
            region = self.buildRegion(headers)
            try:
                self.butler.registry.setDimensionRegion(DataId(fullDataId,
                                                               dimensions=['visit', 'detector', 'instrument'],
                                                               region=region),
                                                        update=False)
                self.visitRegions.setdefault((fullDataId['instrument'], fullDataId['visit']),
                                             []).extend(region.getVertices())
            except IntegrityError:
                # This means that there were already regions for the dimensions in the database, and nothing
                # should be done.
                pass

        return headers, fullDataId

extractDataId()

def lsst.obs.base.gen3.ingest.RawIngestTask.extractDataId	(		self,
			file,
			headers,
			obsInfo
	)

Return the Data ID dictionary that should be used to label a file.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file being ingested (prior to any transfers).
headers : `list` of `~lsst.daf.base.PropertyList`
    All headers returned by `readHeaders()`.
obsInfo : `astro_metadata_translator.ObservationInfo`
    Observational metadata extracted from the headers.

Returns
-------
dataId : `DataId`
    A mapping whose key-value pairs uniquely identify raw datasets.
    Must have ``dataId.dimensions() <= self.dimensions``, with at least
    instrument, exposure, and detector present.

Definition at line 404 of file ingest.py.

    def extractDataId(self, file, headers, obsInfo):
        """Return the Data ID dictionary that should be used to label a file.

        Parameters
        ----------
        file : `str` or path-like object
            Absolute path to the file being ingested (prior to any transfers).
        headers : `list` of `~lsst.daf.base.PropertyList`
            All headers returned by `readHeaders()`.
        obsInfo : `astro_metadata_translator.ObservationInfo`
            Observational metadata extracted from the headers.

        Returns
        -------
        dataId : `DataId`
            A mapping whose key-value pairs uniquely identify raw datasets.
            Must have ``dataId.dimensions() <= self.dimensions``, with at least
            instrument, exposure, and detector present.
        """
        toRemove = set()
        if obsInfo.visit_id is None:
            toRemove.add("visit")
        if obsInfo.physical_filter is None:
            toRemove.add("physical_filter")
        if toRemove:
            dimensions = self.dimensions.toSet().difference(toRemove)
        else:
            dimensions = self.dimensions
        dataId = DataId(
            dimensions=dimensions,
            instrument=obsInfo.instrument,
            exposure=obsInfo.exposure_id,
            visit=obsInfo.visit_id,
            detector=obsInfo.detector_num,
            physical_filter=obsInfo.physical_filter,
        )
        updateExposureEntryFromObsInfo(dataId, obsInfo)
        if obsInfo.visit_id is not None:
            updateVisitEntryFromObsInfo(dataId, obsInfo)
        return dataId

getAllSchemaCatalogs()

def lsst.pipe.base.task.Task.getAllSchemaCatalogs (   self )

inherited

Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.

Returns
-------
schemacatalogs : `dict`
    Keys are butler dataset type, values are a empty catalog (an instance of the appropriate
    lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down
    through all subtasks.

Notes
-----
This method may be called on any task in the hierarchy; it will return the same answer, regardless.

The default implementation should always suffice. If your subtask uses schemas the override
`Task.getSchemaCatalogs`, not this method.

Definition at line 188 of file task.py.

    def getAllSchemaCatalogs(self):
        """Get schema catalogs for all tasks in the hierarchy, combining the results into a single dict.

        Returns
        -------
        schemacatalogs : `dict`
            Keys are butler dataset type, values are a empty catalog (an instance of the appropriate
            lsst.afw.table Catalog type) for all tasks in the hierarchy, from the top-level task down
            through all subtasks.

        Notes
        -----
        This method may be called on any task in the hierarchy; it will return the same answer, regardless.

        The default implementation should always suffice. If your subtask uses schemas the override
        `Task.getSchemaCatalogs`, not this method.
        """
        schemaDict = self.getSchemaCatalogs()
        for subtask in self._taskDict.values():
            schemaDict.update(subtask.getSchemaCatalogs())
        return schemaDict

getDatasetType()

def lsst.obs.base.gen3.ingest.RawIngestTask.getDatasetType

(

self

)

Return the DatasetType of the Datasets ingested by this Task.

Definition at line 141 of file ingest.py.

    def getDatasetType(self):
        """Return the DatasetType of the Datasets ingested by this Task.
        """
        return DatasetType("raw", ("instrument", "detector", "exposure"), "Exposure",
                           universe=self.butler.registry.dimensions)

getFormatter()

def lsst.obs.base.gen3.ingest.RawIngestTask.getFormatter	(		self,
			file,
			headers,
			dataId
	)

Return the Formatter that should be used to read this file after
ingestion.

The default implementation obtains the formatter from the Instrument
class for the given data ID.

Definition at line 445 of file ingest.py.

    def getFormatter(self, file, headers, dataId):
        """Return the Formatter that should be used to read this file after
        ingestion.

        The default implementation obtains the formatter from the Instrument
        class for the given data ID.
        """
        instrument = self.instrumentCache.get(dataId["instrument"])
        if instrument is None:
            instrument = Instrument.factories[dataId["instrument"]]()
            self.instrumentCache[dataId["instrument"]] = instrument
        return instrument.getRawFormatter(dataId)

getFullMetadata()

def lsst.pipe.base.task.Task.getFullMetadata (   self )

inherited

Get metadata for all tasks.

Returns
-------
metadata : `lsst.daf.base.PropertySet`
    The `~lsst.daf.base.PropertySet` keys are the full task name. Values are metadata
    for the top-level task and all subtasks, sub-subtasks, etc..

Notes
-----
The returned metadata includes timing information (if ``@timer.timeMethod`` is used)
and any metadata set by the task. The name of each item consists of the full task name
with ``.`` replaced by ``:``, followed by ``.`` and the name of the item, e.g.::

    topLevelTaskName:subtaskName:subsubtaskName.itemName

using ``:`` in the full task name disambiguates the rare situation that a task has a subtask
and a metadata item with the same name.

Definition at line 210 of file task.py.

    def getFullMetadata(self):
        """Get metadata for all tasks.

        Returns
        -------
        metadata : `lsst.daf.base.PropertySet`
            The `~lsst.daf.base.PropertySet` keys are the full task name. Values are metadata
            for the top-level task and all subtasks, sub-subtasks, etc..

        Notes
        -----
        The returned metadata includes timing information (if ``@timer.timeMethod`` is used)
        and any metadata set by the task. The name of each item consists of the full task name
        with ``.`` replaced by ``:``, followed by ``.`` and the name of the item, e.g.::

            topLevelTaskName:subtaskName:subsubtaskName.itemName

        using ``:`` in the full task name disambiguates the rare situation that a task has a subtask
        and a metadata item with the same name.
        """
        fullMetadata = dafBase.PropertySet()
        for fullName, task in self.getTaskDict().items():
            fullMetadata.set(fullName.replace(".", ":"), task.metadata)
        return fullMetadata

getFullName()

def lsst.pipe.base.task.Task.getFullName (   self )

inherited

Get the task name as a hierarchical name including parent task names.

Returns
-------
fullName : `str`
    The full name consists of the name of the parent task and each subtask separated by periods.
    For example:

    - The full name of top-level task "top" is simply "top".
    - The full name of subtask "sub" of top-level task "top" is "top.sub".
    - The full name of subtask "sub2" of subtask "sub" of top-level task "top" is "top.sub.sub2".

Definition at line 235 of file task.py.

    def getFullName(self):
        """Get the task name as a hierarchical name including parent task names.

        Returns
        -------
        fullName : `str`
            The full name consists of the name of the parent task and each subtask separated by periods.
            For example:

            - The full name of top-level task "top" is simply "top".
            - The full name of subtask "sub" of top-level task "top" is "top.sub".
            - The full name of subtask "sub2" of subtask "sub" of top-level task "top" is "top.sub.sub2".
        """
        return self._fullName

getName()

def lsst.pipe.base.task.Task.getName (   self )

inherited

Get the name of the task.

Returns
-------
taskName : `str`
    Name of the task.

See also
--------
getFullName

Definition at line 250 of file task.py.

    def getName(self):
        """Get the name of the task.

        Returns
        -------
        taskName : `str`
            Name of the task.

        See also
        --------
        getFullName
        """
        return self._name

getSchemaCatalogs()

def lsst.pipe.base.task.Task.getSchemaCatalogs (   self )

inherited

Get the schemas generated by this task.

Returns
-------
schemaCatalogs : `dict`
    Keys are butler dataset type, values are an empty catalog (an instance of the appropriate
    `lsst.afw.table` Catalog type) for this task.

Notes
-----

.. warning::

   Subclasses that use schemas must override this method. The default implemenation returns
   an empty dict.

This method may be called at any time after the Task is constructed, which means that all task
schemas should be computed at construction time, *not* when data is actually processed. This
reflects the philosophy that the schema should not depend on the data.

Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.

See also
--------
Task.getAllSchemaCatalogs

Definition at line 159 of file task.py.

    def getSchemaCatalogs(self):
        """Get the schemas generated by this task.

        Returns
        -------
        schemaCatalogs : `dict`
            Keys are butler dataset type, values are an empty catalog (an instance of the appropriate
            `lsst.afw.table` Catalog type) for this task.

        Notes
        -----

        .. warning::

           Subclasses that use schemas must override this method. The default implemenation returns
           an empty dict.

        This method may be called at any time after the Task is constructed, which means that all task
        schemas should be computed at construction time, *not* when data is actually processed. This
        reflects the philosophy that the schema should not depend on the data.

        Returning catalogs rather than just schemas allows us to save e.g. slots for SourceCatalog as well.

        See also
        --------
        Task.getAllSchemaCatalogs
        """
        return {}

getTaskDict()

def lsst.pipe.base.task.Task.getTaskDict (   self )

inherited

Get a dictionary of all tasks as a shallow copy.

Returns
-------
taskDict : `dict`
    Dictionary containing full task name: task object for the top-level task and all subtasks,
    sub-subtasks, etc..

Definition at line 264 of file task.py.

    def getTaskDict(self):
        """Get a dictionary of all tasks as a shallow copy.

        Returns
        -------
        taskDict : `dict`
            Dictionary containing full task name: task object for the top-level task and all subtasks,
            sub-subtasks, etc..
        """
        return self._taskDict.copy()

ingestFile()

def lsst.obs.base.gen3.ingest.RawIngestTask.ingestFile	(		self,
			file,
			headers,
			dataId,
			run = None
	)

Ingest a single raw file into the repository.

All necessary Dimension entres must already be present.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file to be ingested.
headers : `list` of `~lsst.daf.base.PropertyList`
    Result of calling `readHeaders`.
dataId : `dict`
    Data ID dictionary, as returned by `extractDataId`.
run : `~lsst.daf.butler.Run`, optional
    Run to add the Dataset to; defaults to ``self.butler.run``.

Returns
-------
ref : `DatasetRef`
    Reference to the ingested dataset.

Raises
------
ConflictingDefinitionError
    Raised if the dataset already exists in the registry.

Definition at line 330 of file ingest.py.

    def ingestFile(self, file, headers, dataId, run=None):
        """Ingest a single raw file into the repository.

        All necessary Dimension entres must already be present.

        Parameters
        ----------
        file : `str` or path-like object
            Absolute path to the file to be ingested.
        headers : `list` of `~lsst.daf.base.PropertyList`
            Result of calling `readHeaders`.
        dataId : `dict`
            Data ID dictionary, as returned by `extractDataId`.
        run : `~lsst.daf.butler.Run`, optional
            Run to add the Dataset to; defaults to ``self.butler.run``.

        Returns
        -------
        ref : `DatasetRef`
            Reference to the ingested dataset.

        Raises
        ------
        ConflictingDefinitionError
            Raised if the dataset already exists in the registry.
        """
        if run is not None and run != self.butler.run:
            butler = Butler(butler=self.butler, run=run)
        else:
            butler = self.butler
        try:
            return butler.ingest(file, self.datasetType, dataId, transfer=self.config.transfer,
                                 formatter=self.getFormatter(file, headers, dataId))
        except ConflictingDefinitionError as err:
            raise IngestConflictError("Ingest conflict on {} {}".format(file, dataId)) from err

makeField()

def lsst.pipe.base.task.Task.makeField (   cls,   doc  )

inherited

Make a `lsst.pex.config.ConfigurableField` for this task.

Parameters
----------
doc : `str`
    Help text for the field.

Returns
-------
configurableField : `lsst.pex.config.ConfigurableField`
    A `~ConfigurableField` for this task.

Examples
--------
Provides a convenient way to specify this task is a subtask of another task.

Here is an example of use::

    class OtherTaskConfig(lsst.pex.config.Config)
aSubtask = ATaskClass.makeField("a brief description of what this task does")

Definition at line 329 of file task.py.

    def makeField(cls, doc):
        """Make a `lsst.pex.config.ConfigurableField` for this task.

        Parameters
        ----------
        doc : `str`
            Help text for the field.

        Returns
        -------
        configurableField : `lsst.pex.config.ConfigurableField`
            A `~ConfigurableField` for this task.

        Examples
        --------
        Provides a convenient way to specify this task is a subtask of another task.

        Here is an example of use::

            class OtherTaskConfig(lsst.pex.config.Config)
                aSubtask = ATaskClass.makeField("a brief description of what this task does")
        """
        return ConfigurableField(doc=doc, target=cls)

makeSubtask()

def lsst.pipe.base.task.Task.makeSubtask (   self,   name,   keyArgs  )

inherited

Create a subtask as a new instance as the ``name`` attribute of this task.

Parameters
----------
name : `str`
    Brief name of the subtask.
keyArgs
    Extra keyword arguments used to construct the task. The following arguments are automatically
    provided and cannot be overridden:

    - "config".
    - "parentTask".

Notes
-----
The subtask must be defined by ``Task.config.name``, an instance of pex_config ConfigurableField
or RegistryField.

Definition at line 275 of file task.py.

    def makeSubtask(self, name, **keyArgs):
        """Create a subtask as a new instance as the ``name`` attribute of this task.

        Parameters
        ----------
        name : `str`
            Brief name of the subtask.
        keyArgs
            Extra keyword arguments used to construct the task. The following arguments are automatically
            provided and cannot be overridden:

            - "config".
            - "parentTask".

        Notes
        -----
        The subtask must be defined by ``Task.config.name``, an instance of pex_config ConfigurableField
        or RegistryField.
        """
        taskField = getattr(self.config, name, None)
        if taskField is None:
            raise KeyError("%s's config does not have field %r" % (self.getFullName(), name))
        subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
        setattr(self, name, subtask)

processFile()

def lsst.obs.base.gen3.ingest.RawIngestTask.processFile	(		self,
			file
	)

Ingest a single raw data file after extacting metadata.

This creates any new exposure or visit Dimension entries needed to
identify the ingest file, creates a new Dataset entry in the
Registry and finally ingests the file itself into the Datastore.
Any needed instrument, detector, and physical_filter Dimension entries must
exist in the Registry before `run` is called.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file to be ingested.

Definition at line 366 of file ingest.py.

    def processFile(self, file):
        """Ingest a single raw data file after extacting metadata.

        This creates any new exposure or visit Dimension entries needed to
        identify the ingest file, creates a new Dataset entry in the
        Registry and finally ingests the file itself into the Datastore.
        Any needed instrument, detector, and physical_filter Dimension entries must
        exist in the Registry before `run` is called.

        Parameters
        ----------
        file : `str` or path-like object
            Absolute path to the file to be ingested.
        """
        try:
            headers, dataId = self.ensureDimensions(file)
        except Exception as err:
            raise RuntimeError(f"Unexpected error adding dimensions for {file}.") from err
        # We want ingesting a single file to be atomic even if we are
        # not trying to ingest the list of files atomically.
        with self.butler.transaction():
            try:
                self.ingestFile(file, headers, dataId)
                return
            except IngestConflictError:
                if self.config.conflict == "fail":
                    raise
        if self.config.conflict == "ignore":
            if self.stashRun is not None:
                if self.stashRun.id is None:
                    self.butler.registry.ensureRun(self.stashRun)
                self.log.infof("Conflict on {} ({}); ingesting to stash '{}' instead.",
                               dataId, file, self.config.stash)
                with self.butler.transaction():
                    self.ingestFile(file, headers, dataId, run=self.stashRun)
            else:
                self.log.infof("Conflict on {} ({}); ignoring.", dataId, file)

readHeaders()

def lsst.obs.base.gen3.ingest.RawIngestTask.readHeaders	(		self,
			file
	)

Read and return any relevant headers from the given file.

The default implementation simply reads the header of the first
non-empty HDU, so it always returns a single-element list.

Parameters
----------
file : `str` or path-like object
    Absolute path to the file to be ingested.

Returns
-------
headers : `list` of `~lsst.daf.base.PropertyList`
    Single-element list containing the header of the first
    non-empty HDU.

Definition at line 219 of file ingest.py.

    def readHeaders(self, file):
        """Read and return any relevant headers from the given file.

        The default implementation simply reads the header of the first
        non-empty HDU, so it always returns a single-element list.

        Parameters
        ----------
        file : `str` or path-like object
            Absolute path to the file to be ingested.

        Returns
        -------
        headers : `list` of `~lsst.daf.base.PropertyList`
            Single-element list containing the header of the first
            non-empty HDU.
        """
        phdu = readMetadata(file, 0)
        headers = [merge_headers([phdu, readMetadata(file)], mode="overwrite")]
        for h in headers:
            fix_header(h)
        return headers

run()

def lsst.obs.base.gen3.ingest.RawIngestTask.run	(		self,
			files
	)

Ingest files into a Butler data repository.

This creates any new exposure or visit Dimension entries needed to
identify the ingested files, creates new Dataset entries in the
Registry and finally ingests the files themselves into the Datastore.
Any needed instrument, detector, and physical_filter Dimension entries
must exist in the Registry before `run` is called.

Parameters
----------
files : iterable over `str` or path-like objects
    Paths to the files to be ingested.  Will be made absolute
    if they are not already.

Definition at line 183 of file ingest.py.

    def run(self, files):
        """Ingest files into a Butler data repository.

        This creates any new exposure or visit Dimension entries needed to
        identify the ingested files, creates new Dataset entries in the
        Registry and finally ingests the files themselves into the Datastore.
        Any needed instrument, detector, and physical_filter Dimension entries
        must exist in the Registry before `run` is called.

        Parameters
        ----------
        files : iterable over `str` or path-like objects
            Paths to the files to be ingested.  Will be made absolute
            if they are not already.
        """
        self.butler.registry.registerDatasetType(self.getDatasetType())
        if self.config.onError == "rollback":
            with self.butler.transaction():
                for file in files:
                    self.processFile(os.path.abspath(file))
                if self.config.doAddRegions:
                    self._addVisitRegions()
        elif self.config.onError == "break":
            for file in files:
                self.processFile(os.path.abspath(file))
            if self.config.doAddRegions:
                self._addVisitRegions()
        elif self.config.onError == "continue":
            for file in files:
                try:
                    self.processFile(os.path.abspath(file))
                except Exception as err:
                    self.log.warnf("Error processing '{}': {}", file, err)
            if self.config.doAddRegions:
                self._addVisitRegions()

timer()

def lsst.pipe.base.task.Task.timer (   self,   name,   logLevel = Log.DEBUG  )

inherited

Context manager to log performance data for an arbitrary block of code.

Parameters
----------
name : `str`
    Name of code being timed; data will be logged using item name: ``Start`` and ``End``.
logLevel
    A `lsst.log` level constant.

Examples
--------
Creating a timer context::

    with self.timer("someCodeToTime"):
pass  # code to time

See also
--------
timer.logInfo

Definition at line 301 of file task.py.

    def timer(self, name, logLevel=Log.DEBUG):
        """Context manager to log performance data for an arbitrary block of code.

        Parameters
        ----------
        name : `str`
            Name of code being timed; data will be logged using item name: ``Start`` and ``End``.
        logLevel
            A `lsst.log` level constant.

        Examples
        --------
        Creating a timer context::

            with self.timer("someCodeToTime"):
                pass  # code to time

        See also
        --------
        timer.logInfo
        """
        logInfo(obj=self, prefix=name + "Start", logLevel=logLevel)
        try:
            yield
        finally:
            logInfo(obj=self, prefix=name + "End", logLevel=logLevel)

Member Data Documentation

butler

lsst.obs.base.gen3.ingest.RawIngestTask.butler

Definition at line 149 of file ingest.py.

config

lsst.pipe.base.task.Task.config

inherited

Definition at line 149 of file task.py.

ConfigClass

lsst.obs.base.gen3.ingest.RawIngestTask.ConfigClass = RawIngestConfig

static

Definition at line 137 of file ingest.py.

datasetType

lsst.obs.base.gen3.ingest.RawIngestTask.datasetType

Definition at line 150 of file ingest.py.

dimensionEntriesDone

lsst.obs.base.gen3.ingest.RawIngestTask.dimensionEntriesDone

Definition at line 155 of file ingest.py.

dimensions

lsst.obs.base.gen3.ingest.RawIngestTask.dimensions

Definition at line 151 of file ingest.py.

instrumentCache

lsst.obs.base.gen3.ingest.RawIngestTask.instrumentCache

Definition at line 158 of file ingest.py.

log

lsst.pipe.base.task.Task.log

inherited

Definition at line 148 of file task.py.

metadata

lsst.pipe.base.task.Task.metadata

inherited

Definition at line 121 of file task.py.

stashRun

lsst.obs.base.gen3.ingest.RawIngestTask.stashRun

Definition at line 162 of file ingest.py.

visitRegions

lsst.obs.base.gen3.ingest.RawIngestTask.visitRegions

Definition at line 163 of file ingest.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/Linux64/obs_base/18.1.0-18-gb5d19ff+1/python/lsst/obs/base/gen3/ingest.py

---

Generated on Mon Sep 23 2019 07:03:02 for LSSTApplications by   1.8.13