lsst.pipe.base.graph._loadHelpers.FileLoadHelper Class Reference

Inheritance diagram for lsst.pipe.base.graph._loadHelpers.FileLoadHelper:

Public Member Functions
def	close (self)
QuantumGraph	load (self, Optional[Iterable[int]] nodes=None, Optional[str] graphID=None)

Public Attributes
	fileHandle
	uriObject
	headerSize
	save_version
	taskDefMap
	map
	buffer

Detailed Description

Definition at line 299 of file _loadHelpers.py.

Member Function Documentation

close()

def lsst.pipe.base.graph._loadHelpers.FileLoadHelper.close

(

self

)

Cleans up an instance if needed. Base class does nothing

Reimplemented from lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.

Definition at line 307 of file _loadHelpers.py.

    def close(self):
        if hasattr(self, 'fileHandle'):
            self.fileHandle.close()
 
 
@register_helper(io.IOBase)  # type: ignore

load()

QuantumGraph lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.load (   self, Optional[Iterable[int]]   nodes = None, Optional[str]   graphID = None  )

inherited

Loads in the specified nodes from the graph

Load in the `QuantumGraph` containing only the nodes specified in the
``nodes`` parameter from the graph specified at object creation. If
``nodes`` is None (the default) the whole graph is loaded.

Parameters
----------
nodes : `Iterable` of `int` or `None`
    The nodes to load from the graph, loads all if value is None
    (the default)
graphID : `str` or `None`
    If specified this ID is verified against the loaded graph prior to
    loading any Nodes. This defaults to None in which case no
    validation is done.

Returns
-------
graph : `QuantumGraph`
    The loaded `QuantumGraph` object

Raises
------
ValueError
    Raised if one or more of the nodes requested is not in the
    `QuantumGraph` or if graphID parameter does not match the graph
    being loaded.

Definition at line 163 of file _loadHelpers.py.

    def load(self, nodes: Optional[Iterable[int]] = None, graphID: Optional[str] = None) -> QuantumGraph:
        """Loads in the specified nodes from the graph
 
        Load in the `QuantumGraph` containing only the nodes specified in the
        ``nodes`` parameter from the graph specified at object creation. If
        ``nodes`` is None (the default) the whole graph is loaded.
 
        Parameters
        ----------
        nodes : `Iterable` of `int` or `None`
            The nodes to load from the graph, loads all if value is None
            (the default)
        graphID : `str` or `None`
            If specified this ID is verified against the loaded graph prior to
            loading any Nodes. This defaults to None in which case no
            validation is done.
 
        Returns
        -------
        graph : `QuantumGraph`
            The loaded `QuantumGraph` object
 
        Raises
        ------
        ValueError
            Raised if one or more of the nodes requested is not in the
            `QuantumGraph` or if graphID parameter does not match the graph
            being loaded.
        """
        # need to import here to avoid cyclic imports
        from . import QuantumGraph
        if graphID is not None and self._buildId != graphID:
            raise ValueError('graphID does not match that of the graph being loaded')
        # Read in specified nodes, or all the nodes
        if nodes is None:
            nodes = list(self.map.keys())
            # if all nodes are to be read, force the reader from the base class
            # that will read all they bytes in one go
            _readBytes = functools.partial(DefaultLoadHelper._readBytes, self)
        else:
            # only some bytes are being read using the reader specialized for
            # this class
            # create a set to ensure nodes are only loaded once
            nodes = set(nodes)
            # verify that all nodes requested are in the graph
            remainder = nodes - self.map.keys()
            if remainder:
                raise ValueError("Nodes {remainder} were requested, but could not be found in the input "
                                 "graph")
            _readBytes = self._readBytes
        # create a container for loaded data
        quanta: DefaultDict[TaskDef, Set[Quantum]] = defaultdict(set)
        quantumToNodeId: Dict[Quantum, NodeId] = {}
        loadedTaskDef = {}
        # loop over the nodes specified above
        for node in nodes:
            # Get the bytes to read from the map
            start, stop = self.map[node]
            start += self.headerSize
            stop += self.headerSize
 
            # read the specified bytes, will be overloaded by subclasses
            # bytes are compressed, so decompress them
            dump = lzma.decompress(_readBytes(start, stop))
 
            # reconstruct node
            qNode = pickle.loads(dump)
 
            # read the saved node, name. If it has been loaded, attach it, if
            # not read in the taskDef first, and then load it
            nodeTask = qNode.taskDef
            if nodeTask not in loadedTaskDef:
                # Get the byte ranges corresponding to this taskDef
                start, stop = self.taskDefMap[nodeTask]
                start += self.headerSize
                stop += self.headerSize
 
                # load the taskDef, this method call will be overloaded by
                # subclasses.
                # bytes are compressed, so decompress them
                taskDef = pickle.loads(lzma.decompress(_readBytes(start, stop)))
                loadedTaskDef[nodeTask] = taskDef
            # Explicitly overload the "frozen-ness" of nodes to attach the
            # taskDef back into the un-persisted node
            object.__setattr__(qNode, 'taskDef', loadedTaskDef[nodeTask])
            quanta[qNode.taskDef].add(qNode.quantum)
 
            # record the node for later processing
            quantumToNodeId[qNode.quantum] = qNode.nodeId
 
        # construct an empty new QuantumGraph object, and run the associated
        # creation method with the un-persisted data
        qGraph = object.__new__(QuantumGraph)
        qGraph._buildGraphs(quanta, _quantumToNodeId=quantumToNodeId, _buildId=self._buildId)
        return qGraph
 

Member Data Documentation

buffer

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.buffer

inherited

Definition at line 269 of file _loadHelpers.py.

fileHandle

lsst.pipe.base.graph._loadHelpers.FileLoadHelper.fileHandle

Definition at line 303 of file _loadHelpers.py.

headerSize

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.headerSize

inherited

Definition at line 113 of file _loadHelpers.py.

map

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.map

inherited

Definition at line 161 of file _loadHelpers.py.

save_version

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.save_version

inherited

Definition at line 142 of file _loadHelpers.py.

taskDefMap

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.taskDefMap

inherited

Definition at line 154 of file _loadHelpers.py.

uriObject

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.uriObject

inherited

Definition at line 108 of file _loadHelpers.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-0.4.3/Linux64/pipe_base/22.0.1+94e66cc9ed/python/lsst/pipe/base/graph/_loadHelpers.py