lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper Class Reference

Inheritance diagram for lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper:

Public Member Functions
def	__init__ (self, Union[ButlerURI, io.IO[bytes]] uriObject)
QuantumGraph	load (self, Optional[Iterable[int]] nodes=None, Optional[str] graphID=None)
def	close (self)

Public Attributes
	uriObject
	headerSize
	save_version
	taskDefMap
	map
	buffer

Detailed Description

Default load helper for `QuantumGraph` save files

This class, and its subclasses, are used to unpack a quantum graph save
file. This file is a binary representation of the graph in a format that
allows individual nodes to be loaded without needing to load the entire
file.

This default implementation has the interface to load select nodes
from disk, but actually always loads the entire save file and simply
returns what nodes (or all) are requested. This is intended to serve for
all cases where there is a read method on the input parameter, but it is
unknown how to read select bytes of the stream. It is the responsibility of
sub classes to implement the method responsible for loading individual
bytes from the stream.

Parameters
----------
uriObject : `~lsst.daf.butler.ButlerURI` or `io.IO` of bytes
    This is the object that will be used to retrieve the raw bytes of the
    save.

Raises
------
ValueError
    Raised if the specified file contains the wrong file signature and is
    not a `QuantumGraph` save

Definition at line 79 of file _loadHelpers.py.

Constructor & Destructor Documentation

__init__()

def lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.__init__	(		self,
		Union[ButlerURI, io.IO[bytes]]	uriObject
	)

Definition at line 107 of file _loadHelpers.py.

    def __init__(self, uriObject: Union[ButlerURI, io.IO[bytes]]):
        self.uriObject = uriObject
 
        preambleSize, taskDefSize, nodeSize = self._readSizes()
 
        # Recode the total header size
        self.headerSize = preambleSize + taskDefSize + nodeSize
 
        self._readByteMappings(preambleSize, self.headerSize, taskDefSize)
 

Member Function Documentation

close()

def lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.close

(

self

)

Cleans up an instance if needed. Base class does nothing

Reimplemented in lsst.pipe.base.graph._loadHelpers.FileLoadHelper.

Definition at line 272 of file _loadHelpers.py.

    def close(self):
        """Cleans up an instance if needed. Base class does nothing
        """
        pass
 
 
@register_helper(ButlerS3URI)

load()

QuantumGraph lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.load	(		self,
		Optional[Iterable[int]]	nodes = None,
		Optional[str]	graphID = None
	)

Loads in the specified nodes from the graph

Load in the `QuantumGraph` containing only the nodes specified in the
``nodes`` parameter from the graph specified at object creation. If
``nodes`` is None (the default) the whole graph is loaded.

Parameters
----------
nodes : `Iterable` of `int` or `None`
    The nodes to load from the graph, loads all if value is None
    (the default)
graphID : `str` or `None`
    If specified this ID is verified against the loaded graph prior to
    loading any Nodes. This defaults to None in which case no
    validation is done.

Returns
-------
graph : `QuantumGraph`
    The loaded `QuantumGraph` object

Raises
------
ValueError
    Raised if one or more of the nodes requested is not in the
    `QuantumGraph` or if graphID parameter does not match the graph
    being loaded.

Definition at line 163 of file _loadHelpers.py.

    def load(self, nodes: Optional[Iterable[int]] = None, graphID: Optional[str] = None) -> QuantumGraph:
        """Loads in the specified nodes from the graph
 
        Load in the `QuantumGraph` containing only the nodes specified in the
        ``nodes`` parameter from the graph specified at object creation. If
        ``nodes`` is None (the default) the whole graph is loaded.
 
        Parameters
        ----------
        nodes : `Iterable` of `int` or `None`
            The nodes to load from the graph, loads all if value is None
            (the default)
        graphID : `str` or `None`
            If specified this ID is verified against the loaded graph prior to
            loading any Nodes. This defaults to None in which case no
            validation is done.
 
        Returns
        -------
        graph : `QuantumGraph`
            The loaded `QuantumGraph` object
 
        Raises
        ------
        ValueError
            Raised if one or more of the nodes requested is not in the
            `QuantumGraph` or if graphID parameter does not match the graph
            being loaded.
        """
        # need to import here to avoid cyclic imports
        from . import QuantumGraph
        if graphID is not None and self._buildId != graphID:
            raise ValueError('graphID does not match that of the graph being loaded')
        # Read in specified nodes, or all the nodes
        if nodes is None:
            nodes = list(self.map.keys())
            # if all nodes are to be read, force the reader from the base class
            # that will read all they bytes in one go
            _readBytes = functools.partial(DefaultLoadHelper._readBytes, self)
        else:
            # only some bytes are being read using the reader specialized for
            # this class
            # create a set to ensure nodes are only loaded once
            nodes = set(nodes)
            # verify that all nodes requested are in the graph
            remainder = nodes - self.map.keys()
            if remainder:
                raise ValueError("Nodes {remainder} were requested, but could not be found in the input "
                                 "graph")
            _readBytes = self._readBytes
        # create a container for loaded data
        quanta: DefaultDict[TaskDef, Set[Quantum]] = defaultdict(set)
        quantumToNodeId: Dict[Quantum, NodeId] = {}
        loadedTaskDef = {}
        # loop over the nodes specified above
        for node in nodes:
            # Get the bytes to read from the map
            start, stop = self.map[node]
            start += self.headerSize
            stop += self.headerSize
 
            # read the specified bytes, will be overloaded by subclasses
            # bytes are compressed, so decompress them
            dump = lzma.decompress(_readBytes(start, stop))
 
            # reconstruct node
            qNode = pickle.loads(dump)
 
            # read the saved node, name. If it has been loaded, attach it, if
            # not read in the taskDef first, and then load it
            nodeTask = qNode.taskDef
            if nodeTask not in loadedTaskDef:
                # Get the byte ranges corresponding to this taskDef
                start, stop = self.taskDefMap[nodeTask]
                start += self.headerSize
                stop += self.headerSize
 
                # load the taskDef, this method call will be overloaded by
                # subclasses.
                # bytes are compressed, so decompress them
                taskDef = pickle.loads(lzma.decompress(_readBytes(start, stop)))
                loadedTaskDef[nodeTask] = taskDef
            # Explicitly overload the "frozen-ness" of nodes to attach the
            # taskDef back into the un-persisted node
            object.__setattr__(qNode, 'taskDef', loadedTaskDef[nodeTask])
            quanta[qNode.taskDef].add(qNode.quantum)
 
            # record the node for later processing
            quantumToNodeId[qNode.quantum] = qNode.nodeId
 
        # construct an empty new QuantumGraph object, and run the associated
        # creation method with the un-persisted data
        qGraph = object.__new__(QuantumGraph)
        qGraph._buildGraphs(quanta, _quantumToNodeId=quantumToNodeId, _buildId=self._buildId)
        return qGraph
 

Member Data Documentation

buffer

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.buffer

Definition at line 269 of file _loadHelpers.py.

headerSize

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.headerSize

Definition at line 113 of file _loadHelpers.py.

map

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.map

Definition at line 161 of file _loadHelpers.py.

save_version

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.save_version

Definition at line 142 of file _loadHelpers.py.

taskDefMap

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.taskDefMap

Definition at line 154 of file _loadHelpers.py.

uriObject

lsst.pipe.base.graph._loadHelpers.DefaultLoadHelper.uriObject

Definition at line 108 of file _loadHelpers.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-0.4.3/Linux64/pipe_base/22.0.1+94e66cc9ed/python/lsst/pipe/base/graph/_loadHelpers.py