lsst.pipe.base.graph.QuantumGraph Class Reference

Inheritance diagram for lsst.pipe.base.graph.QuantumGraph:

Public Member Functions
def	__init__ (self, iterable=None)
def	load (cls, file, universe)
def	save (self, file)
def	quanta (self)
def	quantaAsQgraph (self)
def	countQuanta (self)
def	traverse (self)

Public Attributes
	initInputs
	initIntermediates
	initOutputs

Detailed Description

QuantumGraph is a sequence of `QuantumGraphTaskNodes` objects.

Typically the order of the tasks in the list will be the same as the
order of tasks in a pipeline (obviously depends on the code which
constructs graph).

Parameters
----------
iterable : iterable of `QuantumGraphTaskNodes`, optional
    Initial sequence of per-task nodes.

Definition at line 120 of file graph.py.

Constructor & Destructor Documentation

__init__()

def lsst.pipe.base.graph.QuantumGraph.__init__	(		self,
			iterable = None
	)

Definition at line 132 of file graph.py.

    def __init__(self, iterable=None):
        list.__init__(self, iterable or [])
        self.initInputs = NamedKeyDict()
        self.initIntermediates = NamedKeyDict()
        self.initOutputs = NamedKeyDict()
 

Member Function Documentation

countQuanta()

def lsst.pipe.base.graph.QuantumGraph.countQuanta

(

self

)

Return total count of quanta in a graph.

Returns
-------
count : `int`
    Number of quanta in a graph.

Definition at line 239 of file graph.py.

    def countQuanta(self):
        """Return total count of quanta in a graph.
 
        Returns
        -------
        count : `int`
            Number of quanta in a graph.
        """
        return sum(len(taskNodes.quanta) for taskNodes in self)
 

load()

def lsst.pipe.base.graph.QuantumGraph.load	(		cls,
			file,
			universe
	)

Read QuantumGraph from a file that was made by `save`.

Parameters
----------
file : `io.BufferedIOBase`
    File with pickle data open in binary mode.
universe: `~lsst.daf.butler.DimensionUniverse`
    DimensionUniverse instance, not used by the method itself but
    needed to ensure that registry data structures are initialized.

Returns
-------
graph : `QuantumGraph`
    Resulting QuantumGraph instance.

Raises
------
TypeError
    Raised if pickle contains instance of a type other than
    QuantumGraph.

Notes
-----
Reading Quanta from pickle requires existence of singleton
DimensionUniverse which is usually instantiated during Registry
initializaion. To make sure that DimensionUniverse exists this method
accepts dummy DimensionUniverse argument.

Definition at line 160 of file graph.py.

    def load(cls, file, universe):
        """Read QuantumGraph from a file that was made by `save`.
 
        Parameters
        ----------
        file : `io.BufferedIOBase`
            File with pickle data open in binary mode.
        universe: `~lsst.daf.butler.DimensionUniverse`
            DimensionUniverse instance, not used by the method itself but
            needed to ensure that registry data structures are initialized.
 
        Returns
        -------
        graph : `QuantumGraph`
            Resulting QuantumGraph instance.
 
        Raises
        ------
        TypeError
            Raised if pickle contains instance of a type other than
            QuantumGraph.
 
        Notes
        -----
        Reading Quanta from pickle requires existence of singleton
        DimensionUniverse which is usually instantiated during Registry
        initializaion. To make sure that DimensionUniverse exists this method
        accepts dummy DimensionUniverse argument.
        """
        qgraph = pickle.load(file)
        if not isinstance(qgraph, QuantumGraph):
            raise TypeError(f"QuantumGraph pickle file has contains unexpected object type: {type(qgraph)}")
        return qgraph
 

quanta()

def lsst.pipe.base.graph.QuantumGraph.quanta

(

self

)

Iterator over quanta in a graph.

Quanta are returned in unspecified order.

Yields
------
taskDef : `TaskDef`
    Task definition for a Quantum.
quantum : `~lsst.daf.butler.Quantum`
    Single quantum.

Definition at line 207 of file graph.py.

    def quanta(self):
        """Iterator over quanta in a graph.
 
        Quanta are returned in unspecified order.
 
        Yields
        ------
        taskDef : `TaskDef`
            Task definition for a Quantum.
        quantum : `~lsst.daf.butler.Quantum`
            Single quantum.
        """
        for taskNodes in self:
            taskDef = taskNodes.taskDef
            for quantum in taskNodes.quanta:
                yield taskDef, quantum
 

quantaAsQgraph()

def lsst.pipe.base.graph.QuantumGraph.quantaAsQgraph

(

self

)

Iterator over quanta in a graph.

QuantumGraph containing individual quanta are returned.

Yields
------
graph : `QuantumGraph`

Definition at line 224 of file graph.py.

    def quantaAsQgraph(self):
        """Iterator over quanta in a graph.
 
        QuantumGraph containing individual quanta are returned.
 
        Yields
        ------
        graph : `QuantumGraph`
        """
        for qdata in self.traverse():
            node = QuantumGraphTaskNodes(qdata.taskDef, [qdata.quantum],
                                         qdata.quantum.initInputs, qdata.quantum.outputs)
            graph = QuantumGraph([node])
            yield graph
 

save()

def lsst.pipe.base.graph.QuantumGraph.save	(		self,
			file
	)

Save QuantumGraph to a file.

Presently we store QuantumGraph in pickle format, this could
potentially change in the future if better format is found.

Parameters
----------
file : `io.BufferedIOBase`
    File to write pickle data open in binary mode.

Definition at line 194 of file graph.py.

    def save(self, file):
        """Save QuantumGraph to a file.
 
        Presently we store QuantumGraph in pickle format, this could
        potentially change in the future if better format is found.
 
        Parameters
        ----------
        file : `io.BufferedIOBase`
            File to write pickle data open in binary mode.
        """
        pickle.dump(self, file)
 

traverse()

def lsst.pipe.base.graph.QuantumGraph.traverse

(

self

)

Return topologically ordered Quanta and their dependencies.

This method iterates over all Quanta in topological order, enumerating
them during iteration. Returned `QuantumIterData` object contains
Quantum instance, its ``index`` and the ``index`` of all its
prerequsites (Quanta that produce inputs for this Quantum):

- the ``index`` values are generated by an iteration of a
  QuantumGraph, and are not intrinsic to the QuantumGraph
- during iteration, each ID will appear in index before it ever
  appears in dependencies.

Yields
------
quantumData : `QuantumIterData`

Definition at line 249 of file graph.py.

    def traverse(self):
        """Return topologically ordered Quanta and their dependencies.
 
        This method iterates over all Quanta in topological order, enumerating
        them during iteration. Returned `QuantumIterData` object contains
        Quantum instance, its ``index`` and the ``index`` of all its
        prerequsites (Quanta that produce inputs for this Quantum):
 
        - the ``index`` values are generated by an iteration of a
          QuantumGraph, and are not intrinsic to the QuantumGraph
        - during iteration, each ID will appear in index before it ever
          appears in dependencies.
 
        Yields
        ------
        quantumData : `QuantumIterData`
        """
 
        def orderedTaskNodes(graph):
            """Return topologically ordered task nodes.
 
            Yields
            ------
            nodes : `QuantumGraphTaskNodes`
            """
            # Tasks in a graph are probably topologically sorted already but there
            # is no guarantee for that. Just re-construct Pipeline and order tasks
            # in a pipeline using existing method.
            nodesMap = {id(item.taskDef): item for item in graph}
            pipeline = orderPipeline([item.taskDef for item in graph])
            for taskDef in pipeline:
                yield nodesMap[id(taskDef)]
 
        index = 0
        outputs = {}  # maps (DatasetType.name, dataId) to its producing quantum index
        for nodes in orderedTaskNodes(self):
            for quantum in nodes.quanta:
 
                # Find quantum dependencies (must be in `outputs` already)
                prereq = []
                for dataRef in chain.from_iterable(quantum.predictedInputs.values()):
                    # if data exists in butler then `id` is not None
                    if dataRef.id is None:
                        # Get the base name if this is a component
                        name, component = dataRef.datasetType.nameAndComponent()
                        key = (name, dataRef.dataId)
                        try:
                            prereq.append(outputs[key])
                        except KeyError:
                            # The Quantum that makes our inputs is not in the graph,
                            # this could happen if we run on a "split graph" which is
                            # usually just one quantum. Check for number of Quanta
                            # in a graph and ignore error if it's just one.
                            # TODO: This code has to be removed or replaced with
                            # something more generic
                            if not (len(self) == 1 and len(self[0].quanta) == 1):
                                raise
 
                # Update `outputs` with this quantum outputs
                for dataRef in chain.from_iterable(quantum.outputs.values()):
                    key = (dataRef.datasetType.name, dataRef.dataId)
                    outputs[key] = index
 
                yield QuantumIterData(index=index, quantum=quantum, taskDef=nodes.taskDef,
                                      dependencies=frozenset(prereq))
                index += 1

Member Data Documentation

initInputs

lsst.pipe.base.graph.QuantumGraph.initInputs

Definition at line 134 of file graph.py.

initIntermediates

lsst.pipe.base.graph.QuantumGraph.initIntermediates

Definition at line 135 of file graph.py.

initOutputs

lsst.pipe.base.graph.QuantumGraph.initOutputs

Definition at line 136 of file graph.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/cb4e2dc/Linux64/pipe_base/20.0.0-14-g1ce627f+450400e286/python/lsst/pipe/base/graph.py