Running an example interSlice communication pipeline: examples/ring/

This example is found under pex_harness/examples/ring/ .

The default parameters for this example are for execution on a single node with a central Pipeline and 5 Slices. The launchPipeline.py derives the parameters from the MPI "machine file" nodelist.scr, which contains the entry

lsst8.ncsa.uiuc.edu:6

The parameter value 6 that appears after the hostname specifies the total number of MPI processes on the host, and equals the sum of the 1 Pipeline process and the 5 Slices. To increase the number of Slices in the MPI ring, simply increase this value.

Prior to running the example, the full path to the directory pex_harness/examples/stages should be added to the PYTHONPATH:

% export PYTHONPATH ${PWD}/../stages:${PYTHONPATH}    (bash)
% setenv PYTHONPATH ${PWD}/../stages:${PYTHONPATH}    (tcsh)

The example pipeline can be launched by placing a policy file and a runid on the command line of launchPipeline.py, for example,

% launchPipeline.py ring.paf test_2390

To understand the operation of this pipeline we can examine pieces of the ring.paf policy file :

localLogMode: true
executionMode: "oneloop"
logThreshold: -3
eventBrokerHost: "lsst8.ncsa.uiuc.edu"

shareDataOn: true

topology: {
    type: "ring"
    param1: "clockwise"
    # param1: "counterclockwise"
}

appStage: {
    stageName: "lsst.pexhexamples.apps.SyncSetupStage"
    eventTopic: "None"
    stagePolicy: "None"
    shareData: false
}

appStage: {
    stageName: "lsst.pexhexamples.apps.SyncTestStage"
    eventTopic: "None"
    stagePolicy: "None"
    shareData: true
}

InterSlice communication is enabled globally with shareDataOn: true and the topology is set to a ring with attribute clockwise. In this setup we have the Slices in a line: 0 1 2 3 4 with the ends connected, hence, Slice 0 will have neighbors 4 and 1, Slice 1 will have neighbors 0 and 2, etc. The communication will have the character that Slice 0 receives from Slice 4 and sends to Slice 1, Slice 1 receives from Slice 0 and sends to Slice 2, etc. Because the option localLogMode: "Yes" is set, after execution logs files Pipeline.log, Slice0.log, Slice1.log, ... will be found in the working directory.

The pipeline contains two Stages, SyncSetupStage and SyncTestStage. Within its process() method SyncSetupStage places a PropertySet on the Clipboard under the key "rankKey" and marks it as Shared:

propertySet = dafBase.PropertySet()

propertySet.setInt("sliceRank", self._rank)
propertySet.setString("Level", "Debug")

self.activeClipboard.put("rankKey", propertySet)

self.activeClipboard.setShared("rankKey", True)

The next Stage SyncTestStage has the attribute shareData: true set, which ensures that all items marked shared on the Clipboard will be communicated prior to the execution of the process() method for this Stage. The received PropertySet is placed on the Clipboard under a key tagged with the rank of the neighbor Slice, e.g., rankKey-0 for the case where Slice 1 receives data from Slice 0.