LSST Applications
21.0.0-172-gfb10e10a+18fedfabac,22.0.0+297cba6710,22.0.0+80564b0ff1,22.0.0+8d77f4f51a,22.0.0+a28f4c53b1,22.0.0+dcf3732eb2,22.0.1-1-g7d6de66+2a20fdde0d,22.0.1-1-g8e32f31+297cba6710,22.0.1-1-geca5380+7fa3b7d9b6,22.0.1-12-g44dc1dc+2a20fdde0d,22.0.1-15-g6a90155+515f58c32b,22.0.1-16-g9282f48+790f5f2caa,22.0.1-2-g92698f7+dcf3732eb2,22.0.1-2-ga9b0f51+7fa3b7d9b6,22.0.1-2-gd1925c9+bf4f0e694f,22.0.1-24-g1ad7a390+a9625a72a8,22.0.1-25-g5bf6245+3ad8ecd50b,22.0.1-25-gb120d7b+8b5510f75f,22.0.1-27-g97737f7+2a20fdde0d,22.0.1-32-gf62ce7b1+aa4237961e,22.0.1-4-g0b3f228+2a20fdde0d,22.0.1-4-g243d05b+871c1b8305,22.0.1-4-g3a563be+32dcf1063f,22.0.1-4-g44f2e3d+9e4ab0f4fa,22.0.1-42-gca6935d93+ba5e5ca3eb,22.0.1-5-g15c806e+85460ae5f3,22.0.1-5-g58711c4+611d128589,22.0.1-5-g75bb458+99c117b92f,22.0.1-6-g1c63a23+7fa3b7d9b6,22.0.1-6-g50866e6+84ff5a128b,22.0.1-6-g8d3140d+720564cf76,22.0.1-6-gd805d02+cc5644f571,22.0.1-8-ge5750ce+85460ae5f3,master-g6e05de7fdc+babf819c66,master-g99da0e417a+8d77f4f51a,w.2021.48
LSST Data Management Base Package
|
Public Member Functions | |
def | __init__ (self, *args, **kwargs) |
def | __del__ (self) |
def | log (self, msg, *args) |
def | command (self, cmd) |
def | map (self, context, func, dataList, *args, **kwargs) |
Scatter work to slaves and gather the results. More... | |
def | reduce (self, context, reducer, func, dataList, *args, **kwargs) |
Scatter work to slaves and reduce the results. More... | |
def | mapNoBalance (self, context, func, dataList, *args, **kwargs) |
Scatter work to slaves and gather the results. More... | |
def | reduceNoBalance (self, context, reducer, func, dataList, *args, **kwargs) |
Scatter work to slaves and reduce the results. More... | |
def | mapToPrevious (self, context, func, dataList, *args, **kwargs) |
Scatter work to the same target as before. More... | |
def | reduceToPrevious (self, context, reducer, func, dataList, *args, **kwargs) |
Reduction where work goes to the same target as before. More... | |
def | storeSet (self, context, **kwargs) |
Store data on slave for a particular context. More... | |
def | storeDel (self, context, *nameList) |
def | storeClear (self, context) |
def | cacheClear (self, context) |
def | cacheList (self, context) |
def | storeList (self, context) |
def | exit (self) |
def | isMaster (self) |
def | __call__ (cls, *args, **kwargs) |
Public Attributes | |
root | |
size | |
comm | |
rank | |
debugger | |
node | |
Master node instance of MPI process pool Only the master node should instantiate this. WARNING: You should not let a pool instance hang around at program termination, as the garbage collection behaves differently, and may cause a segmentation fault (signal 11).
def lsst.ctrl.pool.pool.PoolMaster.__init__ | ( | self, | |
* | args, | ||
** | kwargs | ||
) |
def lsst.ctrl.pool.pool.PoolMaster.__del__ | ( | self | ) |
|
inherited |
def lsst.ctrl.pool.pool.PoolMaster.cacheClear | ( | self, | |
context | |||
) |
Reset cache for a particular context on master and slaves
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.cacheList | ( | self, | |
context | |||
) |
List cache contents for a particular context on master and slaves
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.command | ( | self, | |
cmd | |||
) |
Send command to slaves A command is the name of the PoolSlave method they should run.
def lsst.ctrl.pool.pool.PoolMaster.exit | ( | self | ) |
|
inherited |
def lsst.ctrl.pool.pool.PoolMaster.log | ( | self, | |
msg, | |||
* | args | ||
) |
Log a debugging message
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
Definition at line 644 of file pool.py.
def lsst.ctrl.pool.pool.PoolMaster.map | ( | self, | |
context, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to slaves and gather the results.
Work is distributed dynamically, so that slaves that finish quickly will receive more work. Each slave applies the function to the data they're provided. The slaves may optionally be passed a cache instance, which they can use to store data for subsequent executions (to ensure subsequent data is distributed in the same pattern as before, use the 'mapToPrevious' method). The cache also contains data that has been stored on the slaves. The 'func' signature should be func(cache, data, *args, **kwargs) if 'context' is non-None; otherwise func(data, *args, **kwargs). @param context: Namespace for cache @param func: function for slaves to run; must be picklable @param dataList: List of data to distribute to slaves; must be picklable @param args: List of constant arguments @param kwargs: Dict of constant arguments @return list of results from applying 'func' to dataList
Definition at line 656 of file pool.py.
def lsst.ctrl.pool.pool.PoolMaster.mapNoBalance | ( | self, | |
context, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to slaves and gather the results.
Work is distributed statically, so there is no load balancing. Each slave applies the function to the data they're provided. The slaves may optionally be passed a cache instance, which they can store data in for subsequent executions (to ensure subsequent data is distributed in the same pattern as before, use the 'mapToPrevious' method). The cache also contains data that has been stored on the slaves. The 'func' signature should be func(cache, data, *args, **kwargs) if 'context' is true; otherwise func(data, *args, **kwargs). @param context: Namespace for cache @param func: function for slaves to run; must be picklable @param dataList: List of data to distribute to slaves; must be picklable @param args: List of constant arguments @param kwargs: Dict of constant arguments @return list of results from applying 'func' to dataList
Definition at line 763 of file pool.py.
def lsst.ctrl.pool.pool.PoolMaster.mapToPrevious | ( | self, | |
context, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to the same target as before.
Work is distributed so that each slave handles the same indices in the dataList as when 'map' was called. This allows the right data to go to the right cache. It is assumed that the dataList is the same length as when it was passed to 'map'. The 'func' signature should be func(cache, data, *args, **kwargs). @param context: Namespace for cache @param func: function for slaves to run; must be picklable @param dataList: List of data to distribute to slaves; must be picklable @param args: List of constant arguments @param kwargs: Dict of constant arguments @return list of results from applying 'func' to dataList
Definition at line 885 of file pool.py.
def lsst.ctrl.pool.pool.PoolMaster.reduce | ( | self, | |
context, | |||
reducer, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to slaves and reduce the results.
Work is distributed dynamically, so that slaves that finish quickly will receive more work. Each slave applies the function to the data they're provided. The slaves may optionally be passed a cache instance, which they can use to store data for subsequent executions (to ensure subsequent data is distributed in the same pattern as before, use the 'mapToPrevious' method). The cache also contains data that has been stored on the slaves. The 'func' signature should be func(cache, data, *args, **kwargs) if 'context' is non-None; otherwise func(data, *args, **kwargs). The 'reducer' signature should be reducer(old, new). If the 'reducer' is None, then we will return the full list of results @param context: Namespace for cache @param reducer: function for master to run to reduce slave results; or None @param func: function for slaves to run; must be picklable @param dataList: List of data to distribute to slaves; must be picklable @param args: List of constant arguments @param kwargs: Dict of constant arguments @return reduced result (if reducer is non-None) or list of results from applying 'func' to dataList
Definition at line 683 of file pool.py.
def lsst.ctrl.pool.pool.PoolMaster.reduceNoBalance | ( | self, | |
context, | |||
reducer, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Scatter work to slaves and reduce the results.
Work is distributed statically, so there is no load balancing. Each slave applies the function to the data they're provided. The slaves may optionally be passed a cache instance, which they can store data in for subsequent executions (to ensure subsequent data is distributed in the same pattern as before, use the 'mapToPrevious' method). The cache also contains data that has been stored on the slaves. The 'func' signature should be func(cache, data, *args, **kwargs) if 'context' is true; otherwise func(data, *args, **kwargs). The 'reducer' signature should be reducer(old, new). If the 'reducer' is None, then we will return the full list of results @param context: Namespace for cache @param reducer: function for master to run to reduce slave results; or None @param func: function for slaves to run; must be picklable @param dataList: List of data to distribute to slaves; must be picklable @param args: List of constant arguments @param kwargs: Dict of constant arguments @return reduced result (if reducer is non-None) or list of results from applying 'func' to dataList
Definition at line 789 of file pool.py.
def lsst.ctrl.pool.pool.PoolMaster.reduceToPrevious | ( | self, | |
context, | |||
reducer, | |||
func, | |||
dataList, | |||
* | args, | ||
** | kwargs | ||
) |
Reduction where work goes to the same target as before.
Work is distributed so that each slave handles the same indices in the dataList as when 'map' was called. This allows the right data to go to the right cache. It is assumed that the dataList is the same length as when it was passed to 'map'. The 'func' signature should be func(cache, data, *args, **kwargs). The 'reducer' signature should be reducer(old, new). If the 'reducer' is None, then we will return the full list of results @param context: Namespace for cache @param reducer: function for master to run to reduce slave results; or None @param func: function for slaves to run; must be picklable @param dataList: List of data to distribute to slaves; must be picklable @param args: List of constant arguments @param kwargs: Dict of constant arguments @return reduced result (if reducer is non-None) or list of results from applying 'func' to dataList
Definition at line 908 of file pool.py.
def lsst.ctrl.pool.pool.PoolMaster.storeClear | ( | self, | |
context | |||
) |
Reset data store for a particular context on master and slaves
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.storeDel | ( | self, | |
context, | |||
* | nameList | ||
) |
Delete stored data on slave for a particular context
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
Definition at line 1007 of file pool.py.
def lsst.ctrl.pool.pool.PoolMaster.storeList | ( | self, | |
context | |||
) |
List store contents for a particular context on master and slaves
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
def lsst.ctrl.pool.pool.PoolMaster.storeSet | ( | self, | |
context, | |||
** | kwargs | ||
) |
Store data on slave for a particular context.
The data is made available to functions through the cache. The stored data differs from the cache in that it is identical for all operations, whereas the cache is specific to the data being operated upon. @param context: namespace for store @param kwargs: dict of name=value pairs
Reimplemented from lsst.ctrl.pool.pool.PoolNode.
Definition at line 989 of file pool.py.