LSSTApplications  10.0+286,10.0+36,10.0+46,10.0-2-g4f67435,10.1+152,10.1+37,11.0,11.0+1,11.0-1-g47edd16,11.0-1-g60db491,11.0-1-g7418c06,11.0-2-g04d2804,11.0-2-g68503cd,11.0-2-g818369d,11.0-2-gb8b8ce7
LSSTDataManagementBasePackage
Classes | Functions | Variables
lsst.pipe.base.cmdLineTask Namespace Reference

Classes

class  TaskRunner
 Run a command-line task, using multiprocessing if requested. More...
 
class  ButlerInitializedTaskRunner
 A TaskRunner for CmdLineTasks that require a 'butler' keyword argument to be passed to their constructor. More...
 
class  CmdLineTask
 Base class for command-line tasks: tasks that may be executed from the command line. More...
 

Functions

def _poolFunctionWrapper
 
def _runPool
 
def profile
 Context manager for profiling with cProfile. More...
 

Variables

list __all__ = ["CmdLineTask", "TaskRunner", "ButlerInitializedTaskRunner"]
 

Function Documentation

def lsst.pipe.base.cmdLineTask._poolFunctionWrapper (   function,
  arg 
)
private 
Wrapper around function to catch exceptions that don't inherit from Exception

Such exceptions aren't caught by multiprocessing, which causes the slave
process to crash and you end up hitting the timeout.

Definition at line 37 of file cmdLineTask.py.

37 
38 def _poolFunctionWrapper(function, arg):
39  """Wrapper around function to catch exceptions that don't inherit from Exception
40 
41  Such exceptions aren't caught by multiprocessing, which causes the slave
42  process to crash and you end up hitting the timeout.
43  """
44  try:
45  return function(arg)
46  except Exception:
47  raise # No worries
48  except:
49  # Need to wrap the exception with something multiprocessing will recognise
50  cls, exc, tb = sys.exc_info()
51  log = getDefaultLog()
52  log.warn("Unhandled exception %s (%s):\n%s" % (cls.__name__, exc, traceback.format_exc()))
53  raise Exception("Unhandled exception: %s (%s)" % (cls.__name__, exc))
def lsst.pipe.base.cmdLineTask._runPool (   pool,
  timeout,
  function,
  iterable 
)
private 
Wrapper around pool.map_async, to handle timeout

This is required so as to trigger an immediate interrupt on the KeyboardInterrupt (Ctrl-C); see
http://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool

Further wraps the function in _poolFunctionWrapper to catch exceptions
that don't inherit from Exception.

Definition at line 54 of file cmdLineTask.py.

54 
55 def _runPool(pool, timeout, function, iterable):
56  """Wrapper around pool.map_async, to handle timeout
57 
58  This is required so as to trigger an immediate interrupt on the KeyboardInterrupt (Ctrl-C); see
59  http://stackoverflow.com/questions/1408356/keyboard-interrupts-with-pythons-multiprocessing-pool
60 
61  Further wraps the function in _poolFunctionWrapper to catch exceptions
62  that don't inherit from Exception.
63  """
64  return pool.map_async(functools.partial(_poolFunctionWrapper, function), iterable).get(timeout)
65 
@contextlib.contextmanager
def lsst.pipe.base.cmdLineTask.profile (   filename,
  log = None 
)

Context manager for profiling with cProfile.

Parameters
filenamefilename to which to write profile (profiling disabled if None or empty)
loglog object for logging the profile operations

If profiling is enabled, the context manager returns the cProfile.Profile object (otherwise it returns None), which allows additional control over profiling. You can obtain this using the "as" clause, e.g.: 

with profile(filename) as prof:
    runYourCodeHere()

The output cumulative profile can be printed with a command-line like: 

python -c 'import pstats; pstats.Stats("<filename>").sort_stats("cumtime").print_stats(30)'

Definition at line 66 of file cmdLineTask.py.

66 
67 def profile(filename, log=None):
68  """!Context manager for profiling with cProfile
69 
70  @param filename filename to which to write profile (profiling disabled if None or empty)
71  @param log log object for logging the profile operations
72 
73  If profiling is enabled, the context manager returns the cProfile.Profile object (otherwise
74  it returns None), which allows additional control over profiling. You can obtain this using
75  the "as" clause, e.g.:
76 
77  with profile(filename) as prof:
78  runYourCodeHere()
79 
80  The output cumulative profile can be printed with a command-line like:
81 
82  python -c 'import pstats; pstats.Stats("<filename>").sort_stats("cumtime").print_stats(30)'
83  """
84  if not filename:
85  # Nothing to do
86  yield
87  return
88  from cProfile import Profile
89  profile = Profile()
90  if log is not None:
91  log.info("Enabling cProfile profiling")
92  profile.enable()
93  yield profile
94  profile.disable()
95  profile.dump_stats(filename)
96  if log is not None:
97  log.info("cProfile stats written to %s" % filename)
98 
lsst.pipe.base.cmdLineTask.profile
def profile
Context manager for profiling with cProfile.
Definition: cmdLineTask.py:66

Variable Documentation

list lsst.pipe.base.cmdLineTask.__all__ = ["CmdLineTask", "TaskRunner", "ButlerInitializedTaskRunner"]

Definition at line 35 of file cmdLineTask.py.

Generated on Thu Sep 24 2015 02:29:42 for LSSTApplications by   doxygen 1.8.5