LSST Applications  21.0.0-172-gfb10e10a+18fedfabac,22.0.0+297cba6710,22.0.0+80564b0ff1,22.0.0+8d77f4f51a,22.0.0+a28f4c53b1,22.0.0+dcf3732eb2,22.0.1-1-g7d6de66+2a20fdde0d,22.0.1-1-g8e32f31+297cba6710,22.0.1-1-geca5380+7fa3b7d9b6,22.0.1-12-g44dc1dc+2a20fdde0d,22.0.1-15-g6a90155+515f58c32b,22.0.1-16-g9282f48+790f5f2caa,22.0.1-2-g92698f7+dcf3732eb2,22.0.1-2-ga9b0f51+7fa3b7d9b6,22.0.1-2-gd1925c9+bf4f0e694f,22.0.1-24-g1ad7a390+a9625a72a8,22.0.1-25-g5bf6245+3ad8ecd50b,22.0.1-25-gb120d7b+8b5510f75f,22.0.1-27-g97737f7+2a20fdde0d,22.0.1-32-gf62ce7b1+aa4237961e,22.0.1-4-g0b3f228+2a20fdde0d,22.0.1-4-g243d05b+871c1b8305,22.0.1-4-g3a563be+32dcf1063f,22.0.1-4-g44f2e3d+9e4ab0f4fa,22.0.1-42-gca6935d93+ba5e5ca3eb,22.0.1-5-g15c806e+85460ae5f3,22.0.1-5-g58711c4+611d128589,22.0.1-5-g75bb458+99c117b92f,22.0.1-6-g1c63a23+7fa3b7d9b6,22.0.1-6-g50866e6+84ff5a128b,22.0.1-6-g8d3140d+720564cf76,22.0.1-6-gd805d02+cc5644f571,22.0.1-8-ge5750ce+85460ae5f3,master-g6e05de7fdc+babf819c66,master-g99da0e417a+8d77f4f51a,w.2021.48
LSST Data Management Base Package
Public Types | Public Member Functions | Public Attributes | List of all members
lsst::cpputils::python::WrapperCollection Class Referencefinal

A helper class for subdividing pybind11 module across multiple translation units (i.e. More...

#include <python.h>

Public Types

using WrapperCallback = std::function< void(pybind11::module &)>
 Function handle type used to hold deferred wrapper declaration functions. More...
 
using WrapperCallback = std::function< void(pybind11::module &)>
 Function handle type used to hold deferred wrapper declaration functions. More...
 

Public Member Functions

 WrapperCollection (pybind11::module module_, std::string const &package)
 Construct a new WrapperCollection. More...
 
 WrapperCollection (WrapperCollection &&other) noexcept
 
 WrapperCollection (WrapperCollection const &)=delete
 
WrapperCollectionoperator= (WrapperCollection const &)=delete
 
WrapperCollectionoperator= (WrapperCollection &&)=delete
 
 ~WrapperCollection () noexcept
 
WrapperCollection makeSubmodule (std::string const &name)
 Create a WrapperCollection for a submodule defined in the same binary. More...
 
void collectSubmodule (WrapperCollection &&submodule)
 Merge deferred definitions in the given submodule into the parent WrapperCollection. More...
 
void addInheritanceDependency (std::string const &name)
 Indicate an external module that provides a base class for a subsequent addType call. More...
 
void addSignatureDependency (std::string const &name)
 Indicate an external module that provides a type used in function/method signatures. More...
 
void wrap (WrapperCallback function)
 Add a set of wrappers without defining a class. More...
 
template<typename PyType , typename ClassWrapperCallback >
PyType wrapType (PyType cls, ClassWrapperCallback function, bool setModuleName=true)
 Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called. More...
 
template<typename CxxException , typename CxxBase >
auto wrapException (std::string const &pyName, std::string const &pyBase, bool setModuleName=true)
 Wrap a C++ exception as a Python exception. More...
 
void finish ()
 Invoke all deferred wrapper-declaring callables. More...
 
 WrapperCollection (pybind11::module module_, std::string const &package)
 Construct a new WrapperCollection. More...
 
 WrapperCollection (WrapperCollection &&other) noexcept
 
 WrapperCollection (WrapperCollection const &)=delete
 
WrapperCollectionoperator= (WrapperCollection const &)=delete
 
WrapperCollectionoperator= (WrapperCollection &&)=delete
 
 ~WrapperCollection () noexcept
 
WrapperCollection makeSubmodule (std::string const &name)
 Create a WrapperCollection for a submodule defined in the same binary. More...
 
void collectSubmodule (WrapperCollection &&submodule)
 Merge deferred definitions in the given submodule into the parent WrapperCollection. More...
 
void addInheritanceDependency (std::string const &name)
 Indicate an external module that provides a base class for a subsequent addType call. More...
 
void addSignatureDependency (std::string const &name)
 Indicate an external module that provides a type used in function/method signatures. More...
 
void wrap (WrapperCallback function)
 Add a set of wrappers without defining a class. More...
 
template<typename PyType , typename ClassWrapperCallback >
PyType wrapType (PyType cls, ClassWrapperCallback function, bool setModuleName=true)
 Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called. More...
 
template<typename CxxException , typename CxxBase >
auto wrapException (std::string const &pyName, std::string const &pyBase, bool setModuleName=true)
 Wrap a C++ exception as a Python exception. More...
 
void finish ()
 Invoke all deferred wrapper-declaring callables. More...
 

Public Attributes

pybind11::module module
 The module object passed to the PYBIND11_MODULE block that contains this WrapperCollection. More...
 

Detailed Description

A helper class for subdividing pybind11 module across multiple translation units (i.e.

source files).

Merging wrappers for different classes into a single compiled module can dramatically decrease the total size of the binary, but putting the source for multiple wrappers into a single file slows down incremental rebuilds and makes editing unwieldy. The right approach is to define wrappers in different source files and link them into a single module at build time. In simple cases, that's quite straightforward: pybind11 declarations are just regular C++ statements, and you can factor them out into different functions in different source files.

That approach doesn't work so well when the classes being wrapped are interdependent, because bindings are only guaranteed to work when all types used in a wrapped method signature have been declared to pybind11 before the method using them is itself declared. Naively, then, each source file would thus have to have multiple wrapper-declaring functions, so all type-wrapping functions could be executed before any method-wrapping functions. Of course, each type-wrapping function would also have to pass its type object to at least one method-wrapping function (to wrap the types own methods), and the result is a tangled mess of wrapper-declaring functions that obfuscate the code with a lot of boilerplate.

WrapperCollection provides a way out of that by allowing type wrappers and their associated methods to be declared at a single point, but the method wrappers wrapped in a lambda to defer their execution. A single WrapperCollection instance is typically constructed at the beginning of a PYBIND11_MODULE block, then passed by reference to wrapper-declaring functions defined in other source files. As type and method wrappers are added to the WrapperCollection by those functions, the types are registered immediately, and the method-wrapping lambdas are collected. After all wrapper-declaring functions have been called, finish() is called at the end of the PYBIND11_MODULE block to execute the collecting method-wrapping lambdas.

Typical usage:

// _mypackage.cc
void wrapClassA(WrapperCollection & wrappers);
void wrapClassB(WrapperCollection & wrappers);
PYBIND11_MODULE(_mypackage, module) {
WrapperCollection wrappers(module, "mypackage");
wrapClassA(wrappers);
wrapClassB(wrappers);
wrappers.finish();
}
pybind11::module module
The module object passed to the PYBIND11_MODULE block that contains this WrapperCollection.
Definition: python.h:448
WrapperCollection(pybind11::module module_, std::string const &package)
Construct a new WrapperCollection.
Definition: python.h:264
PYBIND11_MODULE(_cpputils, mod)
Definition: _cpputils.cc:32
// _ClassA.cc
void wrapClassA(WrapperCollection & wrappers) {
wrappers.wrapType(
py::class_<ClassA>(wrappers.module, "ClassA"),
[](auto & mod, auto & cls) {
cls.def("methodOnClassA", &methodOnClassA);
}
);
}
// _ClassB.cc
void wrapClassB(WrapperCollection & wrappers) {
wrappers.wrapType(
py::class_<ClassB>(wrappers.module, "ClassB"),
[](auto & mod, auto & cls) {
cls.def("methodOnClassB", &methodOnClassB);
mod.def("freeFunction", &freeFunction);
}
);
}

Note that we recommend the use of universal lambdas (i.e. auto & parameters) to reduce verbosity.

Definition at line 242 of file python.h.

Member Typedef Documentation

◆ WrapperCallback [1/2]

Function handle type used to hold deferred wrapper declaration functions.

Definition at line 247 of file python.h.

◆ WrapperCallback [2/2]

Function handle type used to hold deferred wrapper declaration functions.

Definition at line 247 of file python.h.

Constructor & Destructor Documentation

◆ WrapperCollection() [1/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( pybind11::module  module_,
std::string const &  package 
)
inlineexplicit

Construct a new WrapperCollection.

A WrapperCollection should be constructed at or near the top of a PYBIND11_MODULE block.

Parameters
[in]module_Module instance passed to the PYBIND11_MODULE macro.
[in]packageString name of the package all wrapped classes should appear to be from (by resetting their __module__ attribute). Note that this can lead to problems if classes are not also lifted into the package namespace in its __init__.py (in addition to confusing users, this will prevent unpickling from working).

Definition at line 264 of file python.h.

264  :
265  module(module_),
266  _package(package)
267  {}

◆ WrapperCollection() [2/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( WrapperCollection &&  other)
inlinenoexcept

Definition at line 270 of file python.h.

270  :
271  module(std::move(other.module)),
272  _package(std::move(other._package)),
273  _dependencies(std::move(other._dependencies)),
274  _definitions(std::move(other._definitions))
275  {}
T move(T... args)

◆ WrapperCollection() [3/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( WrapperCollection const &  )
delete

◆ ~WrapperCollection() [1/2]

lsst::cpputils::python::WrapperCollection::~WrapperCollection ( )
inlinenoexcept

Definition at line 282 of file python.h.

282  {
283  if (std::uncaught_exceptions()==0 && !_definitions.empty()) {
284  PyErr_SetString(PyExc_ImportError,
285  "WrapperCollection::finish() not called; module definition incomplete.");
286  PyErr_WriteUnraisable(module.ptr());
287  }
288  }

◆ WrapperCollection() [4/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( pybind11::module  module_,
std::string const &  package 
)
inlineexplicit

Construct a new WrapperCollection.

A WrapperCollection should be constructed at or near the top of a PYBIND11_MODULE block.

Parameters
[in]module_Module instance passed to the PYBIND11_MODULE macro.
[in]packageString name of the package all wrapped classes should appear to be from (by resetting their __module__ attribute). Note that this can lead to problems if classes are not also lifted into the package namespace in its __init__.py (in addition to confusing users, this will prevent unpickling from working).

Definition at line 264 of file python.h.

264  :
265  module(module_),
266  _package(package)
267  {}

◆ WrapperCollection() [5/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( WrapperCollection &&  other)
inlinenoexcept

Definition at line 270 of file python.h.

270  :
271  module(std::move(other.module)),
272  _package(std::move(other._package)),
273  _dependencies(std::move(other._dependencies)),
274  _definitions(std::move(other._definitions))
275  {}

◆ WrapperCollection() [6/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( WrapperCollection const &  )
delete

◆ ~WrapperCollection() [2/2]

lsst::cpputils::python::WrapperCollection::~WrapperCollection ( )
inlinenoexcept

Definition at line 282 of file python.h.

282  {
283  if (std::uncaught_exceptions()==0 && !_definitions.empty()) {
284  PyErr_SetString(PyExc_ImportError,
285  "WrapperCollection::finish() not called; module definition incomplete.");
286  PyErr_WriteUnraisable(module.ptr());
287  }
288  }

Member Function Documentation

◆ addInheritanceDependency() [1/2]

void lsst::cpputils::python::WrapperCollection::addInheritanceDependency ( std::string const &  name)
inline

Indicate an external module that provides a base class for a subsequent addType call.

Dependencies that provide base classes cannot be deferred until after types are declared, and are always imported immediately.

Parameters
[in]nameName of the module to import (absolute).

Definition at line 343 of file python.h.

343  {
344  pybind11::module::import(name.c_str());
345  }
table::Key< std::string > name
Definition: Amplifier.cc:116

◆ addInheritanceDependency() [2/2]

void lsst::cpputils::python::WrapperCollection::addInheritanceDependency ( std::string const &  name)
inline

Indicate an external module that provides a base class for a subsequent addType call.

Dependencies that provide base classes cannot be deferred until after types are declared, and are always imported immediately.

Parameters
[in]nameName of the module to import (absolute).

Definition at line 343 of file python.h.

343  {
344  pybind11::module::import(name.c_str());
345  }

◆ addSignatureDependency() [1/2]

void lsst::cpputils::python::WrapperCollection::addSignatureDependency ( std::string const &  name)
inline

Indicate an external module that provides a type used in function/method signatures.

Dependencies that provide classes are imported after types in the current module are declared and before any methods and free functions in the current module are declared.

Parameters
[in]nameName of the module to import (absolute).

Definition at line 357 of file python.h.

357  {
358  _dependencies.push_back(name);
359  }
T push_back(T... args)

◆ addSignatureDependency() [2/2]

void lsst::cpputils::python::WrapperCollection::addSignatureDependency ( std::string const &  name)
inline

Indicate an external module that provides a type used in function/method signatures.

Dependencies that provide classes are imported after types in the current module are declared and before any methods and free functions in the current module are declared.

Parameters
[in]nameName of the module to import (absolute).

Definition at line 357 of file python.h.

357  {
358  _dependencies.push_back(name);
359  }

◆ collectSubmodule() [1/2]

void lsst::cpputils::python::WrapperCollection::collectSubmodule ( WrapperCollection &&  submodule)
inline

Merge deferred definitions in the given submodule into the parent WrapperCollection.

Parameters
submoduleA WrapperCollection created by makeSubmodule. Will be consumed (and must be an rvalue).

Definition at line 329 of file python.h.

329  {
330  _dependencies.splice(_dependencies.end(), submodule._dependencies);
331  _definitions.splice(_definitions.end(), submodule._definitions);
332  }
T end(T... args)
T splice(T... args)

◆ collectSubmodule() [2/2]

void lsst::cpputils::python::WrapperCollection::collectSubmodule ( WrapperCollection &&  submodule)
inline

Merge deferred definitions in the given submodule into the parent WrapperCollection.

Parameters
submoduleA WrapperCollection created by makeSubmodule. Will be consumed (and must be an rvalue).

Definition at line 329 of file python.h.

329  {
330  _dependencies.splice(_dependencies.end(), submodule._dependencies);
331  _definitions.splice(_definitions.end(), submodule._definitions);
332  }

◆ finish() [1/2]

void lsst::cpputils::python::WrapperCollection::finish ( )
inline

Invoke all deferred wrapper-declaring callables.

finish() should be called exactly once, at or near the end of a PYBIND11_MODULE block.

Definition at line 435 of file python.h.

435  {
436  for (auto dep = _dependencies.begin(); dep != _dependencies.end(); dep = _dependencies.erase(dep)) {
437  pybind11::module::import(dep->c_str());
438  }
439  for (auto def = _definitions.begin(); def != _definitions.end(); def = _definitions.erase(def)) {
440  (def->second)(def->first); // WrapperCallback(module)
441  }
442  }
T begin(T... args)
T erase(T... args)

◆ finish() [2/2]

void lsst::cpputils::python::WrapperCollection::finish ( )
inline

Invoke all deferred wrapper-declaring callables.

finish() should be called exactly once, at or near the end of a PYBIND11_MODULE block.

Definition at line 435 of file python.h.

435  {
436  for (auto dep = _dependencies.begin(); dep != _dependencies.end(); dep = _dependencies.erase(dep)) {
437  pybind11::module::import(dep->c_str());
438  }
439  for (auto def = _definitions.begin(); def != _definitions.end(); def = _definitions.erase(def)) {
440  (def->second)(def->first); // WrapperCallback(module)
441  }
442  }

◆ makeSubmodule() [1/2]

WrapperCollection lsst::cpputils::python::WrapperCollection::makeSubmodule ( std::string const &  name)
inline

Create a WrapperCollection for a submodule defined in the same binary.

WrapperCollections created with makeSubmodule should generally be destroyed by moving them into a call to collectSubmodule; this will cause all deferred definitions to be executed when the parent WrapperCollection's finish() method is called.

Parameters
nameRelative name of the submodule.

Attributes added to the returned WrapperCollection will actually be put in a submodule that adds an underscore prefix to name, with __module__ set with the expectation that they will be lifted into a package without that leading underscore by a line in __init__.py like:

from ._package import _submodule as submodule

This is necessary to make importing _package possible when submodule already exists as a normal (i.e. directory-based) package. Of course, in that case, you'd instead use a submodule/__init__.py with a line like:

from .._package._submodule import *
Returns
a new WrapperCollection instance that sets the __module__ of any classes added to it to {package}.{name}.

Definition at line 318 of file python.h.

318  {
319  return WrapperCollection(module.def_submodule(("_" + name).c_str()), _package + "." + name);
320  }

◆ makeSubmodule() [2/2]

WrapperCollection lsst::cpputils::python::WrapperCollection::makeSubmodule ( std::string const &  name)
inline

Create a WrapperCollection for a submodule defined in the same binary.

WrapperCollections created with makeSubmodule should generally be destroyed by moving them into a call to collectSubmodule; this will cause all deferred definitions to be executed when the parent WrapperCollection's finish() method is called.

Parameters
nameRelative name of the submodule.

Attributes added to the returned WrapperCollection will actually be put in a submodule that adds an underscore prefix to name, with __module__ set with the expectation that they will be lifted into a package without that leading underscore by a line in __init__.py like:

from ._package import _submodule as submodule

This is necessary to make importing _package possible when submodule already exists as a normal (i.e. directory-based) package. Of course, in that case, you'd instead use a submodule/__init__.py with a line like:

from .._package._submodule import *
Returns
a new WrapperCollection instance that sets the __module__ of any classes added to it to {package}.{name}.

Definition at line 318 of file python.h.

318  {
319  return WrapperCollection(module.def_submodule(("_" + name).c_str()), _package + "." + name);
320  }

◆ operator=() [1/4]

WrapperCollection& lsst::cpputils::python::WrapperCollection::operator= ( WrapperCollection &&  )
delete

◆ operator=() [2/4]

WrapperCollection& lsst::cpputils::python::WrapperCollection::operator= ( WrapperCollection &&  )
delete

◆ operator=() [3/4]

WrapperCollection& lsst::cpputils::python::WrapperCollection::operator= ( WrapperCollection const &  )
delete

◆ operator=() [4/4]

WrapperCollection& lsst::cpputils::python::WrapperCollection::operator= ( WrapperCollection const &  )
delete

◆ wrap() [1/2]

void lsst::cpputils::python::WrapperCollection::wrap ( WrapperCallback  function)
inline

Add a set of wrappers without defining a class.

Parameters
[in]functionA callable object that takes a single pybind11::module argument (by reference) and adds pybind11 wrappers to it, to be called later by finish().

Definition at line 369 of file python.h.

369  {
370  _definitions.emplace_back(std::make_pair(module, function));
371  }
T make_pair(T... args)

◆ wrap() [2/2]

void lsst::cpputils::python::WrapperCollection::wrap ( WrapperCallback  function)
inline

Add a set of wrappers without defining a class.

Parameters
[in]functionA callable object that takes a single pybind11::module argument (by reference) and adds pybind11 wrappers to it, to be called later by finish().

Definition at line 369 of file python.h.

369  {
370  _definitions.emplace_back(std::make_pair(module, function));
371  }

◆ wrapException() [1/2]

template<typename CxxException , typename CxxBase >
auto lsst::cpputils::python::WrapperCollection::wrapException ( std::string const &  pyName,
std::string const &  pyBase,
bool  setModuleName = true 
)
inline

Wrap a C++ exception as a Python exception.

Template Parameters
CxxExceptionC++ Exception type to wrap.
CxxBaseBase class of CxxException.
Parameters
[in]pyNamePython name of the new exception.
[in]pyBasePython name of the pex::exceptions Exception type the new exception inherits from.
[in]setModuleNameIf true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.
Returns
a pybind11::class_ instance (template parameters unspecified) representing the Python type of the new exception.

Definition at line 421 of file python.h.

421  {
422  auto cls = pex::exceptions::python::declareException<CxxException, CxxBase>(module, pyName, pyBase);
423  if (setModuleName) {
424  cls.attr("__module__") = _package;
425  }
426  return cls;
427  }

◆ wrapException() [2/2]

template<typename CxxException , typename CxxBase >
auto lsst::cpputils::python::WrapperCollection::wrapException ( std::string const &  pyName,
std::string const &  pyBase,
bool  setModuleName = true 
)
inline

Wrap a C++ exception as a Python exception.

Template Parameters
CxxExceptionC++ Exception type to wrap.
CxxBaseBase class of CxxException.
Parameters
[in]pyNamePython name of the new exception.
[in]pyBasePython name of the pex::exceptions Exception type the new exception inherits from.
[in]setModuleNameIf true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.
Returns
a pybind11::class_ instance (template parameters unspecified) representing the Python type of the new exception.

Definition at line 421 of file python.h.

421  {
422  auto cls = pex::exceptions::python::declareException<CxxException, CxxBase>(module, pyName, pyBase);
423  if (setModuleName) {
424  cls.attr("__module__") = _package;
425  }
426  return cls;
427  }

◆ wrapType() [1/2]

template<typename PyType , typename ClassWrapperCallback >
PyType lsst::cpputils::python::WrapperCollection::wrapType ( PyType  cls,
ClassWrapperCallback  function,
bool  setModuleName = true 
)
inline

Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called.

Parameters
[in]clsA pybind11::class_ or enum_ instance.
[in]functionA callable object that takes a pybind11::module argument and a pybind11::class_ (or enum_) argument (both by reference) and defines wrappers for the class's methods and other attributes. Will be called with this->module and cls by finish().
[in]setModuleNameIf true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.
Returns
the cls argument for convenience

Definition at line 391 of file python.h.

391  {
392  if (setModuleName) {
393  cls.attr("__module__") = _package;
394  }
395  // lambda below is mutable so it can modify the captured `cls` variable
396  wrap(
397  [cls=cls, function=std::move(function)] (pybind11::module & mod) mutable -> void {
398  function(mod, cls);
399  }
400  );
401  return cls;
402  }
void wrap(WrapperCallback function)
Add a set of wrappers without defining a class.
Definition: python.h:369

◆ wrapType() [2/2]

template<typename PyType , typename ClassWrapperCallback >
PyType lsst::cpputils::python::WrapperCollection::wrapType ( PyType  cls,
ClassWrapperCallback  function,
bool  setModuleName = true 
)
inline

Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called.

Parameters
[in]clsA pybind11::class_ or enum_ instance.
[in]functionA callable object that takes a pybind11::module argument and a pybind11::class_ (or enum_) argument (both by reference) and defines wrappers for the class's methods and other attributes. Will be called with this->module and cls by finish().
[in]setModuleNameIf true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.
Returns
the cls argument for convenience

Definition at line 391 of file python.h.

391  {
392  if (setModuleName) {
393  cls.attr("__module__") = _package;
394  }
395  // lambda below is mutable so it can modify the captured `cls` variable
396  wrap(
397  [cls=cls, function=std::move(function)] (pybind11::module & mod) mutable -> void {
398  function(mod, cls);
399  }
400  );
401  return cls;
402  }

Member Data Documentation

◆ module

pybind11::module lsst::cpputils::python::WrapperCollection::module

The module object passed to the PYBIND11_MODULE block that contains this WrapperCollection.

Definition at line 448 of file python.h.


The documentation for this class was generated from the following file: