LSST Applications g0265f82a02+093ff98f47,g02d81e74bb+10352d6f39,g1f3e9fa97e+40b0fc677d,g2079a07aa2+14824f138e,g2bbee38e9b+093ff98f47,g337abbeb29+093ff98f47,g3ddfee87b4+064c02c7ee,g487adcacf7+7e842ebf4b,g50ff169b8f+5929b3527e,g52b1c1532d+a6fc98d2e7,g568d43a26c+1d7ac31cb0,g591dd9f2cf+fb1f69e2ea,g858d7b2824+10352d6f39,g8a8a8dda67+a6fc98d2e7,g8cdfe0ae6a+66d966b544,g99cad8db69+7ce8a7c20a,g9ddcbc5298+d4bad12328,ga1e77700b3+246acaaf9c,ga2e4dd1c03+064c02c7ee,ga8c6da7877+04f6ba86dc,gae46bcf261+093ff98f47,gb0e22166c9+3863383f4c,gba4ed39666+9664299f35,gbb8dafda3b+db40f59a7d,gbeb006f7da+e6a448e96d,gbf5cecdb8a+10352d6f39,gc0f3af6251+10a3fd39cd,gc120e1dc64+5839e50a77,gc28159a63d+093ff98f47,gcf0d15dbbd+064c02c7ee,gd2a12a3803+0c2c227a2b,gdaeeff99f8+a38ce5ea23,ge79ae78c31+093ff98f47,gee10cc3b42+a6fc98d2e7,gf1cff7945b+10352d6f39,w.2024.15
LSST Data Management Base Package
Loading...
Searching...
No Matches
Public Types | Public Member Functions | List of all members
lsst::cpputils::python::WrapperCollection Class Referencefinal

A helper class for subdividing pybind11 module across multiple translation units (i.e. More...

#include <python.h>

Public Types

using WrapperCallback = std::function<void(pybind11::module &)>
 Function handle type used to hold deferred wrapper declaration functions.
 
using WrapperCallback = std::function<void(pybind11::module &)>
 Function handle type used to hold deferred wrapper declaration functions.
 

Public Member Functions

 WrapperCollection (pybind11::module module_, std::string const &package)
 Construct a new WrapperCollection.
 
 WrapperCollection (WrapperCollection &&other) noexcept
 
 WrapperCollection (WrapperCollection const &)=delete
 
WrapperCollectionoperator= (WrapperCollection const &)=delete
 
WrapperCollectionoperator= (WrapperCollection &&)=delete
 
 ~WrapperCollection () noexcept
 
WrapperCollection makeSubmodule (std::string const &name)
 Create a WrapperCollection for a submodule defined in the same binary.
 
void collectSubmodule (WrapperCollection &&submodule)
 Merge deferred definitions in the given submodule into the parent WrapperCollection.
 
void addInheritanceDependency (std::string const &name)
 Indicate an external module that provides a base class for a subsequent addType call.
 
void addSignatureDependency (std::string const &name)
 Indicate an external module that provides a type used in function/method signatures.
 
void wrap (WrapperCallback function)
 Add a set of wrappers without defining a class.
 
template<typename PyType , typename ClassWrapperCallback >
PyType wrapType (PyType cls, ClassWrapperCallback function, bool setModuleName=true)
 Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called.
 
template<typename CxxException , typename CxxBase >
auto wrapException (std::string const &pyName, std::string const &pyBase, bool setModuleName=true)
 Wrap a C++ exception as a Python exception.
 
void finish ()
 Invoke all deferred wrapper-declaring callables.
 
 WrapperCollection (pybind11::module module_, std::string const &package)
 Construct a new WrapperCollection.
 
 WrapperCollection (WrapperCollection &&other) noexcept
 
 WrapperCollection (WrapperCollection const &)=delete
 
WrapperCollectionoperator= (WrapperCollection const &)=delete
 
WrapperCollectionoperator= (WrapperCollection &&)=delete
 
 ~WrapperCollection () noexcept
 
WrapperCollection makeSubmodule (std::string const &name)
 Create a WrapperCollection for a submodule defined in the same binary.
 
void collectSubmodule (WrapperCollection &&submodule)
 Merge deferred definitions in the given submodule into the parent WrapperCollection.
 
void addInheritanceDependency (std::string const &name)
 Indicate an external module that provides a base class for a subsequent addType call.
 
void addSignatureDependency (std::string const &name)
 Indicate an external module that provides a type used in function/method signatures.
 
void wrap (WrapperCallback function)
 Add a set of wrappers without defining a class.
 
template<typename PyType , typename ClassWrapperCallback >
PyType wrapType (PyType cls, ClassWrapperCallback function, bool setModuleName=true)
 Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called.
 
template<typename CxxException , typename CxxBase >
auto wrapException (std::string const &pyName, std::string const &pyBase, bool setModuleName=true)
 Wrap a C++ exception as a Python exception.
 
void finish ()
 Invoke all deferred wrapper-declaring callables.
 

Detailed Description

A helper class for subdividing pybind11 module across multiple translation units (i.e.

source files).

Merging wrappers for different classes into a single compiled module can dramatically decrease the total size of the binary, but putting the source for multiple wrappers into a single file slows down incremental rebuilds and makes editing unwieldy. The right approach is to define wrappers in different source files and link them into a single module at build time. In simple cases, that's quite straightforward: pybind11 declarations are just regular C++ statements, and you can factor them out into different functions in different source files.

That approach doesn't work so well when the classes being wrapped are interdependent, because bindings are only guaranteed to work when all types used in a wrapped method signature have been declared to pybind11 before the method using them is itself declared. Naively, then, each source file would thus have to have multiple wrapper-declaring functions, so all type-wrapping functions could be executed before any method-wrapping functions. Of course, each type-wrapping function would also have to pass its type object to at least one method-wrapping function (to wrap the types own methods), and the result is a tangled mess of wrapper-declaring functions that obfuscate the code with a lot of boilerplate.

WrapperCollection provides a way out of that by allowing type wrappers and their associated methods to be declared at a single point, but the method wrappers wrapped in a lambda to defer their execution. A single WrapperCollection instance is typically constructed at the beginning of a PYBIND11_MODULE block, then passed by reference to wrapper-declaring functions defined in other source files. As type and method wrappers are added to the WrapperCollection by those functions, the types are registered immediately, and the method-wrapping lambdas are collected. After all wrapper-declaring functions have been called, finish() is called at the end of the PYBIND11_MODULE block to execute the collecting method-wrapping lambdas.

Typical usage:

// _mypackage.cc
void wrapClassA(WrapperCollection & wrappers);
void wrapClassB(WrapperCollection & wrappers);
PYBIND11_MODULE(_mypackage, module) {
WrapperCollection wrappers(module, "mypackage");
wrapClassA(wrappers);
wrapClassB(wrappers);
wrappers.finish();
}
A helper class for subdividing pybind11 module across multiple translation units (i....
Definition python.h:242
PYBIND11_MODULE(_cpputils, mod)
Definition _cpputils.cc:32
// _ClassA.cc
void wrapClassA(WrapperCollection & wrappers) {
wrappers.wrapType(
py::class_<ClassA>(wrappers.module, "ClassA"),
[](auto & mod, auto & cls) {
cls.def("methodOnClassA", &methodOnClassA);
}
);
}
PyType wrapType(PyType cls, ClassWrapperCallback function, bool setModuleName=true)
Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() i...
Definition python.h:391
// _ClassB.cc
void wrapClassB(WrapperCollection & wrappers) {
wrappers.wrapType(
py::class_<ClassB>(wrappers.module, "ClassB"),
[](auto & mod, auto & cls) {
cls.def("methodOnClassB", &methodOnClassB);
mod.def("freeFunction", &freeFunction);
}
);
}

Note that we recommend the use of universal lambdas (i.e. auto & parameters) to reduce verbosity.

Examples
imageStatistics.cc.

Definition at line 242 of file python.h.

Member Typedef Documentation

◆ WrapperCallback [1/2]

Function handle type used to hold deferred wrapper declaration functions.

Definition at line 247 of file python.h.

◆ WrapperCallback [2/2]

Function handle type used to hold deferred wrapper declaration functions.

Definition at line 247 of file python.h.

Constructor & Destructor Documentation

◆ WrapperCollection() [1/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( pybind11::module module_,
std::string const & package )
inlineexplicit

Construct a new WrapperCollection.

A WrapperCollection should be constructed at or near the top of a PYBIND11_MODULE block.

Parameters
[in]module_Module instance passed to the PYBIND11_MODULE macro.
[in]packageString name of the package all wrapped classes should appear to be from (by resetting their __module__ attribute). Note that this can lead to problems if classes are not also lifted into the package namespace in its __init__.py (in addition to confusing users, this will prevent unpickling from working).

Definition at line 264 of file python.h.

264 :
265 module(module_),
266 _package(package)
267 {}

◆ WrapperCollection() [2/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( WrapperCollection && other)
inlinenoexcept

Definition at line 270 of file python.h.

270 :
271 module(std::move(other.module)),
272 _package(std::move(other._package)),
273 _dependencies(std::move(other._dependencies)),
274 _definitions(std::move(other._definitions))
275 {}
T move(T... args)

◆ WrapperCollection() [3/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( WrapperCollection const & )
delete

◆ ~WrapperCollection() [1/2]

lsst::cpputils::python::WrapperCollection::~WrapperCollection ( )
inlinenoexcept

Definition at line 282 of file python.h.

282 {
283 if (std::uncaught_exceptions()==0 && !_definitions.empty()) {
284 PyErr_SetString(PyExc_ImportError,
285 "WrapperCollection::finish() not called; module definition incomplete.");
286 PyErr_WriteUnraisable(module.ptr());
287 }
288 }

◆ WrapperCollection() [4/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( pybind11::module module_,
std::string const & package )
inlineexplicit

Construct a new WrapperCollection.

A WrapperCollection should be constructed at or near the top of a PYBIND11_MODULE block.

Parameters
[in]module_Module instance passed to the PYBIND11_MODULE macro.
[in]packageString name of the package all wrapped classes should appear to be from (by resetting their __module__ attribute). Note that this can lead to problems if classes are not also lifted into the package namespace in its __init__.py (in addition to confusing users, this will prevent unpickling from working).

Definition at line 264 of file python.h.

264 :
265 module(module_),
266 _package(package)
267 {}

◆ WrapperCollection() [5/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( WrapperCollection && other)
inlinenoexcept

Definition at line 270 of file python.h.

270 :
271 module(std::move(other.module)),
272 _package(std::move(other._package)),
273 _dependencies(std::move(other._dependencies)),
274 _definitions(std::move(other._definitions))
275 {}

◆ WrapperCollection() [6/6]

lsst::cpputils::python::WrapperCollection::WrapperCollection ( WrapperCollection const & )
delete

◆ ~WrapperCollection() [2/2]

lsst::cpputils::python::WrapperCollection::~WrapperCollection ( )
inlinenoexcept

Definition at line 282 of file python.h.

282 {
283 if (std::uncaught_exceptions()==0 && !_definitions.empty()) {
284 PyErr_SetString(PyExc_ImportError,
285 "WrapperCollection::finish() not called; module definition incomplete.");
286 PyErr_WriteUnraisable(module.ptr());
287 }
288 }

Member Function Documentation

◆ addInheritanceDependency() [1/2]

void lsst::cpputils::python::WrapperCollection::addInheritanceDependency ( std::string const & name)
inline

Indicate an external module that provides a base class for a subsequent addType call.

Dependencies that provide base classes cannot be deferred until after types are declared, and are always imported immediately.

Parameters
[in]nameName of the module to import (absolute).

Definition at line 343 of file python.h.

343 {
344 pybind11::module::import(name.c_str());
345 }

◆ addInheritanceDependency() [2/2]

void lsst::cpputils::python::WrapperCollection::addInheritanceDependency ( std::string const & name)
inline

Indicate an external module that provides a base class for a subsequent addType call.

Dependencies that provide base classes cannot be deferred until after types are declared, and are always imported immediately.

Parameters
[in]nameName of the module to import (absolute).

Definition at line 343 of file python.h.

343 {
344 pybind11::module::import(name.c_str());
345 }

◆ addSignatureDependency() [1/2]

void lsst::cpputils::python::WrapperCollection::addSignatureDependency ( std::string const & name)
inline

Indicate an external module that provides a type used in function/method signatures.

Dependencies that provide classes are imported after types in the current module are declared and before any methods and free functions in the current module are declared.

Parameters
[in]nameName of the module to import (absolute).

Definition at line 357 of file python.h.

357 {
358 _dependencies.push_back(name);
359 }
T push_back(T... args)

◆ addSignatureDependency() [2/2]

void lsst::cpputils::python::WrapperCollection::addSignatureDependency ( std::string const & name)
inline

Indicate an external module that provides a type used in function/method signatures.

Dependencies that provide classes are imported after types in the current module are declared and before any methods and free functions in the current module are declared.

Parameters
[in]nameName of the module to import (absolute).

Definition at line 357 of file python.h.

357 {
358 _dependencies.push_back(name);
359 }

◆ collectSubmodule() [1/2]

void lsst::cpputils::python::WrapperCollection::collectSubmodule ( WrapperCollection && submodule)
inline

Merge deferred definitions in the given submodule into the parent WrapperCollection.

Parameters
submoduleA WrapperCollection created by makeSubmodule. Will be consumed (and must be an rvalue).

Definition at line 329 of file python.h.

329 {
330 _dependencies.splice(_dependencies.end(), submodule._dependencies);
331 _definitions.splice(_definitions.end(), submodule._definitions);
332 }
T end(T... args)
T splice(T... args)

◆ collectSubmodule() [2/2]

void lsst::cpputils::python::WrapperCollection::collectSubmodule ( WrapperCollection && submodule)
inline

Merge deferred definitions in the given submodule into the parent WrapperCollection.

Parameters
submoduleA WrapperCollection created by makeSubmodule. Will be consumed (and must be an rvalue).

Definition at line 329 of file python.h.

329 {
330 _dependencies.splice(_dependencies.end(), submodule._dependencies);
331 _definitions.splice(_definitions.end(), submodule._definitions);
332 }

◆ finish() [1/2]

void lsst::cpputils::python::WrapperCollection::finish ( )
inline

Invoke all deferred wrapper-declaring callables.

finish() should be called exactly once, at or near the end of a PYBIND11_MODULE block.

Definition at line 435 of file python.h.

435 {
436 for (auto dep = _dependencies.begin(); dep != _dependencies.end(); dep = _dependencies.erase(dep)) {
437 pybind11::module::import(dep->c_str());
438 }
439 for (auto def = _definitions.begin(); def != _definitions.end(); def = _definitions.erase(def)) {
440 (def->second)(def->first); // WrapperCallback(module)
441 }
442 }
T begin(T... args)
T erase(T... args)

◆ finish() [2/2]

void lsst::cpputils::python::WrapperCollection::finish ( )
inline

Invoke all deferred wrapper-declaring callables.

finish() should be called exactly once, at or near the end of a PYBIND11_MODULE block.

Definition at line 435 of file python.h.

435 {
436 for (auto dep = _dependencies.begin(); dep != _dependencies.end(); dep = _dependencies.erase(dep)) {
437 pybind11::module::import(dep->c_str());
438 }
439 for (auto def = _definitions.begin(); def != _definitions.end(); def = _definitions.erase(def)) {
440 (def->second)(def->first); // WrapperCallback(module)
441 }
442 }

◆ makeSubmodule() [1/2]

WrapperCollection lsst::cpputils::python::WrapperCollection::makeSubmodule ( std::string const & name)
inline

Create a WrapperCollection for a submodule defined in the same binary.

WrapperCollections created with makeSubmodule should generally be destroyed by moving them into a call to collectSubmodule; this will cause all deferred definitions to be executed when the parent WrapperCollection's finish() method is called.

Parameters
nameRelative name of the submodule.

Attributes added to the returned WrapperCollection will actually be put in a submodule that adds an underscore prefix to name, with __module__ set with the expectation that they will be lifted into a package without that leading underscore by a line in __init__.py like:

from ._package import _submodule as submodule

This is necessary to make importing _package possible when submodule already exists as a normal (i.e. directory-based) package. Of course, in that case, you'd instead use a submodule/__init__.py with a line like:

from .._package._submodule import *
Returns
a new WrapperCollection instance that sets the __module__ of any classes added to it to {package}.{name}.

Definition at line 318 of file python.h.

318 {
319 return WrapperCollection(module.def_submodule(("_" + name).c_str()), _package + "." + name);
320 }
WrapperCollection(pybind11::module module_, std::string const &package)
Construct a new WrapperCollection.
Definition python.h:264

◆ makeSubmodule() [2/2]

WrapperCollection lsst::cpputils::python::WrapperCollection::makeSubmodule ( std::string const & name)
inline

Create a WrapperCollection for a submodule defined in the same binary.

WrapperCollections created with makeSubmodule should generally be destroyed by moving them into a call to collectSubmodule; this will cause all deferred definitions to be executed when the parent WrapperCollection's finish() method is called.

Parameters
nameRelative name of the submodule.

Attributes added to the returned WrapperCollection will actually be put in a submodule that adds an underscore prefix to name, with __module__ set with the expectation that they will be lifted into a package without that leading underscore by a line in __init__.py like:

from ._package import _submodule as submodule

This is necessary to make importing _package possible when submodule already exists as a normal (i.e. directory-based) package. Of course, in that case, you'd instead use a submodule/__init__.py with a line like:

from .._package._submodule import *
Returns
a new WrapperCollection instance that sets the __module__ of any classes added to it to {package}.{name}.

Definition at line 318 of file python.h.

318 {
319 return WrapperCollection(module.def_submodule(("_" + name).c_str()), _package + "." + name);
320 }

◆ operator=() [1/4]

WrapperCollection & lsst::cpputils::python::WrapperCollection::operator= ( WrapperCollection && )
delete

◆ operator=() [2/4]

WrapperCollection & lsst::cpputils::python::WrapperCollection::operator= ( WrapperCollection && )
delete

◆ operator=() [3/4]

WrapperCollection & lsst::cpputils::python::WrapperCollection::operator= ( WrapperCollection const & )
delete

◆ operator=() [4/4]

WrapperCollection & lsst::cpputils::python::WrapperCollection::operator= ( WrapperCollection const & )
delete

◆ wrap() [1/2]

void lsst::cpputils::python::WrapperCollection::wrap ( WrapperCallback function)
inline

Add a set of wrappers without defining a class.

Parameters
[in]functionA callable object that takes a single pybind11::module argument (by reference) and adds pybind11 wrappers to it, to be called later by finish().

Definition at line 369 of file python.h.

369 {
370 _definitions.emplace_back(std::make_pair(module, function));
371 }
T make_pair(T... args)

◆ wrap() [2/2]

void lsst::cpputils::python::WrapperCollection::wrap ( WrapperCallback function)
inline

Add a set of wrappers without defining a class.

Parameters
[in]functionA callable object that takes a single pybind11::module argument (by reference) and adds pybind11 wrappers to it, to be called later by finish().

Definition at line 369 of file python.h.

369 {
370 _definitions.emplace_back(std::make_pair(module, function));
371 }

◆ wrapException() [1/2]

template<typename CxxException , typename CxxBase >
auto lsst::cpputils::python::WrapperCollection::wrapException ( std::string const & pyName,
std::string const & pyBase,
bool setModuleName = true )
inline

Wrap a C++ exception as a Python exception.

Template Parameters
CxxExceptionC++ Exception type to wrap.
CxxBaseBase class of CxxException.
Parameters
[in]pyNamePython name of the new exception.
[in]pyBasePython name of the pex::exceptions Exception type the new exception inherits from.
[in]setModuleNameIf true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.
Returns
a pybind11::class_ instance (template parameters unspecified) representing the Python type of the new exception.

Definition at line 421 of file python.h.

421 {
422 auto cls = pex::exceptions::python::declareException<CxxException, CxxBase>(module, pyName, pyBase);
423 if (setModuleName) {
424 cls.attr("__module__") = _package;
425 }
426 return cls;
427 }

◆ wrapException() [2/2]

template<typename CxxException , typename CxxBase >
auto lsst::cpputils::python::WrapperCollection::wrapException ( std::string const & pyName,
std::string const & pyBase,
bool setModuleName = true )
inline

Wrap a C++ exception as a Python exception.

Template Parameters
CxxExceptionC++ Exception type to wrap.
CxxBaseBase class of CxxException.
Parameters
[in]pyNamePython name of the new exception.
[in]pyBasePython name of the pex::exceptions Exception type the new exception inherits from.
[in]setModuleNameIf true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.
Returns
a pybind11::class_ instance (template parameters unspecified) representing the Python type of the new exception.

Definition at line 421 of file python.h.

421 {
422 auto cls = pex::exceptions::python::declareException<CxxException, CxxBase>(module, pyName, pyBase);
423 if (setModuleName) {
424 cls.attr("__module__") = _package;
425 }
426 return cls;
427 }

◆ wrapType() [1/2]

template<typename PyType , typename ClassWrapperCallback >
PyType lsst::cpputils::python::WrapperCollection::wrapType ( PyType cls,
ClassWrapperCallback function,
bool setModuleName = true )
inline

Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called.

Parameters
[in]clsA pybind11::class_ or enum_ instance.
[in]functionA callable object that takes a pybind11::module argument and a pybind11::class_ (or enum_) argument (both by reference) and defines wrappers for the class's methods and other attributes. Will be called with this->module and cls by finish().
[in]setModuleNameIf true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.
Returns
the cls argument for convenience
Examples
imageStatistics.cc.

Definition at line 391 of file python.h.

391 {
392 if (setModuleName) {
393 cls.attr("__module__") = _package;
394 }
395 // lambda below is mutable so it can modify the captured `cls` variable
396 wrap(
397 [cls=cls, function=std::move(function)] (pybind11::module & mod) mutable -> void {
398 function(mod, cls);
399 }
400 );
401 return cls;
402 }
void wrap(WrapperCallback function)
Add a set of wrappers without defining a class.
Definition python.h:369

◆ wrapType() [2/2]

template<typename PyType , typename ClassWrapperCallback >
PyType lsst::cpputils::python::WrapperCollection::wrapType ( PyType cls,
ClassWrapperCallback function,
bool setModuleName = true )
inline

Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called.

Parameters
[in]clsA pybind11::class_ or enum_ instance.
[in]functionA callable object that takes a pybind11::module argument and a pybind11::class_ (or enum_) argument (both by reference) and defines wrappers for the class's methods and other attributes. Will be called with this->module and cls by finish().
[in]setModuleNameIf true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.
Returns
the cls argument for convenience

Definition at line 391 of file python.h.

391 {
392 if (setModuleName) {
393 cls.attr("__module__") = _package;
394 }
395 // lambda below is mutable so it can modify the captured `cls` variable
396 wrap(
397 [cls=cls, function=std::move(function)] (pybind11::module & mod) mutable -> void {
398 function(mod, cls);
399 }
400 );
401 return cls;
402 }

The documentation for this class was generated from the following files: