lsst::utils::python::WrapperCollection Class Reference

A helper class for subdividing pybind11 module across multiple translation units (i.e. More...

#include <python.h>

Public Types
using	WrapperCallback = std::function< void(pybind11::module &)>
	Function handle type used to hold deferred wrapper declaration functions. More...

Public Member Functions
	WrapperCollection (pybind11::module module_, std::string const &package)
	Construct a new WrapperCollection. More...
	WrapperCollection (WrapperCollection &&other) noexcept
	WrapperCollection (WrapperCollection const &)=delete
WrapperCollection &	operator= (WrapperCollection const &)=delete
WrapperCollection &	operator= (WrapperCollection &&)=delete
	~WrapperCollection () noexcept
WrapperCollection	makeSubmodule (std::string const &name)
	Create a WrapperCollection for a submodule defined in the same binary. More...
void	collectSubmodule (WrapperCollection &&submodule)
	Merge deferred definitions in the given submodule into the parent WrapperCollection. More...
void	addInheritanceDependency (std::string const &name)
	Indicate an external module that provides a base class for a subsequent addType call. More...
void	addSignatureDependency (std::string const &name)
	Indicate an external module that provides a type used in function/method signatures. More...
void	wrap (WrapperCallback function)
	Add a set of wrappers without defining a class. More...
template<typename PyType , typename ClassWrapperCallback >
PyType	wrapType (PyType cls, ClassWrapperCallback function, bool setModuleName=true)
	Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called. More...
template<typename CxxException , typename CxxBase >
auto	wrapException (std::string const &pyName, std::string const &pyBase, bool setModuleName=true)
	Wrap a C++ exception as a Python exception. More...
void	finish ()
	Invoke all deferred wrapper-declaring callables. More...

Public Attributes
pybind11::module	module
	The module object passed to the PYBIND11_MODULE block that contains this WrapperCollection. More...

Detailed Description

A helper class for subdividing pybind11 module across multiple translation units (i.e.

source files).

Merging wrappers for different classes into a single compiled module can dramatically decrease the total size of the binary, but putting the source for multiple wrappers into a single file slows down incremental rebuilds and makes editing unwieldy. The right approach is to define wrappers in different source files and link them into a single module at build time. In simple cases, that's quite straightforward: pybind11 declarations are just regular C++ statements, and you can factor them out into different functions in different source files.

That approach doesn't work so well when the classes being wrapped are interdependent, because bindings are only guaranteed to work when all types used in a wrapped method signature have been declared to pybind11 before the method using them is itself declared. Naively, then, each source file would thus have to have multiple wrapper-declaring functions, so all type-wrapping functions could be executed before any method-wrapping functions. Of course, each type-wrapping function would also have to pass its type object to at least one method-wrapping function (to wrap the types own methods), and the result is a tangled mess of wrapper-declaring functions that obfuscate the code with a lot of boilerplate.

WrapperCollection provides a way out of that by allowing type wrappers and their associated methods to be declared at a single point, but the method wrappers wrapped in a lambda to defer their execution. A single WrapperCollection instance is typically constructed at the beginning of a PYBIND11_MODULE block, then passed by reference to wrapper-declaring functions defined in other source files. As type and method wrappers are added to the WrapperCollection by those functions, the types are registered immediately, and the method-wrapping lambdas are collected. After all wrapper-declaring functions have been called, finish() is called at the end of the PYBIND11_MODULE block to execute the collecting method-wrapping lambdas.

Typical usage:

// _mypackage.cc
 
void wrapClassA(WrapperCollection & wrappers);
void wrapClassB(WrapperCollection & wrappers);
 
PYBIND11_MODULE(_mypackage, module) {
    WrapperCollection wrappers(module, "mypackage");
    wrapClassA(wrappers);
    wrapClassB(wrappers);
    wrappers.finish();
}

// _ClassA.cc
 
void wrapClassA(WrapperCollection & wrappers) {
    wrappers.wrapType(
        py::class_<ClassA>(wrappers.module, "ClassA"),
        [](auto & mod, auto & cls) {
            cls.def("methodOnClassA", &methodOnClassA);
        }
    );
}

// _ClassB.cc
 
void wrapClassB(WrapperCollection & wrappers) {
    wrappers.wrapType(
        py::class_<ClassB>(wrappers.module, "ClassB"),
        [](auto & mod, auto & cls) {
            cls.def("methodOnClassB", &methodOnClassB);
            mod.def("freeFunction", &freeFunction);
        }
    );
}

Note that we recommend the use of universal lambdas (i.e. auto & parameters) to reduce verbosity.

Definition at line 242 of file python.h.

Member Typedef Documentation

WrapperCallback

using lsst::utils::python::WrapperCollection::WrapperCallback = std::function<void(pybind11::module &)>

Function handle type used to hold deferred wrapper declaration functions.

Definition at line 247 of file python.h.

Constructor & Destructor Documentation

WrapperCollection() [1/3]

lsst::utils::python::WrapperCollection::WrapperCollection ( pybind11::module  module_, std::string const &  package  )

inlineexplicit

Construct a new WrapperCollection.

A WrapperCollection should be constructed at or near the top of a PYBIND11_MODULE block.

Parameters

[in]	module_	Module instance passed to the PYBIND11_MODULE macro.
[in]	package	String name of the package all wrapped classes should appear to be from (by resetting their __module__ attribute). Note that this can lead to problems if classes are not also lifted into the package namespace in its __init__.py (in addition to confusing users, this will prevent unpickling from working).

Definition at line 264 of file python.h.

                                                                                  :
        module(module_),
        _package(package)
    {}

WrapperCollection() [2/3]

lsst::utils::python::WrapperCollection::WrapperCollection ( WrapperCollection &&  other )

inlinenoexcept

Definition at line 270 of file python.h.

                                                           :
        module(std::move(other.module)),
        _package(std::move(other._package)),
        _dependencies(std::move(other._dependencies)),
        _definitions(std::move(other._definitions))
    {}

WrapperCollection() [3/3]

lsst::utils::python::WrapperCollection::WrapperCollection ( WrapperCollection const &  )

delete

~WrapperCollection()

lsst::utils::python::WrapperCollection::~WrapperCollection ( )

inlinenoexcept

Definition at line 282 of file python.h.

                                  {
        if (std::uncaught_exceptions()==0 && !_definitions.empty()) {
            PyErr_SetString(PyExc_ImportError,
                            "WrapperCollection::finish() not called; module definition incomplete.");
            PyErr_WriteUnraisable(module.ptr());
        }
    }

Member Function Documentation

addInheritanceDependency()

void lsst::utils::python::WrapperCollection::addInheritanceDependency ( std::string const &  name )

inline

Indicate an external module that provides a base class for a subsequent addType call.

Dependencies that provide base classes cannot be deferred until after types are declared, and are always imported immediately.

Parameters

[in]

name

Name of the module to import (absolute).

Definition at line 343 of file python.h.

                                                          {
        pybind11::module::import(name.c_str());
    }

addSignatureDependency()

void lsst::utils::python::WrapperCollection::addSignatureDependency ( std::string const &  name )

inline

Indicate an external module that provides a type used in function/method signatures.

Dependencies that provide classes are imported after types in the current module are declared and before any methods and free functions in the current module are declared.

Parameters

[in]

name

Name of the module to import (absolute).

Definition at line 357 of file python.h.

                                                        {
        _dependencies.push_back(name);
    }

collectSubmodule()

void lsst::utils::python::WrapperCollection::collectSubmodule ( WrapperCollection &&  submodule )

inline

Merge deferred definitions in the given submodule into the parent WrapperCollection.

Parameters

submodule

A WrapperCollection created by makeSubmodule. Will be consumed (and must be an rvalue).

Definition at line 329 of file python.h.

                                                          {
        _dependencies.splice(_dependencies.end(), submodule._dependencies);
        _definitions.splice(_definitions.end(), submodule._definitions);
    }

finish()

void lsst::utils::python::WrapperCollection::finish ( )

inline

Invoke all deferred wrapper-declaring callables.

finish() should be called exactly once, at or near the end of a PYBIND11_MODULE block.

Definition at line 435 of file python.h.

                  {
        for (auto dep = _dependencies.begin(); dep != _dependencies.end(); dep = _dependencies.erase(dep)) {
            pybind11::module::import(dep->c_str());
        }
        for (auto def = _definitions.begin(); def != _definitions.end(); def = _definitions.erase(def)) {
            (def->second)(def->first);  // WrapperCallback(module)
        }
    }

makeSubmodule()

WrapperCollection lsst::utils::python::WrapperCollection::makeSubmodule ( std::string const &  name )

inline

Create a WrapperCollection for a submodule defined in the same binary.

WrapperCollections created with makeSubmodule should generally be destroyed by moving them into a call to collectSubmodule; this will cause all deferred definitions to be executed when the parent WrapperCollection's finish() method is called.

Parameters

name	Relative name of the submodule.

Attributes added to the returned WrapperCollection will actually be put in a submodule that adds an underscore prefix to name, with __module__ set with the expectation that they will be lifted into a package without that leading underscore by a line in __init__.py like:

from ._package import _submodule as submodule

This is necessary to make importing _package possible when submodule already exists as a normal (i.e. directory-based) package. Of course, in that case, you'd instead use a submodule/__init__.py with a line like:

from .._package._submodule import *

Returns

a new WrapperCollection instance that sets the __module__ of any classes added to it to {package}.{name}.

Definition at line 318 of file python.h.

                                                            {
        return WrapperCollection(module.def_submodule(("_" + name).c_str()), _package + "." + name);
    }

operator=() [1/2]

WrapperCollection& lsst::utils::python::WrapperCollection::operator= ( WrapperCollection &&  )

delete

operator=() [2/2]

WrapperCollection& lsst::utils::python::WrapperCollection::operator= ( WrapperCollection const &  )

delete

wrap()

void lsst::utils::python::WrapperCollection::wrap ( WrapperCallback  function )

inline

Add a set of wrappers without defining a class.

Parameters

[in]

function

A callable object that takes a single pybind11::module argument (by reference) and adds pybind11 wrappers to it, to be called later by finish().

Definition at line 369 of file python.h.

                                        {
        _definitions.emplace_back(std::make_pair(module, function));
    }

wrapException()

template<typename CxxException , typename CxxBase >

auto lsst::utils::python::WrapperCollection::wrapException ( std::string const &  pyName, std::string const &  pyBase, bool  setModuleName = true  )

inline

Wrap a C++ exception as a Python exception.

Template Parameters

CxxException	C++ Exception type to wrap.
CxxBase	Base class of CxxException.

Parameters

[in]	pyName	Python name of the new exception.
[in]	pyBase	Python name of the pex::exceptions Exception type the new exception inherits from.
[in]	setModuleName	If true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.

Returns

a pybind11::class_ instance (template parameters unspecified) representing the Python type of the new exception.

Definition at line 421 of file python.h.

                                                                                                    {
        auto cls = pex::exceptions::python::declareException<CxxException, CxxBase>(module, pyName, pyBase);
        if (setModuleName) {
            cls.attr("__module__") = _package;
        }
        return cls;
    }

wrapType()

template<typename PyType , typename ClassWrapperCallback >

PyType lsst::utils::python::WrapperCollection::wrapType ( PyType  cls, ClassWrapperCallback  function, bool  setModuleName = true  )

inline

Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called.

Parameters

[in]	cls	A pybind11::class_ or enum_ instance.
[in]	function	A callable object that takes a pybind11::module argument and a pybind11::class_ (or enum_) argument (both by reference) and defines wrappers for the class's methods and other attributes. Will be called with this->module and cls by finish().
[in]	setModuleName	If true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.

Returns

the cls argument for convenience

Definition at line 391 of file python.h.

                                                                                        {
        if (setModuleName) {
            cls.attr("__module__") = _package;
        }
        // lambda below is mutable so it can modify the captured `cls` variable
        wrap(
            [cls=cls, function=std::move(function)] (pybind11::module & mod) mutable -> void {
                function(mod, cls);
            }
        );
        return cls;
    }

Member Data Documentation

module

pybind11::module lsst::utils::python::WrapperCollection::module

The module object passed to the PYBIND11_MODULE block that contains this WrapperCollection.

Definition at line 448 of file python.h.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-0.7.0/Linux64/utils/22.0.1-4-g5b7b756+c1283a92b8/include/lsst/utils/python.h