lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT > Class Template Reference

A manager for a set of chunks of a single type. More...

#include <ChunkManagerImpl.h>

Inheritance diagram for lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >:

Public Types
typedef SubManager< MutexT, DataT, TraitsT >	Manager
typedef Manager::Chunk	Chunk

Public Member Functions
	ChunkManagerImpl ()
bool	isVisitInFlight (int const visitId)
void	failVisit (int const visitId)
void	registerVisit (int const visitId)
void	startVisit (std::vector< Chunk > &toRead, std::vector< Chunk > &toWaitFor, int const visitId, std::vector< int > const &chunkIds)
void	waitForOwnership (std::vector< Chunk > &toRead, std::vector< Chunk > &toWaitFor, int const visitId, TimeSpec const &deadline)
void	getChunks (std::vector< Chunk > &chunks, std::vector< int > const &chunkIds)
bool	endVisit (int const visitId, bool const rollback)
void	printVisits (std::ostream &os) const
void	printChunks (std::ostream &os) const
void	printVisit (int const visitId, std::ostream &os) const
void	printChunk (int const chunkId, std::ostream &os) const

Static Public Member Functions
static std::size_t	size ()

Private Member Functions
void	rollbackAllExcept (int const visitId)

Static Private Member Functions
static std::size_t	blocks ()
	Returns the offset of the first data block (relative to its manager). More...

Private Attributes
MutexT	_mutex
Condition< MutexT >	_ownerCondition
VisitTracker	_visits
Manager	_data

Detailed Description

template<typename MutexT, typename DataT, typename TraitsT = DataTraits<DataT>>
class lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >

A manager for a set of chunks of a single type.

Each instance is intended to be a header of (meaning: located in the initial bytes of) a large contiguous memory block M, which it then takes charge of managing. In particular, M will contain all chunk descriptors and data, as well as all data structures required for management tasks (visit tracking, memory allocation, and synchronization). M is initialized by calling the placement new operator with the address of M as a parameter. To avoid any data alignment issues, M should begin at an address that is a multiple of 16 bytes (or some larger power of 2).

Finally, note that instances of this class do not contain a single pointer - when necessary, offsets (relative to some known address, e.g. of the manager instance itself) are stored instead. This, in conjunction with an appropriate choice of mutex type, makes the class suitable for placement into shared memory.

Definition at line 292 of file ChunkManagerImpl.h.

Member Typedef Documentation

template<typename MutexT , typename DataT , typename TraitsT = DataTraits<DataT>>

typedef Manager::Chunk lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::Chunk

Definition at line 295 of file ChunkManagerImpl.h.

template<typename MutexT , typename DataT , typename TraitsT = DataTraits<DataT>>

typedef SubManager<MutexT, DataT, TraitsT> lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::Manager

Definition at line 294 of file ChunkManagerImpl.h.

Constructor & Destructor Documentation

template<typename MutexT , typename DataT , typename TraitsT >

lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::ChunkManagerImpl

(

)

Definition at line 702 of file ChunkManagerImpl.cc.

                                                           :
    _data(reinterpret_cast<unsigned char *>(this), blocks())
{}

Member Function Documentation

template<typename MutexT , typename DataT , typename TraitsT = DataTraits<DataT>>

static std::size_t lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::blocks ( )

inlinestaticprivate

Returns the offset of the first data block (relative to its manager).

Definition at line 299 of file ChunkManagerImpl.h.

                              {
        return (sizeof(ChunkManagerImpl) + 511) & ~static_cast<size_t>(511);
    }

template<typename MutexT , typename DataT , typename TraitsT >

bool lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::endVisit	(	int const	visitId,
		bool const	rollback
	)

Relinquishes ownership of any chunks owned by the given visit (each chunk is passed on to its first interested party that is still in flight) and removes the given visit from the list of in-flight visits.

Parameters

[in]	visitId	The visit to remove from the set of in-flight visits being tracked
[in]	rollback	Flag indicating whether chunks should be rolled back (to the state they were in before being acquired by the given visit), or whether changes should be marked as committed.

Returns

true if the visit existed, was not marked as a failure and was committed, false otherwise.

Definition at line 879 of file ChunkManagerImpl.cc.

  {
    ScopedLock<MutexT> lock(_mutex);
    bool roll = rollback || !_visits.isValid(visitId);
    if (!_visits.erase(visitId)) {
        return false;
    }
    // relinquish chunk ownership: if any chunks change hands, notify all threads
    // waiting on chunk ownership to check whether they can proceed
    if (_data.relinquishOwnership(visitId, roll, _visits)) {
        _ownerCondition.notifyAll();
    }
    return !roll;
}

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::failVisit

(

int const

visitId

)

Marks the given visit a failure. If the given visit has not been previously registered, or has already been marked as failed, then the call has no effect.

Definition at line 720 of file ChunkManagerImpl.cc.

                                                                          {
    ScopedLock<MutexT> lock(_mutex);
    Visit * v = _visits.find(visitId);
    if (v != 0) {
        v->setFailed();
    }
}

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::getChunks	(	std::vector< Chunk > &	chunks,
		std::vector< int > const &	chunkIds
	)

Returns a chunk for each identifier in the given list that corresponds to a managed chunk.

Parameters

[out]	chunks	The list to store chunks in.
[in]	chunkIds	The list of identifiers for which corresponding chunks are desired.

Definition at line 856 of file ChunkManagerImpl.cc.

  {
    ScopedLock<MutexT> lock(_mutex);
    _data.getChunks(chunks, chunkIds);
}

template<typename MutexT , typename DataT , typename TraitsT >

bool lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::isVisitInFlight

(

int const

visitId

)

Returns true if the given visit is in-flight and has not been marked as failed.

Definition at line 709 of file ChunkManagerImpl.cc.

                                                                                {
    ScopedLock<MutexT> lock(_mutex);
    return _visits.isValid(visitId);
}

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::printChunk	(	int const	chunkId,
		std::ostream &	os
	)		const

Definition at line 939 of file ChunkManagerImpl.cc.

                                                                                                   {
    ScopedLock<MutexT> lock(_mutex);
    _data.print(chunkId, os);
}

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::printChunks

(

std::ostream &

os

)

const

Definition at line 924 of file ChunkManagerImpl.cc.

                                                                                {
    ScopedLock<MutexT> lock(_mutex);
    _data.print(os);
}

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::printVisit	(	int const	visitId,
		std::ostream &	os
	)		const

Definition at line 931 of file ChunkManagerImpl.cc.

                                                                                                   {
    ScopedLock<MutexT> lock(_mutex);
    _visits.print(visitId, os);
    _data.printVisit(visitId, os);
}

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::printVisits

(

std::ostream &

os

)

const

Definition at line 917 of file ChunkManagerImpl.cc.

                                                                                {
    ScopedLock<MutexT> lock(_mutex);
    _visits.print(os);
}

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::registerVisit

(

int const

visitId

)

Registers the given visit as in-flight without performing any further action.

Exceptions

lsst::pex::exceptions::InvalidParameterError	Thrown if the given visit is already in-flight.
lsst::pex::exceptions::LengthError	Thrown if too-many visits are currently in-flight.

Definition at line 738 of file ChunkManagerImpl.cc.

                                                                              {
    ScopedLock<MutexT> lock(_mutex);
    if (_visits.find(visitId) != 0) {
        throw LSST_EXCEPT(lsst::pex::exceptions::InvalidParameterError,
            (boost::format("Cannot start processing visit %1%: visit is already in flight") % visitId).str());
    }
    if (_visits.space() == 0) {
        throw LSST_EXCEPT(lsst::pex::exceptions::LengthError,
            (boost::format("Cannot register visit %1%: too many visits in-flight") % visitId).str());
    }
    Visit * v = _visits.insert(visitId);
    assert(v != 0);
}

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::rollbackAllExcept ( int const  visitId )

private

Definition at line 905 of file ChunkManagerImpl.cc.

                                                                                  {
    for (Visit const * v = _visits.begin(); v != _visits.end(); ++v) {
        int id = v->getId();
        if (id >= 0 && id != visitId) {
            _visits.erase(id);
            _data.relinquishOwnership(id, true, _visits);
        }
    }
}

template<typename MutexT , typename DataT , typename TraitsT = DataTraits<DataT>>

static std::size_t lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::size ( )

inlinestatic

Returns the total number of bytes required for a ChunkManagerImpl instance and it's associated pool of memory blocks.

Definition at line 308 of file ChunkManagerImpl.h.

                            {
        return blocks() + Chunk::BLOCK_SIZE * TraitsT::NUM_BLOCKS;
    }

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::startVisit	(	std::vector< Chunk > &	toRead,
		std::vector< Chunk > &	toWaitFor,
		int const	visitId,
		std::vector< int > const &	chunkIds
	)

Begins visit processing by registering the given visit as an interested party of each chunk with identifier in the given list. If any identifier in the list does not have a corresponding chunk, a new chunk (owned by the specified visit) is created.

Note that the toWaitFor and toRead output vectors are cleared immediately on entry to the function. Under the assumption that these vectors are empty to begin with, strong exception safety is guaranteed.

Parameters

[out]	toRead	Set to the list of newly created chunks that must be read from disk.
[out]	toWaitFor	Set to the list of chunks that are already in memory and must be waited on.
[in]	visitId	The visit to begin.
[in]	chunkIds	Identifiers for chunks to register an interest in or create.

Exceptions

lsst::pex::exceptions::InvalidParameterError	Thrown if the given visit is not currently in-flight.
lsst::pex::exceptions::LengthError	Thrown if the chunk manager does not have space to track all the chunks for the visit.

Definition at line 773 of file ChunkManagerImpl.cc.

  {
    toRead.clear();
    toWaitFor.clear();

    // ensure external resources necessary for success are available
    toRead.reserve(chunkIds.size());
    toWaitFor.reserve(chunkIds.size());

    ScopedLock<MutexT> lock(_mutex);
    // ensure internal resources necessary for success are available
    if (_data.space() < static_cast<int>(chunkIds.size())) {
        throw LSST_EXCEPT(lsst::pex::exceptions::LengthError,
                          "requested additional chunks exceed chunk manager capacity");
    }
    if (!_visits.isValid(visitId)) {
        throw LSST_EXCEPT(lsst::pex::exceptions::InvalidParameterError,
            (boost::format("Cannot start processing for visit %1%: visit is not in-flight") % visitId).str());
    }
    // having pre-allocated/checked that there is space for everything,
    // manager state can be modified without throwing
    _data.createOrRegisterInterest(toRead, toWaitFor, visitId, chunkIds);
}

template<typename MutexT , typename DataT , typename TraitsT >

void lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::waitForOwnership	(	std::vector< Chunk > &	toRead,
		std::vector< Chunk > &	toWaitFor,
		int const	visitId,
		TimeSpec const &	deadline
	)

Blocks the calling thread until the given visit owns every one of the given chunks.

Note that the vector toRead passed into the method is assumed to be empty - it is immediately cleared upon entry to the function.

Parameters

[out]	toRead	Set to the list of chunks acquired by the given visit, but which must be re-read from disk.
[in,out]	toWaitFor	The list of chunks that must be owned by the given visit before returning - acquired chunks are removed from the list, so that the list is empty on successfull return.
[in]	visitId	The visit that must wait for chunk ownership.
[in]	deadline	The point in time after which chunk acquisition should be abandoned.

Exceptions

lsst::pex::exceptions::TimeoutError

Thrown if the visit deadline expired while waiting to acquire chunks.

Definition at line 820 of file ChunkManagerImpl.cc.

  {
    toRead.clear();
    toRead.reserve(toWaitFor.size());

    ScopedLock<MutexT> lock(_mutex);
    while (true) {
        if (_data.checkForOwnership(toRead, toWaitFor, visitId)) {
            break; // all chunks belong to the visit - ok to proceed
        }
        // wait for ownership
        if (!_ownerCondition.wait(lock, deadline)) {
            // TODO: this is a short-term DC3a hack, necessary because there is no way for
            // the pipeline framework to communicate exceptions arising outside of the implementation
            // of AP (e.g. in an IOStage) back to the pipeline itself. This results in visits that
            // fail, but never relinquish ownership of their chunks. For now, take the draconian measure
            // of rolling back all visits except the current one. This needs to be re-thought for DC3b!
            rollbackAllExcept(visitId);
            throw LSST_EXCEPT(lsst::pex::exceptions::TimeoutError,
                (boost::format("Deadline for visit %1% expired") % visitId).str());
        }
    }
}

Member Data Documentation

template<typename MutexT , typename DataT , typename TraitsT = DataTraits<DataT>>

Manager lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::_data

private

Definition at line 350 of file ChunkManagerImpl.h.

template<typename MutexT , typename DataT , typename TraitsT = DataTraits<DataT>>

MutexT lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::_mutex

mutableprivate

Definition at line 347 of file ChunkManagerImpl.h.

template<typename MutexT , typename DataT , typename TraitsT = DataTraits<DataT>>

Condition<MutexT> lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::_ownerCondition

private

Definition at line 348 of file ChunkManagerImpl.h.

template<typename MutexT , typename DataT , typename TraitsT = DataTraits<DataT>>

VisitTracker lsst::ap::detail::ChunkManagerImpl< MutexT, DataT, TraitsT >::_visits

private

Definition at line 349 of file ChunkManagerImpl.h.

---

The documentation for this class was generated from the following files:

• /lsst/home/lsstsw/stack/Linux64/ap/master-g60f1e072bb+1/include/lsst/ap/ChunkManagerImpl.h
• /lsst/home/lsstsw/stack/Linux64/ap/master-g60f1e072bb+1/include/lsst/ap/ChunkManagerImpl.cc