Class for database storage with data loading from TSV files. More...

#include <DbTsvStorage.h>

Inheritance diagram for lsst::daf::persistence::DbTsvStorage:

Public Types
typedef std::shared_ptr< DbTsvStorage >	Ptr

typedef std::vector< Ptr >	List

enum	{ magicSentinel = 0xdeadbeef }

typedef unsigned long	memId
	Type of the block's ID. More...

typedef memId(*	memNewCallback) (const memId cid)
	A function used to register a callback. More...

typedef memId(*	memCallback) (const Citizen *ptr)

Public Member Functions
	DbTsvStorage (void)
	Constructor. More...

	~DbTsvStorage (void)
	Minimal destructor. More...

virtual void	setPolicy (lsst::pex::policy::Policy::Ptr policy)
	Allow a policy to be used to configure the DbTsvStorage. More...

virtual void	setPersistLocation (LogicalLocation const &location)
	Set the database location to persist to. More...

virtual void	setRetrieveLocation (LogicalLocation const &location)
	Set the database location to retrieve from. More...

virtual void	startTransaction (void)
	Start a transaction. More...

virtual void	endTransaction (void)
	End a transaction. More...

virtual void	createTableFromTemplate (std::string const &tableName, std::string const &templateName, bool mayAlreadyExist=false)
	Create a new table from an existing template table. More...

virtual void	dropTable (std::string const &tableName)
	Drop a table. More...

virtual void	truncateTable (std::string const &tableName)
	Truncate a table. More...

virtual void	setTableForInsert (std::string const &tableName)
	Set the table to insert rows into. More...

template<typename T >
void	setColumn (std::string const &columnName, T const &value)
	Set the value to insert in a given column. More...

virtual void	setColumnToNull (std::string const &columnName)
	Set a given column to NULL. More...

virtual void	insertRow (void)
	Insert the row. More...

template<typename T >
void	outParam (std::string const &columnName, T *location, bool isExpr=false)
	Request a column in the query output and bind a destination location. More...

template<typename T >
void	condParam (std::string const &paramName, T const &value)
	Bind a value to a WHERE condition parameter. More...

template<typename T >
T const &	getColumnByPos (int pos)
	Get the value of a column of the query result row by position. More...

template<>
void	setColumn (std::string const &columnName, char const &value)

template<>
void	setColumn (std::string const &columnName, double const &value)

template<>
void	setColumn (std::string const &columnName, float const &value)

template<>
void	setColumn (std::string const &columnName, DateTime const &value)

virtual void	executeSql (std::string const &sqlStatement)
	Execute an arbitrary SQL statement. More...

virtual void	setTableForQuery (std::string const &tableName, bool isExpr=false)
	Set the table to query (single-table queries only). More...

virtual void	setTableListForQuery (std::vector< std::string > const &tableNameList)
	Set a list of tables to query (multiple-table queries). More...

virtual void	outColumn (std::string const &columnName, bool isExpr=false)
	Request a column in the query output. More...

virtual void	orderBy (std::string const &expression)
	Request that the query output be sorted by an expression. More...

virtual void	groupBy (std::string const &expression)
	Request that the query output be grouped by an expression. More...

virtual void	setQueryWhere (std::string const &whereClause)
	Set the condition for the WHERE clause of the query. More...

virtual void	query (void)
	Execute the query. More...

virtual bool	next (void)
	Move to the next (first) row of the query result. More...

bool	columnIsNull (int pos)
	Determine if the value of a column is NULL. More...

virtual void	finishQuery (void)
	Indicate that query processing is finished. More...

std::string	repr () const
	Return a string representation of a Citizen. More...

void	markPersistent (void)
	Mark a Citizen as persistent and not destroyed until process end. More...

memId	getId () const
	Return the Citizen's ID. More...

Static Public Member Functions
static Ptr	createInstance (std::string const &name, LogicalLocation const &location, bool persist, lsst::pex::policy::Policy::Ptr policy)
	Create and configure a FormatterStorage subclass instance. More...

static bool	hasBeenCorrupted ()
	Check all allocated blocks for corruption. More...

static memId	getNextMemId ()
	Return the memId of the next object to be allocated. More...

static int	init ()
	Called once when the memory system is being initialised. More...

Census
Provide a list of current Citizens
static int	census (int, memId startingMemId=0)
	How many active Citizens are there? More...

static void	census (std::ostream &stream, memId startingMemId=0)
	Print a list of all active Citizens to stream, sorted by ID. More...

static const std::vector< const Citizen * > *	census ()
	Return a (newly allocated) std::vector of active Citizens sorted by ID. More...

callbackIDs
Set callback Ids. The old Id is returned
static memId	setNewCallbackId (memId id)
	Call the NewCallback when block is allocated. More...

static memId	setDeleteCallbackId (memId id)
	Call the current DeleteCallback when block is deleted. More...

callbacks
Set the New/Delete callback functions; in each case the previously installed callback is returned. These callback functions return a value which is Added to the previously registered id. The default callback functions are called default{New,Delete}Callback; you may want to set a break point in these callbacks from your favourite debugger
static memNewCallback	setNewCallback (memNewCallback func)
	Set the NewCallback function. More...

static memCallback	setDeleteCallback (memCallback func)
	Set the DeleteCallback function. More...

static memCallback	setCorruptionCallback (memCallback func)
	Set the CorruptionCallback function. More...

Protected Member Functions
void	verifyPathName (std::string const &pathName)
	Ensure that all directories along a path exist, creating them if necessary. More...

Private Member Functions
int	_getColumnIndex (std::string const &columnName)
	Get the index of a given column. More...

Private Attributes
bool	_persisting

bool	_saveTemp
	Do not delete temporary TSV file if true. More...

std::string	_tempPath
	Directory pathname for temporary TSV file. More...

char *	_fileName
	Full pathname for temporary TSV file. More...

std::string	_location
	Database location URL. More...

std::string	_tableName

std::map< std::string, int >	_colMap
	Map from column names to positions. More...

std::ostringstream	_convertStream
	Stream to convert to text. More...

std::vector< std::string >	_rowBuffer

std::unique_ptr< std::ofstream >	_osp
	Output TSV stream. More...

Detailed Description

Class for database storage with data loading from TSV files.

Subclass of DbStorage, overriding persistence methods.

Persists data to a database using TSV files as an intermediary for performance. Provides methods for writing rows to a table and retrieving rows from a query.

Definition at line 66 of file DbTsvStorage.h.

Member Typedef Documentation

◆ List

typedef std::vector<Ptr> lsst::daf::persistence::FormatterStorage::List

inherited

Definition at line 63 of file FormatterStorage.h.

◆ memCallback

typedef memId(* lsst::daf::base::Citizen::memCallback) (const Citizen *ptr)

inherited

Definition at line 59 of file Citizen.h.

◆ memId

typedef unsigned long lsst::daf::base::Citizen::memId

inherited

Type of the block's ID.

Definition at line 56 of file Citizen.h.

◆ memNewCallback

typedef memId(* lsst::daf::base::Citizen::memNewCallback) (const memId cid)

inherited

A function used to register a callback.

Definition at line 58 of file Citizen.h.

◆ Ptr

typedef std::shared_ptr<DbTsvStorage> lsst::daf::persistence::DbTsvStorage::Ptr

Definition at line 68 of file DbTsvStorage.h.

Member Enumeration Documentation

◆ anonymous enum

anonymous enum

inherited

Enumerator
magicSentinel

Definition at line 86 of file Citizen.h.

86 { magicSentinel = 0xdeadbeef };

lsst::daf::base::Citizen::magicSentinel

Definition: Citizen.h:86

Constructor & Destructor Documentation

◆ DbTsvStorage()

lsst::daf::persistence::DbTsvStorage::DbTsvStorage ( void )

Constructor.

Definition at line 67 of file DbTsvStorage.cc.

67 : _saveTemp(false) {

68 }

lsst::daf::persistence::DbTsvStorage::_saveTemp

bool _saveTemp

Do not delete temporary TSV file if true.

Definition: DbTsvStorage.h:101

◆ ~DbTsvStorage()

lsst::daf::persistence::DbTsvStorage::~DbTsvStorage ( void )

Minimal destructor.

Definition at line 72 of file DbTsvStorage.cc.

72 {

73 }

Member Function Documentation

◆ _getColumnIndex()

int lsst::daf::persistence::DbTsvStorage::_getColumnIndex ( std::string const & columnName )

private

Get the index of a given column.

Create a new entry in the row buffer if the column hasn't already been seen. May modify the row buffer, so do not call this inside "_rowBuffer[]".

Parameters

[in] columnName Name of the column

Returns: Index of the column in the row buffer

Definition at line 252 of file DbTsvStorage.cc.

                                                              {
     std::map<std::string, int>::iterator i = _colMap.find(columnName);
     if (i == _colMap.end()) {
         _colMap.insert(std::pair<std::string, int>(columnName,
                                                    _rowBuffer.size()));
         _rowBuffer.push_back(std::string());
         return _rowBuffer.size() - 1;
     }
     else {
         return i->second;
     }
 }

◆ census() [1/3]

int lsst::daf::base::Citizen::census	(	int	,
		memId	startingMemId = `0`
	)

staticinherited

How many active Citizens are there?

Parameters

startingMemId Don't print Citizens with lower IDs

Definition at line 267 of file Citizen.cc.

       {
     if (startingMemId == 0) {              // easy
         ReadGuard guard(citizenLock);
         return _activeCitizens.size();
     }
 
     int n = 0;
     ReadGuard guard(citizenLock);
     for (table::iterator cur = _activeCitizens.begin();
          cur != _activeCitizens.end(); cur++) {
         if (cur->first->_CitizenId >= startingMemId) {
             n++;
         }
     }
 
     return n;    
 }

◆ census() [2/3]

void lsst::daf::base::Citizen::census	(	std::ostream &	stream,
		memId	startingMemId = `0`
	)

staticinherited

Print a list of all active Citizens to stream, sorted by ID.

Parameters

stream	stream to print to
startingMemId	Don't print Citizens with lower IDs

Definition at line 290 of file Citizen.cc.

       {
     ReadGuard guard(citizenLock);
 
     std::unique_ptr<std::vector<Citizen const*> const> leaks(Citizen::census());
 
     for (std::vector<Citizen const *>::const_iterator citizen = leaks->begin(), end = leaks->end();
          citizen != end; ++citizen) {
         if ((*citizen)->getId() >= startingMemId) {
             stream << (*citizen)->repr() << "\n";
         }
     }
 }

◆ census() [3/3]

std::vector< dafBase::Citizen const * > const * lsst::daf::base::Citizen::census ( )

staticinherited

Return a (newly allocated) std::vector of active Citizens sorted by ID.

You are responsible for deleting it; or you can say std::unique_ptr<std::vector<Citizen const*> const> leaks(Citizen::census()); and not bother (that becomes std::unique_ptr in C++11)

Definition at line 322 of file Citizen.cc.

                                                                {
     std::vector<Citizen const*>* vec =
         new std::vector<Citizen const*>(0);
     ReadGuard guard(citizenLock);
     vec->reserve(_activeCitizens.size());
 
     for (table::iterator cur = _activeCitizens.begin();
          cur != _activeCitizens.end(); cur++) {
         vec->push_back(dynamic_cast<Citizen const*>(cur->first));
     }
         
     std::sort(vec->begin(), vec->end(), cmpId);
 
     return vec;
 }

◆ columnIsNull()

bool lsst::daf::persistence::DbStorage::columnIsNull ( int pos )

inherited

Determine if the value of a column is NULL.

Parameters

[in] pos Position of the column (starts at 0)

Definition at line 270 of file DbStorage.cc.

                                     {
     return _impl->columnIsNull(pos);
 }

◆ condParam()

template<typename T >

void lsst::daf::persistence::DbTsvStorage::condParam	(	std::string const &	paramName,
		T const &	value
	)

Bind a value to a WHERE condition parameter.

Parameters

[in]	paramName	Name of the parameter (prefixed by ":" in the WHERE clause)
[in]	value	Value to be bound to the parameter.

Definition at line 368 of file DbTsvStorage.cc.

                                                                        {
     DbStorage::condParam<T>(paramName, value);
 }

◆ createInstance()

FormatterStorage::Ptr lsst::daf::persistence::FormatterStorage::createInstance	(	std::string const &	name,
		LogicalLocation const &	location,
		bool	persist,
		lsst::pex::policy::Policy::Ptr	policy
	)

staticinherited

Create and configure a FormatterStorage subclass instance.

Parameters

[in]	name	Name of subclass
[in]	location	Location to persist to or retrieve from
[in]	persist	True if persisting, false if retrieving
[in]	policy	Policy used to configure the FormatterStorage

Returns: Shared pointer to FormatterStorage subclass instance

Definition at line 75 of file FormatterStorage.cc.

                                    {
     FormatterStorage::Ptr storage = StorageRegistry::getRegistry().createInstance(name);
     storage->setPolicy(policy);
     if (persist) {
         storage->setPersistLocation(location);
     }
     else {
         storage->setRetrieveLocation(location);
     }
     return storage;
 }

◆ createTableFromTemplate()

void lsst::daf::persistence::DbTsvStorage::createTableFromTemplate	(	std::string const &	tableName,
		std::string const &	templateName,
		bool	mayAlreadyExist = `false`
	)

virtual

Create a new table from an existing template table.

Parameters

[in]	tableName	Name of the new table
[in]	templateName	Name of the existing template table
[in]	mayAlreadyExist	False (default) if the table must not be present

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 183 of file DbTsvStorage.cc.

                                                               {
     if (_persisting) {
         DbStorage dbs;
         dbs.setPersistLocation(LogicalLocation(_location));
         dbs.startTransaction();
         dbs.createTableFromTemplate(tableName, templateName, mayAlreadyExist);
         dbs.endTransaction();
     }
     else {
         DbStorage::createTableFromTemplate(tableName, templateName,
                                            mayAlreadyExist);
     }
 }

◆ dropTable()

void lsst::daf::persistence::DbTsvStorage::dropTable ( std::string const & tableName )

virtual

Drop a table.

Parameters

[in] tableName Name of the table to drop

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 202 of file DbTsvStorage.cc.

                                                        {
     if (_persisting) {
         DbStorage dbs;
         dbs.setPersistLocation(LogicalLocation(_location));
         dbs.startTransaction();
         dbs.dropTable(tableName);
         dbs.endTransaction();
     }
     else {
         DbStorage::dropTable(tableName);
     }
 }

◆ endTransaction()

void lsst::daf::persistence::DbTsvStorage::endTransaction ( void )

virtual

End a transaction.

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 114 of file DbTsvStorage.cc.

                                       {
     if (!_persisting) {
         DbStorage::endTransaction();
         return;
     }
 
     // close stream
     _osp->close();
 
     MYSQL* db = mysql_init(0);
     if (db == 0) {
         throw LSST_EXCEPT(lsst::pex::exceptions::RuntimeError, "Unable to allocate MySQL connection");
     }
     DbStorageLocation dbLoc(_location);
     unsigned int port = strtoul(dbLoc.getPort().c_str(), 0, 10);
     if (mysql_real_connect(db,
                            dbLoc.getHostname().c_str(),
                            dbLoc.getUsername().c_str(),
                            dbLoc.getPassword().c_str(),
                            dbLoc.getDbName().c_str(),
                            port, 0,
                            CLIENT_COMPRESS | CLIENT_LOCAL_FILES) == 0) {
         throw LSST_EXCEPT(lsst::pex::exceptions::RuntimeError,
             "Unable to connect to MySQL database: " + _location);
     }
     if (mysql_options(db, MYSQL_OPT_LOCAL_INFILE, 0) != 0) {
         throw LSST_EXCEPT(lsst::pex::exceptions::RuntimeError,
             std::string("Unable to set LOCAL INFILE option - * ") +
             mysql_error(db));
     }
 
     std::string query = "LOAD DATA LOCAL INFILE";
     query += " '";
     query += _fileName;
     query += "'";
     query += " REPLACE";
     query += " INTO TABLE `" + _tableName;
     query += "` (";
     for (std::map<std::string, int>::const_iterator it = _colMap.begin();
          it != _colMap.end(); ++it) {
         _rowBuffer[it->second] = it->first;
     }
     for (std::vector<std::string>::const_iterator it = _rowBuffer.begin();
          it != _rowBuffer.end(); ++it) {
         if (it != _rowBuffer.begin()) query += " ,";
         query += *it;
     }
     query += ")";
 
     if (mysql_query(db, query.c_str()) != 0) {
         mysql_close(db);
         throw LSST_EXCEPT(lsst::pex::exceptions::RuntimeError,
             "Unable to load data into database table: " + _tableName +
             "- * " + mysql_error(db));
     }
     mysql_close(db);
 
     // unlink file
     if (!_saveTemp) {
         unlink(_fileName);
     }
     delete[] _fileName;
 }

◆ executeSql()

void lsst::daf::persistence::DbStorage::executeSql ( std::string const & sqlStatement )

virtualinherited

Execute an arbitrary SQL statement.

Use primarily to perform server-side computations or complex DDL.

Parameters

[in] sqlStatement SQL statement to be executed. Must not end in ";".

Definition at line 136 of file DbStorage.cc.

                                                         {
     _impl->executeSql(sqlStatement);
 }

◆ finishQuery()

void lsst::daf::persistence::DbStorage::finishQuery ( void )

virtualinherited

Indicate that query processing is finished.

Must be called after next() returns false; no getColumnByPos() or columnIsNull() calls may be made after this method is called.

Definition at line 278 of file DbStorage.cc.

                                 {
     _impl->finishQuery();
 }

◆ getColumnByPos()

template<typename T >

T const & lsst::daf::persistence::DbTsvStorage::getColumnByPos ( int pos )

Get the value of a column of the query result row by position.

Parameters

[in] pos Position of the column (starts at 0)

Definition at line 376 of file DbTsvStorage.cc.

                                              {
     return DbStorage::getColumnByPos<T>(pos);
 }

◆ getId()

dafBase::Citizen::memId lsst::daf::base::Citizen::getId ( ) const

inherited

Return the Citizen's ID.

Definition at line 224 of file Citizen.cc.

                                                 {
     return _CitizenId;
 }

◆ getNextMemId()

dafBase::Citizen::memId lsst::daf::base::Citizen::getNextMemId ( )

staticinherited

Return the memId of the next object to be allocated.

Definition at line 229 of file Citizen.cc.

                                                  {
     return _nextMemId();
 }

◆ groupBy()

void lsst::daf::persistence::DbStorage::groupBy ( std::string const & expression )

virtualinherited

Request that the query output be grouped by an expression.

Parameters

[in] expression Text of the SQL expression

Definition at line 233 of file DbStorage.cc.

                                                    {
     _impl->groupBy(expression);
 }

◆ hasBeenCorrupted()

bool lsst::daf::base::Citizen::hasBeenCorrupted ( )

staticinherited

Check all allocated blocks for corruption.

Definition at line 353 of file Citizen.cc.

                                       {
     ReadGuard guard(citizenLock);
     for (table::iterator cur = _activeCitizens.begin();
          cur != _activeCitizens.end(); cur++) {
         if (cur->first->_hasBeenCorrupted()) {
             return true;
         }
     }
     for (table::iterator cur = _persistentCitizens.begin();
          cur != _persistentCitizens.end(); cur++) {
         if (cur->first->_hasBeenCorrupted()) {
             return true;
         }
     }
 
     return false;
 }

◆ init()

int lsst::daf::base::Citizen::init ( )

staticinherited

Called once when the memory system is being initialised.

Definition at line 214 of file Citizen.cc.

                          {
     volatile int dummy = 1;
     return dummy;
 }

◆ insertRow()

void lsst::daf::persistence::DbTsvStorage::insertRow ( void )

virtual

Insert the row.

Row values must have been set with setColumn() calls.

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 334 of file DbTsvStorage.cc.

                                  {
     // Output row to stream
     for (std::vector<std::string>::const_iterator i = _rowBuffer.begin();
          i != _rowBuffer.end(); ++i) {
         if (i != _rowBuffer.begin()) *_osp << '\t';
         *_osp << *i;
     }
     // \todo Optimization: if columns all outputted, just put out endl
     *_osp << std::endl;
 }

◆ markPersistent()

void lsst::daf::base::Citizen::markPersistent ( void )

inherited

Mark a Citizen as persistent and not destroyed until process end.

Definition at line 254 of file Citizen.cc.

                                         {
     WriteGuard guard(citizenLock);
     _persistentCitizens[this] = _activeCitizens[this];
     _activeCitizens.erase(this);
 }

◆ next()

bool lsst::daf::persistence::DbStorage::next ( void )

virtualinherited

Move to the next (first) row of the query result.

Returns: false if no more rows

Definition at line 255 of file DbStorage.cc.

                          {
     return _impl->next();
 }

◆ orderBy()

void lsst::daf::persistence::DbStorage::orderBy ( std::string const & expression )

virtualinherited

Request that the query output be sorted by an expression.

Multiple expressions may be specified, in order.

Parameters

[in] expression Text of the SQL expression

Definition at line 226 of file DbStorage.cc.

                                                    {
     _impl->orderBy(expression);
 }

◆ outColumn()

void lsst::daf::persistence::DbStorage::outColumn	(	std::string const &	columnName,
		bool	isExpr = `false`
	)

virtualinherited

Request a column in the query output.

Parameters

[in]	columnName	Name of the column
[in]	isExpr	True if the name is actually an expression

The order of outColumn() calls is the order of appearance in the output row. Use either outColumn() or outParam() but not both.

Definition at line 194 of file DbStorage.cc.

                                                                   {
     _impl->outColumn(columnName, isExpr);
 }

◆ outParam()

template<typename T >

void lsst::daf::persistence::DbTsvStorage::outParam	(	std::string const &	columnName,
		T *	location,
		bool	isExpr = `false`
	)

Request a column in the query output and bind a destination location.

Parameters

[in]	columnName	Name of the column
[in]	location	Pointer to the destination
[in]	isExpr	True if the name is actually an expression

The order of outParam() calls is the order of appearance in the output row. Use either outColumn() or outParam() but not both.

Definition at line 357 of file DbTsvStorage.cc.

                                          {
     DbStorage::outParam<T>(columnName, location, isExpr);
 }

◆ query()

void lsst::daf::persistence::DbStorage::query ( void )

virtualinherited

Execute the query.

Definition at line 248 of file DbStorage.cc.

                           {
     _impl->query();
 }

◆ repr()

std::string lsst::daf::base::Citizen::repr ( ) const

inherited

Return a string representation of a Citizen.

Definition at line 245 of file Citizen.cc.

                                      {
     return boost::str(boost::format("%d: %08x %s")
                       % _CitizenId
                       % this
                       % lsst::utils::demangleType(_typeName)
                      );
 }

◆ setColumn() [1/5]

template<typename T >

void lsst::daf::persistence::DbTsvStorage::setColumn	(	std::string const &	columnName,
		T const &	value
	)

Set the value to insert in a given column.

Parameters

[in]	columnName	Name of the column
[in]	value	Value to set in the column

Definition at line 270 of file DbTsvStorage.cc.

                                                                         {
     int colIndex = _getColumnIndex(columnName);
 
     // set value in row buffer
     // \todo quote value as appropriate
     _convertStream.str(std::string());
     _convertStream << value;
     _rowBuffer[colIndex] = _convertStream.str();
 
     // \todo Optimization: if next column, output now, plus any others saved
 }

◆ setColumn() [2/5]

template<>

void lsst::daf::persistence::DbTsvStorage::setColumn	(	std::string const &	columnName,
		char const &	value
	)

Definition at line 284 of file DbTsvStorage.cc.

                                                 {
     int colIndex = _getColumnIndex(columnName);
     _convertStream.str(std::string());
     _convertStream << static_cast<int>(value);
     _rowBuffer[colIndex] = _convertStream.str();
 }

◆ setColumn() [3/5]

template<>

void lsst::daf::persistence::DbTsvStorage::setColumn	(	std::string const &	columnName,
		double const &	value
	)

Definition at line 294 of file DbTsvStorage.cc.

                                                   {
     int colIndex = _getColumnIndex(columnName);
     _convertStream.str(std::string());
     _convertStream << std::setprecision(17) << value;
     _rowBuffer[colIndex] = _convertStream.str();
 }

◆ setColumn() [4/5]

template<>

void lsst::daf::persistence::DbTsvStorage::setColumn	(	std::string const &	columnName,
		float const &	value
	)

Definition at line 303 of file DbTsvStorage.cc.

                                                  {
     int colIndex = _getColumnIndex(columnName);
     _convertStream.str(std::string());
     _convertStream << std::setprecision(9) << value;
     _rowBuffer[colIndex] = _convertStream.str();
 }

◆ setColumn() [5/5]

template<>

void lsst::daf::persistence::DbTsvStorage::setColumn	(	std::string const &	columnName,
		DateTime const &	value
	)

Definition at line 313 of file DbTsvStorage.cc.

                                                     {
     int colIndex = _getColumnIndex(columnName);
     _convertStream.str(std::string());
     struct tm t = value.gmtime(DateTime::UTC);
     char buf[20];
     strftime(buf, sizeof(buf), "%F %T", &t);
     _rowBuffer[colIndex] = std::string(buf);
 }

◆ setColumnToNull()

void lsst::daf::persistence::DbTsvStorage::setColumnToNull ( std::string const & columnName )

virtual

Set a given column to NULL.

Parameters

[in] columnName Name of the column

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 326 of file DbTsvStorage.cc.

                                                               {
     int colIndex = _getColumnIndex(columnName);
     _rowBuffer[colIndex] = "\\N";
 }

◆ setCorruptionCallback()

dafBase::Citizen::memCallback lsst::daf::base::Citizen::setCorruptionCallback ( Citizen::memCallback func )

staticinherited

Set the CorruptionCallback function.

Parameters

func	function be called when block is found to be corrupted

Definition at line 432 of file Citizen.cc.

                                                      {
     Citizen::memCallback old = _corruptionCallback;
     _corruptionCallback = func;
 
     return old;
 }

◆ setDeleteCallback()

dafBase::Citizen::memCallback lsst::daf::base::Citizen::setDeleteCallback ( Citizen::memCallback func )

staticinherited

Set the DeleteCallback function.

Parameters

func	function be called when desired block is deleted

Definition at line 422 of file Citizen.cc.

       {
     Citizen::memCallback old = _deleteCallback;
     _deleteCallback = func;
 
     return old;
 }

◆ setDeleteCallbackId()

dafBase::Citizen::memId lsst::daf::base::Citizen::setDeleteCallbackId ( Citizen::memId id )

staticinherited

Call the current DeleteCallback when block is deleted.

Parameters

id	Desired ID

Definition at line 387 of file Citizen.cc.

       {
     WriteGuard guard(citizenLock);
     Citizen::memId oldId = _deleteId;
     _deleteId = id;
 
     return oldId;
 }

◆ setNewCallback()

dafBase::Citizen::memNewCallback lsst::daf::base::Citizen::setNewCallback ( Citizen::memNewCallback func )

staticinherited

Set the NewCallback function.

Parameters

func	The new function to be called when a designated block is allocated

Definition at line 412 of file Citizen.cc.

       {
     Citizen::memNewCallback old = _newCallback;
     _newCallback = func;
 
     return old;
 }

◆ setNewCallbackId()

dafBase::Citizen::memId lsst::daf::base::Citizen::setNewCallbackId ( Citizen::memId id )

staticinherited

Call the NewCallback when block is allocated.

Parameters

id	Desired ID

Definition at line 376 of file Citizen.cc.

       {
     WriteGuard guard(citizenLock);
     Citizen::memId oldId = _newId;
     _newId = id;
 
     return oldId;
 }

◆ setPersistLocation()

void lsst::daf::persistence::DbTsvStorage::setPersistLocation ( LogicalLocation const & location )

virtual

Set the database location to persist to.

Parameters

[in] location Database connection string to insert to.

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 91 of file DbTsvStorage.cc.

                                                                      {
     _persisting = true;
     _location = location.locString();
     // Set the timezone for any DATE/TIME/DATETIME fields.
     setenv("TZ", "UTC", 1);
 }

◆ setPolicy()

void lsst::daf::persistence::DbTsvStorage::setPolicy ( lsst::pex::policy::Policy::Ptr policy )

virtual

Allow a policy to be used to configure the DbTsvStorage.

Parameters

[in] policy

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 78 of file DbTsvStorage.cc.

                                                           {
     _tempPath = "/tmp";
     if (policy && policy->exists("TempPath")) {
         _tempPath = policy->getString("TempPath");
     }
     if (policy && policy->exists("SaveTemp") && policy->getBool("SaveTemp")) {
         _saveTemp = true;
     }
 }

◆ setQueryWhere()

void lsst::daf::persistence::DbStorage::setQueryWhere ( std::string const & whereClause )

virtualinherited

Set the condition for the WHERE clause of the query.

Parameters

[in] whereClause SQL text of the WHERE clause

May include join conditions.

Definition at line 242 of file DbStorage.cc.

                                                           {
     _impl->setQueryWhere(whereClause);
 }

◆ setRetrieveLocation()

void lsst::daf::persistence::DbTsvStorage::setRetrieveLocation ( LogicalLocation const & location )

virtual

Set the database location to retrieve from.

Parameters

[in] location Database connection string to retrieve from.

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 101 of file DbTsvStorage.cc.

                                                                       {
     _persisting = false;
     DbStorage::setRetrieveLocation(location);
 }

◆ setTableForInsert()

void lsst::daf::persistence::DbTsvStorage::setTableForInsert ( std::string const & tableName )

virtual

Set the table to insert rows into.

Parameters

[in] tableName Name of the table

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 234 of file DbTsvStorage.cc.

                                                                {
     _tableName = tableName;
     std::string templ = _tempPath + "/" + tableName + ".XXXXXX";
     _fileName = new char[templ.size() + 1];
     strncpy(_fileName, templ.c_str(), templ.size());
     _fileName[templ.size()] = '\0';
     int fd = mkstemp(_fileName);
     // \todo check for errors
     close(fd);
     _osp.reset(new std::ofstream(_fileName));
 }

◆ setTableForQuery()

void lsst::daf::persistence::DbStorage::setTableForQuery	(	std::string const &	tableName,
		bool	isExpr = `false`
	)

virtualinherited

Set the table to query (single-table queries only).

Parameters

[in]	tableName	Name of the table
[in]	isExpr	True if the name is actually a table expression

Definition at line 175 of file DbStorage.cc.

                                                                         {
     _impl->setTableForQuery(tableName, isExpr);
 }

◆ setTableListForQuery()

void lsst::daf::persistence::DbStorage::setTableListForQuery ( std::vector< std::string > const & tableNameList )

virtualinherited

Set a list of tables to query (multiple-table queries).

Parameters

[in] tableNameList Vector of names of tables

Definition at line 182 of file DbStorage.cc.

                                                {
     _impl->setTableListForQuery(tableNameList);
 }

◆ startTransaction()

void lsst::daf::persistence::DbTsvStorage::startTransaction ( void )

virtual

Start a transaction.

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 108 of file DbTsvStorage.cc.

                                         {
     if (!_persisting) DbStorage::startTransaction();
 }

◆ truncateTable()

void lsst::daf::persistence::DbTsvStorage::truncateTable ( std::string const & tableName )

virtual

Truncate a table.

Parameters

[in] tableName Name of the table to truncate

Reimplemented from lsst::daf::persistence::DbStorage.

Definition at line 218 of file DbTsvStorage.cc.

                                                            {
     if (_persisting) {
         DbStorage dbs;
         dbs.setPersistLocation(LogicalLocation(_location));
         dbs.startTransaction();
         dbs.truncateTable(tableName);
         dbs.endTransaction();
     }
     else {
         DbStorage::truncateTable(tableName);
     }
 }

◆ verifyPathName()

void lsst::daf::persistence::FormatterStorage::verifyPathName ( std::string const & name )

protectedinherited

Ensure that all directories along a path exist, creating them if necessary.

Parameters

[in] name Pathname to file to be created

Definition at line 93 of file FormatterStorage.cc.

                                                            {
     // Get the directory by stripping off anything after the last slash.
     std::string::size_type pos = name.find_last_of('/');
     if (pos == std::string::npos) return;
     std::string dirName = name.substr(0, pos);
 
     // Check to see if the directory exists.
     struct stat buf;
     int ret = ::stat(dirName.c_str(), &buf);
 
     if (ret == -1 && errno == ENOENT) {
         // It doesn't; check its parent and then create it.
         verifyPathName(dirName);
 
         ret = ::mkdir(dirName.c_str(), 0777);
 
         // If it already exists, we're OK; otherwise, throw an exception.
         if (ret == -1 && errno != EEXIST) {
             throw LSST_EXCEPT(lsst::pex::exceptions::RuntimeError,
                 dirName + ": Error creating directory = " + std::strerror(errno));
         }
     }
     else if (ret == -1) {
         // We couldn't read the (existing) directory for some reason.
         throw LSST_EXCEPT(lsst::pex::exceptions::RuntimeError,
             dirName + ": Error searching for directory = " + std::strerror(errno));
     }
     else if (!S_ISDIR(buf.st_mode)) {
         // It's not a directory.
         throw LSST_EXCEPT(lsst::pex::exceptions::RuntimeError,
             dirName + ": Non-directory in path");
     }
 }

Member Data Documentation

◆ _colMap

std::map<std::string, int> lsst::daf::persistence::DbTsvStorage::_colMap

private

Map from column names to positions.

Definition at line 106 of file DbTsvStorage.h.

◆ _convertStream

std::ostringstream lsst::daf::persistence::DbTsvStorage::_convertStream

private

Stream to convert to text.

Definition at line 107 of file DbTsvStorage.h.

◆ _fileName

char* lsst::daf::persistence::DbTsvStorage::_fileName

private

Full pathname for temporary TSV file.

Definition at line 103 of file DbTsvStorage.h.

◆ _location

std::string lsst::daf::persistence::DbTsvStorage::_location

private

Database location URL.

Definition at line 104 of file DbTsvStorage.h.

◆ _osp

std::unique_ptr<std::ofstream> lsst::daf::persistence::DbTsvStorage::_osp

private

Output TSV stream.

Definition at line 109 of file DbTsvStorage.h.

◆ _persisting

bool lsst::daf::persistence::DbTsvStorage::_persisting

private

Definition at line 100 of file DbTsvStorage.h.

◆ _rowBuffer

std::vector<std::string> lsst::daf::persistence::DbTsvStorage::_rowBuffer

private

Definition at line 108 of file DbTsvStorage.h.

◆ _saveTemp

bool lsst::daf::persistence::DbTsvStorage::_saveTemp

private

Do not delete temporary TSV file if true.

Definition at line 101 of file DbTsvStorage.h.

◆ _tableName

std::string lsst::daf::persistence::DbTsvStorage::_tableName

private

Definition at line 105 of file DbTsvStorage.h.

◆ _tempPath

std::string lsst::daf::persistence::DbTsvStorage::_tempPath

private

Directory pathname for temporary TSV file.

Definition at line 102 of file DbTsvStorage.h.

The documentation for this class was generated from the following files:

/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/daf_persistence/15.0/include/lsst/daf/persistence/DbTsvStorage.h
/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/daf_persistence/15.0/src/DbTsvStorage.cc

Public Types

Public Member Functions

Static Public Member Functions

Protected Member Functions

Private Member Functions

Private Attributes

Detailed Description

Member Typedef Documentation

◆ List

◆ memCallback

◆ memId

◆ memNewCallback

◆ Ptr

Member Enumeration Documentation

◆ anonymous enum

Constructor & Destructor Documentation

◆ DbTsvStorage()

◆ ~DbTsvStorage()

Member Function Documentation

◆ _getColumnIndex()

◆ census() [1/3]

◆ census() [2/3]

◆ census() [3/3]

◆ columnIsNull()

◆ condParam()

◆ createInstance()

◆ createTableFromTemplate()

◆ dropTable()

◆ endTransaction()

◆ executeSql()

◆ finishQuery()

◆ getColumnByPos()

◆ getId()

◆ getNextMemId()

◆ groupBy()

◆ hasBeenCorrupted()

◆ init()

◆ insertRow()

◆ markPersistent()

◆ next()

◆ orderBy()

◆ outColumn()

◆ outParam()

◆ query()

◆ repr()

◆ setColumn() [1/5]

◆ setColumn() [2/5]

◆ setColumn() [3/5]

◆ setColumn() [4/5]

◆ setColumn() [5/5]

◆ setColumnToNull()

◆ setCorruptionCallback()

◆ setDeleteCallback()

◆ setDeleteCallbackId()

◆ setNewCallback()

◆ setNewCallbackId()

◆ setPersistLocation()

◆ setPolicy()

◆ setQueryWhere()

◆ setRetrieveLocation()

◆ setTableForInsert()

◆ setTableForQuery()

◆ setTableListForQuery()

◆ startTransaction()

◆ truncateTable()

◆ verifyPathName()

Member Data Documentation

◆ _colMap

◆ _convertStream

◆ _fileName

◆ _location

◆ _osp

◆ _persisting

◆ _rowBuffer

◆ _saveTemp

◆ _tableName

◆ _tempPath