lsst.daf.persistence.registries.PgsqlRegistry Class Reference

Inheritance diagram for lsst.daf.persistence.registries.PgsqlRegistry:

Public Member Functions
def	__init__ (self, location)
def	lookup (self, *args, **kwargs)
def	lookup (self, lookupProperties, reference, dataId, **kwargs)
def	executeQuery (self, returnFields, joinClause, whereFields, range, values)

Static Public Member Functions
def	readYaml (location)
def	create (location)

Public Attributes
	root
	conn

Static Public Attributes
string	placeHolder = "%s"

Detailed Description

A PostgreSQL-based registry

Definition at line 456 of file registries.py.

Constructor & Destructor Documentation

__init__()

def lsst.daf.persistence.registries.PgsqlRegistry.__init__	(		self,
			location
	)

Constructor

Parameters
----------
location : `str`
    Path to PostgreSQL configuration file.

Reimplemented from lsst.daf.persistence.registries.SqlRegistry.

Definition at line 460 of file registries.py.

    def __init__(self, location):
        """Constructor
 
        Parameters
        ----------
        location : `str`
            Path to PostgreSQL configuration file.
        """
        if not havePgsql:
            raise RuntimeError("Cannot use PgsqlRegistry: could not import psycopg2")
        config = self.readYaml(location)
        self._config = config
        conn = pgsql.connect(host=config["host"], port=config["port"], database=config["database"],
                             user=config["user"], password=config["password"])
        self.root = location
        SqlRegistry.__init__(self, conn)
 

Member Function Documentation

create()

def lsst.daf.persistence.registries.Registry.create (   location )

staticinherited

Create a registry object of an appropriate type.
@param location (string) Path or URL for registry, or None if
                         unavailable

Definition at line 71 of file registries.py.

    def create(location):
        """Create a registry object of an appropriate type.
        @param location (string) Path or URL for registry, or None if
                                 unavailable"""
 
        if location is None:
            return
 
        # if re.match(r'.*\.registry', location):
        #     return FileRegistry(location)
 
        if location.endswith(".pgsql"):
            return PgsqlRegistry(location)
 
        # look for an sqlite3 registry
        if re.match(r'.*\.sqlite3', location):
            if not haveSqlite3:
                raise RuntimeError("sqlite3 registry specified (%s), but unable to import sqlite3 module" %
                                   (location,))
            registry = SqliteRegistry(location)
            if registry.conn is None:
                return None
            return registry
 
        # if re.match(r'mysql:', location):
        #     return DbRegistry(location)
        # return FsRegistry(location)
 
        # next try to create a PosixRegistry
        if os.path.isdir(location):
            return PosixRegistry(root=location)
 
        raise RuntimeError("Unable to create registry using location: " + location)
 
 

executeQuery()

def lsst.daf.persistence.registries.SqlRegistry.executeQuery (   self,   returnFields,   joinClause,   whereFields,   range,   values  )

inherited

Extract metadata from the registry.
@param returnFields (list of strings) Metadata fields to be extracted.
@param joinClause   (list of strings) Tables in which metadata fields
                    are located.
@param whereFields  (list of tuples) First tuple element is metadata
                    field to query; second is the value that field
                    must have (often '?').
@param range        (tuple) Value, lower limit, and upper limit for a
                    range condition on the metadata.  Any of these can
                    be metadata fields.
@param values       (tuple) Tuple of values to be substituted for '?'
                    characters in the whereFields values or the range
                    values.
@return (list of tuples) All sets of field values that meet the
        criteria

Definition at line 401 of file registries.py.

    def executeQuery(self, returnFields, joinClause, whereFields, range, values):
        """Extract metadata from the registry.
        @param returnFields (list of strings) Metadata fields to be extracted.
        @param joinClause   (list of strings) Tables in which metadata fields
                            are located.
        @param whereFields  (list of tuples) First tuple element is metadata
                            field to query; second is the value that field
                            must have (often '?').
        @param range        (tuple) Value, lower limit, and upper limit for a
                            range condition on the metadata.  Any of these can
                            be metadata fields.
        @param values       (tuple) Tuple of values to be substituted for '?'
                            characters in the whereFields values or the range
                            values.
        @return (list of tuples) All sets of field values that meet the
                criteria"""
        if not self.conn:
            return None
        cmd = "SELECT DISTINCT "
        cmd += ", ".join(returnFields)
        cmd += " FROM " + " NATURAL JOIN ".join(joinClause)
        whereList = []
        if whereFields:
            for k, v in whereFields:
                whereList.append("(%s = %s)" % (k, v))
        if range is not None:
            whereList.append("(%s BETWEEN %s AND %s)" % range)
        if len(whereList) > 0:
            cmd += " WHERE " + " AND ".join(whereList)
        cursor = self.conn.cursor()
        cursor.execute(cmd, values)
        return [row for row in cursor.fetchall()]
 
 

lookup() [1/2]

def lsst.daf.persistence.registries.PgsqlRegistry.lookup	(		self,
		*	args,
		**	kwargs
	)

Definition at line 525 of file registries.py.

    def lookup(self, *args, **kwargs):
        try:
            return SqlRegistry.lookup(self, *args, **kwargs)
        except Exception:
            self.conn.rollback()
            raise

lookup() [2/2]

def lsst.daf.persistence.registries.SqlRegistry.lookup (   self,   lookupProperties,   reference,   dataId, **  kwargs  )

inherited

Perform a lookup in the registry.

Return values are refined by the values in dataId.
Returns a list of values that match keys in lookupProperties.
e.g. if the template is 'raw/raw_v%(visit)d_f%(filter)s.fits.gz', and
dataId={'visit':1}, and lookupProperties is ['filter'], and the
filesystem under self.root has exactly one file 'raw/raw_v1_fg.fits.gz'
then the return value will be [('g',)]

:param lookupProperties:
:param dataId: must be a key/value iterable. Keys must be string.
If value is a string then will look for elements in the repository that match value for value.
If value is a 2-item iterable then will look for elements in the repository where the value is between
the values of value[0] and value[1].
:param reference: other data types that may be used to search for values.
:param **kwargs: nothing needed for sqlite lookup
:return: a list of values that match keys in lookupProperties.

Definition at line 370 of file registries.py.

    def lookup(self, lookupProperties, reference, dataId, **kwargs):
        """Perform a lookup in the registry.
 
        Return values are refined by the values in dataId.
        Returns a list of values that match keys in lookupProperties.
        e.g. if the template is 'raw/raw_v%(visit)d_f%(filter)s.fits.gz', and
        dataId={'visit':1}, and lookupProperties is ['filter'], and the
        filesystem under self.root has exactly one file 'raw/raw_v1_fg.fits.gz'
        then the return value will be [('g',)]
 
        :param lookupProperties:
        :param dataId: must be a key/value iterable. Keys must be string.
        If value is a string then will look for elements in the repository that match value for value.
        If value is a 2-item iterable then will look for elements in the repository where the value is between
        the values of value[0] and value[1].
        :param reference: other data types that may be used to search for values.
        :param **kwargs: nothing needed for sqlite lookup
        :return: a list of values that match keys in lookupProperties.
        """
        if not self.conn:
            return None
 
        # input variable sanitization:
        reference = sequencify(reference)
        lookupProperties = sequencify(lookupProperties)
 
        try:
            return self._lookup(lookupProperties, dataId, reference)
        except sqlite3.OperationalError:  # try again, with extra checking of the dataId keys
            return self._lookup(lookupProperties, dataId, reference, checkColumns=True)
 

readYaml()

def lsst.daf.persistence.registries.PgsqlRegistry.readYaml (   location )

static

Read YAML configuration file

The YAML configuration file should contain:
* host : host name for database connection
* port : port for database connection
* user : user name for database connection
* database : database name

It may also contain:
* password : password for database connection

The optional entries are set to `None` in the output configuration.

Parameters
----------
location : `str`
    Path to PostgreSQL YAML config file.

Returns
-------
config : `dict`
    Configuration

Definition at line 478 of file registries.py.

    def readYaml(location):
        """Read YAML configuration file
 
        The YAML configuration file should contain:
        * host : host name for database connection
        * port : port for database connection
        * user : user name for database connection
        * database : database name
 
        It may also contain:
        * password : password for database connection
 
        The optional entries are set to `None` in the output configuration.
 
        Parameters
        ----------
        location : `str`
            Path to PostgreSQL YAML config file.
 
        Returns
        -------
        config : `dict`
            Configuration
        """
        try:
            # PyYAML >=5.1 prefers a different loader
            loader = yaml.UnsafeLoader
        except AttributeError:
            loader = yaml.Loader
        with open(location) as ff:
            data = yaml.load(ff, Loader=loader)
        requireKeys = set(["host", "port", "database", "user"])
        optionalKeys = set(["password"])
        haveKeys = set(data.keys())
        if haveKeys - optionalKeys != requireKeys:
            raise RuntimeError(
                "PostgreSQL YAML configuration (%s) should contain only %s, and may contain 'password', "
                "but this contains: %s" %
                (location, ",".join("'%s'" % key for key in requireKeys),
                 ",".join("'%s'" % key for key in data.keys()))
            )
        for key in optionalKeys:
            if key not in data:
                data[key] = None
 
        return data
 

Member Data Documentation

conn

lsst.daf.persistence.registries.SqlRegistry.conn

inherited

Definition at line 313 of file registries.py.

placeHolder

string lsst.daf.persistence.registries.PgsqlRegistry.placeHolder = "%s"

static

Definition at line 458 of file registries.py.

root

lsst.daf.persistence.registries.PgsqlRegistry.root

Definition at line 474 of file registries.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-0.7.0/Linux64/daf_persistence/22.0.1-4-g243d05b+871c1b8305/python/lsst/daf/persistence/registries.py