lsst.daf.persistence.registries.PosixRegistry Class Reference

Inheritance diagram for lsst.daf.persistence.registries.PosixRegistry:

Classes
class	LookupData

Public Member Functions
def	__init__ (self, root)
def	lookup (self, lookupProperties, reference, dataId, **kwargs)

Static Public Member Functions
def	getHduNumber (template, dataId)
def	lookupMetadata (filepath, template, lookupData, storage)
def	lookupFitsMetadata (filepath, template, lookupData, dataId)
def	create (location)

Public Attributes
	root

Detailed Description

A glob-based filesystem registry

Definition at line 106 of file registries.py.

Constructor & Destructor Documentation

__init__()

def lsst.daf.persistence.registries.PosixRegistry.__init__	(		self,
			root
	)

Definition at line 109 of file registries.py.

    def __init__(self, root):
        Registry.__init__(self)
        self.root = root
 

Member Function Documentation

create()

def lsst.daf.persistence.registries.Registry.create (   location )

staticinherited

Create a registry object of an appropriate type.
@param location (string) Path or URL for registry, or None if
                         unavailable

Definition at line 71 of file registries.py.

    def create(location):
        """Create a registry object of an appropriate type.
        @param location (string) Path or URL for registry, or None if
                                 unavailable"""
 
        if location is None:
            return
 
        # if re.match(r'.*\.registry', location):
        #     return FileRegistry(location)
 
        if location.endswith(".pgsql"):
            return PgsqlRegistry(location)
 
        # look for an sqlite3 registry
        if re.match(r'.*\.sqlite3', location):
            if not haveSqlite3:
                raise RuntimeError("sqlite3 registry specified (%s), but unable to import sqlite3 module" %
                                   (location,))
            registry = SqliteRegistry(location)
            if registry.conn is None:
                return None
            return registry
 
        # if re.match(r'mysql:', location):
        #     return DbRegistry(location)
        # return FsRegistry(location)
 
        # next try to create a PosixRegistry
        if os.path.isdir(location):
            return PosixRegistry(root=location)
 
        raise RuntimeError("Unable to create registry using location: " + location)
 
 

getHduNumber()

def lsst.daf.persistence.registries.PosixRegistry.getHduNumber (   template,   dataId  )

static

Looks up the HDU number for a given template+dataId.
:param template: template with HDU specifier (ends with brackets and an
identifier that can be populated by a key-value pair in dataId.
e.g. "%(visit)07d/instcal%(visit)07d.fits.fz[%(ccdnum)d]"
:param dataId: dictionary that hopefully has a key-value pair whose key
matches (has the same name) as the key specifier in the template.
:return: the HDU specified by the template+dataId pair, or None if the
HDU can not be determined.

Definition at line 114 of file registries.py.

    def getHduNumber(template, dataId):
        """Looks up the HDU number for a given template+dataId.
        :param template: template with HDU specifier (ends with brackets and an
        identifier that can be populated by a key-value pair in dataId.
        e.g. "%(visit)07d/instcal%(visit)07d.fits.fz[%(ccdnum)d]"
        :param dataId: dictionary that hopefully has a key-value pair whose key
        matches (has the same name) as the key specifier in the template.
        :return: the HDU specified by the template+dataId pair, or None if the
        HDU can not be determined.
        """
        # sanity check that the template at least ends with a brace.
        if not template.endswith(']'):
            return None
 
        # get the key (with formatting) out of the brances
        hduKey = template[template.rfind('[') + 1:template.rfind(']')]
        # extract the key name from the formatting
        hduKey = hduKey[hduKey.rfind('(') + 1:hduKey.rfind(')')]
 
        if hduKey in dataId:
            return dataId[hduKey]
        return None
 

lookup()

def lsst.daf.persistence.registries.PosixRegistry.lookup	(		self,
			lookupProperties,
			reference,
			dataId,
		**	kwargs
	)

Perform a lookup in the registry.

Return values are refined by the values in dataId.
Returns a list of values that match keys in lookupProperties.
e.g. if the template is 'raw/raw_v%(visit)d_f%(filter)s.fits.gz', and
dataId={'visit':1}, and lookupProperties is ['filter'], and the
filesystem under self.root has exactly one file 'raw/raw_v1_fg.fits.gz'
then the return value will be [('g',)]

:param lookupProperties: keys whose values will be returned.
:param reference: other data types that may be used to search for values.
:param dataId: must be an iterable. Keys must be string.
If value is a string then will look for elements in the repository that match value for key.
If value is a 2-item iterable then will look for elements in the repository are between (inclusive)
the first and second items in the value.
:param **kwargs: keys required for the posix registry to search for items. If required keys are not
provide will return an empty list.
'template': required. template parameter (typically from a policy) that can be used to look for files
'storage': optional. Needed to look for metadata in files. Currently supported values: 'FitsStorage'.
:return: a list of values that match keys in lookupProperties.

Definition at line 192 of file registries.py.

    def lookup(self, lookupProperties, reference, dataId, **kwargs):
        """Perform a lookup in the registry.
 
        Return values are refined by the values in dataId.
        Returns a list of values that match keys in lookupProperties.
        e.g. if the template is 'raw/raw_v%(visit)d_f%(filter)s.fits.gz', and
        dataId={'visit':1}, and lookupProperties is ['filter'], and the
        filesystem under self.root has exactly one file 'raw/raw_v1_fg.fits.gz'
        then the return value will be [('g',)]
 
        :param lookupProperties: keys whose values will be returned.
        :param reference: other data types that may be used to search for values.
        :param dataId: must be an iterable. Keys must be string.
        If value is a string then will look for elements in the repository that match value for key.
        If value is a 2-item iterable then will look for elements in the repository are between (inclusive)
        the first and second items in the value.
        :param **kwargs: keys required for the posix registry to search for items. If required keys are not
        provide will return an empty list.
        'template': required. template parameter (typically from a policy) that can be used to look for files
        'storage': optional. Needed to look for metadata in files. Currently supported values: 'FitsStorage'.
        :return: a list of values that match keys in lookupProperties.
        """
        # required kwargs:
        if 'template' in kwargs:
            template = kwargs['template']
        else:
            return []
        # optional kwargs:
        storage = kwargs['storage'] if 'storage' in kwargs else None
 
        lookupData = PosixRegistry.LookupData(lookupProperties, dataId)
        scanner = fsScanner.FsScanner(template)
        allPaths = scanner.processPath(self.root)
        retItems = []  # one item for each found file that matches
        for path, foundProperties in allPaths.items():
            # check for dataId keys that are not present in found properties
            # search for those keys in metadata of file at path
            # if present, check for matching values
            # if not present, file can not match, do not use it.
            lookupData.setFoundItems(foundProperties)
            if 'incomplete' == lookupData.status():
                PosixRegistry.lookupMetadata(os.path.join(self.root, path), template, lookupData, storage)
            if 'match' == lookupData.status():
                ll = tuple(lookupData.foundItems[key] for key in lookupData.lookupProperties)
                retItems.append(ll)
        return retItems
 

lookupFitsMetadata()

def lsst.daf.persistence.registries.PosixRegistry.lookupFitsMetadata (   filepath,   template,   lookupData,   dataId  )

static

Look up metadata in a fits file.
Will try to discover the correct HDU to look in by testing if the
template has a value in brackets at the end.
If the HDU is specified but the metadata key is not discovered in
that HDU, will look in the primary HDU before giving up.
:param filepath: path to the file
:param template: template that was used to discover the file. This can
be used to look up the correct HDU as needed.
:param lookupData: an instance if LookupData that contains the
lookupProperties, the dataId, and the data that has been found so far.
Will be updated with new information as discovered.
:param dataId:
:return:

Definition at line 247 of file registries.py.

    def lookupFitsMetadata(filepath, template, lookupData, dataId):
        """Look up metadata in a fits file.
        Will try to discover the correct HDU to look in by testing if the
        template has a value in brackets at the end.
        If the HDU is specified but the metadata key is not discovered in
        that HDU, will look in the primary HDU before giving up.
        :param filepath: path to the file
        :param template: template that was used to discover the file. This can
        be used to look up the correct HDU as needed.
        :param lookupData: an instance if LookupData that contains the
        lookupProperties, the dataId, and the data that has been found so far.
        Will be updated with new information as discovered.
        :param dataId:
        :return:
        """
        try:
            hdulist = astropy.io.fits.open(filepath, memmap=True)
        except IOError:
            return
        hduNumber = PosixRegistry.getHduNumber(template=template, dataId=dataId)
        if hduNumber is not None and hduNumber < len(hdulist):
            hdu = hdulist[hduNumber]
        else:
            hdu = None
        if len(hdulist) > 0:
            primaryHdu = hdulist[0]
        else:
            primaryHdu = None
 
        for property in lookupData.getMissingKeys():
            propertyValue = None
            if hdu is not None and property in hdu.header:
                propertyValue = hdu.header[property]
            # if the value is not in the indicated HDU, try the primary HDU:
            elif primaryHdu is not None and property in primaryHdu.header:
                propertyValue = primaryHdu.header[property]
            lookupData.addFoundItems({property: propertyValue})
 
 

lookupMetadata()

def lsst.daf.persistence.registries.PosixRegistry.lookupMetadata (   filepath,   template,   lookupData,   storage  )

static

Dispatcher for looking up metadata in a file of a given storage type

Definition at line 240 of file registries.py.

    def lookupMetadata(filepath, template, lookupData, storage):
        """Dispatcher for looking up metadata in a file of a given storage type
        """
        if storage == 'FitsStorage':
            PosixRegistry.lookupFitsMetadata(filepath, template, lookupData, storage)
 

Member Data Documentation

root

lsst.daf.persistence.registries.PosixRegistry.root

Definition at line 111 of file registries.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-0.7.0/Linux64/daf_persistence/22.0.1-4-g243d05b+871c1b8305/python/lsst/daf/persistence/registries.py