lsst.datarel.datasetScanner.DatasetScanner Class Reference

Inheritance diagram for lsst.datarel.datasetScanner.DatasetScanner:

Public Member Functions
def	__init__ (self, dataset, camera, cameraMapper)
def	walk (self, root, rules=None)

Detailed Description

File system scanner for a dataset known to a camera mapper.

Definition at line 399 of file datasetScanner.py.

Constructor & Destructor Documentation

__init__()

def lsst.datarel.datasetScanner.DatasetScanner.__init__	(		self,
			dataset,
			camera,
			cameraMapper
	)

Definition at line 403 of file datasetScanner.py.

    def __init__(self, dataset, camera, cameraMapper):
        if not isinstance(cameraMapper, lsst.daf.butlerUtils.CameraMapper):
            raise TypeError('Expecting a lsst.daf.butlerUtils.CameraMapper!')
        if dataset not in cameraMapper.mappings:
            raise NotFoundError('Unknown dataset ' + str(dataset))
        HfsScanner.__init__(self, cameraMapper.mappings[dataset].template)
        camera = camera.lower()
        if camera not in _keyTypes:
            raise RuntimeError('{} camera not supported yet'.format(camera))
        for k in self._formatKeys:
            if k not in _keyTypes[camera]:
                raise RuntimeError('{} is not a valid dataId key for camera {}'.format(k, camera))
            self._formatKeys[k].munge = _mungeFunctions[camera]

Member Function Documentation

walk()

def lsst.datarel.datasetScanner.HfsScanner.walk (   self,   root,   rules = None  )

inherited

Generator that descends the given root directory in top-down
fashion, matching paths corresponding to the template and satisfying
the given rule list. The generator yields tuples of the form
(path, dataId), where path is a dataset file name relative to root,
and dataId is a key value dictionary identifying the file.

Definition at line 281 of file datasetScanner.py.

    def walk(self, root, rules=None):
        """Generator that descends the given root directory in top-down
        fashion, matching paths corresponding to the template and satisfying
        the given rule list. The generator yields tuples of the form
        (path, dataId), where path is a dataset file name relative to root,
        and dataId is a key value dictionary identifying the file.
        """
        oneFound = False
        while os.path.exists(root) and not oneFound:
            stack = [(0, root, rules, {})]
            while stack:
                depth, path, rules, dataId = stack.pop()
                if os.path.isfile(path):
                    continue
                pc = self._pathComponents[depth]
                if pc.simple:
                    # No need to list directory contents
                    entries = [pc.regex]
                    if not os.path.exists(os.path.join(path, pc.regex)):
                        continue
                else:
                    entries = os.listdir(path)
                depth += 1
                for e in entries:
                    subRules = rules
                    subDataId = dataId
                    if not pc.simple:
                        # make sure e matches path component regular expression
                        m = pc.regex.match(e)
                        if not m:
                            continue
                        # got a match - update dataId with new key values (if any)
                        try:
                            for i, k in enumerate(pc.keys):
                                subDataId = self._formatKeys[k].munge(k, m.group(i + 1), subDataId)
                        except:
                            # Munger raises if value is invalid for key, so
                            # not really a match
                            continue
                        if subRules and pc.keys:
                            # have dataId rules and saw new keys; filter rule list
                            for k in subDataId:
                                newRules = []
                                for r in subRules:
                                    if k not in r or subDataId[k] in r[k]:
                                        newRules.append(r)
                                subRules = newRules
                            if not subRules:
                                continue  # no rules matched
                    # Have path matching template and at least one rule
                    p = os.path.join(path, e)
                    if depth < len(self._pathComponents):
                        # recurse
                        stack.append((depth, p, subRules, subDataId))
                    elif depth == len(self._pathComponents):
                        if os.path.isfile(p):
                            # found a matching file, yield it
                            yield os.path.relpath(p, root), subDataId
                            oneFound = True
            # end while stack
            root = os.path.join(root, "_parent")


# -- Camera specific dataId mungers ----

---

The documentation for this class was generated from the following file:

• /home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/datarel/14.0+77/python/lsst/datarel/datasetScanner.py

---

Generated on Sat Apr 7 2018 06:24:20 for LSSTApplications by   1.8.13