lsst.daf.persistence.safeFileIo Namespace Reference

Classes
class	DoNotWrite
class	FileForWriteOnceCompareSameFailure
class	SafeLockedFileForWrite

Functions
def	safeMakeDir (directory)
def	setFileMode (filename)
def	FileForWriteOnceCompareSame (name)
def	SafeFile (name)
def	SafeFilename (name)
def	SafeLockedFileForRead (name)

Function Documentation

FileForWriteOnceCompareSame()

def lsst.daf.persistence.safeFileIo.FileForWriteOnceCompareSame

(

name

)

Context manager to get a file that can be written only once and all other writes will succeed only if
they match the initial write.

The context manager provides a temporary file object. After the user is done, the temporary file becomes
the permanent file if the file at name does not already exist. If the file at name does exist the
temporary file is compared to the file at name. If they are the same then this is good and the temp file
is silently thrown away. If they are not the same then a runtime error is raised.

Definition at line 65 of file safeFileIo.py.

def FileForWriteOnceCompareSame(name):
    """Context manager to get a file that can be written only once and all other writes will succeed only if
    they match the initial write.
 
    The context manager provides a temporary file object. After the user is done, the temporary file becomes
    the permanent file if the file at name does not already exist. If the file at name does exist the
    temporary file is compared to the file at name. If they are the same then this is good and the temp file
    is silently thrown away. If they are not the same then a runtime error is raised.
    """
    outDir, outName = os.path.split(name)
    safeMakeDir(outDir)
    temp = tempfile.NamedTemporaryFile(mode="w", dir=outDir, prefix=outName, delete=False)
    try:
        yield temp
    finally:
        try:
            temp.close()
            # If the symlink cannot be created then it will raise. If it can't be created because a file at
            # 'name' already exists then we'll do a compare-same check.
            os.symlink(temp.name, name)
            # If the symlink was created then this is the process that created the first instance of the
            # file, and we know its contents match. Move the temp file over the symlink.
            os.rename(temp.name, name)
            # At this point, we know the file has just been created. Set permissions according to the
            # current umask.
            setFileMode(name)
        except OSError as e:
            if e.errno != errno.EEXIST:
                raise e
            filesMatch = filecmp.cmp(temp.name, name, shallow=False)
            os.remove(temp.name)
            if filesMatch:
                # if the files match then the compare-same check succeeded and we can silently return.
                return
            else:
                # if the files do not match then the calling code was trying to write a non-matching file over
                # the previous file, maybe it's a race condition? In any event, raise a runtime error.
                raise FileForWriteOnceCompareSameFailure("Written file does not match existing file.")
 
 
@contextmanager

SafeFile()

def lsst.daf.persistence.safeFileIo.SafeFile

(

name

)

Context manager to create a file in a manner avoiding race conditions

The context manager provides a temporary file object. After the user is done,
we move that file into the desired place and close the fd to avoid resource
leakage.

Definition at line 106 of file safeFileIo.py.

def SafeFile(name):
    """Context manager to create a file in a manner avoiding race conditions
 
    The context manager provides a temporary file object. After the user is done,
    we move that file into the desired place and close the fd to avoid resource
    leakage.
    """
    outDir, outName = os.path.split(name)
    safeMakeDir(outDir)
    doWrite = True
    with tempfile.NamedTemporaryFile(mode="w", dir=outDir, prefix=outName, delete=False) as temp:
        try:
            yield temp
        except DoNotWrite:
            doWrite = False
        finally:
            if doWrite:
                os.rename(temp.name, name)
                setFileMode(name)
 
 
@contextmanager

SafeFilename()

def lsst.daf.persistence.safeFileIo.SafeFilename

(

name

)

Context manager for creating a file in a manner avoiding race conditions

The context manager provides a temporary filename with no open file descriptors
(as this can cause trouble on some systems). After the user is done, we move the
file into the desired place.

Definition at line 128 of file safeFileIo.py.

def SafeFilename(name):
    """Context manager for creating a file in a manner avoiding race conditions
 
    The context manager provides a temporary filename with no open file descriptors
    (as this can cause trouble on some systems). After the user is done, we move the
    file into the desired place.
    """
    outDir, outName = os.path.split(name)
    safeMakeDir(outDir)
    temp = tempfile.NamedTemporaryFile(mode="w", dir=outDir, prefix=outName, delete=False)
    tempName = temp.name
    temp.close()  # We don't use the fd, just want a filename
    try:
        yield tempName
    finally:
        os.rename(tempName, name)
        setFileMode(name)
 
 
@contextmanager

SafeLockedFileForRead()

def lsst.daf.persistence.safeFileIo.SafeLockedFileForRead

(

name

)

Context manager for reading a file that may be locked with an exclusive lock via
SafeLockedFileForWrite. This will first acquire a shared lock before returning the file. When the file is
closed the shared lock will be unlocked.

Parameters
----------
name : string
    The file name to be opened, may include path.

Yields
------
file object
    The file to be read from.

Definition at line 148 of file safeFileIo.py.

def SafeLockedFileForRead(name):
    """Context manager for reading a file that may be locked with an exclusive lock via
    SafeLockedFileForWrite. This will first acquire a shared lock before returning the file. When the file is
    closed the shared lock will be unlocked.
 
    Parameters
    ----------
    name : string
        The file name to be opened, may include path.
 
    Yields
    ------
    file object
        The file to be read from.
    """
    log = Log.getLogger("daf.persistence.butler")
    try:
        with open(name, 'r') as f:
            log.debug("Acquiring shared lock on {}".format(name))
            fcntl.flock(f, fcntl.LOCK_SH)
            log.debug("Acquired shared lock on {}".format(name))
            yield f
    finally:
        log.debug("Releasing shared lock on {}".format(name))
 
 

safeMakeDir()

def lsst.daf.persistence.safeFileIo.safeMakeDir

(

directory

)

Make a directory in a manner avoiding race conditions

Definition at line 39 of file safeFileIo.py.

def safeMakeDir(directory):
    """Make a directory in a manner avoiding race conditions"""
    if directory != "" and not os.path.exists(directory):
        try:
            os.makedirs(directory)
        except OSError as e:
            # Don't fail if directory exists due to race
            if e.errno != errno.EEXIST:
                raise e
 
 

setFileMode()

def lsst.daf.persistence.safeFileIo.setFileMode

(

filename

)

Set a file mode according to the user's umask

Definition at line 50 of file safeFileIo.py.

def setFileMode(filename):
    """Set a file mode according to the user's umask"""
    # Get the current umask, which we can only do by setting it and then reverting to the original.
    umask = os.umask(0o077)
    os.umask(umask)
    # chmod the new file to match what it would have been if it hadn't started life as a temporary
    # file (which have more restricted permissions).
    os.chmod(filename, (~umask & 0o666))