LSST Applications  21.0.0-172-gfb10e10a+18fedfabac,22.0.0+297cba6710,22.0.0+80564b0ff1,22.0.0+8d77f4f51a,22.0.0+a28f4c53b1,22.0.0+dcf3732eb2,22.0.1-1-g7d6de66+2a20fdde0d,22.0.1-1-g8e32f31+297cba6710,22.0.1-1-geca5380+7fa3b7d9b6,22.0.1-12-g44dc1dc+2a20fdde0d,22.0.1-15-g6a90155+515f58c32b,22.0.1-16-g9282f48+790f5f2caa,22.0.1-2-g92698f7+dcf3732eb2,22.0.1-2-ga9b0f51+7fa3b7d9b6,22.0.1-2-gd1925c9+bf4f0e694f,22.0.1-24-g1ad7a390+a9625a72a8,22.0.1-25-g5bf6245+3ad8ecd50b,22.0.1-25-gb120d7b+8b5510f75f,22.0.1-27-g97737f7+2a20fdde0d,22.0.1-32-gf62ce7b1+aa4237961e,22.0.1-4-g0b3f228+2a20fdde0d,22.0.1-4-g243d05b+871c1b8305,22.0.1-4-g3a563be+32dcf1063f,22.0.1-4-g44f2e3d+9e4ab0f4fa,22.0.1-42-gca6935d93+ba5e5ca3eb,22.0.1-5-g15c806e+85460ae5f3,22.0.1-5-g58711c4+611d128589,22.0.1-5-g75bb458+99c117b92f,22.0.1-6-g1c63a23+7fa3b7d9b6,22.0.1-6-g50866e6+84ff5a128b,22.0.1-6-g8d3140d+720564cf76,22.0.1-6-gd805d02+cc5644f571,22.0.1-8-ge5750ce+85460ae5f3,master-g6e05de7fdc+babf819c66,master-g99da0e417a+8d77f4f51a,w.2021.48
LSST Data Management Base Package
Public Member Functions | Static Public Attributes | List of all members
lsst.base.packages.Packages Class Reference

Public Member Functions

def __init__ (self, packages)
 
def fromSystem (cls)
 
def fromBytes (cls, data, format)
 
def read (cls, filename)
 
def toBytes (self, format)
 
def write (self, filename)
 
def __len__ (self)
 
def __str__ (self)
 
def __repr__ (self)
 
def __contains__ (self, pkg)
 
def __iter__ (self)
 
def __eq__ (self, other)
 
def update (self, other)
 
def extra (self, other)
 
def missing (self, other)
 
def difference (self, other)
 

Static Public Attributes

dictionary formats
 

Detailed Description

A table of packages and their versions.

There are a few different types of packages, and their versions are
collected in different ways:

1. Run-time libraries (e.g., cfitsio, fftw): we get their version from
   interrogating the dynamic library
2. Python modules (e.g., afw, numpy; galsim is also in this group even
   though we only use it through the library, because no version
   information is currently provided through the library): we get their
   version from the ``__version__`` module variable. Note that this means
   that we're only aware of modules that have already been imported.
3. Other packages provide no run-time accessible version information (e.g.,
   astrometry_net): we get their version from interrogating the
   environment.  Currently, that means EUPS; if EUPS is replaced or dropped
   then we'll need to consider an alternative means of getting this version
   information.
4. Local versions of packages (a non-installed EUPS package, selected with
   ``setup -r /path/to/package``): we identify these through the
   environment (EUPS again) and use as a version the path supplemented with
   the ``git`` SHA and, if the git repo isn't clean, an MD5 of the diff.

These package versions are collected and stored in a Packages object, which
provides useful comparison and persistence features.

Example usage:

.. code-block:: python

    from lsst.base import Packages
    pkgs = Packages.fromSystem()
    print("Current packages:", pkgs)
    old = Packages.read("/path/to/packages.pickle")
    print("Old packages:", old)
    print("Missing packages compared to before:", pkgs.missing(old))
    print("Extra packages compared to before:", pkgs.extra(old))
    print("Different packages: ", pkgs.difference(old))
    old.update(pkgs)  # Include any new packages in the old
    old.write("/path/to/packages.pickle")

Parameters
----------
packages : `dict`
    A mapping {package: version} where both keys and values are type `str`.

Notes
-----
This is essentially a wrapper around a dict with some conveniences.

Definition at line 262 of file packages.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.base.packages.Packages.__init__ (   self,
  packages 
)

Definition at line 317 of file packages.py.

317  def __init__(self, packages):
318  assert isinstance(packages, Mapping)
319  self._packages = packages
320  self._names = set(packages.keys())
321 
set
daf::base::PropertySet * set
Definition: fits.cc:912

Member Function Documentation

◆ __contains__()

def lsst.base.packages.Packages.__contains__ (   self,
  pkg 
)

Definition at line 438 of file packages.py.

438  def __contains__(self, pkg):
439  return pkg in self._packages
440 

◆ __eq__()

def lsst.base.packages.Packages.__eq__ (   self,
  other 
)

Definition at line 444 of file packages.py.

444  def __eq__(self, other):
445  if not isinstance(other, type(self)):
446  return False
447 
448  return self._packages == other._packages
449 
type
table::Key< int > type
Definition: Detector.cc:163

◆ __iter__()

def lsst.base.packages.Packages.__iter__ (   self)

Definition at line 441 of file packages.py.

441  def __iter__(self):
442  return iter(self._packages)
443 

◆ __len__()

def lsst.base.packages.Packages.__len__ (   self)

Definition at line 424 of file packages.py.

424  def __len__(self):
425  return len(self._packages)
426 

◆ __repr__()

def lsst.base.packages.Packages.__repr__ (   self)

Definition at line 435 of file packages.py.

435  def __repr__(self):
436  return "%s(%s)" % (self.__class__.__name__, repr(self._packages))
437 

◆ __str__()

def lsst.base.packages.Packages.__str__ (   self)

Definition at line 427 of file packages.py.

427  def __str__(self):
428  ss = "%s({\n" % self.__class__.__name__
429  # Sort alphabetically by module name, for convenience in reading
430  ss += ",\n".join("%s: %s" % (repr(prod), repr(self._packages[prod])) for
431  prod in sorted(self._names))
432  ss += ",\n})"
433  return ss
434 

◆ difference()

def lsst.base.packages.Packages.difference (   self,
  other 
) 
Get packages in symmetric difference of self and another `Packages`
object.

Parameters
----------
other : `Packages`
    Other packages to compare against.

Returns
-------
difference : `dict`
    Packages in symmetric difference.  Keys (type `str`) are package
    names; values (type `str`) are their versions.

Definition at line 497 of file packages.py.

497  def difference(self, other):
498  """Get packages in symmetric difference of self and another `Packages`
499  object.
500 
501  Parameters
502  ----------
503  other : `Packages`
504  Other packages to compare against.
505 
506  Returns
507  -------
508  difference : `dict`
509  Packages in symmetric difference. Keys (type `str`) are package
510  names; values (type `str`) are their versions.
511  """
512  return {pkg: (self._packages[pkg], other._packages[pkg]) for
513  pkg in self._names & other._names if self._packages[pkg] != other._packages[pkg]}
514 
515 
516 # Register YAML representers
517 

◆ extra()

def lsst.base.packages.Packages.extra (   self,
  other 
) 
Get packages in self but not in another `Packages` object.

Parameters
----------
other : `Packages`
    Other packages to compare against.

Returns
-------
extra : `dict`
    Extra packages. Keys (type `str`) are package names; values
    (type `str`) are their versions.

Definition at line 465 of file packages.py.

465  def extra(self, other):
466  """Get packages in self but not in another `Packages` object.
467 
468  Parameters
469  ----------
470  other : `Packages`
471  Other packages to compare against.
472 
473  Returns
474  -------
475  extra : `dict`
476  Extra packages. Keys (type `str`) are package names; values
477  (type `str`) are their versions.
478  """
479  return {pkg: self._packages[pkg] for pkg in self._names - other._names}
480 

◆ fromBytes()

def lsst.base.packages.Packages.fromBytes (   cls,
  data,
  format 
) 
Construct the object from a byte representation.

Parameters
----------
data : `bytes`
    The serialized form of this object in bytes.
format : `str`
    The format of those bytes. Can be ``yaml`` or ``pickle``.

Definition at line 341 of file packages.py.

341  def fromBytes(cls, data, format):
342  """Construct the object from a byte representation.
343 
344  Parameters
345  ----------
346  data : `bytes`
347  The serialized form of this object in bytes.
348  format : `str`
349  The format of those bytes. Can be ``yaml`` or ``pickle``.
350  """
351  if format == "pickle":
352  new = pickle.loads(data)
353  elif format == "yaml":
354  new = yaml.load(data, Loader=yaml.SafeLoader)
355  else:
356  raise ValueError(f"Unexpected serialization format given: {format}")
357  if not isinstance(new, cls):
358  raise TypeError(f"Extracted object of class '{type(new)}' but expected '{cls}'")
359  return new
360 

◆ fromSystem()

def lsst.base.packages.Packages.fromSystem (   cls) 
Construct a `Packages` by examining the system.

Determine packages by examining python's `sys.modules`, runtime
libraries and EUPS.

Returns
-------
packages : `Packages`

Definition at line 323 of file packages.py.

323  def fromSystem(cls):
324  """Construct a `Packages` by examining the system.
325 
326  Determine packages by examining python's `sys.modules`, runtime
327  libraries and EUPS.
328 
329  Returns
330  -------
331  packages : `Packages`
332  """
333  packages = {}
334  packages.update(getPythonPackages())
335  packages.update(getCondaPackages())
336  packages.update(getRuntimeVersions())
337  packages.update(getEnvironmentPackages()) # Should be last, to override products with LOCAL versions
338  return cls(packages)
339 
lsst::base.packages.getEnvironmentPackages
def getEnvironmentPackages()
Definition: packages.py:154
lsst::base.packages.getCondaPackages
def getCondaPackages()
Definition: packages.py:223
lsst::base.packages.getPythonPackages
def getPythonPackages()
Definition: packages.py:92
lsst::base::getRuntimeVersions
std::map< std::string, std::string > getRuntimeVersions()
Return version strings for dependencies.
Definition: versions.cc:54

◆ missing()

def lsst.base.packages.Packages.missing (   self,
  other 
) 
Get packages in another `Packages` object but missing from self.

Parameters
----------
other : `Packages`
    Other packages to compare against.

Returns
-------
missing : `dict`
    Missing packages. Keys (type `str`) are package names; values
    (type `str`) are their versions.

Definition at line 481 of file packages.py.

481  def missing(self, other):
482  """Get packages in another `Packages` object but missing from self.
483 
484  Parameters
485  ----------
486  other : `Packages`
487  Other packages to compare against.
488 
489  Returns
490  -------
491  missing : `dict`
492  Missing packages. Keys (type `str`) are package names; values
493  (type `str`) are their versions.
494  """
495  return {pkg: other._packages[pkg] for pkg in other._names - self._names}
496 

◆ read()

def lsst.base.packages.Packages.read (   cls,
  filename 
) 
Read packages from filename.

Parameters
----------
filename : `str`
    Filename from which to read. The format is determined from the
    file extension.  Currently support ``.pickle``, ``.pkl``
    and ``.yaml``.

Returns
-------
packages : `Packages`

Definition at line 362 of file packages.py.

362  def read(cls, filename):
363  """Read packages from filename.
364 
365  Parameters
366  ----------
367  filename : `str`
368  Filename from which to read. The format is determined from the
369  file extension. Currently support ``.pickle``, ``.pkl``
370  and ``.yaml``.
371 
372  Returns
373  -------
374  packages : `Packages`
375  """
376  _, ext = os.path.splitext(filename)
377  if ext not in cls.formats:
378  raise ValueError(f"Format from {ext} extension in file {filename} not recognized")
379  with open(filename, "rb") as ff:
380  # We assume that these classes are tiny so there is no
381  # substantive memory impact by reading the entire file up front
382  data = ff.read()
383  return cls.fromBytes(data, cls.formats[ext])
384 
read
std::shared_ptr< table::io::Persistable > read(table::io::InputArchive const &archive, table::io::CatalogVector const &catalogs) const override
Definition: warpExposure.cc:0

◆ toBytes()

def lsst.base.packages.Packages.toBytes (   self,
  format 
) 
Convert the object to a serialized bytes form using the
specified format.

Parameters
----------
format : `str`
    Format to use when serializing. Can be ``yaml`` or ``pickle``.

Returns
-------
data : `bytes`
    Byte string representing the serialized object.

Definition at line 385 of file packages.py.

385  def toBytes(self, format):
386  """Convert the object to a serialized bytes form using the
387  specified format.
388 
389  Parameters
390  ----------
391  format : `str`
392  Format to use when serializing. Can be ``yaml`` or ``pickle``.
393 
394  Returns
395  -------
396  data : `bytes`
397  Byte string representing the serialized object.
398  """
399  if format == "pickle":
400  return pickle.dumps(self)
401  elif format == "yaml":
402  return yaml.dump(self).encode("utf-8")
403  else:
404  raise ValueError(f"Unexpected serialization format requested: {format}")
405 
lsst::sphgeom::python::encode
pybind11::bytes encode(Region const &self)
Encode a Region as a pybind11 bytes object.
Definition: utils.h:53

◆ update()

def lsst.base.packages.Packages.update (   self,
  other 
) 
Update packages with contents of another set of packages.

Parameters
----------
other : `Packages`
    Other packages to merge with self.

Notes
-----
No check is made to see if we're clobbering anything.

Definition at line 450 of file packages.py.

450  def update(self, other):
451  """Update packages with contents of another set of packages.
452 
453  Parameters
454  ----------
455  other : `Packages`
456  Other packages to merge with self.
457 
458  Notes
459  -----
460  No check is made to see if we're clobbering anything.
461  """
462  self._packages.update(other._packages)
463  self._names.update(other._names)
464 

◆ write()

def lsst.base.packages.Packages.write (   self,
  filename 
) 
Write to file.

Parameters
----------
filename : `str`
    Filename to which to write. The format of the data file
    is determined from the file extension. Currently supports
    ``.pickle`` and ``.yaml``

Definition at line 406 of file packages.py.

406  def write(self, filename):
407  """Write to file.
408 
409  Parameters
410  ----------
411  filename : `str`
412  Filename to which to write. The format of the data file
413  is determined from the file extension. Currently supports
414  ``.pickle`` and ``.yaml``
415  """
416  _, ext = os.path.splitext(filename)
417  if ext not in self.formats:
418  raise ValueError(f"Format from {ext} extension in file {filename} not recognized")
419  with open(filename, "wb") as ff:
420  # Assumes that the bytes serialization of this object is
421  # relatively small.
422  ff.write(self.toBytes(self.formats[ext]))
423 
lsst::afw::image::write
void write(OutputArchiveHandle &handle) const override

Member Data Documentation

◆ formats

dictionary lsst.base.packages.Packages.formats
static
Initial value:
= {".pkl": "pickle",
".pickle": "pickle",
".yaml": "yaml"}

Definition at line 313 of file packages.py.

The documentation for this class was generated from the following file:
Generated on Fri Nov 26 2021 10:16:17 for LSST Applications by doxygen 1.9.1