lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase Class Reference

Inheritance diagram for lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase:

Public Member Functions
def	instrumentClassName (self)
def	instrumentClass (self)
def	instrumentName (self)
def	setUp (self)
def	tearDown (self)
def	check_raw (self, gen3Butler, exposure, detector)
def	check_calibs (self, calibName, calibIds, gen3Butler)
def	check_defects (self, gen3Butler, detectors)
def	check_refcat (self, gen3Butler)
def	check_collections (self, gen3Butler)
def	test_convert (self)

Public Attributes
	gen3root
	gen2Butler

Static Public Attributes
string	gen2root = ""
	gen2calib = None
	config = None
list	biases = []
string	biasName = "bias"
list	flats = []
string	flatName = "flat"
list	darks = []
string	darkName = "dark"
dictionary	kwargs = {}
list	refcats = []
	collections = set()
string	detectorKey = "ccd"
string	exposureKey = "visit"
string	calibFilterType = "physical_filter"
	skymapName = None
	skymapConfig = None

Detailed Description

Test the `butler convert` command.

Subclass this, and then `lsst.utils.tests.TestCase` and set the below
attributes.  Uses the `butler convert` command line command to do the
conversion.

Definition at line 42 of file convertTests.py.

Member Function Documentation

check_calibs()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.check_calibs	(		self,
			calibName,
			calibIds,
			gen3Butler
	)

Test that we can get converted bias/dark/flat from the gen3 repo.

Note: because there is no clear way to get calibrations from a gen2
repo, we just test that the thing we got is an ExposureF here, and
assume that formatter testing is handled properly elsewhere.

Parameters
----------
calibName : `str`
    The name of the calibration to attempt to get ("bias", "flat").
calibIds : `list` of `dict`
    The list of calibration dataIds to get.
gen3Butler : `lsst.daf.butler.Butler`
    The Butler to use to get the data.

Definition at line 201 of file convertTests.py.

    def check_calibs(self, calibName, calibIds, gen3Butler):
        """Test that we can get converted bias/dark/flat from the gen3 repo.
 
        Note: because there is no clear way to get calibrations from a gen2
        repo, we just test that the thing we got is an ExposureF here, and
        assume that formatter testing is handled properly elsewhere.
 
        Parameters
        ----------
        calibName : `str`
            The name of the calibration to attempt to get ("bias", "flat").
        calibIds : `list` of `dict`
            The list of calibration dataIds to get.
        gen3Butler : `lsst.daf.butler.Butler`
            The Butler to use to get the data.
        """
        if not calibIds:
            return
        collection = self.instrumentClass.makeCalibrationCollectionName()
        with self.subTest(dtype=calibName):
            datasets = {}
            for assoc in gen3Butler.registry.queryDatasetAssociations(calibName, collections=collection):
                # There will in general be multiple refs for each data ID
                # (for different validity ranges), but we'll just test one
                # anyway to keep the test fast, so we don't care that this
                # might overwrite.
                datasets[assoc.ref.dataId] = assoc.ref
            for dataId in calibIds:
                standardizedDataId = lsst.daf.butler.DataCoordinate.standardize(
                    dataId,
                    universe=gen3Butler.registry.dimensions
                )
                with self.subTest(dataId=standardizedDataId):
                    gen3Exposure = gen3Butler.getDirect(datasets[standardizedDataId])
                    self.assertIsInstance(gen3Exposure, lsst.afw.image.ExposureF)
 

check_collections()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.check_collections	(		self,
			gen3Butler
	)

Test that the correct set of collections is in the gen3 repo.

Parameters
----------
gen3Butler : `lsst.daf.butler.Butler`
    The Butler to be tested.

Definition at line 278 of file convertTests.py.

    def check_collections(self, gen3Butler):
        """Test that the correct set of collections is in the gen3 repo.
 
        Parameters
        ----------
        gen3Butler : `lsst.daf.butler.Butler`
            The Butler to be tested.
        """
        # We use assertGreaterEqual because the conversion code may create
        # multiple RUNS and combine them into CHAINED collections as an
        # implementation detail; we only care that the high-level collections
        # exist one way or another.
        self.assertGreaterEqual(set(gen3Butler.registry.queryCollections()), set(self.collections),
                                f"Compare with expected collections ({self.collections})")
 

check_defects()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.check_defects	(		self,
			gen3Butler,
			detectors
	)

Test that we can get converted defects from the gen3 repo.

Parameters
----------
gen3Butler : `lsst.daf.butler.Butler`
    The Butler to be tested.
detectors : `list` of `int`
    The detector identifiers to ``get`` from the gen3 butler.

Definition at line 237 of file convertTests.py.

    def check_defects(self, gen3Butler, detectors):
        """Test that we can get converted defects from the gen3 repo.
 
        Parameters
        ----------
        gen3Butler : `lsst.daf.butler.Butler`
            The Butler to be tested.
        detectors : `list` of `int`
            The detector identifiers to ``get`` from the gen3 butler.
        """
        collection = self.instrumentClass.makeCalibrationCollectionName()
        datasets = {}
        for assoc in gen3Butler.registry.queryDatasetAssociations("defects", collections=collection):
            # There will in general be multiple refs for each data ID
            # (for different validity ranges), but we'll just test one
            # anyway to keep the test fast, so we don't care that this
            # might overwrite.
            datasets[assoc.ref.dataId] = assoc.ref
        for detector in detectors:
            dataId = lsst.daf.butler.DataCoordinate.standardize(
                detector=detector, instrument=self.instrumentName,
                universe=gen3Butler.registry.dimensions
            )
            if dataId in datasets:
                gen3Defects = gen3Butler.getDirect(datasets[dataId])
                self.assertIsInstance(gen3Defects, lsst.ip.isr.Defects)
 

check_raw()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.check_raw	(		self,
			gen3Butler,
			exposure,
			detector
	)

Check that a raw was converted correctly.

Parameters
----------
gen3Butler : `lsst.daf.butler.Butler`
    The Butler to be tested.
exposure : `int`
    The exposure/vist identifier ``get`` from both butlers.
detector : `int`
    The detector identifier to ``get`` from both butlers.

Definition at line 175 of file convertTests.py.

    def check_raw(self, gen3Butler, exposure, detector):
        """Check that a raw was converted correctly.
 
        Parameters
        ----------
        gen3Butler : `lsst.daf.butler.Butler`
            The Butler to be tested.
        exposure : `int`
            The exposure/vist identifier ``get`` from both butlers.
        detector : `int`
            The detector identifier to ``get`` from both butlers.
        """
        dataIdGen2 = {self.detectorKey: detector, self.exposureKey: exposure}
        try:
            gen2Exposure = self.gen2Butler.get("raw", dataId=dataIdGen2)
        except lsst.daf.persistence.butlerExceptions.NoResults:
            # ignore datasets that don't actually exist in the gen2 butler.
            return
        dataIdGen3 = dict(detector=detector, exposure=exposure, instrument=self.instrumentName)
        gen3Exposure = gen3Butler.get("raw", dataId=dataIdGen3)
        # Check that we got an Exposure, but not what type; there is
        # inconsistency between different obs packages.
        self.assertIsInstance(gen3Exposure, lsst.afw.image.Exposure)
        self.assertEqual(gen3Exposure.getInfo().getDetector().getId(), detector)
        self.assertMaskedImagesEqual(gen2Exposure.maskedImage, gen3Exposure.maskedImage)
 

check_refcat()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.check_refcat	(		self,
			gen3Butler
	)

Test that each expected refcat is in the gen3 repo.

Parameters
----------
gen3Butler : `lsst.daf.butler.Butler`
    The Butler to be tested.

Definition at line 264 of file convertTests.py.

    def check_refcat(self, gen3Butler):
        """Test that each expected refcat is in the gen3 repo.
 
        Parameters
        ----------
        gen3Butler : `lsst.daf.butler.Butler`
            The Butler to be tested.
        """
        if len(self.refcats) > 0:
            for refcat in self.refcats:
                query = gen3Butler.registry.queryDatasets(refcat, collections=["refcats"])
                self.assertGreater(len(list(query)), 0,
                                   msg=f"refcat={refcat} has no entries in collection 'refcats'.")
 

instrumentClass()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.instrumentClass

(

self

)

The instrument class.

Definition at line 70 of file convertTests.py.

    def instrumentClass(self):
        """The instrument class."""
        return doImport(self.instrumentClassName)
 

instrumentClassName()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.instrumentClassName

(

self

)

Full path to the `Instrument` class of the data to be converted,
e.g. ``lsst.obs.decam.DarkEnergyCamera``.

Returns
-------
className : `str`
    The fully qualified instrument class name.

Definition at line 58 of file convertTests.py.

    def instrumentClassName(self):
        """Full path to the `Instrument` class of the data to be converted,
        e.g. ``lsst.obs.decam.DarkEnergyCamera``.
 
        Returns
        -------
        className : `str`
            The fully qualified instrument class name.
        """
        pass
 

instrumentName()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.instrumentName

(

self

)

Name of the instrument for the gen3 registry, e.g. "DECam".

Returns
-------
name : `str`
    The name of the instrument.

Definition at line 75 of file convertTests.py.

    def instrumentName(self):
        """Name of the instrument for the gen3 registry, e.g. "DECam".
 
        Returns
        -------
        name : `str`
            The name of the instrument.
        """
        return self.instrumentClass.getName()
 

setUp()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.setUp

(

self

)

Definition at line 140 of file convertTests.py.

    def setUp(self):
        self.gen3root = tempfile.mkdtemp()
        self.gen2Butler = lsst.daf.persistence.Butler(root=self.gen2root, calibRoot=self.gen2calib)
        self.collections = set(type(self).collections)
        self.collections.add(self.instrumentClass.makeDefaultRawIngestRunName())
        if len(self.refcats) > 0:
            self.collections.add("refcats")
        if self.skymapName is not None:
            self.collections.add("skymaps")
 
        # We always write a default calibration collection
        # containing at least the camera
        self.collections.add(self.instrumentClass.makeCollectionName("calib"))
 

tearDown()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.tearDown

(

self

)

Definition at line 154 of file convertTests.py.

    def tearDown(self):
        shutil.rmtree(self.gen3root, ignore_errors=True)
 

test_convert()

def lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.test_convert

(

self

)

Test that all data are converted correctly.

Definition at line 293 of file convertTests.py.

    def test_convert(self):
        """Test that all data are converted correctly.
        """
        self._run_convert()
        gen3Butler = lsst.daf.butler.Butler(self.gen3root,
                                            collections=self.instrumentClass.makeDefaultRawIngestRunName())
        self.check_collections(gen3Butler)
 
        # check every raw detector that the gen2 butler knows about
        detectors = self.gen2Butler.queryMetadata("raw", self.detectorKey)
        exposures = self.gen2Butler.queryMetadata("raw", self.exposureKey)
        for exposure, detector in itertools.product(exposures, detectors):
            with self.subTest(mode="raw", exposure=exposure, detector=detector):
                self.check_raw(gen3Butler, exposure, detector)
 
        self.check_refcat(gen3Butler)
        self.check_defects(gen3Butler, detectors)
        self.check_calibs(self.biasName, self.biases, gen3Butler)
        self.check_calibs(self.flatName, self.flats, gen3Butler)
        self.check_calibs(self.darkName, self.darks, gen3Butler)
 
 

Member Data Documentation

biases

list lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.biases = []

static

Definition at line 89 of file convertTests.py.

biasName

string lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.biasName = "bias"

static

Definition at line 92 of file convertTests.py.

calibFilterType

string lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.calibFilterType = "physical_filter"

static

Definition at line 130 of file convertTests.py.

collections

lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.collections = set()

static

Definition at line 115 of file convertTests.py.

config

lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.config = None

static

Definition at line 85 of file convertTests.py.

darkName

string lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.darkName = "dark"

static

Definition at line 104 of file convertTests.py.

darks

list lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.darks = []

static

Definition at line 101 of file convertTests.py.

detectorKey

string lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.detectorKey = "ccd"

static

Definition at line 124 of file convertTests.py.

exposureKey

string lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.exposureKey = "visit"

static

Definition at line 127 of file convertTests.py.

flatName

string lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.flatName = "flat"

static

Definition at line 98 of file convertTests.py.

flats

list lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.flats = []

static

Definition at line 95 of file convertTests.py.

gen2Butler

lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.gen2Butler

Definition at line 142 of file convertTests.py.

gen2calib

lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.gen2calib = None

static

Definition at line 53 of file convertTests.py.

gen2root

string lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.gen2root = ""

static

Definition at line 50 of file convertTests.py.

gen3root

lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.gen3root

Definition at line 141 of file convertTests.py.

kwargs

dictionary lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.kwargs = {}

static

Definition at line 107 of file convertTests.py.

refcats

list lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.refcats = []

static

Definition at line 111 of file convertTests.py.

skymapConfig

lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.skymapConfig = None

static

Definition at line 137 of file convertTests.py.

skymapName

lsst.obs.base.gen2to3.convertTests.ConvertGen2To3TestCase.skymapName = None

static

Definition at line 134 of file convertTests.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/0.2.1/Linux64/obs_base/21.0.0-27-gbbd0d29+ae871e0f33/python/lsst/obs/base/gen2to3/convertTests.py