lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask Class Reference

Inheritance diagram for lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask:

Public Member Functions
	__init__ (self, *output_dir=None, **kwargs)
	run (self, inputFiles)
	makeSchema (self, dtype)

Public Attributes
	base_dir
	output_dir
	ingest_table_file
	indexer
	config
	file_reader
	log

Static Public Attributes
bool	canMultiprocess = False
	ConfigClass = ConvertReferenceCatalogConfig

Protected Member Functions
	_writeIngestHelperFile (self, result)
	_writeConfig (self)
	_getOutputFilenames (self, htm)
	_writeMasterSchema (self, inputfile)
	_reduce_kwargs (self)

Static Protected Attributes
str	_DefaultName = 'ConvertReferenceCatalogTask'

Detailed Description

Class for producing HTM-indexed reference catalogs from external
catalog data.

This implements an indexing scheme based on hierarchical triangular
mesh (HTM). The term index really means breaking the catalog into
localized chunks called shards.  In this case each shard contains
the entries from the catalog in a single HTM trixel

For producing catalogs this task makes the following assumptions
about the input catalogs:

- RA, Dec are in decimal degrees.
- Epoch is available in a column, in a format supported by astropy.time.Time.
- There are either no off-diagonal covariance terms, or there are all the
  five-dimensional covariance terms (between RA, Dec, proper motion, and
  parallax). In the latter case, a custom ``ConvertRefcatManager`` must
  exist to handle the covariance terms.

Parameters
----------
output_dir : `str`
    The path to write the output files to, in a subdirectory defined by
    ``DatasetConfig.ref_dataset_name``.

Definition at line 471 of file convertReferenceCatalog.py.

Constructor & Destructor Documentation

__init__()

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.__init__	(		self,
		*	output_dir = None,
		**	kwargs )

Definition at line 500 of file convertReferenceCatalog.py.

    def __init__(self, *, output_dir=None, **kwargs):
        super().__init__(**kwargs)
        if output_dir is None:
            raise RuntimeError("Must specify output_dir.")
        self.base_dir = output_dir
        self.output_dir = os.path.join(output_dir, self.config.dataset_config.ref_dataset_name)
        self.ingest_table_file = os.path.join(self.base_dir, "filename_to_htm.ecsv")
        self.indexer = IndexerRegistry[self.config.dataset_config.indexer.name](
            self.config.dataset_config.indexer.active)
        self.makeSubtask('file_reader')
 

Member Function Documentation

_getOutputFilenames()

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask._getOutputFilenames ( self, htm )

protected

Get filenames from the butler for each output htm pixel.

Parameters
----------
htm : `lsst.sphgeom.HtmPixelization`
    The HTM pixelization scheme to be used to build filenames.

Returns
-------
filenames : `list [str]`
    List of filenames to write each HTM pixel to.

Definition at line 559 of file convertReferenceCatalog.py.

    def _getOutputFilenames(self, htm):
        """Get filenames from the butler for each output htm pixel.
 
        Parameters
        ----------
        htm : `lsst.sphgeom.HtmPixelization`
            The HTM pixelization scheme to be used to build filenames.
 
        Returns
        -------
        filenames : `list [str]`
            List of filenames to write each HTM pixel to.
        """
        filenames = {}
        start, end = htm.universe()[0]
        path = os.path.join(self.output_dir, f"{self.indexer.htm}.fits")
        base = os.path.join(os.path.dirname(path), "%d"+os.path.splitext(path)[1])
        for pixelId in range(start, end):
            filenames[pixelId] = base % pixelId
 
        return filenames
 

_reduce_kwargs()

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask._reduce_kwargs ( self )

protected

Definition at line 631 of file convertReferenceCatalog.py.

    def _reduce_kwargs(self):
        # Need to be able to pickle this class to use the multiprocess manager.
        kwargs = super()._reduce_kwargs()
        kwargs['output_dir'] = self.base_dir
        return kwargs
 
 

_writeConfig()

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask._writeConfig ( self )

protected

Write the config that was used to generate the refcat.

Definition at line 553 of file convertReferenceCatalog.py.

    def _writeConfig(self):
        """Write the config that was used to generate the refcat."""
        filename = os.path.join(self.output_dir, "config.py")
        with open(filename, 'w') as file:
            self.config.dataset_config.saveToStream(file)
 

_writeIngestHelperFile()

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask._writeIngestHelperFile ( self, result )

protected

Write the astropy table containing the htm->filename relationship,
used for the ``butler ingest-files`` command after this task completes.

Definition at line 543 of file convertReferenceCatalog.py.

    def _writeIngestHelperFile(self, result):
        """Write the astropy table containing the htm->filename relationship,
        used for the ``butler ingest-files`` command after this task completes.
        """
        dimension = f"htm{self.config.dataset_config.indexer.active.depth}"
        table = astropy.table.Table(names=("filename", dimension), dtype=('str', 'int'))
        for key in result:
            table.add_row((result[key], key))
        table.write(self.ingest_table_file)
 

_writeMasterSchema()

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask._writeMasterSchema ( self, inputfile )

protected

Generate and save the master catalog schema.

Parameters
----------
inputfile : `str`
    An input file to read to get the input dtype.

Definition at line 614 of file convertReferenceCatalog.py.

    def _writeMasterSchema(self, inputfile):
        """Generate and save the master catalog schema.
 
        Parameters
        ----------
        inputfile : `str`
            An input file to read to get the input dtype.
        """
        arr = self.file_reader.run(inputfile)
        schema, key_map = self.makeSchema(arr.dtype)
 
        catalog = lsst.afw.table.SimpleCatalog(schema)
        addRefCatMetadata(catalog)
        outputfile = os.path.join(self.output_dir, "master_schema.fits")
        catalog.writeFits(outputfile)
        return schema, key_map
 

makeSchema()

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.makeSchema	(		self,
			dtype )

Make the schema to use in constructing the persisted catalogs.

Parameters
----------
dtype : `numpy.dtype`
    Data type describing each entry in ``config.extra_col_names``
    for the catalogs being ingested.

Returns
-------
schemaAndKeyMap : `tuple` of (`lsst.afw.table.Schema`, `dict`)
    A tuple containing two items:
    - The schema for the output source catalog.
    - A map of catalog keys to use in filling the record

Definition at line 581 of file convertReferenceCatalog.py.

    def makeSchema(self, dtype):
        """Make the schema to use in constructing the persisted catalogs.
 
        Parameters
        ----------
        dtype : `numpy.dtype`
            Data type describing each entry in ``config.extra_col_names``
            for the catalogs being ingested.
 
        Returns
        -------
        schemaAndKeyMap : `tuple` of (`lsst.afw.table.Schema`, `dict`)
            A tuple containing two items:
            - The schema for the output source catalog.
            - A map of catalog keys to use in filling the record
        """
        # make a schema with the standard fields
        schema = _makeSchema(
            filterNameList=self.config.mag_column_list,
            addCentroid=False,
            addIsPhotometric=bool(self.config.is_photometric_name),
            addIsResolved=bool(self.config.is_resolved_name),
            addIsVariable=bool(self.config.is_variable_name),
            fullPositionInformation=self.config.full_position_information,
        )
        keysToSkip = set(("id", "centroid_x", "centroid_y", "hasCentroid"))
        key_map = {fieldName: schema[fieldName].asKey() for fieldName in schema.getOrderedNames()
                   if fieldName not in keysToSkip}
 
        _addExtraColumnsToSchema(schema, dtype, self.config.extra_col_names, key_map)
 
        return schema, key_map
 

run()

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.run	(		self,
			inputFiles )

Index a set of files comprising a reference catalog.

Outputs are persisted in the butler repository.

Parameters
----------
inputFiles : `list`
    A list of file paths to read.

Definition at line 511 of file convertReferenceCatalog.py.

    def run(self, inputFiles):
        """Index a set of files comprising a reference catalog.
 
        Outputs are persisted in the butler repository.
 
        Parameters
        ----------
        inputFiles : `list`
            A list of file paths to read.
        """
        # Create the output path, if it doesn't exist; fail if the path exists:
        # we don't want to accidentally append to existing files.
        pathlib.Path(self.output_dir).mkdir(exist_ok=False)
 
        schema, key_map = self._writeMasterSchema(inputFiles[0])
        # create an HTM we can interrogate about pixel ids
        htm = lsst.sphgeom.HtmPixelization(self.indexer.htm.get_depth())
        filenames = self._getOutputFilenames(htm)
        worker = self.config.manager.target(filenames,
                                            self.config,
                                            self.file_reader,
                                            self.indexer,
                                            schema,
                                            key_map,
                                            htm.universe()[0],
                                            addRefCatMetadata,
                                            self.log)
        result = worker.run(inputFiles)
 
        self._writeConfig()
        self._writeIngestHelperFile(result)
 

Member Data Documentation

_DefaultName

str lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask._DefaultName = 'ConvertReferenceCatalogTask'

staticprotected

Definition at line 498 of file convertReferenceCatalog.py.

base_dir

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.base_dir

Definition at line 504 of file convertReferenceCatalog.py.

canMultiprocess

bool lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.canMultiprocess = False

static

Definition at line 496 of file convertReferenceCatalog.py.

config

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.config

Definition at line 530 of file convertReferenceCatalog.py.

ConfigClass

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.ConfigClass = ConvertReferenceCatalogConfig

static

Definition at line 497 of file convertReferenceCatalog.py.

file_reader

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.file_reader

Definition at line 531 of file convertReferenceCatalog.py.

indexer

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.indexer

Definition at line 507 of file convertReferenceCatalog.py.

ingest_table_file

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.ingest_table_file

Definition at line 506 of file convertReferenceCatalog.py.

log

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.log

Definition at line 537 of file convertReferenceCatalog.py.

output_dir

lsst.meas.algorithms.convertReferenceCatalog.ConvertReferenceCatalogTask.output_dir

Definition at line 505 of file convertReferenceCatalog.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-8.0.0/Linux64/meas_algorithms/ge6526c86ff+3a7c1ac5f1/python/lsst/meas/algorithms/convertReferenceCatalog.py