LSST Applications 24.1.5,g02d81e74bb+fa3a7a026e,g180d380827+a53a32eff8,g2079a07aa2+86d27d4dc4,g2305ad1205+c0501b3732,g295015adf3+7d3e92f0ec,g2bbee38e9b+0e5473021a,g337abbeb29+0e5473021a,g33d1c0ed96+0e5473021a,g3a166c0a6a+0e5473021a,g3ddfee87b4+5dd1654d75,g48712c4677+3bf1020dcb,g487adcacf7+065c13d9cf,g50ff169b8f+96c6868917,g52b1c1532d+585e252eca,g591dd9f2cf+d7ac436cfb,g5a732f18d5+53520f316c,g64a986408d+fa3a7a026e,g858d7b2824+fa3a7a026e,g8a8a8dda67+585e252eca,g99cad8db69+a5a909b84f,g9ddcbc5298+9a081db1e4,ga1e77700b3+15fc3df1f7,ga8c6da7877+4cf350ccb2,gb0e22166c9+60f28cb32d,gba4ed39666+c2a2e4ac27,gbb8dafda3b+f991a0b59f,gc120e1dc64+9ccbfdb8be,gc28159a63d+0e5473021a,gcf0d15dbbd+5dd1654d75,gd96a1ce819+42fd0ee607,gdaeeff99f8+f9a426f77a,ge6526c86ff+0d71447b4b,ge79ae78c31+0e5473021a,gee10cc3b42+585e252eca,gff1a9f87cc+fa3a7a026e
LSST Data Management Base Package
|
Public Member Functions | |
__init__ (self, filenames, config, file_reader, indexer, schema, key_map, htmRange, addRefCatMetadata, log) | |
run (self, inputFiles) | |
getCatalog (self, pixelId, schema, nNewElements) | |
Static Public Member Functions | |
computeCoord (row, ra_name, dec_name) | |
Public Attributes | |
filenames | |
config | |
file_reader | |
indexer | |
schema | |
key_map | |
htmRange | |
addRefCatMetadata | |
log | |
coord_err_unit | |
nInputFiles | |
Static Public Attributes | |
ConfigClass = ConvertRefcatManagerConfig | |
Protected Member Functions | |
_convertOneFile (self, filename, fileLocks) | |
_doOnePixel (self, inputData, matchedPixels, pixelId, fluxes, coordErr) | |
_setIds (self, inputData, catalog) | |
_getCoordErr (self, inputData) | |
_setFlags (self, record, row) | |
_getFluxes (self, inputData) | |
_setProperMotion (self, record, row) | |
_setParallax (self, record, row) | |
_epochToMjdTai (self, nativeEpoch) | |
_setCoordinateCovariance (self, record, row) | |
_setExtra (self, record, row) | |
_fillRecord (self, record, row) | |
Static Protected Attributes | |
list | _flags = ['photometric', 'resolved', 'variable'] |
str | _DefaultName = 'convertRefcatManager' |
Convert a reference catalog from external files into the LSST HTM sharded format, using a multiprocessing Pool to speed up the work. Parameters ---------- filenames : `dict` [`int`, `str`] The HTM pixel id and filenames to convert the catalog into. config : `lsst.meas.algorithms.ConvertReferenceCatalogConfig` The Task configuration holding the field names. file_reader : `lsst.pipe.base.Task` The file reader to use to load the files. indexer : `lsst.meas.algorithms.HtmIndexer` The class used to compute the HTM pixel per coordinate. schema : `lsst.afw.table.Schema` The schema of the output catalog. key_map : `dict` [`str`, `lsst.afw.table.Key`] The mapping from output field names to keys in the Schema. htmRange : `tuple` [`int`] The start and end HTM pixel ids. addRefCatMetadata : callable A function called to add extra metadata to each output Catalog. log : `lsst.log.Log` or `logging.Logger` The log to send messages to.
Definition at line 54 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.__init__ | ( | self, | |
filenames, | |||
config, | |||
file_reader, | |||
indexer, | |||
schema, | |||
key_map, | |||
htmRange, | |||
addRefCatMetadata, | |||
log ) |
Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager.
Definition at line 84 of file convertRefcatManager.py.
|
protected |
Read and process one file, and write its records to the correct indexed files, while handling exceptions in a useful way so that they don't get swallowed by the multiprocess pool. Parameters ---------- filename : `str` The file to process. fileLocks : `dict` [`int`, `multiprocessing.Lock`] A Lock for each HTM pixel; each pixel gets one file written, and we need to block when one process is accessing that file. Returns ------- pixels, files : `list` [`int`] The pixel ids that were written to.
Definition at line 134 of file convertRefcatManager.py.
|
protected |
Process one HTM pixel, appending to an existing catalog or creating a new catalog, as needed. Parameters ---------- inputData : `numpy.ndarray` The data from one input file. matchedPixels : `numpy.ndarray` The row-matched pixel indexes corresponding to ``inputData``. pixelId : `int` The pixel index we are currently processing. fluxes : `dict` [`str`, `numpy.ndarray`] The values that will go into the flux and fluxErr fields in the output catalog. coordErr : `dict` [`str`, `numpy.ndarray`] The values that will go into the coord_raErr, coord_decErr, and coord_ra_dec_Cov fields in the output catalog (in radians).
Definition at line 185 of file convertRefcatManager.py.
|
protected |
Convert an epoch in native format to TAI MJD (a float).
Definition at line 398 of file convertRefcatManager.py.
|
protected |
Fill a record in an indexed catalog to be persisted. Parameters ---------- record : `lsst.afw.table.SimpleRecord` Row from indexed catalog to modify. row : structured `numpy.array` Row from catalog being converted.
Definition at line 444 of file convertRefcatManager.py.
|
protected |
Compute the ra/dec error fields that will go into the output catalog. Parameters ---------- inputData : `numpy.ndarray` The input data to compute fluxes for. Returns ------- coordErr : `dict` [`str`, `numpy.ndarray`] The values that will go into the coord_raErr, coord_decErr, fields in the output catalog (in radians). Notes ----- This does not handle the ra/dec covariance field, ``coord_ra_coord_dec_Cov``. That field is handled in `_setCoordinateCovariance`.
Definition at line 294 of file convertRefcatManager.py.
|
protected |
Compute the flux fields that will go into the output catalog. Parameters ---------- inputData : `numpy.ndarray` The input data to compute fluxes for. Returns ------- fluxes : `dict` [`str`, `numpy.ndarray`] The values that will go into the flux and fluxErr fields in the output catalog.
Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager, and lsst.meas.algorithms.convertRefcatManager.ConvertGaiaXpManager.
Definition at line 338 of file convertRefcatManager.py.
|
protected |
Set the off-diagonal position covariance in a record of an indexed catalog. There is no generic way to determine covariance. Override this method in a subclass specialized for your dataset. Parameters ---------- record : `lsst.afw.table.SimpleRecord` Row from indexed catalog to modify. row : structured `numpy.array` Row from catalog being converted.
Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager.
Definition at line 404 of file convertRefcatManager.py.
|
protected |
Set extra data fields in a record of an indexed catalog. Parameters ---------- record : `lsst.afw.table.SimpleRecord` Row from indexed catalog to modify. row : structured `numpy.array` Row from catalog being converted.
Definition at line 421 of file convertRefcatManager.py.
|
protected |
Set flags in an output record. Parameters ---------- record : `lsst.afw.table.SimpleRecord` Row from indexed catalog to modify. row : `numpy.ndarray` Row from catalog being converted.
Definition at line 322 of file convertRefcatManager.py.
|
protected |
Fill the `id` field of catalog with a running index, filling the last values up to the length of ``inputData``. Fill with `self.config.id_name` if specified, otherwise use the global running counter value. Parameters ---------- inputData : `numpy.ndarray` The input data that is being processed. catalog : `lsst.afw.table.SimpleCatalog` The output catalog to fill the ids.
Definition at line 223 of file convertRefcatManager.py.
|
protected |
Set the parallax fields in a record of a refcat.
Definition at line 389 of file convertRefcatManager.py.
|
protected |
Set proper motion fields in a record of an indexed catalog. The proper motions are read from the specified columns, scaled appropriately, and installed in the appropriate columns of the output. Parameters ---------- record : `lsst.afw.table.SimpleRecord` Row from indexed catalog to modify. row : structured `numpy.array` Row from catalog being converted.
Definition at line 365 of file convertRefcatManager.py.
|
static |
Create an ICRS coord. from a row of a catalog being converted. Parameters ---------- row : `numpy.ndarray` Row from catalog being converted. ra_name : `str` Name of RA key in catalog being converted. dec_name : `str` Name of Dec key in catalog being converted. Returns ------- coord : `lsst.geom.SpherePoint` ICRS coordinate.
Definition at line 275 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.getCatalog | ( | self, | |
pixelId, | |||
schema, | |||
nNewElements ) |
Get a catalog from disk or create it if it doesn't exist. Parameters ---------- pixelId : `dict` Identifier for catalog to retrieve schema : `lsst.afw.table.Schema` Schema to use in catalog creation it does not exist. nNewElements : `int` The number of new elements that will be added to the catalog, so space can be preallocated. Returns ------- catalog : `lsst.afw.table.SimpleCatalog` The new or read-and-resized catalog specified by `dataId`.
Definition at line 246 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.run | ( | self, | |
inputFiles ) |
Index a set of input files from a reference catalog, and write the output to the appropriate filenames, in parallel. Parameters ---------- inputFiles : `list` A list of file paths to read data from. Returns ------- output : `dict` [`int`, `str`] The htm ids and the filenames that were written to.
Definition at line 100 of file convertRefcatManager.py.
|
staticprotected |
Definition at line 81 of file convertRefcatManager.py.
|
staticprotected |
Definition at line 80 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.addRefCatMetadata |
Definition at line 93 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.config |
Definition at line 87 of file convertRefcatManager.py.
|
static |
Definition at line 82 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.coord_err_unit |
Definition at line 98 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.file_reader |
Definition at line 88 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.filenames |
Definition at line 86 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.htmRange |
Definition at line 92 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.indexer |
Definition at line 89 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.key_map |
Definition at line 91 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.log |
Definition at line 94 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.nInputFiles |
Definition at line 115 of file convertRefcatManager.py.
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.schema |
Definition at line 90 of file convertRefcatManager.py.