LSST Applications  21.0.0-172-gfb10e10a+18fedfabac,22.0.0+297cba6710,22.0.0+80564b0ff1,22.0.0+8d77f4f51a,22.0.0+a28f4c53b1,22.0.0+dcf3732eb2,22.0.1-1-g7d6de66+2a20fdde0d,22.0.1-1-g8e32f31+297cba6710,22.0.1-1-geca5380+7fa3b7d9b6,22.0.1-12-g44dc1dc+2a20fdde0d,22.0.1-15-g6a90155+515f58c32b,22.0.1-16-g9282f48+790f5f2caa,22.0.1-2-g92698f7+dcf3732eb2,22.0.1-2-ga9b0f51+7fa3b7d9b6,22.0.1-2-gd1925c9+bf4f0e694f,22.0.1-24-g1ad7a390+a9625a72a8,22.0.1-25-g5bf6245+3ad8ecd50b,22.0.1-25-gb120d7b+8b5510f75f,22.0.1-27-g97737f7+2a20fdde0d,22.0.1-32-gf62ce7b1+aa4237961e,22.0.1-4-g0b3f228+2a20fdde0d,22.0.1-4-g243d05b+871c1b8305,22.0.1-4-g3a563be+32dcf1063f,22.0.1-4-g44f2e3d+9e4ab0f4fa,22.0.1-42-gca6935d93+ba5e5ca3eb,22.0.1-5-g15c806e+85460ae5f3,22.0.1-5-g58711c4+611d128589,22.0.1-5-g75bb458+99c117b92f,22.0.1-6-g1c63a23+7fa3b7d9b6,22.0.1-6-g50866e6+84ff5a128b,22.0.1-6-g8d3140d+720564cf76,22.0.1-6-gd805d02+cc5644f571,22.0.1-8-ge5750ce+85460ae5f3,master-g6e05de7fdc+babf819c66,master-g99da0e417a+8d77f4f51a,w.2021.48
LSST Data Management Base Package
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | List of all members
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager Class Reference
Inheritance diagram for lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager:
lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager

Public Member Functions

def __init__ (self, filenames, config, file_reader, indexer, schema, key_map, htmRange, addRefCatMetadata, log)
 
def run (self, inputFiles)
 
def getCatalog (self, pixelId, schema, nNewElements)
 

Static Public Member Functions

def computeCoord (row, ra_name, dec_name)
 

Public Attributes

 filenames
 
 config
 
 file_reader
 
 indexer
 
 schema
 
 key_map
 
 htmRange
 
 addRefCatMetadata
 
 log
 
 coord_err_unit
 
 nInputFiles
 

Static Public Attributes

 ConfigClass = ConvertRefcatManagerConfig
 

Detailed Description

Convert a reference catalog from external files into the LSST HTM sharded
format, using a multiprocessing Pool to speed up the work.

Parameters
----------
filenames : `dict` [`int`, `str`]
    The HTM pixel id and filenames to convert the catalog into.
config : `lsst.meas.algorithms.ConvertReferenceCatalogConfig`
    The Task configuration holding the field names.
file_reader : `lsst.pipe.base.Task`
    The file reader to use to load the files.
indexer : `lsst.meas.algorithms.HtmIndexer`
    The class used to compute the HTM pixel per coordinate.
schema : `lsst.afw.table.Schema`
    The schema of the output catalog.
key_map : `dict` [`str`, `lsst.afw.table.Key`]
    The mapping from output field names to keys in the Schema.
htmRange : `tuple` [`int`]
    The start and end HTM pixel ids.
addRefCatMetadata : callable
    A function called to add extra metadata to each output Catalog.
log : `lsst.log.Log` or `logging.Logger`
    The log to send messages to.

Definition at line 53 of file convertRefcatManager.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.__init__ (   self,
  filenames,
  config,
  file_reader,
  indexer,
  schema,
  key_map,
  htmRange,
  addRefCatMetadata,
  log 
)

Definition at line 83 of file convertRefcatManager.py.

84  schema, key_map, htmRange, addRefCatMetadata, log):
85  self.filenames = filenames
86  self.config = config
87  self.file_reader = file_reader
88  self.indexer = indexer
89  self.schema = schema
90  self.key_map = key_map
91  self.htmRange = htmRange
92  self.addRefCatMetadata = addRefCatMetadata
93  self.log = log
94  if self.config.coord_err_unit is not None:
95  # cache this to speed up coordinate conversions
96  self.coord_err_unit = u.Unit(self.config.coord_err_unit)
97 

Member Function Documentation

◆ computeCoord()

def lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.computeCoord (   row,
  ra_name,
  dec_name 
)
static
Create an ICRS coord. from a row of a catalog being converted.

Parameters
----------
row : `numpy.ndarray`
    Row from catalog being converted.
ra_name : `str`
    Name of RA key in catalog being converted.
dec_name : `str`
    Name of Dec key in catalog being converted.

Returns
-------
coord : `lsst.geom.SpherePoint`
    ICRS coordinate.

Definition at line 257 of file convertRefcatManager.py.

257  def computeCoord(row, ra_name, dec_name):
258  """Create an ICRS coord. from a row of a catalog being converted.
259 
260  Parameters
261  ----------
262  row : `numpy.ndarray`
263  Row from catalog being converted.
264  ra_name : `str`
265  Name of RA key in catalog being converted.
266  dec_name : `str`
267  Name of Dec key in catalog being converted.
268 
269  Returns
270  -------
271  coord : `lsst.geom.SpherePoint`
272  ICRS coordinate.
273  """
274  return lsst.geom.SpherePoint(row[ra_name], row[dec_name], lsst.geom.degrees)
275 
Point in an unspecified spherical coordinate system.
Definition: SpherePoint.h:57

◆ getCatalog()

def lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.getCatalog (   self,
  pixelId,
  schema,
  nNewElements 
)
Get a catalog from disk or create it if it doesn't exist.

Parameters
----------
pixelId : `dict`
    Identifier for catalog to retrieve
schema : `lsst.afw.table.Schema`
    Schema to use in catalog creation it does not exist.
nNewElements : `int`
    The number of new elements that will be added to the catalog,
    so space can be preallocated.

Returns
-------
catalog : `lsst.afw.table.SimpleCatalog`
    The new or read-and-resized catalog specified by `dataId`.

Definition at line 228 of file convertRefcatManager.py.

228  def getCatalog(self, pixelId, schema, nNewElements):
229  """Get a catalog from disk or create it if it doesn't exist.
230 
231  Parameters
232  ----------
233  pixelId : `dict`
234  Identifier for catalog to retrieve
235  schema : `lsst.afw.table.Schema`
236  Schema to use in catalog creation it does not exist.
237  nNewElements : `int`
238  The number of new elements that will be added to the catalog,
239  so space can be preallocated.
240 
241  Returns
242  -------
243  catalog : `lsst.afw.table.SimpleCatalog`
244  The new or read-and-resized catalog specified by `dataId`.
245  """
246  # This is safe, because we lock on this file before getCatalog is called.
247  if os.path.isfile(self.filenames[pixelId]):
248  catalog = afwTable.SimpleCatalog.readFits(self.filenames[pixelId])
249  catalog.resize(len(catalog) + nNewElements)
250  return catalog.copy(deep=True) # ensure contiguity, so that column-assignment works
251  catalog = afwTable.SimpleCatalog(schema)
252  catalog.resize(nNewElements)
253  self.addRefCatMetadata(catalog)
254  return catalog
255 
Custom catalog class for record/table subclasses that are guaranteed to have an ID,...
Definition: SortedCatalog.h:42

◆ run()

def lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.run (   self,
  inputFiles 
)
Index a set of input files from a reference catalog, and write the
output to the appropriate filenames, in parallel.

Parameters
----------
inputFiles : `list`
    A list of file paths to read data from.

Returns
-------
output : `dict` [`int`, `str`]
    The htm ids and the filenames that were written to.

Definition at line 98 of file convertRefcatManager.py.

98  def run(self, inputFiles):
99  """Index a set of input files from a reference catalog, and write the
100  output to the appropriate filenames, in parallel.
101 
102  Parameters
103  ----------
104  inputFiles : `list`
105  A list of file paths to read data from.
106 
107  Returns
108  -------
109  output : `dict` [`int`, `str`]
110  The htm ids and the filenames that were written to.
111  """
112  global COUNTER, FILE_PROGRESS
113  self.nInputFiles = len(inputFiles)
114 
115  with multiprocessing.Manager() as manager:
116  COUNTER.value = 0
117  FILE_PROGRESS.value = 0
118  fileLocks = manager.dict()
119  self.log.info("Creating %s file locks.", self.htmRange[1] - self.htmRange[0])
120  for i in range(self.htmRange[0], self.htmRange[1]):
121  fileLocks[i] = manager.Lock()
122  self.log.info("File locks created.")
123  with multiprocessing.Pool(self.config.n_processes) as pool:
124  result = pool.starmap(self._convertOneFile, zip(inputFiles, itertools.repeat(fileLocks)))
125  return {id: self.filenames[id] for item in result for id in item}
126 
def run(self, coaddExposures, bbox, wcs)
Definition: getTemplate.py:603

Member Data Documentation

◆ addRefCatMetadata

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.addRefCatMetadata

Definition at line 92 of file convertRefcatManager.py.

◆ config

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.config

Definition at line 86 of file convertRefcatManager.py.

◆ ConfigClass

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.ConfigClass = ConvertRefcatManagerConfig
static

Definition at line 81 of file convertRefcatManager.py.

◆ coord_err_unit

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.coord_err_unit

Definition at line 96 of file convertRefcatManager.py.

◆ file_reader

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.file_reader

Definition at line 87 of file convertRefcatManager.py.

◆ filenames

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.filenames

Definition at line 85 of file convertRefcatManager.py.

◆ htmRange

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.htmRange

Definition at line 91 of file convertRefcatManager.py.

◆ indexer

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.indexer

Definition at line 88 of file convertRefcatManager.py.

◆ key_map

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.key_map

Definition at line 90 of file convertRefcatManager.py.

◆ log

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.log

Definition at line 93 of file convertRefcatManager.py.

◆ nInputFiles

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.nInputFiles

Definition at line 113 of file convertRefcatManager.py.

◆ schema

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.schema

Definition at line 89 of file convertRefcatManager.py.


The documentation for this class was generated from the following file: