LSST Applications  21.0.0-141-gec8a224e+c9dc52cf11,22.0.0+052faf71bd,22.0.0+1c4650f311,22.0.0+40ce427c77,22.0.0+5b6c068b1a,22.0.0+7589c3a021,22.0.0+b84b5806bd,22.0.1-1-g7d6de66+961d22230d,22.0.1-1-g87000a6+314cd8b7ea,22.0.1-1-g8760c09+052faf71bd,22.0.1-1-g8e32f31+5b6c068b1a,22.0.1-10-g779eefa+7d186d72ce,22.0.1-12-g3bd7ecb+bec1274511,22.0.1-15-g63cc0c1+2a7037787d,22.0.1-17-ge5a99e88+2652af7731,22.0.1-19-g88addfe+961d22230d,22.0.1-2-g1cb3e5b+9c7a6aab6f,22.0.1-2-g8ef0a89+961d22230d,22.0.1-2-g92698f7+1c4650f311,22.0.1-2-ga9b0f51+052faf71bd,22.0.1-2-gb66926d+5b6c068b1a,22.0.1-2-gcb770ba+0723a13595,22.0.1-2-ge470956+36d961e575,22.0.1-21-g532228a4+2ac85e833c,22.0.1-29-g184b6e44e+8b185d4e2d,22.0.1-3-g59f966b+89b97c2637,22.0.1-3-g8c1d971+f90df4c6d0,22.0.1-3-g997b569+d69a7aa2f8,22.0.1-3-gaaec9c0+4d194bf81c,22.0.1-4-g1930a60+283d9d2f1a,22.0.1-4-g5b7b756+c1283a92b8,22.0.1-4-g8623105+961d22230d,22.0.1-7-gba73697+283d9d2f1a,22.0.1-8-g47d23f5+39e151d95f,master-g5f2689bdc5+40ce427c77,w.2021.38
LSST Data Management Base Package
Public Member Functions | Static Public Member Functions | Public Attributes | List of all members
lsst.meas.algorithms.ingestIndexManager.IngestIndexManager Class Reference
Inheritance diagram for lsst.meas.algorithms.ingestIndexManager.IngestIndexManager:
lsst.meas.algorithms.ingestIndexManager.IngestGaiaManager

Public Member Functions

def __init__ (self, filenames, config, file_reader, indexer, schema, key_map, htmRange, addRefCatMetadata, log)
 
def run (self, inputFiles)
 
def getCatalog (self, pixelId, schema, nNewElements)
 

Static Public Member Functions

def computeCoord (row, ra_name, dec_name)
 

Public Attributes

 filenames
 
 config
 
 file_reader
 
 indexer
 
 schema
 
 key_map
 
 htmRange
 
 addRefCatMetadata
 
 log
 
 coord_err_unit
 
 nInputFiles
 

Detailed Description

Ingest a reference catalog from external files into a butler repository,
using a multiprocessing Pool to speed up the work.

Parameters
----------
filenames : `dict` [`int`, `str`]
    The HTM pixel id and filenames to ingest the catalog into.
config : `lsst.meas.algorithms.IngestIndexedReferenceConfig`
    The Task configuration holding the field names.
file_reader : `lsst.pipe.base.Task`
    The file reader to use to load the files.
indexer : `lsst.meas.algorithms.HtmIndexer`
    The class used to compute the HTM pixel per coordinate.
schema : `lsst.afw.table.Schema`
    The schema of the output catalog.
key_map : `dict` [`str`, `lsst.afw.table.Key`]
    The mapping from output field names to keys in the Schema.
htmRange : `tuple` [`int`]
    The start and end HTM pixel ids.
addRefCatMetadata : callable
    A function called to add extra metadata to each output Catalog.
log : `lsst.log.Log`
    The log to send messages to.

Definition at line 45 of file ingestIndexManager.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.__init__ (   self,
  filenames,
  config,
  file_reader,
  indexer,
  schema,
  key_map,
  htmRange,
  addRefCatMetadata,
  log 
)

Definition at line 73 of file ingestIndexManager.py.

74  schema, key_map, htmRange, addRefCatMetadata, log):
75  self.filenames = filenames
76  self.config = config
77  self.file_reader = file_reader
78  self.indexer = indexer
79  self.schema = schema
80  self.key_map = key_map
81  self.htmRange = htmRange
82  self.addRefCatMetadata = addRefCatMetadata
83  self.log = log
84  if self.config.coord_err_unit is not None:
85  # cache this to speed up coordinate conversions
86  self.coord_err_unit = u.Unit(self.config.coord_err_unit)
87 

Member Function Documentation

◆ computeCoord()

def lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.computeCoord (   row,
  ra_name,
  dec_name 
)
static 
Create an ICRS coord. from a row of a catalog being ingested.

Parameters
----------
row : `numpy.ndarray`
    Row from catalog being ingested.
ra_name : `str`
    Name of RA key in catalog being ingested.
dec_name : `str`
    Name of Dec key in catalog being ingested.

Returns
-------
coord : `lsst.geom.SpherePoint`
    ICRS coordinate.

Definition at line 235 of file ingestIndexManager.py.

235  def computeCoord(row, ra_name, dec_name):
236  """Create an ICRS coord. from a row of a catalog being ingested.
237 
238  Parameters
239  ----------
240  row : `numpy.ndarray`
241  Row from catalog being ingested.
242  ra_name : `str`
243  Name of RA key in catalog being ingested.
244  dec_name : `str`
245  Name of Dec key in catalog being ingested.
246 
247  Returns
248  -------
249  coord : `lsst.geom.SpherePoint`
250  ICRS coordinate.
251  """
252  return lsst.geom.SpherePoint(row[ra_name], row[dec_name], lsst.geom.degrees)
253 
lsst::geom::SpherePoint
Point in an unspecified spherical coordinate system.
Definition: SpherePoint.h:57

◆ getCatalog()

def lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.getCatalog (   self,
  pixelId,
  schema,
  nNewElements 
) 
Get a catalog from disk or create it if it doesn't exist.

Parameters
----------
pixelId : `dict`
    Identifier for catalog to retrieve
schema : `lsst.afw.table.Schema`
    Schema to use in catalog creation it does not exist.
nNewElements : `int`
    The number of new elements that will be added to the catalog,
    so space can be preallocated.

Returns
-------
catalog : `lsst.afw.table.SimpleCatalog`
    The new or read-and-resized catalog specified by `dataId`.

Definition at line 206 of file ingestIndexManager.py.

206  def getCatalog(self, pixelId, schema, nNewElements):
207  """Get a catalog from disk or create it if it doesn't exist.
208 
209  Parameters
210  ----------
211  pixelId : `dict`
212  Identifier for catalog to retrieve
213  schema : `lsst.afw.table.Schema`
214  Schema to use in catalog creation it does not exist.
215  nNewElements : `int`
216  The number of new elements that will be added to the catalog,
217  so space can be preallocated.
218 
219  Returns
220  -------
221  catalog : `lsst.afw.table.SimpleCatalog`
222  The new or read-and-resized catalog specified by `dataId`.
223  """
224  # This is safe, because we lock on this file before getCatalog is called.
225  if os.path.isfile(self.filenames[pixelId]):
226  catalog = afwTable.SimpleCatalog.readFits(self.filenames[pixelId])
227  catalog.resize(len(catalog) + nNewElements)
228  return catalog.copy(deep=True) # ensure contiguity, so that column-assignment works
229  catalog = afwTable.SimpleCatalog(schema)
230  catalog.resize(nNewElements)
231  self.addRefCatMetadata(catalog)
232  return catalog
233 
lsst::afw::table::SortedCatalogT
Custom catalog class for record/table subclasses that are guaranteed to have an ID,...
Definition: SortedCatalog.h:42

◆ run()

def lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.run (   self,
  inputFiles 
) 
Index a set of input files from a reference catalog, and write the
output to the appropriate filenames, in parallel.

Parameters
----------
inputFiles : `list`
    A list of file paths to read data from.

Definition at line 88 of file ingestIndexManager.py.

88  def run(self, inputFiles):
89  """Index a set of input files from a reference catalog, and write the
90  output to the appropriate filenames, in parallel.
91 
92  Parameters
93  ----------
94  inputFiles : `list`
95  A list of file paths to read data from.
96  """
97  global COUNTER, FILE_PROGRESS
98  self.nInputFiles = len(inputFiles)
99 
100  with multiprocessing.Manager() as manager:
101  COUNTER.value = 0
102  FILE_PROGRESS.value = 0
103  fileLocks = manager.dict()
104  self.log.info("Creating %s file locks.", self.htmRange[1] - self.htmRange[0])
105  for i in range(self.htmRange[0], self.htmRange[1]):
106  fileLocks[i] = manager.Lock()
107  self.log.info("File locks created.")
108  with multiprocessing.Pool(self.config.n_processes) as pool:
109  pool.starmap(self._ingestOneFile, zip(inputFiles, itertools.repeat(fileLocks)))
110 
lsst::ip::diffim.getTemplate.run
def run(self, coaddExposures, bbox, wcs)
Definition: getTemplate.py:603

Member Data Documentation

◆ addRefCatMetadata

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.addRefCatMetadata

Definition at line 82 of file ingestIndexManager.py.

◆ config

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.config

Definition at line 76 of file ingestIndexManager.py.

◆ coord_err_unit

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.coord_err_unit

Definition at line 86 of file ingestIndexManager.py.

◆ file_reader

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.file_reader

Definition at line 77 of file ingestIndexManager.py.

◆ filenames

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.filenames

Definition at line 75 of file ingestIndexManager.py.

◆ htmRange

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.htmRange

Definition at line 81 of file ingestIndexManager.py.

◆ indexer

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.indexer

Definition at line 78 of file ingestIndexManager.py.

◆ key_map

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.key_map

Definition at line 80 of file ingestIndexManager.py.

◆ log

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.log

Definition at line 83 of file ingestIndexManager.py.

◆ nInputFiles

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.nInputFiles

Definition at line 98 of file ingestIndexManager.py.

◆ schema

lsst.meas.algorithms.ingestIndexManager.IngestIndexManager.schema

Definition at line 79 of file ingestIndexManager.py.

The documentation for this class was generated from the following file:
Generated on Fri Sep 17 2021 07:21:21 for LSST Applications by doxygen 1.9.1