LSST Applications 24.1.5,g02d81e74bb+fa3a7a026e,g180d380827+a53a32eff8,g2079a07aa2+86d27d4dc4,g2305ad1205+c0501b3732,g295015adf3+7d3e92f0ec,g2bbee38e9b+0e5473021a,g337abbeb29+0e5473021a,g33d1c0ed96+0e5473021a,g3a166c0a6a+0e5473021a,g3ddfee87b4+5dd1654d75,g48712c4677+3bf1020dcb,g487adcacf7+065c13d9cf,g50ff169b8f+96c6868917,g52b1c1532d+585e252eca,g591dd9f2cf+d7ac436cfb,g5a732f18d5+53520f316c,g64a986408d+fa3a7a026e,g858d7b2824+fa3a7a026e,g8a8a8dda67+585e252eca,g99cad8db69+a5a909b84f,g9ddcbc5298+9a081db1e4,ga1e77700b3+15fc3df1f7,ga8c6da7877+4cf350ccb2,gb0e22166c9+60f28cb32d,gba4ed39666+c2a2e4ac27,gbb8dafda3b+f991a0b59f,gc120e1dc64+9ccbfdb8be,gc28159a63d+0e5473021a,gcf0d15dbbd+5dd1654d75,gd96a1ce819+42fd0ee607,gdaeeff99f8+f9a426f77a,ge6526c86ff+0d71447b4b,ge79ae78c31+0e5473021a,gee10cc3b42+585e252eca,gff1a9f87cc+fa3a7a026e
LSST Data Management Base Package
Loading...
Searching...
No Matches
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Protected Member Functions | Static Protected Attributes | List of all members
lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager Class Reference
Inheritance diagram for lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager:
lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager lsst.meas.algorithms.convertRefcatManager.ConvertGaiaXpManager

Public Member Functions

 __init__ (self, filenames, config, file_reader, indexer, schema, key_map, htmRange, addRefCatMetadata, log)
 
 run (self, inputFiles)
 
 getCatalog (self, pixelId, schema, nNewElements)
 

Static Public Member Functions

 computeCoord (row, ra_name, dec_name)
 

Public Attributes

 filenames
 
 config
 
 file_reader
 
 indexer
 
 schema
 
 key_map
 
 htmRange
 
 addRefCatMetadata
 
 log
 
 coord_err_unit
 
 nInputFiles
 

Static Public Attributes

 ConfigClass = ConvertRefcatManagerConfig
 

Protected Member Functions

 _convertOneFile (self, filename, fileLocks)
 
 _doOnePixel (self, inputData, matchedPixels, pixelId, fluxes, coordErr)
 
 _setIds (self, inputData, catalog)
 
 _getCoordErr (self, inputData)
 
 _setFlags (self, record, row)
 
 _getFluxes (self, inputData)
 
 _setProperMotion (self, record, row)
 
 _setParallax (self, record, row)
 
 _epochToMjdTai (self, nativeEpoch)
 
 _setCoordinateCovariance (self, record, row)
 
 _setExtra (self, record, row)
 
 _fillRecord (self, record, row)
 

Static Protected Attributes

list _flags = ['photometric', 'resolved', 'variable']
 
str _DefaultName = 'convertRefcatManager'
 

Detailed Description

Convert a reference catalog from external files into the LSST HTM sharded
format, using a multiprocessing Pool to speed up the work.

Parameters
----------
filenames : `dict` [`int`, `str`]
    The HTM pixel id and filenames to convert the catalog into.
config : `lsst.meas.algorithms.ConvertReferenceCatalogConfig`
    The Task configuration holding the field names.
file_reader : `lsst.pipe.base.Task`
    The file reader to use to load the files.
indexer : `lsst.meas.algorithms.HtmIndexer`
    The class used to compute the HTM pixel per coordinate.
schema : `lsst.afw.table.Schema`
    The schema of the output catalog.
key_map : `dict` [`str`, `lsst.afw.table.Key`]
    The mapping from output field names to keys in the Schema.
htmRange : `tuple` [`int`]
    The start and end HTM pixel ids.
addRefCatMetadata : callable
    A function called to add extra metadata to each output Catalog.
log : `lsst.log.Log` or `logging.Logger`
    The log to send messages to.

Definition at line 54 of file convertRefcatManager.py.

Constructor & Destructor Documentation

◆ __init__()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.__init__ ( self,
filenames,
config,
file_reader,
indexer,
schema,
key_map,
htmRange,
addRefCatMetadata,
log )

Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager.

Definition at line 84 of file convertRefcatManager.py.

85 schema, key_map, htmRange, addRefCatMetadata, log):
86 self.filenames = filenames
87 self.config = config
88 self.file_reader = file_reader
89 self.indexer = indexer
90 self.schema = schema
91 self.key_map = key_map
92 self.htmRange = htmRange
93 self.addRefCatMetadata = addRefCatMetadata
94 self.log = log
95
96 if self.config.coord_err_unit is not None:
97 # cache this to speed up coordinate conversions.
98 self.coord_err_unit = u.Unit(self.config.coord_err_unit)
99

Member Function Documentation

◆ _convertOneFile()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._convertOneFile ( self,
filename,
fileLocks )
protected
Read and process one file, and write its records to the correct
indexed files, while handling exceptions in a useful way so that they
don't get swallowed by the multiprocess pool.

Parameters
----------
filename : `str`
    The file to process.
fileLocks : `dict` [`int`, `multiprocessing.Lock`]
    A Lock for each HTM pixel; each pixel gets one file written, and
    we need to block when one process is accessing that file.

Returns
-------
pixels, files : `list` [`int`]
    The pixel ids that were written to.

Definition at line 134 of file convertRefcatManager.py.

134 def _convertOneFile(self, filename, fileLocks):
135 """Read and process one file, and write its records to the correct
136 indexed files, while handling exceptions in a useful way so that they
137 don't get swallowed by the multiprocess pool.
138
139 Parameters
140 ----------
141 filename : `str`
142 The file to process.
143 fileLocks : `dict` [`int`, `multiprocessing.Lock`]
144 A Lock for each HTM pixel; each pixel gets one file written, and
145 we need to block when one process is accessing that file.
146
147 Returns
148 -------
149 pixels, files : `list` [`int`]
150 The pixel ids that were written to.
151 """
152 global FILE_PROGRESS
153 self.log.debug("Processing: %s", filename)
154 inputData = self.file_reader.run(filename)
155 try:
156 fluxes = self._getFluxes(inputData)
157 coordErr = self._getCoordErr(inputData)
158 matchedPixels = self.indexer.indexPoints(inputData[self.config.ra_name],
159 inputData[self.config.dec_name])
160 except Exception:
161 self.log.error("Failure preparing data for: %s", filename)
162 raise
163
164 pixel_ids = set(matchedPixels)
165 for pixelId in pixel_ids:
166 with fileLocks[pixelId]:
167 try:
168 self._doOnePixel(inputData, matchedPixels, pixelId, fluxes, coordErr)
169 except Exception:
170 self.log.error("Failure processing filename: %s, pixelId: %s",
171 filename, pixelId)
172 raise
173 with FILE_PROGRESS.get_lock():
174 oldPercent = 100 * FILE_PROGRESS.value / self.nInputFiles
175 FILE_PROGRESS.value += 1
176 percent = 100 * FILE_PROGRESS.value / self.nInputFiles
177 # only log each "new percent"
178 if np.floor(percent) - np.floor(oldPercent) >= 1:
179 self.log.info("Completed %d / %d files: %d %% complete ",
180 FILE_PROGRESS.value,
181 self.nInputFiles,
182 percent)
183 return pixel_ids
184
daf::base::PropertySet * set
Definition fits.cc:931

◆ _doOnePixel()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._doOnePixel ( self,
inputData,
matchedPixels,
pixelId,
fluxes,
coordErr )
protected
Process one HTM pixel, appending to an existing catalog or creating
a new catalog, as needed.

Parameters
----------
inputData : `numpy.ndarray`
    The data from one input file.
matchedPixels : `numpy.ndarray`
    The row-matched pixel indexes corresponding to ``inputData``.
pixelId : `int`
    The pixel index we are currently processing.
fluxes : `dict` [`str`, `numpy.ndarray`]
    The values that will go into the flux and fluxErr fields in the
    output catalog.
coordErr : `dict` [`str`, `numpy.ndarray`]
    The values that will go into the coord_raErr, coord_decErr, and
    coord_ra_dec_Cov fields in the output catalog (in radians).

Definition at line 185 of file convertRefcatManager.py.

185 def _doOnePixel(self, inputData, matchedPixels, pixelId, fluxes, coordErr):
186 """Process one HTM pixel, appending to an existing catalog or creating
187 a new catalog, as needed.
188
189 Parameters
190 ----------
191 inputData : `numpy.ndarray`
192 The data from one input file.
193 matchedPixels : `numpy.ndarray`
194 The row-matched pixel indexes corresponding to ``inputData``.
195 pixelId : `int`
196 The pixel index we are currently processing.
197 fluxes : `dict` [`str`, `numpy.ndarray`]
198 The values that will go into the flux and fluxErr fields in the
199 output catalog.
200 coordErr : `dict` [`str`, `numpy.ndarray`]
201 The values that will go into the coord_raErr, coord_decErr, and
202 coord_ra_dec_Cov fields in the output catalog (in radians).
203 """
204 idx = np.where(matchedPixels == pixelId)[0]
205 catalog = self.getCatalog(pixelId, self.schema, len(idx))
206 for outputRow, inputRow in zip(catalog[-len(idx):], inputData[idx]):
207 self._fillRecord(outputRow, inputRow)
208
209 global COUNTER
210 with COUNTER.get_lock():
211 self._setIds(inputData[idx], catalog)
212
213 # set fluxes from the pre-computed array
214 for name, array in fluxes.items():
215 catalog[self.key_map[name]][-len(idx):] = array[idx]
216
217 # set coordinate errors from the pre-computed array
218 for name, array in coordErr.items():
219 catalog[name][-len(idx):] = array[idx]
220
221 catalog.writeFits(self.filenames[pixelId])
222

◆ _epochToMjdTai()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._epochToMjdTai ( self,
nativeEpoch )
protected
Convert an epoch in native format to TAI MJD (a float).

Definition at line 398 of file convertRefcatManager.py.

398 def _epochToMjdTai(self, nativeEpoch):
399 """Convert an epoch in native format to TAI MJD (a float).
400 """
401 return astropy.time.Time(nativeEpoch, format=self.config.epoch_format,
402 scale=self.config.epoch_scale).tai.mjd
403

◆ _fillRecord()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._fillRecord ( self,
record,
row )
protected
Fill a record in an indexed catalog to be persisted.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : structured `numpy.array`
    Row from catalog being converted.

Definition at line 444 of file convertRefcatManager.py.

444 def _fillRecord(self, record, row):
445 """Fill a record in an indexed catalog to be persisted.
446
447 Parameters
448 ----------
449 record : `lsst.afw.table.SimpleRecord`
450 Row from indexed catalog to modify.
451 row : structured `numpy.array`
452 Row from catalog being converted.
453 """
454 record.setCoord(self.computeCoord(row, self.config.ra_name, self.config.dec_name))
455
456 self._setFlags(record, row)
457 if self.config.full_position_information:
458 self._setProperMotion(record, row)
459 self._setParallax(record, row)
460 self._setCoordinateCovariance(record, row)
461 self._setExtra(record, row)
462
463

◆ _getCoordErr()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._getCoordErr ( self,
inputData )
protected
Compute the ra/dec error fields that will go into the output catalog.

Parameters
----------
inputData : `numpy.ndarray`
    The input data to compute fluxes for.

Returns
-------
coordErr : `dict` [`str`, `numpy.ndarray`]
    The values that will go into the coord_raErr, coord_decErr, fields
    in the output catalog (in radians).

Notes
-----
This does not handle the ra/dec covariance field,
``coord_ra_coord_dec_Cov``. That field is handled in
`_setCoordinateCovariance`.

Definition at line 294 of file convertRefcatManager.py.

294 def _getCoordErr(self, inputData, ):
295 """Compute the ra/dec error fields that will go into the output catalog.
296
297 Parameters
298 ----------
299 inputData : `numpy.ndarray`
300 The input data to compute fluxes for.
301
302 Returns
303 -------
304 coordErr : `dict` [`str`, `numpy.ndarray`]
305 The values that will go into the coord_raErr, coord_decErr, fields
306 in the output catalog (in radians).
307
308 Notes
309 -----
310 This does not handle the ra/dec covariance field,
311 ``coord_ra_coord_dec_Cov``. That field is handled in
312 `_setCoordinateCovariance`.
313 """
314 result = {}
315 if hasattr(self, "coord_err_unit"):
316 result['coord_raErr'] = u.Quantity(inputData[self.config.ra_err_name],
317 self.coord_err_unit).to_value(u.radian)
318 result['coord_decErr'] = u.Quantity(inputData[self.config.dec_err_name],
319 self.coord_err_unit).to_value(u.radian)
320 return result
321

◆ _getFluxes()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._getFluxes ( self,
inputData )
protected
Compute the flux fields that will go into the output catalog.

Parameters
----------
inputData : `numpy.ndarray`
    The input data to compute fluxes for.

Returns
-------
fluxes : `dict` [`str`, `numpy.ndarray`]
    The values that will go into the flux and fluxErr fields in the
    output catalog.

Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager, and lsst.meas.algorithms.convertRefcatManager.ConvertGaiaXpManager.

Definition at line 338 of file convertRefcatManager.py.

338 def _getFluxes(self, inputData):
339 """Compute the flux fields that will go into the output catalog.
340
341 Parameters
342 ----------
343 inputData : `numpy.ndarray`
344 The input data to compute fluxes for.
345
346 Returns
347 -------
348 fluxes : `dict` [`str`, `numpy.ndarray`]
349 The values that will go into the flux and fluxErr fields in the
350 output catalog.
351 """
352 result = {}
353 for item in self.config.mag_column_list:
354 result[item+'_flux'] = (inputData[item]*u.ABmag).to_value(u.nJy)
355 if len(self.config.mag_err_column_map) > 0:
356 for err_key in self.config.mag_err_column_map.keys():
357 error_col_name = self.config.mag_err_column_map[err_key]
358 # TODO: multiply by 1e9 here until we have a replacement (see DM-16903)
359 # NOTE: copy the arrays because the numpy strides may not be useable by C++.
360 fluxErr = fluxErrFromABMagErr(inputData[error_col_name].copy(),
361 inputData[err_key].copy())*1e9
362 result[err_key+'_fluxErr'] = fluxErr
363 return result
364

◆ _setCoordinateCovariance()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setCoordinateCovariance ( self,
record,
row )
protected
Set the off-diagonal position covariance in a record of an indexed
catalog.

There is no generic way to determine covariance. Override this method
in a subclass specialized for your dataset.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : structured `numpy.array`
    Row from catalog being converted.

Reimplemented in lsst.meas.algorithms.convertRefcatManager.ConvertGaiaManager.

Definition at line 404 of file convertRefcatManager.py.

404 def _setCoordinateCovariance(self, record, row):
405 """Set the off-diagonal position covariance in a record of an indexed
406 catalog.
407
408 There is no generic way to determine covariance. Override this method
409 in a subclass specialized for your dataset.
410
411 Parameters
412 ----------
413 record : `lsst.afw.table.SimpleRecord`
414 Row from indexed catalog to modify.
415 row : structured `numpy.array`
416 Row from catalog being converted.
417 """
418 raise NotImplementedError("There is no default method for setting the covariance. Override this "
419 "method in a subclass specialized for your dataset.")
420

◆ _setExtra()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setExtra ( self,
record,
row )
protected
Set extra data fields in a record of an indexed catalog.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : structured `numpy.array`
    Row from catalog being converted.

Definition at line 421 of file convertRefcatManager.py.

421 def _setExtra(self, record, row):
422 """Set extra data fields in a record of an indexed catalog.
423
424 Parameters
425 ----------
426 record : `lsst.afw.table.SimpleRecord`
427 Row from indexed catalog to modify.
428 row : structured `numpy.array`
429 Row from catalog being converted.
430 """
431 for extra_col in self.config.extra_col_names:
432 value = row[extra_col]
433 # If data read from a text file contains string like entires,
434 # numpy stores this as its own internal type, a numpy.str_
435 # object. This seems to be a consequence of how numpy stores
436 # string like objects in fixed column arrays. This checks
437 # if any of the values to be added to the catalog are numpy
438 # string types, and if they are, casts them to a python string
439 # which is what the python c++ records expect
440 if isinstance(value, np.str_):
441 value = str(value)
442 record.set(self.key_map[extra_col], value)
443

◆ _setFlags()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setFlags ( self,
record,
row )
protected
Set flags in an output record.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : `numpy.ndarray`
    Row from catalog being converted.

Definition at line 322 of file convertRefcatManager.py.

322 def _setFlags(self, record, row):
323 """Set flags in an output record.
324
325 Parameters
326 ----------
327 record : `lsst.afw.table.SimpleRecord`
328 Row from indexed catalog to modify.
329 row : `numpy.ndarray`
330 Row from catalog being converted.
331 """
332 names = record.schema.getNames()
333 for flag in self._flags:
334 if flag in names:
335 attr_name = 'is_{}_name'.format(flag)
336 record.set(self.key_map[flag], bool(row[getattr(self.config, attr_name)]))
337

◆ _setIds()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setIds ( self,
inputData,
catalog )
protected
Fill the `id` field of catalog with a running index, filling the
last values up to the length of ``inputData``.

Fill with `self.config.id_name` if specified, otherwise use the
global running counter value.

Parameters
----------
inputData : `numpy.ndarray`
    The input data that is being processed.
catalog : `lsst.afw.table.SimpleCatalog`
    The output catalog to fill the ids.

Definition at line 223 of file convertRefcatManager.py.

223 def _setIds(self, inputData, catalog):
224 """Fill the `id` field of catalog with a running index, filling the
225 last values up to the length of ``inputData``.
226
227 Fill with `self.config.id_name` if specified, otherwise use the
228 global running counter value.
229
230 Parameters
231 ----------
232 inputData : `numpy.ndarray`
233 The input data that is being processed.
234 catalog : `lsst.afw.table.SimpleCatalog`
235 The output catalog to fill the ids.
236 """
237 global COUNTER
238 size = len(inputData)
239 if self.config.id_name:
240 catalog['id'][-size:] = inputData[self.config.id_name]
241 else:
242 idEnd = COUNTER.value + size
243 catalog['id'][-size:] = np.arange(COUNTER.value, idEnd)
244 COUNTER.value = idEnd
245

◆ _setParallax()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setParallax ( self,
record,
row )
protected
Set the parallax fields in a record of a refcat.

Definition at line 389 of file convertRefcatManager.py.

389 def _setParallax(self, record, row):
390 """Set the parallax fields in a record of a refcat.
391 """
392 if self.config.parallax_name is None:
393 return
394 scale = self.config.parallax_scale*lsst.geom.milliarcseconds
395 record.set(self.key_map['parallax'], row[self.config.parallax_name]*scale)
396 record.set(self.key_map['parallaxErr'], row[self.config.parallax_err_name]*scale)
397

◆ _setProperMotion()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._setProperMotion ( self,
record,
row )
protected
Set proper motion fields in a record of an indexed catalog.

The proper motions are read from the specified columns,
scaled appropriately, and installed in the appropriate
columns of the output.

Parameters
----------
record : `lsst.afw.table.SimpleRecord`
    Row from indexed catalog to modify.
row : structured `numpy.array`
    Row from catalog being converted.

Definition at line 365 of file convertRefcatManager.py.

365 def _setProperMotion(self, record, row):
366 """Set proper motion fields in a record of an indexed catalog.
367
368 The proper motions are read from the specified columns,
369 scaled appropriately, and installed in the appropriate
370 columns of the output.
371
372 Parameters
373 ----------
374 record : `lsst.afw.table.SimpleRecord`
375 Row from indexed catalog to modify.
376 row : structured `numpy.array`
377 Row from catalog being converted.
378 """
379 if self.config.pm_ra_name is None: # ConvertReferenceCatalogConfig.validate ensures all or none
380 return
381 radPerOriginal = np.radians(self.config.pm_scale)/(3600*1000)
382 record.set(self.key_map["pm_ra"], row[self.config.pm_ra_name]*radPerOriginal*lsst.geom.radians)
383 record.set(self.key_map["pm_dec"], row[self.config.pm_dec_name]*radPerOriginal*lsst.geom.radians)
384 record.set(self.key_map["epoch"], self._epochToMjdTai(row[self.config.epoch_name]))
385 if self.config.pm_ra_err_name is not None: # pm_dec_err_name also, by validation
386 record.set(self.key_map["pm_raErr"], row[self.config.pm_ra_err_name]*radPerOriginal)
387 record.set(self.key_map["pm_decErr"], row[self.config.pm_dec_err_name]*radPerOriginal)
388

◆ computeCoord()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.computeCoord ( row,
ra_name,
dec_name )
static
Create an ICRS coord. from a row of a catalog being converted.

Parameters
----------
row : `numpy.ndarray`
    Row from catalog being converted.
ra_name : `str`
    Name of RA key in catalog being converted.
dec_name : `str`
    Name of Dec key in catalog being converted.

Returns
-------
coord : `lsst.geom.SpherePoint`
    ICRS coordinate.

Definition at line 275 of file convertRefcatManager.py.

275 def computeCoord(row, ra_name, dec_name):
276 """Create an ICRS coord. from a row of a catalog being converted.
277
278 Parameters
279 ----------
280 row : `numpy.ndarray`
281 Row from catalog being converted.
282 ra_name : `str`
283 Name of RA key in catalog being converted.
284 dec_name : `str`
285 Name of Dec key in catalog being converted.
286
287 Returns
288 -------
289 coord : `lsst.geom.SpherePoint`
290 ICRS coordinate.
291 """
292 return lsst.geom.SpherePoint(row[ra_name], row[dec_name], lsst.geom.degrees)
293
Point in an unspecified spherical coordinate system.
Definition SpherePoint.h:57

◆ getCatalog()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.getCatalog ( self,
pixelId,
schema,
nNewElements )
Get a catalog from disk or create it if it doesn't exist.

Parameters
----------
pixelId : `dict`
    Identifier for catalog to retrieve
schema : `lsst.afw.table.Schema`
    Schema to use in catalog creation it does not exist.
nNewElements : `int`
    The number of new elements that will be added to the catalog,
    so space can be preallocated.

Returns
-------
catalog : `lsst.afw.table.SimpleCatalog`
    The new or read-and-resized catalog specified by `dataId`.

Definition at line 246 of file convertRefcatManager.py.

246 def getCatalog(self, pixelId, schema, nNewElements):
247 """Get a catalog from disk or create it if it doesn't exist.
248
249 Parameters
250 ----------
251 pixelId : `dict`
252 Identifier for catalog to retrieve
253 schema : `lsst.afw.table.Schema`
254 Schema to use in catalog creation it does not exist.
255 nNewElements : `int`
256 The number of new elements that will be added to the catalog,
257 so space can be preallocated.
258
259 Returns
260 -------
261 catalog : `lsst.afw.table.SimpleCatalog`
262 The new or read-and-resized catalog specified by `dataId`.
263 """
264 # This is safe, because we lock on this file before getCatalog is called.
265 if os.path.isfile(self.filenames[pixelId]):
266 catalog = afwTable.SimpleCatalog.readFits(self.filenames[pixelId])
267 catalog.resize(len(catalog) + nNewElements)
268 return catalog.copy(deep=True) # ensure contiguity, so that column-assignment works
269 catalog = afwTable.SimpleCatalog(schema)
270 catalog.resize(nNewElements)
271 self.addRefCatMetadata(catalog)
272 return catalog
273
Custom catalog class for record/table subclasses that are guaranteed to have an ID,...

◆ run()

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.run ( self,
inputFiles )
Index a set of input files from a reference catalog, and write the
output to the appropriate filenames, in parallel.

Parameters
----------
inputFiles : `list`
    A list of file paths to read data from.

Returns
-------
output : `dict` [`int`, `str`]
    The htm ids and the filenames that were written to.

Definition at line 100 of file convertRefcatManager.py.

100 def run(self, inputFiles):
101 """Index a set of input files from a reference catalog, and write the
102 output to the appropriate filenames, in parallel.
103
104 Parameters
105 ----------
106 inputFiles : `list`
107 A list of file paths to read data from.
108
109 Returns
110 -------
111 output : `dict` [`int`, `str`]
112 The htm ids and the filenames that were written to.
113 """
114 global COUNTER, FILE_PROGRESS
115 self.nInputFiles = len(inputFiles)
116
117 with multiprocessing.Manager() as manager:
118 COUNTER.value = 0
119 FILE_PROGRESS.value = 0
120 fileLocks = manager.dict()
121 self.log.info("Creating %s file locks.", self.htmRange[1] - self.htmRange[0])
122 for i in range(self.htmRange[0], self.htmRange[1]):
123 fileLocks[i] = manager.Lock()
124 self.log.info("File locks created.")
125
126 start_time = time.perf_counter()
127 with multiprocessing.Pool(self.config.n_processes) as pool:
128 result = pool.starmap(self._convertOneFile, zip(inputFiles, itertools.repeat(fileLocks)))
129 end_time = time.perf_counter()
130 self.log.info("Finished writing files. Elapsed time: %.2f seconds", end_time-start_time)
131
132 return {id: self.filenames[id] for item in result for id in item}
133

Member Data Documentation

◆ _DefaultName

str lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._DefaultName = 'convertRefcatManager'
staticprotected

Definition at line 81 of file convertRefcatManager.py.

◆ _flags

list lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager._flags = ['photometric', 'resolved', 'variable']
staticprotected

Definition at line 80 of file convertRefcatManager.py.

◆ addRefCatMetadata

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.addRefCatMetadata

Definition at line 93 of file convertRefcatManager.py.

◆ config

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.config

Definition at line 87 of file convertRefcatManager.py.

◆ ConfigClass

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.ConfigClass = ConvertRefcatManagerConfig
static

Definition at line 82 of file convertRefcatManager.py.

◆ coord_err_unit

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.coord_err_unit

Definition at line 98 of file convertRefcatManager.py.

◆ file_reader

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.file_reader

Definition at line 88 of file convertRefcatManager.py.

◆ filenames

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.filenames

Definition at line 86 of file convertRefcatManager.py.

◆ htmRange

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.htmRange

Definition at line 92 of file convertRefcatManager.py.

◆ indexer

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.indexer

Definition at line 89 of file convertRefcatManager.py.

◆ key_map

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.key_map

Definition at line 91 of file convertRefcatManager.py.

◆ log

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.log

Definition at line 94 of file convertRefcatManager.py.

◆ nInputFiles

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.nInputFiles

Definition at line 115 of file convertRefcatManager.py.

◆ schema

lsst.meas.algorithms.convertRefcatManager.ConvertRefcatManager.schema

Definition at line 90 of file convertRefcatManager.py.


The documentation for this class was generated from the following file: