LSSTApplications  18.0.0+64,19.0.0+35,19.0.0+37,19.0.0+39,19.0.0+4,19.0.0+44,19.0.0+8,19.0.0-1-g20d9b18+20,19.0.0-1-g49a97f9+2,19.0.0-1-g5549ca4+3,19.0.0-1-g8c57eb9+20,19.0.0-1-ga72da6b+2,19.0.0-1-gbfe0924+22,19.0.0-1-ge272bc4+20,19.0.0-1-gefe1d0d+16,19.0.0-10-g17493b7+4,19.0.0-13-g154200d1e+1,19.0.0-2-g0d9f9cd+36,19.0.0-2-g260436e+20,19.0.0-2-g9675b69+2,19.0.0-2-g9b11441+24,19.0.0-2-gde8e5e3+2,19.0.0-2-gf01c5b1,19.0.0-2-gff6972b+4,19.0.0-21-gaaa92db+5,19.0.0-28-g53bcf5f6+3,19.0.0-3-g27e4659+5,19.0.0-3-g6513920+23,19.0.0-3-gce3f959+9,19.0.0-5-g9aa49c1+1,19.0.0-6-gb5bb71e+13,19.0.0-7-g2f7a0e4+4,19.0.0-7-g686a884+3,19.0.0-7-ga57c4689+11,19.0.0-7-gfdb4f68+3,19.0.0-9-g0ae078d+3,w.2020.08
LSSTDataManagementBasePackage
Public Member Functions | Public Attributes | Static Public Attributes | List of all members
lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask Class Reference
Inheritance diagram for lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask:
lsst.meas.algorithms.ingestIndexReferenceTask.IngestGaiaReferenceTask

Public Member Functions

def __init__ (self, args, butler=None, kwargs)
 
def createIndexedCatalog (self, inputFiles)
 
def makeSchema (self, dtype)
 

Public Attributes

 butler
 
 indexer
 
 IngestManager
 

Static Public Attributes

bool canMultiprocess = False
 
 ConfigClass = IngestIndexedReferenceConfig
 
 RunnerClass = IngestReferenceRunner
 

Detailed Description

Class for producing and loading indexed reference catalogs.

This implements an indexing scheme based on hierarchical triangular
mesh (HTM). The term index really means breaking the catalog into
localized chunks called shards.  In this case each shard contains
the entries from the catalog in a single HTM trixel

For producing catalogs this task makes the following assumptions
about the input catalogs:
- RA, Dec, RA error and Dec error are all in decimal degrees.
- Epoch is available in a column, in a format supported by astropy.time.Time.
- There are no off-diagonal covariance terms, such as covariance
    between RA and Dec, or between PM RA and PM Dec. Gaia is a well
    known example of a catalog that has such terms, and thus should not
    be ingested with this task.

Parameters
----------
butler : `lsst.daf.persistence.Butler`
    Data butler for reading and writing catalogs

Definition at line 274 of file ingestIndexReferenceTask.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask.__init__ (   self,
  args,
  butler = None,
  kwargs 
)

Definition at line 311 of file ingestIndexReferenceTask.py.

311  def __init__(self, *args, butler=None, **kwargs):
312  self.butler = butler
313  super().__init__(*args, **kwargs)
314  self.indexer = IndexerRegistry[self.config.dataset_config.indexer.name](
315  self.config.dataset_config.indexer.active)
316  self.makeSubtask('file_reader')
317  self.IngestManager = ingestIndexManager.IngestIndexManager
318 

Member Function Documentation

◆ createIndexedCatalog()

def lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask.createIndexedCatalog (   self,
  inputFiles 
)
Index a set of files comprising a reference catalog.

Outputs are persisted in the butler repository.

Parameters
----------
inputFiles : `list`
    A list of file paths to read.

Definition at line 319 of file ingestIndexReferenceTask.py.

319  def createIndexedCatalog(self, inputFiles):
320  """Index a set of files comprising a reference catalog.
321 
322  Outputs are persisted in the butler repository.
323 
324  Parameters
325  ----------
326  inputFiles : `list`
327  A list of file paths to read.
328  """
329  schema, key_map = self._saveMasterSchema(inputFiles[0])
330  # create an HTM we can interrogate about pixel ids
331  htm = lsst.sphgeom.HtmPixelization(self.indexer.htm.get_depth())
332  filenames = self._getButlerFilenames(htm)
333  worker = self.IngestManager(filenames,
334  self.config,
335  self.file_reader,
336  self.indexer,
337  schema,
338  key_map,
339  htm.universe()[0],
340  addRefCatMetadata,
341  self.log)
342  worker.run(inputFiles)
343 
344  # write the config that was used to generate the refcat
345  dataId = self.indexer.makeDataId(None, self.config.dataset_config.ref_dataset_name)
346  self.butler.put(self.config.dataset_config, 'ref_cat_config', dataId=dataId)
347 
HtmPixelization provides HTM indexing of points and regions.

◆ makeSchema()

def lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask.makeSchema (   self,
  dtype 
)
Make the schema to use in constructing the persisted catalogs.

Parameters
----------
dtype : `numpy.dtype`
    Data type describing each entry in ``config.extra_col_names``
    for the catalogs being ingested.

Returns
-------
schemaAndKeyMap : `tuple` of (`lsst.afw.table.Schema`, `dict`)
    A tuple containing two items:
    - The schema for the output source catalog.
    - A map of catalog keys to use in filling the record

Definition at line 379 of file ingestIndexReferenceTask.py.

379  def makeSchema(self, dtype):
380  """Make the schema to use in constructing the persisted catalogs.
381 
382  Parameters
383  ----------
384  dtype : `numpy.dtype`
385  Data type describing each entry in ``config.extra_col_names``
386  for the catalogs being ingested.
387 
388  Returns
389  -------
390  schemaAndKeyMap : `tuple` of (`lsst.afw.table.Schema`, `dict`)
391  A tuple containing two items:
392  - The schema for the output source catalog.
393  - A map of catalog keys to use in filling the record
394  """
395  # make a schema with the standard fields
396  schema = LoadReferenceObjectsTask.makeMinimalSchema(
397  filterNameList=self.config.mag_column_list,
398  addCentroid=False,
399  addIsPhotometric=bool(self.config.is_photometric_name),
400  addIsResolved=bool(self.config.is_resolved_name),
401  addIsVariable=bool(self.config.is_variable_name),
402  coordErrDim=2 if bool(self.config.ra_err_name) else 0,
403  addProperMotion=2 if bool(self.config.pm_ra_name) else 0,
404  properMotionErrDim=2 if bool(self.config.pm_ra_err_name) else 0,
405  addParallax=bool(self.config.parallax_name),
406  )
407  keysToSkip = set(("id", "centroid_x", "centroid_y", "hasCentroid"))
408  key_map = {fieldName: schema[fieldName].asKey() for fieldName in schema.getOrderedNames()
409  if fieldName not in keysToSkip}
410 
411  def addField(name):
412  if dtype[name].kind == 'U':
413  # dealing with a string like thing. Need to get type and size.
414  at_size = dtype[name].itemsize
415  return schema.addField(name, type=str, size=at_size)
416  else:
417  at_type = dtype[name].type
418  return schema.addField(name, at_type)
419 
420  for col in self.config.extra_col_names:
421  key_map[col] = addField(col)
422  return schema, key_map
423 
424 
daf::base::PropertySet * set
Definition: fits.cc:902

Member Data Documentation

◆ butler

lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask.butler

Definition at line 312 of file ingestIndexReferenceTask.py.

◆ canMultiprocess

bool lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask.canMultiprocess = False
static

Definition at line 296 of file ingestIndexReferenceTask.py.

◆ ConfigClass

lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask.ConfigClass = IngestIndexedReferenceConfig
static

Definition at line 297 of file ingestIndexReferenceTask.py.

◆ indexer

lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask.indexer

Definition at line 314 of file ingestIndexReferenceTask.py.

◆ IngestManager

lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask.IngestManager

Definition at line 317 of file ingestIndexReferenceTask.py.

◆ RunnerClass

lsst.meas.algorithms.ingestIndexReferenceTask.IngestIndexedReferenceTask.RunnerClass = IngestReferenceRunner
static

Definition at line 298 of file ingestIndexReferenceTask.py.


The documentation for this class was generated from the following file: