lsst.pipe.tasks.postprocess.TransformCatalogBaseTask Class Reference

Inheritance diagram for lsst.pipe.tasks.postprocess.TransformCatalogBaseTask:

Public Member Functions
def	outputDataset (self)
def	inputDataset (self)
def	ConfigClass (self)
def	__init__ (self, *args, **kwargs)
def	runQuantum (self, butlerQC, inputRefs, outputRefs)
def	runDataRef (self, dataRef)
def	run (self, parq, funcs=None, dataId=None, band=None)
def	getFunctors (self)
def	getAnalysis (self, parq, funcs=None, band=None)
def	transform (self, band, parq, funcs, dataId)
def	write (self, df, parqRef)
def	writeMetadata (self, dataRef)

Public Attributes
	funcs

Detailed Description

Base class for transforming/standardizing a catalog

by applying functors that convert units and apply calibrations.
The purpose of this task is to perform a set of computations on
an input `ParquetTable` dataset (such as `deepCoadd_obj`) and write the
results to a new dataset (which needs to be declared in an `outputDataset`
attribute).

The calculations to be performed are defined in a YAML file that specifies
a set of functors to be computed, provided as
a `--functorFile` config parameter.  An example of such a YAML file
is the following:

    funcs:
        psfMag:
            functor: Mag
            args:
                - base_PsfFlux
            filt: HSC-G
            dataset: meas
        cmodel_magDiff:
            functor: MagDiff
            args:
                - modelfit_CModel
                - base_PsfFlux
            filt: HSC-G
        gauss_magDiff:
            functor: MagDiff
            args:
                - base_GaussianFlux
                - base_PsfFlux
            filt: HSC-G
        count:
            functor: Column
            args:
                - base_InputCount_value
            filt: HSC-G
        deconvolved_moments:
            functor: DeconvolvedMoments
            filt: HSC-G
            dataset: forced_src
    refFlags:
        - calib_psfUsed
        - merge_measurement_i
        - merge_measurement_r
        - merge_measurement_z
        - merge_measurement_y
        - merge_measurement_g
        - base_PixelFlags_flag_inexact_psfCenter
        - detect_isPrimary

The names for each entry under "func" will become the names of columns in the
output dataset.  All the functors referenced are defined in `lsst.pipe.tasks.functors`.
Positional arguments to be passed to each functor are in the `args` list,
and any additional entries for each column other than "functor" or "args" (e.g., `'filt'`,
`'dataset'`) are treated as keyword arguments to be passed to the functor initialization.

The "flags" entry is the default shortcut for `Column` functors.
All columns listed under "flags" will be copied to the output table
untransformed. They can be of any datatype.
In the special case of transforming a multi-level oject table with
band and dataset indices (deepCoadd_obj), these will be taked from the
`meas` dataset and exploded out per band.

There are two special shortcuts that only apply when transforming
multi-level Object (deepCoadd_obj) tables:
 -  The "refFlags" entry is shortcut for `Column` functor
    taken from the `'ref'` dataset if transforming an ObjectTable.
 -  The "forcedFlags" entry is shortcut for `Column` functors.
    taken from the ``forced_src`` dataset if transforming an ObjectTable.
    These are expanded out per band.


This task uses the `lsst.pipe.tasks.postprocess.PostprocessAnalysis` object
to organize and excecute the calculations.

Definition at line 558 of file postprocess.py.

Constructor & Destructor Documentation

__init__()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.__init__	(		self,
		*	args,
		**	kwargs
	)

Definition at line 652 of file postprocess.py.

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if self.config.functorFile:
            self.log.info('Loading tranform functor definitions from %s',
                          self.config.functorFile)
            self.funcs = CompositeFunctor.from_file(self.config.functorFile)
            self.funcs.update(dict(PostprocessAnalysis._defaultFuncs))
        else:
            self.funcs = None
 

Member Function Documentation

ConfigClass()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.ConfigClass

(

self

)

Definition at line 649 of file postprocess.py.

    def ConfigClass(self):
        raise NotImplementedError('Subclass must define "ConfigClass" attribute')
 

getAnalysis()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.getAnalysis	(		self,
			parq,
			funcs = None,
			band = None
	)

Definition at line 712 of file postprocess.py.

    def getAnalysis(self, parq, funcs=None, band=None):
        if funcs is None:
            funcs = self.funcs
        analysis = PostprocessAnalysis(parq, funcs, filt=band)
        return analysis
 

getFunctors()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.getFunctors

(

self

)

Definition at line 709 of file postprocess.py.

    def getFunctors(self):
        return self.funcs
 

inputDataset()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.inputDataset

(

self

)

Definition at line 645 of file postprocess.py.

    def inputDataset(self):
        raise NotImplementedError('Subclass must define "inputDataset" attribute')
 

outputDataset()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.outputDataset

(

self

)

Definition at line 641 of file postprocess.py.

    def outputDataset(self):
        raise NotImplementedError('Subclass must define "outputDataset" attribute')
 

run()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.run	(		self,
			parq,
			funcs = None,
			dataId = None,
			band = None
	)

Do postprocessing calculations

Takes a `ParquetTable` object and dataId,
returns a dataframe with results of postprocessing calculations.

Parameters
----------
parq : `lsst.pipe.tasks.parquetTable.ParquetTable`
    ParquetTable from which calculations are done.
funcs : `lsst.pipe.tasks.functors.Functors`
    Functors to apply to the table's columns
dataId : dict, optional
    Used to add a `patchId` column to the output dataframe.
band : `str`, optional
    Filter band that is being processed.

Returns
------
    `pandas.DataFrame`

Definition at line 681 of file postprocess.py.

    def run(self, parq, funcs=None, dataId=None, band=None):
        """Do postprocessing calculations
 
        Takes a `ParquetTable` object and dataId,
        returns a dataframe with results of postprocessing calculations.
 
        Parameters
        ----------
        parq : `lsst.pipe.tasks.parquetTable.ParquetTable`
            ParquetTable from which calculations are done.
        funcs : `lsst.pipe.tasks.functors.Functors`
            Functors to apply to the table's columns
        dataId : dict, optional
            Used to add a `patchId` column to the output dataframe.
        band : `str`, optional
            Filter band that is being processed.
 
        Returns
        ------
            `pandas.DataFrame`
 
        """
        self.log.info("Transforming/standardizing the source table dataId: %s", dataId)
 
        df = self.transform(band, parq, funcs, dataId).df
        self.log.info("Made a table of %d columns and %d rows", len(df.columns), len(df))
        return df
 

runDataRef()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.runDataRef	(		self,
			dataRef
	)

Definition at line 672 of file postprocess.py.

    def runDataRef(self, dataRef):
        parq = dataRef.get()
        if self.funcs is None:
            raise ValueError("config.functorFile is None. "
                             "Must be a valid path to yaml in order to run as a CommandlineTask.")
        df = self.run(parq, funcs=self.funcs, dataId=dataRef.dataId)
        self.write(df, dataRef)
        return df
 

runQuantum()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.runQuantum	(		self,
			butlerQC,
			inputRefs,
			outputRefs
	)

Definition at line 662 of file postprocess.py.

    def runQuantum(self, butlerQC, inputRefs, outputRefs):
        inputs = butlerQC.get(inputRefs)
        if self.funcs is None:
            raise ValueError("config.functorFile is None. "
                             "Must be a valid path to yaml in order to run Task as a PipelineTask.")
        result = self.run(parq=inputs['inputCatalog'], funcs=self.funcs,
                          dataId=outputRefs.outputCatalog.dataId.full)
        outputs = pipeBase.Struct(outputCatalog=result)
        butlerQC.put(outputs, outputRefs)
 

transform()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.transform	(		self,
			band,
			parq,
			funcs,
			dataId
	)

Definition at line 718 of file postprocess.py.

    def transform(self, band, parq, funcs, dataId):
        analysis = self.getAnalysis(parq, funcs=funcs, band=band)
        df = analysis.df
        if dataId is not None:
            for key, value in dataId.items():
                df[str(key)] = value
 
        if self.config.primaryKey:
            if df.index.name != self.config.primaryKey and self.config.primaryKey in df:
                df.reset_index(inplace=True, drop=True)
                df.set_index(self.config.primaryKey, inplace=True)
 
        return pipeBase.Struct(
            df=df,
            analysis=analysis
        )
 

write()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.write	(		self,
			df,
			parqRef
	)

Definition at line 735 of file postprocess.py.

    def write(self, df, parqRef):
        parqRef.put(ParquetTable(dataFrame=df), self.outputDataset)
 

writeMetadata()

def lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.writeMetadata	(		self,
			dataRef
	)

No metadata to write.

Definition at line 738 of file postprocess.py.

    def writeMetadata(self, dataRef):
        """No metadata to write.
        """
        pass
 
 

Member Data Documentation

funcs

lsst.pipe.tasks.postprocess.TransformCatalogBaseTask.funcs

Definition at line 657 of file postprocess.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-0.7.0/Linux64/pipe_tasks/21.0.0-172-gfb10e10a+18fedfabac/python/lsst/pipe/tasks/postprocess.py