LSST Applications  21.0.0-172-gfb10e10a+18fedfabac,22.0.0+297cba6710,22.0.0+80564b0ff1,22.0.0+8d77f4f51a,22.0.0+a28f4c53b1,22.0.0+dcf3732eb2,22.0.1-1-g7d6de66+2a20fdde0d,22.0.1-1-g8e32f31+297cba6710,22.0.1-1-geca5380+7fa3b7d9b6,22.0.1-12-g44dc1dc+2a20fdde0d,22.0.1-15-g6a90155+515f58c32b,22.0.1-16-g9282f48+790f5f2caa,22.0.1-2-g92698f7+dcf3732eb2,22.0.1-2-ga9b0f51+7fa3b7d9b6,22.0.1-2-gd1925c9+bf4f0e694f,22.0.1-24-g1ad7a390+a9625a72a8,22.0.1-25-g5bf6245+3ad8ecd50b,22.0.1-25-gb120d7b+8b5510f75f,22.0.1-27-g97737f7+2a20fdde0d,22.0.1-32-gf62ce7b1+aa4237961e,22.0.1-4-g0b3f228+2a20fdde0d,22.0.1-4-g243d05b+871c1b8305,22.0.1-4-g3a563be+32dcf1063f,22.0.1-4-g44f2e3d+9e4ab0f4fa,22.0.1-42-gca6935d93+ba5e5ca3eb,22.0.1-5-g15c806e+85460ae5f3,22.0.1-5-g58711c4+611d128589,22.0.1-5-g75bb458+99c117b92f,22.0.1-6-g1c63a23+7fa3b7d9b6,22.0.1-6-g50866e6+84ff5a128b,22.0.1-6-g8d3140d+720564cf76,22.0.1-6-gd805d02+cc5644f571,22.0.1-8-ge5750ce+85460ae5f3,master-g6e05de7fdc+babf819c66,master-g99da0e417a+8d77f4f51a,w.2021.48
LSST Data Management Base Package
Public Member Functions | Public Attributes | List of all members
lsst.jointcal.utils.JointcalStatistics Class Reference

Public Member Functions

def __init__ (self, match_radius=0.1 *arcseconds, flux_limit=100.0, do_photometry=True, do_astrometry=True, verbose=False)
 
def compute_rms (self, data_refs, reference)
 
def make_plots (self, data_refs, old_wcs_list, name='', interactive=False, per_ccd_plot=False, outdir='.plots')
 

Public Attributes

 match_radius
 
 flux_limit
 
 do_photometry
 
 do_astrometry
 
 verbose
 
 log
 
 filters
 
 visits_per_dataRef
 
 old_source
 
 new_source
 
 new_PA1
 
 old_dist_total
 
 new_dist_total
 
 old_rms
 
 new_rms
 
 old_ref
 
 new_ref
 
 old_mag
 
 new_mag
 
 faint
 
 bright
 
 old_weighted_rms
 
 new_weighted_rms
 
 old_PA1
 

Detailed Description

Compute statistics on jointcal-processed data, and optionally generate plots.

Notes
-----
Instantiate JointcalStatistics and call compute_rms() to get the relevant
statistics for e.g. unittests, and call make_plots() to generate a suite of
diagnostic plots.

Definition at line 50 of file utils.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.jointcal.utils.JointcalStatistics.__init__ (   self,
  match_radius = 0.1*arcseconds,
  flux_limit = 100.0,
  do_photometry = True,
  do_astrometry = True,
  verbose = False 
)
Parameters
----------
match_radius : lsst.geom.Angle
    match sources within this radius for RMS statistics
flux_limit : float
    Signal/Noise (flux/fluxErr) for sources to be included in the RMS cross-match.
    100 is a balance between good centroids and enough sources.
do_photometry : bool, optional
    Perform calculations/make plots for photometric metrics.
do_astrometry : bool, optional
    Perform calculations/make plots for astrometric metrics.
verbose : bool, optional
    Print extra things

Definition at line 61 of file utils.py.

63  verbose=False):
64  """
65  Parameters
66  ----------
67  match_radius : lsst.geom.Angle
68  match sources within this radius for RMS statistics
69  flux_limit : float
70  Signal/Noise (flux/fluxErr) for sources to be included in the RMS cross-match.
71  100 is a balance between good centroids and enough sources.
72  do_photometry : bool, optional
73  Perform calculations/make plots for photometric metrics.
74  do_astrometry : bool, optional
75  Perform calculations/make plots for astrometric metrics.
76  verbose : bool, optional
77  Print extra things
78  """
79  self.match_radius = match_radius
80  self.flux_limit = flux_limit
81  self.do_photometry = do_photometry
82  self.do_astrometry = do_astrometry
83  self.verbose = verbose
84  self.log = _LOG.getChild('JointcalStatistics')
85 

Member Function Documentation

◆ compute_rms()

def lsst.jointcal.utils.JointcalStatistics.compute_rms (   self,
  data_refs,
  reference 
)
Match all data_refs to compute the RMS, for all detections above self.flux_limit.

Parameters
----------
data_refs : list of lsst.daf.persistence.butlerSubset.ButlerDataRef
    A list of data refs to do the calculations between.
reference : lsst reference catalog
    reference catalog to do absolute matching against.

Returns
-------
namedtuple:
    astropy.Quantity
        Post-jointcal relative RMS of the matched sources.
    astropy.Quantity
        Post-jointcal absolute RMS of matched sources.
    float
        Post-jointcal photometric repeatability (PA1 from the SRD).

Definition at line 86 of file utils.py.

86  def compute_rms(self, data_refs, reference):
87  """
88  Match all data_refs to compute the RMS, for all detections above self.flux_limit.
89 
90  Parameters
91  ----------
92  data_refs : list of lsst.daf.persistence.butlerSubset.ButlerDataRef
93  A list of data refs to do the calculations between.
94  reference : lsst reference catalog
95  reference catalog to do absolute matching against.
96 
97  Returns
98  -------
99  namedtuple:
100  astropy.Quantity
101  Post-jointcal relative RMS of the matched sources.
102  astropy.Quantity
103  Post-jointcal absolute RMS of matched sources.
104  float
105  Post-jointcal photometric repeatability (PA1 from the SRD).
106  """
107 
108  # DECAM doesn't have "filter" in its registry, so we have to get the filter names directly.
109  self.filters = [ref.get('calexp_filterLabel').bandLabel for ref in data_refs]
110  self.visits_per_dataRef = [ref.dataId['visit'] for ref in data_refs]
111 
112  def compute(catalogs, photoCalibs):
113  """Compute the relative and absolute matches in distance and flux."""
114  visit_catalogs = self._make_visit_catalogs(catalogs, self.visits_per_dataRef)
115  catalogs = [visit_catalogs[x] for x in self.visits_per_dataRef]
116  # use the first catalog as the relative reference catalog
117  # NOTE: The "first" catalog depends on the original ordering of the data_refs.
118  # NOTE: Thus, because I'm doing a many-1 match in _make_match_dict,
119  # the number of matches (and thus the details of the match statistics)
120  # will change if the data_refs are ordered differently.
121  # All the more reason to use a proper n-way matcher here. See: DM-8664
122  refcat = catalogs[0]
123  refcalib = photoCalibs[0] if photoCalibs != [] else None
124  dist_rel, flux_rel, ref_flux_rel, source_rel = self._make_match_dict(refcat,
125  catalogs[1:],
126  photoCalibs[1:],
127  refcalib=refcalib)
128  dist_abs, flux_abs, ref_flux_abs, source_abs = self._make_match_dict(reference,
129  catalogs,
130  photoCalibs)
131  dist = MatchDict(dist_rel, dist_abs)
132  flux = MatchDict(flux_rel, flux_abs)
133  ref_flux = MatchDict(ref_flux_rel, ref_flux_abs)
134  source = MatchDict(source_rel, source_abs)
135  return dist, flux, ref_flux, source
136 
137  old_cats = [ref.get('src') for ref in data_refs]
138  # NOTE: build photoCalibs from existing old Calib objects.
139  # TODO: we can make this a listcomp again once DM-10153 is finished.
140  old_calibs = []
141  if self.do_photometry:
142  old_calibs = [ref.get('calexp_photoCalib') for ref in data_refs]
143 
144  self.old_dist, self.old_flux, self.old_ref_flux, self.old_source = compute(old_cats, old_calibs)
145 
146  # Update coordinates with the new wcs, and get the new photoCalibs.
147  new_cats = [ref.get('src') for ref in data_refs]
148  new_wcss = []
149  if self.do_astrometry:
150  new_wcss = [ref.get('jointcal_wcs') for ref in data_refs]
151  new_calibs = []
152  if self.do_photometry:
153  new_calibs = [ref.get('jointcal_photoCalib') for ref in data_refs]
154  if self.do_astrometry:
155  for wcs, cat in zip(new_wcss, new_cats):
156  # update in-place the object coordinates based on the new wcs
158 
159  self.new_dist, self.new_flux, self.new_ref_flux, self.new_source = compute(new_cats, new_calibs)
160 
161  if self.verbose:
162  print('old, new relative distance matches:',
163  len(self.old_dist.relative), len(self.new_dist.relative))
164  print('old, new absolute distance matches:',
165  len(self.old_dist.absolute), len(self.new_dist.absolute))
166  print('old, new relative flux matches:',
167  len(self.old_flux.relative), len(self.new_flux.relative))
168  print('old, new absolute flux matches:',
169  len(self.old_flux.absolute), len(self.new_flux.absolute))
170 
171  if self.do_photometry:
172  self._photometric_rms()
173  else:
174  self.new_PA1 = None
175 
176  def rms_total(data):
177  """Compute the total rms across all sources."""
178  total = sum(sum(dd**2) for dd in data.values())
179  n = sum(len(dd) for dd in data.values())
180  return np.sqrt(total/n)
181 
182  if self.do_astrometry:
183  self.old_dist_total = MatchDict(*(tuple(map(rms_total, self.old_dist))*u.radian).to(u.arcsecond))
184  self.new_dist_total = MatchDict(*(tuple(map(rms_total, self.new_dist))*u.radian).to(u.arcsecond))
185  else:
186  self.old_dist_total = MatchDict(None, None)
187  self.new_dist_total = MatchDict(None, None)
188 
189  if self.verbose:
190  if self.do_astrometry:
191  print("relative, absolute astrometry:",
192  self.new_dist_total.relative,
193  self.new_dist_total.absolute)
194  if self.do_photometry:
195  print("Photometric Accuracy (PA1):", self.new_PA1)
196  Rms_result = collections.namedtuple("Rms_result", ["dist_relative", "dist_absolute", "pa1"])
197  return Rms_result(self.new_dist_total.relative, self.new_dist_total.absolute, self.new_PA1)
198 
table::Key< int > to
void updateSourceCoords(geom::SkyWcs const &wcs, SourceCollection &sourceList)
Update sky coordinates in a collection of source objects.
Definition: wcsUtils.cc:95

◆ make_plots()

def lsst.jointcal.utils.JointcalStatistics.make_plots (   self,
  data_refs,
  old_wcs_list,
  name = '',
  interactive = False,
  per_ccd_plot = False,
  outdir = '.plots' 
)
Make plots of various quantites to help with debugging.
Requires that `compute_rms()` was run first.

Parameters
----------
data_refs : list of lsst.daf.persistence.butlerSubset.ButlerDataRef
    A list of data refs to do the calculations between.
old_wcs_list : list of lsst.afw.image.wcs.Wcs
    A list of the old (pre-jointcal) WCSs, one-to-one corresponding to data_refs.
name : str
    Name to include in plot titles and save files.
interactive : bool
    Turn on matplotlib interactive mode and drop into a debugger when
    plotting is finished. Otherwise, use a non-interactive backend.
per_ccd_plot : bool
    Plot the WCS per CCD (takes longer and generates many plots for a large camera)
outdir : str
    directory to save plots to.

Definition at line 199 of file utils.py.

200  name='', interactive=False, per_ccd_plot=False, outdir='.plots'):
201  """
202  Make plots of various quantites to help with debugging.
203  Requires that `compute_rms()` was run first.
204 
205  Parameters
206  ----------
207  data_refs : list of lsst.daf.persistence.butlerSubset.ButlerDataRef
208  A list of data refs to do the calculations between.
209  old_wcs_list : list of lsst.afw.image.wcs.Wcs
210  A list of the old (pre-jointcal) WCSs, one-to-one corresponding to data_refs.
211  name : str
212  Name to include in plot titles and save files.
213  interactive : bool
214  Turn on matplotlib interactive mode and drop into a debugger when
215  plotting is finished. Otherwise, use a non-interactive backend.
216  per_ccd_plot : bool
217  Plot the WCS per CCD (takes longer and generates many plots for a large camera)
218  outdir : str
219  directory to save plots to.
220  """
221  import matplotlib
222 
223  if not interactive:
224  # Use a non-interactive backend for faster plotting.
225  matplotlib.use('pdf')
226 
227  import matplotlib.pyplot as plt
228  import astropy.visualization
229  # make quantities behave nicely when plotted.
230  astropy.visualization.quantity_support()
231  if interactive:
232  plt.ion()
233 
234  self.log.info("N data_refs: %d", len(data_refs))
235 
236  if self.do_photometry:
237  plot_flux_distributions(plt, self.old_mag, self.new_mag,
238  self.old_weighted_rms, self.new_weighted_rms,
239  self.faint, self.bright, self.old_PA1, self.new_PA1,
240  name=name, outdir=outdir)
241  self.log.info("Photometric accuracy (old, new): {:.2e} {:.2e}".format(self.old_PA1,
242  self.new_PA1))
243 
244  def rms_per_source(data):
245  """Each element of data must already be the "delta" of whatever measurement."""
246  return (np.sqrt([np.mean(dd**2) for dd in data.values()])*u.radian).to(u.arcsecond)
247 
248  if self.do_astrometry:
249  old_dist_rms = MatchDict(*(tuple(map(rms_per_source, self.old_dist))))
250  new_dist_rms = MatchDict(*(tuple(map(rms_per_source, self.new_dist))))
251 
252  self.log.info("relative RMS (old, new): {:.2e} {:.2e}".format(self.old_dist_total.relative,
253  self.new_dist_total.relative))
254  self.log.info("absolute RMS (old, new): {:.2e} {:.2e}".format(self.old_dist_total.absolute,
255  self.new_dist_total.absolute))
256  plot_rms_histogram(plt, old_dist_rms.relative, old_dist_rms.absolute,
257  new_dist_rms.relative, new_dist_rms.absolute,
258  self.old_dist_total.relative, self.old_dist_total.absolute,
259  self.new_dist_total.relative, self.new_dist_total.absolute,
260  name=name, outdir=outdir)
261 
262  plot_all_wcs_deltas(plt, data_refs, self.visits_per_dataRef, old_wcs_list,
263  per_ccd_plot=per_ccd_plot,
264  name=name, outdir=outdir)
265 
266  if interactive:
267  plt.show()
268  import pdb
269  pdb.set_trace()
270 
def plot_all_wcs_deltas(plt, data_refs, visits, old_wcs_list, per_ccd_plot=False, name='', outdir='.plots')
Definition: utils.py:500
def plot_flux_distributions(plt, old_mag, new_mag, old_weighted_rms, new_weighted_rms, faint, bright, old_PA1, new_PA1, name='', outdir='.plots')
Definition: utils.py:439
def plot_rms_histogram(plt, old_rms_relative, old_rms_absolute, new_rms_relative, new_rms_absolute, old_rel_total, old_abs_total, new_rel_total, new_abs_total, name="", outdir='.plots')
Definition: utils.py:719
def format(config, name=None, writeSourceLine=True, prefix="", verbose=False)
Definition: history.py:174

Member Data Documentation

◆ bright

lsst.jointcal.utils.JointcalStatistics.bright

Definition at line 304 of file utils.py.

◆ do_astrometry

lsst.jointcal.utils.JointcalStatistics.do_astrometry

Definition at line 82 of file utils.py.

◆ do_photometry

lsst.jointcal.utils.JointcalStatistics.do_photometry

Definition at line 81 of file utils.py.

◆ faint

lsst.jointcal.utils.JointcalStatistics.faint

Definition at line 303 of file utils.py.

◆ filters

lsst.jointcal.utils.JointcalStatistics.filters

Definition at line 109 of file utils.py.

◆ flux_limit

lsst.jointcal.utils.JointcalStatistics.flux_limit

Definition at line 80 of file utils.py.

◆ log

lsst.jointcal.utils.JointcalStatistics.log

Definition at line 84 of file utils.py.

◆ match_radius

lsst.jointcal.utils.JointcalStatistics.match_radius

Definition at line 79 of file utils.py.

◆ new_dist_total

lsst.jointcal.utils.JointcalStatistics.new_dist_total

Definition at line 184 of file utils.py.

◆ new_mag

lsst.jointcal.utils.JointcalStatistics.new_mag

Definition at line 292 of file utils.py.

◆ new_PA1

lsst.jointcal.utils.JointcalStatistics.new_PA1

Definition at line 174 of file utils.py.

◆ new_ref

lsst.jointcal.utils.JointcalStatistics.new_ref

Definition at line 290 of file utils.py.

◆ new_rms

lsst.jointcal.utils.JointcalStatistics.new_rms

Definition at line 286 of file utils.py.

◆ new_source

lsst.jointcal.utils.JointcalStatistics.new_source

Definition at line 159 of file utils.py.

◆ new_weighted_rms

lsst.jointcal.utils.JointcalStatistics.new_weighted_rms

Definition at line 310 of file utils.py.

◆ old_dist_total

lsst.jointcal.utils.JointcalStatistics.old_dist_total

Definition at line 183 of file utils.py.

◆ old_mag

lsst.jointcal.utils.JointcalStatistics.old_mag

Definition at line 291 of file utils.py.

◆ old_PA1

lsst.jointcal.utils.JointcalStatistics.old_PA1

Definition at line 311 of file utils.py.

◆ old_ref

lsst.jointcal.utils.JointcalStatistics.old_ref

Definition at line 289 of file utils.py.

◆ old_rms

lsst.jointcal.utils.JointcalStatistics.old_rms

Definition at line 285 of file utils.py.

◆ old_source

lsst.jointcal.utils.JointcalStatistics.old_source

Definition at line 144 of file utils.py.

◆ old_weighted_rms

lsst.jointcal.utils.JointcalStatistics.old_weighted_rms

Definition at line 309 of file utils.py.

◆ verbose

lsst.jointcal.utils.JointcalStatistics.verbose

Definition at line 83 of file utils.py.

◆ visits_per_dataRef

lsst.jointcal.utils.JointcalStatistics.visits_per_dataRef

Definition at line 110 of file utils.py.


The documentation for this class was generated from the following file: