LSSTApplications  19.0.0-10-g920eed2,19.0.0-11-g48a0200+2,19.0.0-18-gfc4e62b+11,19.0.0-2-g3b2f90d+2,19.0.0-2-gd671419+5,19.0.0-20-g5a5a17ab+9,19.0.0-21-g2644856+11,19.0.0-22-gc5dc5b1+6,19.0.0-23-gdc29a50+3,19.0.0-24-g923e380+11,19.0.0-25-g6c8df7140,19.0.0-28-g9b887e2,19.0.0-3-g2b32d65+5,19.0.0-3-g8227491+10,19.0.0-3-g9c54d0d+10,19.0.0-3-gca68e65+6,19.0.0-3-gcfc5f51+5,19.0.0-3-ge110943+9,19.0.0-3-ge74d124,19.0.0-3-gfe04aa6+11,19.0.0-4-g06f5963+5,19.0.0-4-g3d16501+11,19.0.0-4-g4a9c019+5,19.0.0-4-g5a8b323,19.0.0-4-g66397f0+1,19.0.0-4-g8278b9b+1,19.0.0-4-g8557e14,19.0.0-4-g8964aba+11,19.0.0-4-ge404a01+10,19.0.0-5-g40f3a5a,19.0.0-5-g4db63b3,19.0.0-5-gfb03ce7+11,19.0.0-6-gbaebbfb+10,19.0.0-60-gafafd468+11,19.0.0-67-g3ab1e6e,19.0.0-7-g039c0b5+9,19.0.0-7-gbea9075+4,19.0.0-7-gc567de5+11,19.0.0-8-g3a3ce09+6,19.0.0-9-g463f923+10,w.2020.21
LSSTDataManagementBasePackage
Public Member Functions | Public Attributes | Static Public Attributes | List of all members
lsst.meas.base.noiseReplacer.NoiseReplacer Class Reference

Public Member Functions

def __init__ (self, config, exposure, footprints, noiseImage=None, exposureId=None, log=None)
 
def insertSource (self, id)
 
def removeSource (self, id)
 
def end (self)
 
def getNoiseGenerator (self, exposure, noiseImage, noiseMeanVar, exposureId=None)
 

Public Attributes

 noiseSource
 
 noiseOffset
 
 noiseSeedMultiplier
 
 noiseGenMean
 
 noiseGenStd
 
 removeplanes
 
 otherbitmask
 
 heavies
 
 heavyNoise
 

Static Public Attributes

 ConfigClass
 
 exposure
 
 footprints
 
 log
 

Detailed Description

Replace sources with noise during measurement.

Parameters
----------
config : `NoiseReplacerConfig`
    Configuration.
exposure : `lsst.afw.image.Exposure`
    Image in which sources will be replaced by noise. During operation,
    the image will be modified in-place to replace all sources. At the end
    of the measurment procedure, the original sources will be replaced.
footprints : `dict`
    Mapping of ``id`` to a tuple of ``(parent, Footprint)``. When used in
    single-frame measurement, ``id`` is the source ID, but in forced
    photometry this is the reference ID (as that is used to determine
    deblend families).
noiseImage : `lsst.afw.image.ImageF`
    An image used as a predictable noise replacement source. Used during
    testing only.
log : `lsst.log.log.log.Log`, optional
    Logger to use for status messages; no status messages will be recorded
    if `None`.

Notes
-----
When measuring a source (or the children associated with a parent source),
this class is used to replace its neighbors with noise, using the
deblender's definition of the sources as stored in
`~lsst.afw.detection.heavyFootprint.HeavyFootprint`\ s attached to the
`~lsst.afw.table.SourceRecord`\ s.  The algorithm works as follows:

#. All pixels in the source `~lsst.afw.detection.Footprint`\ s are replaced
   with artificially generated noise (in `NoiseReplacer.__init__`).
#. Before each source is measured, we restore the original pixel data by
   inserting that source's
   `~lsst.afw.detection.heavyFootprint.HeavyFootprint` (from the deblender)
   into the image.
#. After measurement, we again replace the source pixels with (the same)
   artificial noise.
#. After measuring all sources, the image is returned to its original
   state.

This is a functional copy of the code in the older
``ReplaceWithNoiseTask``, but with a slightly different API needed for the
new measurement framework; note that it is not an `~lsst.pipe.base.Task`,
as the lifetime of a ``NoiseReplacer`` now corresponds to a single
exposure, not an entire processing run.

When processing the ``footprints`` parameter, this routine should create
`~lsst.afw.detection.heavyFootprint.HeavyFootprint`\ s for any non-Heavy
`~lsst.afw.detection.Footprint`\ s, and replace them in the dictionary. It
should then create a dict of
`~lsst.afw.detection.heavyFootprint.HeavyFootprint`\ s containing noise,
but only for parent objects, then replace all sources with noise. This
should ignore any footprints that lay outside the bounding box of the
exposure, and clip those that lie on the border.

As the code currently stands, the heavy footprint for a deblended object
must be available from the input catalog.  If it is not, it cannot be
reproduced here. In that case, the topmost parent in the objects parent
chain must be used. The heavy footprint for that source is created in
this class from the masked image.

Definition at line 57 of file noiseReplacer.py.

Constructor & Destructor Documentation

◆ __init__()

def lsst.meas.base.noiseReplacer.NoiseReplacer.__init__ (   self,
  config,
  exposure,
  footprints,
  noiseImage = None,
  exposureId = None,
  log = None 
)

Definition at line 135 of file noiseReplacer.py.

135  def __init__(self, config, exposure, footprints, noiseImage=None, exposureId=None, log=None):
136  noiseMeanVar = None
137  self.noiseSource = config.noiseSource
138  self.noiseOffset = config.noiseOffset
139  self.noiseSeedMultiplier = config.noiseSeedMultiplier
140  self.noiseGenMean = None
141  self.noiseGenStd = None
142  self.log = log
143 
144  # creates heavies, replaces all footprints with noise
145  # We need the source table to be sorted by ID to do the parent lookups
146  self.exposure = exposure
147  self.footprints = footprints
148  mi = exposure.getMaskedImage()
149  im = mi.getImage()
150  mask = mi.getMask()
151  # Add temporary Mask planes for THISDET and OTHERDET
152  self.removeplanes = []
153  bitmasks = []
154  for maskname in ['THISDET', 'OTHERDET']:
155  try:
156  # does it already exist?
157  plane = mask.getMaskPlane(maskname)
158  if self.log:
159  self.log.debug('Mask plane "%s" already existed', maskname)
160  except Exception:
161  # if not, add it; we should delete it when done.
162  plane = mask.addMaskPlane(maskname)
163  self.removeplanes.append(maskname)
164  mask.clearMaskPlane(plane)
165  bitmask = mask.getPlaneBitMask(maskname)
166  bitmasks.append(bitmask)
167  if self.log:
168  self.log.debug('Mask plane "%s": plane %i, bitmask %i = 0x%x',
169  maskname, plane, bitmask, bitmask)
170  self.thisbitmask, self.otherbitmask = bitmasks
171  del bitmasks
172  self.heavies = {}
173  # Start by creating HeavyFootprints for each source which has no parent
174  # and just use them for children which do not already have heavy footprints.
175  # If a heavy footprint is available for a child, we will use it. Otherwise,
176  # we use the first parent in the parent chain which has a heavy footprint,
177  # which with the one level deblender will alway be the topmost parent
178  # NOTE: heavy footprints get destroyed by the transform process in forcedPhotImage.py,
179  # so they are never available for forced measurements.
180 
181  # Create in the dict heavies = {id:heavyfootprint}
182  for id, fp in footprints.items():
183  if fp[1].isHeavy():
184  self.heavies[id] = fp[1]
185  elif fp[0] == 0:
186  self.heavies[id] = afwDet.makeHeavyFootprint(fp[1], mi)
187 
188  # ## FIXME: the heavy footprint includes the mask
189  # ## and variance planes, which we shouldn't need
190  # ## (I don't think we ever want to modify them in
191  # ## the input image). Copying them around is
192  # ## wasteful.
193 
194  # We now create a noise HeavyFootprint for each source with has a heavy footprint.
195  # We'll put the noise footprints in a dict heavyNoise = {id:heavyNoiseFootprint}
196  self.heavyNoise = {}
197  noisegen = self.getNoiseGenerator(exposure, noiseImage, noiseMeanVar, exposureId=exposureId)
198  # The noiseGenMean and Std are used by the unit tests
199  self.noiseGenMean = noisegen.mean
200  self.noiseGenStd = noisegen.std
201  if self.log:
202  self.log.debug('Using noise generator: %s', str(noisegen))
203  for id in self.heavies:
204  fp = footprints[id][1]
205  noiseFp = noisegen.getHeavyFootprint(fp)
206  self.heavyNoise[id] = noiseFp
207  # Also insert the noisy footprint into the image now.
208  # Notice that we're just inserting it into "im", ie,
209  # the Image, not the MaskedImage.
210  noiseFp.insert(im)
211  # Also set the OTHERDET bit
212  fp.spans.setMask(mask, self.otherbitmask)
213 

Member Function Documentation

◆ end()

def lsst.meas.base.noiseReplacer.NoiseReplacer.end (   self)
End the NoiseReplacer.

Restores original data to the exposure from the heavies dictionary and
the mask planes to their original state.

Definition at line 274 of file noiseReplacer.py.

274  def end(self):
275  """End the NoiseReplacer.
276 
277  Restores original data to the exposure from the heavies dictionary and
278  the mask planes to their original state.
279  """
280  # restores original image, cleans up temporaries
281  # (ie, replace all the top-level pixels)
282  mi = self.exposure.getMaskedImage()
283  im = mi.getImage()
284  mask = mi.getMask()
285  for id in self.footprints.keys():
286  if self.footprints[id][0] != 0:
287  continue
288  self.heavies[id].insert(im)
289  for maskname in self.removeplanes:
290  mask.removeAndClearMaskPlane(maskname, True)
291 
292  del self.removeplanes
293  del self.thisbitmask
294  del self.otherbitmask
295  del self.heavies
296  del self.heavyNoise
297 

◆ getNoiseGenerator()

def lsst.meas.base.noiseReplacer.NoiseReplacer.getNoiseGenerator (   self,
  exposure,
  noiseImage,
  noiseMeanVar,
  exposureId = None 
)
Return a generator of artificial noise.

Returns
-------
noiseGenerator : `lsst.afw.image.noiseReplacer.NoiseGenerator`

Definition at line 298 of file noiseReplacer.py.

298  def getNoiseGenerator(self, exposure, noiseImage, noiseMeanVar, exposureId=None):
299  """Return a generator of artificial noise.
300 
301  Returns
302  -------
303  noiseGenerator : `lsst.afw.image.noiseReplacer.NoiseGenerator`
304  """
305  if noiseImage is not None:
306  return ImageNoiseGenerator(noiseImage)
307  rand = None
308  if self.noiseSeedMultiplier:
309  # default plugin, our seed
310  if exposureId is not None and exposureId != 0:
311  seed = exposureId*self.noiseSeedMultiplier
312  else:
313  seed = self.noiseSeedMultiplier
314  rand = afwMath.Random(afwMath.Random.MT19937, seed)
315  if noiseMeanVar is not None:
316  try:
317  # Assume noiseMeanVar is an iterable of floats
318  noiseMean, noiseVar = noiseMeanVar
319  noiseMean = float(noiseMean)
320  noiseVar = float(noiseVar)
321  noiseStd = math.sqrt(noiseVar)
322  if self.log:
323  self.log.debug('Using passed-in noise mean = %g, variance = %g -> stdev %g',
324  noiseMean, noiseVar, noiseStd)
325  return FixedGaussianNoiseGenerator(noiseMean, noiseStd, rand=rand)
326  except Exception:
327  if self.log:
328  self.log.debug('Failed to cast passed-in noiseMeanVar to floats: %s',
329  str(noiseMeanVar))
330  offset = self.noiseOffset
331  noiseSource = self.noiseSource
332 
333  if noiseSource == 'meta':
334  # check the exposure metadata
335  meta = exposure.getMetadata()
336  # this key name correspond to SubtractBackgroundTask() in meas_algorithms
337  try:
338  bgMean = meta.getAsDouble('BGMEAN')
339  # We would have to adjust for GAIN if ip_isr didn't make it 1.0
340  noiseStd = math.sqrt(bgMean)
341  if self.log:
342  self.log.debug('Using noise variance = (BGMEAN = %g) from exposure metadata',
343  bgMean)
344  return FixedGaussianNoiseGenerator(offset, noiseStd, rand=rand)
345  except Exception:
346  if self.log:
347  self.log.debug('Failed to get BGMEAN from exposure metadata')
348 
349  if noiseSource == 'variance':
350  if self.log:
351  self.log.debug('Will draw noise according to the variance plane.')
352  var = exposure.getMaskedImage().getVariance()
353  return VariancePlaneNoiseGenerator(var, mean=offset, rand=rand)
354 
355  # Compute an image-wide clipped variance.
356  im = exposure.getMaskedImage().getImage()
357  s = afwMath.makeStatistics(im, afwMath.MEANCLIP | afwMath.STDEVCLIP)
358  noiseMean = s.getValue(afwMath.MEANCLIP)
359  noiseStd = s.getValue(afwMath.STDEVCLIP)
360  if self.log:
361  self.log.debug("Measured from image: clipped mean = %g, stdev = %g",
362  noiseMean, noiseStd)
363  return FixedGaussianNoiseGenerator(noiseMean + offset, noiseStd, rand=rand)
364 
365 

◆ insertSource()

def lsst.meas.base.noiseReplacer.NoiseReplacer.insertSource (   self,
  id 
)
Insert the heavy footprint of a given source into the exposure.

Parameters
----------
id : `int`
    ID of the source to insert from original dictionary of footprints.

Notes
-----
Also adjusts the mask plane to show the source of this footprint.

Definition at line 214 of file noiseReplacer.py.

214  def insertSource(self, id):
215  """Insert the heavy footprint of a given source into the exposure.
216 
217  Parameters
218  ----------
219  id : `int`
220  ID of the source to insert from original dictionary of footprints.
221 
222  Notes
223  -----
224  Also adjusts the mask plane to show the source of this footprint.
225  """
226  # Copy this source's pixels into the image
227  mi = self.exposure.getMaskedImage()
228  im = mi.getImage()
229  mask = mi.getMask()
230  # usedid can point either to this source, or to the first parent in the
231  # parent chain which has a heavy footprint (or to the topmost parent,
232  # which always has one)
233  usedid = id
234  while self.footprints[usedid][0] != 0 and usedid not in self.heavies:
235  usedid = self.footprints[usedid][0]
236  fp = self.heavies[usedid]
237  fp.insert(im)
238  fp.spans.setMask(mask, self.thisbitmask)
239  fp.spans.clearMask(mask, self.otherbitmask)
240 

◆ removeSource()

def lsst.meas.base.noiseReplacer.NoiseReplacer.removeSource (   self,
  id 
)
Replace the heavy footprint of a given source with noise.

The same artificial noise is used as in the original replacement.

Parameters
----------
id : `int`
    ID of the source to replace from original dictionary of footprints.

Notes
-----
Also restores the mask plane.

Definition at line 241 of file noiseReplacer.py.

241  def removeSource(self, id):
242  """Replace the heavy footprint of a given source with noise.
243 
244  The same artificial noise is used as in the original replacement.
245 
246  Parameters
247  ----------
248  id : `int`
249  ID of the source to replace from original dictionary of footprints.
250 
251  Notes
252  -----
253  Also restores the mask plane.
254  """
255  # remove a single source
256  # (Replace this source's pixels by noise again.)
257  # Do this by finding the source's top-level ancestor
258  mi = self.exposure.getMaskedImage()
259  im = mi.getImage()
260  mask = mi.getMask()
261 
262  # use the same algorithm as in remove Source to find the heavy noise footprint
263  # which will undo what insertSource(id) does
264  usedid = id
265  while self.footprints[usedid][0] != 0 and usedid not in self.heavies:
266  usedid = self.footprints[usedid][0]
267  # Re-insert the noise pixels
268  fp = self.heavyNoise[usedid]
269  fp.insert(im)
270  # Clear the THISDET mask plane.
271  fp.spans.clearMask(mask, self.thisbitmask)
272  fp.spans.setMask(mask, self.otherbitmask)
273 

Member Data Documentation

◆ ConfigClass

lsst.meas.base.noiseReplacer.NoiseReplacer.ConfigClass
static

Definition at line 121 of file noiseReplacer.py.

◆ exposure

lsst.meas.base.noiseReplacer.NoiseReplacer.exposure
static

Definition at line 123 of file noiseReplacer.py.

◆ footprints

lsst.meas.base.noiseReplacer.NoiseReplacer.footprints
static

Definition at line 127 of file noiseReplacer.py.

◆ heavies

lsst.meas.base.noiseReplacer.NoiseReplacer.heavies

Definition at line 172 of file noiseReplacer.py.

◆ heavyNoise

lsst.meas.base.noiseReplacer.NoiseReplacer.heavyNoise

Definition at line 196 of file noiseReplacer.py.

◆ log

lsst.meas.base.noiseReplacer.NoiseReplacer.log
static

Definition at line 131 of file noiseReplacer.py.

◆ noiseGenMean

lsst.meas.base.noiseReplacer.NoiseReplacer.noiseGenMean

Definition at line 140 of file noiseReplacer.py.

◆ noiseGenStd

lsst.meas.base.noiseReplacer.NoiseReplacer.noiseGenStd

Definition at line 141 of file noiseReplacer.py.

◆ noiseOffset

lsst.meas.base.noiseReplacer.NoiseReplacer.noiseOffset

Definition at line 138 of file noiseReplacer.py.

◆ noiseSeedMultiplier

lsst.meas.base.noiseReplacer.NoiseReplacer.noiseSeedMultiplier

Definition at line 139 of file noiseReplacer.py.

◆ noiseSource

lsst.meas.base.noiseReplacer.NoiseReplacer.noiseSource

Definition at line 137 of file noiseReplacer.py.

◆ otherbitmask

lsst.meas.base.noiseReplacer.NoiseReplacer.otherbitmask

Definition at line 170 of file noiseReplacer.py.

◆ removeplanes

lsst.meas.base.noiseReplacer.NoiseReplacer.removeplanes

Definition at line 152 of file noiseReplacer.py.


The documentation for this class was generated from the following file:
lsst::afw::detection::makeHeavyFootprint
HeavyFootprint< ImagePixelT, MaskPixelT, VariancePixelT > makeHeavyFootprint(Footprint const &foot, lsst::afw::image::MaskedImage< ImagePixelT, MaskPixelT, VariancePixelT > const &img, HeavyFootprintCtrl const *ctrl=NULL)
Create a HeavyFootprint with footprint defined by the given Footprint and pixel values from the given...
Definition: HeavyFootprint.h:148
lsst.gdb.afw.printers.debug
bool debug
Definition: printers.py:9
ast::append
std::shared_ptr< FrameSet > append(FrameSet const &first, FrameSet const &second)
Construct a FrameSet that performs two transformations in series.
Definition: functional.cc:33
lsst::afw::math::makeStatistics
Statistics makeStatistics(lsst::afw::math::MaskedVector< EntryT > const &mv, std::vector< WeightPixel > const &vweights, int const flags, StatisticsControl const &sctrl=StatisticsControl())
The makeStatistics() overload to handle lsst::afw::math::MaskedVector<>
Definition: Statistics.h:520
astshim.keyMap.keyMapContinued.keys
def keys(self)
Definition: keyMapContinued.py:6
end
int end
Definition: BoundedField.cc:105
lsst::afw::math::Random
A class that can be used to generate sequences of random numbers according to a number of different a...
Definition: Random.h:57