lsst.jointcal.check_logged_chi2.LogParser Class Reference

Public Member Functions
	__init__ (self, plot=True, verbose=True)
	__call__ (self, logfile)

Public Attributes
	matcher
	plot
	verbose
	fig
	section_start
	section_end

Protected Member Functions
	_find_chi2_increase (self, chi2Data, title, label, threshold=1)
	_extract_chi2 (self, opened_log, section)
	_plot (self, astrometry, photometry, title)
	_plot_axes (self, ax0, ax1, chi2Data, palette, label="")

Detailed Description

Parse a jointcal logfile to extract chi2 values and plot them.

Call the instance with the path to a file to check it for anamolous chi2
and output plots to your current directory.

Parameters
----------
plot : `bool`
    Make plots for each file (saved to the current working directory)?
verbose : `bool`
    Print extra updates during processing?

Definition at line 76 of file check_logged_chi2.py.

Constructor & Destructor Documentation

__init__()

lsst.jointcal.check_logged_chi2.LogParser.__init__	(		self,
			plot = True,
			verbose = True )

Definition at line 89 of file check_logged_chi2.py.

    def __init__(self, plot=True, verbose=True):
        # This regular expression extracts the chi2 values, and the "kind" of
        # chi2 (e.g. "Initial", "Fit iteration").
        # Chi2 values in the log look like this, for example:
        # jointcal INFO: Initial chi2/ndof : 2.50373e+16/532674=4.7003e+10
        chi2_re = "jointcal INFO: (?P<kind>.+) chi2/ndof : (?P<chi2>.+)/(?P<ndof>.+)=(?P<reduced_chi2>.+)"
        self.matcher = re.compile(chi2_re)
        self.plot = plot
        self.verbose = verbose
 
        # Reuse the Figure to speed up plotting and save memory.
        self.fig = plt.figure(figsize=(15, 8))
 
        # How to find the beginning and end of the relevant parts of the log
        # to scan for chi2 values.
        self.section_start = {"astrometry": "Starting astrometric fitting...",
                              "photometry": "Starting photometric fitting..."}
        self.section_end = {"astrometry": "Updating WCS for visit:",
                            "photometry": "Updating PhotoCalib for visit:"}
 

Member Function Documentation

__call__()

lsst.jointcal.check_logged_chi2.LogParser.__call__	(		self,
			logfile )

Parse logfile to extract chi2 values and generate and save plots.

The plot output is written to the current directory, with the name
derived from the basename of ``logfile``.

Parameters
----------
logfile : `str`
    The filename of the jointcal log to process.

Definition at line 109 of file check_logged_chi2.py.

    def __call__(self, logfile):
        """Parse logfile to extract chi2 values and generate and save plots.
 
        The plot output is written to the current directory, with the name
        derived from the basename of ``logfile``.
 
        Parameters
        ----------
        logfile : `str`
            The filename of the jointcal log to process.
        """
        title = os.path.basename(logfile)
        if self.verbose:
            print("Processing:", title)
 
        with open(logfile) as opened_log:
            # Astrometry is always run first, so we can scan for that until the
            # end of that section, and then continue scanning for photometry.
            astrometry = self._extract_chi2(opened_log, "astrometry")
            increased = self._find_chi2_increase(astrometry, title, "astrometry")
            photometry = self._extract_chi2(opened_log, "photometry")
            increased |= self._find_chi2_increase(photometry, title, "photometry")
 
        if astrometry is None and photometry is None and self.verbose:
            print(f"WARNING: No chi2 values found in {logfile}.")
 
        if increased or self.plot:
            self._plot(astrometry, photometry, title)
            plotfile = f"{os.path.splitext(title)[0]}.png"
            plt.savefig(plotfile, bbox_inches="tight")
            print("Saved plot:", plotfile)
 

_extract_chi2()

lsst.jointcal.check_logged_chi2.LogParser._extract_chi2 ( self, opened_log, section )

protected

Return the values extracted from the chi2 statements in the logfile.

Definition at line 158 of file check_logged_chi2.py.

    def _extract_chi2(self, opened_log, section):
        """Return the values extracted from the chi2 statements in the logfile.
        """
        start = self.section_start[section]
        end = self.section_end[section]
        kind = []
        chi2 = []
        ndof = []
        reduced = []
        # Skip over lines until we get to the section start line.
        for line in opened_log:
            if start in line:
                break
 
        for line in opened_log:
            # Stop parsing at the section end line.
            if end in line:
                break
            if "chi2" in line:
                match = self.matcher.search(line)
                if match is not None:
                    kind.append(match.group("kind"))
                    chi2.append(match.group("chi2"))
                    ndof.append(match.group("ndof"))
                    reduced.append(match.group("reduced_chi2"))
 
        # No chi2 values were found (e.g., photometry wasn't run).
        if len(kind) == 0:
            return None
 
        return Chi2Data(kind, np.array(chi2, dtype=np.float64),
                        np.array(ndof, dtype=int), np.array(reduced, dtype=np.float64))
 

_find_chi2_increase()

lsst.jointcal.check_logged_chi2.LogParser._find_chi2_increase ( self, chi2Data, title, label, threshold = 1 )

protected

Return True and print a message if the raw chi2 increases
markedly.

Definition at line 141 of file check_logged_chi2.py.

    def _find_chi2_increase(self, chi2Data, title, label, threshold=1):
        """Return True and print a message if the raw chi2 increases
        markedly.
        """
        if chi2Data is None:
            return False
        diff = np.diff(chi2Data.raw)
        ratio = diff/chi2Data.raw[:-1]
        if np.any(ratio > threshold):
            increased = np.where(ratio > threshold)[0]
            print(f"{title} has increasing {label} chi2:")
            for x in zip(chi2Data.raw[increased], chi2Data.raw[increased + 1],
                         ratio[increased], diff[increased]):
                print(f"{x[0]:.6} -> {x[1]:.6} (ratio: {x[2]:.6}, diff: {x[3]:.6})")
            return True
        return False
 

_plot()

lsst.jointcal.check_logged_chi2.LogParser._plot ( self, astrometry, photometry, title )

protected

Generate plots of chi2 values.

Parameters
----------
astrometry : `Chi2Data` or None
    The as-read astrometry data, or None if there is none to plot.
photometry : `Chi2Data` or None
    The as-read photometry data, or None if there is none to plot.
title : `str`
    Title for the whole plot.

Definition at line 191 of file check_logged_chi2.py.

    def _plot(self, astrometry, photometry, title):
        """Generate plots of chi2 values.
 
        Parameters
        ----------
        astrometry : `Chi2Data` or None
            The as-read astrometry data, or None if there is none to plot.
        photometry : `Chi2Data` or None
            The as-read photometry data, or None if there is none to plot.
        title : `str`
            Title for the whole plot.
        """
        palette = itertools.cycle(sns.color_palette())
 
        self.fig.clf()
        ax0, ax1 = self.fig.subplots(ncols=2, gridspec_kw={"wspace": 0.05})
 
        self.fig.suptitle(title)
        # Use a log scale if any of the chi2 values are very large.
        if max(getattr(astrometry, "raw", [0])) > 100 or max(getattr(photometry, "raw", [0])) > 100:
            ax0.set_yscale("log")
        ax1.yaxis.set_label_position("right")
        ax1.yaxis.tick_right()
 
        if astrometry is not None:
            patch1, patch2 = self._plot_axes(ax0, ax1, astrometry, palette, label="astrometry")
 
        if photometry is not None:
            patch3, patch4 = self._plot_axes(ax0, ax1, photometry, palette, label="photometry")
 
        # Let matplotlib figure out the best legend location: if there is data
        # in the "upper right", we definitely want to see it.
        handles, labels = ax0.get_legend_handles_labels()
        ax1.legend(handles, labels)
 

_plot_axes()

lsst.jointcal.check_logged_chi2.LogParser._plot_axes ( self, ax0, ax1, chi2Data, palette, label = "" )

protected

Make the chi2 and degrees of freedom subplots.

Definition at line 226 of file check_logged_chi2.py.

    def _plot_axes(self, ax0, ax1, chi2Data, palette, label=""):
        """Make the chi2 and degrees of freedom subplots."""
        xrange = np.arange(0, len(chi2Data.raw), dtype=float)
 
        # mark chi2=1
        ax0.axhline(1, color='grey', ls='--')
        # mark the separation between initialization and iteration
        ax0.axvline(chi2Data.init_count-0.5, color='grey', lw=0.9)
        color = next(palette)
        patch1 = ax0.plot(xrange[:chi2Data.init_count], chi2Data.raw[:chi2Data.init_count], '*', ms=10,
                          label=f"{label} pre-init", color=color)
        patch2 = ax0.plot(xrange[chi2Data.init_count:], chi2Data.raw[chi2Data.init_count:], 'o', ms=10,
                          label=f"{label} post-init", color=color)
        patch1 = ax0.plot(xrange[:chi2Data.init_count], chi2Data.reduced[:chi2Data.init_count], '*',
                          markerfacecolor="none", ms=10, color=color)
        patch2 = ax0.plot(xrange[chi2Data.init_count:], chi2Data.reduced[chi2Data.init_count:], 'o',
                          markerfacecolor="none", ms=10, label=f"{label} reduced", color=color)
 
        ax0.set_xlabel("Iteration #", fontsize=20)
        ax0.set_ylabel(r"$\chi ^2$", fontsize=20)
 
        # mark the separation between initialization and iteration
        ax1.axvline(chi2Data.init_count-0.5, color='grey', lw=0.9)
        ax1.plot(xrange[:chi2Data.init_count], chi2Data.ndof[:chi2Data.init_count], '*', ms=10,
                 label="pre-init", color=color)
        ax1.plot(xrange[chi2Data.init_count:], chi2Data.ndof[chi2Data.init_count:], 'o', ms=10,
                 label="post-init", color=color)
 
        ax1.set_xlabel("Iteration #", fontsize=20)
        ax1.set_ylabel("# degrees of freedom", fontsize=20)
 
        return patch1[0], patch2[0]
 
 

Member Data Documentation

fig

lsst.jointcal.check_logged_chi2.LogParser.fig

Definition at line 100 of file check_logged_chi2.py.

matcher

lsst.jointcal.check_logged_chi2.LogParser.matcher

Definition at line 95 of file check_logged_chi2.py.

plot

lsst.jointcal.check_logged_chi2.LogParser.plot

Definition at line 96 of file check_logged_chi2.py.

section_end

lsst.jointcal.check_logged_chi2.LogParser.section_end

Definition at line 106 of file check_logged_chi2.py.

section_start

lsst.jointcal.check_logged_chi2.LogParser.section_start

Definition at line 104 of file check_logged_chi2.py.

verbose

lsst.jointcal.check_logged_chi2.LogParser.verbose

Definition at line 97 of file check_logged_chi2.py.

---

The documentation for this class was generated from the following file:

• /j/snowflake/release/lsstsw/stack/lsst-scipipe-8.0.0/Linux64/jointcal/gbb8dafda3b+6681f309db/python/lsst/jointcal/check_logged_chi2.py