Inheritance diagram for lsst.jointcal.jointcal.JointcalTask:
Public Member Functions | |
| __init__ (self, **kwargs) | |
| runQuantum (self, butlerQC, inputRefs, outputRefs) | |
| run (self, inputSourceTableVisit, inputVisitSummary, inputCamera, tract=None) | |
Public Attributes | |
| astrometryRefObjLoader | |
| photometryRefObjLoader | |
| job | |
| focalPlaneBBox | |
| config | |
| sourceSelector | |
| log | |
Static Public Attributes | |
| ConfigClass = JointcalConfig | |
Protected Member Functions | |
| _put_metrics (self, butlerQC, job, outputRefs) | |
| _put_output (self, butlerQC, outputs, outputRefs, camera, setter) | |
| _load_data (self, inputSourceTableVisit, inputVisitSummary, associations, jointcalControl, camera) | |
| _make_one_input_data (self, visitRecord, catalog, detectorDict) | |
| _build_ccdImage (self, data, associations, jointcalControl) | |
| _getDebugPath (self, filename) | |
| _prep_sky (self, associations, filters) | |
| _get_refcat_coordinate_error_override (self, refCat, name) | |
| _compute_proper_motion_epoch (self, ccdImageList) | |
| _do_load_refcat_and_fit (self, associations, defaultFilter, center, radius, tract="", match_cut=3.0, reject_bad_fluxes=False, *name="", refObjLoader=None, referenceSelector=None, fit_function=None, epoch=None) | |
| _load_reference_catalog (self, refObjLoader, referenceSelector, center, radius, filterLabel, applyColorterms=False, epoch=None) | |
| _check_star_lists (self, associations, name) | |
| _logChi2AndValidate (self, associations, fit, model, chi2Label, writeChi2Name=None) | |
| _fit_photometry (self, associations, dataName=None) | |
| _fit_astrometry (self, associations, dataName=None) | |
| _check_stars (self, associations) | |
| _iterate_fit (self, associations, fitter, max_steps, name, whatToFit, dataName="", sigmaRelativeTolerance=0, doRankUpdate=True, doLineSearch=False) | |
| _make_output (self, ccdImageList, model, func) | |
Static Protected Attributes | |
| str | _DefaultName = "jointcal" |
Detailed Description
Astrometricly and photometricly calibrate across multiple visits of the same field.
Definition at line 495 of file jointcal.py.
Constructor & Destructor Documentation
◆ __init__()
| lsst.jointcal.jointcal.JointcalTask.__init__ | ( | self, | |
| ** | kwargs ) |
Definition at line 503 of file jointcal.py.
503 def __init__(self, **kwargs):
504 super().__init__(**kwargs)
505 self.makeSubtask("sourceSelector")
506 if self.config.doAstrometry:
507 self.makeSubtask("astrometryReferenceSelector")
508 else:
509 self.astrometryRefObjLoader = None
510 if self.config.doPhotometry:
511 self.makeSubtask("photometryReferenceSelector")
512 else:
513 self.photometryRefObjLoader = None
514
515 # To hold various computed metrics for use by tests
516 self.job = Job.load_metrics_package(subset='jointcal')
517
Member Function Documentation
◆ _build_ccdImage()
|
protected |
Extract the necessary things from this catalog+metadata to add a new
ccdImage.
Parameters
----------
data : `JointcalInputData`
The loaded input data.
associations : `lsst.jointcal.Associations`
Object to add the info to, to construct a new CcdImage
jointcalControl : `jointcal.JointcalControl`
Control object for associations management
Returns
------
namedtuple or `None`
``wcs``
The TAN WCS of this image, read from the calexp
(`lsst.afw.geom.SkyWcs`).
``key``
A key to identify this dataRef by its visit and ccd ids
(`namedtuple`).
`None`
if there are no sources in the loaded catalog.
Definition at line 768 of file jointcal.py.
768 def _build_ccdImage(self, data, associations, jointcalControl):
769 """
770 Extract the necessary things from this catalog+metadata to add a new
771 ccdImage.
772
773 Parameters
774 ----------
775 data : `JointcalInputData`
776 The loaded input data.
777 associations : `lsst.jointcal.Associations`
778 Object to add the info to, to construct a new CcdImage
779 jointcalControl : `jointcal.JointcalControl`
780 Control object for associations management
781
782 Returns
783 ------
784 namedtuple or `None`
785 ``wcs``
786 The TAN WCS of this image, read from the calexp
787 (`lsst.afw.geom.SkyWcs`).
788 ``key``
789 A key to identify this dataRef by its visit and ccd ids
790 (`namedtuple`).
791 `None`
792 if there are no sources in the loaded catalog.
793 """
794 if len(data.catalog) == 0:
795 self.log.warning("No sources selected in visit %s ccd %s", data.visit, data.detector.getId())
796 return None
797
798 associations.createCcdImage(data.catalog,
799 data.wcs,
800 data.visitInfo,
801 data.bbox,
802 data.filter.physicalLabel,
803 data.photoCalib,
804 data.detector,
805 data.visit,
806 data.detector.getId(),
807 jointcalControl)
808
809 Result = collections.namedtuple('Result_from_build_CcdImage', ('wcs', 'key'))
810 Key = collections.namedtuple('Key', ('visit', 'ccd'))
811 return Result(data.wcs, Key(data.visit, data.detector.getId()))
812
◆ _check_star_lists()
|
protected |
Definition at line 1047 of file jointcal.py.
1047 def _check_star_lists(self, associations, name):
1048 # TODO: these should be len(blah), but we need this properly wrapped first.
1049 if associations.nCcdImagesValidForFit() == 0:
1050 raise RuntimeError('No images in the ccdImageList!')
1051 if associations.fittedStarListSize() == 0:
1052 raise RuntimeError('No stars in the {} fittedStarList!'.format(name))
1053 if associations.refStarListSize() == 0:
1054 raise RuntimeError('No stars in the {} reference star list!'.format(name))
1055
◆ _check_stars()
|
protected |
Count measured and reference stars per ccd and warn/log them.
Definition at line 1305 of file jointcal.py.
1305 def _check_stars(self, associations):
1306 """Count measured and reference stars per ccd and warn/log them."""
1307 for ccdImage in associations.getCcdImageList():
1308 nMeasuredStars, nRefStars = ccdImage.countStars()
1309 self.log.debug("ccdImage %s has %s measured and %s reference stars",
1310 ccdImage.getName(), nMeasuredStars, nRefStars)
1311 if nMeasuredStars < self.config.minMeasuredStarsPerCcd:
1312 self.log.warning("ccdImage %s has only %s measuredStars (desired %s)",
1313 ccdImage.getName(), nMeasuredStars, self.config.minMeasuredStarsPerCcd)
1314 if nRefStars < self.config.minRefStarsPerCcd:
1315 self.log.warning("ccdImage %s has only %s RefStars (desired %s)",
1316 ccdImage.getName(), nRefStars, self.config.minRefStarsPerCcd)
1317
◆ _compute_proper_motion_epoch()
|
protected |
Return the proper motion correction epoch of the provided images.
Parameters
----------
ccdImageList : `list` [`lsst.jointcal.CcdImage`]
The images to compute the appropriate epoch for.
Returns
-------
epoch : `astropy.time.Time`
The date to use for proper motion corrections.
Definition at line 887 of file jointcal.py.
887 def _compute_proper_motion_epoch(self, ccdImageList):
888 """Return the proper motion correction epoch of the provided images.
889
890 Parameters
891 ----------
892 ccdImageList : `list` [`lsst.jointcal.CcdImage`]
893 The images to compute the appropriate epoch for.
894
895 Returns
896 -------
897 epoch : `astropy.time.Time`
898 The date to use for proper motion corrections.
899 """
900 return astropy.time.Time(np.mean([ccdImage.getEpoch() for ccdImage in ccdImageList]),
901 format="jyear",
902 scale="tai")
903
◆ _do_load_refcat_and_fit()
|
protected |
Load reference catalog, perform the fit, and return the result.
Parameters
----------
associations : `lsst.jointcal.Associations`
The star/reference star associations to fit.
defaultFilter : `lsst.afw.image.FilterLabel`
filter to load from reference catalog.
center : `lsst.geom.SpherePoint`
ICRS center of field to load from reference catalog.
radius : `lsst.geom.Angle`
On-sky radius to load from reference catalog.
name : `str`
Name of thing being fit: "astrometry" or "photometry".
refObjLoader : `lsst.meas.algorithms.ReferenceObjectLoader`
Reference object loader to use to load a reference catalog.
referenceSelector : `lsst.meas.algorithms.ReferenceSourceSelectorTask`
Selector to use to pick objects from the loaded reference catalog.
fit_function : callable
Function to call to perform fit (takes Associations object).
tract : `str`, optional
Name of tract currently being fit.
match_cut : `float`, optional
Radius in arcseconds to find cross-catalog matches to during
associations.associateCatalogs.
reject_bad_fluxes : `bool`, optional
Reject refCat sources with NaN/inf flux or NaN/0 fluxErr.
epoch : `astropy.time.Time`, optional
Epoch to which to correct refcat proper motion and parallax,
or `None` to not apply such corrections.
Returns
-------
result : `Photometry` or `Astrometry`
Result of `fit_function()`
Definition at line 904 of file jointcal.py.
908 fit_function=None, epoch=None):
909 """Load reference catalog, perform the fit, and return the result.
910
911 Parameters
912 ----------
913 associations : `lsst.jointcal.Associations`
914 The star/reference star associations to fit.
915 defaultFilter : `lsst.afw.image.FilterLabel`
916 filter to load from reference catalog.
917 center : `lsst.geom.SpherePoint`
918 ICRS center of field to load from reference catalog.
919 radius : `lsst.geom.Angle`
920 On-sky radius to load from reference catalog.
921 name : `str`
922 Name of thing being fit: "astrometry" or "photometry".
923 refObjLoader : `lsst.meas.algorithms.ReferenceObjectLoader`
924 Reference object loader to use to load a reference catalog.
925 referenceSelector : `lsst.meas.algorithms.ReferenceSourceSelectorTask`
926 Selector to use to pick objects from the loaded reference catalog.
927 fit_function : callable
928 Function to call to perform fit (takes Associations object).
929 tract : `str`, optional
930 Name of tract currently being fit.
931 match_cut : `float`, optional
932 Radius in arcseconds to find cross-catalog matches to during
933 associations.associateCatalogs.
934 reject_bad_fluxes : `bool`, optional
935 Reject refCat sources with NaN/inf flux or NaN/0 fluxErr.
936 epoch : `astropy.time.Time`, optional
937 Epoch to which to correct refcat proper motion and parallax,
938 or `None` to not apply such corrections.
939
940 Returns
941 -------
942 result : `Photometry` or `Astrometry`
943 Result of `fit_function()`
944 """
945 self.log.info("====== Now processing %s...", name)
946 # TODO: this should not print "trying to invert a singular transformation:"
947 # if it does that, something's not right about the WCS...
948 associations.associateCatalogs(match_cut)
949 add_measurement(self.job, 'jointcal.%s_matched_fittedStars' % name,
950 associations.fittedStarListSize())
951
952 applyColorterms = False if name.lower() == "astrometry" else self.config.applyColorTerms
953 refCat, fluxField = self._load_reference_catalog(refObjLoader, referenceSelector,
954 center, radius, defaultFilter,
955 applyColorterms=applyColorterms,
956 epoch=epoch)
957 refCoordErr = self._get_refcat_coordinate_error_override(refCat, name)
958
959 associations.collectRefStars(refCat,
960 self.config.matchCut*lsst.geom.arcseconds,
961 fluxField,
962 refCoordinateErr=refCoordErr,
963 rejectBadFluxes=reject_bad_fluxes)
964 add_measurement(self.job, 'jointcal.%s_collected_refStars' % name,
965 associations.refStarListSize())
966
967 associations.prepareFittedStars(self.config.minMeasurements)
968
969 self._check_star_lists(associations, name)
970 add_measurement(self.job, 'jointcal.%s_prepared_refStars' % name,
971 associations.nFittedStarsWithAssociatedRefStar())
972 add_measurement(self.job, 'jointcal.%s_prepared_fittedStars' % name,
973 associations.fittedStarListSize())
974 add_measurement(self.job, 'jointcal.%s_prepared_ccdImages' % name,
975 associations.nCcdImagesValidForFit())
976
977 fit_profile_file = 'jointcal_fit_%s.prof'%name if self.config.detailedProfile else ''
978 dataName = "{}_{}".format(tract, defaultFilter.physicalLabel)
979 with lsst.utils.timer.profile(fit_profile_file):
980 result = fit_function(associations, dataName)
981 # TODO DM-12446: turn this into a "butler save" somehow.
982 # Save reference and measurement chi2 contributions for this data
983 if self.config.writeChi2FilesInitialFinal:
984 baseName = self._getDebugPath(f"{name}_final_chi2-{dataName}")
985 result.fit.saveChi2Contributions(baseName+"{type}")
986 self.log.info("Wrote chi2 contributions files: %s", baseName)
987
988 return result
989
◆ _fit_astrometry()
|
protected |
Fit the astrometric data.
Parameters
----------
associations : `lsst.jointcal.Associations`
The star/reference star associations to fit.
dataName : `str`
Name of the data being processed (e.g. "1234_HSC-Y"), for
identifying debugging files.
Returns
-------
fit_result : `namedtuple`
fit : `lsst.jointcal.AstrometryFit`
The astrometric fitter used to perform the fit.
model : `lsst.jointcal.AstrometryModel`
The astrometric model that was fit.
sky_to_tan_projection : `lsst.jointcal.ProjectionHandler`
The model for the sky to tangent plane projection that was used in the fit.
Definition at line 1205 of file jointcal.py.
1205 def _fit_astrometry(self, associations, dataName=None):
1206 """
1207 Fit the astrometric data.
1208
1209 Parameters
1210 ----------
1211 associations : `lsst.jointcal.Associations`
1212 The star/reference star associations to fit.
1213 dataName : `str`
1214 Name of the data being processed (e.g. "1234_HSC-Y"), for
1215 identifying debugging files.
1216
1217 Returns
1218 -------
1219 fit_result : `namedtuple`
1220 fit : `lsst.jointcal.AstrometryFit`
1221 The astrometric fitter used to perform the fit.
1222 model : `lsst.jointcal.AstrometryModel`
1223 The astrometric model that was fit.
1224 sky_to_tan_projection : `lsst.jointcal.ProjectionHandler`
1225 The model for the sky to tangent plane projection that was used in the fit.
1226 """
1227
1228 self.log.info("=== Starting astrometric fitting...")
1229
1230 associations.deprojectFittedStars()
1231
1232 # NOTE: need to return sky_to_tan_projection so that it doesn't get garbage collected.
1233 # TODO: could we package sky_to_tan_projection and model together so we don't have to manage
1234 # them so carefully?
1235 sky_to_tan_projection = lsst.jointcal.OneTPPerVisitHandler(associations.getCcdImageList())
1236
1237 if self.config.astrometryModel == "constrained":
1238 model = lsst.jointcal.ConstrainedAstrometryModel(associations.getCcdImageList(),
1239 sky_to_tan_projection,
1240 chipOrder=self.config.astrometryChipOrder,
1241 visitOrder=self.config.astrometryVisitOrder)
1242 elif self.config.astrometryModel == "simple":
1243 model = lsst.jointcal.SimpleAstrometryModel(associations.getCcdImageList(),
1244 sky_to_tan_projection,
1245 self.config.useInputWcs,
1246 nNotFit=0,
1247 order=self.config.astrometrySimpleOrder)
1248
1249 fit = lsst.jointcal.AstrometryFit(associations, model, self.config.positionErrorPedestal)
1250 # TODO DM-12446: turn this into a "butler save" somehow.
1251 # Save reference and measurement chi2 contributions for this data
1252 if self.config.writeChi2FilesInitialFinal:
1253 baseName = f"astrometry_initial_chi2-{dataName}"
1254 else:
1255 baseName = None
1256 if self.config.writeInitialModel:
1257 fullpath = self._getDebugPath(f"initial_astrometry_model-{dataName}.txt")
1258 writeModel(model, fullpath, self.log)
1259 self._logChi2AndValidate(associations, fit, model, "Initial", writeChi2Name=baseName)
1260
1261 def getChi2Name(whatToFit):
1262 if self.config.writeChi2FilesOuterLoop:
1263 return f"astrometry_init-%s_chi2-{dataName}" % whatToFit
1264 else:
1265 return None
1266
1267 if self.config.writeInitMatrix:
1268 dumpMatrixFile = self._getDebugPath(f"astrometry_preinit-{dataName}")
1269 else:
1270 dumpMatrixFile = ""
1271 # The constrained model needs the visit transform fit first; the chip
1272 # transform is initialized from the detector's cameraGeom, so it's close.
1273 if self.config.astrometryModel == "constrained":
1274 fit.minimize("DistortionsVisit", dumpMatrixFile=dumpMatrixFile)
1275 self._logChi2AndValidate(associations, fit, model, "Initialize DistortionsVisit",
1276 writeChi2Name=getChi2Name("DistortionsVisit"))
1277 dumpMatrixFile = "" # so we don't redo the output on the next step
1278
1279 fit.minimize("Distortions", dumpMatrixFile=dumpMatrixFile)
1280 self._logChi2AndValidate(associations, fit, model, "Initialize Distortions",
1281 writeChi2Name=getChi2Name("Distortions"))
1282
1283 fit.minimize("Positions")
1284 self._logChi2AndValidate(associations, fit, model, "Initialize Positions",
1285 writeChi2Name=getChi2Name("Positions"))
1286
1287 fit.minimize("Distortions Positions")
1288 self._logChi2AndValidate(associations, fit, model, "Initialize DistortionsPositions",
1289 writeChi2Name=getChi2Name("DistortionsPositions"))
1290
1291 chi2 = self._iterate_fit(associations,
1292 fit,
1293 self.config.maxAstrometrySteps,
1294 "astrometry",
1295 "Distortions Positions",
1296 sigmaRelativeTolerance=self.config.astrometryOutlierRelativeTolerance,
1297 doRankUpdate=self.config.astrometryDoRankUpdate,
1298 dataName=dataName)
1299
1300 add_measurement(self.job, 'jointcal.astrometry_final_chi2', chi2.chi2)
1301 add_measurement(self.job, 'jointcal.astrometry_final_ndof', chi2.ndof)
1302
1303 return Astrometry(fit, model, sky_to_tan_projection)
1304
Class that handles the astrometric least squares problem.
Definition AstrometryFit.h:78
A multi-component model, fitting mappings for sensors and visits simultaneously.
Definition ConstrainedAstrometryModel.h:62
A projection handler in which all CCDs from the same visit have the same tangent point.
Definition ProjectionHandler.h:80
A model where there is one independent transform per CcdImage.
Definition SimpleAstrometryModel.h:62
◆ _fit_photometry()
|
protected |
Fit the photometric data.
Parameters
----------
associations : `lsst.jointcal.Associations`
The star/reference star associations to fit.
dataName : `str`
Name of the data being processed (e.g. "1234_HSC-Y"), for
identifying debugging files.
Returns
-------
fit_result : `namedtuple`
fit : `lsst.jointcal.PhotometryFit`
The photometric fitter used to perform the fit.
model : `lsst.jointcal.PhotometryModel`
The photometric model that was fit.
Definition at line 1099 of file jointcal.py.
1099 def _fit_photometry(self, associations, dataName=None):
1100 """
1101 Fit the photometric data.
1102
1103 Parameters
1104 ----------
1105 associations : `lsst.jointcal.Associations`
1106 The star/reference star associations to fit.
1107 dataName : `str`
1108 Name of the data being processed (e.g. "1234_HSC-Y"), for
1109 identifying debugging files.
1110
1111 Returns
1112 -------
1113 fit_result : `namedtuple`
1114 fit : `lsst.jointcal.PhotometryFit`
1115 The photometric fitter used to perform the fit.
1116 model : `lsst.jointcal.PhotometryModel`
1117 The photometric model that was fit.
1118 """
1119 self.log.info("=== Starting photometric fitting...")
1120
1121 # TODO: should use pex.config.RegistryField here (see DM-9195)
1122 if self.config.photometryModel == "constrainedFlux":
1123 model = lsst.jointcal.ConstrainedFluxModel(associations.getCcdImageList(),
1124 self.focalPlaneBBox,
1125 visitOrder=self.config.photometryVisitOrder,
1126 errorPedestal=self.config.photometryErrorPedestal)
1127 # potentially nonlinear problem, so we may need a line search to converge.
1128 doLineSearch = self.config.allowLineSearch
1129 elif self.config.photometryModel == "constrainedMagnitude":
1130 model = lsst.jointcal.ConstrainedMagnitudeModel(associations.getCcdImageList(),
1131 self.focalPlaneBBox,
1132 visitOrder=self.config.photometryVisitOrder,
1133 errorPedestal=self.config.photometryErrorPedestal)
1134 # potentially nonlinear problem, so we may need a line search to converge.
1135 doLineSearch = self.config.allowLineSearch
1136 elif self.config.photometryModel == "simpleFlux":
1137 model = lsst.jointcal.SimpleFluxModel(associations.getCcdImageList(),
1138 errorPedestal=self.config.photometryErrorPedestal)
1139 doLineSearch = False # purely linear in model parameters, so no line search needed
1140 elif self.config.photometryModel == "simpleMagnitude":
1141 model = lsst.jointcal.SimpleMagnitudeModel(associations.getCcdImageList(),
1142 errorPedestal=self.config.photometryErrorPedestal)
1143 doLineSearch = False # purely linear in model parameters, so no line search needed
1144
1145 fit = lsst.jointcal.PhotometryFit(associations, model)
1146 # TODO DM-12446: turn this into a "butler save" somehow.
1147 # Save reference and measurement chi2 contributions for this data
1148 if self.config.writeChi2FilesInitialFinal:
1149 baseName = f"photometry_initial_chi2-{dataName}"
1150 else:
1151 baseName = None
1152 if self.config.writeInitialModel:
1153 fullpath = self._getDebugPath(f"initial_photometry_model-{dataName}.txt")
1154 writeModel(model, fullpath, self.log)
1155 self._logChi2AndValidate(associations, fit, model, "Initialized", writeChi2Name=baseName)
1156
1157 def getChi2Name(whatToFit):
1158 if self.config.writeChi2FilesOuterLoop:
1159 return f"photometry_init-%s_chi2-{dataName}" % whatToFit
1160 else:
1161 return None
1162
1163 # The constrained model needs the visit transform fit first; the chip
1164 # transform is initialized from the singleFrame PhotoCalib, so it's close.
1165 if self.config.writeInitMatrix:
1166 dumpMatrixFile = self._getDebugPath(f"photometry_preinit-{dataName}")
1167 else:
1168 dumpMatrixFile = ""
1169 if self.config.photometryModel.startswith("constrained"):
1170 # no line search: should be purely (or nearly) linear,
1171 # and we want a large step size to initialize with.
1172 fit.minimize("ModelVisit", dumpMatrixFile=dumpMatrixFile)
1173 self._logChi2AndValidate(associations, fit, model, "Initialize ModelVisit",
1174 writeChi2Name=getChi2Name("ModelVisit"))
1175 dumpMatrixFile = "" # so we don't redo the output on the next step
1176
1177 fit.minimize("Model", doLineSearch=doLineSearch, dumpMatrixFile=dumpMatrixFile)
1178 self._logChi2AndValidate(associations, fit, model, "Initialize Model",
1179 writeChi2Name=getChi2Name("Model"))
1180
1181 fit.minimize("Fluxes") # no line search: always purely linear.
1182 self._logChi2AndValidate(associations, fit, model, "Initialize Fluxes",
1183 writeChi2Name=getChi2Name("Fluxes"))
1184
1185 fit.minimize("Model Fluxes", doLineSearch=doLineSearch)
1186 self._logChi2AndValidate(associations, fit, model, "Initialize ModelFluxes",
1187 writeChi2Name=getChi2Name("ModelFluxes"))
1188
1189 model.freezeErrorTransform()
1190 self.log.debug("Photometry error scales are frozen.")
1191
1192 chi2 = self._iterate_fit(associations,
1193 fit,
1194 self.config.maxPhotometrySteps,
1195 "photometry",
1196 "Model Fluxes",
1197 doRankUpdate=self.config.photometryDoRankUpdate,
1198 doLineSearch=doLineSearch,
1199 dataName=dataName)
1200
1201 add_measurement(self.job, 'jointcal.photometry_final_chi2', chi2.chi2)
1202 add_measurement(self.job, 'jointcal.photometry_final_ndof', chi2.ndof)
1203 return Photometry(fit, model)
1204
Class that handles the photometric least squares problem.
Definition PhotometryFit.h:46
◆ _get_refcat_coordinate_error_override()
|
protected |
Check whether we should override the refcat coordinate errors, and
return the overridden error if necessary.
Parameters
----------
refCat : `lsst.afw.table.SimpleCatalog`
The reference catalog to check for a ``coord_raErr`` field.
name : `str`
Whether we are doing "astrometry" or "photometry".
Returns
-------
refCoordErr : `float`
The refcat coordinate error to use, or NaN if we are not overriding
those fields.
Raises
------
lsst.pex.config.FieldValidationError
Raised if the refcat does not contain coordinate errors and
``config.astrometryReferenceErr`` is not set.
Definition at line 840 of file jointcal.py.
840 def _get_refcat_coordinate_error_override(self, refCat, name):
841 """Check whether we should override the refcat coordinate errors, and
842 return the overridden error if necessary.
843
844 Parameters
845 ----------
846 refCat : `lsst.afw.table.SimpleCatalog`
847 The reference catalog to check for a ``coord_raErr`` field.
848 name : `str`
849 Whether we are doing "astrometry" or "photometry".
850
851 Returns
852 -------
853 refCoordErr : `float`
854 The refcat coordinate error to use, or NaN if we are not overriding
855 those fields.
856
857 Raises
858 ------
859 lsst.pex.config.FieldValidationError
860 Raised if the refcat does not contain coordinate errors and
861 ``config.astrometryReferenceErr`` is not set.
862 """
863 # This value doesn't matter for photometry, so just set something to
864 # keep old refcats from causing problems.
865 if name.lower() == "photometry":
866 if 'coord_raErr' not in refCat.schema:
867 return 100
868 else:
869 return float('nan')
870
871 if self.config.astrometryReferenceErr is None and 'coord_raErr' not in refCat.schema:
872 msg = ("Reference catalog does not contain coordinate errors, "
873 "and config.astrometryReferenceErr not supplied.")
874 raise pexConfig.FieldValidationError(JointcalConfig.astrometryReferenceErr,
875 self.config,
876 msg)
877
878 if self.config.astrometryReferenceErr is not None and 'coord_raErr' in refCat.schema:
879 self.log.warning("Overriding reference catalog coordinate errors with %f/coordinate [mas]",
880 self.config.astrometryReferenceErr)
881
882 if self.config.astrometryReferenceErr is None:
883 return float('nan')
884 else:
885 return self.config.astrometryReferenceErr
886
◆ _getDebugPath()
|
protected |
Constructs a path to filename using the configured debug path.
Definition at line 813 of file jointcal.py.
813 def _getDebugPath(self, filename):
814 """Constructs a path to filename using the configured debug path.
815 """
816 return os.path.join(self.config.debugOutputPath, filename)
817
◆ _iterate_fit()
|
protected |
Run fitter.minimize up to max_steps times, returning the final chi2.
Parameters
----------
associations : `lsst.jointcal.Associations`
The star/reference star associations to fit.
fitter : `lsst.jointcal.FitterBase`
The fitter to use for minimization.
max_steps : `int`
Maximum number of steps to run outlier rejection before declaring
convergence failure.
name : {'photometry' or 'astrometry'}
What type of data are we fitting (for logs and debugging files).
whatToFit : `str`
Passed to ``fitter.minimize()`` to define the parameters to fit.
dataName : `str`, optional
Descriptive name for this dataset (e.g. tract and filter),
for debugging.
sigmaRelativeTolerance : `float`, optional
Convergence tolerance for the fractional change in the chi2 cut
level for determining outliers. If set to zero, iterations will
continue until there are no outliers.
doRankUpdate : `bool`, optional
Do an Eigen rank update during minimization, or recompute the full
matrix and gradient?
doLineSearch : `bool`, optional
Do a line search for the optimum step during minimization?
Returns
-------
chi2: `lsst.jointcal.Chi2Statistic`
The final chi2 after the fit converges, or is forced to end.
Raises
------
FloatingPointError
Raised if the fitter fails with a non-finite value.
RuntimeError
Raised if the fitter fails for some other reason;
log messages will provide further details.
Definition at line 1318 of file jointcal.py.
1322 doLineSearch=False):
1323 """Run fitter.minimize up to max_steps times, returning the final chi2.
1324
1325 Parameters
1326 ----------
1327 associations : `lsst.jointcal.Associations`
1328 The star/reference star associations to fit.
1329 fitter : `lsst.jointcal.FitterBase`
1330 The fitter to use for minimization.
1331 max_steps : `int`
1332 Maximum number of steps to run outlier rejection before declaring
1333 convergence failure.
1334 name : {'photometry' or 'astrometry'}
1335 What type of data are we fitting (for logs and debugging files).
1336 whatToFit : `str`
1337 Passed to ``fitter.minimize()`` to define the parameters to fit.
1338 dataName : `str`, optional
1339 Descriptive name for this dataset (e.g. tract and filter),
1340 for debugging.
1341 sigmaRelativeTolerance : `float`, optional
1342 Convergence tolerance for the fractional change in the chi2 cut
1343 level for determining outliers. If set to zero, iterations will
1344 continue until there are no outliers.
1345 doRankUpdate : `bool`, optional
1346 Do an Eigen rank update during minimization, or recompute the full
1347 matrix and gradient?
1348 doLineSearch : `bool`, optional
1349 Do a line search for the optimum step during minimization?
1350
1351 Returns
1352 -------
1353 chi2: `lsst.jointcal.Chi2Statistic`
1354 The final chi2 after the fit converges, or is forced to end.
1355
1356 Raises
1357 ------
1358 FloatingPointError
1359 Raised if the fitter fails with a non-finite value.
1360 RuntimeError
1361 Raised if the fitter fails for some other reason;
1362 log messages will provide further details.
1363 """
1364 if self.config.writeInitMatrix:
1365 dumpMatrixFile = self._getDebugPath(f"{name}_postinit-{dataName}")
1366 else:
1367 dumpMatrixFile = ""
1368 oldChi2 = lsst.jointcal.Chi2Statistic()
1369 oldChi2.chi2 = float("inf")
1370 for i in range(max_steps):
1371 if self.config.writeChi2FilesOuterLoop:
1372 writeChi2Name = f"{name}_iterate_{i}_chi2-{dataName}"
1373 else:
1374 writeChi2Name = None
1375 result = fitter.minimize(whatToFit,
1376 self.config.outlierRejectSigma,
1377 sigmaRelativeTolerance=sigmaRelativeTolerance,
1378 doRankUpdate=doRankUpdate,
1379 doLineSearch=doLineSearch,
1380 dumpMatrixFile=dumpMatrixFile)
1381 dumpMatrixFile = "" # clear it so we don't write the matrix again.
1382 chi2 = self._logChi2AndValidate(associations, fitter, fitter.getModel(),
1383 f"Fit iteration {i}", writeChi2Name=writeChi2Name)
1384
1385 if result == MinimizeResult.Converged:
1386 if doRankUpdate:
1387 self.log.debug("fit has converged - no more outliers - redo minimization "
1388 "one more time in case we have lost accuracy in rank update.")
1389 # Redo minimization one more time in case we have lost accuracy in rank update
1390 result = fitter.minimize(whatToFit, self.config.outlierRejectSigma,
1391 sigmaRelativeTolerance=sigmaRelativeTolerance)
1392 chi2 = self._logChi2AndValidate(associations, fitter, fitter.getModel(), "Fit completed")
1393
1394 # log a message for a large final chi2, TODO: DM-15247 for something better
1395 if chi2.chi2/chi2.ndof >= 4.0:
1396 self.log.error("Potentially bad fit: High chi-squared/ndof.")
1397
1398 break
1399 elif result == MinimizeResult.Chi2Increased:
1400 self.log.warning("Still some outliers remaining but chi2 increased - retry")
1401 # Check whether the increase was large enough to cause trouble.
1402 chi2Ratio = chi2.chi2 / oldChi2.chi2
1403 if chi2Ratio > 1.5:
1404 self.log.warning('Significant chi2 increase by a factor of %.4g / %.4g = %.4g',
1405 chi2.chi2, oldChi2.chi2, chi2Ratio)
1406 # Based on a variety of HSC jointcal logs (see DM-25779), it
1407 # appears that chi2 increases more than a factor of ~2 always
1408 # result in the fit diverging rapidly and ending at chi2 > 1e10.
1409 # Using 10 as the "failure" threshold gives some room between
1410 # leaving a warning and bailing early.
1411 if chi2Ratio > 10:
1412 msg = ("Large chi2 increase between steps: fit likely cannot converge."
1413 " Try setting one or more of the `writeChi2*` config fields and looking"
1414 " at how individual star chi2-values evolve during the fit.")
1415 raise RuntimeError(msg)
1416 oldChi2 = chi2
1417 elif result == MinimizeResult.NonFinite:
1418 filename = self._getDebugPath("{}_failure-nonfinite_chi2-{}.csv".format(name, dataName))
1419 # TODO DM-12446: turn this into a "butler save" somehow.
1420 fitter.saveChi2Contributions(filename+"{type}")
1421 msg = "Nonfinite value in chi2 minimization, cannot complete fit. Dumped star tables to: {}"
1422 raise FloatingPointError(msg.format(filename))
1423 elif result == MinimizeResult.Failed:
1424 raise RuntimeError("Chi2 minimization failure, cannot complete fit.")
1425 else:
1426 raise RuntimeError("Unxepected return code from minimize().")
1427 else:
1428 self.log.error("%s failed to converge after %d steps"%(name, max_steps))
1429
1430 return chi2
1431
◆ _load_data()
|
protected |
Read the data that jointcal needs to run.
Modifies ``associations`` in-place with the loaded data.
Parameters
----------
inputSourceTableVisit : `list` [`lsst.daf.butler.DeferredDatasetHandle`]
References to visit-level DataFrames to load the catalog data from.
inputVisitSummary : `list` [`lsst.daf.butler.DeferredDatasetHandle`]
Visit-level exposure summary catalog with metadata.
associations : `lsst.jointcal.Associations`
Object to add the loaded data to by constructing new CcdImages.
jointcalControl : `jointcal.JointcalControl`
Control object for C++ associations management.
camera : `lsst.afw.cameraGeom.Camera`
Camera object for detector geometry.
Returns
-------
oldWcsList: `list` [`lsst.afw.geom.SkyWcs`]
The original WCS of the input data, to aid in writing tests.
bands : `list` [`str`]
The filter bands of each input dataset.
Definition at line 677 of file jointcal.py.
678 jointcalControl, camera):
679 """Read the data that jointcal needs to run.
680
681 Modifies ``associations`` in-place with the loaded data.
682
683 Parameters
684 ----------
685 inputSourceTableVisit : `list` [`lsst.daf.butler.DeferredDatasetHandle`]
686 References to visit-level DataFrames to load the catalog data from.
687 inputVisitSummary : `list` [`lsst.daf.butler.DeferredDatasetHandle`]
688 Visit-level exposure summary catalog with metadata.
689 associations : `lsst.jointcal.Associations`
690 Object to add the loaded data to by constructing new CcdImages.
691 jointcalControl : `jointcal.JointcalControl`
692 Control object for C++ associations management.
693 camera : `lsst.afw.cameraGeom.Camera`
694 Camera object for detector geometry.
695
696 Returns
697 -------
698 oldWcsList: `list` [`lsst.afw.geom.SkyWcs`]
699 The original WCS of the input data, to aid in writing tests.
700 bands : `list` [`str`]
701 The filter bands of each input dataset.
702 """
703 oldWcsList = []
704 filters = []
705 load_cat_profile_file = 'jointcal_load_data.prof' if self.config.detailedProfile else ''
706 with lsst.utils.timer.profile(load_cat_profile_file):
707 table = make_schema_table() # every detector catalog has the same layout
708 # No guarantee that the input is in the same order of visits, so we have to map one of them.
709 catalogMap = {ref.dataId['visit']: i for i, ref in enumerate(inputSourceTableVisit)}
710 detectorDict = {detector.getId(): detector for detector in camera}
711
712 columns = None
713
714 for visitSummaryRef in inputVisitSummary:
715 visitSummary = visitSummaryRef.get()
716
717 dataRef = inputSourceTableVisit[catalogMap[visitSummaryRef.dataId['visit']]]
718 if columns is None:
719 inColumns = dataRef.get(component='columns')
720 columns, ixxColumns = get_sourceTable_visit_columns(inColumns,
721 self.config,
722 self.sourceSelector)
723 visitCatalog = dataRef.get(parameters={'columns': columns})
724
725 selected = self.sourceSelector.run(visitCatalog)
726 if len(selected.sourceCat) == 0:
727 self.log.warning("No sources selected in visit %s. Skipping...",
728 visitSummary["visit"][0])
729 continue
730
731 # Build a CcdImage for each detector in this visit.
732 detectors = {id: index for index, id in enumerate(visitSummary['id'])}
733 for id, index in detectors.items():
734 catalog = extract_detector_catalog_from_visit_catalog(table,
735 selected.sourceCat,
736 id,
737 ixxColumns,
738 self.config.sourceFluxType,
739 self.log)
740 if catalog is None:
741 continue
742 data = self._make_one_input_data(visitSummary[index], catalog, detectorDict)
743 result = self._build_ccdImage(data, associations, jointcalControl)
744 if result is not None:
745 oldWcsList.append(result.wcs)
746 # A visit has only one band, so we can just use the first.
747 filters.append(data.filter)
748 if len(filters) == 0:
749 raise RuntimeError("No data to process: did source selector remove all sources?")
750 filters = collections.Counter(filters)
751
752 return oldWcsList, filters
753
◆ _load_reference_catalog()
|
protected |
Load the necessary reference catalog sources, convert fluxes to
correct units, and apply color term corrections if requested.
Parameters
----------
refObjLoader : `lsst.meas.algorithms.ReferenceObjectLoader`
The reference catalog loader to use to get the data.
referenceSelector : `lsst.meas.algorithms.ReferenceSourceSelectorTask`
Source selector to apply to loaded reference catalog.
center : `lsst.geom.SpherePoint`
The center around which to load sources.
radius : `lsst.geom.Angle`
The radius around ``center`` to load sources in.
filterLabel : `lsst.afw.image.FilterLabel`
The camera filter to load fluxes for.
applyColorterms : `bool`
Apply colorterm corrections to the refcat for ``filterName``?
epoch : `astropy.time.Time`, optional
Epoch to which to correct refcat proper motion and parallax,
or `None` to not apply such corrections.
Returns
-------
refCat : `lsst.afw.table.SimpleCatalog`
The loaded reference catalog.
fluxField : `str`
The name of the reference catalog flux field appropriate for ``filterName``.
Definition at line 990 of file jointcal.py.
991 applyColorterms=False, epoch=None):
992 """Load the necessary reference catalog sources, convert fluxes to
993 correct units, and apply color term corrections if requested.
994
995 Parameters
996 ----------
997 refObjLoader : `lsst.meas.algorithms.ReferenceObjectLoader`
998 The reference catalog loader to use to get the data.
999 referenceSelector : `lsst.meas.algorithms.ReferenceSourceSelectorTask`
1000 Source selector to apply to loaded reference catalog.
1001 center : `lsst.geom.SpherePoint`
1002 The center around which to load sources.
1003 radius : `lsst.geom.Angle`
1004 The radius around ``center`` to load sources in.
1005 filterLabel : `lsst.afw.image.FilterLabel`
1006 The camera filter to load fluxes for.
1007 applyColorterms : `bool`
1008 Apply colorterm corrections to the refcat for ``filterName``?
1009 epoch : `astropy.time.Time`, optional
1010 Epoch to which to correct refcat proper motion and parallax,
1011 or `None` to not apply such corrections.
1012
1013 Returns
1014 -------
1015 refCat : `lsst.afw.table.SimpleCatalog`
1016 The loaded reference catalog.
1017 fluxField : `str`
1018 The name of the reference catalog flux field appropriate for ``filterName``.
1019 """
1020 skyCircle = refObjLoader.loadSkyCircle(center,
1021 radius,
1022 filterLabel.bandLabel,
1023 epoch=epoch)
1024
1025 selected = referenceSelector.run(skyCircle.refCat)
1026 # Need memory contiguity to get reference filters as a vector.
1027 if not selected.sourceCat.isContiguous():
1028 refCat = selected.sourceCat.copy(deep=True)
1029 else:
1030 refCat = selected.sourceCat
1031
1032 if applyColorterms:
1033 refCatName = refObjLoader.name
1034 self.log.info("Applying color terms for physical filter=%r reference catalog=%s",
1035 filterLabel.physicalLabel, refCatName)
1036 colorterm = self.config.colorterms.getColorterm(filterLabel.physicalLabel,
1037 refCatName,
1038 doRaise=True)
1039
1040 refMag, refMagErr = colorterm.getCorrectedMagnitudes(refCat)
1041 refCat[skyCircle.fluxField] = u.Magnitude(refMag, u.ABmag).to_value(u.nJy)
1042 # TODO: I didn't want to use this, but I'll deal with it in DM-16903
1043 refCat[skyCircle.fluxField+'Err'] = fluxErrFromABMagErr(refMagErr, refMag) * 1e9
1044
1045 return refCat, skyCircle.fluxField
1046
◆ _logChi2AndValidate()
|
protected |
Compute chi2, log it, validate the model, and return chi2.
Parameters
----------
associations : `lsst.jointcal.Associations`
The star/reference star associations to fit.
fit : `lsst.jointcal.FitterBase`
The fitter to use for minimization.
model : `lsst.jointcal.Model`
The model being fit.
chi2Label : `str`
Label to describe the chi2 (e.g. "Initialized", "Final").
writeChi2Name : `str`, optional
Filename prefix to write the chi2 contributions to.
Do not supply an extension: an appropriate one will be added.
Returns
-------
chi2: `lsst.jointcal.Chi2Accumulator`
The chi2 object for the current fitter and model.
Raises
------
FloatingPointError
Raised if chi2 is infinite or NaN.
ValueError
Raised if the model is not valid.
Definition at line 1056 of file jointcal.py.
1056 def _logChi2AndValidate(self, associations, fit, model, chi2Label, writeChi2Name=None):
1057 """Compute chi2, log it, validate the model, and return chi2.
1058
1059 Parameters
1060 ----------
1061 associations : `lsst.jointcal.Associations`
1062 The star/reference star associations to fit.
1063 fit : `lsst.jointcal.FitterBase`
1064 The fitter to use for minimization.
1065 model : `lsst.jointcal.Model`
1066 The model being fit.
1067 chi2Label : `str`
1068 Label to describe the chi2 (e.g. "Initialized", "Final").
1069 writeChi2Name : `str`, optional
1070 Filename prefix to write the chi2 contributions to.
1071 Do not supply an extension: an appropriate one will be added.
1072
1073 Returns
1074 -------
1075 chi2: `lsst.jointcal.Chi2Accumulator`
1076 The chi2 object for the current fitter and model.
1077
1078 Raises
1079 ------
1080 FloatingPointError
1081 Raised if chi2 is infinite or NaN.
1082 ValueError
1083 Raised if the model is not valid.
1084 """
1085 if writeChi2Name is not None:
1086 fullpath = self._getDebugPath(writeChi2Name)
1087 fit.saveChi2Contributions(fullpath+"{type}")
1088 self.log.info("Wrote chi2 contributions files: %s", fullpath)
1089
1090 chi2 = fit.computeChi2()
1091 self.log.info("%s %s", chi2Label, chi2)
1092 self._check_stars(associations)
1093 if not np.isfinite(chi2.chi2):
1094 raise FloatingPointError(f'{chi2Label} chi2 is invalid: {chi2}')
1095 if not model.validate(associations.getCcdImageList(), chi2.ndof):
1096 raise ValueError("Model is not valid: check log messages for warnings.")
1097 return chi2
1098
◆ _make_one_input_data()
|
protected |
Return a data structure for this detector+visit.
Definition at line 754 of file jointcal.py.
754 def _make_one_input_data(self, visitRecord, catalog, detectorDict):
755 """Return a data structure for this detector+visit."""
756 return JointcalInputData(visit=visitRecord['visit'],
757 catalog=catalog,
758 visitInfo=visitRecord.getVisitInfo(),
759 detector=detectorDict[visitRecord.getId()],
760 photoCalib=visitRecord.getPhotoCalib(),
761 wcs=visitRecord.getWcs(),
762 bbox=visitRecord.getBBox(),
763 # ExposureRecord doesn't have a FilterLabel yet,
764 # so we have to make one.
766 physical=visitRecord['physical_filter']))
767
A group of labels for a filter in an exposure or coadd.
Definition FilterLabel.h:58
◆ _make_output()
|
protected |
Return the internal jointcal models converted to the afw
structures that will be saved to disk.
Parameters
----------
ccdImageList : `lsst.jointcal.CcdImageList`
The list of CcdImages to get the output for.
model : `lsst.jointcal.AstrometryModel` or `lsst.jointcal.PhotometryModel`
The internal jointcal model to convert for each `lsst.jointcal.CcdImage`.
func : `str`
The name of the function to call on ``model`` to get the converted
structure. Must accept an `lsst.jointcal.CcdImage`.
Returns
-------
output : `dict` [`tuple`, `lsst.jointcal.AstrometryModel`] or
`dict` [`tuple`, `lsst.jointcal.PhotometryModel`]
The data to be saved, keyed on (visit, detector).
Definition at line 1432 of file jointcal.py.
1432 def _make_output(self, ccdImageList, model, func):
1433 """Return the internal jointcal models converted to the afw
1434 structures that will be saved to disk.
1435
1436 Parameters
1437 ----------
1438 ccdImageList : `lsst.jointcal.CcdImageList`
1439 The list of CcdImages to get the output for.
1440 model : `lsst.jointcal.AstrometryModel` or `lsst.jointcal.PhotometryModel`
1441 The internal jointcal model to convert for each `lsst.jointcal.CcdImage`.
1442 func : `str`
1443 The name of the function to call on ``model`` to get the converted
1444 structure. Must accept an `lsst.jointcal.CcdImage`.
1445
1446 Returns
1447 -------
1448 output : `dict` [`tuple`, `lsst.jointcal.AstrometryModel`] or
1449 `dict` [`tuple`, `lsst.jointcal.PhotometryModel`]
1450 The data to be saved, keyed on (visit, detector).
1451 """
1452 output = {}
1453 for ccdImage in ccdImageList:
1454 ccd = ccdImage.ccdId
1455 visit = ccdImage.visit
1456 self.log.debug("%s for visit: %d, ccd: %d", func, visit, ccd)
1457 output[(visit, ccd)] = getattr(model, func)(ccdImage)
1458 return output
1459
1460
◆ _prep_sky()
|
protected |
Prepare on-sky and other data that must be computed after data has been read.
Definition at line 818 of file jointcal.py.
818 def _prep_sky(self, associations, filters):
819 """Prepare on-sky and other data that must be computed after data has
820 been read.
821 """
822 associations.computeCommonTangentPoint()
823
824 boundingCircle = associations.computeBoundingCircle()
825 center = lsst.geom.SpherePoint(boundingCircle.getCenter())
826 radius = lsst.geom.Angle(boundingCircle.getOpeningAngle().asRadians(), lsst.geom.radians)
827
828 self.log.info(f"Data has center={center} with radius={radius.asDegrees()} degrees.")
829
830 # Determine a default filter band associated with the catalog. See DM-9093
831 defaultFilter = filters.most_common(1)[0][0]
832 self.log.debug("Using '%s' filter for reference flux", defaultFilter.physicalLabel)
833
834 # compute and set the reference epoch of the observations, for proper motion corrections
835 epoch = self._compute_proper_motion_epoch(associations.getCcdImageList())
836 associations.setEpoch(epoch.jyear)
837
838 return boundingCircle, center, radius, defaultFilter, epoch
839
Point in an unspecified spherical coordinate system.
Definition SpherePoint.h:57
◆ _put_metrics()
|
protected |
Persist all measured metrics stored in a job.
Parameters
----------
butlerQC : `lsst.pipe.base.QuantumContext`
A butler which is specialized to operate in the context of a
`lsst.daf.butler.Quantum`; This is the input to `runQuantum`.
job : `lsst.verify.job.Job`
Measurements of metrics to persist.
outputRefs : `list` [`lsst.pipe.base.OutputQuantizedConnection`]
The DatasetRefs to persist the data to.
Definition at line 549 of file jointcal.py.
549 def _put_metrics(self, butlerQC, job, outputRefs):
550 """Persist all measured metrics stored in a job.
551
552 Parameters
553 ----------
554 butlerQC : `lsst.pipe.base.QuantumContext`
555 A butler which is specialized to operate in the context of a
556 `lsst.daf.butler.Quantum`; This is the input to `runQuantum`.
557 job : `lsst.verify.job.Job`
558 Measurements of metrics to persist.
559 outputRefs : `list` [`lsst.pipe.base.OutputQuantizedConnection`]
560 The DatasetRefs to persist the data to.
561 """
562 for key in job.measurements.keys():
563 butlerQC.put(job.measurements[key], getattr(outputRefs, key.fqn.replace('jointcal.', '')))
564
◆ _put_output()
|
protected |
Persist the output datasets to their appropriate datarefs.
Parameters
----------
butlerQC : `lsst.pipe.base.QuantumContext`
A butler which is specialized to operate in the context of a
`lsst.daf.butler.Quantum`; This is the input to `runQuantum`.
outputs : `dict` [`tuple`, `lsst.afw.geom.SkyWcs`] or
`dict` [`tuple, `lsst.afw.image.PhotoCalib`]
The fitted objects to persist.
outputRefs : `list` [`lsst.pipe.base.OutputQuantizedConnection`]
The DatasetRefs to persist the data to.
camera : `lsst.afw.cameraGeom.Camera`
The camera for this instrument, to get detector ids from.
setter : `str`
The method to call on the ExposureCatalog to set each output.
Definition at line 565 of file jointcal.py.
565 def _put_output(self, butlerQC, outputs, outputRefs, camera, setter):
566 """Persist the output datasets to their appropriate datarefs.
567
568 Parameters
569 ----------
570 butlerQC : `lsst.pipe.base.QuantumContext`
571 A butler which is specialized to operate in the context of a
572 `lsst.daf.butler.Quantum`; This is the input to `runQuantum`.
573 outputs : `dict` [`tuple`, `lsst.afw.geom.SkyWcs`] or
574 `dict` [`tuple, `lsst.afw.image.PhotoCalib`]
575 The fitted objects to persist.
576 outputRefs : `list` [`lsst.pipe.base.OutputQuantizedConnection`]
577 The DatasetRefs to persist the data to.
578 camera : `lsst.afw.cameraGeom.Camera`
579 The camera for this instrument, to get detector ids from.
580 setter : `str`
581 The method to call on the ExposureCatalog to set each output.
582 """
583 schema = lsst.afw.table.ExposureTable.makeMinimalSchema()
584 schema.addField('visit', type='L', doc='Visit number')
585
586 def new_catalog(visit, size):
587 """Return an catalog ready to be filled with appropriate output."""
588 catalog = lsst.afw.table.ExposureCatalog(schema)
589 catalog.resize(size)
590 catalog['visit'] = visit
591 metadata = lsst.daf.base.PropertyList()
592 metadata.add("COMMENT", "Catalog id is detector id, sorted.")
593 metadata.add("COMMENT", "Only detectors with data have entries.")
594 return catalog
595
596 # count how many detectors have output for each visit
597 detectors_per_visit = collections.defaultdict(int)
598 for key in outputs:
599 # key is (visit, detector_id), and we only need visit here
600 detectors_per_visit[key[0]] += 1
601
602 for ref in outputRefs:
603 visit = ref.dataId['visit']
604 catalog = new_catalog(visit, detectors_per_visit[visit])
605 # Iterate over every detector and skip the ones we don't have output for.
606 i = 0
607 for detector in camera:
608 detectorId = detector.getId()
609 key = (ref.dataId['visit'], detectorId)
610 if key not in outputs:
611 # skip detectors we don't have output for
612 self.log.debug("No %s output for detector %s in visit %s",
613 setter[3:], detectorId, visit)
614 continue
615
616 catalog[i].setId(detectorId)
617 getattr(catalog[i], setter)(outputs[key])
618 i += 1
619
620 catalog.sort() # ensure that the detectors are in sorted order, for fast lookups
621 butlerQC.put(catalog, ref)
622 self.log.info("Wrote %s detectors to %s", i, ref)
623
Custom catalog class for ExposureRecord/Table.
Definition Exposure.h:310
static Schema makeMinimalSchema()
Return a minimal schema for Exposure tables and records.
Definition Exposure.h:216
Class for storing ordered metadata with comments.
Definition PropertyList.h:68
◆ run()
| lsst.jointcal.jointcal.JointcalTask.run | ( | self, | |
| inputSourceTableVisit, | |||
| inputVisitSummary, | |||
| inputCamera, | |||
| tract = None ) |
Definition at line 624 of file jointcal.py.
624 def run(self, inputSourceTableVisit, inputVisitSummary, inputCamera, tract=None):
625 # Docstring inherited.
626
627 # We take values out of the Parquet table, and put them in "flux_",
628 # and the config.sourceFluxType field is used during that extraction,
629 # so just use "flux" here.
630 sourceFluxField = "flux"
631 jointcalControl = lsst.jointcal.JointcalControl(sourceFluxField)
632 associations = lsst.jointcal.Associations()
633 self.focalPlaneBBox = inputCamera.getFpBBox()
634 oldWcsList, bands = self._load_data(inputSourceTableVisit,
635 inputVisitSummary,
636 associations,
637 jointcalControl,
638 inputCamera)
639
640 boundingCircle, center, radius, defaultFilter, epoch = self._prep_sky(associations, bands)
641
642 if self.config.doAstrometry:
643 astrometry = self._do_load_refcat_and_fit(associations, defaultFilter, center, radius,
644 name="astrometry",
645 refObjLoader=self.astrometryRefObjLoader,
646 referenceSelector=self.astrometryReferenceSelector,
647 fit_function=self._fit_astrometry,
648 tract=tract,
649 epoch=epoch)
650 astrometry_output = self._make_output(associations.getCcdImageList(),
651 astrometry.model,
652 "makeSkyWcs")
653 else:
654 astrometry_output = None
655
656 if self.config.doPhotometry:
657 photometry = self._do_load_refcat_and_fit(associations, defaultFilter, center, radius,
658 name="photometry",
659 refObjLoader=self.photometryRefObjLoader,
660 referenceSelector=self.photometryReferenceSelector,
661 fit_function=self._fit_photometry,
662 tract=tract,
663 epoch=epoch,
664 reject_bad_fluxes=True)
665 photometry_output = self._make_output(associations.getCcdImageList(),
666 photometry.model,
667 "toPhotoCalib")
668 else:
669 photometry_output = None
670
671 return pipeBase.Struct(outputWcs=astrometry_output,
672 outputPhotoCalib=photometry_output,
673 job=self.job,
674 astrometryRefObjLoader=self.astrometryRefObjLoader,
675 photometryRefObjLoader=self.photometryRefObjLoader)
676
The class that implements the relations between MeasuredStar and FittedStar.
Definition Associations.h:55
◆ runQuantum()
| lsst.jointcal.jointcal.JointcalTask.runQuantum | ( | self, | |
| butlerQC, | |||
| inputRefs, | |||
| outputRefs ) |
Definition at line 518 of file jointcal.py.
518 def runQuantum(self, butlerQC, inputRefs, outputRefs):
519 # We override runQuantum to set up the refObjLoaders and write the
520 # outputs to the correct refs.
521 inputs = butlerQC.get(inputRefs)
522 # We want the tract number for writing debug files
523 tract = butlerQC.quantum.dataId['tract']
524 if self.config.doAstrometry:
525 self.astrometryRefObjLoader = ReferenceObjectLoader(
526 dataIds=[ref.datasetRef.dataId
527 for ref in inputRefs.astrometryRefCat],
528 refCats=inputs.pop('astrometryRefCat'),
529 config=self.config.astrometryRefObjLoader,
530 name=self.config.connections.astrometryRefCat,
531 log=self.log)
532 if self.config.doPhotometry:
533 self.photometryRefObjLoader = ReferenceObjectLoader(
534 dataIds=[ref.datasetRef.dataId
535 for ref in inputRefs.photometryRefCat],
536 refCats=inputs.pop('photometryRefCat'),
537 config=self.config.photometryRefObjLoader,
538 name=self.config.connections.photometryRefCat,
539 log=self.log)
540 outputs = self.run(**inputs, tract=tract)
541 self._put_metrics(butlerQC, outputs.job, outputRefs)
542 if self.config.doAstrometry:
543 self._put_output(butlerQC, outputs.outputWcs, outputRefs.outputWcs,
544 inputs['inputCamera'], "setWcs")
545 if self.config.doPhotometry:
546 self._put_output(butlerQC, outputs.outputPhotoCalib, outputRefs.outputPhotoCalib,
547 inputs['inputCamera'], "setPhotoCalib")
548
Member Data Documentation
◆ _DefaultName
|
staticprotected |
Definition at line 501 of file jointcal.py.
◆ astrometryRefObjLoader
| lsst.jointcal.jointcal.JointcalTask.astrometryRefObjLoader |
Definition at line 509 of file jointcal.py.
◆ config
| lsst.jointcal.jointcal.JointcalTask.config |
Definition at line 721 of file jointcal.py.
◆ ConfigClass
|
static |
Definition at line 500 of file jointcal.py.
◆ focalPlaneBBox
| lsst.jointcal.jointcal.JointcalTask.focalPlaneBBox |
Definition at line 633 of file jointcal.py.
◆ job
| lsst.jointcal.jointcal.JointcalTask.job |
Definition at line 516 of file jointcal.py.
◆ log
| lsst.jointcal.jointcal.JointcalTask.log |
Definition at line 739 of file jointcal.py.
◆ photometryRefObjLoader
| lsst.jointcal.jointcal.JointcalTask.photometryRefObjLoader |
Definition at line 513 of file jointcal.py.
◆ sourceSelector
| lsst.jointcal.jointcal.JointcalTask.sourceSelector |
Definition at line 722 of file jointcal.py.
The documentation for this class was generated from the following file:
- /j/snowflake/release/lsstsw/stack/lsst-scipipe-9.0.0/Linux64/jointcal/g82479be7b0+d730eedb7d/python/lsst/jointcal/jointcal.py
