LSST Applications g0265f82a02+c6dfa2ddaf,g1162b98a3f+ffe7eabc7e,g2079a07aa2+1b2e822518,g2bbee38e9b+c6dfa2ddaf,g337abbeb29+c6dfa2ddaf,g36da64cc00+ea84795170,g3ddfee87b4+955a963fd8,g50ff169b8f+2eb0e556e8,g52b1c1532d+90ebb246c7,g555ede804d+955a963fd8,g591dd9f2cf+bac198a2cb,g5ec818987f+420292cfeb,g858d7b2824+d6c9a0a3b8,g876c692160+aabc49a3c3,g8a8a8dda67+90ebb246c7,g8cdfe0ae6a+4fd9e222a8,g99cad8db69+e6cd765486,g9ddcbc5298+a1346535a5,ga1e77700b3+df8f93165b,ga8c6da7877+acd47f83f4,gae46bcf261+c6dfa2ddaf,gb0e22166c9+8634eb87fb,gb3f2274832+12c8382528,gba4ed39666+1ac82b564f,gbb8dafda3b+0574160a1f,gbeb006f7da+dea2fbb49f,gc28159a63d+c6dfa2ddaf,gc86a011abf+d6c9a0a3b8,gcf0d15dbbd+955a963fd8,gdaeeff99f8+1cafcb7cd4,gdc0c513512+d6c9a0a3b8,ge79ae78c31+c6dfa2ddaf,geb67518f79+ba1859f325,gee10cc3b42+90ebb246c7,gf1cff7945b+d6c9a0a3b8,w.2024.13
LSST Data Management Base Package
Loading...
Searching...
No Matches
Public Member Functions | List of all members
lsst.meas.base._id_generator.IdGenerator Class Reference
Inheritance diagram for lsst.meas.base._id_generator.IdGenerator:
lsst.meas.base._id_generator.FullIdGenerator

Public Member Functions

int catalog_id (self)
 
str __str__ (self)
 
IdFactory make_table_id_factory (self)
 
SourceCatalog make_source_catalog (self, Schema schema)
 
np.ndarray arange (self, *args, **kwargs)
 
Callable[[int], tuple[DataCoordinate, int]] unpacker_from_config (cls, BaseIdGeneratorConfig config, DataCoordinate fixed)
 
Callable[[int], tuple[int, DataCoordinate, int]] unpacker_from_dimension_packer (cls, DimensionPacker dimension_packer, int n_releases=DEFAULT_N_RELEASES)
 

Detailed Description

A helper class for packing some combination of a data ID, a per-data-ID
counter, and a release ID into a single 64-bit integer.

As an object frequently passed into code that otherwise has no knowledge of
its own data ID, `IdGenerator` also implements ``__str__`` to provide a
human-readable representation of the data ID for use in logs and exception
messages, with a suitable fallback when no data ID was provided to it.

Notes
-----
Instances of this class are expected to usually be created via
configuration, which will return a derived instance.  This pattern starts
with one of `DetectorExposureIdGeneratorConfig`,
`DetectorVisitIdGeneratorConfig`, and `SkyMapIdGeneratorConfig` (which have
the same interface), and looks something this:

    from lsst.meas.base import DetectorVisitIdGeneratorConfig
    from lsst.pex.config import Config
    from lsst.pipe.base import PipelineTask

    class SomeTaskConfig(PipelineTaskConfig, ...):
        id_generator = DetectorVisitIdGeneratorConfig.make_field()

    class SomeTask(PipelineTaskTask):

        ConfigClass = SomeTaskConfig

        ...

        def runQuantum(self, ..., data_id: DataCoordinate):
            id_generator = self.config.apply(data_id)
            catalog = id_generator.make_source_catalog(self.schema) ...

There is no requirement that `IdGenerator` instances be constructed in
`PipelineTask.runQuantum` methods and passed to the ``run`` method, but
this is the most common approach.

Code that wishes to instead unpack these record IDs to obtain the release
ID, data ID and counter value should use the same config (often loaded from
the ``Butler``) and pass a fully-expanded data ID identifying only a
particular ``skymap`` or ``instrument`` to `unpacker_from_config`::

    config = butler.get("some_task_config")
    catalog = butler.get("some_output_catalog", given_data_id)
    unpacker = IdGenerator.unpacker_from_config(
        config.id_generator, butler.registry.expandDataId(skymap="HSC"),
    )
    release_id, embedded_data_id, counter = unpacker(catalog[0]["id"])
    assert embedded_data_id == given_data_id

This example is a bit contrived, as the ability to reconstruct the data ID
is really only useful when you don't have it already, such as when the
record ID is obtained from some further-processed version of the original
table (such as a SQL database), and in that context the right config to
load will not be obvious unless it has been carefully documented.

Simple instances of the base class that do not include a data ID may also
be constructed by calling the constructor directly::

    id_generator = IdGenerator()

These IDs may not be unpacked, but they also don't need to be, because
they're just the per-catalog "counter" integer already.

See Also
--------
:ref:`lsst.meas.base-generating-source-and-object-ids`

Definition at line 213 of file _id_generator.py.

Member Function Documentation

◆ __str__()

str lsst.meas.base._id_generator.IdGenerator.__str__ ( self)
Return a human-readable representation of the data ID (or a note
about its absence) for use in log and error messages.

Reimplemented in lsst.meas.base._id_generator.FullIdGenerator.

Definition at line 293 of file _id_generator.py.

293 def __str__(self) -> str:
294 """Return a human-readable representation of the data ID (or a note
295 about its absence) for use in log and error messages.
296 """
297 return "[no data ID]"
298

◆ arange()

np.ndarray lsst.meas.base._id_generator.IdGenerator.arange ( self,
* args,
** kwargs )
Generate an array of integer IDs for this catalog.

All parameters are forwarded to `numpy.arange` to generate an array of
per-catalog counter integers.  These are then combined with the
`catalog_id`` to form the returned array.

The IDs generated by `arange` will be equivalent to those generated by
`make_table_id_factory` (and by extension, `make_source_catalog`) only
if the counter integers start with ``1``, not ``0``, because that's
what `~lsst.afw.table.IdFactory` does.

Reimplemented in lsst.meas.base._id_generator.FullIdGenerator.

Definition at line 314 of file _id_generator.py.

314 def arange(self, *args, **kwargs) -> np.ndarray:
315 """Generate an array of integer IDs for this catalog.
316
317 All parameters are forwarded to `numpy.arange` to generate an array of
318 per-catalog counter integers. These are then combined with the
319 `catalog_id`` to form the returned array.
320
321 The IDs generated by `arange` will be equivalent to those generated by
322 `make_table_id_factory` (and by extension, `make_source_catalog`) only
323 if the counter integers start with ``1``, not ``0``, because that's
324 what `~lsst.afw.table.IdFactory` does.
325 """
326 return np.arange(*args, **kwargs)
327

◆ catalog_id()

int lsst.meas.base._id_generator.IdGenerator.catalog_id ( self)
The integer identifier for the full catalog with this data ID, not
just one of its rows (`int`).

This combines the packed data ID and release ID, but not the
counter.

Reimplemented in lsst.meas.base._id_generator.FullIdGenerator.

Definition at line 284 of file _id_generator.py.

284 def catalog_id(self) -> int:
285 """The integer identifier for the full catalog with this data ID, not
286 just one of its rows (`int`).
287
288 This combines the packed data ID and release ID, but not the
289 counter.
290 """
291 return 0
292

◆ make_source_catalog()

SourceCatalog lsst.meas.base._id_generator.IdGenerator.make_source_catalog ( self,
Schema schema )
Construct a empty catalog object with an ID factory.

This is a convenience function for the common pattern of calling
`make_table_id_factory`, constructing a `~lsst.afw.table.SourceTable`
from that, and then constructing an (empty)
`~lsst.afw.table.SourceCatalog` from that.

Definition at line 303 of file _id_generator.py.

303 def make_source_catalog(self, schema: Schema) -> SourceCatalog:
304 """Construct a empty catalog object with an ID factory.
305
306 This is a convenience function for the common pattern of calling
307 `make_table_id_factory`, constructing a `~lsst.afw.table.SourceTable`
308 from that, and then constructing an (empty)
309 `~lsst.afw.table.SourceCatalog` from that.
310 """
311 table = SourceTable.make(schema, self.make_table_id_factory())
312 return SourceCatalog(table)
313

◆ make_table_id_factory()

IdFactory lsst.meas.base._id_generator.IdGenerator.make_table_id_factory ( self)
Construct a new `lsst.afw.table.IdFactory` for this catalog.

Reimplemented in lsst.meas.base._id_generator.FullIdGenerator.

Definition at line 299 of file _id_generator.py.

299 def make_table_id_factory(self) -> IdFactory:
300 """Construct a new `lsst.afw.table.IdFactory` for this catalog."""
301 return IdFactory.makeSimple()
302

◆ unpacker_from_config()

Callable[[int], tuple[DataCoordinate, int]] lsst.meas.base._id_generator.IdGenerator.unpacker_from_config ( cls,
BaseIdGeneratorConfig config,
DataCoordinate fixed )
Return a callable that unpacks the IDs generated by this class,
from a config field.

Parameters
----------
config : `BaseIdGeneratorConfig`
    Configuration for an ID generator.
fixed : `DataCoordinate`
    Data ID identifying the dimensions that are considered fixed by the
    `IdGenerator` that produced the IDs: usually just ``instrument`` or
    ``skymap``, depending on the configuration. For most configurations
    this will need to be a fully-expanded data ID.

Returns
-------
unpacker
    Callable that takes a single `int` argument (an ID generated by an
    identically-configured `IdGenerator`) and returns a tuple of:

    - release_id: the integer that identifies a data release or
      similar (`int`);
    - data_id : the data ID used to initialize the original ID
      generator (`DataCoordinate`);
    - counter : the counter part of the original ID (`int`).

Notes
-----
This method cannot be used on IDs generated without a data ID.

Definition at line 329 of file _id_generator.py.

333 ) -> Callable[[int], tuple[DataCoordinate, int]]:
334 """Return a callable that unpacks the IDs generated by this class,
335 from a config field.
336
337 Parameters
338 ----------
339 config : `BaseIdGeneratorConfig`
340 Configuration for an ID generator.
341 fixed : `DataCoordinate`
342 Data ID identifying the dimensions that are considered fixed by the
343 `IdGenerator` that produced the IDs: usually just ``instrument`` or
344 ``skymap``, depending on the configuration. For most configurations
345 this will need to be a fully-expanded data ID.
346
347 Returns
348 -------
349 unpacker
350 Callable that takes a single `int` argument (an ID generated by an
351 identically-configured `IdGenerator`) and returns a tuple of:
352
353 - release_id: the integer that identifies a data release or
354 similar (`int`);
355 - data_id : the data ID used to initialize the original ID
356 generator (`DataCoordinate`);
357 - counter : the counter part of the original ID (`int`).
358
359 Notes
360 -----
361 This method cannot be used on IDs generated without a data ID.
362 """
363 packer = config._make_dimension_packer(fixed)
364 return cls.unpacker_from_dimension_packer(packer, config.n_releases)
365

◆ unpacker_from_dimension_packer()

Callable[[int], tuple[int, DataCoordinate, int]] lsst.meas.base._id_generator.IdGenerator.unpacker_from_dimension_packer ( cls,
DimensionPacker dimension_packer,
int n_releases = DEFAULT_N_RELEASES )
Return a callable that unpacks the IDs generated by this class,
from a `lsst.daf.butler.DimensionPacker` instance.

Parameters
----------
dimension_packer : `lsst.daf.butler.DimensionPacker`
    Dimension packer used to construct the original
    `DimensionPackerIdGenerator`.
n_releases : `int`, optional
    Number of (contiguous, starting from zero) ``release_id`` values to
    reserve space for. One (not zero) is used to reserve no space.

Returns
-------
unpacker
    Callable that takes a single `int` argument (an ID generated by an
    identically-constructed `DimensionPackerIdGenerator`) and returns a
    tuple of:

    - release_id: the integer that identifies a data release or
      similar (`int`);
    - data_id : the data ID used to initialize the original ID
      generator (`DataCoordinate`);
    - counter : the counter part of the original ID (`int`).

Notes
-----
This method cannot be used on IDs generated with no data ID.

Definition at line 367 of file _id_generator.py.

371 ) -> Callable[[int], tuple[int, DataCoordinate, int]]:
372 """Return a callable that unpacks the IDs generated by this class,
373 from a `lsst.daf.butler.DimensionPacker` instance.
374
375 Parameters
376 ----------
377 dimension_packer : `lsst.daf.butler.DimensionPacker`
378 Dimension packer used to construct the original
379 `DimensionPackerIdGenerator`.
380 n_releases : `int`, optional
381 Number of (contiguous, starting from zero) ``release_id`` values to
382 reserve space for. One (not zero) is used to reserve no space.
383
384 Returns
385 -------
386 unpacker
387 Callable that takes a single `int` argument (an ID generated by an
388 identically-constructed `DimensionPackerIdGenerator`) and returns a
389 tuple of:
390
391 - release_id: the integer that identifies a data release or
392 similar (`int`);
393 - data_id : the data ID used to initialize the original ID
394 generator (`DataCoordinate`);
395 - counter : the counter part of the original ID (`int`).
396
397 Notes
398 -----
399 This method cannot be used on IDs generated with no data ID.
400 """
401 bits = _IdGeneratorBits(dimension_packer, n_releases)
402
403 def unpack(record_id: int) -> tuple[int, DataCoordinate, int]:
404 rest, counter = divmod(record_id, bits.n_counters)
405 rest, packed_data_id = divmod(rest, bits.n_data_ids)
406 rest, release_id = divmod(rest, bits.n_data_ids)
407 if rest:
408 raise ValueError(
409 f"Unexpected overall factor {rest} in record_id {record_id}, "
410 f"after extracting packed_data_id={packed_data_id}, counter={counter}, and "
411 f"release_id={release_id}."
412 )
413 data_id = bits.packer.unpack(packed_data_id)
414 return release_id, data_id, counter
415
416 return unpack
417
418

The documentation for this class was generated from the following file: