Imaging systems have traditionally been designed to mimic the human eye and produce visually interpretable measurements. Modern imaging systems, however, process raw measurements computationally before or instead of human viewing. As a result, the information content of raw measurements is more important than their visual interpretability. Despite its importance, current approaches for evaluating imaging system performance do not quantify information: they instead either use alternative metrics that assess specific aspects of measurement quality or assess measurements indirectly through performance on secondary tasks.
We developed theoretical foundations and a practical method to directly quantify mutual information between noisy measurements and unknown objects. By fitting a probabilistic model to a dataset of measurements and leveraging known noise physics, our method can estimate how much object information imaging systems encode. We demonstrated that our method is:
What is information theory and how does it apply to imaging systems?
We model imaging systems (encoders) as functions that transform objects into images, which are then captured with noise by a detector. Using just these noisy measurements and knowledge of the noise characteristics, we can evaluate system quality and guide improvements.
Information quantify how well we can distinguish different objects using noisy measurements. When noise increases, multiple objects could have produced the same measurement, making them harder to tell apart and reducing the information content.
Information accounts not only for noise, but for all factors that affect measurement quality—including resolution, sampling, color sensitivity, and optical aberrations. It provides a unified metric that captures how these factors combine to determine system performance.
To estimate information, we measure the total variability in measurements and subtract out the portion caused by noise.
Mathematically, this can be written as the difference of two entropies:
These quantities depend on two key probability distributions:
For most imaging systems, we know or can measure the noise distribution $\color{#189EE8}{p(\mathbf{y} \mid \mathbf{x})}$. For example, the random arrival times of photons are known to follow a Poisson distribution. This lets us directly calculate $\color{#189EE8}{H(\mathbf{Y} \mid \mathbf{X})}$.
The measurement distribution $\color{#38AD07}{p(\mathbf{y})}$ is trickier - it depends on both the objects and the imaging system itself. We learn this distribution from data by fitting a model $\color{#38AD07}{p_\theta(\mathbf{y})}$ to a dataset of measurements.
Our approach provides an upper bound on the true information content. Since any estimate will be higher than the true value, different models can be compared by choosing the one that gives the lowest estimate.
We tested our information estimation framework across diverse imaging applications. It matched traditional decoder-based evaluation, without their complex reconstruction algorithms or ground truth data requirements. Higher-information measurements consistently produced better results across all tasks: reconstructing color photos, imaging black holes with radio telescopes, capturing scenes with lensless cameras, and predicting the phenotypic characteristics of cells from microscopy images. This validates our framework as a simpler, faster, and more versatile approach to evaluating imaging systems.
Information content can do more than evaluate existing systems—it can guide the design of better ones. Our Information-Driven Encoder Analysis Learning (IDEAL) method automatically optimizes imaging system parameters to maximize information capture. Unlike traditional approaches that require training complex image reconstruction algorithms, IDEAL directly optimizes the imaging system using only the measurement information content.
For example, below a color photography filter is designed using gradient ascent on information estimates.
IDEAL matched the performance of end-to-end design, the prevailing approach that jointly trains hardware and image reconstruction algorithms. However, IDEAL avoids end-to-end's memory and compute requirements the potential for vanishing gradients that arise from backpropagating through complex reconstruction algorithms.
@article{pinkard2024informationdrivendesignimagingsystems,
title={Information-driven design of imaging systems},
author={Henry Pinkard and Leyla Kabuli and Eric Markley and Tiffany Chien and Jiantao Jiao and Laura Waller},
year={2024},
eprint={2405.20559},
archivePrefix={arXiv},
primaryClass={physics.optics},
url={https://arxiv.org/abs/2405.20559},
}