A Toy Model of the Structure of Information Encoding

How well can an encoder distribute signals in a space so that they remain distinguishable after distortion? To answer this, we define a minimal model. Objects live on a 1D periodic domain. Each is encoded into a signal by a bandlimited, nonnegative convolution kernel (a point spread function), as shown in (a).

The encoding maps each object to a point in a constrained region of a finite-dimensional space. Integrating the encoded signal over sampling intervals yields one coordinate per interval. In physical systems like cameras, this quantity is energy (e.g. photon counts), so we use that term here. These coordinates are nonnegative and sum to at most unit energy (the positive orthant of the L1 unit ball). Bandlimitedness further restricts the reachable region: convolution with a finite-bandwidth kernel cannot concentrate all energy at a single point, so the valid signals (green in (b)) occupy only a subset of this region.

The encoder is not free to map objects to arbitrary points in signal space. The kernel applies the same transformation to every object, so the encoder has limited expressivity, and its range depends on the object itself. In a physical system like a camera, this is concrete: a single lens cannot simultaneously make a dim scene appear bright and a bright scene appear dim.

Since the convolution kernel can only disperse energy, not reconcentrate it, a concentrated object (a single delta function) gives the encoder maximum freedom: it can produce a broad range of output signals by choosing different kernels. A dispersed object (many delta functions) constrains the encoder to a smaller region of output space, even at the same total energy.

For each of three object types below (two deltas, one delta, eight deltas), the colored region on the right shows the set of encoded signals reachable by any kernel. As the object becomes more dispersed, the reachable region shrinks.

The boundary of each reachable region was found by optimization (gradient descent toward target points in signal space). If the optimizer were getting stuck in local minima, the true reachable region could be larger than what we show. To rule this out, we repeated the optimization from many random initializations and optimized toward randomly sampled feasible points; both approaches consistently recovered the same boundaries.

The constraints above have a direct consequence: the encoder cannot produce the distribution of signals that achieves the channel capacity. We call the resulting information loss encoder inefficiency.

Under additive Gaussian noise, the optimal distribution is uniform over the output space. The optimal distribution (green, top left) fills the space uniformly (it appears nonuniform because this is a 2D projection of a higher-dimensional space). The best physically achievable distribution (blue, bottom left) is concentrated in a subregion.

The information gap grows with the number of degrees of freedom, the space-bandwidth product (right). Physical constraints become increasingly restrictive in higher dimensions.

The optimal encoder here was found by maximizing mutual information under a Gaussian approximation (using the IDEAL framework from the paper). This approximation may not be tight, so the specific value of encoder inefficiency could change with a better optimizer. But encoder inefficiency itself must exist: no physical encoder can freely rearrange where energy goes. This is a first-pass estimate. Our later work (IDEAL-IO) explored a PixelCNN-based approach that may produce more accurate estimates.

Encoded information scales predictably with three system parameters. We vary SNR, bandwidth, and sampling density independently, each across several object distributions (a), and measure the effect on mutual information (b):

• Signal-to-noise ratio (left): Information grows logarithmically with SNR. Sparser sources (deltas) encode more efficiently than dense ones (white noise) at every SNR level.
• Degrees of freedom (center): Information grows approximately linearly with bandwidth at source-dependent rates.
• Sampling density (right): Oversampling beyond the Nyquist rate still increases information, but with diminishing returns. Higher SNR amplifies the benefit of oversampling.

These scaling relationships are empirical observations from this physically constrained 1D model. They are consistent with classical results for unconstrained channels, but we have not proven them analytically for the constrained case.

The preceding sections characterized encoding without specifying a task. Here we focus on a classic problem in optics: two-point resolution. Given a noisy signal, determine whether the source is one point or two closely spaced points.

In this simple case, we need exactly 1 bit of information to distinguish the two hypotheses, and we can analytically compute the relationship between information and task performance. The object is convolved with a kernel, captured with noise, and fed to an optimal binary classifier (a). The classifier's accuracy is a monotonic function of mutual information, rising from chance (0.5) at zero information to perfect (1.0) at one bit (b).

Resolution and SNR jointly determine encoded information (c). Iso-information contours trace curves through this space: the same information can come from high resolution with high noise, or low resolution with low noise. The insets show how signal distributions change across regimes. Resolution separates the two signal peaks; SNR controls how well that separation survives noise.

The results above were derived for a simple physical system: 1D convolution with bandlimited, nonnegative kernels. But the phenomena they reveal are not specific to imaging. Any system that encodes high-dimensional objects into finite-dimensional signals under constraints faces the same geometric problem: how the object and the encoder's expressivity together determine how well signals fill the space, and how much information survives.

Noise, interference, and discretization arise across a wide range of encoding systems, from communication channels to neural networks. Studying how these constraints shape signal geometry in other domains could reveal similar structure, and similar tradeoffs, to those found here.