<!– –>

**Figure 1**: (left) LED Array Microscope constructed using a standard

commercial microscope and an LED array. (middle) Close up on the LED array dome

mounted on the microscope. (right) LED array displaying patterns at 100Hz.

Computational imaging systems marry the design of hardware and image

reconstruction. For example, in optical microscopy, tomographic,

super-resolution, and phase imaging systems can be constructed from

simple hardware modifications to a commercial microscope (Fig. 1) and

computational reconstruction. Traditionally, we require a large number of

measurements to recover the above quantities; however, for live cell imaging

applications, we are limited in the number of measurements we can acquire due

to motion. Naturally, we want to know what are the best measurements to acquire.

In this post, we highlight our latest work that learns the experimental design to maximize the performance of a

non-linear computational imaging system.

## Motivation

Standard microscopes usually image the absorption contrast of the sample;

however, most biological cells are weakly absorbing. Stains or dyes can be used

to observe contrast, but this may be prohibitively invasive for live cell

biology. With a computational imaging system it is possible to image other

intrinsic optical properties of the sample such as phase (i.e. refractive index)

to provide us with a strong mechanism for contrast (Fig. 2) and quantitative

information about the sample.

**Figure 2**: Mouse Fibroblast Cells: (left) absorption contrast image has poor

contrast, (right) quantitative phase image has strong structural contrast.

Using a standard microscope, the illumination source can be replaced with a

programmable LED array to construct an LED array microscope (Fig 1.). Images

taken under different LED’s illumination will encode information about the

sample’s phase (specifically, different parts of the sample’s

spatial spectrum) into the intensity measurements. A large set of measurements (10s to

100s) can then be combined by solving an inverse problem to quantitatively

reconstruct the phase of the sample, possibly with higher resolution than the

native resolution of the microscope, or in 3D.

Due to the multiple measurement requirement, obtaining high quality

reconstructions is traded off with poor temporal resolution. So when sample

motion is present, poor temporal resolution will degrade image quality due to

motion blur. To improve upon this tradeoff, designing how to capture a smaller

set of measurements would enable high quality reconstruction at an improved

temporal resolution. But what are the best patterns to display on the LED array?

What is the most efficient way to encode information?

**Figure 3**: LED Array Microscope System: (top) Measurement formation

process using a LED array microscope. (bottom) Regularized inverse problem

solved for image reconstruction.

Designing the LED array patterns for a set of measurements is complicated. Many

computational imaging systems, including this one, are non-linear in how the

hardware encodes information, how the computation reconstructs information, or

both. Conventional design methods, (e.g. optimal design,

spectral analysis) almost all consider design for linear system models and linear

reconstructions, thus will not necessarily improve performance for systems when the

processes are non-linear.

In this post, we consider a new computational imaging framework, Physics-based Learned Design [1],

that learns the experimental design to maximize the overall

performance of the microscope given a system model, a computational reconstruction,

and a dataset. Here, we consider using the LED array microscope as our hardware system and regularized phase retrieval as our computational reconstruction (Fig. 3). Both of these systems are non-linear and thus are difficult to design using conventional methods.

To demonstrate our method, we learn an experimental design for Quantitative

Phase Imaging using a small simulated dataset and in experiment show

similar reconstruction quality to a more sophisticated method using only a fraction of the measurements.

## Optimal Experiment Design and Phase Retrieval

Optimal experiment design methods consider minimizing the variance (mean

square error) of an unbiased estimator or equivalently maximizing its

information (the reciprocal of the variance). Usually, methods optimize

real-valued summary statistic (e.g. trace, determinant) of the variance,

especially when estimating several parameters. However, such design methods are only optimal for linear unbiased estimators (e.g.

least squares) and will not necessarily be optimal for non-linear biased

estimators. In our setting, we want to estimate the phase image by solving a regularized

phase retrieval problem (both a non-linear and biased estimator).

We want to retrieve the sample’s optical properties, $mathbfx$, from several

non-linear measurements, $mathbfy_k$, by minimizing a data consistency

term, $mathcalD$, and a prior term, $mathcalP$, (e.g. sparsity, total-variation).

This type of cost function can be iteratively minimized via gradient-based optimization (e.g. accelerated proximal gradient descent or

alternating direction method of multipliers). Here we choose accelerated

proximal gradient descent.

**Algorithm 1**: Accelerated Proximal Gradient Descent, $mathbfx$ is the

current estimate, $mu$ is the acceleration term, $mathbfw,mathbfz$ are

intermediate variables, and $N$ is the number of iterations. Each iteration

consists of a gradient update (green), proximal update (pink), and an

acceleration update (orange).

## Physics-based Network

Our goal is to learn the experimental design (i.e. how to best encode information) for the

non-linear computational imaging system outlined in figure 3. However, as we discussed conventional

design methods will not work, so let us rethink the problem. Consider *unrolling*

the iterations of the optimizer in Alg. 1 into a network where each layer of the network

is an iteration of the optimizer. Within each layer we have operations which

perform a gradient update, a proximal update, and an acceleration update. Now,

we have a network that incorporates the non-linear image formation process

and the prior information from the reconstruction.

**Figure 4**: Unrolled Physics-based Network: (left) takes several camera

measurements parameterized by the hardware design, $C$, as input and (right)

outputs the reconstructed phase image, $mathbfx^(N)$, which is compared to the

ground truth, $mathbfx’$. Each layer of the network corresponds to an

iteration of the gradient-based image reconstruction. Within each layer, there is a

gradient update (green), a proximal update (pink), and an acceleration update (orange).

We are not the first to consider unrolled physics-based networks. A work in

another imaging field considers unrolling the iterations of their image

reconstruction to form a network and then learns quantities in the

reconstruction that replace the proximal update [2]. In the past year, the

mechanics of unrolling the iterations of an image reconstruction pipeline

have rapidly grown in popularity.

## Physics-based Learned Design

For our context, we want to learn how to turn on the LEDs during each

measurement. Specifically, we learn the brightness of each LED, such that the

image reconstruction error is minimized. Now, the experimental design can be posed

as a supervised learning problem,

where, our loss function is the $L_2$ distance between the reconstructed phase image,

$mathbfx^(N)$, and the ground truth, $mathbfx’$, over a small simulated dataset.

The LED brightnesses, $C in mathbbR^M times T$ for $M$ measurements and $T$ LEDs, are

constrained to be positive so that our learned designs can

be feasibly implemented in hardware. Finally, because the network is only parameterized by

a few variables, we can efficiently

learn them using a small dataset ($100$ examples)! We minimize

the loss function using projected gradient descent.

## Experiments

Our experimental context is cell imaging, so we use 100 cell images (100px by

100px) as our dataset (90 training examples / 10 testing examples). We learn the

experimental design parameters for acquiring only two measurements. In figure 5, we compare

the phase image reconstructions using our learned design [1] and the traditional design

against the phase image reconstructed using a validation method. Using only a fraction of the measurements,

our learned designs can reconstruct a phase image with a similar quality to that of the

validation method, while the traditional design’s phase image is degraded in quality.

**Figure 5**: Result Comparison: Quantitative Phase Images of Mouse

Fibroblast Cells reconstructed using (left) traditional design, (middle) our

learned design, and (right) a validation method using many more measurements. Below is

a cross section comparison of the three reconstructions through the nucleus of a

cell.

## Conclusion

Optimal experimental design methods are useful when the system is linear; however, their

design will not necessarily improve the performance when the system is non-linear. We

have shown we can optimize the experimental design for a non-linear computational imaging system

using supervised learning and an unrolling physics-based network. Looking forward, we will consider

analyzing larger more complex systems with different measurement constraints, as

well as, structured ways to learn parts of the image reconstruction.

## References

[1] Kellman, M. R., Bostan, E., Repina, N., & Waller, L. (2018). Physics-based

Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging.

arXiv preprint arXiv:1808.03571.

[2] Hammernik, K., Klatzer, T., Kobler, E., Recht, M. P., Sodickson, D. K.,

Pock, T., & Knoll, F. (2018). Learning a variational network for reconstruction

of accelerated MRI data. Magnetic resonance in medicine, 79(6), 3055-3071.