Online Convolutional Dictionary Learning for Multimodal Imaging

Figure 1: Sparse Representation in a Convolutional Dictionary.

Multimodal imaging systems acquire several measurements of an object using multiple distinct sensing modalities. Often, the data acquired from the sensors is jointly processed to improve the imaging quality in one or more of the acquired modalities. Such imaging methods have the potential to enable new capabilities in traditional sensing systems, providing complementary sources of information about the object. Some applications of multimodal imaging include remote sensing [Dalla Mura et al., 2015], combining modalities such as synthetic aperture radar, LIDAR, optical and thermal range, multispectral and hyperspectral devices; biomedical imaging, for instance [Fatakdawala and et al., 2013], where fluorescence lifetime imaging, ultrasound backscatter microscopy, and photoacoustic imaging are combined for imaging cancer in live patients, or [Guérit et al., 2016], where CT based geometrical priors are used for blind deconvolution of PET images; and high-resolution depth sensing [Diebel and Thrun, 2005] (also considered in this work) where information from an intensity image is used to increase a depthmap resolution.

Figure 2: Depth-Intensity Multimodal Inverse Problem.

We consider a joint imaging inverse problem with multiple noisy linear measurements as represented on Figure 2. The target images correspond to the same physical object viewed from different modalities. For example, each image may represent a different color channel, spectral band, or a type of sensor. The key insight used in our work is that information about a single modality exists, in some form, in other modalities. This information can be exploited to improve the quality of multimodal imaging, as long as it can be extracted from the measurements.

Figure 3: Illustration of the proposed multimodal imaging method.

In this work, we propose a novel approach based on jointly sparse representation of multimodal images. Specifically, we are interested in learning data-adaptive convolutional dictionaries for both reconstructing and representing the signals. The convolutional dictionary learning is performed in an unsupervised way, i.e., given only the linear measurements of the signals. The main benefits of a convolutional approach are that the resulting dictionary is translation invariant and leads to a sparse representation over the entire image. This, however, comes with the increase in the computational cost, which we address by developing a new online convolutional dictionary learning method suitable for working with large-scale datasets. This online formulation behaves, in some way, as a dynamical system, illustrated in Figure 3, where internal state variables are used to keep track of previously reconstructed signals, in a memory-efficient way. If a non-stationary data stream is given to the online learning algorithm, the latter is able to automatically adapt the convolutional dictionary to the changing data distribution. Our key contributions are summarized as follows:

  • We provide a new formulation for multimodal computational imaging, incorporating a convolutional joint sparsity prior and a TV regularizer. In this formulation, the high resolution images are determined by solving an optimization problem, where the regularizer exploits the redundancies across different modalities.
  • We develop an online convolutional dictionary learning algorithm. By accommodating an additional TV regularizer in the cost, the algorithm is able to learn the convolutional dictionary in an unsupervised fashion, directly from the noisy measurements. The dictionary is convolutional which provides several advantages over a patch-based approach, such as translation invariance and the fact that the size of the atoms does not restrict the size of the associated regularized inverse problem. The online nature of the algorithm makes it appealing for applications that entail very large datasets or dynamically streamed, possibly non-stationary, datasets.
  • We validate our approach for joint intensity-depth imaging on two numerical experiments. The first one compares the proposed approach with two competitive intensity-depth fusion algorithms, in an extensive set of simulations over various examples coming from the Middlebury dataset [Scharstein et al., 2014], which provides accurate groundtruth depthmaps and high-resolution intensity images. The second experiment aims at demonstrating the dynamic capability of the online algorithm by presenting the results of reconstruction over a depth-intensity video sequence, during which the joint convolutional dictionary visibly improves itself, and the reconstructed depthmap quality, with time.

Related References:

  • Degraux, K., Kamilov, U. S., Boufounos, P. T., and Liu, D. (2017).
    "Online Convolutional Dictionary Learning for Multimodal Imaging."
    In 2017 IEEE International Conference on Image Processing (ICIP), September 2017. IEEE. arXiv:1706.04256.

Other References:

  • Dalla Mura, M., Prasad, S., Pacifici, F., Gamba, P., Chanussot, J., and Benediktsson, J. A. (2015). Challenges and Opportunities of Multimodality and Data Fusion in Remote Sensing. Proceedings of the IEEE, 103(9):1585–1601.
  • Fatakdawala, H. and et al. (2013). Multimodal in vivo imaging of oral cancer using fluorescence lifetime, photoacoustic and ultrasound techniques. Biomed. Opt. Express, 4(9):1724–1741.
  • Guérit, S., González, A., Bol, A., Lee, J. A., and Jacques, L. (2016). Blind Deconvolution of PET Images using Anatomical Priors. Proceedings of the third "international Traveling Workshop on Interactions between Sparse models and Technology" iTWIST'16.
  • Diebel, J. and Thrun, S. (2005). An application of Markov random field to range sensing. In Prov. Advances in Neural Information Processing Systems, volume 18, pages 291–298, Vancouver, BC.
  • Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešic ́, N., Wang, X., and Westling, P. (2014). High-resolution stereo datasets with subpixel-accurate ground truth. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8753(1):31–42.


  • Ulugbek S. Kamilov (Washington University in St. Louis, MO, USA, previously MERL, Cambridge, MA, USA)
  • Petros T. Boufounos (MERL, Cambridge, MA, USA)
  • Dehong Liu (MERL, Cambridge, MA, USA)


ImageProcessing ComputationalImaging DictionaryLearning MultimodalImaging ConvolutionalRepresentations DepthSensing SensingImaging InverseProblem ResearchCategoryAll

ISPGroup Participant(s)

Last updated October 23, 2017, at 04:49 PM