better_image_representation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    },
    "colab": {
      "name": "better_image_representation.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/djbradyAtOpticalSciencesArizona/multiframeEstimation/blob/main/better_image_representation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fRqwMeIPi8nH"
      },
      "source": [
        "# Better Image Representation - Learn from NeRF<sup>1</sup>  and SRN<sup>2</sup> \n",
        "\n",
        "Thoughts from discussion with Qian   \n",
        "Minghao   \n",
        "\n",
        "Mar 31, 2021\n",
        "\n",
        "djb update 4 april 2021\n",
        "\n",
        "(SNR is the followup to DeepVoxels<sup>3</sup>)   \n",
        "1: [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis](https://arxiv.org/abs/2003.08934)   \n",
        "2: [Scene representation networks: Continuous 3D-structure-aware neural scene representations](https://papers.nips.cc/paper/2019/hash/b5dc4e5d9b495d0196f61d45b26ef33e-Abstract.html)   \n",
        "3: [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://vsitzmann.github.io/deepvoxels/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YjZ-ltp2i8nK"
      },
      "source": [
        "## NeRF and SRN: implicit 3D representation.\n",
        "\n",
        "These two neural networks has similar input, output, and pipeline. The pipeline can be written as:\n",
        "\\begin{align*}\n",
        "    \\mathbf{D} &= \\mathcal{F}(\\{\\mathbf{X}\\}) \\\\\n",
        "    \\mathbf{Y} &= \\mathcal{G}(\\mathbf{D}, \\mathbf{V})\n",
        "\\end{align*}\n",
        "Where \n",
        "- $\\{\\mathbf{X}\\}$: Input, 100/many 2D images   \n",
        "- $\\mathbf{D}$: Representation, one data chunk (array or weights) representing this single/class 3D object   \n",
        "- $\\{\\mathbf{Y}\\}$: Shown output, 2D images of the object from given view points $\\{\\mathbf{V}\\}$   \n",
        "- $\\mathcal{F}$: can be regarded as fusion and compression\n",
        "- $\\mathcal{G}$: can be regarded as extraction and projection.\n",
        "\n",
        "Like shown in this DeepVoxels' demostration image. $\\{\\mathbf{X}\\}$ are the images of the vase taken from different viewpoints, shown in top-left. $\\mathbf{D}$ is drawn as a colorful cube. With given viewpoints $\\{\\mathbf{V}\\}$, output $\\{\\mathbf{Y}\\}$ are rendered from $\\mathbf{D}$, as shown in bottom-right.\n",
        "![DeepVoxels](https://github.com/djbradyAtOpticalSciencesArizona/multiframeEstimation/blob/main/imgs/mh_image_representation/DeepVoxels_sketch.png?raw=1)\n",
        "\n",
        "SRN and NeRF has $\\mathbf{D}$ in different format, thus different $\\mathcal{F}$ and $\\mathcal{G}$.    \n",
        "NeRF's $\\mathbf{D}$ is some weights for the network structure $\\mathcal{G}$, and its $\\mathcal{F}$ is a training process.   \n",
        "SRN's $\\mathbf{D}$ is more complex, as it aims to work for a class of objects, where a latent vector $z$ chooses which object to render.\n",
        "\n",
        "The input $\\{\\mathbf{X}\\}$ has much redundancy. A human can grab a decent understanding of a 3D object if we show him/her just 3 or 4 images from different view points. The process $\\{\\mathbf{X}\\} \\rightarrow \\mathbf{D}$ is more or less an overposed problem. As a result, the representation $\\mathbf{D}$ takes even smaller disk space than input $\\{\\mathbf{X}\\}$.\n",
        "\n",
        "The shown output $\\{\\mathbf{Y}\\}$ can be regarded as projections of the 3D object, and it lies in the same space of input $\\{\\mathbf{X}\\}$. This makes it easier to train the network, as we may split a set of $\\{\\mathbf{X}\\}$ to a input and groundtruth data.\n",
        "\n",
        "By the way, I notice that $\\{\\mathbf{X}\\}$ and $\\{\\mathbf{Y}\\}$ are all 2D images. Why? My opinion is that this is the restriction of our sensor and display. We have 2D sensors, we hav 2D displays. I bet that the shown output won't be images, if we have commercial available 3D displays."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9aaGSmn7i8nO"
      },
      "source": [
        "## Straight forward 2D analogy\n",
        "\n",
        "However, it's not that easy to find a proper analogy for 2D images. \n",
        "\n",
        "As in SRN, it manages to find a function $\\Phi: (x,y,z) \\rightarrow \\mathbf{v}$, where $(x,y,z)$ is a 3D coordinate, and $\\mathbf{v}$ is a representation vector containing color and radiance information. Then it uses $\\Phi$ to derive $\\mathcal{G}$.   \n",
        "NeRF works similarly, but trace values along rays, not points.  \n",
        "\n",
        "Of course we can try the most straight forward analogy, finding a function $\\Phi: (x,y) \\rightarrow \\mathbf{v}$. Similar method has been raised by CPPN<sup>1</sup>, which uses neural network to perform a function $\\Phi: (x,y) \\rightarrow v$. That paper use it as a pattern generator, instead of a representation or compression. I'm still scanning papers cited CPPN.  \n",
        "Anyway, we can train such a network, and it should be a reasonable inpainting algorithm, as we can train on existing pixels, and predict missing pixels.    \n",
        "However, though we may make arbitrary dense $(x,y)$ grid, I don't think it would easily handle super-resolution tasks. As mentioned in SRN paper, they tried to make denser $(x,y,z)$ grid, but the resulting voxel array didn't contain much high frequency information beyond input images.\n",
        "\n",
        "A main difference between the 3D algorithms and this straight forward 2D analogy, in my opinion, is the relation between representation $\\mathbf{D}$ and shown output $\\{\\mathbf{Y}\\}$.   \n",
        "In the 3D algorithms, with a specific viewpoint $\\mathbf{V}$, we can project $\\mathbf{D}$ to $\\mathbf{Y}$. The reverse projection is very hard. What's more, though different $\\mathbf{Y}$s are closely correlated, its hard to infer one from another.   \n",
        "In the 2D analogy, the output a 2D pixel map is itself a fuction $\\mathbf{I}: (x,y) \\rightarrow v$ with band-limited output,  which is almost identical to the proposed representation $\\Phi: (x,y) \\rightarrow v$. What's more, the output can be fully represented with just one image, not a set of images. Thus, there's not much superiority in using a function to represent an image. \n",
        "\n",
        "1: [Compositional Pattern Producing Networks: A Novel Abstraction of Development](https://link.springer.com/content/pdf/10.1007/s10710-007-9028-8.pdf)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4d4sLkELi8nO"
      },
      "source": [
        "## Expand a 2D image\n",
        "\n",
        "A possible way out is to \"expand\" a 2D image. \n",
        "\n",
        "For example, we can add color channels. We can estimate a implicit representation $\\Phi: (x,y, \\lambda) \\rightarrow \\mathbf{v}$ from multiframe input or CASSI-style input.    \n",
        "In this way, the images with different $\\lambda$ are closely correlated, but not easily transfered. The implicit representation $\\Phi$ can easily project to a image with $\\lambda$, and the reverse is hard. That should be a better analogy.   \n",
        "I don't know if there's already paper about it. I'm searching them.\n",
        "\n",
        "Of course we can think of more ways to expand an image. A existing example estimating BTF (Bidirectional Texture Function) with neural network<sup>1</sup>. I'm still thinking for more options.\n",
        "\n",
        "1: [Neural BTF Compression and Interpolation](http://rgl.epfl.ch/publications/Rainer2019Neural)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eLAQnG4HjEue"
      },
      "source": [
        "# Neural Image Representation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TSYJAgxEj4_p"
      },
      "source": [
        "A imaging system consists of three components: \n",
        " \n",
        "\n",
        "1.   the analog digital layer that converts the physical field distribution into discrete numbers. This may be, for example, a focal lens combined with a color filter array. At the end of this layer, we measure g= Hf, where f is the object radiance distribution. various nonlinearities could generalize this mapping to g=H(f)\n",
        "2.   a coding layer that translates the measured data into features. For example, jpeg or mpeg coding is such a layer. The D layer referenced above is this layer\n",
        "3. a decoding layer that returns the estimate of the unknown field . \n",
        "\n",
        "Conventionally, the coding layer is considered only for compression. But deep voxels uses the coding layer to help with image estimation. We cannot estimate the image directly from measured data, better results are obtained if we find features before training for inversion. \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bqu3FIUquNE1"
      },
      "source": [
        "# Thoughts and plans\n",
        "#### after discussion with Minghao about Dr. Brady's new notes \n",
        "#### Qian\n",
        "#### 4/6/21\n",
        "\n",
        "\n",
        "## Thoughts\n",
        "\n",
        "Feature representation is useful when all desired pixel maps have redundancy but cannot be mutually transformed with ease, e.g., viewpoints in view synthesis tasks $(x, y, z)$, channels in hyper-spectral tasks $(x, y, \\lambda)$, frames in video tasks $(x, y, t)$. Those are all high-dimensional (HD) data that span more than spatial dimensions.\n",
        "\n",
        "Similar to the input of view synthesis task, input data of a Multiframe system is also HD as it involves time dimension. In view of the advantages of \"deep voxel\" representation, we should be able to improve components 2 & 3 by finding a better representation of a multi-frame scene than pixel maps. \n",
        "\n",
        "A problem in multi-frame estimation tasks like demosaicing and super-resolution is that only one good-looking version of the final result is necessary, not all possible estimations. For example, a system that produces all demosaiced estimates from the same image isn't more interesting than a system that only produces one of the best estimations. These lessen the necessity of a demosaicing feature or super-resolution feature that involves a third \"demosaicing dimension\" or \"resolution dimension\". \n",
        "\n",
        "I think a practical way to make meaningful outputs is to still combine those tasks with time/channel/viewpoint. For example, as multiframe naturally involves time, we can design a demosaicing system that takes sensor data of timestamps 1-T and a target timestamp t, and outputs a colored pixel map at time t. Or a demosaicing system that takes sensor data of timestamps 1-T, and outputs all colored pixel maps at time 1-T. The intermediate feature should be a compact representation that can be used to estimate all potential outputs, which are colored but have redundancy in contents.\n",
        "\n",
        "Given the new definition of the coding layer, the estimation may get involved in components 2, 3, or both. If we leave estimation to component 2 solely, the feature should be able to produce all pixel maps that we want nearly losslessly. If we only estimate in component 3, the feature should prepare the input data in a better position to accomplish the estimation task. The latter one may require task-specific and physically-based features (e.g., deep voxel/multiplane image (MPI) for view synthesis and [deep voxel flow](https://openaccess.thecvf.com/content_iccv_2017/html/Liu_Video_Frame_Synthesis_ICCV_2017_paper.html) for video interpolation) In tasks like demosaicing, super-resolution, etc., physically-based feature representation are more ambiguous. I'll investigate its feasibility.\n",
        "\n",
        "<!---## Plans\n",
        "\n",
        "I plan to start with a basic demosaicing system following the thought mentioned above. Have to admit that although the general components are the same (coding layer & decoding), it has many variants when implementing. First try letting coding layer do compression & estimation and decoding to do reconstruction. Train a decoding layer first, freeze it, and train a coding layer on top of that.\n",
        "\n",
        "data size: *n_frames x n_channels x H x W*\n",
        "\n",
        "* input: multiframe bayer data X, *16 x 1 x 64 x 64*\n",
        "* output: colored estimation compact representation Y_comp, size undefined.\n",
        "\n",
        "1. Generate 16-frame video libraries to get ground-truth frames: multiframe colored estimation Y, *16 x 3 x 512 x 512*, using blender\n",
        "2. using the forward model to generate input bayer data\n",
        "3. prepare Y_comp. Design and train a decoder inside an autoencoder. Inputs and outputs are Y and the loss can be MSE.\n",
        "4. Design and train estimation network, mapping from X -> Y_comp.\n",
        "\n",
        "Till now 1 and 2 are finished.\n",
        "\n",
        "\n",
        "<font size=1> Note about Component 1: check operations that are allowed or feasible in an optical system.</font> --->"
      ]
    }
  ]
}