Applied Mathematics & Systems Laboratory - Information Processing Group
Image, Texture, Video & Structure Completion
Nikos Paragios
http://cermics.enpc.fr/~paragios MAS Ecole Centrale de Paris Paris, France Joint Work with : Cedric Allene (ESIEE/CNRS/CERTIS)
Applied Mathematics & Systems Laboratory - Information Processing Group
Outline Problem
Statement
Literature
Review (inpainting in the form of interpolation problem)
Completion On
in the form of a labeling problem using MRFs
the selection of candidate patches and on their positions
Optimization (Brief)
of the Cost function
Sketch Extensions to Video and Structure
Discussion
Applied Mathematics & Systems Laboratory - Information Processing Group
Problem Statement Inpainting
consists of modifying a partially destroyed image towards its ancient form
In
a way that makes the process nondetectable for an observer that hasn’t seen the original image
Applied Mathematics & Systems Laboratory - Information Processing Group
Inpainting by the Propagation of Information According
to conservators from museums, an automated inpainting process should satisfy the following conditions: i The global picture determines how to fill in the gap i The
structure of the area surrounding the missing part is continued into it through the prolongation of the ones arriving at the missing part boundaries
i The
different regions within the missing part are filled in with color that matches the ones of the line at the boundaries that was used to fill in the information
i Small
details that are not part of the structure (texture) are added once the filling in procedure has been completed.
Applied Mathematics & Systems Laboratory - Information Processing Group
Image Inpainting through PDEs Let
us consider an image with a missing part:
Inpainting
that
consists of creating a sequence of images: and
such
One
can consider a general form of this algorithm and write a partial differential equation of the following nature:
Let
us consider the information to be propagated and denote such information with as well as the propagation direction
Then
a quite simple method to perform such propagation is through the following PDE:
Applied Mathematics & Systems Laboratory - Information Processing Group
Image Inpainting
That
leads to following third order PDE
Applied Mathematics & Systems Laboratory - Information Processing Group
Variational Methods for Filling-In (Elastica-based Model ) Consider
upper Through
That
an image , then one can determine the image using its (or lower) level set according to: the following reconstruction process
consists of separating image in iso-intensity lines (or level sets)
The
Euler’s elastica model consists of defining a cost functional that given two T-junction points and and their tangents , seeks a smooth curve between the two points according to the following cost function
Where
the minimum is taken along all curves joining the two points
Applied Mathematics & Systems Laboratory - Information Processing Group
Image Inpainting through the Euler Elastica
Applied Mathematics & Systems Laboratory - Information Processing Group
Joint Interpolation of Vector Fields and Gray Levels Both,
the propagation information model & level-lines Euler’s elastica approach they are mostly based on interpolation of intensities (even though some geometry is used in the case of level lines)
One
can consider a more efficient approach that does not only constrain the appearance but also the geometry of the filling in part
To
this end, the principle of image continuation of good continuation that refers to the joint intensity/orientation space,
Let
be a vector field with the directions of the gradient for the original image that do satisfy the following condition:
Subject
to the constraint:
Applied Mathematics & Systems Laboratory - Information Processing Group
Joint Interpolation of Vector Fields and Gray Levels
Applied Mathematics & Systems Laboratory - Information Processing Group
Problems with PDE approaches Good
continuation Principle is reasonable in the case of uniform images
Good
continuation principle provides good results if the missing content has small volume
Convergence,
as well recovering the optimal solution of the minimization of the cost function might be two problematic issues since we are living in convex spaces
Introducing
metrics and content that goes beyond images is problematic, the definition of good continuation in several metrics doesn’t exist with texture, video, & structure being the examples
Applied Mathematics & Systems Laboratory - Information Processing Group
Completion Using Concepts from Game Theory
Applied Mathematics & Systems Laboratory - Information Processing Group
Defining the Problem Inpainting
consists of modifying a partially destroyed image towards its ancient form
In
a way that makes the process nondetectable for an observer that hasn’t seen the original image
Without
loss of generality since the information is not available at the region to be inpainted we can assume that the missing content is present elsewhere in the image
Let
us consider n possible candidates to fill in the content for a particular segment (randomly selected from the image content) [tetris/lego bars].
Applied Mathematics & Systems Laboratory - Information Processing Group
Defining the Problem Inpainting
the consists of finding for every position the most probable configuration
“Such
configuration on one hand should be in agreement with the existing content; that in the simple image completion problem can be enforced through the minimization of
While
more general metrics can be used, taking into account richest image content that can also encode geometry, importance of boundaries etc.
Applied Mathematics & Systems Laboratory - Information Processing Group
Defining the Problem Preserving
discontinuities and contrast are important for the human eye and therefore particular attention is to be paid when introducing content in these areas
Therefore,
we can modify the image term to encode such perception from biological vision systems through the penalization of errors in areas with important gradients.
Solving i we
such an optimization problem is ill-posed,
have pixel-wise constraints that do not encode a global coherence of the solution i Pixels will be labeled in an independent fashion based on the best match with the existing content i Missing content in areas with non-overlap with the existing content will be falsely recovered
Applied Mathematics & Systems Laboratory - Information Processing Group
Defining the Problem Such
Image Attraction term should be combined with a smoothness labeling term;
Since
we are considering image patches to fill in the gaps, it is natural to assume that within the inpainted region, neighborhood pixels are filled in from the same patch
Using
the markovian properties, one can introduce local potentials that force neighborhood pixels to be filled in from the same patch
Through
the addition of penalties in terms of discrepancies between the local labeling process
Applied Mathematics & Systems Laboratory - Information Processing Group
Defining the Problem However
the missing content is partially constructed and therefore data support is not available for areas with significant distance from the borders of the inpainted region?
Therefore
we can relax the constraint and consider the notion of progressive stitching where in areas with existing content we use the distance between this content and the candidate seeds
While
in areas with non-overlapping content, the use of already inpainted content is considered
Distance from the borders of the inpainted region
Applied Mathematics & Systems Laboratory - Information Processing Group
Defining the Problem Such Its
modification introduces in the process the notion of time.
interpretation is simple;
i First
areas with strongly overlapping content and strong discontinuities (boundaries) will be inpainted i Then areas with strongly overlapping content and less important boundaries will be filled in i And last, completion will be performed through stitching on already stitched components using the same principles and moving towards from areas close to the existing content to the ones further away;
i While
in all stages of the reconstruction, pixels in the local neighborhood will be forced to be inpainted from the same patch
Applied Mathematics & Systems Laboratory - Information Processing Group
An Example… If
we have candidate seeds, a minimization method we can perform inpainting through the lowest potential of the cost function, that refers to discrete optimization;
Applied Mathematics & Systems Laboratory - Information Processing Group
On the selection of Seeds Finding
appropriate candidates for the completion is one of the two most critical components of our approach;
We
are based on the principle that missing content is not necessarily in the vicinity of the destroyed region
We
can consider the seed selection problem as a random variable; that consists of i The
center of the seed (in the image) i Its from (mostly rectangular) as well as its dimensions and orientation Such
a problem consists of finding given an origin point in the impainted regions a number of candidate patches in the image domain that potentially have similar content with one that is missing and the one that is partially present
Applied Mathematics & Systems Laboratory - Information Processing Group
On the Selection of Candidate Seeds Such
seeds selection can be in a probabilistic formulation;
i Given
an origin point, create a number of perturbations [in terms of the position of the seed, its form and its orientation] according to some known distribution i Once new candidate seeds have been determined, evaluate the probability of containing information to complete the missing content; i Consider a number of the most probable seeds and repeat the process until a number of seeds have been selected of varying form, position and orientation with content potentially interesting to complete the missing one
Applied Mathematics & Systems Laboratory - Information Processing Group
Gaussian PDF estimation Compute
to t: If
the present state of system using the observations from 1, or determine the probability
we assume that the density function is known at t-1: one can consider the Bayes rule
That
then
can be re-written using the Chapman-Kolmogorov equation
While
one can claim a number of approaches to estimate this density (incremental EM – otherwise infinite memory is needed) a lack exists on efficient numerical techniques
Applied Mathematics & Systems Laboratory - Information Processing Group
Particle Filters Particle
filters, which are sequential Monte-Carlo techniques, estimate the Bayesian posterior probability density with a set of samples
Let A
us consider M random measures:
particle filter is a non linear approximation of this density consists of
: With
being weights that reflect the importance of
the different samples Once
a set of samples has been drawn, can be computed out of the observation for each sample, and the estimation of the posteriori pdf can be sequentially updated.
Applied Mathematics & Systems Laboratory - Information Processing Group
And a Demonstration…
Applied Mathematics & Systems Laboratory - Information Processing Group
On the optimization of the Cost function Suboptimal
Methods
i Simulated
Anealing i Iterated Conditional Modes i Highest Confidence First Semi-Optimal
Methods
i Graph-based
Optimal
optimization
methods
i Meanfield
anealing i Belief Propagation Networks
Applied Mathematics & Systems Laboratory - Information Processing Group
Graph Cuts A
graph
consists of:
iA
set of nodes i A set of links/directed edges that connect these nodes i Two special terminal nodes often called source and sink that have a different nature than the rest of the nodes i Each
graph-edge is assigned to a positive weight i The cost of an edge in one direction can differ from the one of the opposite direction i An edge is called a t-link if it connects a non-terminal node with a terminal and n-link if it connects two nonterminal nodes
Applied Mathematics & Systems Laboratory - Information Processing Group
The Min-Cut Problem An
cut disjoint subsets
The
Cost of a cut “boundary edges”
refers to a partition of the graph nodes in two and such that the source is in and the sink in is the sum of cost weights of the such that and
The
MINIMUM cut problem is to the find the cut that has minimum cost among all possible cuts
One
of the fundamental results in combinatorial optimization is that the MIN-CUT problem is equivalent with the MAX-FLOW problem.
Applied Mathematics & Systems Laboratory - Information Processing Group
The Max-Flow Problem Consider
to be a water source, and the graph a network of directed pipes with capacities equal to the edge weights
Maximum
flow is the maximum amount of water that can be sent from the source to the sink
The
theorem of Ford-Fulkerson states that the maximum flow from the source to the destination saturates a set of edges in the graph dividing the nodes into two disjoint sets corresponding to the minimum cut,
Two
type of algorithms to solve the max-flow problem
i Push-relabel
techniques i Augmenting path methods
Applied Mathematics & Systems Laboratory - Information Processing Group
Expansion move algorithm Input labeling f
Red expansion move from f
i Find
red expansion move that most decreases E
i Move
there, then find the best blue expansion move, etc i Done when no α-expansion move decreases the energy, for any label α i Many
nice theoretical properties
Applied Mathematics & Systems Laboratory - Information Processing Group
In Practice
Applied Mathematics & Systems Laboratory - Information Processing Group
In Practice
Applied Mathematics & Systems Laboratory - Information Processing Group
The Case of Video & Structure Video The
Can be considered as a three dimensional volume
exact same concept can be used, where now during the construction of the graph in the temporal domain, object correspondences driven from the optical flow computation can be used to determine the costs of the cut in this direction
Applied Mathematics & Systems Laboratory - Information Processing Group
Discussion Image,
Texture , Video & Structure Completion was a formulated as a MAP problem
Image
terms, as well as smoothness considerations were used to fill in the missing content
Random
walks, particle filters and statistical hypotheses testing were used to determine candidate completion structures in the image
Combinatorial
optimization and the alpha expansion algorithm was used to recover the optimal configuration in terms of labels
The
method is rather general and can be used to deal with higher dimensions
The
method is favorable compared to PDEs since no assumption on the image content is made.
Applied Mathematics & Systems Laboratory - Information Processing Group
Future Work Replacing
the Video Content through coupling with optical flow estimation (we are almost there)
Stereo
reconstruction through similar fitting process…(start working on
that) Pre-reconstruction
of a number of basic elements (3D patches) and the complete reconstruction of an unseen scene through le”going” these patches to the image content
Augmentation
of archeological scenes to fill in the missing content from either different locations of the same monument, or components from different monuments
DO
NOT FORGET TO GO TO THE NEXT SLIDE!!!
Applied Mathematics & Systems Laboratory - Information Processing Group
Advertisement Section The Handbook of Mathematical Models in Computer Vision Springer (2005), ISBN 0387263713, 596 pages Editors i
Nikos Paragios
i
Yunmei Chen
i
Olivier Faugeras
http://cermics.enpc.fr/~paragios Chapter
3: PDE-Based Image and Surface
Inpainting, Bertalmio, Caselles, Haro, and Sapiro Chapter
5: Graph Cuts in Vision and
Graphics: Theories and Applications, Boykov and Veksler