Image Inpainting

Optimization of the Cost function .... Solving such an optimization problem is ill-posed, ... Therefore we can relax the constraint and consider the notion of.
3MB taille 5 téléchargements 312 vues
Applied Mathematics & Systems Laboratory - Information Processing Group

Image, Texture, Video & Structure Completion

Nikos Paragios

http://cermics.enpc.fr/~paragios MAS Ecole Centrale de Paris Paris, France Joint Work with : Cedric Allene (ESIEE/CNRS/CERTIS)

Applied Mathematics & Systems Laboratory - Information Processing Group

Outline ‰ Problem

Statement

‰ Literature

Review (inpainting in the form of interpolation problem)

‰ Completion ‰ On

in the form of a labeling problem using MRFs

the selection of candidate patches and on their positions

‰ Optimization ‰ (Brief)

of the Cost function

Sketch Extensions to Video and Structure

‰ Discussion

Applied Mathematics & Systems Laboratory - Information Processing Group

Problem Statement ‰ Inpainting

consists of modifying a partially destroyed image towards its ancient form

‰ In

a way that makes the process nondetectable for an observer that hasn’t seen the original image

Applied Mathematics & Systems Laboratory - Information Processing Group

Inpainting by the Propagation of Information ‰ According

to conservators from museums, an automated inpainting process should satisfy the following conditions: i The global picture determines how to fill in the gap i The

structure of the area surrounding the missing part is continued into it through the prolongation of the ones arriving at the missing part boundaries

i The

different regions within the missing part are filled in with color that matches the ones of the line at the boundaries that was used to fill in the information

i Small

details that are not part of the structure (texture) are added once the filling in procedure has been completed.

Applied Mathematics & Systems Laboratory - Information Processing Group

Image Inpainting through PDEs ‰ Let

us consider an image with a missing part:

‰ Inpainting

that

consists of creating a sequence of images: and

such

‰ One

can consider a general form of this algorithm and write a partial differential equation of the following nature:

‰ Let

us consider the information to be propagated and denote such information with as well as the propagation direction

‰ Then

a quite simple method to perform such propagation is through the following PDE:

Applied Mathematics & Systems Laboratory - Information Processing Group

Image Inpainting

‰ That

leads to following third order PDE

Applied Mathematics & Systems Laboratory - Information Processing Group

Variational Methods for Filling-In (Elastica-based Model ) ‰ Consider

upper ‰ Through

‰ That

an image , then one can determine the image using its (or lower) level set according to: the following reconstruction process

consists of separating image in iso-intensity lines (or level sets)

‰ The

Euler’s elastica model consists of defining a cost functional that given two T-junction points and and their tangents , seeks a smooth curve between the two points according to the following cost function

‰ Where

the minimum is taken along all curves joining the two points

Applied Mathematics & Systems Laboratory - Information Processing Group

Image Inpainting through the Euler Elastica

Applied Mathematics & Systems Laboratory - Information Processing Group

Joint Interpolation of Vector Fields and Gray Levels ‰ Both,

the propagation information model & level-lines Euler’s elastica approach they are mostly based on interpolation of intensities (even though some geometry is used in the case of level lines)

‰ One

can consider a more efficient approach that does not only constrain the appearance but also the geometry of the filling in part

‰ To

this end, the principle of image continuation of good continuation that refers to the joint intensity/orientation space,

‰ Let

be a vector field with the directions of the gradient for the original image that do satisfy the following condition:

‰ Subject

to the constraint:

Applied Mathematics & Systems Laboratory - Information Processing Group

Joint Interpolation of Vector Fields and Gray Levels

Applied Mathematics & Systems Laboratory - Information Processing Group

Problems with PDE approaches ‰ Good

continuation Principle is reasonable in the case of uniform images

‰ Good

continuation principle provides good results if the missing content has small volume

‰ Convergence,

as well recovering the optimal solution of the minimization of the cost function might be two problematic issues since we are living in convex spaces

‰ Introducing

metrics and content that goes beyond images is problematic, the definition of good continuation in several metrics doesn’t exist with texture, video, & structure being the examples

Applied Mathematics & Systems Laboratory - Information Processing Group

Completion Using Concepts from Game Theory

Applied Mathematics & Systems Laboratory - Information Processing Group

Defining the Problem ‰ Inpainting

consists of modifying a partially destroyed image towards its ancient form

‰ In

a way that makes the process nondetectable for an observer that hasn’t seen the original image

‰ Without

loss of generality since the information is not available at the region to be inpainted we can assume that the missing content is present elsewhere in the image

‰ Let

us consider n possible candidates to fill in the content for a particular segment (randomly selected from the image content) [tetris/lego bars].

Applied Mathematics & Systems Laboratory - Information Processing Group

Defining the Problem ‰ Inpainting

the consists of finding for every position the most probable configuration

‰ “Such

configuration on one hand should be in agreement with the existing content; that in the simple image completion problem can be enforced through the minimization of

‰ While

more general metrics can be used, taking into account richest image content that can also encode geometry, importance of boundaries etc.

Applied Mathematics & Systems Laboratory - Information Processing Group

Defining the Problem ‰ Preserving

discontinuities and contrast are important for the human eye and therefore particular attention is to be paid when introducing content in these areas

‰ Therefore,

we can modify the image term to encode such perception from biological vision systems through the penalization of errors in areas with important gradients.

‰ Solving i we

such an optimization problem is ill-posed,

have pixel-wise constraints that do not encode a global coherence of the solution i Pixels will be labeled in an independent fashion based on the best match with the existing content i Missing content in areas with non-overlap with the existing content will be falsely recovered

Applied Mathematics & Systems Laboratory - Information Processing Group

Defining the Problem ‰ Such

Image Attraction term should be combined with a smoothness labeling term;

‰ Since

we are considering image patches to fill in the gaps, it is natural to assume that within the inpainted region, neighborhood pixels are filled in from the same patch

‰ Using

the markovian properties, one can introduce local potentials that force neighborhood pixels to be filled in from the same patch

‰ Through

the addition of penalties in terms of discrepancies between the local labeling process

Applied Mathematics & Systems Laboratory - Information Processing Group

Defining the Problem ‰ However

the missing content is partially constructed and therefore data support is not available for areas with significant distance from the borders of the inpainted region?

‰ Therefore

we can relax the constraint and consider the notion of progressive stitching where in areas with existing content we use the distance between this content and the candidate seeds

‰ While

in areas with non-overlapping content, the use of already inpainted content is considered

Distance from the borders of the inpainted region

Applied Mathematics & Systems Laboratory - Information Processing Group

Defining the Problem ‰ Such ‰ Its

modification introduces in the process the notion of time.

interpretation is simple;

i First

areas with strongly overlapping content and strong discontinuities (boundaries) will be inpainted i Then areas with strongly overlapping content and less important boundaries will be filled in i And last, completion will be performed through stitching on already stitched components using the same principles and moving towards from areas close to the existing content to the ones further away;

i While

in all stages of the reconstruction, pixels in the local neighborhood will be forced to be inpainted from the same patch

Applied Mathematics & Systems Laboratory - Information Processing Group

An Example… ‰ If

we have candidate seeds, a minimization method we can perform inpainting through the lowest potential of the cost function, that refers to discrete optimization;

Applied Mathematics & Systems Laboratory - Information Processing Group

On the selection of Seeds ‰ Finding

appropriate candidates for the completion is one of the two most critical components of our approach;

‰ We

are based on the principle that missing content is not necessarily in the vicinity of the destroyed region

‰ We

can consider the seed selection problem as a random variable; that consists of i The

center of the seed (in the image) i Its from (mostly rectangular) as well as its dimensions and orientation ‰ Such

a problem consists of finding given an origin point in the impainted regions a number of candidate patches in the image domain that potentially have similar content with one that is missing and the one that is partially present

Applied Mathematics & Systems Laboratory - Information Processing Group

On the Selection of Candidate Seeds ‰ Such

seeds selection can be in a probabilistic formulation;

i Given

an origin point, create a number of perturbations [in terms of the position of the seed, its form and its orientation] according to some known distribution i Once new candidate seeds have been determined, evaluate the probability of containing information to complete the missing content; i Consider a number of the most probable seeds and repeat the process until a number of seeds have been selected of varying form, position and orientation with content potentially interesting to complete the missing one

Applied Mathematics & Systems Laboratory - Information Processing Group

Gaussian PDF estimation ‰ Compute

to t: ‰ If

the present state of system using the observations from 1, or determine the probability

we assume that the density function is known at t-1: one can consider the Bayes rule

‰ That

then

can be re-written using the Chapman-Kolmogorov equation

‰ While

one can claim a number of approaches to estimate this density (incremental EM – otherwise infinite memory is needed) a lack exists on efficient numerical techniques

Applied Mathematics & Systems Laboratory - Information Processing Group

Particle Filters ‰ Particle

filters, which are sequential Monte-Carlo techniques, estimate the Bayesian posterior probability density with a set of samples

‰ Let ‰A

us consider M random measures:

particle filter is a non linear approximation of this density consists of

: ‰ With

being weights that reflect the importance of

the different samples ‰ Once

a set of samples has been drawn, can be computed out of the observation for each sample, and the estimation of the posteriori pdf can be sequentially updated.

Applied Mathematics & Systems Laboratory - Information Processing Group

And a Demonstration…

Applied Mathematics & Systems Laboratory - Information Processing Group

On the optimization of the Cost function ‰ Suboptimal

Methods

i Simulated

Anealing i Iterated Conditional Modes i Highest Confidence First ‰ Semi-Optimal

Methods

i Graph-based

‰ Optimal

optimization

methods

i Meanfield

anealing i Belief Propagation Networks

Applied Mathematics & Systems Laboratory - Information Processing Group

Graph Cuts ‰A

graph

consists of:

iA

set of nodes i A set of links/directed edges that connect these nodes i Two special terminal nodes often called source and sink that have a different nature than the rest of the nodes i Each

graph-edge is assigned to a positive weight i The cost of an edge in one direction can differ from the one of the opposite direction i An edge is called a t-link if it connects a non-terminal node with a terminal and n-link if it connects two nonterminal nodes

Applied Mathematics & Systems Laboratory - Information Processing Group

The Min-Cut Problem ‰ An

cut disjoint subsets

‰ The

Cost of a cut “boundary edges”

refers to a partition of the graph nodes in two and such that the source is in and the sink in is the sum of cost weights of the such that and

‰ The

MINIMUM cut problem is to the find the cut that has minimum cost among all possible cuts

‰ One

of the fundamental results in combinatorial optimization is that the MIN-CUT problem is equivalent with the MAX-FLOW problem.

Applied Mathematics & Systems Laboratory - Information Processing Group

The Max-Flow Problem ‰ Consider

to be a water source, and the graph a network of directed pipes with capacities equal to the edge weights

‰ Maximum

flow is the maximum amount of water that can be sent from the source to the sink

‰ The

theorem of Ford-Fulkerson states that the maximum flow from the source to the destination saturates a set of edges in the graph dividing the nodes into two disjoint sets corresponding to the minimum cut,

‰ Two

type of algorithms to solve the max-flow problem

i Push-relabel

techniques i Augmenting path methods

Applied Mathematics & Systems Laboratory - Information Processing Group

Expansion move algorithm Input labeling f

Red expansion move from f

i Find

red expansion move that most decreases E

i Move

there, then find the best blue expansion move, etc i Done when no α-expansion move decreases the energy, for any label α i Many

nice theoretical properties

Applied Mathematics & Systems Laboratory - Information Processing Group

In Practice

Applied Mathematics & Systems Laboratory - Information Processing Group

In Practice

Applied Mathematics & Systems Laboratory - Information Processing Group

The Case of Video & Structure ‰ Video ‰ The

Can be considered as a three dimensional volume

exact same concept can be used, where now during the construction of the graph in the temporal domain, object correspondences driven from the optical flow computation can be used to determine the costs of the cut in this direction

Applied Mathematics & Systems Laboratory - Information Processing Group

Discussion ‰ Image,

Texture , Video & Structure Completion was a formulated as a MAP problem

‰ Image

terms, as well as smoothness considerations were used to fill in the missing content

‰ Random

walks, particle filters and statistical hypotheses testing were used to determine candidate completion structures in the image

‰ Combinatorial

optimization and the alpha expansion algorithm was used to recover the optimal configuration in terms of labels

‰ The

method is rather general and can be used to deal with higher dimensions

‰ The

method is favorable compared to PDEs since no assumption on the image content is made.

Applied Mathematics & Systems Laboratory - Information Processing Group

Future Work ‰ Replacing

the Video Content through coupling with optical flow estimation (we are almost there)

‰ Stereo

reconstruction through similar fitting process…(start working on

that) ‰ Pre-reconstruction

of a number of basic elements (3D patches) and the complete reconstruction of an unseen scene through le”going” these patches to the image content

‰ Augmentation

of archeological scenes to fill in the missing content from either different locations of the same monument, or components from different monuments

‰ DO

NOT FORGET TO GO TO THE NEXT SLIDE!!!

Applied Mathematics & Systems Laboratory - Information Processing Group

Advertisement Section The Handbook of Mathematical Models in Computer Vision Springer (2005), ISBN 0387263713, 596 pages ‰ Editors i

Nikos Paragios

i

Yunmei Chen

i

Olivier Faugeras

http://cermics.enpc.fr/~paragios ‰ Chapter

3: PDE-Based Image and Surface

Inpainting, Bertalmio, Caselles, Haro, and Sapiro ‰ Chapter

5: Graph Cuts in Vision and

Graphics: Theories and Applications, Boykov and Veksler