Image restoration: edge preserving — Convex penalties —
Jean-Fran¸cois Giovannelli
[email protected] Groupe Signal – Image Laboratoire de l’Int´egration du Mat´eriau au Syst`eme Univ. Bordeaux – CNRS – BINP
1 / 32
Topics Image restoration, deconvolution Motivating examples: medical, astrophysical, industrial,. . . Various problems: Fourier synthesis, deconvolution,. . . Missing information: ill-posed character and regularisation
Three types of regularised inversion 1
Quadratic penalties and linear solutions Closed-form expression Computation through FFT Numerical optimisation, gradient algorithm
2
Non-quadratic penalties and edge preservation Half-quadratic approaches, including computation through FFT Numerical optimisation, gradient algorithm
3
Constraints: positivity and support Augmented Lagrangian and ADMM
Bayesian strategy: a few incursions Tuning hyperparameters, instrument parameters,. . . Hidden / latent parameters, segmentation, detection,. . . 2 / 32
Convolution / Deconvolution y = Hx + ε = h ? x + ε ε x
H
+
y
b = Xb(y) x Restoration, deconvolution-denoising General problem: ill-posed inverse problems, i.e., lack of information Methodology: regularisation, i.e., information compensation Specificity of the inversion / reconstruction / restoration methods Trade off and tuning parameters
Limited quality results 3 / 32
Competition: Adequation to data Compare observations y and model output Hx Unknown: x Known: H and y
Comparison experiment – model x
H
Hx y
Quadratic criterion: distance observation – model output JLS (x) = ky − Hxk
2
4 / 32
Competition: Smoothness prior
Data insufficiently informative Account for prior information 1 2
Question of compromise, of competition Specificity of methods
Here: smoothness of images Quadratic penalty of the gray level “gradient” (or other linear combinations) X P(x) = (xp − xq )2 p∼q 2
= kDxk
5 / 32
Quadratic penalty: criterion and solution
Criterion including quadratic penalty 2
2
ky − Hxk + µ kDxk X 2 = ky − Hxk + µ (xp − xq )2
JPLS (x)
=
p∼q
Restored image bPLS x t
t
=
arg min JPLS (x) x t
bPLS (H H + µD D) x
=
H y
bPLS x
=
(H t H + µD t D)−1 H t y
6 / 32
Computational aspects and implementation Various options and many relationships. . . Direct calculus, compact (closed) form, matrix inversion Algorithms for linear system solution Gauss, Gauss-Jordan Substitution Triangularisation,. . .
Special algorithms, especially for 1D case Recursive least squares Kalman smoother or filter (and fast versions,. . . )
Numerical optimisation gradient descent: descent in the direction opposite to the gradient. . . . . . and various modifications
Diagonalization Circulant approximation and diagonalization by FFT 7 / 32
Solution from quadratic penalty
Input
Data
Quadratic penalty
Quadratic penalty
Circulant approx
Non-circulant
8 / 32
Edge preserving approaches
Limited resolution of linear solutions Desirable: less “smoothing” around “discontinuities” Ambivalence: Smoothing (homogeneous regions) Heightening, enhancement, sharpening (discontinuities, edges)
. . . and new compromise, trade off, conciliation
Algorithmic structure Based on the linear solution (based on quadratic criterion) FFT (Wiener-Hunt) Kalman smoother or filter (and its fast versions, factorised,. . . ) Numerical optimisation (gradient,. . . )
9 / 32
Edge preservation and non-quadratic penalties Restored image still defined as the minimiser. . . b = arg min J(x) x x
. . . of a penalised criterion . . . 2
J(x) = ky − Hxk + µ P(x) . . . once again penalising “small variations” but less penalising for “discontinuities” X ϕ(xp − xq ) P(x) = p∼q
ϕ(δ) = δ 2
ϕ(δ) = . . . ?
Ambivalence: new compromise, trade off, conciliation Smoothing (homogeneous regions) Heightening, enhancement, sharpening (discontinuities, edges) 10 / 32
Typical potentials Again ϕ(δ) ∼ δ 2 for small δ Behaviour for large δ 1
Horizontal asymptote
2
Horizontal parabolic behaviour
3
Oblique (slant) asymptote
4
Vertical parabolic behaviour Alpha = 0.50 et Seuil = 1.00
Q
2.5
Huber H and L G and McC
2
B and Z
Potentiel
1.5
1
0.5
0 −5
−4
−3
−2
−1 0 1 Variable Delta
2
3
4
5
11 / 32
Four major types of potentials 1
Horizontal asymptote ϕ(δ) ∼ 1 [Blake and Zisserman (87), Geman and McClure (87)] ϕ(δ) = αs 2 min 1, (δ/s)2
;
ϕ(δ) = αs 2
(δ/s)2 1 + (δ/s)2
2
Horizontal parabolic behaviour ϕ(δ) ∼ log |δ| [Hebert and Leahy (89)] ϕ(δ) = αs 2 log 1 + (δ/s)2
3
Oblique (slant) asymptote ϕ(δ) ∼ |δ| Huber, hyperbolic, Log hyperbolic-cosine , fair function ( q δ2 if | δ | 6 s 2 2 ϕ(δ) = α ; ϕ(δ) = 2αs 1 + [δ/s] − 1 2s | δ | − s 2 if | δ | > s
4
Vertical parabolic behaviour ϕ(δ) ∼ δ 2 Wiener-Hunt solution ϕ(δ) = αδ 2 12 / 32
Potentials with oblique asymptote (L2 / L1 ): details Alpha = 0.50 et Seuil = 1.00 2.5
Potentiel
2 1.5 1 0.5 0 −4
−3
−2
−1
0 1 Variable Delta
(
2
3
2
[δ/s] 2 | δ |/s − 1
4
if | δ | 6 s if | δ | > s
Huber :
ϕ(δ)
=
αs 2
Hyperbolic :
ϕ(δ)
=
2αs 2
LogCosh :
ϕ(δ)
=
2αs 2 log cosh (| δ |/s)
FairFunction :
ϕ(δ)
=
2αs 2 [| δ |/s − log (1 + | δ |/s)]
q
2 1 + [δ/s] − 1
13 / 32
More general non-quadratic penalties (1) Differences, higher order derivatives, generalizations,. . . X P(x) = ϕ(xn+1 − xn ) n
P(x)
=
X
ϕ(xn+1 − 2xn + xn−1 )
n
P(x)
=
X
=
X
ϕ(α xn+1 − xn + α0 xn−1 )
n
P(x)
ϕ(αtn x)
n
Linear combinations (wavelet, other-stuff-in-et,. . . dictionaries,. . . ) ! X X X t P(x) = ϕ(wn x) = ϕ wnm xm n
n
m
Redundant or not Link with Haar wavelet and other 14 / 32
More general non-quadratic penalties (2) 2D aspects: derivative, finite differences, gradient approximations X P(x) = ϕ(xp − xq ) p∼q
=
X
ϕ(xn+1,m − xn,m ) +
n,m
X
ϕ(xn,m+1 − xn,m )
n,m
Notion of neighborhood and Markov field Any highpass filter, contour detector (Prewitt, Sobel,. . . ) Linear combinations: wavelet, contourlet and other-stuff-in-et,. . .
Other possibilities (slightly different) Enforcement towards a known shape X P(x) = ϕ(xp − x¯p ) p
Separable penalty P(x) =
X
ϕ(xp )
p 15 / 32
Penalised least squares solution A reminder of the criterion and the restored image X 2 J(x) = ky − Hxk + µ ϕs (xp − xq ) p∼q
b = arg min J(x) x x
with ϕs one of the mentioned (non-quadratic) potentials and two hyperparameters: µ and s
Non-quadratic criterion Non-linear gradient No closed-form expression
Two questions Practical computation: numerical optimisation algorithm,. . . Minimiser: existence, uniqueness,. . . continuity 16 / 32
Convexity and existence-uniquiness Convex set RN , RN+ , intervals of RN ,. . . Properties: intersection, convex envelope, projection,. . .
Strictly convex criterion, convex criterion, Θ(u) = u 2 , Θ(u) = kuk2 , Θ(u) = | u |, Huber,. . . Properties: sum of convex function, level sets,. . .
Key result Set of minimisers of convex criterion on a convex set is a convex set Strict convexity unique minimiser
Application ϕ convex =⇒ J convex
In the following developments, potential ϕs Huber or hyperbolic: convex (strict) guarantees In addition: non-convex no guarantee (although, sometimes. . . ) 17 / 32
Half-quadratic enchantement (start) Reminder of the criterion 2
J(x) = ky − Hxk + µ
X
ϕ(xp − xq )
p∼q
Minimisation based on quadratic Original idea of [Geman + Yang, 95] Set of auxiliary variables bpq so that: ϕ(δpq ) ←→ δ 2pq
ϕ(δ) = inf b
1 (δ − b)2 + ζ(b) 2
With appropriate ζ(b):
Extended criterion 2 ˜ J(x, b) = ky − Hxk + µ
X1 p∼q
2
2
[(xp − xq ) − bpq ] + ζ(bpq )
und nat¨ urlich: ˜ J(x) = inf J(x, b) b
18 / 32
Legendre-Fenchel Transform (LFT) or Convex Conjugate (CC) Definition of TLF or CC Consider f : R −→ R strictly convex once (or twice) differentiable The TLF or CC is the function f ? : R −→ R defined by: f ? (t) =
sup [ xt − f (x) ] x∈R
Remark f ? (0) = sup [ −f (x) ] = − inf [ f (x) ] x∈R
x∈R
∀t, x ∈ R,
xt − f (x) 6 f ? (t)
∀t, x ∈ R,
f ? (t) + f (x) > xt 19 / 32
LFT: some shift and dilatation / contraction properties f ? (t) =
sup [ xt − f (x) ] x∈R
Vertical: shift-dilatation (α ∈ R and β ∈ R?+ ) ( g (x) = α + β f (x) g ? (t) = β f ? (t/β) − α Horizontal: shift (x0 ∈ R) and dilatation (γ ∈ R?+ ) ( ( g (x) = f (x − x0 ) g (x) = f (γx) g ? (t) = f ? (t) − tx0 g ? (t) = f ? (t/γ) Specific case α = 0, β = 1 /
x0 = 0 /
γ=1 20 / 32
LFT: a first example Quadratic case (α ∈ R and β ∈ R?+ ) Let us consider the function 1 f (x) = α + β(x − x0 )2 2 We look for the CC f ? (t) = sup [ xt − f (x) ] x∈R
Let us denote gt (x) = xt − f (x) = xt − α + β(x − x0 )2 / 2 The derivative reads: gt0 (x) = t − β(x − x0 ) And the second derivative is: gt (x)00 = −β By nullification of gt0 (x): x¯ = x0 + t/β Then by substitution: f ? (t) = gt (¯ x)
1 1 2 2 f (t) = (t + βx0 ) − α + βx0 2β 2 ?
21 / 32
LFT: a generic result for explicitation (a) A swiss army formula: Legendre formula f ? (t) = sup [ xt − f (x) ] x∈R
Let us denote gt (x) = xt − f (x) The derivative reads: gt0 (x) = t − f 0 (x) And the second derivative is: gt (x)00 = −f 00 (x) By nullification of gt0 (x): t − f 0 (¯ x)
=
0
x¯
=
f
0−1
(t) = χ(t)
Then by substitution: f ? (t) = gt (¯ x ) = t¯ x − f (¯ x ) = tχ(t) − f [χ(t)]
22 / 32
LFT: a generic result for explicitation (b) Derivatives Convex conjugate made explicit f ? (t) = tχ(t) − f [χ(t)] with χ = f
0−1
The derivative reads: 0
f ? (t)
=
χ(t) + tχ0 (t) − χ0 (t) f 0 [χ(t)]
=
χ(t)
=
f
0−1
(t)
And the second derivative is: 00
f ? (t) = χ(t)0 =
1 f [χ(t)] 00
>0
hence f ? is convex
23 / 32
LFT: a key result Double conjugate Transformed twice gives back the original f ?? (x) = f (x) f ?? (t) = sup [ xt − f ? (x) ] Let us note ht (x) = xt − f ? (x) and nullify the derivative: 0 x¯
0
= ht0 (¯ x ) = t − f ? (¯ x) = t − f =
−1
χ
0−1
(¯ x ) = t − χ(¯ x)
0
(t) = f (t)
By substitution f ?? (t) = ht (¯ x)
= x¯t − f ? (¯ x) = x¯t − [¯ x χ(¯ x ) − f (χ(¯ x ))] = x¯t − x¯t + f (t) = f (t) 24 / 32
Outcome for LFT: “un th´eor`eme vivant” Definition Let us consider f : R −→ R strictly convex once (or twice) differentiable f ? (t) =
sup [ xt − f (x) ] x∈R
Properties f ? (t) = tχ(t) − f [χ(t)] with χ = f 0
f? =f f
?00
0−1
0−1
=χ 00
(t) = 1 / f [χ(t)] f ? is convex f ?? (x) = f (x)
25 / 32
A theorem in action: half-quadratic (beginning) Problem statement Consider a potential ϕ, convex or not, and look for ζ such that ϕ(δ) = inf (δ − b)2 /2 + ζ(b) b∈R
Let us define g such that it is strictly convex: g (δ) = δ 2 /2 − ϕ(δ) Consider its CC: g ? (b)
=
sup [ bδ − g (δ) ] sup ϕ(δ) − (δ − b)2 /2 + b 2 /2
δ∈R
=
δ∈R
Let us set (reason explained on the next slide): ζ(b) = g ? (b) − b 2 /2 = sup ϕ(δ) − (δ − b)2 /2 δ∈R 26 / 32
A theorem in action: half-quadratic (middle) Take advantage of g = g ?? g (δ)
=
δ 2 /2 − ϕ(δ)
=
ϕ(δ)
= =
g ?? (δ) sup [ bδ − g ? (δ) ] b∈R 2
δ /2 − sup [bδ − g ? (δ)] b∈R inf (δ − b)2 /2 + ζ(b)
b∈R
The icing on the cake, we have the minimiser: ¯ 2 /2 + ζ(b) 0 = (b¯ − δ) + ζ 0 (b) ¯ = g ?0 (b) ¯ −δ 0 = (δ − b) then:
0−1 b¯ = g ? (δ) = g 0 (δ) = δ − ϕ0 (δ)
27 / 32
A theorem in action: half-quadratic (ending)
Reminder: original criterion and extended criterion X 2 J(x) = ky − Hxk + µ ϕ(xp − xq ) p∼q 2 ˜ J(x, b) = ky − Hxk + µ
X1 p∼q
2
2
[(xp − xq ) − bpq ] + ζ(bpq )
Algorithmic strategy: alternating minimisation 1
2
˜ e(b) = arg minx J(x, Minimisation w.r.t. x for fixed b: x b) Quadratic problem ˜ Minimisation w.r.t. b for fixed x: e b(x) = arg minb J(x, b) Separated and explicit update
28 / 32
Image update Non-separable but quadratic ˜ J(x)
2
= ky − Hxk + µ
X1 p∼q
2
2
2
[(xp − xq ) − bpq ] 2
= ky − Hxk + µ ¯ kDx − bk Multivariate system of equations e (H t H + µ ¯D t D) x
= H ty + µ ¯ Dt b
Minimiser e x
=
(H t H + µ ¯D t D)−1 (H t y + µ ¯ D t b)
Numerical aspects Descent algorithm (including inner-iterations) Circulant case and FFT (without inner-iterations) 29 / 32
Auxiliary variables update Separable (parallel computation), explicit (no inner-iterations), easy Each b = bpq
↔ δ = δpq = xp − xq
Update: e b = δ − ϕ0 (δ) Huber: e b = δ [1 − 2α min (1; s/δ)] 2α Geman & McClure: e b =δ 1− (s/δ)2 ... Alpha = 0.40 et Seuil = 2.00
Facteur de contraction
1 0.8 0.6 Huber H and L G and McC B and Z
0.4 0.2 0 −10
−5
0 Variable Delta
5
10
30 / 32
Result L2 / L1 (Huber)
Input
Observations
Quadratic penalty
L2-L1 (Huber) penalty
31 / 32
Synthesis and extensions Synthesis Image deconvolution Also available for non-invariant linear direct model And also available for the case of signal. . .
Edge preserving and non-quadratic penalties Gradient of gray levels (and other transform) Convex (and differentiable) case and also some non-convex cases
Numerical computations Half-quadratic approach: separable and FFT Numerical optimisation (gradient,. . . )
Extensions Including constraints
better image resolution (next lecture)
Hyperparameters estimation, instrument parameter estimation,. . . 32 / 32