Image restoration — Convex approaches: penalties and constraints —
Introduction and examples
Jean-Fran¸cois Giovannelli
[email protected] Groupe Signal – Image Laboratoire de l’Int´egration du Mat´eriau au Syst`eme Univ. Bordeaux – CNRS – BINP
1 / 34
Topics Image restoration, deconvolution Motivating examples: medical, astrophysical, industrial,. . . Various problems: Fourier synthesis, deconvolution,. . . Missing information: ill-posed character and regularisation
Three types of regularised inversion 1
Quadratic penalties and linear solutions Closed-form expression Computation through FFT Numerical optimisation, gradient algorithm
2
Non-quadratic penalties and edge preservation Half-quadratic approaches, including computation through FFT Numerical optimisation, gradient algorithm
3
Constraints: positivity and support Augmented Lagrangian and ADMM
Bayesian strategy: a few incursions Tuning hyperparameters, instrument parameters,. . . Hidden / latent parameters, segmentation, detection,. . . 2 / 34
Interferometry: principles of measurement Physical principle [Thompson, Moran, Swenson, 2001] Antenna array
large aperture
Frequency band, e.g., 164 MHz Couple of antennas interference
Antenna positions
Fourier plane
0
Frequency (v )
North-South (km)
Picture site (NRH)
one measure in the Fourier plane
−1
−2
20
40
60
80
100
120
−1
0
1
2
3
West-Est (km)
20
40
60
80
100
120
Frequency (u)
Knowledge of the sun, magnetic activity, eruptions, sunspots,. . . Forecast of sun events and their impact,. . . 3 / 34
Interferometry: illustration True map ES
True map PS
Dirty beam
Dirty map ES
Dirty map PS
Dirty map PS + ES
4 / 34
MRI: principles of measurement Physical principle [Alaux 92] spin precession, f ∝ kBk
Intense magnetic field B
Gradient B = B0 + B(x) coding space - frequency Proton density signal amplitude
0.5 0.4 0.3 0.2 0.1 0 −0.1
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2 0.1
−0.3
0
0
−0.1
−0.1
−0.1
−0.4
−0.2
−0.2
−0.2
−0.3
−0.3
−0.3
−0.5 −0.4
GE Phantom
0.5
0.1
−0.2
−0.2
0
0.2
0.4
0.6
Frequency coverage
0.2 0.1 0
−0.4
−0.4
−0.4
−0.5
−0.5
−0.5
−0.4
−0.2
0
0.2
0.4
0.6
−0.5 −0.4 −0.3 −0.2 −0.1
0
0.1
0.2
0.3
0.4
0.5
−0.4
−0.2
0
0.2
0.4
0.6
Other acquisition schemes
Medical imaging, morphological and functional, neurology,. . . Fast MRI, cardiovascular applications, flow imaging,. . . 5 / 34
MRI: illustration
Instrument response −1 100
Phantom (true)
−2 50−3
200
“Observations”
−4
100
300 −5 150
400
−6 200
−7
500 100
200
300
400
500 250
Cross-section
300
−0.5 350
−1.5 400
−2.5
450
−3.5
500 50
−4.5
100
200
300
400
100
150
200
250
300
350
400
450
500
500
6 / 34
X-ray tomography (scanner): principles of measurement Physical principle X-rays absorption
radiography
Rotation around the objet
a set of radiographies (sinogram)
Radon transform
Materials analysis and characterization, airport security,. . . Medical imaging: diagnosis, therapeutic follow-ups,. . . 7 / 34
X-ray tomography: illustration
8 / 34
Hydrogeology and source identification Physical principle Source: chemical, radioactive, odor,. . . Transport phenomena in porous media Groundwater sensors (drilling)
5
5
(ZNS) y
x 10
5
(1) 5
4
4
3
3
2
2
1
x
0 0
(ZS)
5
5
(2)
1
0
z
x 10
x 10
100 t[j] (3)
200
0 5
5
4
4
3
3
2
2
1
x 10
100 t[j] (4)
200
100 t[j]
200
1
0
0 0
100 t[j]
200
0
Monitoring: electricity generation, chemical industry,. . . Knowledge for its own sake: subsoils, transportation, geology,. . . 9 / 34
Ultrasonic imaging Physical principle Interaction: ultrasonic wave ↔ medium of interest Acoustic impedance: inhomogeneity, discontinuity, medium change deplacement
faisceau acoustique interfaces
emission : transducteur
reception : 10 10
Temps (microsecondes)
11 12
micros.
20
13
30
14
40
15
50
16 60
2
4
6
8 mm
10
12
14
17 100
200
300
400
Traces
Industrial control: very-early cracks detection (nuclear plants,. . . ) Non destructive evaluation: aeronautics, aerospace,. . . Tissue characterisation, medical imaging,. . . 10 / 34
Seismic reflection method Physical principle Interaction: mechanical wave ↔ medium of interest Acoustic impedance: inhomogeneity, discontinuity, medium change ? source
geophones
Trace z
Ondelette h
AAAAAAAA AAAAAAAA AAAAAAAA AAAAAAAA t
Reflectivite r
t
t
Mineral and oil exploration,. . . Knowledge of subsoils and geology,. . .
11 / 34
Optical imaging (and infrared, thermography)
Physical principle Fundamentals of optics (geometrical and physical) stain CCD sensors or bolometers spatial and time response Public space surveillance (car traffic, marine salvage,. . . ) Satellite imaging: astronomy, remote sensing, environment Night vision, smokes / fogs / clouds, bad weather conditions
12 / 34
Digital photography and demosaicing
Physical principle Matrix / filter / Bayer mosaic: red, green, blue Chrominance and luminance Holiday pictures Surveillance (public space, car traffic,. . . )
13 / 34
The case of super-resolution
Physical principle Time series of images
over-resolved images
Sub-pixel motion ∼ over-sampling Motion estimation + Restoration Same applications. . . with higher resolution
14 / 34
And other imaging. . . fields, modalities, problems,. . . Fields Astronomy, geology, hydrology,. . . Thermography, fluid mechanics, transport phenomena,. . . Medical: diagnosis, prognosis, theranostics,. . . Remote sensing, airborne imaging,. . . Surveillance, security,. . . Non destructive evaluation, control,. . . Computer vision, under bad conditions,. . . Photography, games, recreational activities, leisures,. . . ... Health, knowledge, leisure,. . . Aerospace, aeronautics, transport, energy, industry,. . .
15 / 34
And other imaging. . . fields, modalities, problems,. . . Modalities Interferometry (radio, optical, coherent,. . . ) Magnetic Resonance Imaging Tomography based on X-ray, optical wavelength, tera-Hertz,. . . Ultrasonic imaging, sound, mechanical Holography Polarimetry: optical and other Synthetic aperture radars Microscopy, atomic force microscopy ... Essentially “wave ↔ matter” interaction
16 / 34
And other imaging. . . fields, modalities, problems,. . . “Signal – Image” problems Denoising Edge / contrast enhancement Missing data (inpainting, interpolation,. . . ) Deconvolution Inverse Radon Fourier synthesis ... And also: Segmentation Detection of impulsions, salient points,. . . ...
In the following lectures: deconvolution-denoising 17 / 34
Inversion: standard question y = H(x) + ε = Hx + ε = h ? x + ε ε x
H
+
y
b = Xb(y) x Restoration, deconvolution-denoising Genaral problem: ill-posed inverse problems, i.e., lack of information Methodology: regularisation, i.e., information compensation Specificity of the inversion / reconstruction / restoration methods Trade off and tuning parameters
Limited quality results 18 / 34
Inversion: advanced question y = H(x) + ε = Hx + ε = h ? x + ε ε, γ x, γ, `
H, θ
h
+
y
b = Xb(y) x i b b b, γ b , θ, x ` = Xb(y)
More estimation problems Hyperparameters, tuning parameters: unsupervised Instrument parameters (resp. response): myopic (resp. blind) Hidden variables: edges, regions, singular points,. . . : augmented Different models for image, noise, response,. . . : model selection 19 / 34
Issues and framework Inverse problems Instrument model, direct / forward model Involves physical principles of the phenomenom at stake the acquisition system, the sensor
Inverse undo the degradations, surpass natural resolution from consequences to causes restore / rebuild / retrieve
Ill-posed / ill-conditioned character and regularisation Framework Direct model linear and shift invariant, i.e., convolutive including additive error (model and measurement)
Regularisation through penalties and constraints Criterion optimisation and convexity 20 / 34
Some historical landmarks Quadratic approaches and linear filtering ∼ 60 Phillips, Twomey, Tikhonov Kalman Hunt (and Wiener ∼ 40)
Extension: discrete hidden variables ∼ 80 Kormylo & Mendel (impulsions, peaks,. . . ) Geman & Geman (lines, contours, edges,. . . ) Besag, Graffigne, Descombes (regions, labels,. . . )
Convex penalties (also hidden variables,. . . ) ∼ 90 L2 − L1 , Huber, hyperbolic: Sauer, Blanc-Fraud, Idier. . . L1 : Alliney-Ruzinsky, Taylor ∼ 79, Yarlagadda ∼ 85 . . . And. . . L1 -boom ∼ 2005
Back to more complex models ∼ 2000 Unsupervised, myopic, semi-blind, blind Stochastic sampling (MCMC, Metropolis-Hastings. . . ) 21 / 34
Example due to Hunt (“square” response) [1970]
Convolutive model (sample averaging): y = h ? x + ε
0.3 1.2
1.2 0.25
1
1 0.2
0.8
0.8 0.15
0.6
0.6 0.1
0.4
0.4 0.05
0.2
0.2 0
0
0 −0.05
−0.2
−0.2 0
20
40
60
80
Input (x)
100
120
−0.1
−10
−8
−6
−4
−2
0
2
4
Response (h)
6
8
10
0
20
40
60
80
100
120
Output (y)
22 / 34
Example: photographed photographer (“square” response) Convolutive model (pixels averaging): y = h ? x + ε ◦
◦
◦
◦
Fourier domain: y (ν) = h(ν) x(ν) + ε(ν) 1 0.8 0.6 0.4 0.2 0 −0.5
−0.4
−0.3
−0.2
−0.1
0
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
−0.5
−0.4
−0.3
−0.2
−0.1
0
0.1
0.2
0.3
0.4
0.5 −0.5
Spatial response
−0.4
−0.3
−0.2
−0.1
0
Frequency response
Input (x)
Output (y)
23 / 34
Example: photographed photographer (“motion blur”) Convolutive model (pixels averaging): y = h ? x + ε ◦
◦
◦
◦
Fourier domain: y (ν) = h(ν) x(ν) + ε(ν) −0.5
−0.4
−0.3
−0.2
−0.1
0
0.1
0.2
0.3
0.4
0.5 −0.5
Spatial response
−0.4
−0.3
−0.2
−0.1
0
0.1
0.2
0.3
0.4
0.5
Frequency response
Input (x)
Output (y) 24 / 34
Convolution equation (discrete time / space) Examples of response Square
Motion
Gaussian
Diffraction
Punctual
Convolutive model z(n)
=
+P X
h(p) x(n − p)
p=−P
z(n, m)
=
+Q +P X X
h(p, q) x(n − p, m − q)
p=−P q=−Q
Response: h(p, q) or h(p) impulse response, convolution kernel,. . . . . . point spread function, stain image 25 / 34
Convolution equation (discrete time, 1D ): matrix form Linear matricial relation: z = Hx Shift invariance Tœplitz structure Short response band structure . . . zn−1
H =
=
zn+1
=
. . . ... ... ... ... . . .
=
zn
hP 0 0 0 0 0
. . .. . . ... hP 0 0 0 0 . . .
hP xn−P + · · · + h1 xn−1 + h0 xn + h−1 xn+1 + · · · + h−P xn+P
h0 ... hP 0 0 0
. . . ... h0 ... hP 0 0 . . .
h−P ... h0 ... hP 0
. . . 0 h−P ... h0 ... hP . . .
0 0 h−P ... h0 ...
. . . 0 0 0 h−P ... h0 . . .
0 0 0 0 h−P ...
. . . 0 0 0 0 0 h−P . . .
... ... ... ...
See exercises regarding Tœplitz and circulant matrices. . . 26 / 34
Short and important incursion in “continuous statement“
More realistic modelling of physical phenomenon Continuous variable convolution (1D and 2D) Observations Sampling (discretization) of output Finite number of samples
Decomposition of unknown object Again “discrete” convolution
27 / 34
Convolution equation: continuous variable Convolutive integral equation Z z(t)
=
z(u, v )
=
x(τ ) h(t − τ ) dτ ZZ
x(u 0 , v 0 ) h(u − u 0 , v − y 0 ) du 0 dv 0
More generally: Fredholm integral equation (first kind) Z z(t)
=
x(τ ) h(t, τ ) dτ ZZ
z(u, v )
=
x(u 0 , v 0 ) h(u, u 0 , v , y 0 ) du 0 dv 0
28 / 34
Continuous convolution and discrete observations
Convolutive integral equation Z for t ∈ R :
+∞
x(τ ) h(t − τ ) dτ
z(t) = −∞
Measurement Discrete data (just sampling, no approximation). . . Z +∞ zn = z(nTs ) = x(τ ) h(nTs − τ ) dτ −∞
. . . and finite number of data: n = 1, 2, . . . , N. Unknown object remains “continuous variable”: x(t), for t ∈ R
29 / 34
Object “decomposition-recomposition”
“General” decomposition of continuous time object X x(τ ) = xk ϕk (τ ) k
Fourier series and finite time extend (finite duration) Cardinal sine and finite bandwidth Spline, wavelets, Gaussian kernel. . . ...
Infinite dimensional linear algebra Hilbert spaces, Sobolev spaces. . . Basis, representations. . . Inner products, norms, projections. . .
30 / 34
Object “de / re - composition”: example of finite bandwidth “General” decomposition of continuous time object X x(τ ) = xk ϕk (τ ) k
Case of shifted version of a basic function ϕ0 ϕk (τ ) = ϕ0 (τ − kδ) Special case with cardinal sine ϕ0 (τ ) = sinc t/δ
sin πu with sinc u = πu
That is the Shannon reconstruction formula X X τ − kδ x(τ ) = xk ϕ0 (τ − kδ) = xk sinc δ k∈Z
k∈Z
. . . with xk = . . .
There is no approximation if. . . 31 / 34
Convolution: continuous
discrete
Given that (discrete observation at time nTs ) Z +∞ zn = x(τ ) h(nTs − τ ) dτ for n = 1, 2, . . . , N −∞
and that (case of shifted version of a basic function ϕ0 ) X x(τ ) = xk ϕ0 (τ − kδ) for τ ∈ R k
We have Z zn
+∞
"
=
# X
−∞
=
X
Z
=
+∞
ϕ0 (τ − kδ) h(nTs − τ ) dτ
xk −∞
k
X
xk ϕ0 (τ − kδ) h(nTs − τ ) dτ
k
Z
+∞
xk −∞
k
|
ϕ0 (τ ) h([nTs − kδ] − τ ) dτ {z } 32 / 34
Convolution: continuous
discrete
Let us denote: h¯ = ϕ0 ? h ¯ h(u)
Z
+∞
ϕ0 (τ ) h(u − τ ) dτ
= −∞
We then have zn
=
X k
=
X
Z
+∞
ϕ0 (τ ) h([nTs − kδ] − τ ) dτ
xk −∞
¯ xk h(nT s − kδ)
k
The zn are given as a function of the xk It is a “discrete linear” transform There is no apprximation
33 / 34
Convolution: continuous
discrete
A specific case when δ = Ts /K X zn = xk h¯ [nTs − kδ] k
=
X
=
X
xk h¯ [nK δ − kδ]
k
xk h¯ [(nK − k)δ]
k
Subsampled discrete convolution
A specific case when δ = Ts , i.e., K = 1 X zn = xk h¯ [(n − k)δ] k
A standard discrete convolution
34 / 34
