'
Bayesian inference for inverses problems in signal and image processing
$
Ali Mohammad-Djafari Ph.D. Students & collaborators: M. Ichir, O. F´eron, P. Brault, A. Mohammadpour, Z. Chama H. Snoussi, F. Humblot, S. Moussaoui Laboratoire des Signaux et Syst`emes ´lec-Ups Cnrs-Supe Sup´elec, Plateau de Moulon 91192 Gif-sur-Yvette, FRANCE.
[email protected] &
http://djafari.free.fr 1
%
'
$
Contents • Inverses problems in image processing • Multi sensor image processing problems • Basics of Bayesian approach • HMM modeling of images • Examples of applications – Single channel image restoration – Fourier synthesis in optical imaging – Multi channel data fusion and joint segmentation – Video movie segmentation with motion estimation – Blind source (image) separation (BSS) – Hyperspectral image segmentation • Bayesian image processing in wavelet domain
• Some results and conclusions &
2
%
'
Inverses problems in image processing
$
• General non linear inverse problem: g(r) = [Hf (r 0 )](r) + ²(r), • Linear model: g(r) =
Z
r = (x, y) ∈ R,
r 0 = (x0 , y 0 ) ∈ R0
f (r 0 )h(r, r 0 ) dr 0 + ²(r) R0
• Linear and translation invariante (convolution) model: Z f (r 0 )h(r − r 0 ) dr 0 + ²(r) = h(r) ∗ f (r) + ²(r) g(r) = R0
• Discretized version g = Hf + ² where g = {g(r), r ∈ R}, ² = {²(r), r ∈ R} and f = {f (r 0 ), r 0 ∈ R0 }
&
3
%
'
Single channel image restoration
$
²(x, y)
? f (x, y) -
h(x, y)
- +
Observation model :
- g(x, y) = f (x, y) ∗ h(x, y) + ²(x, y)
g = Hf + ²
? ⇐=
&
4
%
'
$
Fourier synthesis in optical imaging g(ω) =
Z
£
t
¤
f (r) exp −jω r dr + ²(ω)
• Coherent imaging:
G(g) = g
−→
g = Hf + ²
• Non coherent imaging:
G(g) = |g|
−→
g = H(f ) + ²
g = {g(ω), ω ∈ Ω},
² = {²(ω), ω ∈ Ω}
?
20
40
20
40
⇐=
60
60
80
80
100
100
120
120 20
&
f = {f (r), r ∈ R}
and
40
60
80
100
120
20
5
40
60
80
100
120
%
'
Multi sensor image processing problems
$
• Disjoint multi sensors system: g i = H i f i + ²i ,
i = 1, · · · , M
• General multi input multi output (MIMO) system: N X H ij f j + ²i , i = 1, · · · , M gi = j=1
• General unknown mixing gain MIMO system: N X Aij H j f j + ²i , i = 1, · · · , M gi = j=1
• Blind Sources Separation (BSS) problem: N X Aij f j + ²i , i = 1, · · · , M gi = j=1
where A = {Aij , i = 1, · · · , M, j = 1, · · · , N } is an unknown mixing matrix.
&
6
%
'
$
Multi-spectral image deconvolution ²i (x, y) fi (x, y) -
h(x, y)
Observation model :
? - + -gi (x, y) = fi (x, y) ∗ h(x, y) + ²i (x, y)
g i = Hf i + ²i ,
i = 1, 2, 3
? ⇐= &
7
%
'
Image fusion and joint segmentation
$
Fusion ? =⇒
gi (r) = fi (r) + ²i (r), g(r) = {gi (r), i = 1, M },
g i = {gi (r), r ∈ R},
g(r) = f (r) + ²(r), &
i = 1, · · · , M
8
g = {g i (r), i = 1, M }
g =f +²
%
'
Blind image separation and joint segmentation
gi (r) = f1 (r)
j=1
Aij fj (r) + ²i (r)
g(r) = {gi (r), i = 1, M }
? f2 (r)
PN
$
g1 (r)
g(r) = Af (r) + ²(r), g = {g i (r), i = 1, M }
Separation
g i = {gi (r), r ∈ R},
⇐= g2 (r) f3 (r)
&
9
g = Af + ²
%
'
$
&
%
Segmentation of hyperspectral images
10
'
$
Segmentation of hyperspectral images 50
50
50
50
50
50
100
100
100
100
100
100
150
150
150
150
150
150
200
200
200
200
200
200
250
250
250
250
250
250
300
300
300
300
300
300
350
350
350
350
350
350
400
400
400
400
400
400
450
450
450
450
450
450
500
500
500
500
500
100
200
300
400
500
600
100
50
50
100
100
150
150
200
200
250
250
300
300
350
350
400
400
450
450
200
300
400
500
600
100
200
300
400
500
600
100
200
300
400
500
500
600
100
20000
20000
15000
15000
10000
10000
5000
5000
200
300
400
500
600
300
500 400
500
600
40
60
80
100
120
140
160
180
200
220
0
0 0
20
40
60
80
100
120
140
160
180
200
220
−5000
0
50
gi (r) = fi (r) + ²i (r), g(r) = {gi (r), i = 1, M }, &
600
1000
600 20
500
2000
500
300
400
3000
400
200
300
4000
200
100
200
5000
100
500
100
100
150
200
11
−5000
0
50
100
150
200
250
−1000
0
50
100
150
200
250
i = 1, · · · , M
g i = {gi (r), r ∈ R},
g(r) = f (r) + ²(r),
250
g = {g i (r), i = 1, M }
g =f +²
%
'
Blind image separation and joint segmentation
gi (r) = f1 (r)
j=1
Aij fj (r) + ²i (r)
g(r) = {gi (r), i = 1, M }
? f2 (r)
PN
$
g1 (r)
g(r) = Af (r) + ²(r), g = {g i (r), i = 1, M }
Separation
g i = {gi (r), r ∈ R},
⇐= g2 (r) f3 (r)
&
12
g = Af + ²
%
'
$
Bayesian estimation approach g = Af + ² • Forward model and prior knowledge on the noise • Prior knowledge on f • Bayes rule:
−→
−→
p(g|f )
p(f )
p(f |g) = p(g|f ) p(f ) / p(g) ∝ p(g|f ) p(f )
• Infer on f via p(f |g): – MAP estimator: © ª ª © b f = arg max p(f |g) = arg min − ln p(f |g) f f Z b = f p(f |g) df posterior mean – MSE estimator f
&
13
%
'
Bayesian blind sources separation (BBSS)
$
g = Af + ² • Likelihood
p(g|f , A, θ 1 )
• Priors
p(f |θ 2 ), p(A|θ 3 ) and p(θ)
hyperparameters θ = {θ 1 θ 2 , θ 3 }
• Bayes rule p(f , A, θ|g) ∝ p(g|f , A, θ 1 ) p(f |θ 2 ) p(A|θ 3 ) p(θ) • Generalized a posteriori EM-ML estimator o n ª © b = arg max p(f |g, A, b b = arg max p(A, θ|g) b θ) b θ) −→ f (A, (A , θ ) f • MSE estimator which corresponds to the posterior mean using MCMC Z b = {f b , A, b p(f , A, θ|g) d{f b , A, b b , A, b θ} b θ} b θ}. {f
&
14
%
'
$
HMM modeling of images What they share ?
50 20
100 40
150
Borders & Regions
200 60
250 80
300
350
100
Segmentation
400 120
450
500
140 20
40
60
80
100
120
140
160
180
100
200
300
400
500
600
Hidden variable z(r) z(r) = k, k = 1, · · · , K
50 20
100 40
Rk = {r : z(r) = k}, R = ∪k Rk
150
200 60
250 80
300
Homogeneity in regions
350
100
400 120
450
p(f (r)|z(r) = k) = N (mk , σk2 )
500
140 20
40
60
80
100
120
140
160
180
100
200
Rj k = {r : zj (r) = k},
300
400
500
600
R j = ∪ k Rj k
f j k = {fj (r) : r ∈ Rj k }, &
f j = ∪k f j k . Kj Kj Y Y p(fj (r), r ∈ Rj k ) = N (mj k 1, σj 2k ) p(fj (r), r ∈ R) = k=1
k=1
15
%
'
Main hypothesis:
k=4
$
k=3 k=2 k=1
• Pixels values in different regions of an image are independent. • For pixels values in a given region of an image, two possibilities: – i.i.d.:
p(fj (r)|zj (r) = k) = N (mj k , σj2 k ) p(fj (r), r ∈ Rj k ) = N (mj k 1, σj2 k I)
– Markovien:
k=4 k=3 k=2
k=1
k=4 k=2
k=3
k=1
p(fj (r), r ∈ Rj k ) = N (mj k 1, Σj k )
• For pixels values in different images but in a given common region two possibilities: – i.i.d.: – Markovien:
&
(fj (r)|z(r) = k) independent of (fi (r)|z(r) = k),
i 6= j
p(fj (r)|z(r) = k, f (r)) = p(fj (r)|z(r) = k, fj−1 (r)) 16
%
'
$
Modeling the labels p(fj (r)|zj (r) = k) =
N (mj k , σj2 k )
−→ p(fj (r)) =
X
P (zj (r) = k) N (mj k , σj2 k )
k
• Independent Gaussian Mixture model (IGM), where z j = {zj (r), r ∈ R} are assumed to be independent and X Y pk = 1 and p(z j ) = pk P (zj (r) = k) = pk , with k
k
• Contextual Gaussian Mixture model (CGM): z j Markovien X X δ(zj (r) − zj (s)) p(z j ) ∝ exp α r ∈R s∈V(r ) which is the Potts Markov random feild (PMRF). The parameter α controls the mean value of the regions’ sizes. & 17
%
'
Expressions of likelihood, prior and posterior laws N X
gi =
Ai,j f j + ²i ,
i = 1, · · · , M
−→
$
g = Af + ²
j=1
• Likelihood: p(g|A, f , θ 1 ) =
M Y
p(g i |A, f , Σ² i ) =
i=1
Σ² i = σ² 2i I,
−→
M Y
N (Af , Σ² i )
i=1
θ 1 = {σ² 2i , i = 1, · · · , M }
• HMM for the images: p(f |z, θ 2 ) =
N Y
p(f j |z j , mj , Σj )
j=1
where z = {z j , j = 1, · · · , N } and where we assumed that f j |z j are independent. −→ θ 2 = {(mi k , σj2 k ), j = 1, · · · , N }
&
18
%
'
$
• PMRF for the labels: p(z) ∝
N Y
j=1
exp α
X
X
r ∈R s∈V(r )
δ(zj (r) − zj (s))
• Conjugate priors for the hyperparameters θ = (θ 1 , θ 2 ): θ = {{σ² 2i , i = 1, · · · , M }, {(mi k , σj2 k ), j = 1, · · · , M, k = 1, · · · , K}} p(mj k )
= N (mj k 0 , σj2 k 0 )
p(σj2 k )
= IG(αj0 , βj0 )
p(Σj k )
= IW(αj0 , Λj0 )
p(σ² i )
= IG(αi0 , βi0 )
• Joint posterior law of f , z and θ &
p(f , z, θ|g) ∝ p(g|f , θ 1 ) p(f |z, θ 2 ) p(z|θ 2 ) p(θ) 19
%
'
General MCMC sampling scheme
$
p(f , z, θ|g) ∝ p(g|f , θ 1 ) p(f |z, θ 2 ) p(z|θ 2 ) p(θ) Gibbs sampling: • Generate samples (f , z, θ)(1) , · · · , (f , z, θ)(N ) using ∼ p(f |g, z, θ),
–
f
–
z ∼ p(z|g, f , θ),
–
θ ∼ p(θ|g, f , z),
• Compute any statistics such as mean, median, variance, ... Difficulties: • No mixture, No convolution: • Mixture but No convolution: • Convolution but No mixture: • Mixture and Convolution:
&
g i = f i + ²i , i = 1, · · · , M PN g i = j=1 Aij f j + ²i , i = 1, · · · , M
g i = H i f i + ²i , i = 1, · · · , M PN g i = j=1 Aij H ij f j + ²i , i = 1, · · · , M 20
%
'
$
Examples of applications • No mixture, No convolution applications: – Multi channel image fusion and joint segmentation g i = f i + ²i ,
i = 1, · · · , M,
f i |z independent
– Hyperspectral image segmentation g i = f i + ²i ,
i = 1, · · · , M,
f i |z dependent
– Video movie segmentation with motion estimation g i = f i + ²i ,
i = 1, · · · , M,
f i |z i independent
• Mixture, No convolution applications: – Blind source (image) separation (BSS) and joint segmentation PN g i = j=1 Aij f j + ²i , i = 1, · · · , M, f i |z i independent • Convolution but No mixture applications: – Fourier synthesis in optical imaging – Single channel image restoration &
21
g = H(f ) + ²,
f |z
g = Hf + ²,
f |z
%
'
$
Images fusion and joint segmentation ´ (Olivier FERON) gi (r) = fi (r) + ²i (r)
p(fi (r)|z(r) = k) = N (mi k , σi2 k ) Q p(f |z) = i p(f i |z)
g1
g2 &
−→
b f 1
b z
b f 2 22
%
'
Joint segmentation of hyper-spectral images
$
(Adel MOHAMMADPOUR) gi (r) = fi (r) + ²i (r), r ∈ R, or g i = f i + ²i , i = 1, · · · , M p(f (r)|z(r) = k) = N (m , σ 2 ), k = 1, · · · , K i ik ik Q p(f |z) = i p(f i |z) m follow a Markovian model along the index i ik
&
23
%
'
Segmentation of a video sequence of images
$
(Patrice BRAULT) gi (r) = fi (r) + ²i (r), r ∈ R, or g i = f i + ²i , or g = f + ² p(f (r)|z (r) = k) = N (m , σ 2 ), k = 1, · · · , K i i ik ik Q p(f |z) = i p(f i |z i ) z (r) follow a Markovian model along the index i i
&
24
%
'
Blind image separation and joint segmentation
$
N X gi (r) = Aij fj (r) + ²i (r) −→ g(r) = Af (r) + ²(r) −→ g = Af + ² j=1 Q p(g|f , A, Σ² ) = Q r ∈R p(g(r)|f (r), A) = r ∈R N (Af (r), Σ² ) 2 p(f (r)|z (r) = k) = N (m , σ ), j j j j k k Q Q p(f |z) = r ∈R j p(fj (r)|zj (r)) p(Aij ) = N (A0 ij , σ02 ij ) or p(vect(A)) = N (vect(A0 ), σ02 ij I) P b (r), Σ(r)) b p(f |g, z, θ, A) = N (f with ∈R r t −1 −1 −1 b Σ A + Σ (r)) and Σ(r) = (A z ² ¡ t −1 ¢ f −1 b (r) = Σ(r) b A Σ² g(r) + Σz (r) mz (r)
p(z(r)|g(r), θ, A) ∝ (p(g(r)|z(r), θ, A)) p(z(r)) with t p(g(r)|z(r), θ) = N (Am , AΣ A + Σ² ) z( r ) z( r ) p(A, Σ |f , g, θ) = N (A; A , Σ ) W(Σ −1 ; ν , Σ ) p p p ² ² ²p
&
25
%
26
% &
$ '
Mixing
Segmentation
Separation
'
$
f &
b f
g
27
b z
%
'
Single channel image restoration g(r 0 ) =
Z
$
h(r 0 − r) f (r) dr + ²(r 0 ) −→ g = Hf + ²
Fourier synthesis inverse problem
g(ω) =
Z
exp [−j(ω.r)] f (r) dr + ²(ω) −→ g = Hf + ²
p(²) = N (0, Σ² ) −→ p(g|f ) = N (Hf , Σ² ) with Σ² = σ² 2 I p(f (r)|z(r) = k) = N (mk , σk2 ), b , Σ) b with p(f |z, θ, g) = N (f
k = 1, · · · , K
¡ t −1 ¢ t −1 −1 −1 −1 b b b and f = Σ H Σ² g + Σz mz Σ = (H Σ² H + Σz ) b = arg max {p(f |z, θ, g)} = arg min {J(f )} with Compute f f f 2 P kf k −mk 1k 1 2 J(f ) = σ2 kg − Hf k + k σ2
² k p(z|g, θ) ∝ p(g|z, θ) p(z) with t , Σ ) with Σ = HΣ H + Σ² p(g|z, θ) = N (Hm g g z z Use p(z|g, f , θ) ∝ p(g|f , z, θ) p(z) & 28
%
'
a)
e)
$
Simulation results
20
20
20
20
40
40
40
40
60
60
60
60
80
80
80
80
100
100
100
100
120 20
40
60
80
100
120
b)
120 20
40
60
80
100
120
c)
120 20
40
60
80
100
120
d)
120
20
20
20
20
40
40
40
40
60
60
60
60
80
80
80
80
100
100
100
100
120 20
40
60
80
100
120
f)
120 20
40
60
80
100
120
g)
120 20
40
60
80
100
120
h)
20
40
60
80
100
120
20
40
60
80
100
120
120
a) object, b) exact known support, c) support of the data, d) measured data, e) and f) Results when phase is measured: e) IFT and f) proposed method, g) and h) Results when the phase is not measured but we know the support of the object: g) by Gerchberg-Saxton h) by the proposed method. & 29
%
'
$
Wavelet domain Bayesian image processing g(r) = Af (r) + ²(r) −→ WT −→ g j (r) = Af j (r) + ²j (r) images
Hist. of images
Wavelet coeff.
300
Hist. of wavelet coeff. 2500
250 2000
200 1500
150
1000 100
500 50
0
0
50
100
150
200
250
0 −500
300
600
3000
500
2500
400
2000
300
1500
200
1000
100
500
0
0
50
100
150
200
250
0 −500
300
−400
−300
−200
−100
0
100
200
300
400
500
−400
−300
−200
−100
0
100
200
300
400
500
• multi-resolution computation • Wavelet coefficients can be classified and segmented in only K = 2 classes
&
30
%
'
$
f &
b f
g
31
(GM)
b f
(HMM)
%
'
Conclusion and works in progress
$
Bayesian approach & HMM are appropriate tools for many image processing problems • H. Snoussi : BSS in 1D and 2D either in pixel domain or Fourier transform domain • M. Ichir : BSS with mixture of Gamma and BSS in wavelet domain • S. Moussaoui : BSS for non negative sources with application in spectrometry • O. F´eron : Data and image fusion, Fourier synthesis and inverse problems in microwave imaging • P. Brault : Segmentation of images sequences either directly or in wavelet domain • A. Mohammadpour : Segmentation of hyper-spectral images, • Z. Chama : Image recovery from the Fourier phase (Fourier Synthesis) • F. Humblot : Obtaining a super-resolution image from a set of lower resolution images
&
32
%
