Particle Filters for Visual Tracking
T. Chateau, Pascal Institute, Clermont-Ferrand
1
mardi 29 janvier 13
Content •Particle filtering: a probabilistic framework •SIR particle filter •MCMC particle filter •RJMCMC particle filter
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
2
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Content •Particle filtering: a probabilistic framework •SIR particle filter •MCMC particle filter •RJMCMC particle filter
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
3
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Visual Tracking
y
bird view
State (hidden variable) x
Observation
Particle Filters for Visual Tracking mardi 29 janvier 13
y
x
T. Chateau
4
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Definitions Dynamic Bayesian Network representation
xk−3 xk−2
xk−1
xk
xk+1
xk+2
States
. X = {xk }k=1,...,K
zk−3 zk−2
Particle Filters for Visual Tracking mardi 29 janvier 13
zk−1
zk
zk+1
Observations
zk+2
. Z = {zk }k=1,...,K
T. Chateau
5
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Online Tracking Online Tracking Observation
zk−3 zk−2 State
Particle Filters for Visual Tracking mardi 29 janvier 13
zk−1
zk
zk+1
zk+2
xk−1 xk
T. Chateau
6
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Online Tracking Probabilistic framework
p(Zk |Xk )
zk−3 zk−2 State
p(Xk
Particle Filters for Visual Tracking mardi 29 janvier 13
zk−1
zk
Observation
zk+1
zk+2
xk−1 xk 1 |Z0:k 1 )
p(Xk |Z0:k )
T. Chateau
7
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Sequential Monte-Carlo Inference
p(xk−1 |zk−1 )
p(xk |zk−1 ) Prediction (Chapman Kolmogorov)
p(xk |xk−1 ) Dynamics
Particle Filters for Visual Tracking mardi 29 janvier 13
p(xk |zk ) Update (Bayes)
p(zk |xk ) Likelihood
T. Chateau
8
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Sequential Monte-Carlo Inference
p(xk−1 |zk−1 )
p(xk |zk−1 ) Prediction (Chapman Kolmogorov)
p(xk |xk−1 ) Dynamics
p(xk |zk ) Update (Bayes)
p(zk |xk ) Likelihood
Stochastic representation of distributions
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
9
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Stochastic representation of distributions With a set of particles N 1 X p(X) ⇡ (X N n=1
Xn )
With a set of weighted particles p(X) ⇡
Particle Filters for Visual Tracking mardi 29 janvier 13
N X
⇡ n (X
Xn )
n=1
T. Chateau
10
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Visual Tracking
y
bird view
State (hidden variable) x
Observation y
Particle Filters for Visual Tracking mardi 29 janvier 13
x
T. Chateau
11
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Content •Particle filtering: a probabilistic framework •SIR particle filter •MCMC particle filter •RJMCMC particle filter
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
12
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
SIR Particle Filter
Sampling Importance Resampling
y
x
State distribution at time t-1
Resampling
y
x
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
13
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
SIR Particle Filter
State distribution at time t-1
Prediction
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
14
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
SIR Particle Filter
Predicted distribution at time t
Update
Posterior
S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp. A tutorial on particle filters for on-line non-linear/ non-gaussian bayesian tracking. IEEE Transactions on Signal Processing, 50(2):174–188, Feb. 2002. Particle Filters for Visual Tracking
mardi 29 janvier 13
T. Chateau
15
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
SIR Particle Filter t=1
t=2
t
Target t=3
Particle Filters for Visual Tracking mardi 29 janvier 13
Temporal Filtering
t=4
T. Chateau
16
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
SIR Particle Filter: some examples
• State vector: 2D position and scale (image reference frame) • Dynamics: random step • Observation model: max. of gradients set of points T. Chateau and J. Lapresté. Robust real time tracking of a vehicle by image processing. In IEEE Intelligent Vehicles Symposium,, Parma, Italy, June 2004. Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
17
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
SIR Particle Filter: some examples
• State vector: 2D position and scale (image reference frame) • Dynamics: random step • Observation model: offline learning based model (Haar wavelets) T. Chateau, V. Gay-Belille, F. Chausse, and J. T. Lapresté. Real-time tracking with classifiers. In WDV WDV Workshop on Dynamical Vision at ECCV2006, Grazz, Austria, May 2006. Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
18
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
SIR Particle Filter: some examples
• State vector: 3D position, orientation, steering angle and velocity • Dynamics: driven by a bicycle model • Observation model: background/foreground subtraction, camera and laser range finder • Collaboration with LCPC Nantes
Y. Goyat, T. Chateau, and L. Trassoudaine. Tracking of vehicle trajectory by combining a camera and a laser rangefinder. Springer MVA : Machine Vision and Application, online, March 2009. Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
19
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
SIR Particle Filter: some examples
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
20
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Efficient implementation of SIR Particle Filters
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
21
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Efficient implementation of Particle Filters SIR Particle Filter
d
N /e Particle Filters for Visual Tracking mardi 29 janvier 13
N: number of particles d : size of the state vector T. Chateau
22
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Content •Particle filtering: a probabilistic framework •SIR particle filter •MCMC particle filter •RJMCMC particle filter
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
23
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter p(Xk
Particle Filters for Visual Tracking mardi 29 janvier 13
1 |Zk 1 )
T. Chateau
24
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter p(Xk
1 |Zk 1 )
a Dr
Particle Filters for Visual Tracking mardi 29 janvier 13
w
T. Chateau
24
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter p(Xk
1 |Zk 1 )
a Dr
w
e
v Mo
X0
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
24
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter Markov Chain Monte Carlo
X0
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
25
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter Markov Chain Monte Carlo
a Dr
w
X0
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
25
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter Markov Chain Monte Carlo
X⇤
Move
a Dr
w
X0
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
25
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter Markov Chain Monte Carlo
X⇤
Move
a Dr
w
X0
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
25
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter Markov Chain Monte Carlo
X⇤
Move
a Dr
w
X0
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
25
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter Markov Chain Monte Carlo
X⇤ X0
w
? Compare
Particle Filters for Visual Tracking mardi 29 janvier 13
Move
a Dr
T. Chateau
25
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter Markov Chain Monte Carlo
X⇤ X0
? Compare
a Dr
Move
w
X1
Accept X⇤ or copy X0
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
25
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter Markov Chain Monte Carlo
X⇤ X0
? Compare
a Dr
Move
w
X1
XN
Accept X⇤ or copy X0
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
25
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter Markov Chain Monte Carlo
X⇤ X0
? Compare
a Dr
Move
w
X1
XN
Accept X⇤ or copy X0
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
25
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
!
⇡ ⇤ ... ↵ = min 1, k ⇡t ...
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
!
⇡ ⇤ ... ↵ = min 1, k ⇡t ...
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
!
⇡ ⇤ ... ↵ = min 1, k ⇡t ...
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: marginal move
Metropolis Hasting rule
!
⇡ ⇤ ... ↵ = min 1, k ⇡t ...
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
26
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
MCMC Particle Filter: example
- State vector: 2D location, scale and combination parameters of observation modules (colour, texture, gradient, ...) - Dynamics: random step - Observation function: learning based (Adaboost) - In collaboration with Teb-online
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
27
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Used to track a varying number of objects
. It 1 Xt = {It , xt , ..., xt }
The state is defined into a joint space
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
28
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Reversible Jump Markov Chain Monte Carlo
Approximate the distribution with a variable size A pair of new proposals
X = {x1 , x2 , x3 }
X = {x1 , x2 }
Jump into a lower dimensional space
X = {x1 , x2 , x3 , x4 }
Jump into a higher dimensional space
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
29
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Proposals have to be add:
- object position update - add one object - remove one object
The state is defined into a joint space
Particle Filters for Visual Tracking mardi 29 janvier 13
Data driven
T. Chateau
30
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Add an object: a data driven proposal Background/foreground hypothesis
X⇤
New object position proposal map Background/foreground observation map Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
31
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Real time pedestrian tracking
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
32
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Real time vehicle tracking
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
33
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Simultaneous tracking and categorisation Proposals: - update object position - add/remove one object - update object category One geometric and kinematic model for each category Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
34
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Simultaneous tracking and categorisation Proposals: - update object position - add/remove one object - update object category One geometric and kinematic model for each category Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
34
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Simultaneous tracking and categorisation Proposals: - update object position - add/remove one object - update object category
Particle Filters for Visual Tracking mardi 29 janvier 13
category update proposal matrix
T. Chateau
35
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Simultaneous tracking and categorisation
F. Bardet, T. Chateau, and D. Ramadasan. Unifying real-time multi-vehicle tracking and categorization. In Intelligent Vehicle Symposium, volume 1, 2009. Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
36
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter
Simultaneous tracking, categorisation and context detection Proposals: - update object or sun position position - add/remove one object or sun - update object category
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
37
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter Simultaneous tracking and categorisation
F. Bardet, T. Chateau, and J. Lapresté. Illumination aware mcmc particle filter for long-term outdoor multi-object simultaneous tracking and classification. In ICCV 2009, International Conference on Computer Vision, Tokyo, Japan, 09 2009. Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
38
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
RJMCMC Particle Filter
Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
39
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Efficient implementation of MCMC Particle Filters Multi Proposal MCMC Particle Filter
X
n−1
1−α π n−1 α
n
X πn
X ∗ π∗
X
n−1
1−α π n−1
X1∗ π1∗
Xn n π α Xp∗ πp∗
XP∗ πP∗ p ∈ {1, ..., P } (a): single proposal MCMC
Particle Filters for Visual Tracking mardi 29 janvier 13
(b): P-proposal MCMC
T. Chateau
40
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Efficient implementation of Particle Filters MCMC1PF t=0
0.1
0.1
0.05
0.05
0.05
0
20
40
60
0
0.05
0.05
0.05
20
40
60
0
20
40
60
0
0.1
0.1
0.1
0.05
0.05
0.05
20
40
60
0
20
40
60
0
0.1
0.1
0.1
0.05
0.05
0.05
20
40
60
0
20
40
60
0
0.1
0.1
0.1
0.05
0.05
0.05
0
20
40
Particle Filters for Visual Tracking mardi 29 janvier 13
60
0.1
0
t=4
40
0.1
0
t=3
20
0.1
0
t=2
MCMC4PF
0.1
0
t=1
MCMC2PF
60
0
20
40
60
0
20
40
60
20
40
60
20
40
60
20
40
60
20
40
60
T. Chateau
41
A probabilistic framework
SIR particle filter
MCMC particle filter
RJMCMC particle filter
Conclusion Particle filters are widely used for temporal filtering applications They provide tools able to handle with non linear systems SIR particle filters have to be chosen for low dimensional problems MCMC particle filters with marginal proposal strategy are preferred for high dimensional problems RJMCMC particle filters can be used to manage states with a varying dimension Particle Filters for Visual Tracking mardi 29 janvier 13
T. Chateau
42
