Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Visual Tracking
Content
T. Chateau 1
Introduction to Visual Tracking
2
Modelling Visual Tracking
3
Stochastic Filters for Visual Tracking
4
Some examples
Pascal Institute, UMR 6602, UBP/CNRS/IFMA
October 24, 2012
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Content
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Visual Tracking
1
Introduction to Visual Tracking What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
2
Modelling Visual Tracking
3
Stochastic Filters for Visual Tracking
4
Some examples T. Chateau
Blaise Pascal University
Definition Visual Tracking is the process of locating, identifying, and determining the dynamic configuration of one or many moving (possibly deformable) objects (or parts of objects) in each frame of one or several cameras Human equivalent Follow something with your eyes
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Visual Tracking (before beginning)
On-line and O↵-line Tracking
State Vector The dynamic configuration of the the tracked object at time k is modelled by a State vector denoted:
O↵-line Tracking (Deferred Tracking) Estimation of the state xk uses the entire observation sequence . Z = {zk }k=1,...,K
xk
Available Observations
State Sequence The state sequence is given by the set (sequence) of State vectors, denoted: . X = {xk }k=1,...,K
zk−3 zk−2
zk−1
zk
. Observation: Z = {zk }k=1,...,K
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
On-line and O↵-line Tracking
zk
zk+1
xk T. Chateau
Blaise Pascal University
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Delayed Tracking Estimation of the state needs current, past and a part (delay) of future observation
Available Observations
Available Observations
zk−1
zk+2
On-line and O↵-line Tracking
On-line Tracking Estimation of the state xk uses the current and past observation: z0:k
zk−3 zk−2
zk+1
xk
Observation
T. Chateau
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Blaise Pascal University
zk+2
zk−3 zk−2 xk−2
zk−1
zk
zk+1
delay
T. Chateau
Blaise Pascal University
zk+2
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
On-line and O↵-line Tracking
Why is Visual Tracking Difficult ?
On-line Tracking For robotic applications: estimation of the state xk uses the current and past observation: z0:k
Available Observations
zk−3 zk−2
zk−1
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
zk
zk+1
zk+2
Hidden State The state X is a hidden state and must be deduced from observation Tracking Challenges Object Modeling: how to define what an object is in terms that can be interpreted by a computer ? Appearance Change: The observation of an object changes according to many parameters (illumination conditions, occlusions, shape variation...)
xk T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Kinematic Modelling: How to inject priors on object kinematic and interactions between objects.
Blaise Pascal University
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Tracking Challenges: Object Modelling
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Tracking Challenges: object description Example 1
Generic, discriminative model Build a visual description of the object: Generic enough to encode the entire variability of the object Discriminative enough to separate the object into the images (cluttered background)
T. Chateau
Blaise Pascal University
z1 z . 2 z= z3 ... zN
The object is a pattern:
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Tracking Challenges: object description Example 2 Generic object tracking (example: vehicle tracking):
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Tracking Challenges: object description Example 3 color based tracking:
R
G
B
RGB (marginalized histogram)
The Image is described with a vector of selected ”Haar-like” wavelets T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
The Image is described with a color histogram
Blaise Pascal University
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
What is Visual Tracking ? On-line and O↵-line Tracking Why is Visual Tracking Difficult ?
Tracking Challenges: Appearance Variation Several Illumination Conditions and poses
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Content 1
Introduction to Visual Tracking
2
Modelling Visual Tracking My First Tracker The Toy Example Detection vs Tracking
Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
T. Chateau
Blaise Pascal University
3
Stochastic Filters for Visual Tracking
4
Some examples T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
The Toy Example
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
The Toy Example
Object Tracking from a single static camera Estimation of the 2D position of a moving object
State Vector . xk = {xk , yk } , the position of the gravity center of the object (into the image reference plane) Observation function Based on a di↵erence image : f Compute di↵. image: Idif = Iref Ik k k ref ref Update Ref. image: Ik+1 = ↵.I + (1
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
The Toy Example
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
↵).Ik
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Tracking by Detection Tracking by Detection The state xk at time k depends only on the observation zk at time k
Example of di↵. image
zk
xk
T. Chateau
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Tracking by Detection
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Tracking by Detection
Application to the toy example xk = f (zk ) where f is a function given the position of the foreground pixel which has the most moving neighbours pixels (clustering method not developed here)
Conclusion Tracking by detection needs a function xk = f (zk ) No prior on motion between two images is injected into the algorithm
Matlab demonstration
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Injecting priors on motion
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Injecting priors on motion
Assumption We know (prior) a bound on the relative motion between images Ik 1 and Ik . Consequence The state xk at time k depends only on observation zk at time k and the previous state xk 1 : xk = f (zk , xk zk−3 zk−2
zk−1
zk
The observation function is reduced to a Region of Interest (ROI) around the previous estimated state. Matlab demonstration
1)
zk+1
Application to the toy example
zk+2
xk−1 xk T. Chateau
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Injecting priors on motion
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Classification of Visual Tracking approaches Non-Probabilistic vs Probabilistic Approaches
Conclusion Injecting priors on motion reduces the search state space
Visual Tracking Approaches Non Probabilistic Approaches
The resulting solution is a basic ”tracker”
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Optimisation based approaches
Parametric based Approaches
Other Approaches
Stochastic based Approaches
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Tracking as an optimisation problem
State The State vector is an unknown parameter vector which can be estimated using optimisation techniques :
Probabilistic Approaches
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
example of Tracking solutions using optimisation techniques MeanShift, Comaniciu
ˆ k = arg min E(xk , zk ) x xk 2X
The search space X is often reduced using priors on motion and previous estimation.
T. Chateau
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
example of Tracking solutions using optimisation techniques Localisation, Royer (Lasmea)
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
example of Tracking solutions using optimisation techniques Localisation, Royer (Lasmea)
Manual control
1) Learning step Training video 2) Autonomous control
3D Visual map Realtime localisation
Robot localisation
Current frame Autonomous navigation
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
example of Tracking solutions using optimisation techniques Localisation, Royer (Lasmea)
T. Chateau
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
example of Tracking solutions using optimisation techniques Localisation, Royer (Lasmea)
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
example of Tracking solutions using optimisation techniques
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
example of Tracking solutions using optimisation techniques
Localisation, Royer (Lasmea) Localisation, Royer (Lasmea)
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
example of Tracking solutions using optimisation techniques ESM, Malis, INRIA
T. Chateau
Blaise Pascal University
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Tracking : Other non-probabilistic approaches Using Machine Learning, Williams, RVM, Relevance Vector Machine
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Tracking : Other non-probabilistic approaches Tracking by Detection, V. Lepetit, EFPL
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Probabilistic Approaches to Visual Tracking Random Vectors Both the state X and the observation Z are random vectors: X 2 X and Z 2 Z Joint Probability The Probability of a sate sequence is given by: p(X|Z) = p(x1 ; x2 ; ...; xK |z1 ; z2 ; ...; zK ) ˆ The final output of a Visual Tracking process is an estimate X
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
T. Chateau
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
The Recursive Bayesian Estimation Approach Dynamic Bayesian Network representation First order Markovian assumption: the object configuration at time k, xk , depends only on the previous state Xk 1 .
xk−3 xk−2
xk−1
xk
xk+1
xk+2
States
. X = {xk }k=1,...,K
zk−3 zk−2
zk−1
zk
T. Chateau
zk+1
Observations
zk+2
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Recursive state-space Bayesian estimation approach
Posterior distribution The belief about the current state xk is expressed by a probability distribution: p(xk |zk ): POSTERIOR DISTRIBUTION How to recursively compute p(xk |zk )?
. Z = {zk }k=1,...,K
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
computing p(xk |zk ) from p(xk 1 |zk 1 )
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
computing p(xk |zk ) from p(xk 1 |zk 1 )
A two steps algorithm
Prediction step (dynamical model)
p(xk−1 |zk−1 )
p(xk |zk−1 ) Prediction (Chapman Kolmogorov)
p(xk |zk ) Update (Bayes)
Chapman-Kolmogorov equation: Z p(xk |z1:k 1 ) = p(xk |xk p(xk−1 |zk−1 )
1 )p(xk 1 |z1:k 1 )dxk 1
p(xk |zk−1 ) Prediction (Chapman Kolmogorov)
p(xk |xk−1 )
p(xk |zk ) Update (Bayes)
p(zk |xk ) p(xk |xk−1 )
p(zk |xk )
Dynamics
Dynamics
Likelihood
Likelihood
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
T. Chateau
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
computing p(xk |zk ) from p(xk 1 |zk 1 )
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
computing p(xk |zk ) from p(xk 1 |zk 1 )
Update step Bayes theorem:
Recursive Bayesian filtering distribution p(zk |xk )p(xk |z1:k p(xk |z1:k ) = p(zk |z1:k 1 )
with : p(zk |z1:k
1)
=
Z
1)
p(xk |z1:k ) = C
1
p(zk |xk )
Z
xk
p(xk−1 |zk−1 )
p(zk |xk )p(xk |z1:k
p(xk−1 |zk−1 )
p(xk |zk−1 ) Prediction (Chapman Kolmogorov)
1 )dxk
p(xk |zk )
T. Chateau Dynamics
p(xk |xk
p(xk |zk−1 ) Prediction (Chapman Kolmogorov)
p(xk |xk−1 )
1 )p(xk 1 |z1:k 1 )dxk 1 p(xk |zk )
Update (Bayes)
p(zk |xk )
Update (Bayes) Dynamics
p(xk |xk−1 )
1
Likelihood
p(zk |xk ) Blaise Pascal University Likelihood
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
computing p(xk |zk ) from p(xk 1 |zk 1 )
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Modelling pdf
Partial Conclusion The recursive bayesian filtering distribution provides an efficient solution to compute the posterior at time k (p(xk |zk )) from the posterior at time k 1 (p(xk 1 |zk the dynamic model (p(xk |xk 1 )), and the likelihood (p(zk |xk ))
Parametric and stochastic models
1 ))
,
Operations (integrals, products) on pdf have to be done:
Question how to define probabilities such that operations like product and integration become tractable ? T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
p(xk |zk−1 )
p(xk |xk−1 ) Dynamics
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Stochastic models (Particle filters,...) Particle filters All pdf are approximated by a set of samples.
Kalman filter Assumption: all pdf are modelized with Gaussian
Prediction (Chapman Kolmogorov)
Stochastic approximation of the pdf
T. Chateau
Parametric models (Kalman,...)
p(xk−1 |zk−1 )
Parametric function (Gaussian)
p(xk |zk ) Update (Bayes)
p(zk |xk ) Likelihood
Blaise Pascal University
p(xk−1 |zk−1 )
p(xk |zk−1 ) Prediction (Chapman Kolmogorov)
p(xk |xk−1 ) Dynamics
T. Chateau
p(xk |zk ) Update (Bayes)
p(zk |xk ) Likelihood
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
My First Tracker Classification of Visual Tracking approaches Overview of non probabilistic methods Probabilistic Approaches to Visual Tracking
Probabilistic filters
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
Content
Partial conclusion Kalman filters and derived: we assume that the unknown pdf can be modelized by a parametric function Stochastic solutions: approximation of the pdf by a set of particles.
1
Introduction to Visual Tracking
2
Modelling Visual Tracking
3
Stochastic Filters for Visual Tracking Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
4
Some examples
Next Stochastic approaches to bayesian filter are developed in the next section
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
Approximating pdf(s)
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
Approximating pdf(s) Set of weighted particles model
Set of particles model p(x) ⇡ {xn }n=1,..,N p(x) ⇡
N X
(x
x ) n
n=1
T. Chateau
Blaise Pascal University
p(x) ⇡ {xn , ⇡ n }n=1,..,N p(x) ⇡
N X
⇡ n (x
xn )
n=1
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
SIR (algo. and matlab simulation) (1996)
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
SIR: Conclusion
Condensation Algorithm 0
Init: particles {(x0n , 1/N )}N n=1 according to the initial distribution x0 for k = 1, ..., Kend do Prediction : generation of {(xnk , 1/N )}N n=1 from 0 p(xk |Xk 1 = xkn 1 ) Observation : estimation of the weight vector according to the image {(xnk , ⇡ nk )}N with ⇡ nk / p(zk |xk = xnk ) n=1 0n n n N Sampling : build {(xk 1 , 1/N )}N n=1 from {(x0 , ⇡ 0 )}n=1 using Importance Sampling P . n ˆ k = N1 N Estimation : x n=1 xk end for Output: Estimated state sequence {ˆ xk }k=1,...,Kend T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
MCMC (Markov Chain Monte Carlo)
Conclusion Particle filters approximate non gaussian pdf Condensation is a ”parallel” algorithm. The power of exploration is conditioned by an efficient sampling (many sampling strategies have been proposed) matlab illustration (SIR, CONDENSATION)
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
MCMC (Markov Chain Monte Carlo)
Examples Method: build a chained set of particles (Markov Chain Monte Carlo)
T. Chateau
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
MCMC (Markov Chain Monte Carlo) Method: build a chained set of particles (Markov Chain Monte Carlo)
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
MCMC: Conclusion
Conclusion MCMC approximate non gaussian pdf MCMC are sequential algorithms. Efficient sampling strategies based on partionned sampling can be proposed MCMC are used in for high dimensional tracking problems
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
T. Chateau
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
Multi-object visual tracking MOT
Challenges The state vector has a variable dimension: The exploration process must jump from one dimension to an other.
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
RJMCMC
RJMCMC The size of the state vector is variable according to the pdf associated to the number of objects position updating proposals,
One solution: RJMCMC Reversible Jump Monte-Carlo Markov Chain is a solution to track a varying number of objects
T. Chateau
Blaise Pascal University
dimension move proposals (add an object, remove an object)
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Estimating pdf(s) with a set of samples The SIR algorithm The MCMC (Markov Chain Monte Carlo) algorithm Multi-object visual tracking (MOT)
Conclusion
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Content
Conclusion STATE MODEL
TRACKING ENGINE (KALMAN ,SIR, MCMC)
LIKELIHOOD FUNCTON (OBSERVATI ON)
KINEMATIC MODEL
1
Introduction to Visual Tracking
2
Modelling Visual Tracking
3
Stochastic Filters for Visual Tracking
4
Some examples Tracking a vehicle from a static camera Context Solution
Multi-object tracking Context Solution
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
What do we want to do ?
We want to estimate velocity and steering angle of the vehicle.
Bicycle kinematic model
Visual tracking of a vehicle from a static camera The dynamic model of the object is known. We want to estimate velocity and steering angle of the vehicle. L
L
Car
T. Chateau
Context Solution Tracking with Classifiers Introduction to Visual Tracking Tracking a vehicle from a static camera The classifier Modelling Visual Tracking Multi-object tracking Stochastic Filters for Visual Tracking Tracking with classifiers Temporal Tracking Some examples Results
What do we want to do ?
Visual tracking of a vehicle from a static camera The dynamic model of the object is known.
L
Tracking with classifiers T. Chateau Blaise Pascal University
δ
Blaise Pascal University
L
Car Bicycle kinematic model
T. Chateau
δ
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Tracking scheme
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Kinematic model Bicycle model L
Pt : position (size 2) STATE MODEL
βt : orientation δt: steering angle vt: velocity
Car Bicycle kinematic model
BACKGROUND/FOREGROUND
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
KINEMATIC MODEL
v: velocity x, y : position
BICYCLE MODEL
: orientation
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Observation function
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Observation function
T. Chateau
δ
x˙ = v. cos y˙ = v. sin ˙ = v . tan L
TRACKING ENGINE SIR
LIKELIHOOD FUNCTON (OBSERVATI ON)
L
Blaise Pascal University
T. Chateau
Blaise Pascal University
(1)
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Observation function
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Results Results
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Results
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Pr´ecision (cm) Speed km/hr 40 60 80
Results
T. Chateau
Blaise Pascal University
Vision ave/std 0.25/0.18 0.19/0.16 0.18/0.15
Rangefinder ave/std 0.65/0.54 0.72/0.67 0.33/0.22
T. Chateau
Sensor merge ave/std 0.17/0.10 0.09/0.06 0.14/0.10
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Multi-object tracking
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
What do we want to do ?
Visual tracking of a varying number of objects real time tracking and identification of a variable number of objects in 3D,
Multi-object tracking
no prior knowledge of objects appearance, robustness to partial and heavy occlusions , robustness to heavy scale changes, no prior knowledge of object entrance locations.
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
What do we want to do ?
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
What do we want to do ?
Visual tracking of a varying number of objects real time tracking and identification of a variable number of objects in 3D,
Visual tracking of a varying number of objects real time tracking and identification of a variable number of objects in 3D,
no prior knowledge of objects appearance,
no prior knowledge of objects appearance,
robustness to partial and heavy occlusions ,
robustness to partial and heavy occlusions ,
robustness to heavy scale changes,
robustness to heavy scale changes,
no prior knowledge of object entrance locations.
no prior knowledge of object entrance locations.
T. Chateau
T. Chateau
Blaise Pascal University
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
What do we want to do ?
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
What do we want to do ?
Visual tracking of a varying number of objects real time tracking and identification of a variable number of objects in 3D,
Visual tracking of a varying number of objects real time tracking and identification of a variable number of objects in 3D,
no prior knowledge of objects appearance,
no prior knowledge of objects appearance,
robustness to partial and heavy occlusions ,
robustness to partial and heavy occlusions ,
robustness to heavy scale changes,
robustness to heavy scale changes,
no prior knowledge of object entrance locations.
no prior knowledge of object entrance locations.
T. Chateau
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Tracking scheme
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Proposals . Xk = {n, x2k , x3k , ..., xnk } xk : state of one object STATE MODEL
RJMCMC Proposals Enter: add an object (data driven) Leave: remove an object (data driven)
TRACKING ENGINE RJMCMC
Object position Update: choose and object and propose a spatial move associated to this object.
LIKELIHOOD FUNCTON (OBSERVATI ON)
KINEMATIC MODEL
BACKGROUND/FOREGROUND
Several solutions
T. Chateau
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Results
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Tracking with classifiers
Tracking with classifiers
Demonstration
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
What do we want to do ?
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
What do we want to do ?
Visual tracking of an object class from a moving camera We want to recognize and track a moving object from a moving monocular camera, at realtime (30fps). We want to track and recognize an object using only a generic model.
T. Chateau
T. Chateau
Blaise Pascal University
Visual tracking of an object class from a moving camera We want to recognize and track a moving object from a moving monocular camera, at realtime (30fps). We want to track and recognize an object using only a generic model.
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Tracking scheme
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Why is it a challenge ? A moving object C: position (size 2) s: scale factor
Variation of the appearence of the object (3D object)
STATE MODEL
Variation of the illumination conditions
TRACKING ENGINE SIR
LIKELIHOOD FUNCTON (OBSERVATI ON)
KINEMATIC MODEL
ADABOOST CLASSIFIER
Cluttered background
AR Model
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Why is it a challenge ?
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Why is it a challenge ?
A moving camera Background/Foreground segmentation can not be done easily.
Recognize an object from a generic model This is an object recognition problem
?
Difficult with a moving camera
T. Chateau
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Why is it a challenge ?
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
A solution ? using classifiers into a probabilistic tracking framework
Track an object at a realtime framerate This is a realtime tracking problem
Bring together recent classifiers (Adaboost, SVM) and a particle filter Recent Classifier SVM or AdaBoost
Probabilistic tracking framework Particle Filter
a Classifier based likelihood Observation function Particle Filter (COPF)
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Using Classifiers for Real-Time Tracking: why ? O✏ine learning of the object to be tracked Trackers can be designed to track categories of objects (pedestrians, vehicles),
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Using Classifiers for Real-Time Tracking: why ? O✏ine learning of the object to be tracked Objects are modelized with a collection of views, representing variation of the object appearance,
STATISTICAL LEARNING
T. Chateau
Blaise Pascal University
STATISTICAL LEARNING
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Overview of the method
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Overview of the method t=0
Initialisation where is the object in the first frame ?
Initialisation where is the object in the first frame ?
t++
t++
CLASSIFIER
Prediction Predict the new object position
Prediction Predict the new object position
Observation Update the object position using a likelihood function based on classifier
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
The Observation Function
output of a classifier m(f )
output of a classifier m(f )
Assumption : m(f ) 2]
Assumption : m(f ) 2]
1; +1[ with
m(f 1 ) < m(f 2 ) ! P (class|f 1 ) < P (class|f 2 )
?
1 1 + exp(A.m(f ) + B) Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
1; +1[ with
m(f 1 ) < m(f 2 ) ! P (class|f 1 ) < P (class|f 2 )
Platt scaling, 1999 Estimate a sigmoid, from a learning database in order to produce calibrated probabilities from the output of the classifier :
T. Chateau
CLASSIFIER
Particle FilterBlaise Pascal University T. Chateau
The Observation Function
P (positive|m(f )) =
?
Observation Update the object position using a likelihood function based on classifier
Particle FilterBlaise Pascal University T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Convert to probabilities
t=0
(2)
Platt scaling, 1999 Estimate a sigmoid, from a learning database in order to produce calibrated probabilities from the output of the classifier : P (positive|m(f )) =
T. Chateau
1 1 + exp(A.m(f ) + B) Blaise Pascal University
(2)
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Platt scaling
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Platt scaling
Estimation of the sigmoid parameters A et B are two parameters to be estimated from a learning database (mi , yi ) with (yi 2 {0; 1})
Estimation of the sigmoid parameters A et B are two parameters to be estimated from a learning database (mi , yi ) with (yi 2 {0; 1})
non-linear estimation minimization of the cross-entropy error function: X argmin(A,B) { yi log(pi ) + (1 yi ) log(1
non-linear estimation minimization of the cross-entropy error function: X argmin(A,B) { yi log(pi ) + (1 yi ) log(1
pi )},
(3)
i
with
with
1 pi = 1 + exp(A.mi + B) T. Chateau
1 1 + exp(A.mi + B)
T. Chateau
0.25
Neg. Pos.
p1
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Neg. Pos.
p1 0.2
Haar based wavelets
0.5 0.15
0.4
+1 0.3
+1
0.1
+1
0.2 0.05
-10
0
10
20
SVM score (margin)
30
0.9
0.9
0.8
0.8
0.7
0.7
p(y=1|x)
1
0.6 0.5 0.4
0.3
0.3
0.2
0.2
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Adaboost score
0.9
p(y=1|x)
-20
0 0.1
1
-1
+1
+1
-1
-1
+1
-1
-20
-10
0
10
20
30
0 0
SVM score (margin)
T. Chateau
A Window of interest is descibed by a vector of the output of five filters applied to several positions and scales. Large number of features example: For 3 scales and a 128 ⇥ 64 pixels image, the number of features is about 40000.
0.1
0.1 0 -30
+1
+1
0 -30
0.4
-1 -1
0.1
0.5
(4)
Features
0.7
0.6
(3)
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Results 0.6
pi =
(4)
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
pi )},
i
0.2
0.4
0.6
Adaboost score
0.8
Blaise Pascal University
1
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Features
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
features reduction
Haar based wavelets +1 +1 +1
-1
+1
-1
+1
+1
-1
-1
+1
-1
-1 +1
A Window of interest is descibed by a vector of the output of five filters applied to several positions and scales. Large number of features example: For 3 scales and a 128 ⇥ 64 pixels image, the number of features is about 40000.
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
features reduction
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
features reduction
Using Adaboost Weak Classifier: one threshold for each feature
Using Adaboost Weak Classifier: one threshold for each feature
Evaluation Fonction: minimization of the number of samples assigned to the bad class
Evaluation Fonction: minimization of the number of samples assigned to the bad class
Stopping criteria: number of features to retain (rounds of boosting)
Stopping criteria: number of features to retain (rounds of boosting)
T. Chateau
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
features reduction
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Some of the selected features
Using Adaboost Weak Classifier: one threshold for each feature Evaluation Fonction: minimization of the number of samples assigned to the bad class Stopping criteria: number of features to retain (rounds of boosting)
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Result for a recognition step (Adaboost)
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Result for a recognition step (Adaboost) 1 0.9 feature vector dimension 50 75 100
0.8 0.7 0.6 0.5
-1
+1
...
...
0.4 0.3 0.2 0.1 00
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Roc curve for three different sizes of feature vector Using a pedestrian database with 2000 samples for training and 2000 samples for testing l
T. Chateau
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Classifier score evolution near the true position of the object
. Particle filter : p(Xt |Z0:t ) = {(Xnt , ⇡tn )}N n=1 State vector . Xt = (ct , ct
0.8
5
0.6 0.4 10
-10 -10 -5 0 5 horizontal trans. (pixels)
Xt+1 = AXt + Bvt , vt ⇠ N (0, ⌃)
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
(6)
Likelihood function
10
P (Zt |Xt = Xnt ) = T. Chateau
(5)
Dynamics
0 -5
0.2 -10 -5 0 5 horizontal trans. (pixels)
1 , st , st 1 )
. with c = (x, y) location of a bounding box whitin the image and associated scale factor st
SVM 10
Margin
Score
Adaboost 1
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Blaise Pascal University
1 n o ⇤ (cn + sn W)) + B ˆ ˆ 1 + exp A.m(F t t T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
. Particle filter : p(Xt |Z0:t ) = {(Xnt , ⇡tn )}N n=1
(7)
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
. Particle filter : p(Xt |Z0:t ) = {(Xnt , ⇡tn )}N n=1
State vector
State vector . Xt = (ct , ct
1 , st , st 1 )
. Xt = (ct , ct
(5)
1 , st , st 1 )
(5)
. with c = (x, y) location of a bounding box whitin the image and associated scale factor st
. with c = (x, y) location of a bounding box whitin the image and associated scale factor st
Dynamics
Dynamics Xt+1 = AXt + Bvt , vt ⇠ N (0, ⌃)
(6)
Likelihood function P (Zt |Xt = Xnt ) =
Xt+1 = AXt + Bvt , vt ⇠ N (0, ⌃)
(6)
Likelihood function 1 n o ⇤ (cn + sn W)) + B ˆ ˆ 1 + exp A.m(F t t T. Chateau
Blaise Pascal University
(7)
P (Zt |Xt = Xnt ) =
1 n o ⇤ (cn + sn W)) + B ˆ ˆ 1 + exp A.m(F t t T. Chateau
Blaise Pascal University
(7)
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
100 80 60 0
50
100 150 #image (a)
200
250
SVM Adaboost
120 100 80 60 0
100 #image (b)
200
(c) Im#001
(d) Im#261
im #010
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
im #020 The number of gravity center of particles is restricted the set of particles
Blaise Pascal University
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Tracking a generic object (videos)
-150 -100 -50 0 50 horizontal position
(b)
T. Chateau
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Tracking a generic object: occlusion (videos)
(a)
STATISTICAL LEARNING
Blaise Pascal University
(c)
(e)
(g)
100-150 -100 -50 0 50 horizontal position
(d)
T. Chateau
100
p(Xt |Zt )
some particles
p(Xt |Zt )
im #001
140 SVM Adaboost
vertical position
1/10 particles displayed 600 particles 100 particles
120
p(Xt | Zt )
3
The classifier is called for a set of regular positions and scales whitin the image A set of particles is initialized from the positions ans scales associated to the highest outputs of the classifier The number of particles is restricted when the particle distribution is around the object to be tracked.
p(Xt |Zt )
2
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Comparing SVM and Adaboost classifiers
horizontal position
Initialization 1
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
-150 -100 -50 0 50 horizontal position
100
(f)
Blaise Pascal University
-150 -100
-50 0 50 horizontal position
(h)
100
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Tracking a generic object: occlusion (videos)
(a) (36.2s)
(b) (37.0s)
(c) (37.5s)
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Tracking a specific object (videos)
(d) (38.5s)
STATISTICAL LEARNING
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Parallel implementation: the Babylon Project
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
The Babylon Project
Firefire 400Mbps dedicated for video broadcasting
1 node with 2 CPU
Ethernet switch Gigabit dedicated to nodes communications
Node: XServe PPC G5 Bi-processor SIMD achived by the AltiVec Extention T. Chateau
Blaise Pascal University
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Parallel implementation
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Parallel implementation
N2 Time consuming
Prediction
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Observation
N1
Prediction
Resampling
One iteration
One iteration
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Parallel implementation
T. Chateau
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Blaise Pascal University
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Parallel implementation
N1
N1
N2 N1
wi
N2 N1
N1
N14 Prediction
Observation
One iteration
T. Chateau
N14 Prediction
Observation
One iteration
Blaise Pascal University
T. Chateau
Blaise Pascal University
Resampling
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
Parallel implementation
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
3D Tracking
N1
wi
N2 N1
N1
N14 Observation
Prediction
Resampling
One iteration
T. Chateau
Blaise Pascal University
Introduction to Visual Tracking Modelling Visual Tracking Stochastic Filters for Visual Tracking Some examples
T. Chateau
Tracking a vehicle from a static camera Multi-object tracking Tracking with classifiers
3D Tracking
Conclusion STATE MODEL
Performances Seq. Par. Gain
200 0.0609s 16.40 0.0231s 43.31 ⇥2.7
Blaise Pascal University
500 0.1439s 6.95 0.0265s 37.72 ⇥5.42
1000 0.2874s 3.48 0.0313s 31.9 ⇥9.16
5000 1.6393s 0.61 0.0858s 11.66 ⇥19.43
10000 3.8462s 0.26 0.1567s 6.38 ⇥24.5
20 FPS with 2000 particles . Linear evolution of performances according to the number of nodes. T. Chateau
Blaise Pascal University
TRACKING ENGINE (KALMAN ,SIR, MCMC)
LIKELIHOOD FUNCTON (OBSERVATI ON)
KINEMATIC MODEL
Thanks to F. Bardet, D. Ramadasan, C. Tournayre, G. Jacob, E. Royer,....
T. Chateau
Blaise Pascal University
