Machine Vision and Applications (2003) 13: 201–212

Machine Vision and Applications c Springer-Verlag 2003


Motion compensated film restoration O. Buisson1 , S. Boukir2 , B. Besserer2 1 2

Institut National de l’Audiovisuel, 4 Av. de l’Europe, 94366 Bry sur Marne Cedex, France Laboratoire L3i, Universit´e de La Rochelle, Avenue Michel Cr´epeau, 17042 La Rochelle cedex 1, France; e-mail: [email protected], Tel.: +33-5-46458284, Fax: +33-5-46458242

Received: 9 July 2000 / Accepted: 13 January 2002

Abstract. Motion picture films are susceptible to local degradations such as dust spots. Other deteriorations are global such as intensity and spatial jitter. It is obvious that motion needs to be compensated for before the detection/correction of such local and dynamic defects. Therefore, we propose a hierarchical motion estimation method ideally suited for high resolution film sequences. This recursive block-based motion estimator relies on an adaptive search strategy and Radon projections to improve processing speed. The localization of dust particles then becomes straightforward. Thus, it is achieved by simple inter-frame differences between the current image and motion compensated successive and preceding frames. However, the detection of spatial and intensity jitter requires a specific process taking advantage of the high temporal correlation in the image sequence. In this paper, we present our motion compensation-based algorithms for removing dust spots, spatial and intensity jitter in degraded motion pictures. Experimental results are presented showing the usefulness of our motion estimator for film restoration at reasonable computational costs. Key words: Motion estimation – Block matching – Motion picture restoration – Dust spot removal – Spatial and intensity jitter removal

1 Introduction Motion picture films can suffer from a variety of degradations which reduce their usefulness. Until the 1950s, all films were made in Cellulose Nitrate which becomes unstable with time. Modern film is also subject to degradation due mainly to repeated projection or bad preservation. Film restoration is therefore seen to have useful applications in the entertainment market. Nowadays, traditional or computer-aided restoration of classic old films is very expensive, and takes several months for the treatment of just one movie. However, digital restoration lets us expect results that are beyond today’s limiCorrespondence to: S. Boukir

tations (such as automated processing, correction of photographed dust spots and scratches, removing large defects, etc.) in far less time than using traditional techniques [Mor95,Ros96,Sut96,Bui97b,Dec97b,Kok98,Cho98]. A large number of films are digitized for TV broadcasting, using telecine. Usually, the print goes through a wet gate and the low output resolution (720 × 576 in Europe) allows low-pass or median filtering [Alp90,Pit90,Vel90]. Dust spots and scratches will virtually disappear, and large artifacts are removed with specialized (and real-time for video resolution) hardware. Digital processing will be different if the frames are shot back again onto 35 mm film after processing. For motion picture film restoration, full quality digital processing on (2000×1500) images is needed over the whole footage. Therefore, processing power has to be cheap, processing time as short as possible, and the restoration process should be automated, to reduce exploitation costs. The classical techniques of digital image restoration [Ozk92,Bra95] cannot respect these constraints for a variety of reasons. They are mainly based on the video format, so if we apply them on 2K images, they will involve a high computational cost. Our digital restoration technique is a three-stage process. The first stage is a detection process, running without human interaction on the raw image sequence. Motion compensation is computed during this step, and relevant information stored. During the second step, an operator checks the results of the detection process. The last step is also automatic. According to parameters set up by the operator, the image sequence is processed again. Using image masks and data generated during the detection step, artifacts are removed using an efficient interpolation technique. In this paper, we present a motion estimation method ideally suited for the processing of large images [Bui97b,Bui97a]. This algorithm is the core of many of our detection algorithms: dust, spatial and intensity jitter detectors. We focus then on the treatment of these three defects, showing the usefulness of our motion compensation technique for film restoration. While the localization of dust particles becomes straightforward after motion compensation, the detection of spatial and intensity jitter requires a specific process that takes advantage of the high temporal correlation in the image sequence.

202

O. Buisson et al.: Motion compensated film restoration

2 Block-based motion estimation There are two main methods to estimate the optical flow of a noisy image sequence: • Pre-filter the noisy sequence and use a classical motion estimation such as block matching, regression, etc. [Sri85,Haa92]. • Develop a motion estimation algorithm which is robust to noise or image alteration [Bla92,Bab97]. We have chosen the first solution for three reasons: 1. It is difficult to know the real sensitivity to noise or to image alteration of a motion estimator. 2. If robustness is ensured, this is achieved with high computational costs, even at SIF resolution (352 × 288). 3. In a high resolution image sequence, large displacements may occur, up to 200 or 300 pixels. One of the best solutions to quickly estimate such motions is the use of a hierarchical structure [Bur81,Ana89,Ber92,Jol94]. The filtering process is then included in the construction of this image pyramid. Having organized image information in a hierarchical manner, a recursive block-matching technique is used to estimate the optical flow.

2.1 Hierarchical structure The basic idea is to create a pyramidal representation of an image [Ana89] using the following algorithm: if (x mod 2 = 0) and (y mod 2 = 0) then I l+1 (x/2, y/2, t) = f ∗ I l (x, y, t) fi where ∗ denotes the convolution operator and f is a given filter. I l is interpreted as a family of images, where l indicates the level of resolution (or scale). The larger l, the more blurred the original image I is, finally showing the larger structures in the image. Our hierarchical image structure is built using a low-pass filtering such that film grain and deteriorations disappear at higher levels of the pyramid. Indeed, such high spatial frequencies disturb the motion estimation process.

2.2 Hierarchical motion estimation Our method combines the principle of hierarchical motion estimation with a block-matching algorithm [Bui97b,Bui97a]. In first step, motion is estimated allowing only a coarse velocity field to result and, in lower hierarchical levels, the details of the vector field are calculated as relatively small update around the estimated vector resulting from the previous higher level. At each level, displacements are estimated using a recursive block-matching algorithm [Sri85]. For each pixel of the current grid, we search for the displacement vector which

yields the minimum value of the following criterion, assuming a linear variation of the luminance:
 E(p, d, t) = A(p).I(pi , t) + B(p) p ∈Wp i

  −I(pi + d(p), t + dt)

(1)

where Wp is a neighboring window of n × n pixels centered at pixel p, d(p) is the displacement of p from time t to t + dt and A(p), B(p) are the image brightness variation parameters at pixel p. Notice that intensity variations generally occur in large areas of the images, so they will be visible in the higher levels of the pyramid. Therefore, to speed up the motion estimation process, we compute these parameters only in the higher levels of the pyramid. In Gennert and Negahdaripour [Gen87], these coefficients are estimated for all the pixels of the image, thus involving high computational cost. More formally, our recursive search approach consists of the following steps.First, for each pixel p of level l, the brightness parameters Ali , Bil are estimated, at each iteration i, as follows:    V ar(I l (p + dli−1 (p), t + dt))  l Ai (p) = V ar(I l (p, t))   l l l Bi (p) = Ai (p).µ(I (p, t)) − µ(I l (p + dli−1 (p), t + dt)) (2) where µ(I l (p, t)), V ar(I l (p, t)) (respectively, µ(I l (p + dli−1 (p), t + dt)), V ar(I l (p + dli−1 (p), t + dt))) represent the mean and variance of the intensities of m × m pixels centered at pixel p of image I l (t) (respectively, at pixel p + dli−1 (p) of image I l (t + dt)). This is a simple and fast technique to compute the image brightness model parameters. Indeed, means and variances may be easily pre-calculated. Therefore, the linear model coefficients can be estimated using at most four arithmetic operations at each iteration. This technique is much faster than the least squares technique used by Gennert [Gen87]. Having estimated coefficients Al (p) and B l (p), we can then estimate the displacement dl (p) at pixel p. First, the estimated displacement from the higher level is used as the prediction of the present location: dl0 (p) = dl+1 (p) ∗ 2

(3)

To minimize the calculation effort, rather than doing a full block matching search, we check only five vectors (around the predicted position) in the first step, and at the very most three vectors in the following steps. Figure 1 illustrates this procedure. Then, the best displacement candidate, δ l1 ∈ {(0, 0), (−1, 0), (1, 0), (0, 1), (0, −1)}, is selected according to criterion E(p, dl0 (p) + δ l1 , t). The current displacement is then updated: dl1 = dl0 + δ l1 (4) In the next search steps, three new candidates are evaluated. Their position depends upon the best previous candidate.

O. Buisson et al.: Motion compensated film restoration

203

Cy

EP (p, d, t) =

4

n/2


(Px (px , py + v, t)

v=−n/2

3

−Px (px + dx , py + dy + v, t + dt))2

4

2

4

3

4

1

1

2

3

1

1

2

1

2

0

1

−1

+

−Py (px + dx + u, py + dy , t + dt))2

x

0

1

2

3

Fig. 1. Recursive block matching search

For example (see Fig. 1): δ li−1 = (1, 0) ⇒ δ li ∈ {(0, 0), (1, 0), (0, 1), (0, −1)}

(5)

where i denotes the search iteration number, and displacement (0, 0) is related to the best previously selected candidate. Note that the candidates that have already been checked do not need further evaluation. The current displacement is then updated with the best candidate: dli = dli−1 + δ li .

(7)

where Px and Py represent the horizontal Radon projection (along the x-axis) and the vertical Radon projection (along the y-axis), respectively:  n/2 Px (p, t) = u=−n/2 I(px + u, py , t) (8) n/2 Py (p, t) = v=−n/2 I(px , py + v, t)

C −1

(Py (px + u, py , t)

u=−n/2

−2 −2

n/2


(6)

The updating process is stopped at the moment at which the update falls under a threshold, in case the previous selected candidate remains the best (local minimum), or after a fixed number of iterations. Finally, we have designed an adaptive search strategy under which all possible vectors do not need to be checked, thus providing a fast block-matching search. For a maximum displacement magnitude of ±3 pixels, this method checks only 20 candidates, while an exhaustive method checks 49. So, the processing speed increases by a factor of almost 2.5. This algorithm incorporates both computational simplicity and good performance. Indeed, motion compensation is achieved for a (2000 × 1500) image in less than 30 seconds on a 700 MHz PC. To further minimize the processing speed, an appealing approach is to reduce complete two-dimensional information to one-dimensional information [Anc93]. Usually, one does not use gray values directly, but rather some filtered version of the image. So, giving appropriate transformations (along the x- and y-axes, respectively), a block of n × n pixels is reduced to two n-component vectors. The key aspect of this approach is its reduction of the complexity of the problem, while maintaining a good estimation of the original issue. To enhance the speed of our block matching technique, we use a new error function based on these attractive 2D–1D transformations. More exactly, we rely on Radon projections [San88]. Considering n × n blocks, Ep may be expressed as:

The computation of criterion EP requires only 6n − 1 operations, while E needs 3n2 − 1 operations. Applied to the whole image, such an optimization greatly improves our motion estimation processing performance. However, while this novel criterion significantly improves the processing speed as compared to the previously used criterion, it does not provide the same quality of motion estimation. Indeed, the use of compact information instead of complete data involves some errors in the motion estimation. An error of ε pixels at a level l of the pyramid generates an error of ε∗2l pixels at level 0. The larger l, the more significant are the estimation errors. Therefore, to alleviate the propagation of these errors over the pyramid, we do not use the same criterion at each level of the pyramid: • at high levels lh  [k, L] (L is the higher level of the pyramid), the minimization is based on the error function E; • at lower levels ll  [0, k − 1], the minimization is based on the error function EP . While increasing the processing speed by a factor of almost 2, this technique does not decrease the quality of our spatial and intensity jitter detectors. However, the detection of dust spots (on high resolution sequences) is significantly better using the former algorithm (without Radon projections). In the following, we present our techniques for removing dust spots, spatial and intensity jitter. The detection of these three categories of defects is performed on motion compensated film sequences. Thus, our motion estimator is used in the first step. Notice that we do not automatically handle scene changes in the film sequences – the segmentation step into sequences of the same scene is performed by an operator. 3 Dust spot removal Dust spots and thin hair are primarily caused by careless handling, poorly maintained equipment and improper storage. They are randomly spread over each frame, and disturb the viewer when too numerous (see Fig. 2). The gray levels of these artifacts tend to the black (i.e. dust particles in a positive to positive copy) or to the white (negative to positive copy). They typically can be found on one frame only, and will therefore present a strong temporal discontinuity in image brightness. However, we cannot use simple frame subtraction or “XORing” to detect them, because within a sequence,

204

O. Buisson et al.: Motion compensated film restoration

camera and actors or scene elements move around, objects may overlap with other objects and/or background details (socalled occlusions or disocclusions). Thus, our dynamic dust detector relies on both motion compensation and gray-level conservation. The same scheme has been adopted in several works [Ros96,Cho98,Kok98], differing mainly in the motion estimation step, but most of them are based on the video format. 3.1 Dust spot detection Once the optical motion flow has been correctly estimated, the next frame may be rebuilt without any deteriorations and a simple inter-frame difference may then be used for the detection process. Outliers, usually corresponding to deteriorations, occlusions or disocclusions, are detected when this difference is higher than a threshold S. These outliers are potential deteriorations.

Fig. 3. Dust spot detection on “La belle et la bˆete” frame

low as possible to preserve the visual quality of the processed movie.

3.2 Dust spot correction After localization of the deteriorations, the damaged areas are reconstructed using our restoration method. This detection/correction scheme has been also used by other work related to film restoration [Mor95,Dec97a,Cho98,Kok98]. Indeed, classical methods such as low-pass filters or median filters [Nie87,Pit90,Alp90] are not appropriate tools for high quality restoration, because they deteriorate high frequency components of the images. Our image reconstruction method is a polynomial interpolation based on a cubic polynomial: Fig. 2. Frame of “La belle et la bˆete” film by Jean Cocteau (1946)

IP I (x, y) =

3
3


akl xk y l

(9)

k=0 l=0

To deal with occlusions and disocclusions, we use a third image in our estimation scheme. The same process as described above is performed between the image at time t and the image at time t − dt. Common spurious points from the two independent motion estimation and comparison processes are selected as deteriorations. Let us emphasize that the double evidence provided by “ANDing” the two detection results permits us to easily reject the false detections due to occlusions or disocclusions. Indeed, occlusions or disocclusions occurring between frames I(t) and I(t − dt) could not be the same as those appearing between frames I(t) and I(t + dt). Figure 3 shows the result of our dust spot detector on one frame of “La belle et la bˆete” sequence (see Fig. 2). To evaluate the performances of our dust spot detector, a statistical study has been carried out on an undamaged film sequence which has been artificially deteriorated [Bui97a]. This investigation has led to the following detection rates: 2.7% of false detections and 18% of undetected defects. Notice that it is obviously preferable to miss a few dust spots than to remove scene features. The false detection rate then has to be as

which is the most simple model to approximate low frequency components of an image. To estimate the model coefficients akl , we use a leastsquares technique whose input data are the coordinates (xi , yi ) of uncorrupted pixels Pi belonging to a pixel block P around the defect. The resulting estimates

akl are then used to reconstruct the distortion area using RP I (xj , yj ) =

3
3


akl xkj yjl , (xj , yj ) ∈ Q

(10)

k=0 l=0

where Q denotes the set of deteriorated pixels to reconstruct. Our dust spot removal method yields good results on the “La belle et la bˆete” frame shown in Fig. 4. It can be seen that most impulsive distortions are removed, leaving non-degraded areas of the frame untouched. Details of this sequence before and after restoration are shown in Figs. 5 and 6. This technique has been validated on the whole high resolution (2000×1500)

O. Buisson et al.: Motion compensated film restoration

205

This assumption states that the global motion of the image sequence is continuous. On the contrary, spatial instabilities involve a discontinuous global motion, which is manifested by rapid temporal changes in the sequence. Notice that this dynamic noise is not of the same intensity over the sequence. Our method consists of first estimating the global motion of the deteriorated image sequence, and then searching for the possible discontinuities in the resulting motion field. We define the global motion between two frames as the most frequent apparent motion. Indeed, in a motion picture film sequence, most of the objects of the scene are stationary. Besides, even if the moving edges are more numerous than stationary ones, they could not have the same motions. Therefore, a non-zero global motion should be due to a pan motion and/or to spatial jitter. 4.1 Estimation of the global motion Fig. 4. Dust spot removal on “La belle et la bˆete” frame

“La belle et la bˆete” sequence (60 frames). This is illustrated by the video-clip “dustCompare.mpeg” (see http://perso.univlr.fr/sboukir/FilmRestoration.htm), showing a sample of the sequence before (left part) and after (right part) restoration. Other results obtained on another historical film sequence are also depicted in Figs. 8 and 9. Notice that our simple interpolation method reconstructs only the low frequencies of the damaged areas. Furthermore, due to its spatial nature, our technique is well suited only for small local defects (dust, hair, etc.). Figure 7 shows an enlarged part of frame 44 before and after removal of a large dust blob (see the bottom center of Fig. 6). The interpolated area is still somewhat visible because the defect is close to a light edge boundary. Hence, one naturally turns to spatio-temporal methods, but at a greater cost and risk of error. Indeed, this requires an accurate motion-compensation method, a difficult issue, especially on noisy image sequences.

Let us consider a sequence S of n images I(0), I(1), ..., I(n− 1) affected by spatial jitter. The global motion between each successive pair of images I(t) and I(t − 1) is estimated using the following steps: 1. Histograms Hx and Hy , respectively related to edge points displacements dx (t) and dy (t) between frame I(t) and I(t − 1), are computed. More precisely, these histograms cumulate the displacements along the x and y-axes of every edge point of the current image. Of course, this dynamic information is provided by our motion estimator. 2. The peaks Dgx (t) and Dgy (t) obtained in the two preceding histograms are related to the most frequent motion occurring between the two images. The global motion is then Dg(t) = (Dgx (t), Dgy (t))t . The same process is applied to the whole image sequence, and leads to a set of  global motion vectors : Dg(1), Dg(2), ..., Dg(n − 1) . 4.2 Spatial jitter correction

4 Stabilization Unlike dust particles and scratches, which are local deteriorations, spatial jitter is global. This artifact introduces unusual temporally rapid changes caused by damage at the corners of the perforations, misalignment of film in the projector (or scanner) gate, or improper film guidance in old cameras. In contrast to the work done in dust removal [Mor95,Ros96,Dec97a,Cho98,Kok98], spatial jitter has been addressed very little in the literature. The problem is not a negligible one, especially in old motion pictures which usually suffer from such disturbing dynamic noise. To our knowledge, the only reported work related to frame jitter removal is that described in Suter and Richardson [Sut96]. The basic strategy of this simple technique is to calculate the cross-correlation between frames by Fourier transforming the frames. The peaks of the cross-correlation are then sought to estimate the shifts. The stabilization of scanned images in the presence of vibrations, a problem similar to the removal of spatial jitter in motion picture films, is tackled in Wolberg and Loce [Wol96]. In this section, we describe our spatial jitter removal algorithm, which relies on a common smoothness assumption.

We can easily establish a relation between global motion and spatial jitter:

 (11) Dg(t) = P g(t) + (t) − P g(t − 1) + (t − 1) where P g(t) = (P gx (t), P gy (t))t represents the global position of the stationary edge points of the current image I(t) and (t) = (x (t), y (t)) the spatial jitter of this image. From (11), we can derive a more tractable relation: Sg(t) =

t


 Dg(i) = P g(t) + (t) − P g(0) + (0) (12)

i=1

which requires another constraint to estimate the spatial instabilities (t). Having no a priori information about these defects, we consider a widely used smoothing assumption in motion analysis: the global motion is continuous over the sequence. Therefore, we can adopt a simple polynomial representation for the components P gx (t), P gy (t) of the global position:  d P gx (t) = k=0 ak tk d (13) P gy (t) = k=0 bk tk

206

Fig. 5. Part of frame 22 of “La belle et la bˆete” sequence before and after correction

Fig. 6. Part of frame 44 of “La belle et la bˆete” sequence before and after correction

Fig. 7. Enlarged part of the previous image before and after correction

O. Buisson et al.: Motion compensated film restoration

O. Buisson et al.: Motion compensated film restoration

207

Fig. 8. Part of frame 12 of the “Building” sequence before and after correction

Fig. 9. Details from frame 16 of the “Building” sequence before and after correction

where d denotes the order of the polynomial model. Incorporating these equations in (12) leads to: 

d Sgx (t) = k=1 ak tk + x (t) − x (0) d Sgy (t) = k=1 bk tk + y (t) − y (0)

(14)

which may be expressed as two linear systems:   −1 Sgx (t0 )  Sgx (t1 )   −1    = .  ..   ..  . Sgx (tN ) −1

t0 t1 .. . tN

  −1 Sgy (t0 )  Sgy (t1 )   −1    = .  ..   ..  . Sgy (tN ) −1

t0 t1 .. . tN





    . . . td0
x (0)
x (t0 ) d    . . . t1   a1  
x (t1 )   .  .  +  .  . . . ..   ..   .. 
x (tN ) ad . . . tdN (15)     . . . td0
y (0)
y (t0 )     . . . td1    b1  
y (t1 )  ..   ..  +  ..  ··· .  .   . 
y (tN ) bd . . . tdN (16)

which may then be written as follows : 

Sg x = T .A + x Sg y = T .B + y

(17)

Assuming that x and y are Gaussian noise, a least squares technique is used to estimate the two sets of parameters A and

B. From (17), we deduce: 

= (T t .T )−1 .T t .Sg A x

= (T t .T )−1 .T t .Sg B y

(18)

Therefore, the estimation of Sg(t) becomes straightforward:   x (t) = d

ak tk −

x (0) Sg k=1 (19) d

 y (0) Sg y (t) = k=1 bk tk −

leading finally to the estimation of the spatial jitter: 

(t) = Sg(t) − Sg(t)

(20)

Having determined the misalignment vector (t) for each frame I(t), spatial jitter is easily corrected by a shift of − (t). In our experiments, we use a cubic polynomial approximation (d = 3) to cover most of the possible global motions occurring in a real world scene. Of course, this linear approximation is used locally in the sequence, and concerns only a limited number of images. Therefore, we use N/2 preceding frames and N/2 (N > d) following frames. Our method has been validated on a high resolution (2000 × 1500) “Fr`eres Lumi`ere” historical film sequence (40 frames). Figure 10 shows the first and last frames of

208

O. Buisson et al.: Motion compensated film restoration

Fig. 10. First and last frame of a “Fr`eres Lumi`ere” film no. 805. 6e Bataillon de chasseurs alpins – Artillerie de montagne : chargement de mulets (1897)

Fig. 11. Global motion over the “Fr`eres Lumi`ere” sequence before and after correction (smoothest curve)

O. Buisson et al.: Motion compensated film restoration

Fig. 12. A synthetic sequence (50 frames)

Fig. 13. Surrounding circles for the mean intensity calculation

Fig. 14. Mean intensity over the synthetic sequence (left). Mean intensity over the synthetic sequence with artificial flicker (right)

209

210

O. Buisson et al.: Motion compensated film restoration

Fig. 15. Artificial flicker

this sequence, while Fig. 11 depicts its global motion behavior before and after the removal of spatial jitter. The videoclip “jitterCompare.mpeg” (available from http://perso.univlr.fr/sboukir/FilmRestoration.htm) shows the sequence before (left part) and after (right part) frame jitter removal. The results obtained show a significant alleviation of the misalignment of the frames.

5 Brightness correction

This weighted mean intensity has been calculated for each frame of the previous synthetic sequence, leading to a smooth temporal behavior of the mean intensity over the sequence depicted in Fig. 14 (left part). Of course, in this simple illustration, the temporal mean intensity variation is only due to the motion of the object. However, these brightness variations may also be caused by light changes or flicker. Therefore, instead of a smooth mean intensity curve, we should have a noisy curve such as that depicted on the right-hand side of Fig. 14. This curve has been obtained by adding an artificial intensity jitter (see Fig. 15) to the previous synthetic sequence. Our flicker detector operates as follows. First, the (noisy) mono-dimensional signal µ(Xc , Yc , t) related to the mean intensity over the whole motion compensated sequence is filtered using a low-pass filter. The difference between the original and filtered signals then provides the flicker E(Xc , Yc , t) at the center (Xc , Yc ) of the surrounding circle of each frame of the sequence. Usually, though, brightness variations caused by flicker are not constant over a frame. Therefore, we estimate this dynamic noise at several centers uniformly spread over each frame of the sequence, the corresponding circles being overlapped. Finally, we estimate the intensity jitter E(X, Y, t) at each pixel (X, Y ) of frame I(t) using an interpolation of the intensity jitters Ei (Xci , Y ci , t) computed at the different circle centers (Xci , Y ci ) of I(t):

Intensity jitter is usually due to jerky shutter control and film transport by old hand crank cameras. In recent movies, the cause is often unsynchronized light sources. Sometimes, a gray shadow due to mishandling during the chemical development process induces similar intensity variation. This dynamic noise has also almost been ignored in the literature. In Wu and Suter [Wu 95], brightness correction is carried out after the image sequences are broken into segments, where each segment has the same scene. A simple brightness modification is then used to remove intensity jitter in each sequence. This method consists merely of adding each pixel a fixed amount to adjust the mean gray level of a given frame to the average of the mean intensity in the whole sequence. Our intensity jitter (or flicker) correction algorithm may be illustrated by a simple example showing a synthetic sequence representing a simple object moving in a white background (see Fig. 12). The brightness variations may be modeled by the following expression: |X −Xc | − |Yk −Yc | − kτx τy .e k I(Xk , Yk , t).e µ(Xc , Yc , t) = Y −Y | | − |Xk −Xc | − kτ c τx y .e ke (21) which calculates the mean intensity of the pixels (Xk , Yk ) belonging to a circle CI centered at (Xc , Yc ) (cf Fig. 13). This expression uses the distance of each pixel to the surrounding circle center as a weight value. Thus, the closest pixels to the circle center will have a greater influence in the mean intensity computation than the farthest ones. Notice that the coefficients τx and τy are used for practical considerations (for example, |Xk −Xc |

at 0.5 when |Xk − Xc | = R/2, to set the weight e− τx with R, the radius of the circle CI ).

E(X, Y, t) =

|X−Xci |

−

− τx .e i E(Xci , Y ci , t).e ci | − |X−Xci | − |Y −Y τy τx .e ie

|Y −Y ci | τy

(22) The correction of flicker is then simply achieved by extracting the former brightness error from each pixel. Our method has been validated on a video film sequence (80 frames) damaged by a disturbing flicker. Figure 16 shows the first and last frames of this sequence. Unfortunately, these dynamic artifacts cannot readily be seen in static frames. However, they are objectionable when the frames are played in sequence as the film runs (see top part of “flickerCompare.mpeg” (see http://perso.univlr.fr/sboukir/FilmRestoration.htm)), and are significantly reduced using our flicker correction method (see bottom part of “flickerCompare.mpeg”). 6 Conclusion We have presented efficient methods for the removal of dust spots, spatial and intensity jitter in degraded motion picture films. First, the images of the sequence are motion compensated using a recursive block-based motion estimator. This hierarchical algorithm is well suited for high resolution film sequences. The detection of dust spots which are impulsive defects is then easily achieved by the double evidence of two inter-frame differences. The detectors of spatial and intensity jitter are both based on smoothness assumptions. The detection of such dynamic noise is achieved using a simple difference between a smoothed global motion (respectively mean intensity) of the sequence and the noisy estimated global motion (respectively the noisy estimated mean intensity). Despite our encouraging results, there are several avenues for future work. The question of how to achieve an accurate and

O. Buisson et al.: Motion compensated film restoration

211 Acknowledgements. This work is supported by CENTRIMAGE (Laboratoires Neyrac Films - Cit´elab), Paris, France, under contract C58.

References [Alp90]

[Ana89]

[Anc93]

[Bab97]

[Ber92]

[Bla92] [Bra95]

[Bui97a]

[Bui97b]

Fig. 16. Two successive frames of “Gar¸con de caf´e” film sequence (1931)

[Bur81]

[Cho98]

robust estimation of the motion field without compromising the processing speed is a key aspect of our ongoing research. Indeed, one of the most crucial aspects of any restoration technique is to detect and account for motion. We have developed a strategy that involves hierarchical block matching assuming a locally very simple motion model. However, this approach cannot accurately model all of the motion in a realistic scene. On the other hand, even with the reduced cost/increasing capacity of digital systems, existing robust motion estimation techniques are still too costly to handle high resolution motion sequences. Likewise, there are situations where the implicit assumptions behind our dynamic artifact detection methods will fail to be met, thus requiring more sophisticated techniques. In particular, the search for more accurate models for these film noises, including film grain, is another important field of investigation in future work. Nonetheless, we have demonstrated that the methods we have implemented are useful techniques one can employ to restore film and video material at reasonable costs.

[Dec97a]

[Dec97b]

[Gen87]

[Haa92] [Jol94] [Kok98]

[Mor95]

Alp B, Haavisto P, Jarske T, Oistamo K (1990) Medianbased algorithms for image sequence processing. SPIE Visual Commun Image Process, Vol. 1360, pp. 122–133 Anandan P (1989) A computational framework and an algorithm for the measurement of visual motion. Int J Comput Vision 2:283–310 Ancona N, Poggio T (1993) Optical flow from 1D correlation: application to a simple time-to-crash detector. ICCV ’93, IEEE International Conference on Computer Vision, Berlin, Vol. 93, pp. 209–214, May 1993 Bab-Hadiashar A, Suter D (1997) Optic flow calculation using robust statistics. CVPR’97, IEEE International Conference on Computer Vision and Pattern Recognition, Puerto Rico, USA, pp 988–993 Bergen J, Anadan P, Hanna K, Hingorami R (1992) Hierarchical model-based motion estimation. Proceedings European Conference on ComputerVision, Santa Margherita Ligure, Italy, Springer Black MJ (1992) Robust incremental optical flow. PhD thesis, University of Yale, USA Brailean JC (1995) Simultaneous recursive displacement estimation and restoration of noisy-blurred image sequences. IEEE Trans Image Process 4(9) Buisson O (1997) Analyse de s´equences d’images haute r´esolution. Application a` la restauration num´erique de films cin´ematographiques. PhD thesis, Universit´e de La Rochelle, France Buisson O, Besserer B, Boukir S, Helt F (1997) Deterioration detection for digital film restoration. International Conference on Computer Vision and Pattern Recognition, Puerto Rico, USA, 1:78–84 Burt PJ (1981) Fast filter transform for image processing. CVGIP, Comput Vision Graph and Image Process 16:20– 51 Chong MN, Kalra S, Krishnan D, Laud A (1998) Computerized motion picture restoration system. Proceedings of BroadcastAsia98, pp 153–159 Decenci`ere Ferrandi`ere E (1997) Detection of local defects in old motion pictures. VII National Symposium on Pattern Recognition and Image Analysis, 145–150 Decenci`ere Ferrandi`ere E (1997) Restauration automatique de films anciens. PhD thesis, Ecole Nationale Sup´erieure des Mines de Paris, France Gennert MA, Negahdaripour S (1987) Relaxing the brightness constancy assumption in computing optical flow. Technical Report 975, MIT Artificial Intelligence Laboratory, MIT, MA Haan G (1992) Motion estimation and compensation. PhD thesis, Delft University of Technology, The Netherlands Jolion JM, Rosenfeld A (1994) A pyramid framework for early vision. Kluwer, Dordrecht Kokaram A (1998) Motion picture restoration. Digital algorithms for artefact suppression in degraded motion picture film and video. Springer, Berlin Heidelberg NewYork Morris RD (1995) Image sequence restoration using Gibbs distributions. PhD thesis, University of Cambridge, UK

212 [Nie87]

[Ozk92]

[Pit90] [Ros96]

[San88]

[Sri85]

[Sut96]

[Vel90] [Wol96]

[Wu 95]

O. Buisson et al.: Motion compensated film restoration Nieminen A, Heinonen P, Neuvo Y (1987) A new class of detail-preserving filters for image processing. IEEE Trans PAMI (9):74–90 Ozkan MK, Erdem AT, Sezan MI, Tekalp AM (1992) Efficient multiframe Wiener restoration of blurred and noisy image sequences. IEEE Trans Image Process 1:453–476 Pitas I, Venetsanopoulos AN (1990) Nonlinear digital filters principles and applications. Kulwer, Dordrecht Rosenthaler L, Wittmann A, G¨unzl A, Gschwind R (1996) Restoration of old movie films by digital image processing. IMAGE’COM 96, Bordeaux, France, pp 1–6 Sanz JLC, Hinkle EB, Jain AK (1988) Radon and projection transform-based computer vision. Springer Series in Information Sciences, 16. Springer, Berlin Heidelberg New York Srinivasan R, Rao KR (1985) Predictive coding based on efficient motion estimation. IEEE Trans Commun 33(8):888–896 Suter D, Richardson P (1996) Historical film restoration and video coding. Proceedings of PCS’96, Melbourne, Australia, pp 389–394 Veldhuis R (1990) Restoration of lost samples in digital signals. Prentice Hall, 1990 Wolberg G, Loce RP (1996) Restoration of images scanned in the presence of vibrations. J Electr Imag 5(1):50–65 Wu Y, Suter D (1995) Historical film processing. SPIE Proc Appl Digital Image Process XVIII, 2564:289–300

Olivier Buisson received his Ph.D degree in computer science from the University of La Rochelle, France, in 1997. From 1998 to 1999, he developed colour movie restoration software for the Ex-Machina company. Since 1999, he has joined the research group of the I.N.A (Audiovisual National Institute) in France. He is currently working on BRAVA, a European research project. The aim of this project is to automatically restore video films in real time. His research interests include motion estimation, belief propagation and image indexing. Samia Boukir received the computer science engineering degree from the University of Algiers, Algeria, in 1989 and the PhD degree in computer vision from the University of Rennes 1, France, in 1993. Since 1994, she has been a lecturer at the University of La Rochelle, France, and a member of the L3i research group. Her current research interests include image processing and dynamic scene analysis especially matching, tracking and motion analysis issues. She is currently working on the restoration of archived motion pictures and on the analysis of dance gestures. Bernard Besserer obtained the engineering degree in electronics from the University of Clermont-Ferrand, France, in 1988. He received the PhD degree in applied electronics from the same University in 1993. Since 1993, he has been a lecturer at La Rochelle University and a member of the L3i research group. His major research interests include static and dynamic image analysis, software and hardware optimization for high-resolution image processing, and color science. Since 1995, he worked on digital motion picture restoration including scratch detection, tracking and removal, image stabilization and die fading correction.