IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 11, NO. 4, APRIL 2002

485

Correspondence________________________________________________________________________ Mathematics for Demosaicking H. J. Trussell and Robert E. Hartwig

Abstract—Digital color cameras sample the continuous color spectrum using three or more filters; however, each pixel represents a sample of only one of the color bands. This arrangement is called a mosaic. To produce a full-resolution color image, the recorded image must be processed to estimate the values of the pixels for all the other color bands. This restoration process is often called demosaicking. This paper uses stacked notation to represent the mosaicked image capture and derives the minimum mean square error (MMSE) estimator for the demosaicked image. By making common assumptions, the restoration can be computed in a cost-effective manner. Extensions to the linear method are proposed to allow adaptive behavior. Index Terms—Deblurring, demosaic, digital camera, image restoration, interpolation.

to the general problem of demosaicking blurred images in a cost-effective manner with relatively simple assumptions. The method can be extended to the adaptive case in a straightforward manner. The two-dimensional (2-D) restoration problem has long used stacked notation to represent the symbolic mathematics for restoration. Because we will use the properties of the stacked notation extensively, it is reasonable to do a quick review. Without loss of generality, let us assume an image is defined on a square N 2 N grid. The notation for the spatial dimensions, N , is easily recognized later when we introduce other spectral and sampling dimensions.The single band 2-D image is defined by

X

X = [x1 x 2 ;

x

Digital color cameras require sampling of the image in at least three color bands. Most often, these are in the red, green, and blue spectral regions. In using CCD arrays to capture the color information, an expedient solution to color sampling is to put color filters on each of the sensor elements. A common filter pattern is the Bayer pattern shown in Fig. 1 [1]. This sampling method means that only one color band is sampled at each spatial location. The color information of the other bands at any location must be estimated. The problem of restoration of subsampled color images is called demosaicking.1 This problem is fundamental to the operation of current color digital cameras. There have been many solutions to the problem which take heuristic approaches, that is, they are not based on solving a mathematically defined problem under a stated set of assumptions [2]–[10]. To their credit, these methods work well enough to be the bases for many implementations. While these methods are not based on the classical statistical assumptions, they are based on reasonable assumptions about color images and are locally adaptive. The methods do not consider the additional problem of optical blurring. A large drawback to the classical approach is the complications of dealing with the huge matrices defined by the classical Bayesian or statistical theories. The Bayesian approach has been addressed in [11] for the case with no optical blurring and without achieving a cost-effective solution. The work does not address the sampling pattern in an explicit manner. This paper shows how the classical theories can be applied

...;

xN ]

(1)

where the k are column vectors of length N . The stacked vector representing the image is given by

x = [xT1 xT2 ;

I. INTRODUCTION

;

;

...;

xTN ]T

(2)

which is an N 2 2 1 column vector. Two-dimensional (2-D) convolution for a single band image with a K 2 K point spread function (PSF) is defined by

K 01 01 K

y (m; n)

=

k=0

h(k; l)x(m

0

k; n

0

l)

(3)

l=0

and is written using the stacked notation for the vectors we have

y

(N +K

01) 21 = H(N +K 01) 2N

x

N

21

where the matrix representing the 2-D PSF in stacked form is given by y0; 0 y1; 0

.. . yN +K

y

(N +K

02; 0

y0; 1

01) 21 =

y1; 1

.. . yN +K

02; 1

y0; 2

.. .

yN +K

02; N +K 02 x0; 0

Manuscript received December 14, 1999; revised November 30, 2001. This work was supported by Hewlett-Packard Labs. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Glenn Healey. H. J. Trussell is with the Department of Electrical and Computer Engineering, North Carolina State University, Raleigh, NC 27695-7911 USA (e-mail: [email protected]). R. E. Hartwig is with the Department of Mathematics, North Carolina State University, Raleigh, NC 27695-8205 USA. Publisher Item Identifier S 1057-7149(02)03978-7.

x1; 0

.. .

xN

=

H

(N +K

01) 2N

x1; 1

.. .

xN

1There

is some debate about the spelling of “demosaicking.” The author prefers to follow rules used in similar verbs ending in “ic,” e.g., mosaic—mosaicking, picnic—picnicking, panic—panicking, frolic—frolicking. See the Merriam–Webster Dictionary, online version at http://www.m-w.com. 1057–7149/02$10.00 © 2002 IEEE

01; 0

x0; 1

01; 1

x0; 2

.. .

xN

01; N 01

(4)

486

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 11, NO. 4, APRIL 2002

one-dimensional (1-D) N

wN

= exp(0j 2=N )

W1

=

2 N DFT matrix with the DFT coefficients

0 wN 0 wN 0 wN

0 wN 1 wN 2 wN

0 wN 2 wN 4 wN

.. .

.. .

.. .

. . . w0 . . . w 01 . . . w2 02 : N N N N N

N 01 N 02 0 wN wN wN

The 2-D DFT of

.. .

.. .

...

1 wN

x is denoted by x~ and can be written x 2 x~ 2 = W2 N

1

N

1

where the 2-D DFT matrix is defined by

W2

= W1

W1

:

The symbol represents the Kronecker product (outer product). The convolution can be written in the Fourier domain as

W2y = W2H x = W2H W20 W2x c

Bayer color filter array.

Wx

W

H

where the array (N +K 01) 2N is defined and shown in (5) at the bottom of the page. Since each (N + K 0 1) 2 N block of the large matrix is Toeplitz, the form is called block Toeplitz. For ease of notation in the succeeding math, let us assume that all images have been padded to accommodate the added dimensions of the PSF and denote output image size as N 2 N , i.e., replace N + K 0 1 by N . This will help in keeping track of dimensions of the various matrices. If the convolution of (3) is approximated by a 2-D circulant operation, the matrix will be block circulant. The block circulant is diagonalized by the 2-D DFT. Recall that the 2-D DFT is defined by using the

H

WHW Wf

II. EXTENSIONS TO COLOR SAMPLING The original color image represents the radiant power distribution at a continuum of wavelengths from approximately 400–700 nm. Let us assume it is sufficient to sample the color spectrum at P wavelengths, e.g., P = 31 for 10 nm sampling from 400 to 700 nm is common. Let f n g, 1  n  P , represent the P wavelength bands of a 2-D image.

x

. . . h 01 1

0

0

0

. . . h 01 0

...

0

0 0 0

0 0 0

0 0 0

... ... ...

0 0 0

0 0 0

0 0 0

0 0 0

... ... ...

0 0 0

. . . h0 01 0 0 ... 0 ... . . . h1 01 h0 01 . . . h2 01 h1 01 h0 01 . . .

0 0 0

0

0

0

...

0

0

0

0

...

0

...

.. . .. .

.. .

.. . .. .

.. .

.. . .. .

.. .

.. .

0 0 0

is a diagonal matrix with elements of the 2-D DFT of the base of the block circulant matrix of c on the diagonal. Thus, the term 01 2 is the element-by-element multiplication of the 2 c 2 DFTs of the input signal and PSF.

0

0

0 h0; 1 0 h1; 1 h0; 1 0 h2; 1 h1; 1 h0; 1

.. .

.. .

.. .

0 0 0

H

0

0

.. .

0 0 0

=3

0

0

.. .

0 0 0

1

... ... ... ... ... ... ... ... ... ...

.. .

... ... ...

W

W2H W20 c

h0; 0 0 0 h1; 0 h0; 0 0 h2; 0 h1; 0 h0; 0

.. .

. . . h 01 0 0 0 0 ... 0 h0 0 0 0 ... 0 h1 0 h0 0 0 ... 0 h2 0 h1 0 h0 0 K

;

; ; ;

.. .

.. .

K

.. .

.. .

;

.. .

.. .

.. . .. .

.. .

H

where the subscript c emphasizes the block circular extension of . The terms 2 and 2 are recognized as the DFTs of the output and input, respectively. Note that 101 = (1=N ) 13 . It can be shown that

Wy

Fig. 1.

1

c

; ;

.. . .. .

.. .

;

.. . .. .

.. .

... ... ...

0 0 0

... ... ... ...

0 0 0 0

.. .

.. .

.. .

.. .

K

.. .

.. .

;

.. .

.. .

.. .

0 0 0

0 0 0

0 0 0

... ... ...

0 0 0

0 0 0 0

0 0 0 0

0 0 0 0

... ... ... ...

0 0 0 0

.. .

.. .

0 .. .

.. .

.. .

0 .. .

.. .

.. .

0 .. .

.. .

.. .

.. .

.. .

.. .

.. .

;K

.. .

;K

;K

;K

;K

;K

.. .

.. .

.. .

0

0

0

.. .

.. .

. . . h 01 01 K

;K

(5)

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 11, NO. 4, APRIL 2002

487

The bands can be stacked in the same way as the columns of the 2-D image and the color blurring can be represented in block form

u1 u2

u=

=

.. .

x1 x2

HF C

(A

xP

Note that this is possible since we have assumed equal blurring for each wavelength. This is important since it implies we are concerned only with deblurring the Q-dimensional sensed or recorded signals and not the original P dimensional source signals. It reduces the dimensionality of the problem from P to Q, e.g., from 31 to 3 or 4. This order change can be demonstrated by noting

where the matrix HF C for the full color problem is defined by

H1; 1 H2; 1

H1; 2 H2; 2

... ...

H1; P H1; P

.. .

.. .

.. .

.. .

ST

HP; P

...

where the matrix Hj; k represents the blurring effect of the k th band of the input on the j th band of the output. For virtually all optical blurs, the wavelengths are independent, so the matrix is block diagonal

H1 0 0 H2 .. .

...

.. .

0

0

0 0

...

.. .

.. .

HP

...

where Hk is N 2 N . If the blurs can be approximated by the same matrix, Hk = H for all k , then the large matrix can be represented by the Kronecker (outer) product

HF C

=

IP

S=

...

.. . ...

H)x = (I H)

0

0

0

...

0

0

0

0

0

...

0

0

0

1

0

...

0

0

0

0

0

...

0

0

0

0

0

...

0

0

0

0

0

...

0

0

1

0

0

...

0

0

0

0

0

...

0

0

0

0

1

...

0

0

0

0

0

...

0

0

0

0

0

...

1

.. .

(I

H) x + 

.. .

C1 =

.. .

.. .

.. .

.. .

.. .

:

.. .

The extension to 1 : M sampling, where N is an integral multiple of M , is straightforward with every M th diagonal element equal to unity. Note that we have the constraint M

(6)

The full-resolution forward image formation model can now be written

I

.. .

C1 = diag(0; 1; 0; 1; . . . ; 0; 1):

The Q-band sensed image is obtained from the P-band image by

v = ST

.. .

or

01

k=0

21 = (S 2P IN 2N )xN P 21 :

.. .

and

c = ST r:

N Q

x:

C0 = diag(1; 0; 1; 0; . . . ; 1; 0)

s1Q s2Q .. . sP Q

T Q

I

The diagonal matrices can be represented more concisely by using

and the output of the sensor is given by

v0

ST

1

.. .

A Q band sensor is represented by the matrix ...

(I

C0 =

r = [r1 ; r2 ; . . . ; rP ]:

s12 s22 .. . sP 2

I

Note the change in the dimensions of the identity matrices. Sampling of the analog signal can be represented by subsampling a digital signal. We assume that the original digital signal has been sampled without aliasing. The subsampling caused by the mosaic may introduce aliasing. It is the goal of all demosaicking methods to recover information up to the Nyquist limit of the original sampling. However, any gain over the Nyquist limit of the individual color sampling is an improvement. The 1 : 2 sampling matrix for a one-dimensional (1-D) signal is represented by diagonal matrices

H

where the subscript on the IP represents the size of the identity matrix. The appropriateness of this assumption depends on the wavelength range, the quality of the optics, and the accuracy required in the restoration. For most multielement lens systems, which have some color correction, this is a reasonable assumption. For a single element lens, the blur radius is approximately proportional to the wavelength. This would imply an almost 2 : 1 difference from 400 to 700 nm. However, since the eye is much less sensitive at these extremes than in the center of the range, the assumption is often appropriate for practical use. A source pixel is represented by its power at each wavelength

s11 s21 .. . sP 1

B) (C D) = AC BD v = S H x + : T

.. .

uP

HP; 1 HP; 2

where  represents the measurement or quantization noise. Now, using the property of the multiplication of Kronecker products

Ck = I:

(7)

The 2-D sampling is represented by a collection

Cjk = Cj Ck ;

for 0  j; k

 M 0 1:

The total forward model is now given by

y = C STF C HF C x + 

(8)

488

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 11, NO. 4, APRIL 2002

where STF C = ST I and HF C = I H can be used to represent the full color Kronecker products. The matrix C is a block diagonal

is a severe but common restriction. This is equivalent to assuming all bands are independent, identically distributed

... ...

E fx(u0 ; v0 ; 0 )x(u1 ; v1 ; 1 )g = E fx(u0 ; v0 )x(u1 ; v1 )gE fx(0)x(1 )g: (12)

C00 0 0 C01

C=

.. .

0

= diag

.. .

.. .

0 0 .. .

0 . . . C(M 01)(M 01) C00 ; C01 ; . . . ; C(M 01)(M 01)

(9)

where M 2 is the number of samplings it takes to cover the grid under investigation. The order of the diagonal matrices is arbitrary. Usually M 2  Q and the matrix may include replications of a color band. For a 1 : 2 subsampling in each dimension, M 2 = 4. Nonsquare sampling is possible but we will assume square sampling for ease of notation. For a (RGGB) Bayer pattern, as shown in Fig. 1, two of the bands in the stacked vector are identical, for this example

C00 0 0 0 0 0 C01 0 0 0 C10 0 0 0 0 C11

y=

xr xg : xg xb

(10)

Note that because the order of STF C and HF C can be reversed (all wavelengths blurred identically), we can write

y = C HF C STF C x +  : III. MINIMUM MEAN SQUARE ERROR SOLUTION The minimum mean square error (MMSE)problem is to minimize the expected value of the error, or minimize

8(D) = E fkx 0 Dyk2 g

with respect to the deconvolution matrix D. An equally valid approach is to minimize the posterior probability to obtain the maximum a posteriori probability (MAP) estimate. For Gaussian distributions, which are usually assumed, the MAP and MMSE estimates are identical. The solution is obtained by solving

@ 8(D) @D

+ CK CT ]y :

(13)

Note that the pseudoinverse, denoted by y, must be used since pre and post multiplying by C implies that the matrix in brackets is usually singular. For the full-resolution case C = I and, as such, can be eliminated from the equation, while the pseudoinverse reduces to the inverse. For the subsampled case, it must be retained. Let us consider the first case. A. Full-Resolution Color Problem

D = (K Kxx )(S I) I HT

which yields

01

D = Kxx SF C HFT C CT 1[CHF C STF C Kxx SF C HFT C CT

For the full-resolution case, the matrix C can be eliminated; then by using the Kronecker products, we have

= 02E fxyT g + 2DE fyyT g = 0

D = E fxyT g E fyyT g

That the restriction to separable correlation structures is not realistic can be seen by considering an image of a yellow ball on a green lawn. In the region of the ball, the spatial correlation is characterized by the smooth surface of the ball and the color (wavelength) correlation would be high for wavelengths in the region above 450 nm. In the region of the lawn, the spatial correlation would the characterized by the texture of the grass, while the color correlations would be high only in the 450–550 nm range. For this case, spatial correlation would be correlated to color correlation. This violates the separability assumption. However, most linear restoration methods assume stationary statistics. While stationarity is not realistic, that does not prevent the use of linear filters in the real applications. The linear methods can be improved by allowing adaptive behavior based on local conditions. Methods of using the linear filter discussed in this paper in an adaptive context are discussed in the last section. It is of interest to note how this separability simplifies the computation in two cases. The first case is the common case, where there is no subsampling of the color channels. This full-resolution case is treated in the next section. The main interest of this paper is the simplification of the subsampled color case, which is treated in the succeeding section. In both cases, the solution can be found by substituting for y in (11) from the model and using the assumption that the noise is zero mean and uncorrelated with the image. This gives

:

(11)

The solution appears simple but the matrices are huge. It remains to determine an efficient way to compute the solution. We will begin to do this by using the separability that is represented by the Kronecker products. The products that we will use are

1 (I H)

ST

I (K Kxx)(S I) I HT + K

01

where the inverse matrix exists. Gathering the terms across the Kronecker product yields

D = K S Kxx HT

1

ST K S



HKxx HT

01

+ K

STF C = ST I HF C = I H WF C = I W2 Kxx = K Kxx K = I 2 I:

In order to compute the solution in reasonable time, the FFT will be used. This requires the assumptions that the circulant approximations for the PSF and the autocorrelation matrix are reasonable. This gives

There are three assumptions of note here. The blurs are the same for all wavelengths. The noise is assumed zero mean and white. Its autocorrelation is represented by K . The autocorrelation of the color image Kxx is separable into a wavelength part and a spatial part. This

(W2 HW2 0 W2 KxxW2 0 W2 HT W2 0 ) 0 +(I W2 )K (I W2 0 ) :

: (14)

~ = (I W2 )D(I W2 01 ) D = (K S) (W2 Kxx W2 01 W2 HW2 01 ) ST K S 1

1

1

1

1

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 11, NO. 4, APRIL 2002

Note the Fourier transforms diagonalize the matrices which permits term-by-term multiplication; the matrix T  is 4 2 4 for the Bayer pattern color representation.

SKS

489

The solution of (14) is now modified with the subsampling matrix to give

D = (K S) KxxHT CT C ST K S

B. Subsampled Color Problem The equation to be solved for the subsampled case is given by (13), where 6= . The subsampling matrix cannot be represented as a Kronecker product. Recall that from (7) that the periodic sampling on the diagonal of each block is related by the fact that all spatial positions are sampled exactly once. Furthermore, each subsampling is related to the others by a periodic shift. Assume the first sampling starts at the origin; then the Fourier transform of the k th shifted sampling in one dimension is

C I

C

W1 Ck W101 = 2k W1C0W10123k

W1C0W101 = 12

= 12

1

1



IN=2:

(15)

For other values of M, (15) can be generalized. Since the current cameras use only M = 2, we will concentrate on that case for the remainder of this paper. The extension to 2-D sampling is given by

W2(Ck Cl )W201 = W1Ck W101 W1ClW101:

D~ = (K S) K~ xxH~ T C~ 3 C~ ST K S



:



H~ K~ xx H~ T

+ ~ 











It is the evaluation of the pseudoinverse of the matrix in brackets with which we are most concerned. Let us examine this task in detail. The Fourier transform, coupled with the assumptions on separability, has produced a diagonal N 2 N matrix. We can use this effectively if we reorder the Kronecker products; that is, group the image samples by pixels (each pixel is a 4-vector) rather than by bands (each band is an N 2 vector). The image vector would have the form

y1; 1 y2; 1

C 0 0 0 0 C~ 01 0 0 0 0 C~ 10 0 0 0 0 C~ 11 0 0 C~ 0 C~ 0 0 0 0 C~ 0 C~ 1 0 0 0 C~ 1 C~ 0 0 0 0 0 C~ 1 C~ 1 ~ 00

..

y = yN;. 1 .. .





:

yN; N

y

where j; k = [y (1)j; k ; y (2)j; k ; . . . ; y (4)j; k ]T for the four color bands. The reordering of the Kronecker products is done by multiplication by a unitary permutation matrix [17, p. 408]. The effect, for example, is that instead of



P

If we define the diagonal phase matrix as

IN IN 0 0 0 0 0 IN 21 0 0 0 0 21 IN 0 0 0 21 21

8=

y

D~ = (KS) K~ xx H~T 8(I4 C~ 00)83 8(I4 C~ 00)83 ST K S H~ K~ xx H~ T y ~ 00 83 : +I4 2 IN 8 I4 C



=

K CT

K C~ 3 y : ~ , we Substituting for the Fourier transform of the sampling matrix C 1

The Fourier transform of the full-color sampling matrix (9) is given by

C~ =

+



1

and the 3 represents the complex conjugate. For the case of M = 2 1

HKxxHT

Taking the Fourier transform as before, we have



2k = diag([1; e0j2k=N ; e0j22k=N ; . . . ; e0j2Nk=N ]) 1



obtain

where

IN=2 IN=2 IN=2 IN=2

1

B = ST K S







H~ K~ xx H~ T

I

+ 4



2 2 IN



C

then ~ can be written

C~ 00 C~ = 8 00 0

0 C~ 00 0 0

0 0 C~ 00 0

we have

Bp = PT BP = H~ K~ xx H~ T

0 0 83 0 C~ 00

C~ 00 = 14



IN=2 IN=2 IN=2 IN=2

:

(16)

The above form can be derived by noting that the sampling can be interpreted as a constant function that is interpolated 2 : 1 by interspersing zeroes. It is well known that such interpolation results in replicated spectra. The matrix ~ can be written more concisely as

C

C~ = 8(I4 C~ 00)83:

ST K S

+

I

2 2 N



I4:

Other products are changed accordingly. 2 The reordering allows the N 2 2 N 2 diagonal noise matrix  2 N to be easily used. The reordered matrix p is block diagonal with the k th 4 2 4 block of the form

I

B

where the ~ represents the Fourier transform of the spatial domain quantity. The transform of the 2-D sampling is given by

IN=2 IN=2 IN=2 IN=2



k ST K S + 2 I4

3 HK H

where k is the k th element of = ~ ~ xx ~ T . The reordered solution is now written ~ D~ p = PT DP ~ xx H ~ ~ 00 = K (K S) 8p C I4 83p 8p C~ 00 I4 E C~ 00 I4 83p y

1







(17)

490

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 11, NO. 4, APRIL 2002

where = 3p [( ~ ~ xx ~ T ) ( T  ) + 2 2N 4 ] p and p = T is the permuted diagonal phase matrix. Now let us concentrate on the block diagonal matrix

E 8 HK H P 8P

SKS

I

I 8

8

E

83

E = 8p H~ K~ xx H S K S +  IN I p = diag (E ; E ; . . . EN ) where each Ei is 4 4 and positive definite. In order to obtain compatible submatrices for multiplication, we will partition E as E = F0 F0 ~ where Fi is 2N 2N . We write C I in a form for compatibility IN= A C~ I = IIN= IN= N= ~T

1

T



2



4

1

2

2

2

00

00 4

4

2

2

2

2



where

IN= IN= IN= IN= ~ 4N . The matrix Q = (C A=

A

1 4

2

2

2

2



I

IN= IN= IN= IN= 2

2

2

2



I

F 0 0 F IN= IN= IN= IN=

A

2

2

2

C

I

Gk

( ) 1

2

2



A

2

2

2

2



( ) 1

( ) 2

2

2

2

2

1 16

1 16

=

1

2

2

2

1

2

2

1

2

2

2

1

2

2

G k +G k ( ) 1

( ) 2

Q=

1

1

1

IN= A (F 2

1

I

+

A A

( ) 1

( ) 2

( ) 1

( ) 2

( ) 1

( ) 2

1

1

1

E k0 N +E k0 +E k0 E k0 N +E k0 +E k0 1)

+1

Q=

1 16

(20)

2

:

2

2

1

1

1



diag

AG A; AG A; . . . ; AGN= A 1

2

2

: (22)

Fi , each Gk is defined by Gk = diag E k0 N ; E k0 N ; . . . ; E k0 N N + diag E k0 N N = ; E k0 N N = ; ...; (23) EkN N = k k k = diag B ; B ; . . . ; BN : (24)

Using the definition of (

1)

+1

(

1)

+

( ) 1

( ) 2

(

+1+

1)

+2

2

2

( )

(

(

1)

+2+

1)

1

1 1

1

1

1

1

Q

Qy = 16

1

1

1

The 4 2 4 matrices of is given by

2

1

( ) 1

:

( ) 2

(26)

E

+

E k0

1)N +1+N

(

1)N +1+N=2

(

1)N +1+N=2+N

+2

+

=2

=2 ; 1)N +2+N =2

E k0 (

(

1)N +2+N=2

(

1)N +2+N=2+N

1)

1

1

(21)

F I A), which F F F + F = G = diag G ; G ; . . . ; GN= where each Gi is 4N 4N . Now, Q can be written 1

G k +G k



(

1)

(

A F

2

( )

: (27)

=2 ;

.. .

E k0 N N= + E k0 N N= N = +E k0 N N + E k0 N N N = Finally, the matrix Q, for which we need to compute the pseudoin-

Now, consider the term ( N=2 )( 1 + 2 )( N=2

is block diagonal. Let us partition the sum 1 + 2 , as 1

( ) 2

+

2

1)

(

1)

+

+

(

2+

1)

+

2

+

2

verse, is given by

F ) IN= A 2

2

G

(

and which simplifies to 1

(25)

4

( )

( ) 2

1

G

: (19)

A A A A

2



The summation of the k1 + k2 is block diagonal and can be written in terms of the partitions of the original large matrix k as

we write 2

2

( ) 1

(

(F + F ) IN= Q = IIN= A A (F + F ) IN= N= IN= A (F + F ) IN= IN= A (F + F ) IN=

2

2

( ) 2

( ) ( 2+1)

= diag

A IN= A = IIN= A IN= N=

2

G k = diag B kN= ; B k ; . . . ; BNk : The matrix AGk A has the summation form G k +G k G k +G k AGk A = G k +G k G k +G k

Using the fact that

IN= IN= IN= IN=

( ) 2

and

2

2

( ) 1

4

B k ; B k ; . . . ; BN=k

= diag

(

2

G k 0k 0 G IN= IN= I IN= IN=

I



A of (18), it is expedient

where

1

1

Q=

2

1

(18)

4

~ where matrix is 4N 2 00 4 )E ( 00 4 ) is the part that will require the most effort in deriving the pseudoinverse. It can be written as

Q=

1 16

2

2

2

AG A G IN= IN= AGk A = IN= IN=

Now consider k . Because of the form of to partition k and write



diag

G

2 1



(



(

+

1

1

1

1

1

1

1

1

1

1

( ) 2

y

y



G + G ); 1 1

(

1 2

G ); . . . ; 2 2

:

2

2

(28)

are nonsingular, so the pseudoinverse

diag

1

1

1

1

y

G

(

1 1

+

G )0 ; 1 2

1

G + G )0 ; . . . ; 2 1



(



(

y

1

2

1

( ) 1

1

1

1

GN= + GN= )

G k +G k

1

1

2 2

1

GN= + GN= )0 1

2

2

2

1

(29)

in which the pseudoinverse of the 2 2 2 matrix is

+

2

1

1

1

1

y

=

1 4

1

1

1

1

:

(30)

The estimation process is reduced to inverting N 2 =2 matrices of size 4

2

4:

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 11, NO. 4, APRIL 2002

The term multiplying the pseudoinverse of (17) ~ xx H ~ (K S) 8p C ~ 00 I4 83 T= K p ~ 00 I4 can be handled by the same method of partitioning. The matrix C can be written as

IN IN IN IN

C~ 00 I4 = 14

IN IN IN IN

IN IN IN IN

IN IN IN IN

:

(31)

Let us also partition the diagonal matrix 8p into four N 2 2N 2 diagonal blocks

0 0 2 0 0 0 0

1

8p =

0 0

3

0 0 : 0

0 4

Now, we can write

8p (C~ 00 I4 )8p3 = 14

113 213 313 413

123 223 323 423

133 233 333 433

143 243 : 343 3  4 4

~ xx H ~ into four N 2 =4 2 N 2 =4 diWe partition the diagonal matrix K agonal blocks.

01 0 0 0

0 02 0 0

0 0 0 0 ~ xx H ~ =0= K 03 0 : 0 04 Combining these partitionings, we can write N as a 4 2 4 block matrix, T = [Tij ], where the blocks are defined by Tij = 14 (0i K S)i j ; ij = 1; 2; 3; 4: We note that the Kronecker products, 0i K S, are block diagonal with the P 2 Q blocks. Since Q = 4 for the Bayer case, we partition the diagonal matrices, i j3 , into 4 2 4 diagonal blocks. This shows that T can be written in the form of P 2 4 blocks. This is compatible with the 4 2 4 diagonal block form of the pseudoinverse matrix obtained previously. ~ The result of the partitioned forms of the matrices that make up D of (17) is that the 4 2 1 elements or the reordered Fourier transform of the recorded elements in the vector y are multiplied by P 2 4 matrices. There are only five matrix multiplies required for each 4 2 1 element. y 3 ^ = T8p Q 8p y can be done in a resonable Thus, the computation of x time. The procedure that was used here for the 2 2 2 Bayer sampling array can be extended to any regular subsampling patterns of any size. It is clear that the bookkeeping will be nontrivial. IV. ADAPTIVE EXTENSIONS A well-known disadvantage of linear methods is that they are based on statistical assumptions that are questionable when applied to real imagery. The necessary assumption of stationary signals is clearly violated in the real world. An image whose statistical character is the same over the entire spatial range would be very uninteresting. Different objects within the image necessarily have different statistical characterizations. Otherwise, pattern recognition would be impossible. Despite

491

this shortcoming, linear methods are used successfully in many practical applications. Researchers have recognized this limitation of linear processing and developed several ways to create adaptive versions of linear filters. Basic approaches are to estimate the signal characteristics or noise characteristics of a local area of the image and modify the statistical characteristics of the linear method locally. Methods that section the image and process each section for locally varying correlations, blur, and noise are discussed in [12], [13]. Methods that alter the stochastically based filter based on local edge estimation are discussed in [14]–[16]. The methods that employ local edge estimation use a similar philosophy to the demosaicking methods of [4], [6], [8]. The approach of using a collection of restoration kernels that are optimized for a finite number of edge orientations as in [14] is a straightforward extension of the linear method. It is expected that adaptive methods would outperform purely linear methods. Most adaptive methods have concentrated on the effect of edges in the images. However, most image sensors are photon counting devices, e.g., CCD arrays. The images from these sensors are subject to Poisson noise. This is usually modeled as additive, zero mean Gaussian noise with a variance proportional to the pixel value. This is signal dependent noise in that the variance of the noise depends on the value of the signal. However, in this model, the noise is uncorrelated with the signal. The straightforward approach to handle this case is to use sectioned methods similar to those which address the exact same problem for monochrome images [12]. However, iterative methods are not practical for current digital cameras. An alternative is to create restoration kernels based on a finite collection of signal-to-noise ratios (SNRs). This is analogous to the method of optimizing for a finite number of edge orientations and can be combined with that method. The choice of the appropriate kernel would be made based on local estimates of the SNR. Using the properties of the Poisson distribution, an estimate of the local mean would give an estimate of the local noise variance. Let us consider the creation of restoration kernels. The linear restoration filters derived in the previous section were computed in the frequency domain. To obtain small extent spatial filters, the filters are transformed to the spatial domain and truncated to a desired size. This commonly can be done by windowing the spatial domain filter to produce a small support kernel. The advantage of the linear method presented here is that the statistical characterization of local properties can be used explicitly to create the kernel. V. CONCLUSION Using common assumptions, it has been shown that MMSE restoration of a blurred and subsampled color image can be computed in reasonable time. This demosaicking of a color filter array with a Q 2 Q pattern requires computing the discrete Fourier transform of each spatial color band and only Q 2 Q matrix inverse operations. The method is amenable to adaptive extensions by using a series of spatial kernels based on local image and noise characteristics. Many questions still remain regarding the best uses of this result. The actual implementation of the method also remains to be seen. A comparison with the currently used interpolation methods is of interest. It is almost certain that adaptive method implementation would be needed to make the linear method commercially competitive. While this paper suggests that simple ways exist to make the method adaptive, it is probable that other modifications could produce greater improvements. It is hoped that this work will stimulate further investigation. ACKNOWLEDGMENT The idea for much of this work was a direct result of discussions with D. Taubman and reading material that he produced. The major

492

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 11, NO. 4, APRIL 2002

results of this paper were produced during a visit to Hewlett-Packard Labs under a summer program. REFERENCES [1] B. E. Bayer, “Color imaging array,” U.S. Patent 3 971 065, 1976. [2] D. R. Cok, “Signal processing method and apparatus for producing interpolated chrominance values in a sampled color image signal,” U.S. Patent 4 642 678, 1987. [3] T. W. Freeman, “Median filter for reconstructing missing color samples,” U.S. Patent 4 724 395, 1988. [4] A. C. Laroche, “Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients,” U.S. Patent 5 373 322, 1994. [5] D. R. Cok, “Reconstruction of CCD images using template matching,” in Proc. IS&T Ann. Conf./ICPS, 1994, pp. 380–385. [6] H. R. Hibbard, “Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients,” U.S. Patent 5 382 976, 1995. [7] J. E. Adams, Jr., “Interactions between color plane interpolation and other image processing functions in electronic photography,” Proc. SPIE, vol. 2416, pp. 144–155, 1995. [8] J. F. Hamilton and J. E. Adams, Jr., “Adaptive color plan interpolation in single sensor color electronic camera,” U.S. Patent 5 629 734, 1997.

[9] J. E. Adams, Jr., “Design of practical color filter array interpolation algorithms for digital cameras,” Proc. SPIE, vol. 3028, pp. 117–125, 1997. [10] R. Kimmel, “Demosaicing: Image reconstruction from color CCD samples,” IEEE Trans. Image Processing, vol. 8, pp. 1221–1228, Sept. 1999. [11] D. H. Brainard, “Bayesian method for reconstructing color images from trichromatic samples,” in Proc. IS&T 47th Annu. Meeting, Rochester, NY, 1994, pp. 375–380. [12] H. J. Trussell and B. R. Hunt, “Sectioned methods in image restoration,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-26, pp. 157–164, Apr. 1978. [13] H. J. Trussell and S. Fogel, “Identification and restoration of spatially variant motion blurs in sequential images,” IEEE Trans. Image Processing, vol. 1, pp. 123–126, Jan. 1992. [14] A. M. Tekalp, H. Kaufman, and J. W. Woods, “Edge-adaptive Kalman filtering for image restoration with ringing suppression,” IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 892–899, June 1989. [15] M. R. Banham and A. K. Katsaggelos, “Spatially adaptive wavelet-based multiscale image restoration,” IEEE Trans. Image Processing, vol. 5, pp. 619–634, Apr. 1996. [16] M. R. Azim-Sadjadi, R. Xiao, and Y. Yu, “Neural network decision directed edge-adaptive Kalman filter for image estimation,” IEEE Trans. Image Processing, vol. 8, pp. 589–592, Apr. 1999. [17] P. Lancaster and M. Tismenetsky, The Theory of Matrices, 2nd ed. New York: Academic, 1985.