Use+the+mathematical+relationship+between+the+chroma+signals+in+successive+frames+to+recover+geometry+and+colour+information

Introduction
This is a revised, corrected and expanded description of a proposed method for colour recovery utilising the PAL four frame sequence. Essentially the idea is that the difference in the chroma signal in parts of successive frames where the image is static (no motion or change in colour or brightness) allows us to identify points in the signal from which colour values can be inferred.

A crude experimental implementation of the ideas outlined here has led to some qualified success in recovering some colour from a sample film recording. A description of these results, including some of the practical obstacles, together with a sample image appears here in the Current Work section.

The account begins with a description of my understanding of PAL and the mathematics of the four frame sequence. I am not an expert on this subject and I would be glad to know of any errors or inaccuracies in this. I've also tried to keep it simple and strictly relevant to the colour recovery problem, but I apologise if, despite this, my explanations are still hard to follow!

My (simplified) understanding of PAL
The PAL colour system is designed to be backwardly compatible with monochrome television transmission, so that a monochrome receiver can still display a black and white picture from a colour PAL signal.

A monochrome television signal is an amplitude modulated signal where the amplitude of the signal determines the luminance (brightness) of each point in the picture. Each full frame of the picture is built up from a series of horizontal lines. The lines for each frame are transmitted in a sequence of two //fields//, where a field consists of all the odd, or all the even, lines of a frame. In the British TV system a frame consists of 625 lines, not all of which are actually displayed, and 25 frames (i.e. 50 fields) are displayed each second.

In the PAL colour system the chroma (colour) signal is the sum of two amplitude modulated signals of equal frequency but with a quarter cycle phase difference between them. This signal is in turn added to the luminance signal causing small variations in its amplitude. On a monochrome receiver this results in the small variations in the brightness of the picture which we refer to as the “chroma dot” pattern. A colour receiver, on the other hand, separates out the chroma signal from the underlying luminance using a filter. The values carried in the chroma signal can be retrieved, for example by sampling it at the precise times when the quarter cycle phase difference in the two signals it is comprised of means that one of the signals has a zero value and the other is at maximum or vice-versa. For such decoding of the colour signal to be successful the receiver must have information about the precise phase of its two component signals. This phase information is conveyed by means of a reference “colour burst” sent during part of the transmitted signal that does not contain visible picture information.

The value carried by the luminance signal is referred to as Y. The values carried by two components of the chroma signal are referred to as U and V. In colour television luminance is a sum of weighted values of red, green and blue (R, G and B.) The U and V values are colour difference values based on B - Y and R - Y respectively. From the Y, U and V values a colour receiver can derive the values of red green and blue that make up the displayed picture.

To help compensate for possible phase errors in the transmission of the chroma signal, the PAL system introduces a phase inversion (half cycle phase difference) in the V component (only) of the chroma signal with each line that is transmitted. So for each line of a field the V phase is inverted with respect to the previous line.

My understanding of the mathematics of PAL and the four frame sequence
Simple amplitude modulation of a carrier signal c by a modulating signal m results in a signal whose equation is the product:-

math y = c(A + m) math

where A is a constant. If A has the value 1 the carrier signal is fully present in the resulting signal. If A has the value 0 the carrier is said to be suppressed.

In the case of the amplitude modulated U and V signals which together form the chroma signal, their carrier, which contains no useful information, is suppressed prior to addition to the luminance signal. On a monochrome receiver this effectively reduces the degree to which the picture is affected by variations in luminance due to the chroma signal modulating the luminance signal. In other words it reduces the severity of the “chroma dot” pattern.

If the signal which carries the U colour difference information has an equation like this

math u = U \sin(\omega t) math

where u denotes the value of the signal at time t, and ω is the angular frequency, then the quarter cycle phase difference in the signal which carries the V colour difference information means that it will have an equation like this

math v = V \sin(\omega t - \frac {\pi}{2}) math

which is equivalent to

math v = -V \cos(\omega t) math

The chroma signal is the sum of the U and V signals. Adding the chroma signal to the underlying Y luminance value gives an equation for the value of luminance y actually transmitted (and displayed on a monochrome receiver as the chroma dot pattern) that looks like this

math y = Y + U \sin(\omega t) - V \cos(\omega t) math

However since in PAL the phase of the V signal is inverted on alternate lines, for those lines the equation will look like this

math y = Y + U \sin(\omega t) + V \cos(\omega t) math

The angular frequency ω is the frequency fc of the chroma signal multiplied by 2π. For PAL fc is defined by the equation

math fc = \frac {1135 \times fl}{4} + fr math

where fl is the line frequency which is 625 * 25 = 15625 lines per second and fr is the frame frequency which is 25 frames per second.

This gives fc a value of 4433618.75 cycles per second.

Now if we take any point in a displayed frame, the equivalent point in the subsequent frame is going to be displayed 1/25 of a second later. During that 1/25 of a second the chroma signal will have gone through fc/25 = 177344.75 cycles, so there will be a quarter cycle phase difference between the U signal at any point in the second frame and the U signal at the equivalent point in the first frame.

So if we use t to denote the time elapsed since the start of a frame’s transmission (so that t has the same value for points at equivalent positions in each frame) then if the value for the U signal in the first frame is given by

math u = U \sin(\omega t) math

the value in the second frame will be given by

math u = U \sin(\omega t - \frac {\pi}{2}) = -U \cos(\omega t) math

There will also be a phase difference in the V signal, but due to the fact that there is an odd number (625) of lines in each frame and PAL inverts the phase of the V signal on every alternate transmitted line, any point in the second frame will additionally have a V phase inversion applied to it relative to its equivalent point in the first frame. So if in the first frame

math v = -V \cos(\omega t) math

in the second frame

math v = V \cos(\omega t - \frac {\pi}{2}) = V \sin(\omega t) math

Putting this together, it follows that when the equation for value of luminance y1 in the first frame at a time t relative to the beginning of that frame’s transmission looks like this

math y1 = Y + U \sin(\omega t) - V \cos(\omega t) math

the equation for value of luminance y2 in the second frame at time t relative to the beginning of its transmission will look like this

math y2 = Y - U \cos(\omega t) + V \sin(\omega t) math

We can derive similar equations for the third and fourth frame, after which the sequence repeats itself.

The equations are

math y1 = Y + U \sin(\omega t) - V \cos(\omega t) math math y2 = Y - U \cos(\omega t) + V \sin(\omega t) math math y3 = Y - U \sin(\omega t) + V \cos(\omega t) math math y4 = Y + U \cos(\omega t) - V \sin(\omega t) math

For alternate lines the equations will have the form

math y1 = Y + U \sin(\omega t) + V \cos(\omega t) math math y2 = Y - U \cos(\omega t) - V \sin(\omega t) math math y3 = Y - U \sin(\omega t) - V \cos(\omega t) math math y4 = Y + U \cos(\omega t) + V \sin(\omega t) math

Using the four frame sequence to recover information from film telerecordings
A film telerecording of a television programme captures each frame of the television picture on to a separate frame of film and will capture the chroma dot pattern if it is present. Recovering colour information from the chroma dot pattern is difficult because the telerecording process introduces geometric distortions to the picture, and hence to the time critical chroma signal. Additionally the phase information contained in the reference colour burst is not present.

If the mathematics of the four frame sequence as outlined above is essentially correct then it might be possible to make use of it to recover at least partial geometry, phase and colour information from the picture.

For the method outlined here to be usable, it will firstly be essential that it is possible to track the line structure of the recorded picture. (If this is possible then some of the picture geometry could also presumably be corrected from the line structure, but distortions that stretch and squeeze the chroma signal, and hence affect its timing, would remain.)

Secondly, in general, there would have to be no significant changes in the geometrical distortion taking place over sequences of four frames.

Thirdly the resolution and degree of accuracy with which the luminance of the chroma signal is present in the original picture, captured by the film recording, and sampled by any subsequent digitization must be sufficiently high (or there must be methods that can be used to compensate for this if it isn’t.)

Assuming all the above, the first thing to notice from the equations describing the four frame sequence is that for any point in the original picture that is unchanged (i.e. has the same colour values) in one frame and the next frame but one, the value of Y for that point will be the average of its chroma dot luminance values in those two frames.

For example

math Y = \frac {y1 + y3}{2} math

Using this could give us a way of identifying, with reasonable confidence, parts of the original picture that do not change over a four frame sequence because for those parts the following equation holds

math \frac {y1 + y3}{2} = \frac {y2 + y4}{2} math

This would also give us the values of Y for those parts of the picture and hence give us separation of the chroma signal from the luminance signal in those areas.

If we can use this method to identify parts of the picture that don’t change over four frames then the second thing to notice is that the absolute value of the chroma signal will be the same for these areas across all four frames at precisely the points where the absolute values of sin(ωt) and cos(ωt) are equal. These are the points where

math \omega t = \frac {(2n + 1) \pi}{4} math

(where n is an integer)

So at these points the following equation holds

math math
 * y1 - Y| = |y2 - Y| = |y3 - Y| = |y4 - Y| = (\pm U \pm V)\sin\frac{\pi}{4}

So if we can identify, with sufficient precision, the points within those areas where these absolute values are the same (and non-zero) then we will have recovered potentially useful information about the original geometry of the picture and the phase of the chroma signal. From this we should then be able to recover U and V values for those parts of the picture. (For example, without geometric distortion, at points on each line lying exactly half way between the points identified as above, the absolute value of the chroma signal will be equal to the absolute value of either U or V.)

An example may help to clarify this. Suppose we have two consecutive frames from a sequence and we have separated luminance and chroma for the parts of the original colour picture that are static across those two frames. So we know the values of Y for those parts of the original picture. Suppose we are tracking a line from the static part of the picture in each of those two frames. The equation that describes the line in the first frame may be any one of the eight equations detailed above, depending on whereabouts the line is in the frame and where the frame is within the four frame sequence. It does not actually matter which equation describes the line, as what is important is the relationship between the equations describing the same line in the two consecutive frames. Let’s assume that the equations are:-

math y1 = Y + U \sin(\omega t) - V \cos(\omega t) math math y2 = Y - U \cos(\omega t) + V \sin(\omega t) math

or

math y1 - Y = U \sin(\omega t) - V \cos(\omega t) math math y2 - Y = - U \cos(\omega t) + V \sin(\omega t) math

The known Y term, as we have said, represents the luminance in the original colour picture, whereas y1 and y2 represent the luminance in the first and second frame monochrome respectively where the embedded chroma signal has caused variations in the luminance.

Now consider what happens at the point where

math \omega t = \frac{\pi}{2} math

At this point

math y1 - Y = U math math y2 - Y = V math

and since we know Y, y1 and y2, if we could identify this point, we could infer the values of U and V and hence the colours at this position. There are four such points in each signal cycle. They are the points where

math \omega t = \frac {n \pi}{2} math

(where n is an integer.)

Note that, depending on the value of n, the U value will either be obtainable from frame 1 and the V value from frame 2 or vice versa - in fact they alternate with each increment of n. Similarly, depending on the value of n, the values obtainable will either be +U or -U, and +V or -V.

So, to achieve colour recovery, what we have to do is identify these points. As indicated above, we can do this by identifying the points where the absolute values of the chroma signals are the same in both frames, since the points we want lie exactly halfway between those points (assuming no geometrical distortion.)

The absolute values of the chroma signals are the same when

math \omega t = \frac {(2n + 1) \pi}{4} math

If we identify a point where the chroma signals are equal in value but opposite in sign, for example, assuming our two line equations, the point where

math \omega t = \frac {\pi}{4} math

then at the point corresponding to the next value of n the signals will be equal in both value and sign. Exactly halfway between those two points the U value will be obtainable from the first frame and the V value from the second frame. (This is the case whichever pair of equations happen to describe our line in the two frames.) Conversely if our first identified point is a point where the signals are equal in value and sign, at the halfway point the V value will be obtainable from the first frame and the U value from the second frame.

Unfortunately, although this method lets us obtain the absolute U and V values at the identified points there seems to be no obvious way to determine whether we are getting a +U or a -U (and a +V or a -V.) However the signs alternate in a predictable fashion so if we pick the initial polarities it should be possible to determine all subsequent polarities relative to that. Picking the initial polarities (out of the four possibilities) would then be a matter of trial and error, assuming that, on whatever film sequence we are trying to recover colour from there are clues (eg fleshtones, sky or whatever) to enable us to identify which is the right combination.

To conclude, its worth reiterating that the method described here only describes colour recovery over static parts of a picture (across the frames being utilised) No consideration has been given to the possibility of extrapolating and interpolating this recovered geometry, phase and colour information to infer U and V values for those parts of the picture which do not remain unchanged across the frames in question

-- Andrew Browne 11/06/2007, Revised 27/01/2008