Investigations+into+intra-frame+decoding

Andrew Browne 4th December 2008

When I created this page back in April, I drew a hasty and incorrect conclusion after a limited analysis of the chroma signal pattern in the TOTP film recording. Although I retracted this conclusion in subsequent discussions, I thought, in the interests of accuracy, that it was about time that I corrected this page, especially as the wiki is drawing some interest at the moment due to the successes that others, in particular Richard Russell, have since had with their colour recovery techniques.

In addition to correcting my original conclusion, I have incorporated a couple of images, which, I think, illustrate how the chroma pattern I was examining actually arises. The first is taken from a 1920x576 simulated chroma pattern produced by software, and the second roughly simulates the effect of scanning that pattern at 1920x1080, which is the resolution of our HD scan of the TOTP film recording. I also briefly discuss how features of the HD scan might conceivably be used to help identify the original picture scan line structure, even at a vertical resolution of only 1080 pixels. Unfortunately I have not as yet found the time to investigate whether that idea can actually be taken very far in practice.

Most of the revisions I have made come after the section on the mathematics.


 * Introduction**

I thought it might be useful to write up some of my ongoing investigations into improving on the localised decoding technique which I initially proposed here, and for which an initial proof of concept experimental implementation is described here. The technique as proposed used the phase relationships between the chroma signals at the same positions in consecutive frames to infer the magnitudes of Y, U and V at those positions (assuming Y, U and V are unchanged at those positions across the frames in question.) I initially proposed this approach because the phase relationships between the chroma signals across a four frame sequence are very simple, involving exact integer multiples of a quarter cycle, giving, for example, a precise inversion of the chroma signal between one frame and the next frame but one, and also because we should expect that much of the time there will indeed be no change in Y, U and V at corresponding positions across a four frame sequence. However a potentially significant drawback of this technique is that it relies critically on the digitisation of the frames in question having sampled the chroma signals at the same (to a high degree of precision) relative positions in each frame. This is a pretty tall order, especially when we consider the frequency of the chroma signal and the effect of instability in the film recording. The initial experiment showed that this was indeed something of an issue - although the results were actually better than might have been expected given this consideration. James Insell has proposed instead using the relationships between adjacent scan lines in a single frame to see if better results can be obtained by this approach (which in fact is essentially his original proposal for localised decoding) and one of the things I have been doing recently is looking into this possibility.

I shall begin by looking at what we should theoretically expect the relationships between adjacent scan lines within a single frame to be. Here we have to remember that a frame is composed of two fields, all the odd lines of the frame coming from one field of the frame and the even lines coming from the other. I shall then take a look at what we actually see when we look at our "Top of The Pops" film recording sample.


 * Mathematics of the chroma signal across consecutive lines within a single frame**

Let's take a line of our frame from the first field to be displayed and where the contributions of the Y value and the chroma signal can be described by the equation:-

math y1 = Y + U \sin(\omega t) + V \cos(\omega t) math

where t here denotes time relative to the start of the line.

The next line in the frame will be from the second field. This means that there is a delay corresponding to 313 lines between the display of the first line and the display of the second line. The phase of the V signal will also have inverted 313 times during that period, resulting in a net V phase inversion between the two lines.

As the line frequency is 15625 lines per second and the chroma signal frequency is 4433618.75 cycles per second, a 313 line delay expressed in terms of chroma signal cycles is:-

math \frac{313}{15625} \times 4433618.75 = 8841.2508 math

So there is an approximately quarter cycle phase difference between the U signal on the second line and the U signal on the first line. Taking the V phase switch into account, and again taking t to denote the time relative to the start of the current line, the second line can therefore be describe to close approximation by the equation:-

math y2 = Y - U \cos(\omega t) - V \sin(\omega t) math

The third line is from the first field. Here there is a delay of just 1 line relative to the first line, and there is a V switch also relative to the first line. The delay in terms of chroma signal cycles is 283.7516, giving a phase difference of approximately three quarters and an approximate equation of:-

math y3 = Y + U \cos(\omega t) + V \sin(\omega t) math

So we can see that the chroma signal in line 3 is an almost precise inverse of the signal in line 2 (assuming the lines have the same hue.)

By continuing this reasoning we can derive approximate equations for 8 consecutive lines in a frame, where the odd lines come from one field and the even lines from the other. After 8 lines the sequence repeats intself:-

math y1 = Y + U \sin(\omega t) + V \cos(\omega t) math math y2 = Y - U \cos(\omega t) - V \sin(\omega t) math math y3 = Y + U \cos(\omega t) + V \sin(\omega t) math math y4 = Y + U \sin(\omega t) + V \cos(\omega t) math math y5 = Y - U \sin(\omega t) - V \cos(\omega t) math math y6 = Y + U \cos(\omega t) + V \sin(\omega t) math math y7 = Y - U \cos(\omega t) - V \sin(\omega t) math math y8 = Y - U \sin(\omega t) - V \cos(\omega t) math

Ignoring the Y term and denoting the chroma signal on line 1 by C and the chroma signal on line 3 (where the sine and cosine terms have swapped between U And V) by C' we see that we essentially have the pattern:-

C, -C', C', C, -C, C', -C', -C

So if we start with line 2, for example, we can see the pattern as essentially a sequence of pairs of lines, where the second line of the pair carries the approximate inverse of the chroma signal in the first line of the pair.

If we look at the sample chroma signal patterns near the foot of [|this page], we can see that they do indeed seem to be consistent with this.


 * Examination of the "Top Of The Pops" film recording sample**

The image below shows a 16x enlargement of the pixel pattern in 32 X 30 pixel area of approximately constant colour from the first frame of the HD "Top Of The Pops" film recording sample.



(The top left coordinate of this are in the HD sample is (604, 324). It is part of the square reddish backgound on which the large numeral "1" appears.)

It should be apparent that there is a roughly repeating pattern of luminance values, where the horizontal interval over which the pattern repeats is close to 8 pixels and the vertical interval is close to 15 pixels.

The 8 pixel horizontal interval is consistent with the length of the chroma signal cycle, since we know there are approximately 230.5 chroma cycles in the visible part of a TV picture and an HD image is 1920 pixels wide. As the image above is of an area 32 pixels wide it is therefore roughly 4 chroma signal cyles in width.

The 15 pixel vertical interval is consistent with the 8 line chroma signal pattern we discussed above. As the visible part of a TV picture is 576 lines in height and an HD image is 1080 pixels high, we should expect an eight line band in our sample to have a height of about 15 pixels. As the image above is of an area 30 pixels high, it is therefore roughly the height of 16 scan lines or twice the height of the repeating 8 line pattern.

At this point, I originally wrote:-

"It should also be fairly apparent that the chroma signal pattern over a height of 15 pixels does not match what we should expect to see according to the calculations we made above. We do not seem to be seeing a sequence of pairs of lines where the second line of the pair carries a signal which is the inverse of the first. Instead the nearest inverses of each chroma signal seems to be at a vertical distance of about 7 pixels either side of it, corresponding to a distance of about 4 scan lines.

"Examining our equations shows that each line should indeed have an inverse at a distance of 4 scan lines above and below it. These inverses come from the same field as the line in question. However there should also be an intermediate inverse either immediately above or below each line which comes from the alternate field and these do not appear to be present in our sample. Indeed it looks very much as though our sample only contains a significant chroma signal pattern from one of the two fields in the frame, so that what we are seeing over a vertical distance of 15 pixels are significant chroma signal signal contributions from just 4 of the 8 lines that that distance spans."

This conclusion was quite wrong. I believe the following pair of images shed some light on the appearance of the pattern.

First an enlargment of a 32x16 pixel sample taken from a 1920x576 software generated chroma pattern produced from the colour on the corresponding part of the frame in the colour VT copy of the TOTP episode:-

Here each of the 16 pixel rows corresponds to a picture scan line. For this particular source colour it so happens that the peaks and troughs of the chroma signal pattern on some adjacent lines occur at reasonably similar positions, so we can see that the large "chroma dots" in our film recording are actually formed of two smaller "dots" on neighbouring scan lines.

Now here is a 32x30 pixel sample taken from a resizing of the 1920x576 pattern to 1920x1080 using bilinear sampling:-



This resampling is intended to provide a crude simulation of the effect on the chroma pattern in the film recordings of scanning at 1920x1080. It can be seen that the above pattern does look pretty similar to the pattern in the TOTP recording.

It should also be apparent, even by visual inspection, that the resampling has affected the overall luminance value and caused some distortion of the embeded chroma signal.

A feature that can be seen both in the above image, and in the film recordings is that there are pixel rows where the variation in luminance along the row due to chroma is much less than the neighboring rows above and below. I believe these rows generally correspond to positions where the scanning/sampling is effectively averaging the values from neighbouring scan lines whose chroma signals are in antiphase, in accordance with the pattern of pairs of inverse signals that we identified earlier. The pixel rows on either side of such lines should therefore correspond to the scan lines in question. It is possible that this could be an aid to identifying scan line structure in film recordings at this resolution, but further investigation would be needed to see how well that actually works out in practice.

-- Andrew Browne 8th April 2008, revised 4th December 2008