...continued from Full gamut colour recovery

Richard Russell, 30th May 2008

A significant weakness of my Full Gamut Colour Recovery method is that it needs similar chrominance on both fields. If one field's chroma differs substantially from the other field, the quadrant determination process will fail in that region.

There are two main circumstances when this can occur:

  • A coloured object moving quickly across a differently-coloured (or uncoloured) background, or an uncoloured object moving quickly across a coloured background.

  • A cut between dissimilar frames, when the picture-phase of the Film Recording is wrong (so the cut appears to occur between fields).

I had originally hoped that such circumstances would be rare, but unfortunately they both occur in the Top of the Pops test sequences.

I have been trying to think of a workaround which would preserve the intra-frame nature of my process, but so far without success. It may be necessary either to put up with this shortcoming, or to use information from previous frames to overcome it.

31st May 2008

I am convinced that there is no way of making my process tolerant of the two fields being different (as far as chroma is concerned) whilst remaining intra-frame. The only way to cope with this shortcoming will be somehow to use information from other frames, but before even contemplating that I need to know when it will be necessary. In other words I need some way of determining that the chroma content of the fields is significantly different.

A 'direct' method, such as comparing the content of the lines in one field with the lines in the other field, is ruled out because geometric distortions and vertical sub-Nyquist sampling mean that the two fields cannot be separated (if they could this whole process would be made much easier!). So we need to find a method which works even in the presence of such distortions.

Fortunately there is a way. Referring back to my experiments page, where I showed that the 2D chrominance spectrum includes components at both 72 and 216 cycles-per-picture-height, some useful information can be gleaned from the relative intensities of these two spectral components. It turns out that if the two fields contain similar chroma the 72 c/aph component is smaller than the 216 c/aph component, however if the two fields are very different the relative amplitude of the 72 c/aph component increases.

So it ought to be possible to determine whether the fields differ significantly by measuring the ratio of these two components. For a demonstration that there is promise in such a method consider frame 332 of the 'Magenta Key' sequence:

mag332.jpg

Here you can see that on the left of the image there is fast vertical movement, where the two overlapping regions correspond to the two different fields. Where they overlap the content of the two fields is similar; where they don't, only one field contains chrominance (note that this image demonstrates, contrary to some claims, that the two fields are reasonably equally represented in the Film Recorded frame).

This false colour image shows the ratio between the 72 c/aph and 216 c/aph spectral components:

mag332df.jpg

You can see that the regions where the fields are very different are picked out in magenta and grey, whereas the regions where the fields are similar are mostly green, cyan and blue.

Whether the discrimination is reliable enough to be put to useful purpose remains to be seen.

1st June 2008

Initial experiments with using the information in other frames, to help when the fields are dissimilar, have been disappointing - overall performance worsened rather than improved. Indications are that the geometric distortion isn't sufficiently consistent between frames, quite possibly because of positional instabilities such as film weave.

This doesn't bode well for alternative approaches to Full Gamut Colour Recovery, which may rely on measuring the distortion sufficiently accurately to perform a 'direct' demodulation of the chrominance. I think it likely that they will have no alternative but to measure, and correct for, film instabilities; something I don't want to consider at this stage.

I propose, for the time being, to leave my process as intra-frame. Although it has some limitations (especially when the fields are dissimilar), performance is generally pretty good.

3rd June 2008

I've been fiddling with my process to see if I can squeeze any more performance out of it, and the conclusion I've come to is that I can't! It's not difficult to make it work a little better on a particular frame or two, but invariably another frame or another sequence gets worse at the same time.

There's little doubt that one reason for the less-than-perfect results is the dependence of my method on the two fields being similar, for which I haven't managed to find a workaround. However I suspect another major factor is simply the relatively poor quality of the source material, particularly in respect of geometrical stability and uniformity of focus.

Whilst an alternative approach to Colour Recovery might well be less sensitive to the fields being dissimilar, imperfections in the source material are always going to be a challenge. I look forward to seeing what other methods can achieve, especially if they are an improvement on mine!

I intend to do no more work on my process until some more material becomes available. If anybody wants to play with my software - it runs under Windows - please contact me to request a copy. Sorry, I'm only prepared to release the executable program at this stage, not the source!

7th June 2008

I've been hunting around for an ideal test image, both to demonstrate my Full Gamut Colour Recovery process, and as a useful 'artificial' source image for anybody else working on their own method. I think I've found it in the Boy with Toys test slide; it contains a good selection of saturated colours and some more subtle tones in the background. Here are both the original and the colour-recovered version:

boytoys.jpg boytoys_rtr.jpg

I've uploaded the 'artificial film recorded' (PAL coded) version here so that anybody who has downloaded my Colour Recovery program can confirm the above result at full size, and anybody working on their own method can see how it compares. The file format is raw, headerless, 8-bit luminance only (2,073,600 bytes).

The file was upconverted from a 12 MHz sampling rate so the chroma patterning is by no means immaculate, but then neither is the chroma in genuine Film Recordings! I would expect any promising method to do a good job.

9th June 2008

I have now had the opportunity to try my process on a frame taken from an episode of Doctor Who. It completely fails. No (correct) colour is recovered at all from the entire frame. This is disappointing and very discouraging.

11th June 2008

Lest anybody gets too depressed by the above comment, I should add that the Doctor Who frame that James sent was from the opening titles, with Jon Pertwee's face immersed in the swirly 'video feedback' effect. My suspicion is that my process failed in a similar (but worse) way to the 'JS Lights' TOTP sequence.

Later Scrub that. Apparently the opening titles came off film, so there cannot be large differences between the fields. Therefore the complete failure of my process on this frame remains a mystery.

It occurs to me that I've not had the chance to try my program with a sequence containing a camera pan (or tilt). Given that my process doesn't work well when the two video fields (in a frame) are significantly different, there is a distinct risk that it might object greatly to pans. That might be cause for concern.

My process has a long way to go before it could be usable on 'real' material such as entire programmes, and it may well never be possible to achieve the level of performance that would be required. I'm sorry if, in my enthusiasm to show it working at its best, a different impression has been given.

12th June 2008

James has sent me another frame from the Doctor Who episode, this time a reasonably stationary studio shot with no particularly notable characteristics that might cause concern. Again, my colour recovery program fails completely when presented with this frame. The cross-correlation process, which is supposed to measure the horizontal component of the geometric distortion (and which works well with both the Top of the Pops sequences and the 'artificial' FR images), can't detect the characteristic pattern of chroma dots which should be there.

Although, by eye, there is obviously chroma patterning in this frame it appears to be so different from what it should be that my program cannot recognise it. The picture exhibits severe line-pairing (or mis-interlacing) near the top of the frame, and it may well be that (although less pronounced) the entire frame is affected to such a degree that my process fails. The likelihood is that the entire episode (and possibly the entire series) is similarly affected.

Friday 13th June 2008

Here you can see for yourself the nature of the problem. Firstly here is an X-displacement 'contour map' for one of the Top of the Pops frames (frame 763 of the Magenta Key sequence):

mag763dx.jpg

Each coloured band corresponds to a distortion in the horizontal direction amounting to one (HD) pixel. Here is the equivalent map for the second Doctor Who frame:

drwhodx.jpg

There is scant evidence of any contours at all, and my process simply gives up. The contour map is generated by means of a 2D cross-correlation between the residual chroma in the frame and a distorted (but otherwise 'perfect') subcarrier reference. The absence of any structure in the Doctor Who version indicates that the subcarrier patterning bears little resemblance to what it ought to be.

Quite obviously this is no minor problem with a quick fix!

Later

Some success. Looking closely at the chroma patterning in the Doctor Who frame it is apparent that high vertical frequencies (e.g. 216 c/aph) are greatly attenuated compared to low ones (72 c/aph). One obvious possible explanation for this is that the CRT focus was not very good, especially if the 'spot' was very elliptical with the vertical diameter greater than the horizontal diameter. Another contributory factor could be a degree of 'line pairing', again causing a vertical low-pass-filtering effect.

On that basis I've tried artificially filtering my 'reference' subcarrier so that it too lacks high vertical frequencies, and indeed that considerably improves the results from the 2D cross-correlation so that contour features can be seen. I will experiment with this approach to see what can be achieved, but the initial results have at least dissipated some of my pessimism.

14th June 2008

Another possible reason, in addition to those mentioned above, why the Doctor Who frames lack the high vertical frequencies might be that 'spot wobble' (designed to mask the line structure) has been 'overdone', causing adjacent lines to overlap rather than simply to touch. Any or all of the causes may be present to some degree.

Although pre-filtering the subcarrier reference (so that it doesn't include the 216 c/aph vertical frequency) significantly improves performance with those frames, it comes at a cost: it's much more likely that diagonal luminance edges will be mistaken for chrominance. Unfortunately the consequences for the overall colour recovery process can be quite severe (much worse than just a little 'cross colour').

So, although I have now managed to recover some colour from the Doctor Who frames, results are never going to be as good as with the Top of the Pops sequences - and particularly the 'artificial' Film Recorded frames - which have a sufficiently good vertical frequency response to require both the 72 c/aph and 216 c/aph components to be present. Since luminance is unlikely to contain both those frequencies the likelihood of it being mistaken for chrominance is much reduced.

It remains to be seen whether the degraded performance is good enough for the resulting colour-recovered programme to be usable.

16th June 2008

Here is an image which originated as a 1920 x 1080 scan of the full film width (therefore including both picture and optical soundtrack). The active picture, with a width of about 1408 pixels, has been extracted and stretched to fill the full frame, as required by my process. The residual chroma was in any case relatively poor, and won't have been improved by the additional re-sizing step.

temple_rtr.jpg

This frame has admittedly been carefully chosen as one of the better ones, but it does show what my Colour Recovery process can achieve even when the source is of marginal quality.

17th June 2008

I have uploaded the first minute or so of the Paul Temple episode here (about 50 Mbytes); as usual it's an MPEG4-coded QuickTime movie. The saturation has deliberately been left quite low, because it disguises some of the Colour Recovery errors, and anyway I think it suits the rather grainy material.

This is probably typical of the kind of Colour Recovery quality one might expect from relatively poor source material, using my process. There are a number of errors which would have to be corrected before it would be suitable for broadcast or DVD authoring. Something that would be invaluable in such an editing process would be the ability to flip the signs of U and V within a specified region - is that a facility usually provided?

Continued at Full gamut colour recovery (yet more)...