Algorithmic+approaches+-+thoughts

Andrew Steer - 6 April 2008

Ok, it's about 8-9 years since I wrote my own [|software PAL colour decoder]. I'm a little rusty on it now.

I've just met up with James, and had a chat about where the group has got to. James has supplied me with two 1080 (HD telecine film-scans) frames of the Jimmy Saville image, and two of much the same image without Jimmy. It is unfortunate that there is so much black in these frames, but I guess we have to live with that for now!

The PAL chroma signal in these scans definitely looks quite strong, and I'd say it's very reasonable to say that it should be possible to decode the colour. The question, of course, is //how//.

I still maintain that the ideal way to do it would be to recover the original line structure and precise image geometry (using the embedded PAL signal at 4.43MHz as a guide). I can see that this may be tricky in practice, not least because a 1080 line scan still only barely resolves the original line structure. It would be interesting to see a very high resolution scan of the film to see just what is there... but while possible for single frames, I've no idea whether this would be practical for whole programmes anyway. James says there's some moves afoot on getting a "2K" test-scan made...? But I thought "2K" was only (2048 x 1080) anyway? Recovering the line structure would also have the advantage of recovering 50Hz fields properly, without having to use VidFIRE - which should reduce motion-blur. If there's any chance of doing this, I still think this should be a goal. With a restored image, you've essentially regenerated the original raw PAL signal, so a conventional (software) PAL decoder can bring back the colour.

It also occurs to me that it should be possible to locally detect the "comb" effect you get from motion on merged interlaced fields, and use this as a strong hint for VidFIRE-like processes.

I hear the other Andrew (Andrew Browne) has been working on a "local colour decoding" approach. I know no more about this than is mentioned on the pages here, i.e. not a lot. From my past knowledge of PAL I know that while single lines are coded with varying amplitudes of quadrature "U" and "V" phase subcarrier (and need a phase-locked reference -the colour burst- to make sense of), because of the subtle phase-relationships between lines, looking across several lines, colours make recognisable 2D patterns. As I wrote, on my own [|webpage] years ago, "Of note, U-axis colours (pure yellows or blues) appear as diagonal stripes \ \ \, while V-axis colours (pure reds or greens) appear as diagonals /// / /.// Mixture colours, such as orange (red+yellow) give rise to a superposition of the two, ie cross-hatching." Now, in the absence of a phase reference, you still know that \ \ \ stripes are either yellows OR blues, and / / / are red OR green. There's a 180-degree phase ambiguity. Right now, I'm not sure whether there's any way to resolve this. It may be possible to get some hints from the inter-line or inter-frame comparisons - maybe from mixed U+V colours? Even if it does have to be manually determined, it should only be necessary to do once per shot, because algorithms could follow through from one frame to the next. If a poor quality home VHS recording existed, then this would be sufficient to determine the phase too. It might also be possible to do statistically, eg because flesh tones are more probable than the complementary desaturated ugly green colour.

When I first heard about "local colour decoding" (ie in the absence of external phase reference, owing to absence of burst and image distortion), I was concerned that if you had small areas of colour, or coloured patterns, you might run into great difficulty. I now suspect it might be possible to establish a local phase-lock from the dominant colour, which would then be the reference which enabled nearby smaller patches of colour to be decoded.

I hope it might be possible to locally-recover the horizontal, U, and V slope-angles and use these as a guide to geometrically correcting the image. This will be very difficult with large picture-areas without chroma.

If you can't (or don't) geometrically correct the image, you want to try ("local") decoding on the HD scan at its native resolution. What you could do is to locally regenerate a (2D ???) U/V phase reference - //phase-locked to the image// where-possible. You'd then use digital "product demodulation" to recover the U,V, and luminance signals cleanly. *

Now, when I did PAL before, I was only considering single-fields. In this problem, we have the two fields interleaved (and not easily fully separable). I need to re-discover my copy of "Broadcast Television Fundamentals" by Micheal Tancock - excellent guide for the "analog" background to this project, and familiarise myself with the expected patterns and relations for interleaved fields. Alternatively I can dig out my copy of RT Russells program and generate some references that way.

When I first saw the full res scans, I thought that this might be harder than I thought - that I thought maybe I'd "bitten off more than I could chew". But having written this page, especially the paragraph above marked * I'm getting quite excited about it.

And now to bed...

Andrew Steer - 7 April 2008


 * Further facts and observations**

A) Local recovery of the (2D) U and V axis will yeild diagonal stripe patterns, further: 1) At the nominal vertical positions of the scan-lines, these will be in quadrature (90 degree phase offset) when resolved horizontally 2) In practice owing to the line-doubling from the two interleaved fields, the actual scan-lines will be a half-line above and below the "fit" nominal-position 3) (In areas with mixed U/V colour, this may help with resolving the scanlines)

B) Vertical wandering of the original scan-line structure relative to the HD film-scan would result in (90-degree) "phase-shifts" of the horizontal Fsc, if analysed naively in 1D C) It is intriguing to note that in the non-chroma (white/grey) figures in the right of the picture alternate scan-lines (ie fields) appear to have a different brightness. It would be interesting to do a simple 1D chroma notch filter over the rest of the image and see if this luminance differential is visible over the whole picture. This may be another "clue" to scan-line recovery.

D) Overall, I'm thinking that full/complete scan-line recovery would be nice, but I'm having doubts that it would ever be robust enough. If we relied on it, and it failed (mis-aligned), it would cause a lot of problems. Conversely, "in-situ" (ie native at HD resolution) colour-decoding using 2D filters might sacrifice a little vertical resolution, but should be far more robust.

E) I do think there would be merit in "local" scan-line recovery (or at least scan-line awareness), and that this could significantly improve the signal-to-noise ratio of decoded colour when compared to a more general (scan-line oblivious) 2D chroma/comb filter. It should also improve the filtering out of the colour signal from the target luminance signal while maintaining maximum horizontal luminance resolution (bandwidth). This scan-line awareness might be derived from comparison of a simple 1D horizontal chroma filter of adjacent lines to see which match and which don't and/or by trial-and-error matching of discrete (scan-line) staggered 2D chroma-filters rather than smooth diagonal filters and/or see point (C) above.

Yesterday I was afraid that although "local colour decoding" would get some initial results fast, it might prove to be an evolutionary dead end. I've now come around to the position that "local colour decoding" (as I envisage it in my head - I am unaware of the details of Andrew Browne's approach) is the way forward.

I need to start coding soon, because until I have something to show, all these ideas are merely "talk"! "The proof of the pudding is in the eating!"

...

Just had a further close look at the source-images (HD scans). The subcarrier has a typical pitch of around 8.0 to 8.1 pixels. (This makes the scan roughly equivalent to a 35.7MHz horizontal sampling frequency.) Seemingly the pitched is stretched to as much as 9.0 pixels in the extreme top left of the image though. This gives an impression of the worst-case geometric distortion!

Looking to see what happens at the edges of the scans, I tried applying a gamma 1.5 "correction" to brighten the low-greys. I then noticed that even the 6MB ".BMP" images have still previously undergone a block-transform (JPEG-like) coding at some stage. The compression was evidently very mild, but blocking artifacts are evident in the low greylevels if you look for them. This shouldn't be a problem for now, and for proof-of-concept, but it would be good to have truly "clean" images for serious processing if this Colour Recovery really takes off in future!

Andrew Steer 8th April 2008


 * First software trials**

I have got a head-start here, as I'm working by hacking an old version of my PALcolour program. The downside is that soon I'll probably realise I can't remember what half of it does, or will find one change causes unforeseen side-effects - the eternal problem of tinkering with old code!

Anyway first result:

©BBC - Simple 1D horizontal 4.43MHz notch-filtered section of the HD-scanned sample film image (1:1 pixels) [Edit to add: owing to a historical coding oversight this turned out to be a 2MHz bandwidth notch rather than the standard 1.1MHz - the resulting image is softer than it needs to be.]

Note how the line-structure has become more easily perceived. Clearly the alternate field is slightly weaker in brightness. I've noticed the little black pixels in the (orange) block to the left. I'm not sure exactly where they've come from in my processing (shouldn't really be happening) -- but they're not a major issue at this stage anyway.

I wish I could find my photocopy of Tancock ("Fundamentals of Broadcast Television") - I could do with the interleaved field PAL phasing reference-images! It gets complicated to work out owing to the half-line field-offset. There's a second-hand copy on Amazon for £23... Instead I'll try and work it out from my copy of Rec.470. Or just hack something quickly - trial-and-error in the first instance!

Because it was designed for scan-line accurate encoded source images, my existing code makes hardwired assumptions about interline phase offsets if 2D filtering is used (even basic "delay-line PAL" uses 2-line averaging). That'll have to go - or be seriously rehashed.

I'm trying to get my program to do basic 1D PAL decoding (i.e. no interline averaging or anything) with a fake "burst" stuck on the left hand edge. I was hoping to get "rainbow" colours - i.e. all wrong (not properly phase- or frequency-locked), but showing saturation loosely proportional to that intended, and changing hue abruptly where the original colours change. Instead I'm getting very little colour, even with the saturation at "200%". Something has gone wrong, so I probably need to take a few steps back...

That's it for tonight.

Goodnight!

Andrew Steer 10th April 2008


 * Further quick software hacks**

©BBC

Taking the film image, appending an (incorrect) burst sequence, and whacking the contrast control in my PALcolour up to 400% ... ! [This is a hacked version of PALcolour doing just a 1D decode.]

I know that my burst-sequence is wrong, and that with the geometric distortions the phase won't be preserved, so the rainbow mess is not unexpected ;-) Not much practical use, I know, but roughly what I was expecting - so good in that sense.

©BBC - This version with a hard-coded fixed phase reference (rather than relying on a cut-n-pasted graphic burst) This is giving a bit more saturation "out of the box".

Next step is to apply my phase-locking ideas... this will begin to get us towards "true" colours.

...another day!

Andrew Steer 11th April 2008


 * U/V isolation

**

It's still fairly crude, but I've now introduced a simple 2D filter which separates U and V, preventing them mixing. Naturally enough it also much suppresses horizontal colour-banding. The phasing is still not locked, which means that you can get inverse colours (probably U and V can invert independently) and a loss of saturation. You will however note that some of the colours are now getting closer... in some areas there's more reasonable flesh-tones, and the olive-green of the jacket is trying to show through. The (what should be) orangey background to the big square with the "1" is still some way off.

There's still worse cross-colour (ie coloured stripes on luminance edges) than there should be because I'm not using the filtering properly and am instead relying merely on the stronger (brighter) field dominating the weaker one and then cranking up the saturation. This also leads to worse colour SNR than should be possible. My aim at this stage is proof-of-principle - i.e. can we get the //**correct colours**//, not perfection in terms of optimal image-quality!

As above, but applying a gamma-1.3 "correction" to the black-and-white source before colour-recovery. I might have slightly over-done it, but I think it looks a bit more "TV" and a bit less "film" that way.

That's it for tonight.

Andrew Steer 12th April 2008


 * A further quick hack... as an example, ditch the phase-information**

In this case, I've forced +V and -U phase throughout. This is a trick that Andrew Browne has also used with this series of images - successful in this instance, but of course only gives access to one quarter of the total colour palette(!). It is proof however that if we could get phase-locking then there is good hope for colour recovery. To clarify - this result is from **decoding a single frame in isolation**.

I think it probably is still possible to make further progress with phase-locking (and not constraining the sign of U and V) using more "empirical fudges" ... but you would want to employ more detailed unpicking of the line structure to do it properly (I have several ideas on how to do that too).

As on the previous images, cross-colour (luminance edges being interpreted as colour) such as the red halo on Jimmy's right sleeve, is far worse than it need be owing to a large number of shortcuts I've taken. Tricks to improve this (and generally reduce the chroma noise) would involve -at least partial- awareness of the line-structure.

[ For reference, a colour videotape original of this frame can be found on this page of the wiki Experiments in recreation of line-structure through block-matching ]

1:1 pixel sample of image above

My present routine is essentially a single-pass local-decoder, and for decoding the chroma of any one pixel within the image considers nothing more than the data within a surrounding block of 8 lines high by 33 pixels wide (HD-scan lines/pixels). It rides roughshod over the interleaved fields, and probably only works because one field is stronger than the other. For extracting the luminance, at the moment it is using a simple 1D horizontal 4.43MHz notch filter (with 2MHz bandwidth - it should be 1MHz bw, so throwing away sharpness unnecessarily!).

Implementation of chroma "phase(polarity) locking" is going to require analysis of the image on a larger scale, and probably a multi-pass algorithm. I expect this will entail some significant architecture-changes to the program (but hopefully I can also instigate some major speedups (efficiency improvements) in the process, which will be no bad thing). I really haven't got the time to work on such changes in the next few days.


 * The bigger picture**

What have I learned, what does this tell us about the prospects for largely-automatic colourisation of programs in future?
 * 1) While higher resolution scans might be nice (and certainly for "research" purposes), I reckon the 1080-line scans are "good enough". Improved line-structure-aware algorithms could still go a lot further than I've demonstrated so far, with the source-images we have.
 * 2) Unless there's something clever we can pull out of the interlace and V-switch (doubtful), I don't think we can determine the (frame-global) polarity of U and V algorithmically. A human operator would probably need to select this manually (easy - 1-2 pushbuttons). Provided there are no bad edits in the film (or source video) the U and V phase should propagate predictably through the footage.
 * 3) For images with a good wash of chroma, the colour can probably be (phase)polarity-locked within a frame.
 * 4) For images with distant isolated islands of chroma (especially if separated by black with precious-little line structure) then it may prove difficult to reliably polarity-lock the chroma within a frame. Colours could decode with the correct U and V polarity in some parts of the frame, but inverted in other areas.
 * 5) The problem of (4) can be overcome by using geometry-distortion information from other frames //provided// the global distortions are reasonably stable (or can be tracked, or are very predictable based on average luminance for example) from shot to shot. We would need to be able to predict horizontal position to within a quarter of a 4.43MHz cycle, and vertical position to +/-1 original-video scan-lines. Scan-line locking would help, and this would be simplified if one field does prove to be //consistently// darker than the other.
 * 6) Software could (in theory) know where it has "islands" of chroma-polarity uncertainty, and prompt an operator to mouse-click on them to cycle through the four possible U/V polarity-combinations until it looks right. In most cases you would expect these to be reasonably-large islands, and the operator would only have to intervene once per shot. If the scene cuts back and forth between two camera angles, it ought to be possible to reference older shots (maybe even automatically). Although requiring intervention, this would be vastly less labour-intensive than conventional colourisation.

- Andrew 12/4/2008


 * We have ways of making U and V talk**(!)

In discussions with Andrew Browne, I noticed that both our images have extra-saturated reds and skin-tones when the saturation on the olive-green jacket was about right. I hypothesised that the V signal is being accentuated relative to the U. Tried knocking back the V-gain, and the result is a big improvement... making the skin less saturated and the "orange" square a bit more orangey.


 * Yes, this colour //really is// obtained from the film-scan!** **:-)**

[In this version I also tried a 1MHz notch for the luminance - has given greater sharpness, but predicatably in the top-left of the image where the geometric distortion is worse this has also coupled more residual subcarrier into the luminance.]

The question remains of //why// the V-gain is higher than U. (My basic decoder is based upon all the normal standards and matrixes).

It could be due to something like mild diagonal film-blur or an oval spot or something on the telerecorder (certainly the image is blurred that way in the top-left corner)... or perhaps more-likely may be an artifact of "riding roughshod" over the two interleaved video fields in the processing if something else has introduced an asymmetry of some sort. If the latter, I wonder if this can lead us to some other useful information - like absolute U or V polarity???

Here's the videotaped source reference image:-

Andrew Steer - 13 April 2008


 * Further thoughts...**

I still haven't quite worked out why U should be getting artificially suppressed relative to V. There was talk that some telerecording equipment managed to merge the fields by scanning one at high brightness so that it remained by phosphor persistance to merge with the second field scanned during the film exposure. If this is the case for these scans, it has some interesting consequences... If the two fields (at vastly different brightnesses) remain in register, then that may indicate that the overall EHT regulation and image geometry is very stable... which will be very advantageous. On the other hand, it might just mean the regulation is very good at 25Hz (and not necessarily so at low frequencies). If the fields are very different brightnesses then you might expect a larger (and blurrier) raster spot for the earlier bright field. I wonder what happens if there is a very slight horizontal registration-error between the two fields, whether that would upset the U/V ratios when doing the crude cross-field decoding we're doing. Coming off a mechanical VTR, I imagine an offset of say 1/4 of a PAL cycle (0.05us) may not be surprising...? Coupled with the imbalence in field brightness, might this be the answer we're looking for..?

Interesting: comparing the notch-filtered (and horiz-blurred) image alongside the original HD filmscan, it transpires that the //highest-contrast chroma// lines correspond with the overall slightly //darker// scanlines.

Did another test, subtracting two images from each other. Result appears to indicate very good geometry stability - bodes well for largely-automatic whole-programme decoding without "isolated chroma island" U/V polarity uncertainty. Need to try it for real of course, but looks very hopeful for now.

Difference-image showing good consistancy of image-geometry.

That's all for tonight.

15 April 2008 - Andrew Steer

It doesn't matter, but having read this page http://archive.whoniversity.co.uk/tech/film.htm I'm doubtful that our example recordings were made using stored field techniques. Not only does this page imply that stored field was an early technique and soon superceeded, I am also sceptical as to whether stored field would produce the mostly very good quality we have in these film recordings. Surely stored field would need quite a long persistance phosphor, and we're not seeing other evidence of that. I also suspect the registration of the fields would be worse with stored field. Obviously I will stand corrected by anyone who knows for sure what equipment would have been used at the time.

Andrew's colour-recovery progress-blog continues at Single-frame decoding continued...