6th March 2010

Have finally got round to writing this update detailing recent work!


New Software

First I have improved my QBASIC programs, and the new versions are provided here:

These include bug fixes and I've also introduced some running average filtering for the horizontal distortion values.

I also attempted to create a single-pass process in order to do away with Undist2, and that version is presented here:

The double-pass process is only necessary because I've been unable to find an equation that precisely reverses the pre-distortion factor in the initial references. Instead I've reversed it approximately with Undist1, and then used Undist2 to tweak it for full correction.
The following lines of code in Undist1i were designed to get over this problem, and compensate for the pre-distortion precisely:

z = (16 * VAL(csv$(1))) + VAL(csv$(3)) - 960
v = z - .677157215# + (.00627464474# * z) + (.00000547508015# * (z ^ 2)) - (.0000000472352485# * (z ^ 3))
x1(x) = z + 960 + .677157215# - (.00627464474# * v) - (.00000547508015# * (v ^ 2)) + (.0000000472352485# * (v ^ 3))

These are based on the maths of reference frame transformation, and should work perfectly; but for some reason they completely fail.
Perhaps someone cleverer than me can solve this problem!


Matching Film Recordings to existing domestic VT Recordings

I'm fairly confident that my macro for matching the footage frame-by-frame will work well, and within acceptable processing times.
I have yet to test this, and am currently waiting for a suitable VT image sequence.
If it works it should prove particularly useful for the "Dr. Who" story "Ambassadors Of Death", since the VT copy has large sections of unusable chroma. Matching the film recording to the VT geometry should hopefully give a much better chance of colour recovery with minimal quadrant errors.

Problems may arise around the edges of the frame, since the film recording is cropped relative to the VT. It should only be a concern if there are regions at the edge of frame with no luminance detail for ImageJ to lock onto.

I suspect the processing times on a modern computer would be much faster than I've previously reported.
My computer has a single-core 2GHz CPU, with PC2100 memory, and IDE (UDMA Mode 5) hard-drives.
All these components are now obsolete, and slow down my processing times considerably.

After processing, I can recombine the output image sequence into a Y8 video file using the following AviSynth script processed through Virtual Dub:

c = ImageSource("F:\Output\%01d.bmp", start = 0, end = 1024)
ImmaWrite(c, "output filename.y", type="raw", mode="Grayscale", animation=true)

(This requires that the Immaavs set of dll's is first installed into the system path.)

The Y8 file can then be processed through the CR software.

(There are some annoying quirks of bUnwarpJ when it comes to the output image formats produced. This means I sometimes have to run a batch conversion on the output image sequence, before I can recombine them into a Y8 video. But I won't bore you with the details!)


Field Separation tests

I had hoped my field separations would eliminate CR quadrant assignment errors where the two fields are different, and possibly allow restoration of 50Hz motion.
However I've found that the amount of separation I've accomplished is too small to achieve either of these goals.
It's still possible that a higher resolution film scan could allow more separation; but I fear my discovery that some field separation is possible may remain of intellectual interest only, and have little practical use.


Problems with my undistort process

I realised some time ago that my version of Custom Reference does not work. This is because ImageJ can only work on image files; but the reference files are binaries containing both positive and negative values. When I convert them to images, the negative values become positive, and so the structure of the files is lost.
This could perhaps be solved by transforming and rescaling the values so they're all positive before converting to image format, then applying the custom distortion with ImageJ, and then reversing the earlier transform to reintroduce negative values.
However I don't plan to spend anymore time on this, since it's essentially duplicating Richard's work (the only difference is that my custom references would be more accurate, being based on the B-Spline model, and they would not suffer from Runge's phenomenon).

I have also found that my undistort method, using the above QBASIC programs, does not work well with poorer quality film recordings, and can produce some very distorted output frames.
This is because these film recordings tend to have large regions of missing geometry data, which are impossible to interpolate correctly intra-frame.
These regions tend to appear monochrome following CR; though it in fact produces a faint pattern of multi-coloured noise. My guess is that the chroma was less saturated in these areas, and the chroma patterning is only weakly imprinted on the film.

I attempted to get round this problem by filling in the missing geometry data from a frame elsewhere in the footage where the data is complete. However this doesn't work because the transient changes in horizontal distortion are too large: so you end up with unacceptable picture skews.

The poorer film recordings also have multiple quadrant assignment errors in every single frame.
Suprisingly I have discovered that my undistort process can actually iron-out some of these errors so they no longer appear when CR is applied to the undistorted frame. The number of errors is reduced as a result; but errors are by no means completely eliminated.
When deriving the undistort transform from the best frame I could find (call this Frame A), and applying it to the whole of the footage, I found that there were fewer quadrant errors, and a lot less manual intervention required in terms of "quadrant prompts".
However it also imprinted some of the quadrant errors from Frame A throughout the footage!
I suspect my results were no better than Richard can achieve with his Custom Reference method, and most likely worse because of the imprinting effect.


Universal Reference

I was asked to try and find a Universal Reference, instead of deriving Custom References for each scene.
Whilst this may well work for the y-distortion; I have found that it's not possible for the x-distortion since the variations in geometry are too extreme.

The horizontal distortion images below illustrate the problem:
771.jpg
1024.jpg
63.jpg
118.jpg

The first two images are from the same scene with a locked-off camera shot. There's an actor in the foreground who is moving to the right.
You can see how much data is missing, and how difficult it would be to interpolate.
Notice how much the form of the distortion has changed over a very short period of time. Indeed it often changes rapidly from one frame to the next, especially when you have an object moving in the foreground.
I imagine this must limit the usefulness of Richard's Custom Reference method.

The second two images are taken from another locked off camera-shot, with several actors moving in the middle-distance on the rhs of frame.
Note how much the distortion changes even on the left of frame where there is no movement!

Now if we could find a model that would predict these changes in distortion, it would be extremely useful, as we could use this model to derive a unique undistort transform for every frame. Once processed the footage would be distortion free, and thus CR could be carried out with no quadrant errors (except where there are different colours in each field superimposed at the same point).
Unfortunately finding such a model is an extremely difficult task.
If I were to attempt it, I would first try to find a correlation between picture content and distortion as follows:

Find two test frames with a good spread of colour across them, so we can get reliable geometry data.
Then interpolate any missing geometry data (Undist1 and 2 already do this, producing a file called mod.csv containing a complete set of values).

We ignore the y-displacement, since this is fairly consistent throughout the footage.

We then have the following scalar fields:
For each quadrant block:
D1(x,y) = x-displacement of test frame 1 (in HD pixels)
D2(x,y) = x-displacement of test frame 2 (in HD pixels)
L1(x,y) = luminance of test frame 1 (averaged over each quadrant block)
L2(x,y) = luminance of test frame 2 (averaged over each quadrant block)
f(x,y) = scalar field of weighting constants (since the influence of luma on the distortion may well vary depending on the coordinates)

My hunch is that the distortion will be proportional to the square of the luma, which would give the following equation:
D2(x,y) - D1(x,y) = f(x,y)(L2(x,y)^2 - L1(x,y)^2)

Using our test frame data we can solve for f(x,y), and hence we would have an equation to predict the x-distortion in every frames for that particular piece of footage.
(To be more rigorous, we should really take multiple test frames and fit the data to a generalised polynomial series in L1 and L2, but then it starts to get really complicated!)

However this formula is not good enough!
We can see from the second pair of images above that the distortion changes radically on the lhs, even though the luma remains constant in that area.
I suspect this is because the change in luma on the rhs of frame is affecting the distortion on the lhs!
So we would need to reflect this symmetry in our equation, perhaps as follows:

L1r(x,y) = luminance of test frame 1 reflected around the vertical centre-line of the distortion pattern
L2r(x,y) = luminance of test frame 2 reflected around the vertical centre-line of the distortion pattern

Our equation then becomes:
D2(x,y) - D1(x,y) = f(x,y)(L2(x,y)^2 - L1(x,y)^2) + g(x,y)(L2r(x,y)^2 - L1r(x,y)^2)
where g(x,y) is a second set of weighting constants.

To solve for both f(x,y) and g(x,y) we would need simultaneous equations, so we would have to use at least three test frames.
You can see how impossibly complicated it becomes!!

The make things still worse, here is a horizontal distortion image from a different film recording:
totp_ch08.jpg
This one is not symmetrical about the centre-line; but rather around a diagonal line drawn from bottom left to top right.
This arises because the y-displacements are more diagonally orientated for this particular film recording.

I have concluded that I'm not clever enough to develop a suitable model, even if I had the time (which I don't!)
Perhaps someone else would like to take up the challenge. (As a first step I would try removing the vertical distortion only, and see if I could spot any obvious pattern in the horizontal distortion.)

The thought occurs that someone with an expertise in CRT design would probably find this quite easy to solve.
When designing a tube, you would presumably have a formula for the electric field E(x,y,z,t,i), and use it to calculate the beam deflections to assess how the linearity performs. (N.B. i = the beam current.)
In this case we have the opposite situation: we know what the beam deflections are in our test frame, and we need to deduce a formula for the electric field. This could then be used to calculate beam deflections for all subsequent frames.
The time evolution of the field depends on the waveforms of the driving voltages for the field coils, which in turn depend on the way that particular film recorder has been adjusted by it's various rheostats. The beam current would also affect the field pattern, because the electron beam itself has a field, and the amount of current drawn by the beam probably affects the power supply to the field coils.


17th April 2010

I have now conducted an experiment with my frame-by-frame matching process, using an FR and corresponding domestic VT.
As usual it was harder than I expected it to be!


Cropping problem

The initial results were very poor, and I quickly discovered that the variable cropping was confusing the pattern-matching process.
So I masked off the head-switching area of the VT, as well as a small slither at the top of frame.
I also masked off the right-hand-side of the FR, since (surprisingly) it contained more active picture content than the VT in that region.
(I achived this using multiple instances of the "fill" filter in Virtual Dub.)

This solved the problem, and the pattern-matching began to work well.


Input format

The next problem to solve was choosing a suitable format for the image sequences.
I began by using 8-bit bmp's; but this produced an unexplained rapid fluctuation in the black levels on the output sequence.
So I then switched to 24-bit bmp's, which resolved the issue; but led to a glacial 3.3 minutes per frame processing speed!

Next I tried 8-bit tif's, which reduced the processing time to a more acceptable 1.25 minutes per frame.
However the output tif's were about 11 times larger than the inpit tif's, and could only be opened in ImageJ (other image viewers didn't recognise them). Therefore I had to run ImageJ's batch conversion macro and convert them back to bmp format, before I could recombine them into a raw video-stream.

Since this is not very convenient I tried using 8-bit png format for the image sequences, and this worked perfectly, producing 8-bit output png's that can be directly combined into a raw video-stream. Processing times were circa 1.25 - 1.5 minutes per frame depending on the processing settings used.


Effect on CR

As a control for my experiment I created a simple undistorted version of the FR, based on the chroma of one test frame (similar to Richard's undistort solution).
I then performed CR on the raw FR, the simple undistort of the FR, and the frame-by-frame undistorted version.
Since the idea of frame-by-frame processing is to reduce the amount of manual intervention required, I unchecked the "Quadrant Prompt" option in all cases.

Scene 2 of the footage was particularly problematic for the CR process. The raw footage failed completely to get the right quadrants. The simple undistort produced some improvement; but I was pleased to discover that the frame-by-frame process produced by far the best result. Nearly all the quadrants were correctly selected without any manual intervention.

Scene 3 was a different matter, and on this I found the frame-by-frame undistort introduced new errors, with the simple undistort giving the best results. The problem was an area of low texture immediately behind a moving actor close-up in the foreground. As he moved across the frame, he dragged the geometry of the low-texure area with him. The pattern-matching failed because there was no detail for it to lock onto in this low texture area.

So it may turn out my frame-by-frame process is only useful for improving CR in certain problem scenes.


Geometric differences between the domestic VT and the original broadcast VT

These differences turned out to be more pronounced than I expected, especially in the horizontal direction.
The line-blanking interval on the domestic VT appeared to have been increased and was eating into the active picture content on the right-hand-side of frame. I assume this may have been the result of the PAL-NTSC-PAL conversions?
There was also some horizontal compression of the line timing, and possibly some wow-and-flutter variations from the VT scanner.

To match the original broadcast VT geometry an additional 2nd-pass transformation would be needed.
This may possibly make the footage more CR-friendly still.
I could generate this transform from the chroma; but unfortunately the frame-by-frame pattern-matching produces variations in areas of low texture around the edges of frame, so the 2nd-pass transformation needed would not remain constant across the footage.

Here is an example of the horizontal and vertical distortion remaining after the frame-by-frame processing:
262_(H).jpg
Horiz distortion

262_(V).jpg
Vert distortion



Geometric fluctuations

Unfortunately a side-effect of my frame-by-frame process, is an undesirable rapid fluctuation in geometry on low texture areas, and around the edges of the frame.

The test sequence I'm using is particularly problematic, since it originated from location footage.
This makes it particularly prone to bounce and weave, due to the following sources:
1) The film camera used on location.
2) The original 1970's telecine playout of the location footage.
3) The film camera used in the film recording.
4) The more recent HD telecine transfer (though this should be more stable with modern equipment).

All these sources of positional instability produce a rapid flexing of the geometry that would be disturbing for the viewer.

There are three possible solutions:
1) Deform the domestic VT frame-by-frame to match the FR. This would be useless for CR; but could be useful when using the VT to overlay colour onto the FR. Changes in the geometry of the chroma in untextured areas would not be very noticeable to the viewer. (This would require processing in 24 bit RGB, which would be slow.)

2) Apply film stabilisation to both the VT and FR prior to processing. This would most likely have a negative effect on CR, but this wouldn't matter if you just wanted to do VT colour overlay.

3) Constrain the div and curl derivatives of the undistort vector field. (These measure the amount of divergence and curvature in the field respectively.)

I've experimented with this last method, and found that applying the maximum constraints produces a big improvement in geometric stability; but has a very negative effect on CR.
Am currently trying to find the best compromise settings, to get both good stability and good CR.


This field sequence of this particular test footage was out-of-phase; but this appears to have had little impact on the pattern-recognition process, which was still able to get a good cross-match between the FR and VT frames.


The Undistort Macro I ended up with after all these changes, was the following:

dirf = getDirectory("Choose FR directory ");
dirv = getDirectory("Choose VT directory ");
diro = getDirectory("Choose output directory ");
for (i = 10; i < 1025; i++)
{
j = i + 49;
open(dirv+i+".png");
open(dirf+j+".png");
run("bUnwarpJ", "registration=Mono image_subsample_factor=0 initial_deformation=[Very Coarse] final_deformation=[Fine] divergence_weight=1 curl_weight=1 landmark_weight=0 image_weight=1 stop_threshold=0.01");
save(diro+i+".png");
close();
close();
close();
}

N.B. The line j = i + 49 is required because there is a 49 frame offset in the numbering used for the FR and VT sequences.


Applications of vector and scalar field constraints in CR

The biggest remaining problem with CR is to get rid of the quadrant errors when there is motion that results in differences between the fields.
I've noticed these usually arise not from two different colours overlapping and mixing between the fields; but rather the disturbance in vertical and horizontal displacement values caused by the phase rotation of the vertical measurement.
It may be possible to filter the displacement values to remove these disturbances.

Since the distortion in the FR is represented by a 2D vector field, the most natural way to filter the displacement values for errors is to apply constraints that can act on the vectors directly.
So it may well turn out that constraining the div and curl of the vector field prior to generating a quadrant map, will reduce the number of quadrant errors.

Since curl is itself a vector field, you would probably want to constrain the magnitude of the curl vectors.
The div however, is a scalar field, and therefore you can constrain it directly.

Previously I've tried filtering the displacement values by calculating a running-average; but since that's a 1D filter it's less likely to be effective.


Another possibility is to try filtering in the frequency domain.
I've noticed that lots of ripples appear in the vector field during panning shots (caused by the phase disturbances in the vertical measurement). By splitting the vector field into horizontal and vertical scalar fields, and applying FFT's, it may be possible to remove this ripple using 2D frequency domain filtering.
Alternatively you could constrain the partial derivatives of the scalar fields.

Here is an example of the ripple introduced when the fields are different.
The first image shows the vertical displacement maps for the locked-off shot immediately before the camera pans, and the next image shows what happens after the camera starts to pan:
Ripple_V_1.jpg
Ripple_V_2.jpg


2nd July 2010

Discovered a problem with the png input format: it seems to be producing the same black level fluctuations I've seen previously with other input formats. So I've had to go back to using tiff sequences for 8-bit processing.


Conforming the VT to the FR

Today I concluded my experiment to match a domestic colour VT to the corresponding FR in a frame-by-frame process.
I had assumed this would slow down processing because it requires 24-bit image sequences (obviously!); but then I realised there was no reason to process in 1080X1920, since 576X720 is perfectly adequate.
With this lower resolution, processing times are speeded up to around 36 seconds per frame on my PC (at least 4 times faster on a more up to date PC).
(N.B. An added benefit to 24-bit processing is that there are no input format issues.)

Initially ImageJ produced quite a lot of errors in the geometry of the output frames, with some very odd distortions appearing!
I solved this by adjusting the div and curl constraints to maximum, which fully resolved the issue.

I combined the colour output sequence with the original FR using the following AviSynth script:

ImageSource("F:\FR (SD)(24 bit)(mask 2)\%d.bmp", 3, 722)
lumavid = ConvertToYV12()
ImageSource("F:\Output\%d.bmp", 3, 722)
chromavid = ConvertToYV12()
MergeChroma(lumavid, chromavid)

I'm pleased to say the geometry seemed to line up very well indeed, and I was very happy with the end result.
I noticed some minor colour-bleed; but this was already present on the original domestic VT.
There are also some intermittent problems on the extreme right and left margins of frame, where the colour twitches in and out of alignment. These would probably have to be trimmed off as they're an unavoidable consequence of the pattern-matching process.


Conforming the FR to the VT

I've not had time to make any further progress tweaking the div and curl settings, in order to achieve a more consistent improvement in CR.
I intend to investigate this further when time allows.