26th July 2008

Earlier this week I modified the C++ code for Mohan's DeBarrel filter to allow for mixed negative and positive parameters.
This allowed me to test a reverse-mapping to remove moustache-type distortion from the scan-lines, which produced the following vertical contour-map:
TOTP36_(DeMoustached)_(V).jpg
(C) BBC


Here is the vertical contour-map when using a reverse-mapping for pin-cushion distortion:
TOTPDEPIN_(V).jpg
(C) BBC)


Since there were fewer contours on the moustache reverse-mapping, I knew I was on the right track.
However Mohan's filter uses radial coordinates, which produce a radially-deformed reverse-mapping. What was really required was a Cartesian filter.

Mohan then emailed to let me know he had modified his filter to allow semi-independent x and y distortions, related by a ratio, as well as mixed negative and positive parameters.

Using ratios of 1.5 and 2, I obtained the following vertical contour-maps:
Totp36_(vh=1.50)_(V).jpg
xy ratio = 1.5 (C) BBC

Totp36_(vh=2)_(V).jpg
xy ratio = 2 (C) BBC

This is obviously a big improvement on the radial distortion, although probably not the best that could be achieved.

Mohan's filter requires the parameters to be set by eye, using a grid overlay.
In order to see the scan-lines clearly, I overlaid the grid onto the 288 c/aph waveform.
Then after setting the parameters I substituted the actual frame and performed the distortion.

I used the following polynomial for the radial function:
r' = (-0.01r³ + 0.03r² + 0.98)r

(where r' is the source image coordinate, and r is the corrected image coordinate)
The use of an xy ratio produces and oval-shaped distortion, which would have the effect of reducing the values of these parameters along the vertical axis.

Richard's automated process using CSV files has found this polynomial:
y' - y = 0.0243x - 0.2545y + 0.0018xy - 0.0008y² + 0.0001y³

Hence:
y' = 0.0243x + 0.7455y + 0.0018xy - 0.0008y² + 0.0001y³

(I have eliminated other terms, which are so small they are negligible)

As he's not used a quartic equation for the fitting polynomial, it's not possible to determine from this whether moustache distortion is present.
If the CSV file was relative to Richard's distorted reference, this will ofcourse skew the parameters quite substantially. (The "butterfly distortion" in the horizontal contour-map suggests that it was done relative to the distorted reference.)

The terms in x and xy are most likely due to Distortion B (i.e. distortion caused by the image content, rather than the underlying Distortion A). Since these terms can skew the geometry in either direction depending on the image content, it would be better to find the underlying Distortion A, and use this to deform the reference waveform.
This should produce the best chance of tracking Distortion B.
(N.B. The small cubic term distorts the image in different directions in the left and right halves of frame, and is again probably caused by the unbalanced image content.)

Another reason for using Distortion A on the reference; rather than trying to deform each source frame individually, is that whenever Richard's process makes an error, it would also produce an error in the reverse mapping function for that frame.


Two vivid illustrations of Distortion B are the JS Lights Sequence, and the Paul Temple frame of uniform colour:
In JS Lights, the fast moving spotlights behind Jimmy S. produce an increase in the pin-cushion distortion near the centre of frame, bunching the scan-lines together more strongly.

In the Paul Temple frame, the luminance is noticeably lower on the righthandside of frame, and hence the horizontal contour-map is asymmetric.

I decided to test my reverse mapping on the Paul Temple frame, which produced the following vertical contour-map:
pt450pal_(DeMoustache2)_(V).jpg
(C) BBC


Obviously not a good result: so it seems the reverse-mapping parameters for Distortion A would have to be tailored individually to different sources.


28th July 2008

Richard has kindly given me his distortion parameters for frame 36 of the TOTP footage.
If x' and y' are the distorted coordinates, with respect to an origin in the centre of frame, then the equations are as follows:

x' = - 0.922x + 0.0619y - 0.0006x^2 - 0.0024xy - 0.0017y^2
y' = 0.0243x + 0.7455y + 0.0018xy - 0.0008y^2 + 0.0001y^3

In an attempt to prove that the image content influences the distortion, I have plotted these as a surface manifold:
Richard's_surface1.jpg
Surface plot (low grid density) (C) BBC

Richard's_surface2.jpg
Surface plot (high grid density) (C) BBC

The background grid represents an undistorted surface, and the grid in the foreground is the distorted frame as recorded on the film.

Since the luminance content is top-heavy, the upper half of the frame has become dilated. (Remember higher luminance means higher beam current, and therefore a stronger pull on the beam from the scan-coils: hence you get image dilation.)
This is especially true of the upper left corner, which has the highest luminance content. The upper left extremity of the BBC1 sign is being dragged towards the scan-coils, which is exactly what you'd expect, since this is the closest point between the scanning beam and the coils. (Electric force being proportional to the inverse square of the distance between the charges, emphasises the drag from the scan-coils at this point of closest approach.)

The luminance content is also greater in the lefthandside of the image, and therefore we see a vertical dilation of the left of frame.

I know it's not conclusive, but it's a good indication that a large element of the distortion depends on image content.

(N.B. This distortion is relative to Richard's pre-distorted reference; but the effect of this is simply to stretch the manifold in it's left and right margins, and therefore it doesn't effect my above conclusions.)

Thus the effect of localised "breathing" is likely to be quite significant, especially on dark scenes with disjoint areas of high luminance.
This would be quite easy to test on the JS Lights footage, where the spotlights move rapidly across the frame. I would expect the distortion to follow the spotlight.

Note that the cross-terms in x and y, along with the terms in xy, produce a trapezoid distortion. This gives the illusion of perspective; but is in fact due to the luminance-driven distortions described above.

Finding the mean or underlying distortion will produce the best chance of tracking all the deviations across all frames.
This could be done by averaging the CSV parameters across all frames; or across a basket of representative frames drawn from throughout the footage.

Since I expect the underlying distortion to be largely symmetric, I would expect the assymetric terms to drop out of Richard's equations.
So the cubic terms would disappear, in which case a quartic term could be quite significant in producing the best fitting surface.

The image content would then act as a stresor producing a foliation of manifolds around this underlying topology.
(This is very similar to General Relativity, where space-time manifolds are locally distorted by concentrations of mass/energy.)

If you plot the negative cubic term against the positive squared term in the y distortion equation, it interestingly produces a half-moustache distortion as shown below:
Half_moustache.jpg
(C) BBC


So it seems RIchard's process did pick up the moustache distortion on the lefthandside of frame; but missed it on the right because of the paucity of colour in that region.


30th July 2008

In my last write-up I made several glaring errors in my enthusiasm to publish!

Firstly I got my reference frames mixed up: the equations were in terms of x' and y'; not x and y.
I also got the sign wrong on one term.

To save further confusion, I will use u and v as the source frame coordinates, and x and y as the undistorted destination frame coordinates.
This gives the TOTP equations as:

x = 1.078*u - 0.0619*v + 0.0006*(u^2) + 0.0024*u*v + 0.0017*(v^2)

y = -0.0243*u + 1.2544*v + 0.0007*(u^2) - 0.0018*u*v + 0.0008*(v^2) - 0.0001*(v^3)

However I now think the higher terms are important after all, so a more accurate version of these equations is:

x = 1.078*u - 0.0619*v + 0.00064*(u^2) + 0.0024*u*v + 0.0017*(v^2) - 0.000014*(u^3) + 0.000022*(u^2)*v - 0.000062*u*(v^2) + 0.000035*(v^3)

y = -0.0243*u + 1.2544*v + 0.00068*(u^2) - 0.0018*u*v + 0.0008*(v^2) - 0.000011*u^3 - 0.000028*(u^2)*v - 0.00000011*u*(v^2) - 0.00014*(v^3)

Likewise for the Paul Temple frame we get:

x = -1.962 + 1.0969*u - 0.2221*v - 0.005*(u^2) + 0.0019*u*v + 0.0009*(v^2) + 0.000029*(u^3) + 0.0000075*(u^2)*v - 0.000041*u*(v^2) + 0.000057*(v^3)

y = 9.1406 + 0.186*u + 1.3151*v + 0.00036*(u^2) - 0.00097*u*v - 0.00013*(v^2) - 0.0000089*(u^3) - 0.000021*(u^2)*v + 0.0000038*u*(v^2) - 0.00014*(v^3)

(Note, unlike TOTP these have a realignment of the origin.)

These equations produce the following surfaces.
Richard's_TOTP_surface_(corrected).jpg
TOTP manifold (C) BBC

Paul_Temple_surface_(more_precise).jpg
Paul Temple manifold (C) BBC

These are in fact the undistorted frames relative to the coordinate system of the distorted frames. In other words they show the deformation that needs to be applied to the distorted frames to produce the undistorted frames. (I.e. I got it backwards in my previous write-up!)

You can see that the areas of high luminance are in fact more compressed; not more dilated, and this is the opposite of what I expected.

I don't know the details of the EHT power system and how it affects picture breathing, but I believe it's to do with poor power supply regulation. So could it be that a stronger beam draws more current, taking power away from the scan-coils? This could produce a local contraction of the picture in brighter areas.

Anyway back to the maths:
You can see that the manifolds for both TOTP and Paul Temple are shaped exactly the same, they've just been rotated differently and are viewed at a slightly different perspective.

Assuming that the rotation and perspective are a result of Distortion B (i.e. produced by the image content), you can remove them by taking away the cross terms in u and v, and the terms in u*v.

This leaves you with the following equations for the TOTP footage:

x = 1.078*u + 0.00064*(u^2) + 0.0017*(v^2) - 0.000014*(u^3) + 0.000022*(u^2)*v - 0.000062*u*(v^2) + 0.000035*(v^3)

y = 1.2544*v + 0.00068*(u^2) + 0.0008*(v^2) - 0.000011*u^3 - 0.000028*(u^2)*v - 0.00000011*u*(v^2) - 0.00014*(v^3)

The surface is shown below:
TOTP_Underlying_distortion_(3rd_guess).jpg
(C) BBC


This could be the underlying symmetry.
However, I have yet to see the distortion polynomials for a frame that has more luminance content on the righthandside, or in the lower part of frame. Such frames could produce mirror-image distortions due to the different image content skewing the geometry in the opposite direction.

If this is the case, then we need to remove the assymetric terms from the equations to get the underlying distortion, which leaves us with:

x = 1.078*u + 0.0006*(u^2) - 0.000062*(v^2)*u

y = 1.2544*v + 0.0008*(v^2) - 0.000028*(u^2)*v

Here is the surface plot:
TOTP_Underlying_distortion_(2nd_guess_with_no_quartic_term).jpg
(C) BBC

Obviously the parameters will need fine-tuning to produce the proper scaling; but the basic form is there.

Partly from intuition, as well as observation, I would also add a quartic term to the y distortion, to produce a moustache-type distortion in the scan-lines, thus:

x = 1.078*u + 0.0006*(u^2) - 0.000062*(v^2)*u

y = 1.2544*v + 0.0008*(v^2) - 0.000028*(u^2)*v + 0.00002*(u^4)*(0.0001*v)

(I have no idea of the correct magnitude for this term: so it's a complete guestimate.)

These equations produces the following manifold:
TOTP_Underlying_distortion_(2nd_guess).jpg
(C) BBC


It's also quite interesting to look at the rotation and perspective-free equations with only the cubic terms removed:

x = 1.078*u + 0.00064*(u^2) + 0.0017*(v^2) + 0.000022*(u^2)*v - 0.000062*u*(v^2)

y = 1.2544*v + 0.00068*(u^2) + 0.0008*(v^2) - 0.000028*(u^2)*v - 0.00000011*u*(v^2)

This gets rid of another assymetry, and could be a possible candidate for the underlying distortion.
It's surface is shown below:
TOTP_Underlying_distortion_(no_cubic_terms).jpg
(C) BBC

For these to be tested will require the new version of Richard's software.
The distortions could then be applied to the reference (once I have scaled them suitably).


7th September 2008

Whilst investigating various surface-modelling software, I've been reminded that thus far we have only been looking at a tiny sub-group of all possible 2D surface distortions.
It's very likely that a better surface model could be found which is more naturally in tune with this system.
Then fewer parameters would be required to match the distortion, thus saving on CPU-time.

The EM and Dirac fields (i.e. electrons) both have a natural Lorentz symmetry: so the Lorentz Group could be a suitable candidate.
However the luminance-driven element of the distortion makes it look more like General Relativity to me.

Instead of a 4D space-time distortion being generated by the local mass-distribution, you have a 2D surface distortion being generated by the luminance distribution on the screen.
So the maths should be similar to GR; but in 2 dimensions rather than 4.

This would suggest the Diffeomorphism Group could be a candidate surface model; probably with some constraints applied (e.g. to replace radial symmetry with Cartesian asymmetry).

Another parallel with GR is the fixed element of the distortion. This mirrors the global space-time warping produced by the Universe as a whole (used to define the meaning of an inertial field).

Unfortunately I don't have the expertise in topology to take this any further.

On a more basic level, I do still think it would be a good idea to remove any distortion terms known to be luminance-driven, before warping the reference. This should mimic the effect of averaging the distortion measurements over the whole of the footage to be processed, without actually having to perform this averaging.

I had hoped to determine which terms in the polynomial are luminance-driven; but cannot proceed any further on this without more source material. (Though I've had a fair stab at guessing in my previous write-up.)

To answer Richard's doubts about the origins of the distortion, I would make the following points:

We know from the Poisson Equation in Electrostatics, that a curved charged surface will produce a curved field. So at any given instant the field lines inside the tube will be curved.
The linearity of the scanning beam can only be set for a given picture content. So as soon as the picture content changes, the scan-lines will start to curve again, and the underlying curvature in the field will reassert itself. Thus I don't believe it's possible to completely remove the imprint of the field curvature.
(Similarly any thermal-drift in the compensating circuitry will cause the scan-lines to become curved, and the circuitry of the era was much more thermally sensitive than it is today.)

From the evidence I've seen so far the CRT distortions seem to dominate over distortions produced by the film camera.
So if you could tease out those distortion terms caused by the luminance distribution, then you could rewrite the distortion equation in terms of a luminance field L(x,y). Thus you would have a model based on luma rather than chroma (useful where there's no colour information).


11th December 2008

With the NFT event coming up on Sunday, I thought this would be a good time to write up my latest thoughts on geometric correction.

Taking my cue from General Relativity, I have concluded that the most natural reference frame to use would be the fixed distortion inherent in the film recorder system.
Richard is already doing this to some extent by stretching the right and left margins of the reference signal; but a more rigorous distortion would be required to match the fixed distortion more closely.

I've already outlined above how the fixed and transient distortions may be separated, using Richard's displacement measurements.

Once you have measured and determined the fixed distortion, this could then be used to deform the reference signal, after which the source footage could be run through Richard's software a second time.
The resulting displacement measurements would then correspond only to the transient distortions, which I believe are produced mostly by the luminance distribution.

In Gravitational Field Theory, the space-time manifold expands in areas of high mass-energy. However in this case we appear to have the exact opposite effect: with the 2D manifold contracting in areas of high luminance.
Therefore we can write:

delta_x(u,v) = c/L(u,v)

where (u,v) is the coordinate system of the fixed distortion, delta_x(u,v) is the x-displacement at coordinate (u,v), c is a constant, and L(u,v) is the luminance field.

Thus:

c = L(u,v) * delta_x(u,v)

If I'm correct then c should turn out to be constant across the frame, (and across all other frames).

Similarly we need to determine a constant k for the y displacement function delta_y(u,v).

So once c and k have been determined, we now have a model for reversing the transient distortions in every frame, even where there is no chroma.
The simplicity of this formula makes reversing the geometry less CPU-hungry, and therefore it should be more practical.

Should c and k turn out not to be constants, then we would need to replace them with functions c(u,v) and k(u,v).
Then use a surface fitting technique (such as linear regression) to find the form of these functions. However they should be quite simple.

It's important to note that we're now using three different reference frames:
1) the source footage reference frame, which I'll call (a,b),
2) the reference frame corresponding to the fixed distortion (u,v),
3) the undistorted reference frame (x,y).
So to get from L(a,b) to L(x,y) we need to transform reference frames twice (i.e. we need to apply two consecutive deformations to each frame). In other words we would first reverse the fixed distortion, then reverse the transient distortions.

The resulting footage could be processed through the PAL Transform Decoder, thus recovering the colour information.
That would solve the problem of excessive cross-colour, when the vertical signal at 216 cycles is too attenuated to be useful (e.g. in the "Dr. Who" test footage). Since the Transform Decoder can use temporal frequencies to distinguish chroma from luma.
It would also allow reliable field separation, provided the vertical sampling frequency of the source material was sufficient.

(It's interesting to note that since luminance and displacement are inversely proportional, then plotting the function c/delta_x as luminance should produce the original picture content without even referring to the luma values in the distorted frame.
Not very useful in practice though, since the delta_x values are measured on a block-by-block basis: so the picture definition would be substantially reduced. Also the luma would disappear entirely in monochrome regions of the frame.)

The key point in the above, is that using the most natural coordinate system allows for the simplest reverse mapping function, and is therefore more practical in terms of computer power.
It may still prove to be unachievable for the moment, but this will change once CPU speeds increase in the future.

Some vertical jitter correction would probably need to be applied right at the start of the process, otherwise the Transform Decoder may not be able to lock onto the line structure.

In all the above I'm assuming the film-derived distortions are infrequent and trivial enough to be ignored; but obviously these would produce errors as they're not encompassed in the model.