Undistorting+Image+Sequences

__**25th May 2009**__

This page explains how to undistort image stacks (i.e. video sequences as oppose to test frames).

FIlm stabilisation should first be applied to remove film-gate bounce and weave. I have used the following AviSynth script to achieve this:

loadplugin("rawsource.dll") RawSource("G:\totp_ch03.y", 1920, 1080, "Y8") ConvertToYV12 Stab function Stab (clip clp, int "range", int "dxmax", int "dymax") { range = default(range, 1) dxmax = default(dxmax, 4) dymax = default(dymax, 4) temp = clp.TemporalSoften(7,255,255,25,2) inter = Interleave(temp.Repair(clp.TemporalSoften(1,255,255,25,2)),clp) mdata = DePanEstimate(inter,range=range,trust=0,dxmax=dxmax,dymax=dymax) DePan(inter,data=mdata,offset=-1) SelectEvery(2,0) }

(Where G:\totp_ch03.y is the source clip.)

The clips must then be separated into an image sequence. I use Virtual Dub for this purpose, and create a bmp sequence.

The elastic B-Spline undistort transform is then generated from one of these frames using BUnwarpJ. (See my previous page for a detailed description of how to do this.) The transform is contained in a file, which I will call elastic.txt

Next I use ImageJ to run the following macro:

dir = getDirectory("Choose input directory "); diro = getDirectory("Choose output directory "); for (i = 0; i < 1025; i++) { call("bUnwarpJ_.elasticTransformImageMacro", dir+i+".bmp", dir+i +".bmp", dir+"elastic.txt", diro+i+".bmp"); }

The undistorted image sequence ends up in whichever output directory I selected, and it can then be reassembled into a video sequence.

__**Processing times**__

On my PC it takes a disappointing 25 seconds to process each frame using the above macro. However my CPU is an outdated 2GHz AMD Athlon 2400+, whereas the code has been optimised for multi-core processors: so it should run much faster on a Dual Core machine.

Also for some purposes you may not need process the film in HD resolution. E.g. If you wish to overlay colour from a domestic VT source onto the b/w FR, then you might as well down-convert the film to SD prior to undistort processing, as this should reduce processing time by approximately a factor of five!


 * __Methods for generating the undistort transform__**

This can be produced by one of two methods:

1) A chroma-derived method using Richard Russell's CR software.

2) A luma-derived method using existing VT material cross-matched with the FR.

The first method can result in artefacts such as ringing at the top-of-frame, and tearing on the left and right frame boundaries. These are caused by errors in CR quadrant assignment, which in turn lead to errors in geometry. They can also be caused by large areas of missing geometry information, thus leading to interpolation errors. They can be avoided by careful choice of source frame.

The second method avoids these problems entirely; but it takes much longer to generate the elastic transform file.


 * __Description of the luma-derived method__**

The source and target files are loaded into ImageJ (i.e. a frame from the FR, and the corresponding frame from the VT). The BUnwarpJ plugin is then loaded, and you must ensure that the VT frame is selected in the "Target Image" dropdown.

"Mono" is selected in the "Registration Mode" dropdown, then the "Final Deformation" and "Stop Threshold" settings can be tweaked to adjust the accuracy of the undistort you wish to achieve.

Click "OK" to start deriving the elastic transform. Diagnostic windows will then open, one of which shows a wireframe representation of the cross-matching between the two images. When it's done, it will ask you what filename you wish to save the elastic transform under.

It can take up to an hour to generate the transform on my PC, depending on which settings I've chosen. However for some purposes it may be possible to use the same transform across an entire episode. E.g. If you wish to overlay colour from a domestic VT, then you don't require an ultra-high level of accuracy.


 * __Recap of the chroma-derived method__**

This uses Richard's CR software coupled with my Undist1 and Undist2 QBASIC programs. Undist1 contains an extra block of code to remove the gross distortion in the right and left margins of the film frames (present in all film recordings). Undist2 does not contain this code: therefore you should apply Undist1 first to remove most of the distortion, followed by Undist2 to remove any residual distortion. (N.B. Once you've applied Undist1 to the image, you must ofcourse remember to change the reference files from urefdist.bin/vrefdist.bin to uref-1.bin/vref-1.bin before generating a new csv file to process with Undist2.)

I have discovered that a two-pass process, using Undist1 with a B-Spline Interval of 960, followed by Undist2 with B-Spline of 1920, removes nearly all the distortion.

To apply this undistort efficiently across multiple frames, you need to combine the elastic.txt files from both passes into a single elastic transform (otherwise you'd have to apply two successive undistort transforms to every frame). You can't combine them directly, since they have different B-Spline Intervals, therefore you need to combine the raw transform files instead using the "Compose Raw Transformations" option in BUnwarpJ's I/O menu. The resulting raw file is then converted to an elastic file using "Convert Transformation To Elastic".

I have experimented with various different B-Spline values for this combined transform, and found the best one to be 1079. (N.B. The horizontal distortion is improved most with a value of 1920; but this doesn't work so well for the vertical. So 1079 is probably the best compromise.)


 * __How many frames can be processed with the same elastic transform?__**

This depends for what purpose you wish to use the undistorted footage:

1) For overlay of colour from a domestic VT source: you may get away with using the same transform across an entire episode. Certainly you shouldn't need to recalculate more often than once per camera-shot change.

2) For deinterlacing FR's: providing film stabilisation is applied first, I have found the vertical geometry to be quite stable, over several hundred frames at least.

3) For conventional PAL decoding: you'll probably need to derive a new transform for every film frame. This is not practical at present; though it may be in a few years time. It would require, Colour Recovery, BUnwarpJ and Undist to be integrated in some way, perhaps as plugins for AviSynth. That way you could you could experiment with different multi-pass combinations, with new undistort transforms being generated automatically for each frame.


 * __Diagnostic Images:__**

The following images show the original distortion in one of the TOTP frames, followed by the distortion after 1st-pass processing with Undist1, then 2nd-pass processing with Undist2:

Now let's look at the results of a combined undistort, and compare B-Spline interval settings of 1079 and 1920:

You can see that 1079 gives the best overall result; though 1920 is best for the horizontal correction. I've experimented with lots of different values for the interval; but the others gave much worse results. (Simply increasing the interval by 1 can lead to a surprisingly large change in the diagnostics.) It seems values close to the horizontal or vertical frame resolution give the best results for those particular directions.

Finally let's look at how well a single undistort performs when carried over multiple frames. This undistort transform was derived from Frame 665, and has been applied to Frames 666, 680, 700 and 1024:

You can see that the horizontal distortion drifts a lot more than the vertical; but the undistorted frames are still a lot closer to the correct geometry than they were originally.

__**10th August 2009**__

I have recently been experimenting with matching film recordings to corresponding domestic VT footage: so that colour from the VT can be accurately overlaid onto the monochrome film. To my surprise I've discovered that it is feasible to calculate a new undistort transform for every film frame. This is achieved with the following ImageJ macro:

dirf = getDirectory("Choose FR directory "); dirv = getDirectory("Choose VT directory "); diro = getDirectory("Choose output directory "); for (i = 0; i < 1025; i++) { open(dirv+i+".bmp"); open(dirf+i+".bmp"); run("bUnwarpJ", "registration=Mono image_subsample_factor=0 initial_deformation=[Very Coarse] final_deformation=[Very Fine] divergence_weight=0 curl_weight=0 landmark_weight=0 image_weight=1 consistency_weight=10 stop_threshold=0.5"); save(diro+i+".bmp"); close; close; close; }

Before applying this macro, it is necessary to split the FR and VT files into image sequences, and the VT images must be upscaled to 1920X1080 to match the FR.

It takes about 90-100 seconds to process each frame on my PC; but would be considerably faster on a dual-core machine. Thus it is practical to put this to work on a full-length TV show (especially if you process each episode in parallel on a separate PC). On my machine it would take about 1,000 hours to process a 25 minute episode (with titles removed).

The process is fully automated, and can be left running indefinitely, provided all the source frames are contained within the same folder and consecutively numbered. If you wanted to put each chunk of the film recording into a separate directory, then the macro would have to be slightly modified, otherwise you would need manual intervention at the beginning of each chunk. (It may be necessary to close down firewall and anti-virus software, as I found these were causing ImageJ to crash after several minutes running the macro.)

Reducing the stop_threshold value will increase the accuracy of the undistort; though it also increases processing time. Changing the final_deformation value to [Ultra Fine] will also increase accuracy. It's doubtful either of these tweaks would be necessary, as I believe my settings are accurate enough for purpose.

The reason I've over-estimated the processing time in the past, is because Virtual Dub was producing 24-bit RGB images from the raw film recording chunks. However since the images are monochrome, you only need 8-bit greyscale files, which have three times less information to process.

Another processing option is to derive an undistort transform once every 25 frames, and use this to process the intervening frames. The macro for this is as follows:

dirf = getDirectory("Choose FR directory "); dirv = getDirectory("Choose VT directory "); diro = getDirectory("Choose output directory "); dirt = getDirectory("Select directory containing ImageJ "); for (j = 0; j < 41; j++) { n = 25 * j; open(dirv+n+".bmp"); open(dirf+n+".bmp"); run("bUnwarpJ", "registration=Mono image_subsample_factor=0 initial_deformation=[Very Coarse] final_deformation=[Very Fine] divergence_weight=0 curl_weight=0 landmark_weight=0 image_weight=1 stop_threshold=0.5 save_transformations"); save(diro+n+".bmp"); close; close; close;

open(dirv+n+".bmp"); open(dirf+n+".bmp"); for (k = 1; k < 25; k++) { i = k + (25 * j); call("bUnwarpJ_.elasticTransformImageMacro", dirf+i+".bmp", dirf+i+".bmp", dirt+n+"-1_direct_transf.txt", diro+i+".bmp"); }

close; close; }

Unfortunately this macro only works with 24-bit image files at the moment, due to limitations in the BUnwarpJ plugin. However it still provides a faster processing time overall. I'm looking into ways to get round the plugin limitations.

A final option is to split the footage into separate scenes using automatic scene detection, and derive one transform for each scene. This could be accomplished by minor changes to the above macro.