What are these white pixels … ?

What white pixels ?

These, there, flickering at the cliff wall !

Ummhh …

… I do not know why this has slipped through my countless screenings, but here they were. Full bright white, sometimes colorful, single hot pixels.

What follows now is an anecdote, highlighting the various adventures you sometimes face, when dealing with problems which seem so trivial at first sight. It is almost nostalgic, as these problem patterns tended to appear throughout the whole production and they persist right to the end 🙂

The Discovery

In 3 shots I discovered hot pixels (predominantly white, but at closer inspection also pure black and sometimes colored) in about 20 percent of the frames from 3 distinctive shots. Not many pixels, between 1 to 6, but quite detectable. It needed someone from the outside to point at these and then they were quite obvious and – even more importantly – very distracting.

A thorough search on all the other shots didn’t show any of these problems.

The pixels looks like bugs in the rendering or compositing. They were not associated to a fixed location of the texture or model, but the general area was always the same. So my idea was to pin down the source of the problem and fix it or at least work around it.

Trying to Fix the Compositing

I loaded a raw frame of one of the faulty shots into the compositor blend file and did a compositing run and searched for the hot pixels.

There were none.

This puzzled me. I tried different frames, all with the same result – no hot pixels. I did another test, this time using my render machine and compositing the same frames as on my workstation. To my utter astonishment, some of the frames showed the hot pixels. BUT, other hot pixels then on the picture locked footage.

Each time I ran a compositing run on my render machine, I got different results, but always around 20% faulty frames.The hot pixels always changed specific location, but never the general one. Sometimes a frame was rendered clean, another time it had the pixels.

I know, that my render machine has pre release versions of the Intel Xeon Hexacore processors but I have never seen anything like this. The circumstances which lead to this behavior are rather strange and my be fun to investigate, but not this time. I wanted a true picture locked and error free footage.

The First Workaround

So quite an easy task to fix the problem. Just use the workstation to do the compositing.

That would have been the best option, but alas, I needed ~70GB of extra disk space for the raw frames and my workstation was already full. And no other spare disk space available. All disks at my place are already packed full with the production files, so no way to free some space.

The Next Idea

Well, all in all there were about 90 frames, which showed the hot pixels. So I just could cope with manually painting the errors out. A very tedious type of work but manageable. The first idea was to paint directly over the DPX files, which are fed into the final mix. And here comes the point, were I sometimes hate to work under linux (but only sometimes 🙂 ). This is when I ave to deal with doing image processing for higher than 24bit/pixel formats.

DPX files have 10 bit per color channel per pixel, so using the Gimp is out of the question.

No problem I thought. There’s Krita, much talked about recently, so this should do the trick. My previous experiences with Krita were rather bad, but I thought after 1 year of Krita development I should give it a try for this simple task.

Well, as it turned out, Krita does not support DPX files.

OK, but Krita does support OpenEXR files, which is even better, as I have no loss in color information then.

OK, it turned out that Krita is not able to handle the OpenEXR files generated by blender. It just shows an empty canvas, thats all.

Next Try

Next I tried to convert the DPX files to TIFF files using the imagemagick toolchain. As it turned out, the generated TIFFs did display completely wrong. False colors, wrong gamma, absolute unusable.

It was then, that I remembered, that blender has it own image editor, quite basic but enough for the task at hand. Another extremely nice feature is to load an image as image sequence, and with this it is quite easy to do the correction painting. Using this workflow I was almost glad that Krita had failed.

So I painted over the 3 shots and it tuned out to be a very tedious work. Also very eye tiring. But after an 1.5 hrs work it was done.

So I went to the color correction program I run under Windows to produce the final DPX files.

What a nice surprise: all the painted DPX files turned out pure black. I almost cried …

The Final Resolution

I just couldn’t believe this. I seemed that up until now everything had conspired to prevent me from fixing these hot pixels.

So I went back to blender and this time I loaded in the EXR files and did a paint over of one of the faulty frames. A test with this single frame showed no problems further down the pipeline and I was good to go. The difference in the painting experience of working on float color images compared to the 30bit DPX files is striking. Doing a single stroke on the EXR image takes about 2-3 seconds until you see the result, whereas working on the DPX file, you have immediate feedback. That meant, where I had worked 1.5 hrs on the DPX files, I now had to work for 2.5 hrs.

Did I already mention that this is tedious work ?

In the end, I got all the frames cleaned up and now I have an error free footage, ready to get mixed with the music and sound.

I just hope that these corrected shots never have to be rerendered …