Improved result validation, stability enhancements, future plans
For the past weeks I've been working on a new mechanism that handles our validation of results computed by our users.
We used to have a pixel-by-pixel comparison, with a small error tolerance hard-coded for the different file formats (PNG, EXR). The problem with such a comparison is that between different platforms the same rendering process can result in very small numerical differences. These differences are so small, that in most cases our eye can't distinguish between such results. Unfortunately, doing a pixel-by-pixel validation using their absolute values can quickly trigger the error tresholds, especially for larger images, invalidating results that good.

The solution to this dilemma is to use a computational model that simulates the human visual system. This is known as http://pdiff.sourceforge.net/ypg01.pdfPerceptual Difference. Based on information found through pdiff.sourceforge.net the idea was to implement this technique from the ground up, but after some time and tests I determined that it was better to outsource it to the already working implementation that the tool PerceptualDiff already provides. Apart from being able to use a proven implementation our system also benefits from a lowered memory requirement to do the validation in this manner.


The robot model was made by Arto Harju / Samuli Jomppanen.
The main problem for me to solve became the question: What result image to use as reference? Normally one would use PerceptualDiff in a controlled environment, where the reference image is known beforehand. On Renderfarm.fi it is not the case, we get a number of results from which we can't directly tell if they are actually correct.

Our current system will calculate a special value for each result. The system picks the result with the value closest to the average in the result set. Using the selected reference result all other results will be validated. If PerceptualDiff determines that the images differ perceptually such a result will be invalidated.

The tests we have conducted show that this process is quite robust: even sessions with very low amount of AO samples get properly validated, despite minor differences in results from different platforms.

This means that finally we can render sessions again on all supported platforms instead of assigning them to one operating system only. This will improve the overall performance of our grid for sessions, and everyone will be getting an equal quantity of work.
Other improvements and future plans

During this summer we've been mainly focusing on fixing open issues and improving the service performance. A number of smaller problems have been eradicated (zip corruption that jammed sessions, clients crashing due to library issues, OS X computers not getting work, automatic file deletion after 2 weeks, &c.), and we'll be focusing on this for a while longer. We'll be observing how the new validator behaves and giving it some patches when it shows signs of bugging, although it is expected to perform excellently.
The uploader was given a small feature update which makes login automated once you have done it once. This means that you no longer have to type your email and password every time you wish to upload new work to the farm. This was pushed to the Blender trunk this morning, and will be available in the next Blender version.
In the autumn we'll have the screensaver finally deployed for Windows machines and for other platforms when we have the Windows version working properly. We have also been looking at replacing all our old rendering clients with virtual machines by basing the software on the CernVM developed by the folk at CERN, but it would seem that the VM support for BOINC is not quite what we require, yet. We'll be continuing the work on this if it seems to be functional enough.


More...