Boitho - distributed crawler

Thanks runarb, I'll change my stats now, because they have only just started (mine that is) I added.

Ah, so the crawled pages get full weight but the robots don't. Geez, that's so logical and it didn't even cross my mind! I was splitting the total weight by the parcels, no wonder it wasn't working. :P