I've just started running grub, and I've been noticing that it stops crawling all of a sudden many times. Most of the times it reaches 500 url's processed and just stops.
On the Crawler log it says "Crawling finished", and on the Client control log it seems to be trying to contact the server many times but unsucessfully. It has alot of messages saying "Next attemp to connect will be in xxx second(s)"
I don't understand, because on the client status it saysit's running, and it has 34 workunits queued... so why doesn't it move on to the next workunit after finishing one?