NeoGen
12-04-2005, 03:28 PM
Let's go get it guys!
http://www.majestic12.co.uk/projects/dsearch/download.php
A must have upgrade with important bug related to robots.txt processing fixed and much more CPU friendly archiving that can be controlled using Archiving delays.
Details from history.txt:
v1.2.1 10/12/05
! Fixed problem with some rare urls resulting in a url bucket load failure
+ Better strategy to deal with constantly failing buckets
! Content analyser did not care about delays resulting in 100% CPU usage
! Unified all archiving related delays into one called ... "archiving delay" and added it to profiles
! Revised cache flushing logic
! Fixed robots.txt problem when user-agent specified was actually MJ12bot (DOH!)
Major new release with content analysis technology build-in - this reduces size of uploads considerably, generally by at least 50%+. It is highly recommended to upgrade as soon as possible.
Details from history.txt:
v1.2.0 01/12/05
+ Smart content analysis (ConAn) technology added to node to greatly reduce size of barrels and increase speed
of indexing.
! Improved clustering logic of barrel sorting
+ Blog XML feeds detection added
+ Accelerated timeouts applied to extremely slow sites that remain in nearly completed bucket.
! Changed maximum robots.txt delay to be 10 seconds.
! Sorter is more robust now
! Fixed some urls that timed out that were wrongly classified as "Unknown" error type
! Self-restart when timeouts are high changed to at least 60 mins of runtime
! Better filtering of URLs
! Fixed bug in font size parsing
! Fixed problem with non-specified Content-Lengths after content analysis
! Fixed webserver not sending correct authentication header (when auth was enabled)
! Fixed incorrect progress indication for final stage of archiving when archiver is internal (LZMA)
! Fixed potentially fatal SQL syntax error that could force node into endless loop
! Added possibly fatal database error to list of errors that will cause automatic restart of node
! Fixed auto-restarts due to timeouts that happened too early
! Fixed apparent non-completion of some buckets with 1 url left that resulted in bucket load/unloads
http://www.majestic12.co.uk/projects/dsearch/download.php
A must have upgrade with important bug related to robots.txt processing fixed and much more CPU friendly archiving that can be controlled using Archiving delays.
Details from history.txt:
v1.2.1 10/12/05
! Fixed problem with some rare urls resulting in a url bucket load failure
+ Better strategy to deal with constantly failing buckets
! Content analyser did not care about delays resulting in 100% CPU usage
! Unified all archiving related delays into one called ... "archiving delay" and added it to profiles
! Revised cache flushing logic
! Fixed robots.txt problem when user-agent specified was actually MJ12bot (DOH!)
Major new release with content analysis technology build-in - this reduces size of uploads considerably, generally by at least 50%+. It is highly recommended to upgrade as soon as possible.
Details from history.txt:
v1.2.0 01/12/05
+ Smart content analysis (ConAn) technology added to node to greatly reduce size of barrels and increase speed
of indexing.
! Improved clustering logic of barrel sorting
+ Blog XML feeds detection added
+ Accelerated timeouts applied to extremely slow sites that remain in nearly completed bucket.
! Changed maximum robots.txt delay to be 10 seconds.
! Sorter is more robust now
! Fixed some urls that timed out that were wrongly classified as "Unknown" error type
! Self-restart when timeouts are high changed to at least 60 mins of runtime
! Better filtering of URLs
! Fixed bug in font size parsing
! Fixed problem with non-specified Content-Lengths after content analysis
! Fixed webserver not sending correct authentication header (when auth was enabled)
! Fixed incorrect progress indication for final stage of archiving when archiver is internal (LZMA)
! Fixed potentially fatal SQL syntax error that could force node into endless loop
! Added possibly fatal database error to list of errors that will cause automatic restart of node
! Fixed auto-restarts due to timeouts that happened too early
! Fixed apparent non-completion of some buckets with 1 url left that resulted in bucket load/unloads