Lagu
12-19-2005, 06:55 PM
Alexc has written this announcement:
Current situation
From crawling point of view we are doing extremely well - 50+ mln pages daily is a very high number of pages - 850 mln urls loaded just 3 weeks ago disappeared like sugar in hot water! Congratulations to all as it shows strength of the concept - we are not yet at Google's crawling levels, but we sure achieved far more than Google did in their year 1 of operations - first known index for them was 24 mln pages (Source (http://www-db.stanford.edu/~backrub/google.html))
Its all good, but search engine right now is still at just 45 mln pages - a drop in an ocean comparing to number of crawled pages. BarkerJ summed it up pretty nicely:
In short, our bottleneck to global dominance is currently getting the search engine to process a large amount of data, not a lack of data.
Indeed, there are rivers, lakes, seas and oceans of data, but its being stored right now rather than quickly indexed and added to search engine for daily use - and of course I want to have a genuinly usable search engine that uses crawled data.
So what to do?
In theory it all looks good - there is search engine already and it works, in practice however is that it may work for 50 mln pages, but it wont do for 500 mln - the keyword here is scalability - all components of the system should scale well to billions of pages.
Here are the main 4 stages that make it up:
crawling -> indexing -> merging -> searching
Crawling scales exceptionally very well - this is the most advanced and automated part of the system that works pretty well, no problems here.
Indexing - I've done a lot of work to speed it up in October, and happy to say that on new server "FiddleAbout" (AMD x2 3800, 1 GB dual channel DDR 400) indexing runs very fast and dual cores scale very very well - looks like it should be able to index 45-50 mln pages per 24H. Even better indexing is already very scalable so upcoming 3 boxes will work as part of the whole system, thus giving us ability to index 170-200 mln pages per day. This is more than we crawl now and it will allow to catch up with backlog, so we are doing well in this area.
Merging the last phase before actual searching that takes separate indexed barrels and merges them into searchable index - this area needs big improvements in order to scale to billions of pages. Merging right now is too linear not using more than 1 CPU - this is to be changed to use multiple CPUs and multiple machines, this is the key step to scaling seacrh engine up.
Searching - this needs to be scaled up as well, but luckily main work is done and so long as merging is scaled I expect this part to scale up easily.
So, its the merging's module scalability that needs to be addressed ASAP.
So what about recrawling?
A good search engine is not just a big search engine, but also up to date search engine - some pages change dramatically and people got accusstommed (def sp!) to searching for recent stuff online, and people are very quick to make a conclusion about whether search engine is good or crap - I am pretty harsh in this respect too.
We can only be reasonably up to date if we recrawl - most small search engines can't afford recrawling, so recrawl is not a dirty word, its actually good that we can afford to think in terms of freshness of information, not just its quantity.
As it was mentioned there is a lot of data flowing in every day - 1 TB+ of uncompressed data every day, it has to be well managed (indexed and serached) before we can dig much deeper.
Here are my considerations for recrawl:
1) first 2 bln pages are not as well compressed as they can be now - this means pages occupy 3-5 times more than they can be compressed now, thus recrawling them would free up some disks for index.
2) first pages we crawled were nearer the top of the sites than current deeper pages - they definately need to be indexed
3) some pages are as old as 1 year
4) recrawl gives me breathing space of around 2 months to scale the search engine - at 50 mln per day we can redo 3 bln in just 60 days.
5) new url loads should have much smarter prioritisation - current limit on urls per site is too generic, say Microsoft.com or GeoCities.com should not be subject to the same limit of 15k urls per site as newly formed doorway pages website with lots of junk - in order to be smart we need to index at least 1 bln pages and build info on sites importance.
This is why I think we should start recrawl of existing buckets rather than load more billions.
Exception: national TLDs (top level domains) will still be added, the reason is because there are so few of them comparing to .COMs/.NETs/.ORGs that there is a point to grow them as numbers are too small to create problems anyway. I hope people are okay with this exception.
Also, I want to state here very clearly important thing - the fact that we can recrawl data does NOT diminish in any way important of it being crawled in the first 9 months of this project - I have to admit here that thoughts of quitting this project in the face of impossibility of the task at hand crossed my mind a few times but this ended in April and won't come back - if it was not for people whose participation helped come through the hardest first 6-9 months of this project then it would have never reached this stage.
Will only changed pages send back?
Not right now - this functionality is trickier than it seems as it would require intelligent indexing/merging of multiple barrels with the same name taking only most recent pages from them - this requires all crawled data to be online, which it is not right now, but I expect this to happen in Q1 2006.
The only drawback of not having smart recrawl is that data send back will not be smaller, but right now with Conan active its pretty reasonably small - compare current barrel of 5 MB with 40-50 MB we typically seen in early days of the project.
Conclusion
It is good that we can afford to look into recrawl of data to keep it fresh - many search engines simply can't do that, we are maturing faster than I thought, which is a good thing :)
So what now?
Simple - old buckets will be marked as "uncrawled", and thus automatically put into recrawling. All your stats will remain the same - they won't be wiped out as crawled bucket stays crawled bucket regardless of whether its recrawl or not.[/quote]
Lagu :)
Current situation
From crawling point of view we are doing extremely well - 50+ mln pages daily is a very high number of pages - 850 mln urls loaded just 3 weeks ago disappeared like sugar in hot water! Congratulations to all as it shows strength of the concept - we are not yet at Google's crawling levels, but we sure achieved far more than Google did in their year 1 of operations - first known index for them was 24 mln pages (Source (http://www-db.stanford.edu/~backrub/google.html))
Its all good, but search engine right now is still at just 45 mln pages - a drop in an ocean comparing to number of crawled pages. BarkerJ summed it up pretty nicely:
In short, our bottleneck to global dominance is currently getting the search engine to process a large amount of data, not a lack of data.
Indeed, there are rivers, lakes, seas and oceans of data, but its being stored right now rather than quickly indexed and added to search engine for daily use - and of course I want to have a genuinly usable search engine that uses crawled data.
So what to do?
In theory it all looks good - there is search engine already and it works, in practice however is that it may work for 50 mln pages, but it wont do for 500 mln - the keyword here is scalability - all components of the system should scale well to billions of pages.
Here are the main 4 stages that make it up:
crawling -> indexing -> merging -> searching
Crawling scales exceptionally very well - this is the most advanced and automated part of the system that works pretty well, no problems here.
Indexing - I've done a lot of work to speed it up in October, and happy to say that on new server "FiddleAbout" (AMD x2 3800, 1 GB dual channel DDR 400) indexing runs very fast and dual cores scale very very well - looks like it should be able to index 45-50 mln pages per 24H. Even better indexing is already very scalable so upcoming 3 boxes will work as part of the whole system, thus giving us ability to index 170-200 mln pages per day. This is more than we crawl now and it will allow to catch up with backlog, so we are doing well in this area.
Merging the last phase before actual searching that takes separate indexed barrels and merges them into searchable index - this area needs big improvements in order to scale to billions of pages. Merging right now is too linear not using more than 1 CPU - this is to be changed to use multiple CPUs and multiple machines, this is the key step to scaling seacrh engine up.
Searching - this needs to be scaled up as well, but luckily main work is done and so long as merging is scaled I expect this part to scale up easily.
So, its the merging's module scalability that needs to be addressed ASAP.
So what about recrawling?
A good search engine is not just a big search engine, but also up to date search engine - some pages change dramatically and people got accusstommed (def sp!) to searching for recent stuff online, and people are very quick to make a conclusion about whether search engine is good or crap - I am pretty harsh in this respect too.
We can only be reasonably up to date if we recrawl - most small search engines can't afford recrawling, so recrawl is not a dirty word, its actually good that we can afford to think in terms of freshness of information, not just its quantity.
As it was mentioned there is a lot of data flowing in every day - 1 TB+ of uncompressed data every day, it has to be well managed (indexed and serached) before we can dig much deeper.
Here are my considerations for recrawl:
1) first 2 bln pages are not as well compressed as they can be now - this means pages occupy 3-5 times more than they can be compressed now, thus recrawling them would free up some disks for index.
2) first pages we crawled were nearer the top of the sites than current deeper pages - they definately need to be indexed
3) some pages are as old as 1 year
4) recrawl gives me breathing space of around 2 months to scale the search engine - at 50 mln per day we can redo 3 bln in just 60 days.
5) new url loads should have much smarter prioritisation - current limit on urls per site is too generic, say Microsoft.com or GeoCities.com should not be subject to the same limit of 15k urls per site as newly formed doorway pages website with lots of junk - in order to be smart we need to index at least 1 bln pages and build info on sites importance.
This is why I think we should start recrawl of existing buckets rather than load more billions.
Exception: national TLDs (top level domains) will still be added, the reason is because there are so few of them comparing to .COMs/.NETs/.ORGs that there is a point to grow them as numbers are too small to create problems anyway. I hope people are okay with this exception.
Also, I want to state here very clearly important thing - the fact that we can recrawl data does NOT diminish in any way important of it being crawled in the first 9 months of this project - I have to admit here that thoughts of quitting this project in the face of impossibility of the task at hand crossed my mind a few times but this ended in April and won't come back - if it was not for people whose participation helped come through the hardest first 6-9 months of this project then it would have never reached this stage.
Will only changed pages send back?
Not right now - this functionality is trickier than it seems as it would require intelligent indexing/merging of multiple barrels with the same name taking only most recent pages from them - this requires all crawled data to be online, which it is not right now, but I expect this to happen in Q1 2006.
The only drawback of not having smart recrawl is that data send back will not be smaller, but right now with Conan active its pretty reasonably small - compare current barrel of 5 MB with 40-50 MB we typically seen in early days of the project.
Conclusion
It is good that we can afford to look into recrawl of data to keep it fresh - many search engines simply can't do that, we are maturing faster than I thought, which is a good thing :)
So what now?
Simple - old buckets will be marked as "uncrawled", and thus automatically put into recrawling. All your stats will remain the same - they won't be wiped out as crawled bucket stays crawled bucket regardless of whether its recrawl or not.[/quote]
Lagu :)