Majestic - 12

[Steve Lux]

Thanks for coming back to MJ-12 Sheff. We were rapidly losing ground and ARS Technica would have had us within a month.

Not much action here in MJ-12, nor much team support for our placement, but I've been giving it everything I have these last 6 months to resist losing placement. Folks come buy and crawl for a little while, but don't seem to stay long.

If you haven't yet, you need to upgrade to the newest MJ-12 client. It added a much-needed 20% to my crawling capacity.

[AMD-USR_JL]

Did you say Ars Technica was about to pass us? I'm going to have to start crawling again.


UPDATE: According to Free-Dc, it's just 24 days until they overtake us there too, i hope my bit helps.

[AMD-USR_JL]

I keep on getting an "internalError" saying that there is not enough disk space to upload a barrel. I've got 150GB left on my drive, but it says "internalError", so is it them or me?

[Evil-Dragon]

It's them, Alex needs to change HDD's again.

[Steve Lux]

AMD-USR_JL, thanks for coming back. Your help is much appreciated, but we still need more to keep from falling behind. We are close to pacing them, but ARS Technica is still gaining on us.

[AMD-USR_JL]

According to Free-DC we beat them by 100k yesterday, but right now they've got us by 100k

[AMD-USR_JL]

Thanks for the help Vaughan. Now we have them by almost 300k today.

[Steve Lux]

No kidding, we're starting to look healthier now. With this newer version of MJ-12 I've been able to run 15 crawlers and not lock my system up, something I was unable to do with the older version.

[Steve Lux]

Terry and Dark13Mage, welcome to Majestic-12. Thank you for your assistance. With a bit more help we could hold off ARS and X Grubbers.

If you don't have unlimited bandwidth you should adjust your crawlers and upload/download speed to ensure you don't go over your provider limit.

[gatekeeper53]

Steve it's not my bandwidth it's my router that gets blown out. the bandwidth is unlimited.

[Opteron]

Steve it's not my bandwidth it's my router that gets blown out. the bandwidth is unlimited.

What kind of router do you use gatekeeper53"

[AMD-USR_JL]

It's much appreciated Terry and DarkMage. :) Sorry i hadn't been there for the past couple of days (been doing a lot of BitTorrenting), but i should be back in a few days.:icon_mrgreen:

[Steve Lux]

Steve it's not my bandwidth it's my router that gets blown out. the bandwidth is unlimited.

Terry; that was for any new user who might be interested in joining Majestic-12. I want to encourage team members to help out, but wouldn't want anyone going over their allotted limits.

My ISP account claims to be unlimited, so I'm not worried about how much traffic I do, but I understand that some folks have a monthly limit and I wouldn't want them to get charged extra.

So, if someone has a limit on their monthly unload/download traffic, just start slowly with only a few crawlers and low limits on your upload and download speeds, then watch your traffic per 24 hours, per week and then per month and adjust accordingly to ensure you don't incur charges for exceeding your limit.

[gatekeeper53]

What kind of router do you use gatekeeper53"


It's a Cisco 831. I'll just have to do some reading lol

And Steve there is no limit

[AQUAJOE]

Steve I am here to help with MJ12. I'll do what I can.

I wonder if we could get a few others to run this during the day when they are at work or school. Looks like we could use a boost.

If you set the client up right. This will not take anything away from your CPU crunching. It is easy to check to see if you have set it and will use CPU.
In task manager look for the mj12bar.exe. If that is running it is taking CPU cycles.
I have my machines Set to 20 async workers and 15 buckets. If I go any higher the CPU kicks in.

On some machines that threshold is lower so watch for that and you can crunch away and not affect your other projects.

[Steve Lux]

Yes, I have noticed you've been putting in some nice numbers AquaJoe. At least double what I can do. With a bit more help we could maintain our placement.

[Steve Lux]

MJ-12 seems to be down, or it could just be their stats page isn't working well with their newest version (1.5.0).

[Steve Lux]

Just happened to catch this. Likely we won't see AMD in the lead often - except as a fluke.

1 AMD Users (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=2) 103,234 2,019
2 Knights Who Say Ni! (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=24) 100,693 1,564
3 Rechenkraft.net (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=26) 88,136 1,690
4 Team Norway (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=28) 75,655 1,802
5 X Grubbers Kick Ass (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=12) 67,640 763
6 Team Haiku (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=88) 56,134 1059
7 Ukraine (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=97) 48,298 912
8 French team (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=83) 35,136 514
9 XtremeSystems (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=60) 31,148 563
10 IGA (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=18) 25,687 535
11 Team-Goobee.org (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=113) 9,175 209
12 Ars Technica (http://www.majestic12.co.uk/projects/dsearch/teaminfo.php?id=23) 8,798 185

Total (17 teams (http://www.majestic12.co.uk/projects/dsearch/teams.php)) 683,887 11,815

And on a more personal note, this is likely the only time I'll see this:

Top 12 users (Today)
__#__Nick____________URLs_______
_____________________done_______Data__
_________________________________(MB)__
__1__http://www.majestic12.co.uk/images/flags/small/us.gif Steve Lux (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1284)_______192,706___3,755
__2__http://www.majestic12.co.uk/images/flags/small/england.gif KWSN Grizzly (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=430)_____145,982___1,968
__3__http://www.majestic12.co.uk/images/flags/small/us.gif KAMCOBILL (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=281)_____102,243___1,329
__4__http://www.majestic12.co.uk/images/flags/small/up.gif Oleg82 (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1538)________63,671___1,211
__5__http://www.majestic12.co.uk/images/flags/small/gm.gif guybrush242 (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1698)______61,981__1,295
__6__http://www.majestic12.co.uk/images/flags/small/fi.gif TTT (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=588)___________57,950___1,282
__7__http://www.majestic12.co.uk/images/flags/small/fr.gif valhalla (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1337)_________55,777___860
__8__http://www.majestic12.co.uk/images/flags/small/us.gif kharri1073 (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1573)________51,028___860
__9__http://www.majestic12.co.uk/images/flags/small/no.gif carlfrs (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1208)__________47,210___865
_10__http://www.majestic12.co.uk/images/flags/small/us.gif Elekt (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=351)__________45,827___938
_11__http://www.majestic12.co.uk/images/flags/small/sw.gif Mildew (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=431)_________45,251___829
_12__http://www.majestic12.co.uk/images/flags/small/ca.gif TheNerd (http://www.majestic12.co.uk/projects/dsearch/userinfo.php?id=1832)_________44,095___809

___Total (46 users (http://www.majestic12.co.uk/projects/dsearch/stats.php))1,412,03325,499


At the moment I'm having trouble connecting to MJ-12. Their server seems to be down. These guys usually kick my tail consistantly, I must have had more spare buckets cashed than they did.

For the moment: Were number one!
(until they fix their system that is)

[NeoGen]

I've been slowly running this for the past few months and am already 24th on the team. :)

[vaughan]

I'm unable to upload results today.

[Steve Lux]

Yeh, they are still down, I'm still #1 in the world and AMD Users is still the #1 team in the world. (all because I seemed to have had a bigger cashe of buckets to crawl stored up than most).

Anyhow. This is certainly taking longer to fix than I had expected. Also, I am expecting that many more folks will keep a larger reserve of buckets to crawl.

One good thing about this MJ-12 downtime; CCPROXY.EXE (the Norton's Virus) hasn't been stealing reources away from Einstein crunching.

[Steve Lux]

I'd like to welcome WAR to the party and the team on MJ-12.

[vaio]

Damn U.K limited ISP's hold me back on this :icon_neutral:

[Steve Lux]

You know you can control how many crawlers are active and how fast they crawl, right?

[NeoGen]

I wish more people could run this. I've set MJ-12 to run from 1:00 to 9:00 am every night (happy hour on my ISP) a few months back, along with p2p applications, and bit by bit I've reached the top-20 on the Team. :)

Keep an eye on our #1 member FiddleAbout, he will reach 1 billion URLs very soon! :icon_eek:

[drezha]

I wish more people could run this. I've set MJ-12 to run from 1:00 to 9:00 am every night (happy hour on my ISP) a few months back, along with p2p applications, and bit by bit I've reached the top-20 on the Team. :)


I'm interested in running it...How do I get it to set it running at a certain time and how do I figure out the optimum settings?

Also, will it interfere with my webserver on the same connection? In fact, can I run it on my server as well (both server and main rig?)

Speedtest shows me as 500kb/s download and 71 kb/s upload...Think someone in the house is running P2P mind. As we should have 2Mb broadband.

[sentient_life]

I'm already running Dimes. I'm hesitant to tie up my internet connection too much more.

[liuqyn]

I'm going to try it out on one of my rigs when I get home, will only run from 3-6 am though as that is free time on my isp.

[NeoGen]

I'm interested in running it...How do I get it to set it running at a certain time and how do I figure out the optimum settings?

Also, will it interfere with my webserver on the same connection? In fact, can I run it on my server as well (both server and main rig?)

Speedtest shows me as 500kb/s download and 71 kb/s upload...Think someone in the house is running P2P mind. As we should have 2Mb broadband.
I'm at work right now, so I can't check the exact options, but the MJ-12 client has options to set everything from download/upload speed limit, schedule to run, ammount of worker threads, etc...
There's alot of help of their forums, and ours here as well, if you need to fine-tune it in a specific way.

But beware, it will indeed interfere with the webserver if you set it to full speed, because MJ-12 is a *major* bandwidth hog if you don't set it limits. I believe it's sufficient to run it on only one machine, unless you have more than one internet connections, in which case you could run it in one machine per net connection.
Also, it is advised to have it on a relatively strong machine, because after crawling, it compresses the hundreds of megabytes of results before uploading back. (To save on upload bandwidth). A slower machine might spend too much time on the compressing stage and not be crawling as much as it could.

[drezha]

Cheers.

I'll look it up later then :) I seem to remember I can select preference on what domains to crawl as well. - cant impact crunching much as it'll run on a Quad core and so it'll only need to zip on one core.

[drezha]

Got it installed and finally got it working so it will only crunch at night between 12 and 9.

I'll keep an eye on it to see what happens. Max of 1Gb a day crawling...If it ever gets to that.

[Steve Lux]

Installed a new printer/fax/all-in-one unit early this week. As per the directions I installed the unit in-line with the DSL before it gets to my modem, in order for the fax to work. It has about killed my speed. I'll try troubleshooting the issue this weekend, and may have to forget the fax and hook it up to one of the ports on my router. Until I get it fixed it's a really slow crawl in MJ-12 for me.

Then again, if BOK finds out that some of the top scores credited to our team in the FreeDC site stats are from folks that aren't actually on our team - we'll be off the radar scope anyhow. (Yes, I know he reads this - but it should be fixed anyhow)

[Steve Lux]

Good news for MJ-12'ers;Between the new version (v1.6.0) and Norton's 2008 it seems we can do a bit more work now. I've been able to crawl about 1.5 million URL's per day for the last two days - since upgrading both, and I'm still not done tweaking the settings for my max output. Happily, Norton's (2008) isn't such a resource hog any more - which helps significantly, and also increases my average Proteins output (look out PoorBoy).

Evil-Dragon and junkbuster better up their output, I'm less than 200 million URL's behind you.

(Perhaps a little junk-talking can go a long way?)

[Steve Lux]

Welcome to the MJ-12 party Backwar. Majestic-12 isn't a very popular project, so your help here is greatly appreciated.

[Steve Lux]

Waving at Evil-Dragon as I zoom by.

Keep going junkbuster - I'm coming after ya!

[Frederic Brillouet]

Heheh, Steve's on a roll! Keep on going!

[Steve Lux]

And thanks to Anih coming back and helping with some superior output we may actually be able to stay in the top 10. The Powerful Cows from the Netherlands have really started to show their power in MJ-12.

[Frederic Brillouet]

And thanks to Anih coming back and helping with some superior output we may actually be able to stay in the top 10. The Powerful Cows from the Netherlands have really started to show their power in MJ-12.
yeah one of them has 100 mbit cable with unlimited download and upload. it costs 20 euros there. we have to pay 45 euros for 10 mbit with 12gb download limit (which I will break if the q6600 is able to connect when its finished wu's only on primegrid!)

[Steve Lux]

I am running Einstein @ Home at full speed on both of my system's Opteron 185 cores (and play WoW) and it seems to have no adverse effect on MJ-12 running with max 80 crawlers and a max of 40 open buckets. I'm not using anywhere near that much though as my system won't seem to use more than about 75% - 80% of my ISP's rated capacity. Still, the Opterons being server processors, seem to be able to handle communications based applications far better (~8 times better) than the old 3800+ dual core that initially came with my system.

[Steve Lux]

Was gone on vacation for 9 days. Left my "confuser" on (Bender10's term). Came back to see some old MJ-12 crawlers back in action. A few more heavy hitters, or several more not as heavy hitters and we might be able to take back our placement from Team Haiku.

In any instance; I'm aiming for a billion URL's crawled. Anyone else want to join in the chase?

[sentient_life]

I'm there with ya. Just started up a week ago. I should have joined up a long time ago, with how fast the "credits" come. Haven't noticed any CPU hit, so it's running synchronous with BOINC. I'd like to run it as fast as possible. It's currently set to use 90% of my 5MB downstream, 60 async workers, and 20 open URL buckets. Any other tweaks to get going a bit faster?

[Steve Lux]

Well, I have found that a CPU designed for servers works much more effeciently than other types (I swapped out my multi-media AMD 3800+ dual core with an AMD 185 Opteron about 15 months ago and got about a 7 to 8 fold increase in crawling productivity - even though the board manufacturer said the board wasn't designed for the Opty).

I'm no expert at tweaking the MJ-12 settings, so the best I can suggest is reading their boards. Somehow I went from 3 million URLs/day down to 2.5 million or less after a system restart several months ago. I've tried several tweaks since then, but have not been able to return to my previous output. Reading my MJ-12 log I'm getting quite a few errors, so it looks like I need to go study up on the boards a bit myself to see if I can fix whatever problem my system is having.

[Steve Lux]

Looks like junkbuster, who was our team's #2 in total output has left us for the Dutch Power Cows.

Now they'll gain on us that much more quickly. Ouch!

[sentient_life]

For some reason I'm getting a ton of DNS errors with the client. After moving to KY, this ISP just doesn't seem to like crawling sites as much as my previous location. :(

[Frederic Brillouet]

I can still use majestic 12, but I dont know how my university will react when they discover I've already downloaded 500gb of data.. thats why I've shut down the client :-(

[[NGS]Cpt00Kirk]

Looks like junkbuster, who was our team's #2 in total output has left us for the Dutch Power Cows.

Now they'll gain on us that much more quickly. Ouch!


Well i don't know why he changed!! he is not @ our forum i think :dontknow:

:wav: But we welcome him to the Dutch Power Cows :wav:

May the grass be greener on our fields :icon_mrgreen:

[Frederic Brillouet]

Cpt00Kirk;61626']
Well i don't know why he changed!! he is not @ our forum i think :dontknow:



:wav: But we welcome him to the Dutch Power Cows :wav:



May the grass be greener on our fields :icon_mrgreen:

your grass is just littered with cheese, cheeseheads! (het alom bekende "kaaskoppen" in België)

[sentient_life]

After all this time, we were finally able to figure out what was causing so many DNS errors and other problems browsing the internet. The NAT Filter was enabled on the router. It wasn't the ISP's fault for once. I've turned the Majestic-12 client back on, and it's back to normal -- 90% successes!:blob3: