GPU clients

[Brucifer]

I have been looking at some of the discussion of the GPU issues and experimentation on the PS3Grid thread. And I'm assuming rightly or wrongly that those are all centered around the windows boinc world? I read with interest Bender10's comments on power adjustment loading on his home network by taking crunchers offline after adding GPU's.

Distributed.net shows a nvida GPU client for 64-bit linux. At present it is only working on rc5-72. I have never tried running a GPU client on any project. Don't even have a compatible GPU. Bluntly, I'm totally ignorant on the gpu scene period. Thus, I really don't have any idea where to begin on this. It would be nice if someone that has been playing around with the gpu's has tried the distributed.net client. Lacking that, could someone that is not at clueless as I on the subject take a look at the distributed.net download page for the beta stuff related to the CUDA beta client and then offer a bit of advice. Like what graphic card to get for starters... not the top of the line mind you, but a decently constructed card that is going to be able to handle the generated heat. Or if heat is going to be a concern, what mods for cooling the card need to be taken into account. Supposedly, there is supposed to be a pretty substantial gain in output from the cuda client.

And then that begs the questions related to is it worth while to do it rather than run standard quads, or put the bucks into a PS3 and go that route. Anyone playing with both the PS3 and cuda??

[Bender10]

The PS3 is a solid performer (I have 2). But with the addition of CUDA and GPU's, it's performance for PS3grid is only average. It does 1 wu every 20-23 hours, compared to the output of an 8800GT (~15 hours). $300 for a PS3 (~140 watts), and $100 for a 8800GT (maybe 100 watts).

The Ps3 does pretty good on dnet clients, as IronBits will tell you. It seems to do ok on OGR-NG (I have 1 PS3 on that now), similar to OGR-25

Until we find someone running dnet on a GPU, (to get some indepth info from) you are kinda stuck...

[Bender10]

!!HOLY CRAP!!

I just fired up my *nix 64 GPU box on RC5-72...

It is running 470 Mkeys/s...Is that a lot??

[Brucifer]

yeah that is just rocking right along for one system.

So could you give me a blow by blow of what you did, what you loaded, etc. It's impressive enough that I'll spend the bucks to do it. :icon_mrgreen:

If you run that hummer for four hours, tell me how many work units you complete on that system if you don't mind. Then I can do some comparisons. The mnodes per sec thing is good for the tekkies I guess, but in my old mind I go by work unit count on rc5. The other thing that would be nice to know is what wattage PS you are running, and if you have a power meter, what is the power consumption in watts of that system when it is running the cuda client on rc5.... :)

edit: Also are you running Xwindows or in text mode? And if you are using X, it the gpu providing graphics for the system too, or do you have another card doing that?

[Bender10]

Yeah,

I guess jumped in with a number too fast, anyway...

To make a better test of it, I suspended the PS3grid wu's, to take the extra load off the GPU's. Boinc is still running, Just no GPU tasks.

The 'work-o-meter' in the Dnet client went down from 470 Mkeys/s to a steady 225 Mkeys/s, with 2 GPU tasks crunching.

The 'work-o-meter' has started to go up again, with the GPU tasks suspended. Right now it is at 270 Mkeys/s and climbing. This setup is completing ~50 packets (points??) every 5 minutes. I'll keep track for a couple of hours to get a better idea.

I'm running Ubuntu 8.10 64bit (Xwindows), 660 watt ps, Evga 9800GX2, "kill-o-watt" meter says: 320 watts. That is running Boinc (4 boinc wu's, 2 PS3grid wu's) and RC5-72.

[Brucifer]

This setup is completing ~50 packets (points??) every 5 minutes. I'll keep track for a couple of hours to get a better idea.

I'm running Ubuntu 8.10 64bit (Xwindows), 660 watt ps, Evga 9800GX2, "kill-o-watt" meter says: 320 watts. That is running Boinc (4 boinc wu's, 2 PS3grid wu's) and RC5-72.

Holy crap..... are you really serious here? 50 units every 5 minutes???? Like 600 an hour????? 14,400 a day???? That isn't fast, that is phenomenal. That is just totally unbelieveable :shocked:

[Bender10]

I guess so. I've done ~1,180 packets since I started this test a couple of hours ago.

And part of that time, the RC5 client was sharing the GPU's with the PS3grid wu's. I still need to find out how much the GPU wu's slow down when running with the Rc5 client...

[Brucifer]

At that production rate you shouldn't even waste the time of running rc5 on the cpu cores cause there just isn't any comparison between the two. Run your boinc stuff on the cpu cores for projects that aren't supported with the gpu.

Well, no doubt about it, I've got to go out and buy a graphics card now. Also need to get a stout PS too. So basically for a dedicated rc5 cruncher, all one needs is a system that will support the graphics card and a good ps and just let that rock on. Don't need quads or anything fancy really. Get three or four of those bugger running and shut the other stuff down to conserve power, and there would be some ferocious rc5 crunching.

I guess the question is how long those cards will run like that before they self destruct????

[Bender10]

This is ONLY a TEST. I'm only doing this because you got me curious. Not sure where I will go with this.

Also, My output is DOUBLE what a normal nVidia 9800 would be.

I have a 9800GX2....

2 GPU's on 1 double width card. It uses ~160 watts all on it's own.

About the life span. Keep 'em cool, and they should last. A single 8800GT/9800GT is only ~$100-$120, If yo look for a deal.

[Brucifer]

How much noise does the card make??? How much did the card cost?

[Bender10]

It's in the basement, can't hear it the living room.

[Bender10]

It's in a rack with 4 other computers. It makes a little noise. Hard to say. Sorry.

[Brucifer]

just sitting here laughing my a$$ off.... For those of us that have to sit in the same room with the thing, gotta wear ear plugs??

[Bender10]

No, its not bad. Like a cruncher with 80mm case fans (small high rpm fans, you know what I mean?)...

Update: 3 hours RC5 = 1273 packets. (400 per hour)

I turned on the GPU tasks for the evening, to see how that works (with RC5 running). I'll let you know Monday.

9 hr run: ~200 packets per hour (everything running... 4 Boinc wu's, 2 PS3grid wu's, and RC5)

[Bender10]

I am kind of a dnet noob.

Right now I am running Boinc wu's, PS3grid (gpu) wu's and RC5 on the same box. The RC5 output is reduced due to having to 'share' the gpu with PS3grid (which is also taking a performance hit). So I did 2 '-bench' runs. With RC5 and PS3grid wu's sharing (A). And RC5 running solo on the gpu (Boinc still running) (B).

A results:

[Dec 01 14:08:18 UTC] RC5-72: using core #0 (CUDA 1-pipe).
[Dec 01 14:08:37 UTC] RC5-72: Benchmark for core #0 (CUDA 1-pipe)
0.00:00:16.11 [197,038,435 keys/sec]
[Dec 01 14:08:37 UTC] RC5-72: using core #1 (CUDA 2-pipe).
[Dec 01 14:08:55 UTC] RC5-72: Benchmark for core #1 (CUDA 2-pipe)
0.00:00:16.19 [172,660,517 keys/sec]

B results:

[Dec 01 14:10:59 UTC] RC5-72: using core #0 (CUDA 1-pipe).
[Dec 01 14:11:17 UTC] RC5-72: Benchmark for core #0 (CUDA 1-pipe)
0.00:00:16.14 [241,531,335 keys/sec]
[Dec 01 14:11:17 UTC] RC5-72: using core #1 (CUDA 2-pipe).
[Dec 01 14:11:37 UTC] RC5-72: Benchmark for core #1 (CUDA 2-pipe)
0.00:00:16.84 [202,234,189 keys/sec]

I hope this helps.

[Brucifer]

yep, most definitely, and extremely impressive. In case of "A" it is 15 times faster than a stock q6600 using 4 cores, and on "B" it is 18 times faster....

If you are using 340??watts versus 110 watts roughly, then that is also extremely impressive in the power differential weighted against units completed. Most impressive indeed. I thank you very very much for taking the time to run the tests. Even if the work is cut in half for one GPU rather than your double, it is still most impressive, 8 and 10 times quicker respectively as measured with your core(0) and four q6600 cores. Still a massive power savings versus work crunched.

Thank you!! :icon_mrgreen:

[Bender10]

I'm glad I could help. Your questions got me curious, and since I had a *nix GPU setup, I figured it was worth a go.

Not really scientific, as I don't usually do formal testing, but it worked.

[Brucifer]

Well it was scientific enough for me! :icon_mrgreen: Sure answered what I was wondering about.

Now, I guess I need to load up an Ubuntu system, go buy a graphic card, and then figure out how to install the driver for it.... ??

[Nflight]

I am impressed after reading this post to the forum. There is a lot of potential in running things just a wee bit differently then BOINC which I am really tired of. I know I entered into the Race this month and will run it as long as I can stand it. But I really prefer to run non-boinc projects.

[Brucifer]

@Bender10 -- Okay, I installed Ubuntu 8.10, Also have a 9800GT w/512m/b mem installed. Went to get the dnetc cuda client. First issue.... libraries. So I guess I need a blow by blow for libraries, drivers, etc. Also this isn't KDE erk it's Gnome. Not a gnome guy. The gpu was in the system when I installed Ubuntu, so I'm ASSuming that it recognized the video card???. Where do I get driver, libraries, and anything else needed???

[AMDave]

Brucifer,

I am using Gnome Ubuntu 8.10 on a Phenom 9750 with a 9600GT and the 177.x version of the NVIDIA driver,
which your Ibex should load for you under
System > Administration > Hardware Drivers

I had a quick chat on IRC #distributed and per the note under the client download we "should" be installing the CUDA library first.
That is the correct way.

But then I found this interesting trail (plus others that were similar):
http://forums.nvidia.com/lofiversion/index.php?t81442.html

So, I cheated on the libraries :P
I copied the cudaart library from the BOINC GPUGRID client into my dnetc-cuda folder

(my thinking was "Hey this is an ELF version", and this ".so" file is kind of like the ".so" files for eOn and they live in the same folder, so I'll just try that...)


cp ~/BOINC/projects/www.ps3grid.net/libcudart.so.2 ~/dnetcuda/
and it "just worked"
Awesome.
heh heh heh

I just cleared my GPUGRID cache.
I see Bender10 flying up behind me
Time to get rolling, rolling rolling, RAW HIDE!:icon_twisted:

(Just because I got away with it this time does not mean that it will work with the next version of the client. We will see what happens then)

PS - Gnome is better :P

[AMDave]

This page
http://bugs.distributed.net/show_bug.cgi?id=4030

advises (comment #5) downloading the CUDA libs from
http://developer.nvidia.com/object/cuda.html#downloads

I may have to re-do.
My client won't accept OGR-NG in preferences. :(

[Bender10]

I can help out later today (after werk). I see AMDave has posted some good info. I also should have a procedure for adding the nVidia driver (*nix noobish here also). I am currently using 180.60 (for PS3grid).

And yes AMDave, just copying the libcuda.so(?) file to the dnetc folder worked for me also.

[Bender10]

Oh what the , i'll be lat for work today...

Try this for the driver install. I'm not sure about the "gdm" stuff, that may/may not be necessary.

Download the drivers from the NVIDIA website into a directory (for instance /tmp)


This works with my ubuntu 8.10. But i lost the page with my procedure. You may be able to skip: root login, and start/stop gdm. If the sh "driver file.run" does not work, try adding sudo before sh

1* Press ctrl-alt-F1, a text login screen will appear

* login as root (this might not be necessary in ubuntu, skip?)

2* type "sudo /etc/init.d/gdm stop" to stop the X server

3* change to the dir where you saved the driver file

4* type "sh NVIDIA....pkg.run" to install the driver from the directory where you had downloaded it (use name of driver file)

5* answer yes to all questions from the installer (it will want to re-compile the kernel, this is ok)

6* type "sudo /etc/init.d/gdm start" to start the X Server again

Play with that

[AMDave]

Go to work.

No, Boss.
I haven't seen Bender10 on any forum this morning.
He must be delayed in traffic :D

Psst - don't forget the coffees. Mine is the Large Cap. with none, and a Flat White with *10* sugars for the boss, to sweeten the day ;)

[Bender10]

made it with time to spare (for my morning oatmeal, coffee later..)

[AMDave]

Hey, where's my coffee.
Bah.
Late order, huh. Srry.
Off to get my own.
BRB.
;)

[Bender10]

I'm running OGR-NG on 1 of my PS3 boxes. the other one is now a holiday game/movie machine.

My entrance into RC5 is actually only limping along now. I am only running it on 1 GPU (of my 9800gx2), and that is sharing cycles with PS3grid wu's. I think it's running around 100Mkeys/s. The other half is still full time PS3grid tasks.

[Bender10]

No, I'll get the coffee....may be a few minutes ....hangon...don't get your knickers in a....grumpy old sod...the things I do for a friend...must be half barmy that's me...

[AMDave]

50% of your 9800gx2 is going to take my 9600GT out to the car park and bloody its nose.

Maybe if I close my eyes and wish really hard, I will get one of those for Christmas?

Either way, I will be eating your dust in a couple of days.

Thanks for the comment on the NG.
I was wondering if your GPU client was crunching NG work but apparently not.
The info says NG support is there, but it no crunchy for me...yeti.

[Bender10]

I'm in the process of building another *nix box just for testing (waiting for a MB). As I am a 1 *nix farm so far. Then I will crank up a spare 8800 I have and see how that works with RC5 (so i can get back to PS3grid full time...).

[AMDave]

Uh oh. Then I am toast!
As far as I know the 8800 still beats the 9600GT.

Hey, speaking of toast, thanks for the coffee.
My turn next time ;)

[AMDave]

Maybe if I close my eyes and wish really hard, I will get one of those for Christmas/
On second thoughts, I'd better not push that too far with the cashier.
I did get a new ATI card last month that I haven't got around to fully stress-testing yet, although it has already given a good fps in Cube2.

[Bender10]

Don't worry, It (8800) would not be for full time production, just to test and maybe run on the weekends. I have to curtail my energy usage a bit for a little while until I win the lotto...

It just started to snow. So i guess I should go outside and do some pm's on our vehicle radios...

[Brucifer]

Brucifer,

I am using Gnome Ubuntu 8.10 on a Phenom 9750 with a 9600GT and the 177.x version of the NVIDIA driver,
which your Ibex should load for you under
System > Administration > Hardware Drivers

I had a quick chat on IRC #distributed and per the note under the client download we "should" be installing the CUDA library first.
That is the correct way.

But then I found this interesting trail (plus others that were similar):
http://forums.nvidia.com/lofiversion/index.php?t81442.html

So, I cheated on the libraries :P
I copied the cudaart library from the BOINC GPUGRID client into my dnetc-cuda folder

(my thinking was "Hey this is an ELF version", and this ".so" file is kind of like the ".so" files for eOn and they live in the same folder, so I'll just try that...)


cp ~/BOINC/projects/www.ps3grid.net/libcudart.so.2 ~/dnetcuda/
and it "just worked"
Awesome.
heh heh heh

I just cleared my GPUGRID cache.
I see Bender10 flying up behind me
Time to get rolling, rolling rolling, RAW HIDE!:icon_twisted:

(Just because I got away with it this time does not mean that it will work with the next version of the client. We will see what happens then)

PS - Gnome is better :P

Still trying to wrap the brain around this whole process here. It's like a bunch of things/geeks trying to make this more complicated than it needs to be. Assuming that the only purpose that the 9800 would be in the system for would be for the sole pleasure of the dnetc cuda client, why would one need to install the video driver for the card as it would not be utilized for video. It would seem that the only things that should be needed would be the cuda libraries and the nvida client, and the 9800 card. So what am I missing here???

IRT to your thread above, I downloaded the NVIDA driver and the .run file for OpenSuse 10.3. Where do the libraries come from? Am I supposed to do something to use the .run file? If so, then what? I did now see anywhere that I could download the ~.so library files...

I would prefer to just keep using the plane jane PCI card (non pci-express) for video that I'm using now. sigh.....

[Brucifer]

So is anyone up on the status of the upcoming ATI card (double precision) and ATI's version of cuda? Any projects going to run with it right off the bat?

[vaughan]

Folding@Home. (http://foldingforum.org/viewtopic.php?f=51&t=7471)

"A new ATI core (v1.22) has been released. It includes optimizations made by AMD (Mike Houston) on both the CPU and GPU side. Several bug fixes are also included."

Sorry Brucifer - Windows only. :icon_wink:

I haven't tried it yet due to the PrimeGrid Summer/Winter Solstice Challenge. The strongest ATI card in this household is the 4870 in my son's gaming machine so I doubt I'll be allowed near it. :icon_razz:

In other news, I'd like to know when Seti are going to fix their beta site so that it can accept all the GPU tasks I've crunched for them and my boxen keep trying to upload the completed work. Very frustrating.

[Brucifer]

Ah Mr. Vaughan......................... :icon_mrgreen: Gimme a break :icon_mrgreen: Windows?????? Windoze???

LOL Yep, on the serious side, it looks that way for the time being anyway. I'm sure though that there will soon be linux variant out there, primarily because of the double precision which will allow the possibilities for the GPU to be used in math related projects. And if it doesn't happen, then I guess things will just have to progress along seperate paths. I would honestly believe that double precision is in work for the nvida side of things in view of their efforts in behalf of the scientific community. And that is also what helps me believe that ATI et al will get a linux package out too. Getting back to a few basic things, while windows runs rampant in the business world, there is still a lot of linux there too, and one must not forget that in the university and scientific world there is still heavy Unix utilization.

I do have a windoze box here....:icon_rolleyes: infact I have two of them now. One the spouse uses, and I have one for compatibility issues. Who knows, at some point in this game I might just have to try an ATI card in that system................... :sad5:

LOL

Yea, it's unfortunate indeed that you are having problems getting your crunched work uploaded to the Seti Beta site. Of course being beta that's the bummer part of the deal, but that stuff has to be done in order for the overall effort to move forward.

This is really an interesting time in the DC world as we are witnessing a major change occurring in crunching. And while I'm not a nowdays participating fan of F@H, Dr. Pande deserves a large degree of recognition for his efforts in pushing forward use of the cell and GPU for DC crunching by the "man in the street" (that includes the ladies too) that would not have come along as fast as it has. On the commercial business side of the street Sony/IBM with their contributions with the cell really got things rocking along. And Nvida's release of the CUDA toolkit was a major boost. For some time AMD has been working in the direction of offloading tasks from the CPU, so it surprised me really that after their acquisition of ATI that Nvida beat them to the punch with the development kit on the street for their GPU first.

The other thing that is interesting to me is the bang for the buck perspective in all this. While the higher end GPU's are not cheap, they are still within reach for most of us, and their performance is well worth the cost. Yesiree Bob, this is really an interesting and fun period in DC computing!! :icon_santa:

[Brucifer]

When you get bored, go take a long read on the distributed.net site in the comments stuff on the cuda client bugs thread. Pretty interesting really. Sounds like there should be the rc5 cuda client out for windows before much longer along with the updated linux one also.

[vaughan]

OK, Doomeva and I have been trying to get the RC5 DNetc app to install and run under Ubuntu64 on my Opteron 165. We get this error:

RC5 cuda: CUDA CORE ERROR: setting the device when a process is active is not allowed

We have d/l and installed Nvidia Cuda 180.06 (although the site calls it 180.60 just to confuse us).

We installed Dnetc and configured it with the e-mail addy and it d/l 24 work units. However, it still gives the CUDA CORE ERROR.

Doomeva has stormed off in a huff saying the Dnet programmers are lazy and haven't configured it to look in the directory where Nvidia has put the CUDA files.

Any help appreciated. :sad5:

[Bender10]

I just ran a test with the Windoze client. Works fine, but it is not ready for crunching yeti. It just does random units and will not SCORE them...

But It gave me a chance to -bench another GPU for future use.

[Brucifer]

OK, Doomeva and I have been trying to get the RC5 DNetc app to install and run under Ubuntu64 on my Opteron 165. We get this error:

RC5 cuda: CUDA CORE ERROR: setting the device when a process is active is not allowed

We have d/l and installed Nvidia Cuda 180.06 (although the site calls it 180.60 just to confuse us).

We installed Dnetc and configured it with the e-mail addy and it d/l 24 work units. However, it still gives the CUDA CORE ERROR.

Doomeva has stormed off in a huff saying the Dnet programmers are lazy and haven't configured it to look in the directory where Nvidia has put the CUDA files.

Any help appreciated. :sad5:

Are you using the 64-bit version of Ubuntu? Are you using the 8.10 version of Ubuntu? If you are using the 64-bit, use the 177 version of of the video driver. You also need the cuda library files. PM me your email addy and I will send you a copy of what AMDave sent to me, or get them from Dave if he is handier for you. I will be here all day and will watch for your PM. That should get you running. :)

[Brucifer]

I just ran a test with the Windoze client. Works fine, but it is not ready for crunching yeti. It just does random units and will not SCORE them...

But It gave me a chance to -bench another GPU for future use.

What are you talking here, the boinc client for working on seti beta? Or the in work version on the dnetc one?

[vaughan]

Are you using the 64-bit version of Ubuntu?Yes.
Are you using the 8.10 version of Ubuntu?Yes.
If you are using the 64-bit, use the 177 version of of the video driver.OK, I was using 180.06
You also need the cuda library files.I have them installed and the 3D test "gears" works
PM me your email addy and I will send you a copy of what AMDave sent to me, or get them from Dave if he is handier for you. I will be here all day and will watch for your PM. That should get you running. :)

[AMDave]

tar.gz sent

180.xx should be ok as well

3 days left on this version of the client.
Keeping the eyes peeled for the new version

[vaughan]

Thanks got it.

I'll wake the night owl soon and see if we can get this application sorted. (He went to bed at 3am, maybe he's working in the South African timezone or something :icon_razz: )

[Brucifer]

yeah, I sent it too. I logged on here, saw the PM, and sent the file. Looked in here and see that AMDave was Johnny on the spot and beat me!! :icon_mrgreen:

[vaughan]

Its running nicely. 215M keys per sec. Is that good or what? GPU Temp has gone from 48C to 59C so it appears to be working. :icon_wink:

Thank you for the help.

[Brucifer]

Glad to hear you are up and running. :) What card do you have???

[vaughan]

Inno3D 8800GT

[AMDave]

note from the dnet dev-log - they appear to be on track and alpha testing is going well

Hopefully we can get a beta release before the current beta expires on Tuesday.Sounds promising

on the down side another dev said:
I have been looking at implementing an OGR core, but the OGR code is much harder to understand, so this might take some time.but at least they have it in their sights

good news for liuqyn, the next CUDA release does have a client for Windoze

[liuqyn]

good news, seti cuda not so good yet. didn't really want to fall back to GPUGrid either at this point.

[Brucifer]

So any hot gossip on what the latest and greatest is going to be coming down the road next for both nvida and ati?