AMD(ATI) GPU apps on PrimeGrid

[AMDave]

PrimeGrid just posted some ati clients onto their PPS(Sieve) sub-project.
[/URL]corrected link --> [url]http://www.primegrid.com/forum_thread.php?id=2799 (http://amdusers.com/forum/newthread.php?do=newthread&f=72)

So far they are getting "No work for application" but that is an improvement in itself ;)
The problem has been posted in the thread above and hopefully someone will report it as being fixed soon.

I have attached 3 x HD5770
Are you ready?

[AMDave]

I got 3 out of 4 running.
1 x linux 64, NVIDIA 9600GT + Cuda23 resumed right where it left off.
1 x WinXP 32, ATI HD5770, 10.10 APP 32 driver, inserted the x86 boinc client and created the app_info.xml and it 'just works'
repeated the same process for
1 x Win 7 64, ATI HD5770, 10.10 APP 64 driver, inserted the x86 boinc client and created the app_info.xml and it 'just works'
(32 - there is no Win 64 bit client but the Win 32 bit client works just fine on a 64 WIn+64 BOINC install)

I'll have a go at the linux 64 setup later.
I just ran out of weekend. :(

lessons learned
Use the ATI 10.10 APP version driver.
It 'just works' and it's a lot simpler than fiddling with the Stream 2.2 SDK
The ATI 10.11 APP version does NOT work for these clients (not in my case anyway)
The ATI 10.11 version is NOT officially supported with Stream SDK 2.2.
The Stream SDK page shows the SDK is only tested as working with 10.9, but if you use the APP 10.9 or 10.10 you should not need the SDK anyway.
Looking forward to the linux 64 install when I can make some time free.

[AMDave]

Update - rebirther does have his ATI PPS(Sieve) client running on WIn7-64 with the 10.11 APP version of the driver.
As I posted there, I may have had the non-APP version installed on the machine which does not have the OpenCL components included and hence it would have failed.
The whole setup is version specific, you just have to be very clear about which versions you have installed or are downloading to install.

I can post my app_info.xml, if anyone wants it. Just let me know.

[AMDave]

Oooh. Nice one Dirk Broer.
Just jumped over me in the stats on this sub-project.

[Dirk Broer]

Using a Nvidia GTX260 (original 192 cuda cores variant), scoring 51,259 credits yesterday, and at least 100,800 more today.
But whichever stats I look at, AMDave is still ahead of me. I may have jumped you in daily stats though..

[AMDave]

Phew! I got back past you Dirk :D
Full steam ahead (minus the 5 warmest hours of the day)

Anyone else have their ATI/AMD cards on this yet?

[Dirk Broer]

I just added my beloved HD3870 in an attempt to pass you as the GT9500 in my wife's PC only contributes when she is using it. The HD3870 and the GTX 260 (192 cores) will hopefully outpace your three HD5770s and GT9600 GPU team:icon_razz: So far no work for the HD3870 though...looks like I need AMDave's app_info.xml file

[Dirk Broer]

Looks like I've bitten more than I can chew: there seems to be no way to get a HD38xx to work at an OpenCL application. The private battle between AMDave and me has boosted AMDusers RAC to the 7th current on PrimeGrid, while we are the 20th team in total credits. Just a few more days for "The Knights Who Say Ni", and we will be 19th.

[Tamaster]

Well... I think I'm very close to giving this a shot. I've been getting my HD4850 rigged and rolling and have successfully Benchmarked (http://global.phoronix-test-suite.com/index.php?k=profile&u=tamaster-13018-32407-32281) the OpenCL performance with Stream SDK v2.2 and have upgraded to BOINC 6.10.58 and established a PrimeGrid account.

Now all I need are the last bit of tweaks to get the necessary WUs to 'Come to Papa' and get crunchin'...

I'm hoping one of you that have 'been there, did that' can give me some pointers. I'm on the PG message boards now digging through the cruft to get to those tweaks.

Thanks in advance

[AMDave]

Looks like I've bitten more than I can chew: there seems to be no way to get a HD38xx to work at an OpenCL application. The private battle between AMDave and me has boosted AMDusers RAC to the 7th current on PrimeGrid, while we are the 20th team in total credits. Just a few more days for "The Knights Who Say Ni", and we will be 19th.
BANZAI !

Actually, vaughan is producing a bit more than usual the last couple of days, but we are in there too.

That got really close yesterday you almost got me again.
I lost a machine for 24 hours but I brought it back again with yet-another win7-64 patch
I might be able to make some headway on you now.
I still haven't tackled the *nix AMD64 HD5770 config yet. Looks like I won't get to that until the weekend.

[AMDave]

...
I'm hoping one of you that have 'been there, did that' can give me some pointers. I'm on the PG message boards now digging through the cruft to get to those tweaks.

Thanks in advance
Which OS are you looking at for that box?

/ed - I see it is Ubuntu 10.4 AMD64 per the benchmark outputs. You are a couple of steps ahead of me on that trail. I will watch with interest ;) - ed/

[Dirk Broer]

Hi Tamaster,

Though I am a noob at Linux, and can't let it run Boinc trouble-free longer than 3 weeks or so:icon_redface:, I've come across some tips:

"I'm running the Berkeley version -instead of the Debian/Ubuntu scripts and startup methods- on this one freshly redone machine because there is currently a driver permissions bug in the Debian/Ubuntu set up when trying to do GPU crunching on the ATI HD4xxx & HD5xxx series cards. This may even be specific to running x64 OS & drivers, but I don't have a 32bit setup to test that"

"You have to modify the boinc directory permissions to be able to write in it with your user":
sudo chown -R your_user:your_user /var/lib/boinc-client/"You have to remove and recreate the existing autostart links of boinc-client to postpone it otherwise it will be loaded too early and will not detect the gpu"
sudo update-rc.d -f boinc-client remove
sudo update-rc.d boinc-client defaults 99
and finally you have to modify /etc/default/boinc-client and change the boinc user for your user :
BOINC_USER="your_user"

In fact in ubuntu boinc is started with a custom "boinc" user that does not have sufficient rights to access the video card.
Even when the "boinc" user is added to the "video" group that doesn't change anything the gpu is not detected.
The user launching boinc must have a Xorg session running!

All credit to the DNETC forum:icon_mrgreen:

[Tamaster]

Thanks Dirk, but I've already got BOINC running (and recognizing the GPU as available) by installing the Berkeley version as my un-priviledged user. I've installed (and built) the Stream SDK w/OpenCL in like fashion and have fully tested the ability to execute OpenCL JIT build/execute of sample programs.

I think that the next step is downloading the new ppsieve executable and creating a customized app_info.xml file with the appropriate parameters. That is where I am right now.

I've seen a number of the files created for Winblows but I am still searching for a Linux one.

[Tamaster]

You are a couple of steps ahead of me on that trail. I will watch with interest ;) - ed/

If I meet success, I will post what turned the corner and any customized files that I might have created.

[Dirk Broer]

But other people have BOINC running too, and even can work on CAL/Brook++ applications. That does not mean they can automatically do work on the OpenCL applications.
I can let my Ati HD 3870 work at Milkyway, DNETC and Collatz, but not on Primegrid. I know I can't because a HD 3870 does not seem to have the needed compute shaders, but a HD 4850 like yours has those. According to http://www.primegrid.com/apps.php you do not have to download the application once you have completed the correct set-up, it will be downloaded automatically, but perhaps AMDave knows more?

[Tamaster]

I've got a large batch of the WUs running right now. I've verified with my Kill-A-Watt EZ that the GPU is very active. So are both cores of the CPU. I've had some errors while setting it up, but those are behind me. I could not seem to get the executable via automatic download. I'm running Ken Brazier's TPSieve v2.3c build for OpenCL. After this initial batch, I'll try the automatic download again. Here's the app_info.xml I am using:



<app_info>
<app>
<name>pps_sr2sieve</name>
<user_friendly_name>Proth Prime Search (Sieve)</user_friendly_name>
</app>
<file_info>
<name>primegrid_tpsieve_2.3c_x86_64-ATI-linux-gnu</name>
<executable/>
</file_info>
<app_version>
<app_name>pps_sr2sieve</app_name>
<version_num>130</version_num>
<plan_class>ati13ati</plan_class>
<avg_ncpus>0.05</avg_ncpus>
<max_ncpus>1</max_ncpus>
<flops>1.0e11</flops>
<coproc>
<type>ATI</type>
<count>1</count>
</coproc>
<cmdline></cmdline>
<file_ref>
<file_name>primegrid_tpsieve_2.3c_x86_64-ATI-linux-gnu</file_name>
<main_program/>
</file_ref>
</app_version>
</app_info>
Note: I have renamed the executable for purposes of this experiment.

[AMDave]

Well done, Tamaster!

Dirk, I could not get the M$ clients to download the app either.
I downloaded and inserted the right client for each box and then created the app_info.xml file like Tamaster has done.
I'll do the same for the linux box when I get around to it - if there is still work available in the queue by then.

/ed - Hey Dirk. We got silver badges! Vaughan is now project rank 22 - ed/

[vaughan]

Hi everyone, been busy in real life but have almost all my nvidia GPUs running Primegrid sieving. This has given me the recent boost in output.

I have also given into temptation and paid thru the nose with "Apple Tax" and bought a MacBook Pro 15 inch non-glossy screen with Intel (:icon_twisted:) Core i5 cpu. Its been running some Riesel Problem (sieve) tasks. I still haven't mastered the commands in Mac OS X but at least LogMeIn and BOINC are working.

My AMD/ATI GPUs are running either Milkyway or Collatz with DNetc as backup project.

Now that I reached my personal milestone in NPLB I've returned to BOINC. Most of my CPUs are running Czech project as it gives out 360 points per hour!!!

[Tamaster]

The box is crunchin' pps_sr2sieve WUs at the rate of about every 50 minutes (not sure whether that is good or slow). I've got both CPU cores crunchin' WCG as well. Temps are O.K. I'm thinking about modding and re-flashing the HD4850 BIOS (like I did on my HD2600XTs) to disable any power management 2D vs 3D clock throttling (perhaps even overclock it a touch). Right now, I'm just looking at stability and results. All on 230 watts input. :icon_mrgreen:

[Dirk Broer]

The pps_sr2sieve WUs run in 25:44 on my GTX260 (192 cuda cores). It is supposed to have a computing power of 715 Gflop single precision, as opposed to 1000 Gflop for the HD 4850.
Then again, Boinc reports a mere 477 Gflop peak for the GTX260, as does Cuda-Z. Compared against this Nvidia card, the Cuda-application seems more efficient than the OpenCL for the Ati card.

[Tamaster]

I suspect that there is more going on than that obvious answer, that the compute kernel is somewhat less than optimal (it was a port from Cuda-land, after all) and that I'm CPU bound with a dual-core that isn't capable of keeping the HD4850 fed. Not making excuses, but I think an intelligent version designed from scratch for OpenCL, while taking into consideration the specifics of the underlying Stream Cores and platform architecture might fair better performance-wise. For my machine, BOINC reports only 50 GFlops peak, probably also related to these same issues. I'm actually quite happy with the performance I'm getting, though I also believe that you probably lose a bit of efficiency when moving the application from CAL/Brook to OpenCL, which has the focus of being much more portable. It (OpenCL) is very much at early evolutionary stages right now. You makes your choices and picks your poison.

[Dirk Broer]

I think that the combination Linux with the OpenCL for the Ati card is the problem. While the top-200 Hosts for PrimeGrid features some Linux-boxen and some Ati cards (mostly 58xx series), the combination Linux+Ati is nowhere to be found in the top-200.
1st Linux box comes at #11 with an average of 164,569.07/day using four (!) NVIDIA Tesla C870s and an AMD Opteron 2220 SE. Some wealthy Korean....
1st Ati Card is at #14 with an average of 150,883.50 using two Radeon HD5800 series and an Intel i7980.
The first combination of Linux-box together with Ati GPU comes in at #253. There's a plain Linux-only box that is ranked higher (#201)
This leads to the suspicion that the present OpenCL application is optimized for Intel, and that AMD users under Linux could benefit from a fresh re-compilation to optimise their client for their CPU.

[Tamaster]

This leads to the suspicion that the present OpenCL application is optimized for Intel, and that AMD users under Linux could benefit from a fresh re-compilation to optimise their client for their CPU.

Well... since the source is available and I have the necessary tool chain installed and experience to do just that, I propose an experiment (oooh, I just love these experiments :icon_wink:). When I complete the current batch of WUs, I'll rebuild and grab another batch to run, then compare the results. I also have the Open64 compiler (which I compiled from scratch) installed as well, and if I can fit it in I'll rebuild on that also.

Has anyone else here done this?

[Dirk Broer]

The private battle between AMDave and me has boosted AMDusers RAC to the 7th current on PrimeGrid, while we are the 20th team in total credits. Just a few more days for "The Knights Who Say Ni", and we will be 19th.

Checked today: We are currently 5th at the RAC, having passed the 569-user strong Japanese Team 2ch and the PrimeSearchTeam; and have now risen to the 18th place, having passed both "The Knights Who Say Ni" and Duke University.

/ed - Hey AMDave. Our badges have turned gold, and Vaughan is now project rank 16! Liuqyn is at #141 and steadily climbing, having a RAC# of 52 -Vaughan is even the projects 2nd best at this moment in terms of RAC!- ed/

[Area 51]

Checked today: We are currently 5th at the RAC, having passed the 569-user strong Japanese Team 2ch and the PrimeSearchTeam; and have now risen to the 18th place, having passed both "The Knights Who Say Ni" and Duke University.

/ed - Hey AMDave. Our badges have turned gold, and Vaughan is now project rank 16! Liuqyn is at #141 and steadily climbing, having a RAC# of 52 -Vaughan is even the projects 2nd best at this moment in terms of RAC!- ed/

With only 28 active members, that's pretty impressive...!

[Dirk Broer]

Latest standings:

We have dropped to 7th place in average results, as the Japanese (Team 2ch, 577 users) and French (L'Alliance Francophone, 755 users) read stats as well, and react on them.
In Total Points we are now 17th, having passed Canada (the more-or-less-official Canadian National team, 165 users strong)
Our next objective is team Picard, and then we take on the UK BOINC Team, which will be a thougher nut to crack.

[vaughan]

Half my pharm was down yesterday as my son had to replace several case fans, we took the opportunity to clean out the dust bunnys and re-arrange some network cables. All except the Quadfather are back crunching. The Quadfather has some very squeaky CPU cooler fans so I have to find some suitable Socket F HSFs.

[AMDave]

Hey AMDave. Our badges have turned gold,

My PPS sieve badge just went pink.
I think the red one is at 5m.
Maybe I will find out.

[Area 51]

My PPS sieve badge just went pink.
I think the red one is at 5m.
Maybe I will find out.


I think Sphire is the next colour @ >4,000,000.

<edit>sorry, got my colurs mixed up, I believe it ruby @ >4,000,000</edit>

[bwhite]

Hey!! I got work for pps too. There is now an ATI windows app that takes care of itself!!!

All I did to get it to work was sign up a couple of weeks ago and select ati gpu as my preference. Until today just kept telling me no work...I should check the list of apps.

This morning BOINC manager downloaded the app Proth Prime Search (Sieve) 1.35 (ati13ati) and sent me a bunch of wu.

I'm running an ati hd 6850 with W7 ultimate 64 bit and all seems to be working OK.

[Dirk Broer]

Congratulations an thanks!
I just received a sh#tload of GPU work for my 3870 as well...on a XP32 machine, which crashed as soon as the scheduler got around to start them.
Even though the BOINC Manager sees a usable GPU -my HD3870- PrimeGrid may not be able to use it if it does not support OpenCL.
You might call this a bug of the PrimeGrid client, I think I should not hve received the WUs at all.

[AMDave]

Intrigued by this development I set the linux_AMD64 to recieve work.
It too got a client and a bunch of WUs.

...which started crashing as soon as the scheduler got around to them.
linux_AMD64 *MUST* have the Stream SDK 2.2 manually installed for the client to work.

so. finally pulling out the digit and getting around to it...
a couple of hours later and it is now working (even after a couple of trial reboots)
but definitely not in the configuration that I was expecting

The SDK would not properly compile with the Ubuntu version of the 10.9 driver.
So when AMD says that the Stream SDK 2.2 supports Ubuntu 10.4 it means that it supports the AMD version of the 10.9 driver on Ubuntu 10.4 and not the Ubuntu version of the 10.9 driver on Ubuntu 10.4

Oh. Nice side note here -
the SDK install instructions are VERY unclear about what you should be doing with the "OpenCL™ ICD registration configuration files for Linux"
Like first you need to download an additional zip file that you can find at the foot of the SDK homepage.
Then go ahead and be root and
then don't be copying it down into your root folder like they say, just run the tar command from the root folder and put the path to the file in the command.
Sheesh. Who copies files int a *nix root folder?
Now check your /etc/openCL/vendor/ folder now contains the 2 ati* files ... cool

There was no way that I could get the hot-fix version of 10.9 to work. even after fully removing every trace of the ubuntu ati drivers and config files etc.
Be aware that the supported version of the ATI drivers is 10.9 as per http://developer.amd.com/gpu/ATIStreamSDK/Pages/default.aspx
However they issued a hot fix in late September for the 10.9 driver after a security fix that went out on linux in general as per
Oh, one more thing their support for the drivers is quite wide ranging, however the support for the SDK is not. Read what it says on the SDK homepage and stick to it. If you didn't and you haven't and you still got it working, shake your own hand. I'm too far away to take you down to the pub and yak about it.

So, my working video driver is ATI 10.10 (AMD64)
check flgrxinfo works, glxgears works. Mmm pretty.

(I did try with 10.11 and it failed to install properly also - possibly a bad download. I will do a MD5 check on it later)

Now becoming root, relocating to where I had unpacked the SDK,
export the variables again as per the SDKJ Installation Notes,
then do make and it worked this time without errors.
This time dive down into a samples folder and try out some of the OpenCL goodies.
Oh Very nice.

boinc --> crash crash crash SUSPEND <deep sigh>

add the export variables to my users .bashrc file because they won't get remembered by themselves.

reboot

boinc. SWEEEET!!!!

Spent the next 10 minutes puttiung my desktop back the way I like it - single desktop on dual widescreens at 3840 x 1080 and some huge backgrounds :P

Done.
Don't try this at home unless you are a mental linux addict, have backups of everything on your system, have checked that your backups worked, have several hours to spare and a liberal supply of you favourite brain juice ;)

[AMDave]

1st wu in 35 minutes

/ed - I see a lot of WIn+ATI looming in my rear view. Got this card working just in time! - ed/

[Dirk Broer]

We have dropped to 7th place in average results, as the Japanese (Team 2ch, 577 users) and French (L'Alliance Francophone, 755 users) read stats as well, and react on them.
In Total Points we are now 17th, having passed Canada (the more-or-less-official Canadian National team, 165 users strong)
Our next objective is team Picard, and then we take on the UK BOINC Team, which will be a thougher nut to crack.


While we have dropped to the 9th place in average results, we have risen to the 15th place in total results. Team 'Picard' and BOINC@AUstralia have been passed. We are still in pursuit of the UK BOINC Team, but we might pass the Ukrainian National Team (team 'Ukraine') first. And let's not forget: from 18 till 21 December there's the Winter Solstice Challenge, just that sub-project where our strenght lies......

[Tamaster]

Congrats AMDave for getting your Linux/OpenCL/ATI GPU platform crunchin' (I know how fun that journey can be :icon_mrgreen: )

I'm still looking at the source and files to see how difficult it might be to rebuild the app on the chance there may be a way to squeeze more performance out. Since the build procedure is pretty non-standard, one of the mods I'm looking at is to use autoconf/automake to bring it more in line of the gnu standard build. That's just the first step in what I envision to be a long process of optimization for Stream architecture.

In the mean time, I've experimented with the app_info.xml file in order to coax multiple threads of execution out of the adapter:



<app_info>
<app>
<name>pps_sr2sieve</name>
<user_friendly_name>Proth Prime Search (Sieve)</user_friendly_name>
</app>
<file_info>
<name>primegrid_tpsieve_2.3c_x86_64-ATI-linux-gnu</name>
<executable/>
</file_info>
<app_version>
<app_name>pps_sr2sieve</app_name>
<version_num>130</version_num>
<plan_class>ati13ati</plan_class>
<avg_ncpus>0.55</avg_ncpus>
<max_ncpus>1</max_ncpus>
<flops>1.0e11</flops>
<coproc>
<type>ATI</type>
<count>0.5</count>
</coproc>
<cmdline></cmdline>
<file_ref>
<file_name>primegrid_tpsieve_2.3c_x86_64-ATI-linux-gnu</file_name>
<main_program/>
</file_ref>
</app_version>
</app_info>
NOTE: I've renamed the executable to clarify what's actually being run

I've changed the avg_ncpus to force thread affinity on my dual-core processor and the coproc count to allow for two (2) threads on the adapter. It seems to be working.

[AMDave]

Ooh. Nice multi-thread mod.
I should test | play with | hack some of those before my holiday shutdown. :D

Well done to everyone for the bump in the contest.
Not a bad effort!
Not bad at all. :)

[Tamaster]

On my HD4850, on my platform didn't show any significant improvement. I had thought that the dual-core might have been starving the adapter, but this experiment seems to prove otherwise. Then again, there may be driver level thread locks in play here which would keep this from actually doing what I intended.

Anyway... Made Top 10 today :icon_mrgreen:

[Tamaster]

I finally hit 10 Million on Ken's x86_64 OpenCL/ATI TPSieve v2.3e (that took some time). I think that I've now got a pretty good measure of the actual rate of Flops for this platform (6.0e10). Not what I had hoped for, but pretty good for an standard clocked HD 4850 on a 2.8ghz dual-core (@ 230 watts input) on Linux.

Yes friends, it can be done... :cool:
(and on a budget at that)