Unable to process application "Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)

[sergioclr]

Unable to process application "Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)".

Application starts OK then, about 10-20 seconds later, always ends with "Error while computing".

Please note that "Gamma-ray pulsar search #3 v1.11 (FGRPSSE) run OK" and "Binary Radio
Pulsar Search (Perseus Arm Survey) v1.39 (BRP5-opencl-ati)" also run OK.

Computer info:
GenuineIntel Pentium(R) Dual-Core CPU E6500 @ 2.93GHz [Family 6 Model 23 Stepping 10]
Number of processors 2
Coprocessors CAL AMD Radeon HD 6350/6450/7450/7470 series (Caicos) (1024MB) driver: 1.4.1741
Note: my GPU card is HD6450 (single precision FP)

Operating System Linux Mint 16 (Petra) 64-bit
3.2.0-61-generic
BOINC client version 7.2.42
Memory 3954.38 MB

Stderr output (from Einstein@Home task)

Using OpenCL platform provided by: Advanced Micro Devices, Inc.
Using OpenCL device "Caicos" by: Advanced Micro Devices, Inc.
Max allocation limit: 134217728
% Opening inputfile: ../../projects/einstein.phys.uwm.edu/LATeah0092C.dat
% Total amount of photon times: 10000
% Preparing toplist of length: 5
read_checkpoint(): Couldn't open file 'results.cand.out.cpt': No such file or directory (2)
% fft_size: 33554432 (0x2000000)
% Sky point 1/99
% Creating FFT plan.
Error allocating device memory: 268435456 bytes (error: -61)
21:32:59 (2510): [CRITICAL]: ERROR: MAIN() returned with error '1'

As you can see application tries to allocate 268435456 bytes but driver (?) limits the Max allocation to 134217728 bytes
hence the error "Error allocating device memory: 268435456 bytes (error: -61)"

Actions:
I tried several driver versions, besides the Ubuntu (Mint uses the same driver) driver available in the repository (fglrx-updates version 2:13.101-0ubuntu3),
including the 14.x drivers available at AMD site but none solved the memory allocation limitation.
I have opened a thread at Einstein@Home forum but the analysis has come to a halt due to lack of knowledge regarding the inner workings of a driver.

Additional info:
Dom 25 Mai 2014 15:08:22 BRT | | CAL: ATI GPU 0: AMD Radeon HD 6350/6450/7450/7470 series (Caicos) (CAL version 1.4.1741, 1024MB, 933MB available, 400 GFLOPS peak)
Dom 25 Mai 2014 15:08:22 BRT | | OpenCL: AMD/ATI GPU 0: AMD Radeon HD 6350/6450/7450/7470 series (Caicos) (driver version 1214.3, device version OpenCL 1.2 AMD-APP (1214.3), 1024MB, 933MB available, 400 GFLOPS peak)

>clinfo (truncated)
Number of platforms: 1
Platform Profile: FULL_PROFILE
Platform Version: OpenCL 1.2 AMD-APP (1214.3)
Platform Name: AMD Accelerated Parallel Processing
Platform Vendor: Advanced Micro Devices, Inc.
Platform Extensions: cl_khr_icd cl_amd_event_callback cl_amd_offline_devices
.
.
Platform Name: AMD Accelerated Parallel Processing
Number of devices: 2
Device Type: CL_DEVICE_TYPE_GPU
Device ID: 4098
Board name: AMD Radeon HD 6450
Device Topology: PCI[ B#1, D#0, F#0 ]

Max clock frequency: 625Mhz
Address bits: 32
Max memory allocation: 134217728
.
.
Global memory size: 536870912

From my point of view Global memory size: 536870912 which is 50% of total memory (1024 MB)
but Max memory allocation: 134217728 which is about 13% of total memory.

Question: why 'Global memory size' of 536870912 bytes? The GPU card is supposed to have 1024 MB.

Any suggestions will be greatly appreciated.
Thanks in advance.
Sergio.

[Dirk Broer]

I'd try to upgrade the card to at least a 2GB DDR3 HD 6670 if I were you. Caicos just doesn't cut it here I guess...

Do you use an app_config.xml for Einstein? If so, can you post it?
Something like
<app_config>
<app>
<name>hsgamma_FGRP3</name>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
<app>
<name>einsteinbinary_BRP4G</name>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
<app>
<name>einsteinbinary_BRP5</name>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
</app_config>

Have you set Einstein to crunch 1 or 2 WUs on the GPU at http://einstein.phys.uwm.edu/prefs.php?subset=project ?

[sergioclr]

'Caicos' was the chosen GPU card because of price ( I am on a tight budget), performance (400 GFlops), it is a fan-less card and local availability.
In fact the GPU (Caicos) is working OK, for about one year, and this application (Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)) is the first
one, for single precision FP, that I cannot run with it.

Please note that "Gamma-ray pulsar search #3 v1.11 (FGRPSSE) runs OK" and "Binary Radio
Pulsar Search (Perseus Arm Survey) v1.39 (BRP5-opencl-ati)" also runs OK.

BOINC is installed /etc/default/boinc-client and cc_config.xml, global_prefs_override.xml, gui_rpc_auth.cfg, and remote_hosts.cfg filles are all set to default.

Under Einstein@Home preferences the following applies:

Resource share
Determines the proportion of your computer's resources allocated to this project.
Example: if you participate in two BOINC projects with resource shares of 100 and 200,
the first will get 1/3 of your resources and the second will get 2/3. 100
Use CPU Enforced by version 6.10+ yes
Use ATI GPU Enforced by version 6.10+ yes
Use NVIDIA GPU Enforced by version 6.10+ no
Use INTEL GPUEnforced by version 7.0.27+ no
Is it OK for Einstein@Home and your team (if any) to email you? yes
Should Einstein@Home show your computers on its web site? no
Default computer location home

Graphics setting: frames per second (FPS)
Warning: affects CPU consumption! Default value: 20 20
Graphics setting: render quality
Warning: requires hardware 3D acceleration! Default value: low low
Graphics setting: window width (pixels)
Default value: 800 800
Graphics setting: window height (pixels)
Default value: 600 600

Run only the selected applications
Binary Radio Pulsar Search (Arecibo): yes
Binary Radio Pulsar Search (Arecibo, GPU): no
Binary Radio Pulsar Search (Perseus Arm Survey): yes
Gravitational Wave S6 Directed Search (CasA) : yes
Gamma-ray pulsar search #3: no Please notice that "Gamma-ray pulsar search #3: no" has been set to "no" to avoid unnecessary status of "Error while computing"
Run beta/test application versions?
This helps us develop applications, but it may cause jobs to fail on your computer. no

Run CPU versions of applications for which GPU versions are available yes

GPU utilization factor of BRP apps
DANGEROUS! Only touch this if you are absolutely sure of what you are doing!
Wrong setting might even damage your computer! Use solely on your own risk!
Min: -1.0 / Max: 1.0 / Default: 1.0, negative values will disable GPU tasks of this type 1

GPU utilization factor of FGRP apps
DANGEROUS! Only touch this if you are absolutely sure of what you are doing!
Wrong setting might even damage your computer! Use solely on your own risk!
Min: -1.0 / Max: 1.0 / Default: 1.0, negative values will disable GPU tasks of this type 1

GPU utilization factor of GW apps
DANGEROUS! Only touch this if you are absolutely sure of what you are doing!
Wrong setting might even damage your computer! Use solely on your own risk!
Min: -1.0 / Max: 1.0 / Default: 1.0, negative values will disable GPU tasks of this type 1

If I undestood the question properly the above parameters set Einstein@Home to crunch only one WU by the GPU.

Thanks in advance for helping me to diagnose if this is a solvable issue.
Sergio.

[Dirk Broer]

'Caicos' was the chosen GPU card because of price ( I am on a tight budget), performance (400 GFlops), it is a fan-less card and local availability.
I can understand the budget part, but for a few bucks more there is/was a fanless ASUS or Sapphire HD 6670, doing about 1000 GFlops.
More budgetwise the HD 6570 would also be an option. These might be able to crunch two WUs at a time, on a HD 6450 I wouldn't dare go past one as it has just 160 shaders and two SIMD units
(the HD 6570 and HD 6670 have 480 shaders and 6 SIMD units).

So your system is able to crunch FGRP WUs on the CPU (SSE) and BRP on the GPU (OpenCL), without using a cc_config.xml.
If you stop BOINC and make a new file called cc_config.xml containg the lines I mentioned in the project directory of Einstein,
using gedit -or another as basic as possible text editor- and then reboot, will it then start crunching more than previously?

[Dirk Broer]

Question: why 'Global memory size' of 536870912 bytes? The GPU card is supposed to have 1024 MB.

I guess that it may have to do with 'Number of devices: 2'. I think that the 1024 MB is divided over the two SIMD units that each get 512 MB alloted to them.
As maximum memory allocation size in OpenCL is by default only a quarter of available main memory (http://stackoverflow.com/questions/20408504/maximum-memory-allocation-size-in-opencl-only-a-quarter-of-available-main-memory), you hit that 'Max allocation limit: 134217728' barrier.

You may want to experiment with this command:

export GPU_MAX_HEAP_SIZE=100 (or any other integer value between 0 and 100, but I guess you'd be best of with at least 50 or 75)

in order to get things going for this Einstein subproject. Keep us informed!

Meanwhile a Lubuntu 13.10 system of me is running an Einstein Gamma-ray Pulsar search #3 v1.11 (FGRPopencl-ati) WU on the Radeon HD 7660D that is in its A10-5700. The same system also holds a HD 7750 and a HD 6670.

[Nflight]

Sergioclr (http://www.amdusers.com/forum/member.php/2526-sergioclr) Welcome to the Team Forum, thanks for stopping by!

As for your desired help I found the Admins on the Project Message board (http://einstein.phys.uwm.edu/forum_thread.php?id=10735) just as helpful as Dirk, Although Dirk does have serious experience in both Linux and Working with Einstein, one of our favorite projects here at AMDusers. I do hope we can find a solution. The link to the message boards is to the exact one where you should be able to solve the problem your having, Which I believe will be to change your share to 50% for the best results. I am not great at my system is so new I would rather drag our more technical staff into the mix to answer this be-develing problem your having. :blob3:

[sergioclr]

I have installed a brand new Linux (Mint 17 64-bit) distro into one of my ext4 partitions. It runs smoothly, albeit without being able to process "Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)" due to the Maximum Alloocation Limit. After applying your suggestion (app_config.xml in the /etc/lib/boinc-client/einstein.phys.uwm.edu/projects directory & rebooting) I noticed that "Binary Radio Pulsar Search (Perseus Arm Survey) v1.39 (BRP5-opencl-ati)" is running apparently faster than before, that is, before: 0.5 CPU+1 GPU and now 1 CPU+1 GPU. Still, "Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)" is not able to allocate the required 256 MB.

Comment (regarding app_config.xml): the downside is that previously I was able to run 3 WU's simultaneously (1 CPU, 1 CPU and 0.5 CPU+1 GPU). I understand that the actual scenario is more efficient when running WU BRP5-opencl-ati.

Action plan: I am still waiting (other ext4 partition) for some FRGPSSE WU's to complete before installing a Mint 17 32-bit. Reason: to double check whether 32-bit driver has the same behaviour as the 64-bit driver. Reference: boinc-amd-opencl (https://launchpad.net/ubuntu/trusty/+package/boinc-amd-opencl)



You may want to experiment with this command:

export GPU_MAX_HEAP_SIZE=100 (or any other integer value between 0 and 100, but I guess you'd be best of with at least 50 or 75

Transcription of part of one of my replies in Einstein@Home forum:

Summary:"The AMD APP SDK v2.3 currently defaults to exposing 50% of the physical GPU memory to OpenCL™ applications
For developers who wish to experiment with increasing the amount of physical memory that is accessible to their OpenCL™ applications,
the default 50% setting can be changed by setting the environment variable GPU_MAX_HEAP_SIZE to the percentage of total GPU memory that should be exposed.
For example, if you wanted to set the exposed GPU physical memory size to 75%, you need to the GPU_MAX_HEAP_SIZE environment variable to 75."

I am willing to experiment with GPU_MAX_HEAP_SIZE but I must confess that I don't know how to implement AMD APP SDK v2.3 in my Linux distro.

Thanks for helping me.
Sergio

[sergioclr]

Sergioclr Welcome to the Team Forum, thanks for stopping by!

As for your desired help I found the Admins on the Project Message board just as helpful as Dirk, Although Dirk does have serious experience in both Linux and Working with Einstein, one of our favorite projects here at AMDusers. I do hope we can find a solution. The link to the message boards is to the exact one where you should be able to solve the problem your having, Which I believe will be to change your share to 50% for the best results. I am not great at my system is so new I would rather drag our more technical staff into the mix to answer this be-develing problem your having.

I will experiment with 'Resource Share'.

Thank you Nflight.

[Dirk Broer]

I am willing to experiment with GPU_MAX_HEAP_SIZE but I must confess that I don't know how to implement AMD APP SDK v2.3 in my Linux distro.
It is actually quite simple:

Download it (but the most recent SDK is already v2.9 (http://developer.amd.com/tools-and-sdks/opencl-zone/opencl-tools-sdks/amd-accelerated-parallel-processing-app-sdk/) and Linux Mint is not named as supported OS. In fact they don't go further than Ubuntu 10.0!)
Extract it (using your archive manager)
install it (run the ./*.sh)


My Lubuntu 13.10 box presently runs the subproject in question (>4 hours underway), without having to install the SDK though.
I had some real problems getting my Ubuntu box back to crunch with OpenCL after upgrading it from 13.10 to 14.04 and had to remove files in order to be able to succesfully edit xorg.conf

[sergioclr]

I will make some experiments with Mint 17 32-bit and/or Xubuntu 14.04 32-bit even with the 'ancient' release support limitation.


I guess that it may have to do with 'Number of devices: 2'. I think that the 1024 MB is divided over the two SIMD units that each get 512 MB alloted to them.
As maximum memory allocation size in OpenCL is by default only a quarter of available main memory, you hit that 'Max allocation limit: 134217728' barrier.


Please take a look at Einstein@Home Message Board # 131101
User reports the same error but with "Turks" ( a HD6570, I presume) although he does not specify memory size.

Question: does the Stack Exchange page apply to both GPU AND CPU? If this is the case, I am confused with computer's clinfo display regarding OpenCL and CPU.
I can send the clinfo, via Dropbox link, if you think that further analysis is required.

Off topic note: be aware that in one of the bootable configurations (Xubuntu 14.04 64-bit) I have been receiving random Crash (non-fatal) reports related to Xorg. I have sent all the automatic crash reports but I don't have any answer yet. Xubuntu 12.04 LTS 64-bit is stable and does not report any Xorg related issues.

Thank you.

[Dirk Broer]

"Turks" comes in two main varieties: Turks Pro (HD 6570/HD7570) and Turks XT (HD 6670/HD7670).
I am hoping the HD 6670 on my Lubuntu Box will also get an Einstein WU to see what happens.

OpenCL can also be used by certain CPUs (Intel since SandyBridge, my Llano's and Trinity also support it ).
In a sytem with an AMD discrete graphics card and an OpenCL capable Intel CPU the CPU can report both the Intel 1.1 OpenCL and the AMD 1.2 OpenCL.

Off topic -but not entirely- Ubuntu 14.04 has severe OpenCL issues (both with AMD and nVidia proprietary drivers). I know how to circumvent the AMD problem -see our forum under 'Linux'- and am working on the nVidia.

[sergioclr]

Test results:

1) Mint 17 32-bit installed + updates;
2) boinc-amd-opencl (BOINC version: 7.2.42+dfsg-10, fglrx version is 2:13.350.1-0ubuntu2) installed;
3) AMD APP SDK v2.9 installed as per Installation Notes PDF document;
4) I tried to set SDK "Environment variables" for GPU_MAX_HEAP_SIZE=100 but no matter how much I read the instructions, I couldn't figure out how to do it.
Change or update the environment variables:
i. Open etc/profile; then check the AMDAPPSDKROOT and LD_LIBRARY_PATH settings.
ii. Check /etc/ld.so.conf.d/amdapp_x86.conf and /etc/ld.so.conf.d/amdapp_x86_64.conf .

Testing with CLI (command line interface/term) allows export GPU_MAX_HEAP_SIZE=100 and, according to clinfo, now Maximum Memory allocation is 256MB.

5) After opening BOINC manager and adding Einstein@Home, I added app_config.xml to BOINC Projects directory, stopped BOINC and rebooted the computer.
6) Since GPU_MAX_HEAP_SIZE=100 was executed at CLI, the information was lost after the reboot, so I entered again and checked clinfo.
7) BOINC was started and option "Allow new Tasks" was chosen.
8) To my surprise all the new FGRP3 WU's failed with the same reason as before! Checking clinfo showed Maximum Memory Allocation of 256MB but Stderr (Einstein@Home) site showed:
Using OpenCL platform provided by: Advanced Micro Devices, Inc.
Using OpenCL device "Caicos" by: Advanced Micro Devices, Inc.
Max allocation limit: 134217728

Conclusion: either I am failing to permanently set Environment Variable GPU_MAX_HEAP_SIZE=100 or this limitation (max memory allocation) cannot be modified/adjusted for Caicos 1024MB.
Addtional info on the topic (GPU_MAX_HEAP_SIZE) :
Setting up OpenCL for AMD/ATI devices (http://www.darktable.org/usermanual/ch08s02s05.html.php)
Maximum memory allocation problem (http://devgurus.amd.com/thread/158397)


and finally, to save some money and buy a new GPU (probably one of HD 7500/7600 Series).

Best regards.
Sergio.

[Dirk Broer]

After opening BOINC manager and adding Einstein@Home, I added app_config.xml to BOINC Projects directory, stopped BOINC and rebooted the computer.
That won't do. Stop BOINC altogether and then add or edit the app_config.xml, then reboot, do 'export GPU_MAX_HEAP_SIZE=100' and only then start BOINC.


Addtional info on the topic (GPU_MAX_HEAP_SIZE) :
Setting up OpenCL for AMD/ATI devices (http://www.darktable.org/usermanual/ch08s02s05.html.php)
Maximum memory allocation problem (http://devgurus.amd.com/thread/158397)
Read those and am pissed off myself too. I own three HD 6670 cards..


and finally, to save some money and buy a new GPU (probably one of HD 7500/7600 Series). Best regards, Sergio.

My advice: Buy a 2nd hand FM2 mobo with a A10-5700 or better (A10-5800K or A10-6700/6800K). I've been able to crunch two Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati) WUs with my A10-5700 (http://einstein.phys.uwm.edu/results.php?hostid=11235759&offset=0&show_names=1&state=3&appid=26).
The IGP on the A10s is better than the cards you have in mind. I have bought my A10 plus mobo for less than 100 euro (okay, I've since replaced the mobo with one that supports multiple GPUs).
Mind you: the RAM on AMD FM1 and FM2 platforms should preferably be 1866 MHz or higher (A10-5800/6800K and A10-Kaveri's).

ex-bitcoin miners nowadays offer interesting hardware for next to nought...

[sergioclr]

"Defeat is not an option"

Just to quench my curiosity, I moved the 'Caicos'/HD6450 GPU card to an Inspiron 530 (circa 2008) computer running Windows 7 32-bit.
Operating System Microsoft Windows 7
Home Premium x86 Edition, Service Pack 1, (06.01.7601.00)
BOINC client version 7.2.42
Memory 3326.18 MB
Cache 1024 KB
Swap space 6641.46 MB

After installing AMD driver (Catalyst package 14.4), BOINC was started with one FGRP3 WU and guess what? It runs OK (no error)!.
Note: AMD SDK 2.x and app_config.xml were not installed.

439831720 7182523 8 Jun 2014 18:18:08 UTC 22 Jun 2014 18:18:08 UTC In progress --- --- --- --- Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)

338

clinfo (truncated) shows:
Device Type: CL_DEVICE_TYPE_GPU
Device ID: 4098
Board name: AMD RADEON HD 6450
.
Max memory allocation: 536870912 50% of total memory!
.
Global memory size: 1073741824

It appears that the Maximum Allocation Memory limitation is restricted to Linux distros!

Sergio.

[Nflight]

Congratulations sergioclr - It feels good to succeed! Did Dirk help in anyway to your optimization and achievement goals. As he deserves much more then most people outwardly pour in this Team of Crunchers. Thanks for sharing your good spirits! :blob3:

[Dirk Broer]

I'm glad for you that you found a way to get your HD 6450 productive in the way you wanted to.

I'm most curious about the way the Linux drivers handle OpenCL.

Doing 'export GPU_MAX_HEAP_SIZE=100' doubled both the global memory and the Max memory allocation for all three GPUs in my Ubuntu 14.04 box (the HD 6550D, the 1GB HD 6670 and the 2GB HD 6670) as per clinfo.