Unable to process application "Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)

[sergioclr]

Unable to process application "Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)".

Application starts OK then, about 10-20 seconds later, always ends with "Error while computing".

Please note that "Gamma-ray pulsar search #3 v1.11 (FGRPSSE) run OK" and "Binary Radio
Pulsar Search (Perseus Arm Survey) v1.39 (BRP5-opencl-ati)" also run OK.

Computer info:
GenuineIntel Pentium(R) Dual-Core CPU E6500 @ 2.93GHz [Family 6 Model 23 Stepping 10]
Number of processors 2
Coprocessors CAL AMD Radeon HD 6350/6450/7450/7470 series (Caicos) (1024MB) driver: 1.4.1741
Note: my GPU card is HD6450 (single precision FP)

Operating System Linux Mint 16 (Petra) 64-bit
3.2.0-61-generic
BOINC client version 7.2.42
Memory 3954.38 MB

Stderr output (from Einstein@Home task)

Using OpenCL platform provided by: Advanced Micro Devices, Inc.
Using OpenCL device "Caicos" by: Advanced Micro Devices, Inc.
Max allocation limit: 134217728
% Opening inputfile: ../../projects/einstein.phys.uwm.edu/LATeah0092C.dat
% Total amount of photon times: 10000
% Preparing toplist of length: 5
read_checkpoint(): Couldn't open file 'results.cand.out.cpt': No such file or directory (2)
% fft_size: 33554432 (0x2000000)
% Sky point 1/99
% Creating FFT plan.
Error allocating device memory: 268435456 bytes (error: -61)
21:32:59 (2510): [CRITICAL]: ERROR: MAIN() returned with error '1'

As you can see application tries to allocate 268435456 bytes but driver (?) limits the Max allocation to 134217728 bytes
hence the error "Error allocating device memory: 268435456 bytes (error: -61)"

Actions:
I tried several driver versions, besides the Ubuntu (Mint uses the same driver) driver available in the repository (fglrx-updates version 2:13.101-0ubuntu3),
including the 14.x drivers available at AMD site but none solved the memory allocation limitation.
I have opened a thread at Einstein@Home forum but the analysis has come to a halt due to lack of knowledge regarding the inner workings of a driver.

Additional info:
Dom 25 Mai 2014 15:08:22 BRT | | CAL: ATI GPU 0: AMD Radeon HD 6350/6450/7450/7470 series (Caicos) (CAL version 1.4.1741, 1024MB, 933MB available, 400 GFLOPS peak)
Dom 25 Mai 2014 15:08:22 BRT | | OpenCL: AMD/ATI GPU 0: AMD Radeon HD 6350/6450/7450/7470 series (Caicos) (driver version 1214.3, device version OpenCL 1.2 AMD-APP (1214.3), 1024MB, 933MB available, 400 GFLOPS peak)

>clinfo (truncated)
Number of platforms: 1
Platform Profile: FULL_PROFILE
Platform Version: OpenCL 1.2 AMD-APP (1214.3)
Platform Name: AMD Accelerated Parallel Processing
Platform Vendor: Advanced Micro Devices, Inc.
Platform Extensions: cl_khr_icd cl_amd_event_callback cl_amd_offline_devices
.
.
Platform Name: AMD Accelerated Parallel Processing
Number of devices: 2
Device Type: CL_DEVICE_TYPE_GPU
Device ID: 4098
Board name: AMD Radeon HD 6450
Device Topology: PCI[ B#1, D#0, F#0 ]

Max clock frequency: 625Mhz
Address bits: 32
Max memory allocation: 134217728
.
.
Global memory size: 536870912

From my point of view Global memory size: 536870912 which is 50% of total memory (1024 MB)
but Max memory allocation: 134217728 which is about 13% of total memory.

Question: why 'Global memory size' of 536870912 bytes? The GPU card is supposed to have 1024 MB.

Any suggestions will be greatly appreciated.
Thanks in advance.
Sergio.

[Dirk Broer]

I'd try to upgrade the card to at least a 2GB DDR3 HD 6670 if I were you. Caicos just doesn't cut it here I guess...

Do you use an app_config.xml for Einstein? If so, can you post it?
Something like
<app_config>
<app>
<name>hsgamma_FGRP3</name>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
<app>
<name>einsteinbinary_BRP4G</name>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
<app>
<name>einsteinbinary_BRP5</name>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
</app_config>

Have you set Einstein to crunch 1 or 2 WUs on the GPU at http://einstein.phys.uwm.edu/prefs.php?subset=project ?

[sergioclr]

'Caicos' was the chosen GPU card because of price ( I am on a tight budget), performance (400 GFlops), it is a fan-less card and local availability.
In fact the GPU (Caicos) is working OK, for about one year, and this application (Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)) is the first
one, for single precision FP, that I cannot run with it.

Please note that "Gamma-ray pulsar search #3 v1.11 (FGRPSSE) runs OK" and "Binary Radio
Pulsar Search (Perseus Arm Survey) v1.39 (BRP5-opencl-ati)" also runs OK.

BOINC is installed /etc/default/boinc-client and cc_config.xml, global_prefs_override.xml, gui_rpc_auth.cfg, and remote_hosts.cfg filles are all set to default.

Under Einstein@Home preferences the following applies:

Resource share
Determines the proportion of your computer's resources allocated to this project.
Example: if you participate in two BOINC projects with resource shares of 100 and 200,
the first will get 1/3 of your resources and the second will get 2/3. 100
Use CPU Enforced by version 6.10+ yes
Use ATI GPU Enforced by version 6.10+ yes
Use NVIDIA GPU Enforced by version 6.10+ no
Use INTEL GPUEnforced by version 7.0.27+ no
Is it OK for Einstein@Home and your team (if any) to email you? yes
Should Einstein@Home show your computers on its web site? no
Default computer location home

Graphics setting: frames per second (FPS)
Warning: affects CPU consumption! Default value: 20 20
Graphics setting: render quality
Warning: requires hardware 3D acceleration! Default value: low low
Graphics setting: window width (pixels)
Default value: 800 800
Graphics setting: window height (pixels)
Default value: 600 600

Run only the selected applications
Binary Radio Pulsar Search (Arecibo): yes
Binary Radio Pulsar Search (Arecibo, GPU): no
Binary Radio Pulsar Search (Perseus Arm Survey): yes
Gravitational Wave S6 Directed Search (CasA) : yes
Gamma-ray pulsar search #3: no Please notice that "Gamma-ray pulsar search #3: no" has been set to "no" to avoid unnecessary status of "Error while computing"
Run beta/test application versions?
This helps us develop applications, but it may cause jobs to fail on your computer. no

Run CPU versions of applications for which GPU versions are available yes

GPU utilization factor of BRP apps
DANGEROUS! Only touch this if you are absolutely sure of what you are doing!
Wrong setting might even damage your computer! Use solely on your own risk!
Min: -1.0 / Max: 1.0 / Default: 1.0, negative values will disable GPU tasks of this type 1

GPU utilization factor of FGRP apps
DANGEROUS! Only touch this if you are absolutely sure of what you are doing!
Wrong setting might even damage your computer! Use solely on your own risk!
Min: -1.0 / Max: 1.0 / Default: 1.0, negative values will disable GPU tasks of this type 1

GPU utilization factor of GW apps
DANGEROUS! Only touch this if you are absolutely sure of what you are doing!
Wrong setting might even damage your computer! Use solely on your own risk!
Min: -1.0 / Max: 1.0 / Default: 1.0, negative values will disable GPU tasks of this type 1

If I undestood the question properly the above parameters set Einstein@Home to crunch only one WU by the GPU.

Thanks in advance for helping me to diagnose if this is a solvable issue.
Sergio.

[Dirk Broer]

'Caicos' was the chosen GPU card because of price ( I am on a tight budget), performance (400 GFlops), it is a fan-less card and local availability.

I can understand the budget part, but for a few bucks more there is/was a fanless ASUS or Sapphire HD 6670, doing about 1000 GFlops.
More budgetwise the HD 6570 would also be an option. These might be able to crunch two WUs at a time, on a HD 6450 I wouldn't dare go past one as it has just 160 shaders and two SIMD units
(the HD 6570 and HD 6670 have 480 shaders and 6 SIMD units).

So your system is able to crunch FGRP WUs on the CPU (SSE) and BRP on the GPU (OpenCL), without using a cc_config.xml.
If you stop BOINC and make a new file called cc_config.xml containg the lines I mentioned in the project directory of Einstein,
using gedit -or another as basic as possible text editor- and then reboot, will it then start crunching more than previously?

[Dirk Broer]

Question: why 'Global memory size' of 536870912 bytes? The GPU card is supposed to have 1024 MB.

I guess that it may have to do with 'Number of devices: 2'. I think that the 1024 MB is divided over the two SIMD units that each get 512 MB alloted to them.
As maximum memory allocation size in OpenCL is by default only a quarter of available main memory, you hit that 'Max allocation limit: 134217728' barrier.

You may want to experiment with this command:

export GPU_MAX_HEAP_SIZE=100 (or any other integer value between 0 and 100, but I guess you'd be best of with at least 50 or 75)

in order to get things going for this Einstein subproject. Keep us informed!

Meanwhile a Lubuntu 13.10 system of me is running an Einstein Gamma-ray Pulsar search #3 v1.11 (FGRPopencl-ati) WU on the Radeon HD 7660D that is in its A10-5700. The same system also holds a HD 7750 and a HD 6670.

[Nflight]

Sergioclr Welcome to the Team Forum, thanks for stopping by!

As for your desired help I found the Admins on the Project Message board just as helpful as Dirk, Although Dirk does have serious experience in both Linux and Working with Einstein, one of our favorite projects here at AMDusers. I do hope we can find a solution. The link to the message boards is to the exact one where you should be able to solve the problem your having, Which I believe will be to change your share to 50% for the best results. I am not great at my system is so new I would rather drag our more technical staff into the mix to answer this be-develing problem your having.

[sergioclr]

I have installed a brand new Linux (Mint 17 64-bit) distro into one of my ext4 partitions. It runs smoothly, albeit without being able to process "Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)" due to the Maximum Alloocation Limit. After applying your suggestion (app_config.xml in the /etc/lib/boinc-client/einstein.phys.uwm.edu/projects directory & rebooting) I noticed that "Binary Radio Pulsar Search (Perseus Arm Survey) v1.39 (BRP5-opencl-ati)" is running apparently faster than before, that is, before: 0.5 CPU+1 GPU and now 1 CPU+1 GPU. Still, "Gamma-ray pulsar search #3 v1.11 (FGRPopencl-ati)" is not able to allocate the required 256 MB.

Comment (regarding app_config.xml): the downside is that previously I was able to run 3 WU's simultaneously (1 CPU, 1 CPU and 0.5 CPU+1 GPU). I understand that the actual scenario is more efficient when running WU BRP5-opencl-ati.

Action plan: I am still waiting (other ext4 partition) for some FRGPSSE WU's to complete before installing a Mint 17 32-bit. Reason: to double check whether 32-bit driver has the same behaviour as the 64-bit driver. Reference: boinc-amd-opencl

You may want to experiment with this command:

export GPU_MAX_HEAP_SIZE=100 (or any other integer value between 0 and 100, but I guess you'd be best of with at least 50 or 75

Transcription of part of one of my replies in Einstein@Home forum:

Summary:"The AMD APP SDK v2.3 currently defaults to exposing 50% of the physical GPU memory to OpenCL™ applications
For developers who wish to experiment with increasing the amount of physical memory that is accessible to their OpenCL™ applications,
the default 50% setting can be changed by setting the environment variable GPU_MAX_HEAP_SIZE to the percentage of total GPU memory that should be exposed.
For example, if you wanted to set the exposed GPU physical memory size to 75%, you need to the GPU_MAX_HEAP_SIZE environment variable to 75."

I am willing to experiment with GPU_MAX_HEAP_SIZE but I must confess that I don't know how to implement AMD APP SDK v2.3 in my Linux distro.

Thanks for helping me.
Sergio

[sergioclr]

Sergioclr Welcome to the Team Forum, thanks for stopping by!

As for your desired help I found the Admins on the Project Message board just as helpful as Dirk, Although Dirk does have serious experience in both Linux and Working with Einstein, one of our favorite projects here at AMDusers. I do hope we can find a solution. The link to the message boards is to the exact one where you should be able to solve the problem your having, Which I believe will be to change your share to 50% for the best results. I am not great at my system is so new I would rather drag our more technical staff into the mix to answer this be-develing problem your having.

I will experiment with 'Resource Share'.

Thank you Nflight.

[Dirk Broer]

I am willing to experiment with GPU_MAX_HEAP_SIZE but I must confess that I don't know how to implement AMD APP SDK v2.3 in my Linux distro.

It is actually quite simple:

1. Download it (but the most recent SDK is already v2.9 and Linux Mint is not named as supported OS. In fact they don't go further than Ubuntu 10.0!)
2. Extract it (using your archive manager)
3. install it (run the ./*.sh)

My Lubuntu 13.10 box presently runs the subproject in question (>4 hours underway), without having to install the SDK though.
I had some real problems getting my Ubuntu box back to crunch with OpenCL after upgrading it from 13.10 to 14.04 and had to remove files in order to be able to succesfully edit xorg.conf

[sergioclr]

I will make some experiments with Mint 17 32-bit and/or Xubuntu 14.04 32-bit even with the 'ancient' release support limitation.

I guess that it may have to do with 'Number of devices: 2'. I think that the 1024 MB is divided over the two SIMD units that each get 512 MB alloted to them.
As maximum memory allocation size in OpenCL is by default only a quarter of available main memory, you hit that 'Max allocation limit: 134217728' barrier.

Please take a look at Einstein@Home Message Board # 131101
User reports the same error but with "Turks" ( a HD6570, I presume) although he does not specify memory size.

Question: does the Stack Exchange page apply to both GPU AND CPU? If this is the case, I am confused with computer's clinfo display regarding OpenCL and CPU.
I can send the clinfo, via Dropbox link, if you think that further analysis is required.

Off topic note: be aware that in one of the bootable configurations (Xubuntu 14.04 64-bit) I have been receiving random Crash (non-fatal) reports related to Xorg. I have sent all the automatic crash reports but I don't have any answer yet. Xubuntu 12.04 LTS 64-bit is stable and does not report any Xorg related issues.

Thank you.