Thread Pool/Queuing Software

Look to run a big batch of models at work - approx 30.

I want to run 8 models at once (or 4, depending on RAM usage and hyper threading as it's a Xeon E3-1240V2) but I want all the models to be queued up and "ready to go".
My initial thought was to create 8 (or 4) batch files with the models split equally as much as possible between them and the batch script will progress through them as they go. However, I realised that isn't the most effective method as all the long running models might be put in one batch file and this might take longer to run than the one with a shorter models.

What I'm therefore after is an app that or program that would accept a pool of 8(4) models, run them and when one is complete, move another from the "pool" or queue into position and process. From looks, this is a thread pool? However, I can't see a program to run them really - it's all code to build into apps, rather than scripts.This is a Windows machine.

I found Batch Runner but it's single core only, so not to much different to running a batch file for each core.
Remote access isn't really an option - this is to be designed and run now, for completion after christmas (so it'll be processing from now until then!).

Does anyone know of a program/hack/script that will let me achieve what I need? I'm sure years ago I had a program that would run a new Folding instance when one ended, thus allowing me to create a "queue" for F@H. Though that was probably single core.

from stackoverflow:
http://stackoverflow.com/questions/3...utively-in-vbs

Dave, thanks for the link.

However, that has the same issue as Batch Runner and running them in batch files - that they execute one after another. But on a single core.

Each model uses one core of the available 8. Using this or a batch file would therefore use only 1/8 of the CPU. I would need 8 different batch files to run all 8.

Apparently I can do what I'm after in Linux by using xargs or parallel. Example such as creating a shell script with each line of the command and then passing the command to it as

Code:

---

Models_Batch.sh|xargs -n 1 -P 4

---

seems to do the trick. However, this is a Windows server and xargs isn't on Windows.

Yeah. You need to fork.
You can write your own:
http://stackoverflow.com/questions/1...hild-processes

or try something like gearman -> Gearman.org

Well I solved it the easiest way I could... I moved to Linux. :icon_lol:

Installed a virtual machine (or got IT to install VirtualBox) and set the machine up to run using parallell in the end (rather than xargs) and save the results to a local shared folder. Seems to be working fairly well.

Went with Parallel rather than xargs, soley as with Parallel you can set it display the program it's working on (i.e. it shows the command) so I can see what models it's currently completed or working on. xargs just displayed the output of each program in a jumble (which I can check in each process/threads log file).

Seemed ironic asking about this considering the forum is built around the team using a computational task scheduler software... as that's pretty much what BOINC is!:icon_razz:

Maybe I'll investigate if the work can be farmed out to BOINC based machines at some time in the future :icon_lol:

Have a look at the "anonymous" client wrapper in the BOINC development forum.
It's fairly simple to insert your client app and compile.
Then you can set up your tasks and run it locally under BOINC manager - like the pre-alpha test runs for BOINC apps