betalon.com - simply webdevelopment

About gearman job server

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel and to load balance processing, all the nice things that php isn't able to do out of the box.

Gearman is still in an early state, but it's stable enough that big player like Yahoo! and Digg use it in their productive environment.

How it works?

The gearman daemon is the middleman between the php script which wants to consume a job an the script / process that provides the job.

When you send a job to gearman, the server looks for free resources (worker) that can handle the job, an give it to the first available worker. When the job is done, the worker can send a result back to the client.

You can send as much jobs as you like to the gearman server which process the jobs parallel. For example you have a job that takes 10 seconds to complete, and you have to do it 10 times, with a normal PHP script your script will need 100 seconds to be completed, with gearman it only takes 10 seconds (assumed you have 10 workers for the job), because they can processed parallel. This can be helpful for many usecases, for example you have to consume one or more webservices to render the page and you have to call very much methods to get all the data you need. Every call to a webservice need some time (especially services with much overhead, SOAP, ...), when you can call all methods in parallel you get the required data much faster.

An other usage is to do jobs in the background, that means that you send the job to gearman and you don't have to wait until the job is finished. The disadvantage is that you aren't able to define a callback function, so the further processing of the data must be done by the worker/job.

This is practical for jobs that take much time, for example video converting, and so on.

What you need

• gearmand
• daemon (utility from libslack)
• Net_Gearman (PEAR package)

First install gearman, you can build it from source or install a package of your Linux distribution.

For Debian Lenny you can use apt-get install gearman-server

Then install the deamon utility, i think there is no package at the moment so you have to build it from source.

This is also very easy:

---------------------------------------------------------------------
tar xzf daemon-0.6.3.tar.gz
cd daemon-0.6.3
./config
make
make install
---------------------------------------------------------------------

Optionally you can install the PEAR package Net_Gearman with the command:
pear install Net_Gearman-0.2.3

If you don't have commandline access or you don't like to install it via PEAR the scripts are also available via the project page, and in my example (download at the end of the article) the files are also included.

Now let's start

For my example I think it's useful that you first download my example files, extract it and put it on your server.

Download Now

Now you should have the structure you see on the right side.

The file index.php contains the code that puts the job to the server, worker.php contains the code for the worker.

The file init-worker.sh script is the init script for the workers (worker.php), it starts as much worker threads you like in daemon mode. In this script you have to define some pathes to the scripts and executables, just open the file and fill in the correct pathes.

You can also run the worker with the command php worker.php then you can see the stdout of the script, thats useful for debugging.

In jobs/Test.php one example job is defined which is provided by worker.php.

Run the first test

First be sure that your gearman daemon is running (/etc/init.d/gearman-server start).

Then start the worker threads with ./init-worker.sh start
Be sure that the threads started successfully with ps aux

You shuld see something like this:

Now you can send the first job to your workers via gearman by calling the index.php via your browser.

http://your-host.com/your_directory/index.php

You should see something like the following:

I hope it works for you, if not check if all services run correctly.

Now enjoy the advantages of threaded php scripts :)


Download Now
Example Script