To be honest benchmarking frameworks on a hello world application is a bit unfair. A hello world application requires very little complexity from the the framework. Zend Framework is a very powerful framework that has many strengths. A test should do something realistic. For example connecting to a database, having a configuration file, properly rendering a view, having a model to retrieve data, doing something with it, rendering a layout, trying some plugins or other components like pagination. It requires more work to do such applications, but that is a good way to do a fair benchmark.

Not related to benchmarking but I couldn’t help but notice that Yii benchamarks die in controller actions. It is not best the best thing to do. It does not allow the framework to completely shutdown and do whatever it has to do. For example executing post dispatcher methods.

The benchmarks say that Yii 1.0 is 800% faster with APC than ZendFramework 1.7.2. That is trully amazing. A huge difference. And.. I don’t really believe it is true. So let’s run the same benchmark on our own metal

Let’s start with server configurations.

Yii server:

Operating System: Red Hat Enterprise Linux Server release 5.2
Web Server: Apache httpd 2.0.40
PHP: 5.2.6, any non-essential extensions are disabled
CPU: Dual Intel Xeon 3.2GHz
Main Memory: 2GB
Hard Drive: 73GB 15K RPM SCSI/SAS HDD

Our server:

Operating System: CentOS 5.2
Web Server: Apache/2.2.3 Default configuration
PHP: PHP Version 5.2.6 Default configuration + APC
CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz
Main Memory: 1GB
Hard Drive: 250GB 7200 RPM SATA2

As you can see there is no big difference. Actually our server is slower Less CPU power, less RPMs on HDD, more not needed modules and extensions enabled, less memory.

I used the same APC settings as the Yii folks.

apc.enabled=1
apc.shm_segments=1
apc.optimization=0
apc.shm_size=32
apc.ttl=7200
apc.user_ttl=7200
apc.num_files_hint=1024
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.enable_cli=1
apc.cache_by_default=1

I’ve downloaded Yii 1.0 and zend framework 1.7.2. I’ve setup the same applications the Yii folks provide. Except I’ve removed the die and replaced it with an echo. And changed the ZF bootstrap file to work in no view renderer mode. The bootstrap file looks like this:

set_include_path(dirname(__FILE__));
require_once 'Zend/Loader.php';
Zend_Loader::registerAutoload();$front = Zend_Controller_Front::getInstance();
$front->addControllerDirectory(dirname(__FILE__)."/application/controllers", 'default');
$front->setParam('noViewRenderer', true);
$front->dispatch();

And then I ran the same AB test: “ab -t 30 -c 10 URL” as the Yii guys. I also want to note that I ran the tests a few times to warm up the APC cache. Let’s look at the results:

• Zend Framework 1.7.2: 184 RPS (requests per second)
• Yii 1.0: 275 RPS
• Yii 1.0 with yiilite.php: 235 RPS

And now lets evalue these test results:

1. Even on a slower machine ZendFramework is more than 3 times faster with APC than in the Yii benchmark. (strange)
2. In comparison Yii is not 800% faster than ZF like shown on the Yii benchmark page.

It’s possible I’ve made a mistake somewhere. It would be nice to see more people testing this.