The Developer Day | Staying Curious

Dec/08

31

Benchmarks battle: Yii vs. Zend Framework

While browsing through planet-php.org like every morning I’ve found this blog post promoting Yii framework. Interested I took a look at Yii benchmarks page and got really surprised.

To be honest benchmarking frameworks on a hello world application is a bit unfair. A hello world application requires very little complexity from the the framework. Zend Framework is a very powerful framework that has many strengths. A test should do something realistic. For example connecting to a database, having a configuration file, properly rendering a view, having a model to retrieve data, doing something with it, rendering a layout, trying some plugins or other components like pagination. It requires more work to do such applications, but that is a good way to do a fair benchmark.

Not related to benchmarking but I couldn’t help but notice that Yii benchamarks die in controller actions. It is not best the best thing to do. It does not allow the framework to completely shutdown and do whatever it has to do. For example executing post dispatcher methods.

The benchmarks say that Yii 1.0 is 800% faster with APC than ZendFramework 1.7.2. That is trully amazing. A huge difference. And.. I don’t really believe it is true. So let’s run the same benchmark on our own metal ;)

Let’s start with server configurations.

Yii server:

Operating System: Red Hat Enterprise Linux Server release 5.2
Web Server: Apache httpd 2.0.40
PHP: 5.2.6, any non-essential extensions are disabled
CPU: Dual Intel Xeon 3.2GHz
Main Memory: 2GB
Hard Drive: 73GB 15K RPM SCSI/SAS HDD

Our server:

Operating System: CentOS 5.2
Web Server: Apache/2.2.3 Default configuration
PHP: PHP Version 5.2.6 Default configuration + APC
CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz
Main Memory: 1GB
Hard Drive: 250GB 7200 RPM SATA2

As you can see there is no big difference. Actually our server is slower :( Less CPU power, less RPMs on HDD, more not needed modules and extensions enabled, less memory.

I used the same APC settings as the Yii folks.

apc.enabled=1
apc.shm_segments=1
apc.optimization=0
apc.shm_size=32
apc.ttl=7200
apc.user_ttl=7200
apc.num_files_hint=1024
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.enable_cli=1
apc.cache_by_default=1

I’ve downloaded Yii 1.0 and zend framework 1.7.2. I’ve setup the same applications the Yii folks provide. Except I’ve removed the die and replaced it with an echo. And changed the ZF bootstrap file to work in no view renderer mode. The bootstrap file looks like this:

set_include_path(dirname(__FILE__));
require_once 'Zend/Loader.php';
Zend_Loader::registerAutoload();$front = Zend_Controller_Front::getInstance();
$front->addControllerDirectory(dirname(__FILE__)."/application/controllers", 'default');
$front->setParam('noViewRenderer', true);
$front->dispatch();

And then I ran the same AB test: “ab -t 30 -c 10 URL” as the Yii guys. I also want to note that I ran the tests a few times to warm up the APC cache. Let’s look at the results:

  • Zend Framework 1.7.2: 184 RPS (requests per second)
  • Yii 1.0: 275 RPS
  • Yii 1.0 with yiilite.php: 235 RPS

And now lets evalue these test results:

  1. Even on a slower machine ZendFramework is more than 3 times faster with APC than in the Yii benchmark. (strange)
  2. In comparison Yii is not 800% faster than ZF like shown on the Yii benchmark page.

It’s possible I’ve made a mistake somewhere. It would be nice to see more people testing this.

RSS Feed

12 Comments for Benchmarks battle: Yii vs. Zend Framework

Knut Urdalen | December 31, 2008 at 5:13 PM

Did you remember to restart Apache between your benchmark tests to get a correct comparison?

Check out the bash-script that was used to run these tests as well:
http://www.yiiframework.com/forum/index.php/topic,250.msg1125.html#msg1125

pcdinh | December 31, 2008 at 5:30 PM

I agree. Calling die() in a controller is not a correct way to test a simple “Hello world” printing task.

http:// /p/phpmark/issues/detail?id=1

Retrieving presentation content is part of a framework life cycle because a request is routed through a chain of responsibility. At a node of the chain, content is retrieved but framework still continues to run. die() just stops the execution prematurely. Therefore, some hidden costs are not revealed

Author comment by admin | December 31, 2008 at 5:48 PM

Hi,

I have just recently ran the benchmarks again with restarting apache after every benchmark and i didn’t notice any really visible difference.

Witek Radomski | January 4, 2009 at 1:13 AM

Hey,

Thanks a lot for this post. I had just looked at the Yii benchmarks page and nearly had a heart attack. Thanks for scrutinizing the benchmarks and setting things straight!

Logosfera | February 4, 2009 at 5:46 PM

The “hello world” benchmarks are pretty telling about the framework extra-weight added to the application. Of course there a framework may load everything upfront and make the benchmark pointless form a performance point-of-view but it still tells something about how the framework was built.

I have worked with Kohana framework and CodeIgniter for the same type of application and as the complexity of the application grows the differences in performance grows accordingly. Is not that a developer that creates a “fast” framework will screw up big time when it comes to db interogation or AR implementation. That’s why the feature set should be considered when analysing the “hello world” benchmarks.

It’s not the most informative benchmark but it’s not idiotic either.

Malmoth | February 24, 2009 at 4:32 PM

Thank you for clearing the issue.
I had my doubts about the benchmarks myself, even with a single Hello World application running, but I didn’t go further to test it for myself.
It’s just sad that such misleading benchmark results, if not downright misinforming, find their ways up to the web.

Christian | March 2, 2009 at 12:35 AM

Even if the Yii developers claim, that they have built the fastest framework, you shoudl have al look at http://adventure-php-framework.org/Page/103-Yii-vs-apf/~/sites_demosite_biz-action/setLanguage/lang/en.

This re-run of the Yii RPS benchmark analyzes some aspects mentioned above, too.

Darkredz | July 25, 2009 at 1:55 AM

Hi how about testing out DooPHP? It is built to be flexible for use with Zend Framework rich list of components. And both shares the same new BSD license. Please see if it is full of bu**sh*t.

Darkredz | July 25, 2009 at 1:56 AM

I also didn’t find that Yii is faster than CI or so many times faster than ZF(eventhough ZF is heavy)

deerawan | August 14, 2009 at 8:51 AM

wow…nice testing of yii and zend.

mercury | August 25, 2009 at 1:40 PM

Searching around regarding framework comparison. Can’t believe there is a big differences between the benchmark comparison you have mentioned here and Yii website.
Anyway, there is a new PHP Framework called DooPHP. They claim to be the fastest PHP Framework. Probably you can do some benchmark comparison again?

Sheldmandu | April 8, 2010 at 4:30 AM

I did some comprehensive performance benchmarks of all the latest PHP MVC Frameworks including Yii 1.1.1, Kohana 2.3.4 and Code Igniter 1.7.2 as well as others. The results and analysis of results is here: Comparing PHP MVC Framework Performance

Leave a comment!

<<

>>

Find it!

Theme Design by devolux.org