A very interesting article by Ilya Grigorik on optimizing nginx time to first byte.
Quick summary:

• Use Nginx 1.5.7 and above due to a SSL buffering issue in 1.4
• Apply an MTU patch to nginx to reduce roundripts
• Enable TLS false start on nginx to reduce RTTs further

nginx, performance, ssl, tls Hide

Since Ubuntu 9.04 Jaunty Jackalope Ubuntu ships with EXT4 as the default file system. Surprisingly it makes MySQL writes extremely slow. This post is targeted to developers who work on Linux using MySQL and who would like to optimize MySQL performance.

Disk Performance Tuning

First start by tuning your disk performance. To do that you’ll have to sacrifice data consistency over data write speed. First start by enabling journal_data_writeback on your partition. This will allow to write to disk before updating the EXT4 journal. If your box crashes before updating the journal you might loose new data or some deleted data might reappear.

sudo tune2fs -o journal_data_writeback /dev/sda1 (use the right partition)

Next step is editing your /etc/fstab to change ext4 mounting options. My fstab file looks something like this:

UUID=irrelevant / ext4 errors=remount-ro,noatime,nodiratime,data=writeback,barrier=0,nobh,commit=100,nouser_xattr 0 1

There’s a few non default options added to improve write performance over consistency. Journal data writeback is enabled by data=writeback. The main option which is slowing down MySQL is barrier=0. You could actually change this single option and MySQL write performance would increase dramatically. Disabling this option makes your new data less safe when a system crash happens. Option nobh tries to avoid associating buffer heads and offers a minor performance improvement. Another option commit=100 says that all your updates are written to disk every 100 seconds. The default is 5 seconds. If your machine crashes you’re likely to loose 100 seconds of updates. Large commit values like 100 provide big performance improvements. And the last option nouser_xattr disables extended options on your filesystem and provides a minor performance boost.

Double check your /etc/fstab syntax and reboot.

Tuning MySQL configuration

MySQL configuration settings depend on what database engines you’re using. The most common ones are MyISAM and InnoDB. I will assume that you use both.

Warning! Some of the configuration changes will or might make your database inaccessible. Therefore backup all your databases by dumping them to SQL to a safe location. Make sure to include triggers and stored procedures. Double check that you will be able to reimport your backups and only then proceed further. Some options will make your InnoDB database stop working. I’ll mark those. Also backup your MySQL configuration. Just in case.

MySQL settings depend on how much memory you have. I will assume a normal working station will have 4GB of RAM. Open your MySQL configuration file which on Ubuntu is located at /etc/mysql/my.cnf and set the following options.

transaction-isolation = READ-COMMITTED

As a developer you will probably not have transactions running in parallel. If you don’t care about transactions and still use InnoDB set the isolation level to READ-COMMITED. This will make your transactions only see committed data but won’t prevent phantom rows. Setting it to READ-COMMITED will also improve performance.

key_buffer = 512M

By far the most important option for MyISAM. MyISAM indexes are cached using in the key buffer. It’s usually a good bet to set it from 25% to 40% of memory available. As a developer you might not need that much but do not leave it at a default.

query_cache_size = 256M

Caches query results. Especially useful if your applications don’t have caching.

innodb_buffer_pool_size = 1024M (requires a backup and an import)

InnoDB buffer pool size is the most important option for InnoDB. If your whole database is InnoDB you can try and fit your whole database in memory. If you don’t have that much memory you can generally set 70% - 80% of memory available. On a development box you will probably want to have extra RAM for things like Gnome or your IDE.

innodb_additional_mem_pool_size = 32M
innodb_log_buffer_size = 4M
innodb_log_file_size = 128M

innodb_flush_log_at_trx_commit = 2

This option tells InnoDB to only flush log data every two seconds. On development machines you can set this even higher because the only risk is losing transactions during a system crash. If your development machine crashes you probably won’t care about lost transactions. Experiment!

innodb_flush_method = O_DIRECT

This options tells InnoDB to skip filesystem cache and write straight to disk since InnoDB already has it’s own cache - the buffer pool. You save yourself some RAM.

table_cache = 1024

Caches open tables. Might not be very useful on a single dev box but useful in general on any database server.

myisam_use_mmap = 1

Mmap is a new MyISAM feature available with MySQL 5.1. Should improve MyISAM write/read performance ~6%.

To sum up all the settings on a 4GB work environment:

transaction-isolation = READ-COMMITTED
key_buffer = 512M
query_cache_size = 256M
innodb_buffer_pool_size = 1024M
innodb_additional_mem_pool_size = 32M
innodb_log_buffer_size = 4M
innodb_log_file_size = 128M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
table_cache = 1024
myisam_use_mmap = 1

Buy an SSD disk

This is by far the best upgrade you can do. SSD does not have any moving mechanical parts therefore doing a random read or write is as fast as doing a sequential read or write. My work laptop Lenovo T400 can push 3.5 MB with random writes, 35 MB with sequential writes, 2.6MB with random reads and 38MB with sequential reads per second. The same test with an SSD disk can push 220MB random writes and 330MB random reads with similar numbers for sequential reads and writes. So for IO access you can expect 10 - 100 times performance difference.

Summary

It’s easy to squeeze some extra performance out of your development environment by sacrificing data safety. In my case these changes made our database integration test suites run a lot quicker. So far I haven’t experienced any downsides from the above settings though you have to accept that one day it most likely will. Most of the database settings I’ve mentioned are those considered most when tuning production database servers. My final advice is take everything you read here with a pinch of salt as I am by far not an expert in these matters and everything listed here is gathered from various resources online.

Resources

InnoDB performance optimization basics
Tunning MySQL server after installation
MyISAM MMAP feature
MySQL transaction isolation levels
Why you should ignore key cache hit ratio
Tweaks to boost EXT4 performance
|SSD Benchmarks

innodb, Linux, MyISAM, MySQL, performance, Tuning, Ubuntu Hide

As I mentioned earlier guys from Packt publishing asked me to review a recently published book Zend Framework 1.8 Web Application Development. The title says it all - it’s a book about designing and developing PHP web applications using Zend framework.

This book doesn’t require the reader to be familiar with zend framework and explains all concepts in proper detail, though it will be easier to read the book if the reader is familiar with the framework and/or has experience with MVC and OOP in general. This book should be interesting to all developers who design and develop day to day web applications using MVC frameworks or not yet familiar with them as it may improve their insights towards web applications modelling, testing , optimizations and more.

Even though I am fairly familiar with the framework I found the book to be an interesting, easy read, plentiful of examples explaining the intricacies of the framework.

The first though a very important chapter teaches the concept of bootstrapping using Zend_Application and shows how to write and run a simple hello world program using controllers and views. As well it introduces the use of controller utility methods such as _getParam(), _forward(), _redirect(), action helpers, view helpers which are very valuable and a lot of developers miss them entirely. This chapter also shows the proper use of the response object which also tends to get forgotten.

The second chapter dives straight into the Front Controller pattern explaining how the framework routes, dispatches requests and responds to the client. I have never been bothered to understand the whole thing and was quite surprised to see how simple it all is. It is worth mentioning that this chapter explains in great detail how the router and various routes work and how elegantly it integrates with Zend_Config. Last the chapter covers the request object and it’s external API which provides lot’s of valuable functionality.

From the third chapter author Keith Pope starts building the main application of the entire book, the Storefront. It’s a relatively simple “real life” application that serves the purpose of being an online products catalog. This chapter shows how such an application is structured on a file system and bootstrapped and configured.  Even more the chapter covers the creation of Zend_Log and various logging writers and database profiling. Extremely valuable features that not many developers know of. If every zend framework application would start as the chapter describes I believe a lot more developers would be eager to start their IDE’s 9:00 AM straight.

Next is my personal favorite chapter - Models. Zend Framework does not have a base Model class and there’s a pretty good reason why it doesn’t. Models are specialized for a certain business task and as such it is arguably impossible to make a generic model implementation that would fit all sizes well. It then might raise a question what’s there to read about? It is my personal belief that web applications modelling has lost focus during the years by the ever growing development community. Developers got their minds focused on the next “new” thing. Let it be template engines, ORM’s, rise of active record and ruby on rails, ajax and javascript frameworks. While the majority of the models “out there” are deeply crippled. Hundreds of books were written explaining how to manage the complexity of the problem so a single chapter is a mighty challenge. I believe the author made a great choice by explaining the concept of the fat model, skinny controller, explaining the benefits composition over inheritance, data access layer separation from the business layer and my deepest respect for introducing domain model design, Martin Fowler’s book Patterns of Enterprise Application and Eric Evan’s book Domain Driven Design also known as DDD.

Next five chapters describe the implementation of the Storefront application. Each chapter highlights a major component of the framework. Use of resource autoloaders, plugins, Zend_Form, Zend_Auth, Zend_Acl. Before saying anything else it is very important to say that books rarely ever show hardcore “real life” applications as examples. Even Fowler himself likes to skip certain topics like validation in certain sections of his famous book just because they complicate things too much. This book is no exception to the rule. It’s a pretty straightforward application. There isn’t a single join to another table or a GROUP BY statement in SQL, forms implemented with Zend_Form are rather simple with little if any javascript / ajax. I found it a bit disturbing that class create other classes having both business and factory logic. One of the ugly examples was where a getPrice() method on a product model creates it’s own Taxation service which could not be mocked if that class would need to be unit tested. In most of the cases author provides injection methods for unit testing but I would argue that it does not show class dependencies explicitly which is very valuable for unit testing. Besides that I really enjoyed how the author decided to go with ACL in the domain layer. This would more often than not be implemented in the controllers making the model tightly dependent on the controllers. All in all keeping in mind that it’s a tutorial application introducing the framework I’m highly satisfied of it’s overall quality. Repeating myself. If every zend framework application would be so well written..

Another chapter worth mentioning is regarding optimizations. I was surprised to learn about such things like plugin loader cache, table gateway metadata or various Zend_Cache frontends which I have never bothered to look up. Not to mention widely known tricks using APC, stripping zend framework of all requires and setting up an optimized include path.

And last but not least again one of my favorite topics - testing. I strongly agree with Misko Hevery that test driven development is a skill. It’s definitely not easy to start or learn. One would fool himself to think otherwise.  This topic deserves many books of it’s own. I can only share from my own experience - once I started unit testing applications that I work with, I have never looked back. This chapter explains different types of testing, shows how to setup PHPUnit and provides examples of controller testing using Zend_Test. I believe this chapter deserves more attention on how to do testing with a database in mind, debugging failing controllers, avoiding complicated mocks, implementing continuous integration. But again it is worth to keep in mind that the book is about Zend Framework and not testing in general.

All in all I enjoyed reading this book. I would and will recommend it to my colleagues and friends. I hope that this hopefully not too boring review convinced you to buy the book and learn something new. Once again - big thanks to Packt Publishing for a free book. Happy reading. Over and out.

book, framework, PHP, review, zend Hide

Another great article I’ve found about MySQL character sets and collations that demystifies all the so often found problems in projects made by confused developers. This also can help you to save some space your database takes and improve your queries performance because more of your database can fit into memory.

character-sets, collations, MySQL, sorting Hide

I was really happy to see that most of the Drupalcon 2008 talks videos were published. There are a few general talks that I would like to highlight.

One talk that really impressed me was “Rasmus Lerdorf keynote - Simple is Hard“. I found it to bee very interesting, exciting, funny and shocking. It opened my eyes to some extent not to overdo with abstractions and mega layers and take a look at simple things like transactions per second. The inclued pecl extension just blew my mind. Sadly there isn’t a DLL compiled for windows. But I’m definitely trying out this extension on our applications to see what kind of a mess we have.

There were also a few by Rasmus about PHP security at the end of the presentation that were really interesting. Few other talks worth mentioning: “Indexes and denormalization: keys to scaling sites with massive content” a nice introduction to indexes and how they work. And most importantly how you can’t have indexes on two tables using open source relational databases and how denormalization helps.

“High availability solutions for MySQL: An Overview and practical demo” this one speaks for itself. If you are interested in replication, clusters and that alike this talk is for you.

drupalcon, lerdorf, PHP, rasmus, talks, videos Hide

Here’s an interesting question from the Zend exam practice test:

“Absent any actual need for choosing one method over the other, does passing arrays by value to a read-only function reduce performance compared to passing them by reference?’

I wondered well PHP would have to make a copy of the variable passed so that would be slower than passing it by reference. I was wrong. Turns out PHP uses a lazy-copy mechanism (also called copy-on-write) that does not actually create a copy of a variable until it is modified. And since PHP must create a set of structures that it uses to maintain the reference it is actually “slower” to pass a variable to a read-only function by reference.

Having programmed for quite a few years I could have thought of that myself. Strangely this was not in the study guide.

copy-on-write, lazy-copy, Optimization, PHP Hide

A while ago I was in need to profile one of our PHP applications. I needed to be able to see execution times, memory usage and some kind of a graphical interface to view it all. I remembered a PHP profiler debugger tool named Xdebug. Xdebug works as a PHP Zend extension. If you are a windows user installing it on your machine is very straightforward:

Download the appropriate windows module from xdebug.org. Edit your php.ini file by adding the following line to the extensions section:

zend_extension_ts=”path/xdebug.dll”

Restart your web server and look for a module named Xdebug in your phpinfo() output. If you’re lucky enough you’ll find it there.

After that you need to configure xdebug extension.

;enables the profiler
xdebug.profiler_enable=1;
;specifies profiler output files directory
xdebug.profiler_output_dir=”c:/php/xdebug/”
;specifies profiler output file name pattern.
xdebug.profiler_output_name=”cachegrind.%u.out”
;enables auto tracing on every script startup
xdebug.auto_trace=1
;specifies trace files output direcctory
xdebug.trace_output_dir=”c:/php/xdebug/”
;specifies trace output file name pattern.
xdebug.trace_output_name=”trace.%u”
;enable memory usage tracking
xdebug.show_mem_delta=1

Restart your web server again. Run any of your php applications and you should find files in your specified directories:

trace.1187359095_993350.xt
cachegrind.1187366421_815700.out

Now you can view cachegrind files with WinCacheGrind. The application I tested was spawning few processes of it’s own and the profile information was in one file by default. But apparently something didn’t want to work so I separated each process to a separate file by adding %u to the output file name pattern.

The most interesting part for me was memory usage which you can find in the .xt files. It shows the total memory used after every line of code executed and a delta’s between executions. A tool for viewing/analyzing these trace files would be nice. Did not look for one though. All in all Xdebug is useful to find those nasty bottlenecks. It also has other nice features that help you to daily develop your apps. Nice thing I didn’t try yet is the “Code coverage” test. It might help me find some unused code in our applications.

PHP, profiling, win-cache-grind, xdebug Hide