Posts Tagged ‘performance’

New Magento Minifier version

June 2nd, 2014

A new version of the Mangadreen Minifier is out, 1.0.4. Since the previous post there have been a few tweaks, fixes and performance improvements. The code has also been migrated to GitHub.

Here’s a direct link to the latest version (zipped):

The plugin is currently in production on several Magento CE 1.5, 1.7, 1.8 and 1.9, so it should work on all Magento CE versions after 1.5.

Magento ,

Magento memory leaks and custom options

August 30th, 2012

Most software, if not all, has memory leaks. Some are really bad at collecting garbage, other manage it better, but not enough. Magento had its problems since the beginning, and a lot of them have been fixed in the latest releases (using 1.6 to test this). However, it still has a long way until all major memory problems are gone. A simple test, like the one below, will reveal what I mean (put it in your magento root and run it from the CLI):

require "app/Mage.php";

for($k = 0; $k < 100; $k++) {
    echo $k . "\t" . (memory_get_usage()/1024/1024) . "\n";

This will show the current iteration and the memory usage. Replace “123” with a real product id – first, use a simple product, with no custom options. My result looks like this:

0 - 8.3790
99 - 8.3794

Looks good, very good actually.

Now try it with a product that has custom options. Not more than 5-6 options, but stuff a dozen values in one or two options. My test shows these numbers:

0 - 8.9398
99 - 17.5057


Along the way, you’ll see that the garbage collector kicks in and memory is freed, so max memory usage (for me) doesn’t exceed 20M. Cool, everything seems acceptable, and it seems that the old memory leaks have been fixed (read more over here, here and here ).

Now let’s try something different. Importing a few hundred products from a CSV. All products have custom options, usually the same, but with different values. The code could be summarized like this:

$options = $this->_getOptions($row);
$product = $this->initNewProduct(); //just inits the product with all the required data
        ->setAttributeSetId( $attributeSetId )
        ->setCategoryIds( array($someIds = 123) );
if( $options && count($options) ) {


To me, this looked fine and should have worked. I didn’t care it ran out of memory after 60-70 cycles, I could live with that. However, after approximately 20 hours of intense debugging, trying to figure out why the options aren’t showing up correctly on the frontend I decided to check the products in the admin. I had a huge surprise when I’ve noticed that for the second product, I had the first product’s options and the (correct) second product’s options. For the third, I had the first, second and third option sets. And so on, the last one was a complete mess. At some point Firefox couldn’t even load the interface anymore, there were simply too many options. Now that’s a memory leak for sure!

So, is there a fix? Seems like there was one, tested it for 3 products and it worked fine, no more leftover options. Initially, I was doing this:


but it didn’t work, although gc_collect_cycles() helped keeping the memory growth under control.

So I added the following lines:

unset($product, $options); //instead of just unset($product);

and voila – all custom options have been created correctly. Haven’t really analyzed why this did the trick and why it didn’t work without it. Either way, I’m happy the code works and I hope this will help someone else someday.

Magento , ,

Updated version of the Magento CSS & JS Minifier

May 31st, 2012

I’ve released the first version of the magento css&js minifier more than a year and half ago. Since then, it has been downloaded over 200 times – roughly 10 downloads per month. It’s a decent number, considering its utility and the fact that it hasn’t been promoted or included on Magento Connect. The first version was compatible with all Magento versions greater than 1.4. Starting with Magento 1.7, the extension failed to work on the admin pages, because of the -webkit-keyframes css declarations.

What’s new in version 1.0.1

  • Changed namespace from Oxygen to Mandagreen
  • Upgraded JsMin to version 1.1.2
  • Upgraded CssMin to version 3.0.1
  • Added backend options for converting HSL values, converting font-weight values, converting named colors and replacing variables


Download & Install

Minifier for Magento (1418 downloads) , then unzip it and copy the app/ folder to your Magento root folder.

Logout from the admin if you’re already logged in, then log back in. Go to Cache Management, click on the “Flush cache storage” button, then go to Configuration > Developer and Enable javascript minifier and css minifier under CSS & JS Minifier. Also, make sure that “Merge JavaScript Files” and “Merge CSS Files” are enabled, under JavaScript Settings, and CSS Settings respectively. Go back to Cache Management and click the “Flush Javascript/CSS cache” button.


Upgrading from 0.1.0

If you’re already using the first version of the extension, you’ll need to remove the old files. Go to your magento root folder, then navigate to the etc/modules folder. Remove the file called Oxygen_Minifier.xml. Then, navigate to app/code/local and remove the Oxygen folder (or Oxygen/Minifier, if you have other modules with this namespace).

Next, issue the following mysql command, using the mysql console or PhpMyAdmin:

update core_config_data set path = replace(path, 'oxy_', 'mg') where path like '%oxy_minifier%';

You can now install the module as explained above.



This extension has been tested with all Magento CE versions 1.4 through 1.7.


Supporting the Module

If you enjoy using the Magento CSS & JS Minifier, help us improve it and make it even better. Since this project it entirely open source and free for everyone, we welcome any donations and consider them a sign of appreciation. Donate

We also appreciate your feedback, both positive and negative, as long as they are constructive.

If any of you know how to package an extension for Magento Connect, I would appreciate the help. I’ve given up after 2 hours or repeated failures.

Magento , , ,

When JOINs fail to work

May 6th, 2012

Any developer, junior or senior, should know and understand the basic rules of database normalization since they are the foundation of relational DBMS. Edgar Codd introduced the concept in 1970 and since then, most of the architectures have been following his rules. However, for high-traffic websites, sometime de-normalized is better than normalized, resulting in faster queries, less swap space and less storage space.

Although this post is not exactly about normalization, another common recommendation is to use a join rather than multiple single queries. In other words, rather than doing “select * from a where condition1” then “select * from b where condition2”, if a and b are in a relationship, it’s better to usually run a query like “select * from a inner join b on = b.aid where conditions”. A classic example is when using categories for a certain object, say movies. Let’s say each movie can have a category and you want to display a list of all movies and the category that movie is in. Some developers might write the following php code:

$rs = mysql_query('select * from movies');
while( $row = mysql_fetch_array($rs) ) {
  $rs2 = mysql_query('select * from categories where category_id = ' . $row['category_id'];
  $row2 = mysql_fetch_array($rs2);
  $category = $row2['category_name'];
  echo $row['name'] . "<br />Category: " . $category . "<br /><br />";

but there’s a simpler and (most of the time) better approach:

$rs = mysql_query('select * from movies m inner join categories c on c.category_id = m.category_id');
while( $row = mysql_fetch_array($rs) ) {
  echo $row['name'] . "<br />Category: " . $row['category_name'] . "<br /><br />";

The first approach generates 1 + N queries, where N is the number of movies. If N is a big number, things might take a while… The second approach only takes 1 query, leaving the hard part to the internal RDBMS engine. Well, sometimes, in certain scenarios, the first version is the right choice. It’s THE solution.

Let’s assume there are 4+1 tables – the main one, call it objects, one for tags, one for categories, another one called hits which will track the number of times a certain object has been seen by a user, and the last one for storing the n-to-n relationship between tags and objects, call it >code>object_tags. The ERD is rather simple:

Also, let’s assume we want to show a list of objects, with all their tags, their current category and the number of hits it has. The results will be paginated, only showing 20 items per page. Simple, right? Anyone would tend to write something like this for the first page:

SELECT * FROM objects o 
INNER JOIN categories c ON c.category_id = o.category_id
INNER JOIN object_tags ot ON ot.object_id = o.object_id
INNER JOIN tags t ON t.tag_id = ot.tag_id
INNER JOIN hits h ON h.object_id = o.object_id

This was what I wrote too, more or less. However, after spending many hours trying to figure out why that page was loading in more than 100 seconds, whenever it was loading at all, I’ve learned that sometimes it’s better to break the rules and not follow any recommendations. So I tried the following code, and the results were staggering – less than 1s loading time:

$rs = mysql_query('select * from objects limit 20');
$categories = $tags = array(); //used for locally caching categories and tags
while( $row = mysql_fetch_array($rs) ) {
  if( !isset($categories[$row['category_id']]) {
    $rsCat = mysql_query('SELECT category_name FROM categories WHERE category_id = ' . $row['category_id'] . ' LIMIT 1';
    $rowCat = mysql_fetch_array($rsCat);
    $categories[$row['category_id']] = $rowCat['category_name'];
    unset($rowCat, $rsCat);

  $objectTags = $toLoad = array();
  $rsObjTags = mysql_query('SELECT tag_id FROM object_tags WHERE object_id = ' . $row['object_id']);
  while( $rowObjTags = mysql_fetch_array($rsObjTags) ) {
    if( isset($tags[$rowObjTags['tag_id']]) ) {
      $objectTags[$rowObjTags['tag_id']][] = $tags[$rowObjTags['tag_id']];
    else {
      $toLoad[] = $rowObjTags['tag_id'];
  unset($rsObjTags , $rowObjTags);

  if( count($toLoad) ) {
    $rsTags = 'SELECT tag_id, tag FROM tags WHERE tag_id IN (' . implode(',', $toLoad) . ') LIMIT ' . count($toLoad);
    while( $rowTags  = mysql_fetch_array($rsTags) ) {
      $tags[$rowTags['tag_id']] = $rowTags['tag'];
      $objectTags[$rowTags['tag_id']][] = $rowTags['tag'];
  unset($toLoad, $rsTags, $rowTags);

  $rsHits = 'SELECT today_hits FROM hits WHERE object_id = ' . $row['object_id'] . ' LIMIT 1';
  $hits = mysql_fetch_array($rsHits);

  //ready to display the data
  echo $row['object_name'] . ' is in category ' . $categories[$row['category_id']] . '<br />';
  echo 'Tags: ' . implode(', ', $objectTags) . '<br/>';
  echo 'Hits today: ' . $hits['today_hits'] . '<hr />';

I hope there aren’t too many mistakes in the code, I wrote everything in here without actually testing it.

Seems like a lot of work and totally not optimized for something quite straight-forward. However, let’s assume once more that the tables above have the following number of records:
objects – 50,000
categories – 500
tags – 100,000
objects_tags – 250,000
hits – 20,000,000

In this case, 4 joins would result in a huge dataset, requiring a lot of memory, and eventually ending up on the disk. Also, the LIMIT doesn’t help at all, since the dataset is truncated at the end, after scanning all the necessary rows. However, in the unoptimized version we do benefit from the LIMIT, by only getting a fixed set (20 rows) from 50k rows. The rest of the queries are acceptable because they are all primary key based, they are using “LIMIT” as well (3 out of 4), and because their number is rather small – somewhere between 45 and 80 queries.

Hope this helps someone some day. I’d like to know your thoughts, what would have you done? Any other quick fixes for this problem?


Installing Memcache Server for PHP on Windows

February 15th, 2012

Memcached is a free open source, high-performance, distributed memory object caching system. It is currently used by a lot of websites, including Flickr, Twitter, Youtube, Digg and WordPress.

I’ve been using memcached on a few production servers, but never thought it could come in handy on a Windows development machine – in fact I didn’t even thought it was available on Windows. So, a few days ago, while working on yet another Magento project, I ran a Google search for “memcache windows” and it turned out there are a few Win32 ports of the original version. Cool! It can be used on Windows, now all I need is to find the right PHP extension.

Here’s a step-by-step tutorial on how to get memcached running in PHP on a Windows box. There’s are a bunch of really good tutorials out there, but I think another could only be helpful.

1. Go to and download memcached. I’ve used the 09.03.2008 binaries, memcached 1.2.4.
You can also use v1.2.6 that you can download from – I’ve used this one to update from the previous 1.2.4 and I can confirm that it works.

2. Unzip the downloaded file to any folder (i.e. C:/memcached). I’ve saved in the same place where I have Apache, MySQL and PHP.

3a. If you’re on Windows Vista, navigate to your memcached folder, right click on memcached.exe and click Properties, then click the Compatibility tab. In here, check the “Run this program as an administrator” checkbox.

3. Install the service by running memcached.exe -d install from the command line. (you should be in the folder where you unzipped the file(s), ie: c:/memcached)

By default, memcached has 64M available for caching and runs on port 11211. If you need to change any of these you can do it by editing the Registry – open regedit then search for HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/memcached Server. Change the ImagePath entry from "C:\memcached\memcached.exe" -d runservice to "C:\memcached\memcached.exe" -d runservice -m 256 -p 11222". This gives you 256 megs of memory and change the port to 11222 (on a development machine you probably don’t need to change the port, but who knows). See the full list of memcached options – not sure which ones apply to the Windows version too.

4. Now, that the service is installed and configured, let’s start it: memcached.exe -d start
To test that memcached is running you can try telnet’ing – telnet localhost 11211 (use the port you configured the service to listen to). If you can connect on that port, everything is fine.

5. Memcached is installed, but we need to use in PHP, so we need an extension. Here’s a list of sites where you can download the dll from:

I had to install a VC6 dll, so I’ve downloaded the last one. Unzip it and put it in your php extensions folder. If you don’t know where this folder is, try this command: php -i | find "extension_dir". Add the dll in there, then open php.ini (to find it, you can run php -i | find "Loaded Config") and at the end add something like this:


Now restart Apache.

6. To test that memcached can now be used in PHP, use the following snippet:

$memcache = new Memcache;
if( !$memcache->connect('localhost', 11211)) { //change 11211 with the port your memcached is configured to listen to
  die('Could not connect!');

echo '

If you see an array with a bunch of data in it, you’ve done it. If not, something went wrong at some point. Make sure memcached is running, then make sure you’ve got the correct php settings and extension.

As I mentioned at the beginning of the post, installing memcached was something that just hit me while working on a Magento project – for those familiar with Magento, you know how slow it can be sometimes, especially when you need to constantly refresh the pages to check html/js/css changes. So, I thought memcached would help speed up the parts on which I wasn’t working on – everything except layouts and html block cache. And so it did, development seems MUCH easier now – but all about this in another post, soon to come.

Many thanks for helping me figure this out and (hopefully) speeding up development to:

PHP , ,

Introducing Magento CSS & JS Minifier

September 15th, 2010

As the title says, I’m glad to announce my first public Magento extension (not yet added in the Connect repository). During my 3 years experience with Magento, I’ve worked on a lot of custom extensions, improvements & fixes, but most of them were client-specific, plus they weren’t designed to have a backend interface (with a few exceptions). This one, however, is entirely configurable from the Admin and it’s both simple and effective.


What it does

This quick optimizer parses all javascript & css files included on a page and removes all unnecessary characters. The most simple step is to remove spaces, tabs and new lines – but there’s more than just that. Of course, for small files compression is insignificant, but when you work with almost 600KB and around 30 requests, you can save a lot. Here’s a quick math on one of my Magento installs:

Javascript – 26 requests, 479 KB
CSS – 4 requests, 102 KB

With Magento’s default merging enabled:
Javascript – 1 request, 360 KB (not sure why this is smaller then the 26 summed up, but nvm)
CSS – 1 request, 108.2 KB (same for this one too, but again, nvm)

We’ve already saved 28 requests, which means less overhead – quicker download times for user, less stress on the server.

With the Minifier enabled:
Javascript is 255 KB, which means almost 47% compression
CSS is 92 KB, which means almost 10% compression





  • Need to write javascripts very careful, adding a semicolon after almost everything
  • Have to rewrite of Mage_Core_Model file
  • Have to override two Magento methods
  • Additional processing time (insignificant in my opinion)



This plugin wouldn’t exist if it weren’t for these two outstanding PHP projects:
Joe Scylla’s CssMin
Ryan Grove’s JsMin
Big thanks to both of them.


Download & Install

First, Minifier for Magento (1418 downloads) , then unzip it and copy the app/ folder to your Magento root folder.

Logout from the admin if you’re already logged in, then login. Go to Cache Management, click on the “Flush cache storage” button, then go to Configuration > Developer and enable all the options, as shown in the attached screenshot. Go back to Cache Management and this time click on the “Flush Javascript/CSS cache”.

Go to your store frontend and behold, you’re now using compressed js’s and css’s.

Works on Magento 1.4+

Updated on May 31st, 2012 – all download links in this page refer to the latest version of the plugin. Some explanations on this post might be inaccurate. For the latest details please check new post called Updated version of the Magento CSS & JS Minifier

Magento , , ,

Reducing the number of CSS requests

September 11th, 2009

One common problem with larger/trafficked websites is that they end up serving a considerable amount of css data. Most of the frontend developers use individual style sheets for resets, structure, design, homepage / innerpages and so on. In fact, there’s no other rule here than keeping the css code as maintainable as possible. This also means adding comments and maybe writing every declaration on a single line, using indents.

Another well-known problem is with caching. When the CSS file is changed, most of the times you’ll need to clear the browser cache in order to display the site correctly, with the new changes applied. Well, for users that’s not an option and all the css upgrades should be reflected instantly. I know what you’re saying: there’s Apache’s mod_expire for that, but there are times when you cannot use that or don’t know about it or you don’t know how to get it running. If you fall into this category, this article should solve that.

Let’s take an example first. On the homepage of a site, there are 3 style sheet files requests. Their total size is 46,274 bytes – that’s roughly 46KB. My aim is to only request a single CSS file and make it change its name when anything in those 3 files changes. The site uses templates, so this tutorial will be based on that architecture – for plain ole’ php-html sites, this should be even simpler.

So let’s write a list of what needs to be done:

  1. Reflect any css change into the file name so browser’s cache is instantly invalidated
  2. Combine all requested files into a single file
  3. Create a queue of css files: both templates and controller files should be able to read & write to the queue.

Step 1 – Changes to the Template Model

In the template model, add new public methods for addCss( $cssFile, $media = 'screen') and getCompiledCss($media = 'screen'). Here’s how it could look like:

function addCssFile( $file, $media = 'screen' ) { 
	$this->cssFiles[] = array( 'url' => $file, 'media' => $media ); 
	return $this; 

function getCompiledCss( $media = 'screen' ) {
	if( $this->compiledCss instanceof CompiledCss ) { return $this->compiledCss->getFilename(); }
	if( !count($this->cssFiles) ) { return null; }

	$cssMedia = array();
	foreach( $this->cssFiles as $css ) {
		if( !isset($cssMedia[ $css['media'] ]) ) {
			$cssMedia[ $css['media'] ] = array();
		$cssMedia[ $css['media'] ][] = $css['url']; 
	if( !$media || !isset($cssMedia[$media]) || !count($cssMedia[$media]) ) { return null; }
	$this->compiledCss = new CompiledCss( $cssMedia[$media] );
	return $this->compiledCss->getFilename();

Maybe the getCompiledCss seems complicated, but in fact it’s not. It’s just a proxy method with some lazy-initialization in it. It only checks to see if the CompiledCss object is instantiated (creates it if it’s not), then return the compiled CSS file name. So basically, everything is done in the CompiledCss class.

Step 2 – Changes to the template

All valid and W3C-compliant CSS declarations go in the header of the HTML, so it should be easy to group all CSS requests together and call the getCompiledCss somewhere in the main layout file or in a head template component. Either way, here’s a quick peek at how the templates could look like:

	$this->addCssFile('general.css', 'screen')
		->addCssFile('homepage.css', 'screen')
		->addCssFile('panels.css', 'screen');
<?php if( $this->getCompiledCss('screen') ): ?>
	<link type="text/css" rel="stylesheet" href="<?php echo $this->getCompiledCss('screen'); ?>" media="Screen" />
<?php endif; ?>

Step 3 – CompiledCss Class – main logic

As we’ve seen above, 99% of the logic is done inside this class, so tasks #1 & #2 are its responsibilities. First of all, remember that inside the Template Model there was only a call for getFilename() – let’s take a look at it:

function getFilename() {
	if( null === $this->filename ) {
	return $this->filename;

Again, it’s pretty simple – all it does is to call the compile() method once. Otherwise, if this method is called more than once, it just returns the same file name. As you can see, complie() does the entire job, actually, so on with it:

First, make sure each css file gets compiled only once, then sort their names alphabetically, so that the generated file name is always the same, no matter the order you add the files (I know this can cause some serious issues – but bear with me until the end for problems, solutions & improvements

$this->cssFiles = array_unique($this->cssFiles);

The file name will need to parts: one for recognizing the css files that are “inside” and another one for content checksum – the second part helps generating new file names when the source files are modified.

$filename = $out = '';
foreach( $this->cssFiles as $css ) {
	$filename .= Crc32($css); 
	$cssPath = 'path/to/css/files/' . $css; #whatever path you need to get to the css files
	if( is_file($cssPath) && is_readable($cssPath) ) {
		$out .= file_get_contents($cssPath);
$filename = Crc32($filename);
$checksum = Crc32($out);

We need to cache this compiled css, to be able to serve CSS data. As opposed to the method of requesting a list of css files on a GET request, this one needs to save the parsed css someplace, otherwise decoding the checksums would be almost impossible. So, write the “processed” css data to a file and than load it each time it is requested published here. Here’s the code for actually compressing the css:

$output = preg_replace('#[ \t]+#', ' ', $output); #replace multi-spaces or multi-tabs with a single space
$output = preg_replace('#[\n\r]+#', '', $output); #remove new lines
$output = preg_replace('#([:;])\s+#', '$1', $output); #remove spaces after : and ;
$output = preg_replace('#/\*(.*?)\*/#', '', $output); #remove all comments
return $output;

Step 4 – Final touches

One more step is required for this to work. You need a php file handling the compiled css request. Let’s assume your requests (and compiled file name) look like this: crc32-crc32.css. That means you’re gonna create a mod_rewrite rule in a .htaccess file mapping 8 chars, dash, 8 chars, dot css to a php file, passing both checksums as parameters. Inside the php file, just send some headers and request the file:

header('Content-Type: text/css');
header('Cache-Control: cache');
header('Pragma: cache');
header('Expires: ' . gmdate('r', strtotime('+1 year')));
echo CompiledCss::request( $firstChecksum, $secondChecksum );  

request() logic is quite simple – check for existing cache and display it, otherwise just return null. Usually, if the process runs normally, first we’re gonna get the getFilename() call which saves the cache and returns a valid file name, and just after that the request() method fires.

Compression performance

Like I said in the beginning of this post, the original file size for all three requests for this example was roughly 46K. After compilation it got to 42K – that’s around 91% of the original size, aprox. 10% compression.
Best compression level with this tools was 13%, but the usual rate is 10%. Keep in mind, however, that it depends on how the css file is written. For example, I write each declaration inline, so there are only a few spaces and new lines, but for indented css the compression could reach 20% (tested).

So there you have it. At least 10% improvement, a few extra requests saved, and the ability to invalidate browser cache on-demand. I’ve been using this tool for a while now and I can tell you I have no design/layout problems or conflicts and the total size of css requests have saved enough bandwidth – not to mention that download times are reduced. Add mod_deflate or manual gzip compression to it and it’s gonna download even faster.

Problems, Solutions & Improvements

I know some of you might have seen only problems in this article. Or you might thing this is way too complicated and it’s not needed cause there are other ways (much more simpler and faster) of doing it. So let’s see what problems I’ve identified so far and what you can do:

Problem: if CSS files are sorted and they are not compiled in the same order as they’re given, there might be conflicts, overlapping styles, and in the end the design might be screwed up.
» Solution(s): One of them is to improve your css declarations making them work the same, no matter how they’re parsed. The other solution is to remove the sorting in the compile() method, but be sure to always use the same queue push to get the same filename. For example, pushing a.css, b.css in one place an b.css, a.css in another place will create two separate css cache files.

Problem: crc32 only uses 32 bits, so checksum collisions might occur (although the chances are minimal)
» Solution: Switch to md5 or even sha1 if you think those don’t create collisions.

Problem: when stripping spaces after :;, there are chances to affect strings, like for instance url’s.
» Solution: try to keep your filenames free of : and ;, or, if this isn’t enough, remove this line: $output = preg_replace('#([:;])\s+#', '$1', $output);
The same applies for removing multi-spaces and multi-tabs.

Problem: removing comments might break some IE hacks
» Solution: hacks are not recommended anyway, so try using separate IE6- and IE7+ stylesheets, if nothing else works. Again, removing that line from the output parsing helps, but eventually the compression level would be zero.

Improvement: gzip the css and server gzip content if the client accepts gzip. Try enabling mod_deflate anyway, if load is not really an issue.

Improvement: check you inodes cache settings to help speeding disk reads – as you’ve noticed we’re reading all css files each time a page is requested, so some system tunin might help.

PHP , ,