Archive

Posts Tagged ‘PHP’

Magento memory leaks and custom options

August 30th, 2012

Most software, if not all, has memory leaks. Some are really bad at collecting garbage, other manage it better, but not enough. Magento had its problems since the beginning, and a lot of them have been fixed in the latest releases (using 1.6 to test this). However, it still has a long way until all major memory problems are gone. A simple test, like the one below, will reveal what I mean (put it in your magento root and run it from the CLI):

require "app/Mage.php";
Mage::app();

for($k = 0; $k < 100; $k++) {
    Mage::getModel('catalog/product')->load(123);
    echo $k . "\t" . (memory_get_usage()/1024/1024) . "\n";
}

This will show the current iteration and the memory usage. Replace “123” with a real product id – first, use a simple product, with no custom options. My result looks like this:

0 - 8.3790
...
99 - 8.3794

Looks good, very good actually.

Now try it with a product that has custom options. Not more than 5-6 options, but stuff a dozen values in one or two options. My test shows these numbers:

0 - 8.9398
...
99 - 17.5057

.

Along the way, you’ll see that the garbage collector kicks in and memory is freed, so max memory usage (for me) doesn’t exceed 20M. Cool, everything seems acceptable, and it seems that the old memory leaks have been fixed (read more over here, here and here ).

Now let’s try something different. Importing a few hundred products from a CSV. All products have custom options, usually the same, but with different values. The code could be summarized like this:

$options = $this->_getOptions($row);
$product = $this->initNewProduct(); //just inits the product with all the required data
$product->addData($specificFields)
        ->setAttributeSetId( $attributeSetId )
        ->setTypeId($typeId)
        ->setCategoryIds( array($someIds = 123) );
		
if( $options && count($options) ) {
    $product->setCanSaveCustomOptions(true);
    $product->setProductOptions($options);
}

$product->save();

To me, this looked fine and should have worked. I didn’t care it ran out of memory after 60-70 cycles, I could live with that. However, after approximately 20 hours of intense debugging, trying to figure out why the options aren’t showing up correctly on the frontend I decided to check the products in the admin. I had a huge surprise when I’ve noticed that for the second product, I had the first product’s options and the (correct) second product’s options. For the third, I had the first, second and third option sets. And so on, the last one was a complete mess. At some point Firefox couldn’t even load the interface anymore, there were simply too many options. Now that’s a memory leak for sure!

So, is there a fix? Seems like there was one, tested it for 3 products and it worked fine, no more leftover options. Initially, I was doing this:

$product->clearInstance();
unset($product);
				
gc_collect_cycles();

but it didn’t work, although gc_collect_cycles() helped keeping the memory growth under control.

So I added the following lines:

$product->getOptionInstance()->unsetOptions()->clearInstance();
unset($product, $options); //instead of just unset($product);

and voila – all custom options have been created correctly. Haven’t really analyzed why this did the trick and why it didn’t work without it. Either way, I’m happy the code works and I hope this will help someone else someday.

Magento , ,

Installing Memcache Server for PHP on Windows

February 15th, 2012

Memcached is a free open source, high-performance, distributed memory object caching system. It is currently used by a lot of websites, including Flickr, Twitter, Youtube, Digg and WordPress.

I’ve been using memcached on a few production servers, but never thought it could come in handy on a Windows development machine – in fact I didn’t even thought it was available on Windows. So, a few days ago, while working on yet another Magento project, I ran a Google search for “memcache windows” and it turned out there are a few Win32 ports of the original version. Cool! It can be used on Windows, now all I need is to find the right PHP extension.

Here’s a step-by-step tutorial on how to get memcached running in PHP on a Windows box. There’s are a bunch of really good tutorials out there, but I think another could only be helpful.

1. Go to http://splinedancer.com/memcached-win32/ and download memcached. I’ve used the 09.03.2008 binaries, memcached 1.2.4.
You can also use v1.2.6 that you can download from http://code.jellycan.com/memcached/ – I’ve used this one to update from the previous 1.2.4 and I can confirm that it works.

2. Unzip the downloaded file to any folder (i.e. C:/memcached). I’ve saved in the same place where I have Apache, MySQL and PHP.

3a. If you’re on Windows Vista, navigate to your memcached folder, right click on memcached.exe and click Properties, then click the Compatibility tab. In here, check the “Run this program as an administrator” checkbox.

3. Install the service by running memcached.exe -d install from the command line. (you should be in the folder where you unzipped the file(s), ie: c:/memcached)

By default, memcached has 64M available for caching and runs on port 11211. If you need to change any of these you can do it by editing the Registry – open regedit then search for HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/memcached Server. Change the ImagePath entry from "C:\memcached\memcached.exe" -d runservice to "C:\memcached\memcached.exe" -d runservice -m 256 -p 11222". This gives you 256 megs of memory and change the port to 11222 (on a development machine you probably don’t need to change the port, but who knows). See the full list of memcached options – not sure which ones apply to the Windows version too.

4. Now, that the service is installed and configured, let’s start it: memcached.exe -d start
To test that memcached is running you can try telnet’ing – telnet localhost 11211 (use the port you configured the service to listen to). If you can connect on that port, everything is fine.

5. Memcached is installed, but we need to use in PHP, so we need an extension. Here’s a list of sites where you can download the dll from:
http://downloads.php.net/pierre/
http://www.pureformsolutions.com/pureform.wordpress.com/2008/06/17/php_memcache.dll
http://kromann.info/download.php?strFolder=php5_1-Release_TS&strIndex=PHP5_1
shikii.net/blog/downloads/php_memcache-cvs-20090703-5.3-VC6-x86.zip

I had to install a VC6 dll, so I’ve downloaded the last one. Unzip it and put it in your php extensions folder. If you don’t know where this folder is, try this command: php -i | find "extension_dir". Add the dll in there, then open php.ini (to find it, you can run php -i | find "Loaded Config") and at the end add something like this:

[PHP_MEMCACHE]
extension=php_memcache.dll

Now restart Apache.

6. To test that memcached can now be used in PHP, use the following snippet:

$memcache = new Memcache;
if( !$memcache->connect('localhost', 11211)) { //change 11211 with the port your memcached is configured to listen to
  die('Could not connect!');
}

echo '
';
print_r($memcache->getStats());

If you see an array with a bunch of data in it, you’ve done it. If not, something went wrong at some point. Make sure memcached is running, then make sure you’ve got the correct php settings and extension.

As I mentioned at the beginning of the post, installing memcached was something that just hit me while working on a Magento project – for those familiar with Magento, you know how slow it can be sometimes, especially when you need to constantly refresh the pages to check html/js/css changes. So, I thought memcached would help speed up the parts on which I wasn’t working on – everything except layouts and html block cache. And so it did, development seems MUCH easier now – but all about this in another post, soon to come.

Many thanks for helping me figure this out and (hopefully) speeding up development to:
http://splinedancer.com/memcached-win32/
http://pureform.wordpress.com/2008/01/10/installing-memcache-on-windows-for-php/
http://shikii.net/blog/installing-memcached-for-php-5-3-on-windows-7/

PHP , ,

Workaround for attribute store labels not being displayed in Magento

February 12th, 2012

Here’s a quick workaround for a rather strange issue – sometimes, calling $attribute->getStoreLabel() returns null, although everything looks fine in the admin and in the database. It might be something that is slipping my mind, or it might be a bug, not really sure. However, I had to deal with it and come up with a fix, so I’d thought I’d share.

$_aIds = array();
foreach( $prod->getAttributes() as $attribute ) {
	//need the attribute_id which is missing from the attribute object
	if( !isset($_aIds[$attribute->getAttributeCode()]) ) {
		$_aIds[$attribute->getAttributeCode()] = $attribute->getIdByCode('catalog_product', $attribute->getAttributeCode());
	}
			
	//if there's not store_label try to set it
	if( !$attribute->getStoreLabel() ) {
		$labels = $attribute->getResource()->getStoreLabelsByAttributeId( $_aIds[$attribute->getAttributeCode()] );
		if( array_key_exists(Mage::app()->getStore()->getStoreId(), $labels)) {
			$attribute->setStoreLabel( $labels[Mage::app()->getStore()->getStoreId()] );
		} 
		else {
			$attribute->setStoreLabel( $attribute->getFrontendLabel() );
		}
	}
}
// end workaround

Applies to Magento 1.5.x

Magento ,

Patching the Magento USPS module

January 20th, 2011

Since their last update on Jan 3rd, 2011 USPS has stopped working on all of my Magento distro’s. The reason for that is a change in USPS’ response XML values. Magento has its part of blame here, first for not patching it immediately (which again shows how much they care about the CE users) and second for not coding it properly in the first place – they’re using the method name (i.e. Express Mail) to see if that method is allowed from the admin, instead of using the CLASSID attribute.

This issue has been reported on magentocommerce.com bug tracking system and it’s also discussed in the forum. USPS makes it clear that they’ll discontinue all API versions prior RateV4 ad IntlRateV2:
“All Rate Calculator API integrators are encouraged to migrate to the latest API versions (RateV4, IntlRateV2):
RateV4 and IntlRateV2 will be the only Rate Calculator API versions to offer the full range of new products and functionality
Rate, RateV2, RateV3 and IntlRate will be retired in May 2011, requiring all integrators to migrate to the latest versions”

As explained on the forums, the fix is quite simple, as USPS only added the &lt;sup&gt;&reg;&lt;/sup&gt; which is the encoded (twice 🙂 ) version of the registered sign in a sup tag. Magento values are defined without those so when the code tries to match them it fails. All you have to do is clean that string and you’re good to go. I don’t like to modify core files so the fix involves copying app/code/core/Mage/Usa/Model/Shipping/Carrier/Usps.php to app/code/local/Mage/Usa/Model/Shipping/Carrier/ and working on the local version of Usps.php. Strip the special chars:

$mailService = str_replace('&reg;', '', strip_tags(htmlspecialchars_decode(htmlspecialchars_decode((string)$postage->MailService))));

and

$svcDescription = str_replace('&reg;', '', strip_tags(htmlspecialchars_decode(htmlspecialchars_decode((string)$service->SvcDescription))));

 

Here’s the archive with the patch. Just download it and unzip it in the root of your Magento distribution. Download USPS patch for Magento 1.4.2.0 (8.59 kB)

 

Note: This patch applies to Magento CE 1.4.2.0. For all other versions you will have to do it manually as explained above.

 

Magento, PHP , ,

Adding Customer Comments on Invoice PDFs in Magento (using OneStepCheckout)

January 18th, 2010

I’ve recently installed OneStepCheckout (http://www.onestepcheckout.com/) on a couple of Magento installations. The extension is very nice, really simple to integrate and I expect to see better conversion rates on the checkout process.

One cool thing is that it comes with the option of activating Customer Order Comments – it adds a textarea field on the checkout page, and a box with the customer comments in the admin, when viewing the order.

However, one of my clients requested I added these comments in the invoice PDF’s. So, here’s how to do it:

Step 1
Copy app/code/core/Mage/Sales/Model/Order/Pdf/Invoice.php to app/code/local/Mage/Sales/Model/Order/Pdf/Invoice.php

Step 2
Open the new file and create a new method:

	function insertOscComments(&$page, $order) {
		if( !$order->getOnestepcheckoutCustomercomment() ) { return; }
		$this->y -= 20;
		$page->setFillColor(new Zend_Pdf_Color_Rgb(0.93, 0.92, 0.92));
		$page->setLineColor(new Zend_Pdf_Color_GrayScale(0.5));
		$page->setLineWidth(0.5);
		$page->drawRectangle(25, $this->y, 570, $this->y - 20);
		$page->setFillColor(new Zend_Pdf_Color_Rgb(1, 1, 1));
		$page->drawRectangle(25, $this->y - 20, 570, $this->y - 40);
		
		$page->setFillColor(new Zend_Pdf_Color_RGB(0.1, 0.1, 0.1));
		$page->drawText(Mage::helper('onestepcheckout')->__('Customer Comments'), 35, $this->y - 13, 'UTF-8');
		$page->drawText($order->getOnestepcheckoutCustomercomment(), 33, $this->y - 33, 'UTF-8');
		$this->y -= 50;
	}

Step 3
At the end of method getPdf add a call to the new method you created:

  /* Add totals */
  $this->insertTotals($page, $invoice);

  /* Add OneStepCheckout Customer Comments */
  $this->insertOscComments($page, $order);
}

And that’s all you need.

Magento , ,

Reducing the number of CSS requests

September 11th, 2009

One common problem with larger/trafficked websites is that they end up serving a considerable amount of css data. Most of the frontend developers use individual style sheets for resets, structure, design, homepage / innerpages and so on. In fact, there’s no other rule here than keeping the css code as maintainable as possible. This also means adding comments and maybe writing every declaration on a single line, using indents.

Another well-known problem is with caching. When the CSS file is changed, most of the times you’ll need to clear the browser cache in order to display the site correctly, with the new changes applied. Well, for users that’s not an option and all the css upgrades should be reflected instantly. I know what you’re saying: there’s Apache’s mod_expire for that, but there are times when you cannot use that or don’t know about it or you don’t know how to get it running. If you fall into this category, this article should solve that.

Let’s take an example first. On the homepage of a site, there are 3 style sheet files requests. Their total size is 46,274 bytes – that’s roughly 46KB. My aim is to only request a single CSS file and make it change its name when anything in those 3 files changes. The site uses templates, so this tutorial will be based on that architecture – for plain ole’ php-html sites, this should be even simpler.

So let’s write a list of what needs to be done:

  1. Reflect any css change into the file name so browser’s cache is instantly invalidated
  2. Combine all requested files into a single file
  3. Create a queue of css files: both templates and controller files should be able to read & write to the queue.


Step 1 – Changes to the Template Model

In the template model, add new public methods for addCss( $cssFile, $media = 'screen') and getCompiledCss($media = 'screen'). Here’s how it could look like:

function addCssFile( $file, $media = 'screen' ) { 
	$this->cssFiles[] = array( 'url' => $file, 'media' => $media ); 
	return $this; 
}

function getCompiledCss( $media = 'screen' ) {
	if( $this->compiledCss instanceof CompiledCss ) { return $this->compiledCss->getFilename(); }
	if( !count($this->cssFiles) ) { return null; }

	$cssMedia = array();
	foreach( $this->cssFiles as $css ) {
		if( !isset($cssMedia[ $css['media'] ]) ) {
			$cssMedia[ $css['media'] ] = array();
		}
		$cssMedia[ $css['media'] ][] = $css['url']; 
	}
	if( !$media || !isset($cssMedia[$media]) || !count($cssMedia[$media]) ) { return null; }
	
	$this->compiledCss = new CompiledCss( $cssMedia[$media] );
	return $this->compiledCss->getFilename();
}

Maybe the getCompiledCss seems complicated, but in fact it’s not. It’s just a proxy method with some lazy-initialization in it. It only checks to see if the CompiledCss object is instantiated (creates it if it’s not), then return the compiled CSS file name. So basically, everything is done in the CompiledCss class.


Step 2 – Changes to the template

All valid and W3C-compliant CSS declarations go in the header of the HTML, so it should be easy to group all CSS requests together and call the getCompiledCss somewhere in the main layout file or in a head template component. Either way, here’s a quick peek at how the templates could look like:

<?php 
	$this->addCssFile('general.css', 'screen')
		->addCssFile('homepage.css', 'screen')
		->addCssFile('panels.css', 'screen');
?>
<?php if( $this->getCompiledCss('screen') ): ?>
	<link type="text/css" rel="stylesheet" href="<?php echo $this->getCompiledCss('screen'); ?>" media="Screen" />
<?php endif; ?>


Step 3 – CompiledCss Class – main logic

As we’ve seen above, 99% of the logic is done inside this class, so tasks #1 & #2 are its responsibilities. First of all, remember that inside the Template Model there was only a call for getFilename() – let’s take a look at it:

function getFilename() {
	if( null === $this->filename ) {
		$this->compile();
	}
	return $this->filename;
}

Again, it’s pretty simple – all it does is to call the compile() method once. Otherwise, if this method is called more than once, it just returns the same file name. As you can see, complie() does the entire job, actually, so on with it:

First, make sure each css file gets compiled only once, then sort their names alphabetically, so that the generated file name is always the same, no matter the order you add the files (I know this can cause some serious issues – but bear with me until the end for problems, solutions & improvements

$this->cssFiles = array_unique($this->cssFiles);
asort($this->cssFiles);

The file name will need to parts: one for recognizing the css files that are “inside” and another one for content checksum – the second part helps generating new file names when the source files are modified.

$filename = $out = '';
foreach( $this->cssFiles as $css ) {
	$filename .= Crc32($css); 
	$cssPath = 'path/to/css/files/' . $css; #whatever path you need to get to the css files
	if( is_file($cssPath) && is_readable($cssPath) ) {
		$out .= file_get_contents($cssPath);
	}
}
$filename = Crc32($filename);
$checksum = Crc32($out);

We need to cache this compiled css, to be able to serve CSS data. As opposed to the method of requesting a list of css files on a GET request, this one needs to save the parsed css someplace, otherwise decoding the checksums would be almost impossible. So, write the “processed” css data to a file and than load it each time it is requested published here. Here’s the code for actually compressing the css:

$output = preg_replace('#[ \t]+#', ' ', $output); #replace multi-spaces or multi-tabs with a single space
$output = preg_replace('#[\n\r]+#', '', $output); #remove new lines
$output = preg_replace('#([:;])\s+#', '$1', $output); #remove spaces after : and ;
$output = preg_replace('#/\*(.*?)\*/#', '', $output); #remove all comments
return $output;


Step 4 – Final touches

One more step is required for this to work. You need a php file handling the compiled css request. Let’s assume your requests (and compiled file name) look like this: crc32-crc32.css. That means you’re gonna create a mod_rewrite rule in a .htaccess file mapping 8 chars, dash, 8 chars, dot css to a php file, passing both checksums as parameters. Inside the php file, just send some headers and request the file:

header('Content-Type: text/css');
header('Cache-Control: cache');
header('Pragma: cache');
header('Expires: ' . gmdate('r', strtotime('+1 year')));
echo CompiledCss::request( $firstChecksum, $secondChecksum );  

request() logic is quite simple – check for existing cache and display it, otherwise just return null. Usually, if the process runs normally, first we’re gonna get the getFilename() call which saves the cache and returns a valid file name, and just after that the request() method fires.


Compression performance

Like I said in the beginning of this post, the original file size for all three requests for this example was roughly 46K. After compilation it got to 42K – that’s around 91% of the original size, aprox. 10% compression.
Best compression level with this tools was 13%, but the usual rate is 10%. Keep in mind, however, that it depends on how the css file is written. For example, I write each declaration inline, so there are only a few spaces and new lines, but for indented css the compression could reach 20% (tested).

So there you have it. At least 10% improvement, a few extra requests saved, and the ability to invalidate browser cache on-demand. I’ve been using this tool for a while now and I can tell you I have no design/layout problems or conflicts and the total size of css requests have saved enough bandwidth – not to mention that download times are reduced. Add mod_deflate or manual gzip compression to it and it’s gonna download even faster.


Problems, Solutions & Improvements

I know some of you might have seen only problems in this article. Or you might thing this is way too complicated and it’s not needed cause there are other ways (much more simpler and faster) of doing it. So let’s see what problems I’ve identified so far and what you can do:

Problem: if CSS files are sorted and they are not compiled in the same order as they’re given, there might be conflicts, overlapping styles, and in the end the design might be screwed up.
» Solution(s): One of them is to improve your css declarations making them work the same, no matter how they’re parsed. The other solution is to remove the sorting in the compile() method, but be sure to always use the same queue push to get the same filename. For example, pushing a.css, b.css in one place an b.css, a.css in another place will create two separate css cache files.

Problem: crc32 only uses 32 bits, so checksum collisions might occur (although the chances are minimal)
» Solution: Switch to md5 or even sha1 if you think those don’t create collisions.

Problem: when stripping spaces after :;, there are chances to affect strings, like for instance url’s.
» Solution: try to keep your filenames free of : and ;, or, if this isn’t enough, remove this line: $output = preg_replace('#([:;])\s+#', '$1', $output);
The same applies for removing multi-spaces and multi-tabs.

Problem: removing comments might break some IE hacks
» Solution: hacks are not recommended anyway, so try using separate IE6- and IE7+ stylesheets, if nothing else works. Again, removing that line from the output parsing helps, but eventually the compression level would be zero.

Improvement: gzip the css and server gzip content if the client accepts gzip. Try enabling mod_deflate anyway, if load is not really an issue.

Improvement: check you inodes cache settings to help speeding disk reads – as you’ve noticed we’re reading all css files each time a page is requested, so some system tunin might help.

PHP , ,

Showing all reviews and ratings on a page in Magento

August 6th, 2009

Finally, a new post, and at last it is about Mangeto Commerce (http://www.magentocommerce.com/). In my first Magento how-to you’ll learn how to retrieve all product reviews and show them on a single page, together with the average rating. For this, I assume you have already created a new module and are able to view the page. You’ll only need to manipulate a block and a template.

First, let’s retrieve the reviews collection (this method will go into the block):

	function getReviews() {
		$reviews = Mage::getModel('review/review')->getResourceCollection();
		$reviews->addStoreFilter( Mage::app()->getStore()->getId() )
						->addStatusFilter( Mage_Review_Model_Review::STATUS_APPROVED )
						->setDateOrder()
						->addRateVotes()
						->load();        
		
		return $reviews;
	}

We’re using Mage_Review_Model_Mysql4_Review_Collection, which is a resource model. First setup the collection, filtering by store – we only want to retrieve the product reviews in the current store -, by status – show only approved reviews -, and ordering by date in reverse order, then load the collection.

addRateVotes() helps loading all the ratings/votes for that review. We’re gonna use this collection to compute the average rating.

Next, let’s move on to the template for a second. We’re gonna call the getReviews() method, then iterate through all the reviews. For each review you would probably want to display the title, nickname, date and details, but also the product associated and the user rating. For the first four, things are pretty easy, all you have to do is call getTitle(), getNickname(), getDetail(), getCreatedAt() on each review object.

To display the product name & link, we need to retrieve the product associated with each review – unfortunately I wasn’t able to find a way to join the product tables inside the query for retrieving all the reviews. So, we need to create a helper method inside our block, called getProduct(). We’re gonna use a storage/registry variable called _loadedProducts, so that we avoid loading the same product multiple times.

	function getProduct( Mage_Review_Model_Review $review ) {
		if( !isset($this->_loadedProducts[ $review->getEntityPkValue() ]) ) {
			$this->_loadedProducts[$review->getEntityPkValue()] = Mage::getModel('catalog/product')->load( $review->getEntityPkValue() );
		}
		
		return $this->_loadedProducts[ $review->getEntityPkValue() ];
	}

And inside the template:

<?php $_prod = $this->getProduct( $review ); ?>
<a href="<?php echo $_prod->getProductUrl(); ?>"><?php echo $_prod->getName(); ?></a>

One last thing, if you intend to display the average rating of each review, add another helper method inside the block:

	function getAverageRating( Mage_Review_Model_Review $review ) {
		$avg = 0;
		if( count($review->getRatingVotes()) ) {
			$ratings = array();
			foreach( $review->getRatingVotes() as $rating ) {
				$ratings[] = $rating->getPercent();
			}
			$avg = array_sum($ratings)/count($ratings);
		}
		
		return $avg;
	}

And then call it in the template (in this example we’re using Magento’s default styling):

<div class="rating-box">
	<div class="rating" style="width: <?php echo ceil($this->getAverageRating( $review )); ?>%;"></div>
</div>

That is all! You now have a page where all product reviews can be display.
Things to consider: pagination and cache!

Update: Seems that Magento comes prepared for reviews on products, categories and customers. We only need to load product reviews, so it would be wise to filter by entity. Unfortunately, the current version of Magento doesn’t allow filtering for a certain entity, only by entity and entity PK (which is the product ID in this case). Of course, we could write a decorator and write a method to just add a filter for entity_code = 'product', but the quickest (and dirtiest) way of doing it is by adding a check inside the template foreach loop (or adding a helper method in the block):


if( $review->getEntityId() == 1 ) { continue; }
//1 is the id of the 'product' entity - if you write a method, use a class constant

Note: This has only been tested on Magento 1.3.x – 1.5.x.

Magento ,