ALL countries in checkout, although allowed_countries set correctly

We had the problem, that we have a short list of countries we ship to:

Country list configuration in the backend shows a filtered list

But the country list for the invoice and shipping address in the checkout is complete.

Country select in the checkout shows all countries to choose from

Config

The configuration is fine as you can see above. I checked the cache, because I thought it might be wrong - but the config cache too was fine.

Cache

Digging into the issue shows, that more caches are involved:

\Mage_Checkout_Block_Onepage_Abstract::getCountryOptions
public function getCountryOptions()
{
    $options    = false;
    $useCache   = Mage::app()->useCache('config');
    if ($useCache) {
        $cacheId    = 'DIRECTORY_COUNTRY_SELECT_STORE_' . Mage::app()->getStore()->getCode();
        $cacheTags  = array('config');
        if ($optionsCache = Mage::app()->loadCache($cacheId)) {
            $options = unserialize($optionsCache);
        }
    }

    if ($options == false) {
        $options = $this->getCountryCollection()->toOptionArray();
        if ($useCache) {
            Mage::app()->saveCache(serialize($options), $cacheId, $cacheTags);
        }
    }
    return $options;
}

So I checked the cache entries for DIRECTORY_COUNTRY_SELECT_STORE_DE and found a complete list of countries. So the assumption is: Something "poisons" the cache.

So, what is writing into this cache and maybe we have a broken extension - we don't.

\Mage_XmlConnect_Model_Simplexml_Form_Element_CountryListSelect::_getCountryOptions
\Mage_XmlConnect_Block_Checkout_Address_Form::_getCountryOptions
\Mage_Directory_Block_Data::getCountryHtmlSelect
\Mage_Checkout_Block_Onepage_Abstract::getCountryOptions
\Mage_Catalog_Model_Product_Attribute_Source_Countryofmanufacture::getAllOptions

If we look further into it, we can assume the problem is not from here:

\Mage_XmlConnect_Block_Checkout_Address_Form::_getCountryOptions
\Mage_XmlConnect_Model_Simplexml_Form_Element_CountryListSelect::_getCountryOptions
\Mage_Checkout_Block_Onepage_Abstract::getCountryOptions
\Mage_Directory_Block_Data::getCountryHtmlSelect

because all of them filter the country list via Mage::getModel('directory/country')->getResourceCollection()->loadByStore();, therefore the filter of the config is applied.

\Mage_Catalog_Model_Product_Attribute_Source_Countryofmanufacture

The source model doesn't do correct filtering.

How is the cache filled with wrong data?

This is the tricky part. The only place were I know the source model is used is the backend. But Mage::app()->getStore()->getCode() is always admin. Even when you edit data for another store view.

After digging a while I found out, that when I place a configurable product into the cart one of our extensions reads the country of manufacturer in the frontend and poisons the cache.

I fixed the problem with a rewrite:

class Project_Bugfix_Model_Catalog_Product_Attribute_Source_Countryofmanufacture
    extends Mage_Catalog_Model_Product_Attribute_Source_Countryofmanufacture
{
    /**
     * Get list of all available countries
     *
     * @return mixed
     */
    public function getAllOptions()
    {
        $cacheKey = 'DIRECTORY_COUNTRY_SELECT_STORE_' . Mage::app()->getStore()->getCode();
        if (Mage::app()->useCache('config') && $cache = Mage::app()->loadCache($cacheKey)) {
            $options = unserialize($cache);
        } else {
            if (Mage::app()->getStore()->isAdmin()) {
                $collection = Mage::getModel('directory/country')->getResourceCollection();
            } else {
                $collection = Mage::getModel('directory/country')->getResourceCollection()->loadByStore();
            }
            $options = $collection->toOptionArray();

            if (Mage::app()->useCache('config')) {
                Mage::app()->saveCache(serialize($options), $cacheKey, ['config']);
            }
        }
        return $options;
    }   
}

Custom Product Attributes, Source Models and Flat Tables

How to add an attribute to product_flat_table

Either one of the following attribute settings is true:

  • backend_typ = 'static'
  • is_filterable > 0
  • used_in_product_listing = 1
  • is_used_for_promo_rules= 1
  • used_for_sort_by = 1

Or you add it manually by observing this event:
catalog_product_flat_prepare_columns

Attribute is not added - source models

If you attribute is still not added it might be, because \Mage_Eav_Model_Entity_Attribute_Abstract::getFlatColumns is called on the attribute, which checks wether a source model exists.

public function getFlatColumns()
{
    // If source model exists - get definition from it
    if ($this->usesSource() && $this->getBackendType() != self::TYPE_STATIC) {
        return $this->getSource()->getFlatColums();
    }
    // ...
}

As you can see, getFlatColums is called on your source model, which default implementation is:

// \Mage_Eav_Model_Entity_Attribute_Source_Abstract::getFlatColums
public function getFlatColums()
{
    return array();
}

Therefore your attribute is not added.

And implementation like this might help:

public function getFlatColums()
{
    $attributeCode = $this->getAttribute()->getAttributeCode();
    $column        = [
        'unsigned' => false,
        'default'  => null,
        'extra'    => null,
    ];

    if (Mage::helper('core')->useDbCompatibleMode()) {
        $column['type']    = 'varchar';
        $column['is_null'] = true;
    } else {
        $column['type']     = Varien_Db_Ddl_Table::TYPE_VARCHAR;
        $column['nullable'] = true;
        $column['comment']  = 'Seals column';
    }

    return [$attributeCode => $column];
}

*Update*

How to fill the field

After the final win that the column is created on the flat tables, it was NULL after reindexing. Something is missing and this is:

\Mage_Eav_Model_Entity_Attribute_Source_Abstract::getFlatUpdateSelect

This method has to return a Zend_Db_Select due to this:

\Mage_Catalog_Model_Resource_Product_Flat_Indexer::updateAttribute
// ...
$select = $attribute->getFlatUpdateSelect($storeId);
if ($select instanceof Varien_Db_Select) {
    if ($productIds !== null) {
        $select->where('e.entity_id IN(?)', $productIds);
    }
}
// ...

An example implementation which works for me:

public function getFlatUpdateSelect($store)
{
    return Mage::getResourceSingleton('eav/entity_attribute')
        ->getFlatUpdateSelect($this->getAttribute(), $store);
}

Magento 1 Update - How to

Rico Neitzel told me back in the days, that during Magento updates files gets deleted, therefore it is a bad idea, therefore you want to run a patch (not the ones from Magento, but a patch between the versions) over your Magento installation.

After fiddeling around for a while it is a good idea to document, what I have just done (I updated from 1.9.2.2 to 1.9.3.3):

  1. Get a trusted git repository with all Magento versions you need, I can recommend https://github.com/OpenMage/magento-mirror
  2. Clone it

    git clone https://github.com/OpenMage/magento-mirror.git
    
  3. Make sure your current installation is free of core hacks and all files exists which might get patched

    magento-mirror$ git checkout 1.9.2.2
    production$  rsync --progress -v -r ../magento-mirror/* .
    
  4. You might have now too much files - either you have them already in your gitignore - if not, time to add them.

  5. Create a patch file (diff-index is needed for binary patches)

    magento-mirror$ git checkout 1.9.3.3
    magento-mirror$ git diff-index 92a1142a3 --binary > patch.patch
    
  6. Apply the patch

    production$ git apply --binary ../magento-mirror/patch.patch
    
  7. Now you have a working copy with all the changes. You can review them, commit them, etc.

Magento core_config_data not loaded in backend

Guest Author: Rico Neitzel, http://buro71a.de/

I sometimes have the following strange behaviour in Magento:

Config not loaded in Backend

  1. I do have valid config entries in core_config_data Table
  2. The backend in System -> Config doesn't reflect these config values from the table

On further investigation I saw that this usually only applies to the General Tab and the General Section.

My finding was astonishing! There had been a core_config_data Entry with the following data:

ID scope scope_id path value
123 default 0 general NULL

How did that happen?

I'm pretty sure that I accidentally clicked the "Add new row" Button in my SQL-Tool. That prefilled the fields with its (in SQL defined) default values. The main issue here is the default value for the path field: general

What did it do?

When Magento loads the configuration and applies the database config values this NULL value for general will overwrite all general/* settings and so the backend cannot find anything although it's stored in the DB.

How to solve it?

Dead simple: Remove that broken general entry from the core_config_data table and you're fine :-)

Import from cli/cronjob

Back in the days Vinai Kopp gave a talk about ImportExport module.

In the talk is a code snippet which helps to run importExport from CLI or as a cronjob. Unfortunately it is a screenshot and I typed the snippet already a couple of times. Time to make sure it is copy pastable:

<?php

require 'html/app/Mage.php';

Mage::app();

/** @var $import Mage_ImportExport_Model_Import */
$import = Mage::getModel('importexport/import');
$import->setEntity(Mage_Catalog_Model_Product::ENTITY);
$file = './import.csv';

if (!$file || !file_exists($file)) {  
    echo 'File does not exist.';
    die();
}

$validationResult = $import->validateSource($file);

if ($import->getProcessedRowsCount() <= 0 || !$validationResult) {  
    printf(
        'File %s contains %s corrupt records (from a total of %s)',
        $file,
        $import->getInvalidRowsCount(),
        $import->getProcessedRowsCount(),
        );
    foreach ($import->getErrors() as $type => $lines) {
        $lines = implode(', ', $lines);
        printf("\n:::: $type :::: \nIn Line(s) $lines \n");
    }
    die();
}

$import->importSource();
$import->invalidateIndex();

INSERT INTO, increased auto_increment, but no data

Missing data, failed transactions and unassigned products

Last week my job was to port a module back from Magento 1.9 to Magento 1.4.

Allowed core hacks

In the end the only real problem I had, was that Mage_Core_Model_Resource_Db_Abstract doesn't exist already, therefore I created the file in core with the content:

<?php
class Mage_Core_Model_Resource_Db_Abstract extends Mage_Core_Model_Mysql4_Abstract {}

If you ask me, this is one of the very rare cases, where a core hack is ok. When Magento is updated, the file is overwrittten and exists then. If you create the file in community or local, this doesn't happen automatically.

Raised auto_increment but missing data - no quote

I copied the module from Magento 1.9 to 1.4 without problems, created the core file and tested it. On my first tests, I didn't get a quote item. Whatever I did, my cart was empty. The bad thing I discovered was, that Magento didn't even create a quote. sales_flat_quote is empty. But the auto_increment value raises by 1. I tried to google it, but googleing for INSERT is ignored or quote is not written didn't help a lot. After one day of searching, I found a MySQL forums post which described the same behaviour. The answer was thankfully there to. What happened?

  1. Transaction is started
  2. Quote is written to the database
  3. Autoincrement is increased
  4. Something fails
  5. Rollback.

Ok. But what fails? In the end I discovered, that the Problem was a killed model rewrite, which leads to an unset quote_id on the quote_payment. I have no clue why this happend. But after deleting the model and setting up the old rewrite this problem was solved.

Quote item in database, but cart empty

I thought I was done. Said the client it is only a matter of minutes, but the next test still fails. The cart is still empty. I started investigating again. The quote was created, the item written - and then deleted.

I learned, that when a collection is loaded all products are assigned to the quote items. If the product doesn't exist, the quote item is deleted. So far so understandable.

\Mage_Sales_Model_Resource_Quote_Item_Collection::_afterLoad
\Mage_Sales_Model_Resource_Quote_Item_Collection::_assignProducts

$product = $productCollection->getItemById($item->getProductId());
if ($product) {
    // [...]
} else {
    $item->isDeleted(true);
    $recollectQuote = true;
}

Long story short: The collection is loaded from the flat tables (if activated). My product was missing there.

$productCollection = Mage::getModel('catalog/product')->getCollection()
    ->setStoreId($this->getStoreId())
    ->addIdFilter($this->_productIds)
    ->addAttributeToSelect(Mage::getSingleton('sales/quote_config')->getProductAttributes())
    ->addOptionsToResult()
    ->addStoreFilter()
    ->addUrlRewrite()
    ->addTierPriceData();

Unfortunately I forgot to assign the product to the desired website. We don't test if the product is available on the website, so no error is thrown on adding it to the cart and without happening anything the item is deleted, before anyone even sees it.

Hope that helps someone finding such weird and hard to trace bugs.