Monitor huge mysql imports

I tweeted today about, how to get a progress bar for commands on the bash, in my case: mysql imports of huge files:

pv magento.sql | mysql -uroot -p database

This tipp is proudly presented by Rob

INSERT INTO, increased auto_increment, but no data

Missing data, failed transactions and unassigned products

Last week my job was to port a module back from Magento 1.9 to Magento 1.4.

Allowed core hacks

In the end the only real problem I had, was that Mage_Core_Model_Resource_Db_Abstract doesn't exist already, therefore I created the file in core with the content:

<?php
class Mage_Core_Model_Resource_Db_Abstract extends Mage_Core_Model_Mysql4_Abstract {}

If you ask me, this is one of the very rare cases, where a core hack is ok. When Magento is updated, the file is overwrittten and exists then. If you create the file in community or local, this doesn't happen automatically.

Raised auto_increment but missing data - no quote

I copied the module from Magento 1.9 to 1.4 without problems, created the core file and tested it. On my first tests, I didn't get a quote item. Whatever I did, my cart was empty. The bad thing I discovered was, that Magento didn't even create a quote. sales_flat_quote is empty. But the auto_increment value raises by 1. I tried to google it, but googleing for INSERT is ignored or quote is not written didn't help a lot. After one day of searching, I found a MySQL forums post which described the same behaviour. The answer was thankfully there to. What happened?

  1. Transaction is started
  2. Quote is written to the database
  3. Autoincrement is increased
  4. Something fails
  5. Rollback.

Ok. But what fails? In the end I discovered, that the Problem was a killed model rewrite, which leads to an unset quote_id on the quote_payment. I have no clue why this happend. But after deleting the model and setting up the old rewrite this problem was solved.

Quote item in database, but cart empty

I thought I was done. Said the client it is only a matter of minutes, but the next test still fails. The cart is still empty. I started investigating again. The quote was created, the item written - and then deleted.

I learned, that when a collection is loaded all products are assigned to the quote items. If the product doesn't exist, the quote item is deleted. So far so understandable.

\Mage_Sales_Model_Resource_Quote_Item_Collection::_afterLoad
\Mage_Sales_Model_Resource_Quote_Item_Collection::_assignProducts

$product = $productCollection->getItemById($item->getProductId());
if ($product) {
    // [...]
} else {
    $item->isDeleted(true);
    $recollectQuote = true;
}

Long story short: The collection is loaded from the flat tables (if activated). My product was missing there.

$productCollection = Mage::getModel('catalog/product')->getCollection()
    ->setStoreId($this->getStoreId())
    ->addIdFilter($this->_productIds)
    ->addAttributeToSelect(Mage::getSingleton('sales/quote_config')->getProductAttributes())
    ->addOptionsToResult()
    ->addStoreFilter()
    ->addUrlRewrite()
    ->addTierPriceData();

Unfortunately I forgot to assign the product to the desired website. We don't test if the product is available on the website, so no error is thrown on adding it to the cart and without happening anything the item is deleted, before anyone even sees it.

Hope that helps someone finding such weird and hard to trace bugs.