Magento 1.8 CE Import/Export from Shell on Bitnami

I’m running a Magento Commerce Community Edition store with 17k + skus. Previously, I only had 3k skus so I could use the build in Data Flow Profiles to import and export with ease. However, when I added the extra 14k skus, the web interface would hang. Furthermore, it would require me to leave the browser window open ”

Warning: Please do not close the window during importing/exporting data

” which would preclude me from being able to start the processes at the end of the day and then head home with my laptop.

The following information was sourced from http://stackoverflow.com/questions/11856841/magento-possible-to-run-export-profile-from-shell-ssh and is a way to run your Magento Import or Export process from the shell.

Note: This does NOT require PHP cli. For the purposes of this process I point to my php executable.

<?php

/***********************
 * Import/Export Script to run Import/Export profile
 * from command line or cron. Cleans entries from dataflow_batch_(import|export) table
 ***********************/


$mageconf = './app/etc/local.xml';  // Mage local.xml config
$mageapp  = './app/Mage.php';       // Mage app location
$logfile  = 'export_data.log';      // Import/Export log file

/* uncomment following block when moved to server - to ensure this page is
 * not accessed from anywhere else
 */


//if ($_SERVER['REMOTE_ADDR'] !== '<your server ip address>') {
//   die("You are not a cron job!");
//}


/* System -> Import/Export -> Profiles get profile ID from
 * Magento Import/Export Profiles
 */


$profileId = 9;

/* Post run housekeeping table bloat removal
 * imports use "dataflow_batch_import" table
 * exports use "dataflow_batch_export" table
 */


$table = 'dataflow_batch_export';

/* Scan Magento local.xml file for connection information */

if (file_exists($mageconf)) {

$xml = simplexml_load_file($mageconf, NULL, LIBXML_NOCDATA);

$db['host'] = $xml->global->resources->default_setup->connection->host;
$db['name'] = $xml->global->resources->default_setup->connection->dbname;
$db['user'] = $xml->global->resources->default_setup->connection->username;
$db['pass'] = $xml->global->resources->default_setup->connection->password;
$db['pref'] = $xml->global->resources->db->table_prefix;

}

else {
    Mage::log('Export script failed to open Mage local.xml', null, $logfile);
    exit('Failed to open Mage local.xml');
}


/* Initialize profile to be run as Magento Admin and log start of export */

require_once $mageapp;
umask(0);
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);

$profile = Mage::getModel('dataflow/profile');
$userModel = Mage::getModel('admin/user');
$userModel->setUserId(0);
Mage::getSingleton('admin/session')->setUser($userModel);
$profile->load($profileId);
if (!$profile->getId()) {
    Mage::getSingleton('adminhtml/session')->addError('ERROR: Incorrect profile id');
}

Mage::log('Export ' . $profileId . ' Started.', null, $logfile);

Mage::register('current_convert_profile', $profile);
$profile->run();
$recordCount = 0;
$batchModel = Mage::getSingleton('dataflow/batch');

Mage::log('Export '.$profileId.' Complete. BatchID: '.$batchModel->getId(), null, $logfile);

echo "Export Complete. BatchID: " . $batchModel->getId() . "\n";

/* Connect to Magento database */

sleep(30);

mysql_connect($db['host'], $db['user'], $db['pass']) or die(mysql_error());
mysql_select_db($db['name']) or die(mysql_error());

/* Truncate dataflow_batch_(import|export) table for housecleaning */

$querystring = "TRUNCATE ".$db['pref'].$table;

mysql_query($querystring) or die(mysql_error());

?>

On my initial run, I recieved a “PHP Fatal Error: Allowed memory size of 1034 bytes exhausted” error. So I added the following line to the top of my script:

ini_set("memory_limit","2048M");

I relize this a very large size, but in my case it is my development server that is on a VirtualBox VM running Ubuntu 12.04LTS and Bitnami’s Lamp stack with Magento 1.8 CE. I have 10GB of ram allocated to the machine, so it’s no big deal to set a large memory limit.

This is my actual shell command is run from my php/bin path.

./php /opt/bitnami/apps/magento/htdocs/var/export/export-everything.php