Medieval Programming

Eric made a valid point in his comment that you could remove all includes due to symfony autoloading.

I found the sfBuilders that are responsible for stripping the comments from the generated propel classes and also saw that there is a addIncludes parameter in propel.ini. Lets reuse that and modify the Builders to strip the inline includes and requires.

I admit this is a tiny step, but some propel users are desperately searching for performance tweaks (as I am as well) so I hope this could be of use. I send this patch for inclusion in symfony 1.1, Noël told me that it will not become part of 1.0.x .

So if you want to patch your symfony 1.0.x yourself. here is the diff: propel_includes_10.patch

Enjoy

Carsten came today to me and told me that he had an exciting finding:

$entity =  $myObject->getEntity();
echo($entity);
echo($entity);

is faster than:

echo($myObject->getEntity());
echo($myObject->getEntity());

I couldn’t believe that but it was the truth. When I dug deeper in propel i found this line generated [reformatted]:

public function getEntity($con = null) {
  include_once 'lib/model/om/BaseEntityPeer.php';
  if ($this->aEntity === null && ($this->entity_gid !== null)) {
    $this->aEntity = EntityPeer::retrieveByPK($this->entity_gid, $con);
  }
  return $this->aEntity;
}

so, what is the magic here? as you might know, include_once comes with a performance hit. and here you will get this performance hit each and every time you call the getter. Isn’t that annoying?

I found the guilty propel builder and did a small patch which places the include inside the if (where it is actually needed) and the performance hit is gone. Neat, isn’t it?

The patch can be found here: PHP5ComplexObjectBuilder.patch

Enjoy!

this might be a complaint, or a rant, or something like that.

For some reasons there is about 200 megabytes of data I would like to import into the new web application I am writing.

I thought doing so is easy with propel. Well it actually is, wouldn’t PHP have some strange issues. Perhaps they are propel issues but I guess PHP is also to blame.

I do a mysql_query against my old database and initialize a new propel connection for the new one. I loop over all rows for that table. 63000 rows. it failed with out of memory some times, until I found a memory setting that would allow that loop to complete. hold tight: it is 1GB. The data processed inside this loop weighs 50MB.

I am not really doing complex things. I new my Propel objects, invoke a few setters and then save() them. Using the same connection for all inserts brings already a lot of performance. but using the Propel Objects leaks memory each loop. But more interesting, If I just loop and comment all the propel stuff I also leak memory. or more precise, PHP does. Why does an empty loop over mysql data leak memory? is there an internal loop status object that keeps track of each iteration?

Okay the bulk of the memory leak goes credit to propel. It seems to be that the crossreferenced table-, database- and column maps never free the memory. The PHP garbage collector is unable to collect cyclic island references, so they stay in memory. There are some tickets open in propel trac, so perhaps this will get improved. The only thing I could do now is to explicitly unset all variables I am using myself to limit memory usage and to give PHP that gigabyte for the import. This will be a one time operation, but I wonder if there might be a better way to reduce memory consumption. And yes I would like to keep using the Propel objects.

Here some leaking sample code. The style is not perfect. its just a quick snippet. not that I am using the basePeer::doInsert because I needed here to preserve the ID (which is removed by the save() method); also its a bit faster.

mysql_connect("legacy-db", "olduser", "oldpw");
$data = mysql_query("select oid,title,text from db.table");
$max = mysql_num_rows($data);
 
$databaseManager = new sfDatabaseManager();
$databaseManager->initialize();
$con = Propel::getConnection(EntryPeer::DATABASE_NAME);
try {
  $con->begin();
  $i=0;
  while($row=mysql_fetch_row($data)){
    $i++;
    if ($i % 100 == 0) echo ($i."/".$max."\n");
    $e = new Entry();
    $e->setId($row[0]);
    $e->setTitle(utf8_encode($row[1]));
    $e->setBody(utf8_encode($row[2]));
    $crit=$e->buildCriteria();
    $crit->setDbName(EntryPeer::DATABASE_NAME);
    BasePeer::doInsert($crit, $con);
    unset($e);
    unset($crit);
    unset($row);
  }
  $con->commit();
} catch(PropelException $e) {
  $con->rollback();
  throw $e;
}
mysql_free_result($data);