PHP: What to do about "Allowed memory size exhausted"?

Fatal error: Allowed memory size of 8388608 bytes exhausted
(tried to allocate 12288 bytes)

The first answer you find for this is: raise the limit. Sometimes that is even right. Usually, though, it is a reaction to a file being loaded into memory in one piece when it could have been read line by line — and the difference is a factor of a thousand.

First of all: where do I stand?

Before changing anything, it is worth looking at the numbers:

<?php
  echo ini_get('memory_limit'), "\n";
  echo memory_get_usage(), "\n";
  echo memory_get_usage(true), "\n";
  echo memory_get_peak_usage(true), "\n";
?>
memory_limit           : 512M
memory_get_usage()     : 0.46 MiB
memory_get_usage(true) : 2.00 MiB

The difference between the two needs explaining. Without a parameter you get what PHP is using for your variables. With true you get what PHP has requested from the operating system — and that is the figure the limit is measured against. So for the question “how close am I” true is the more honest one.

memory_get_peak_usage() stays at the high-water mark, even once the memory has long been freed:

peak so far : 6.28 MiB
current     : 0.46 MiB

That is the value you want to write to your log at the end of a script. Because when your script crashes, it was the peak that broke the limit — not the usage at the end.

The one point that usually decides it

The same file, 11.63 MiB, 200,000 lines. Three ways to read it:

file_get_contents        11.64 MiB
file() as an array       22.31 MiB
fgets() line by line      0.01 MiB

The third line is not a typo. Ten kilobytes.

file_get_contents() puts the whole file down as one string, so about the file size as expected. file() needs nearly double, because each of the 200,000 lines becomes its own string with its own bookkeeping. And the streaming route needs practically nothing, because only a single line is in memory at any one time.

<?php
  $fh = fopen($file, 'r');
  while (($line = fgets($fh)) !== false) {
    // process one line
  }
  fclose($fh);
?>

With a 12 MiB file that still makes no difference. With a 2 GiB log file it decides whether your script runs at all — and there no memory_limit = -1 helps either, because the server simply does not have the memory.

The same principle applies elsewhere:

  • Database queries: not fetchAll(), but fetch() in a loop.
  • CSV: fgetcsv() rather than reading the file and splitting it yourself.
  • JSON: a streaming parser for large documents, because json_decode() builds the complete document in memory.
  • Images: GD keeps an image uncompressed in memory, four bytes per pixel. A 4000×3000 photo is maybe 2 MiB as a JPEG — measured, it occupies 46.91 MiB in GD, matching the calculation 4000 × 3000 × 4 bytes. With a default limit of 128 MiB, three such images at once are already too many.

What an array really costs

So that the numbers are less of a surprise:

range(1, 100000)       : 2.00 MiB   -> 21.0 bytes per integer
100,000 short strings  : 5.82 MiB

An integer on its own occupies 8 bytes. In an array it costs 21, because PHP maintains hash table entries and type information for every element.

That explains why 50,000 database rows with ten columns each need so much more memory than the sum of the field contents would suggest.

Copy on write

An effect that makes tracking down the cause harder:

$copy = $big            : 0.00 MiB   <- no copy yet
after one modification  : 4.00 MiB   <- now it was copied

The assignment costs nothing. PHP merely notes that two variables point at the same data. Only when you modify one of them does the copy actually happen.

For debugging that means: memory does not blow up on the line with the assignment, but somewhere further down, on a harmless-looking modification. Anyone who does not know this looks in the wrong place.

Catching the error — you cannot, but you can read it

The obvious attempt:

<?php
  try {
    // use a lot of memory
  } catch (Throwable $e) {
    echo 'caught';   // never reached
  }
?>

Does not work. Exceeding the limit is a fatal error and not an Error object, so there is nothing to catch. The catch branch is not entered, the script simply ends at that point.

What does work:

<?php
  register_shutdown_function(function (): void {
    $f = error_get_last();
    if ($f !== null && $f['type'] === E_ERROR) {
      // log here, clean up, serve an error page
      error_log($f['message'] . ' in ' . $f['file'] . ':' . $f['line']);
    }
  });
?>
[shutdown] Allowed memory size of 8388608 bytes exhausted (tried to allocate 12288 bytes)
[shutdown] in sprengen.php line 16

The function still runs after a fatal error. So you do get the file and the line — and that is exactly the information you need. You just cannot let the script carry on.

The process’s exit code, incidentally, is 255, the same as for any other fatal error.

And when do you raise the limit after all?

There are legitimate cases. An import script that runs once a month and really does have to hold a large data set in memory may have more than a web page:

<?php
  ini_set('memory_limit', '512M');
?>
before : 512M -> after: 256M -> again : 64M

That works at runtime, in both directions. Two caveats: it does not help with a parse error, because the line is never executed (more on that in the article about error messages), and some hosts lock the directive down.

What I do not recommend is memory_limit = -1 on a web server. The limit is not there to annoy you, it is there so that a single mistake — an infinite loop, a query without a LIMIT — does not take the whole server with it. Without a limit you get, instead of a fatal error in one process, a server that starts swapping or gets killed by the OOM killer.

For a command line script that you start yourself, -1 is perfectly fine.

Summary

  • Measure first: log memory_get_peak_usage(true) at the end of the script.
  • Read files line by line rather than in one piece — 0.01 MiB instead of 11.64 MiB in the test.
  • file() needs double the file size.
  • An integer in an array costs 21 bytes, not 8.
  • Copy on write: memory is claimed on modification, not on assignment.
  • The error cannot be caught, but can be read via register_shutdown_function() and error_get_last().
  • memory_limit = -1 belongs on the command line, not on a web server.

About Netcup (advertisement)

The German host Netcup offers, among other things, affordable and powerful web hosting packages, KVM-based root servers and dedicated servers. With our voucher codes you can save even more (6€ off your first order, 30% off all KVM-based root servers, ...).