PHP: How do you parse HTML properly?
You want to pull one piece of information out of an HTML page, and you reach for the first tool that comes to mind: a regular expression. That usually goes wrong. PHP ships with everything you need for this job.
All output comes from a script I ran against PHP 8.5.9 with libxml 2.9.14.
The tool: DOMDocument
You do not need to install anything. The dom, libxml and simplexml
extensions are part of PHP. For HTML you use DOMDocument:
<?php
$dom = new DOMDocument();
$dom->loadHTML($html);
?>
The part many people do not know: loadHTML() copes with broken HTML. Behind
it sits libxml’s HTML parser, which is built for exactly that. Take this
deliberately malformed markup — an unclosed <p>, an attribute without quotes,
uppercase tags, a bare &:
<html><body>
<DIV class=teaser>
<p>First product & accessories
<a href="/a.html">Product A</a>
<p>Second product
<a href='/b.html'>Product B</a>
</DIV>
</body></html>
That loads without trouble.
Do not forget libxml_use_internal_errors
With broken HTML, libxml writes warnings to the log or the output. You usually do
not want that. libxml_use_internal_errors(true) collects the messages instead
of letting them be printed:
<?php
$dom = new DOMDocument();
libxml_use_internal_errors(true);
$dom->loadHTML($html);
$errors = libxml_get_errors(); // in case you do want to look at them
libxml_clear_errors();
?>
For our example above libxml reports exactly one problem
(htmlParseEntityRef: no name, because of the bare &) — and parses on
regardless.
Extracting data with XPath
For searching you use DOMXPath. With it you describe what you want, instead
of describing what the characters around it look like:
<?php
$xpath = new DOMXPath($dom);
foreach ($xpath->query('//div[@class="teaser"]//a') as $a) {
echo $a->textContent . ' -> ' . $a->getAttribute('href') . "\n";
}
?>
Output:
Product A -> /a.html
Product B -> /b.html
You read attributes with getAttribute() and the text with textContent. A
practical example, summing prices from a list:
RS 1000 10.40 EUR
CCX11 23.68 EUR
sum: 34.08 EUR
If you need the markup of a subtree rather than its text, use saveHTML() with a
node as its argument:
<?php
echo $dom->saveHTML($node);
?>
The character encoding trap
And now the bug that can cost you an afternoon. Without a hint about the
encoding, loadHTML() assumes ISO-8859-1. With UTF-8 input you get this:
without charset : Preis in Euro: 10,40 ⬠für Zubehör
There are two ways to fix it. Either the document contains a
<meta charset="utf-8">, in which case libxml detects the encoding itself, or
you prepend an XML declaration to the markup:
<?php
$dom->loadHTML('<?xml encoding="UTF-8">' . $html);
?>
Both give the correct result:
with XML hint : Preis in Euro: 10,40 € für Zubehör
with meta charset : Preis in Euro: 10,40 € für Zubehör
So if your accented characters are being mangled, this is why.
SimpleXML is not an HTML parser
SimpleXML is convenient, but it wants well-formed XML. Pointed at our
broken HTML:
simplexml_load_string($html) => false
For XML — an RSS feed, say — SimpleXML is a good choice. For HTML from the wild it is not.
Why not just use a regular expression?
Because it falls over at the first gust of wind. Take a perfectly reasonable looking expression for finding links:
<?php
preg_match_all('#<a href="([^"]+)">([^<]+)</a>#', $html, $matches);
?>
The result with our example:
regular expression : 1 of 2 links
DOM : 2 of 2 links
The second link uses single instead of double quotes. A single character of difference in the markup, and the expression misses half the results. That is exactly the point: an HTML parser already knows the rules of HTML. A regular expression has to be taught them again every time — and you will always forget a case.
That does not mean regular expressions are forbidden. For very simple extractions from markup whose shape you know, they are fast and perfectly fine. But as soon as nesting, attributes in arbitrary order or somebody else’s markup are involved, use a parser.
What about all those libraries?
On Stack Overflow you will find a long list of libraries for this job. That list has aged, though: several of the projects named there are no longer maintained, others are in security-only mode. And the two best known ones that are not built on libxml are explicitly not recommended in that same answer, because they are slow and memory hungry.
My advice: start with DOMDocument and DOMXPath. It is built in, it is fast,
and you can do everything with it. If you would rather write CSS selectors than
XPath, the
DomCrawler from
the Symfony components is a good and actively maintained addition. It can be used
without the rest of Symfony.
Summary
DOMDocument::loadHTML()handles broken HTML — that is not a myth.libxml_use_internal_errors(true)turns off the shower of warnings.- Use
DOMXPathfor searching. - Without
<meta charset>or<?xml encoding="UTF-8">your accented characters get mangled. SimpleXMLis for XML, not for HTML.- Regular expressions break as soon as the markup changes even slightly.
About Netcup (advertisement)
The German host Netcup offers, among other things, affordable and powerful web hosting packages, KVM-based root servers and dedicated servers. With our voucher codes you can save even more (6€ off your first order, 30% off all KVM-based root servers, ...).