How to iterate over rows in a Pandas DataFrame
pandas is the standard library for tabular data in Python. It offers several ways to walk a DataFrame row by row, and the usual advice is not to. This article shows the ways, what each one costs, and the two behaviours that surprise people.
The three ways
for idx, row in df.iterrows(): # a Series per row
for row in df.itertuples(): # a namedtuple per row
for name, qty in zip(df["name"], df["qty"]): # just the columns you need
iterrows : index=0 name=ada qty=2
itertuples : index=0 name=ada qty=2
zip : name=ada qty=2
All three give the same values. What they cost is not the same at all.
What it costs
20,000 rows, adding two columns together, per single pass:
vectorised df['a'] + df['b'] : 0.044 ms
itertuples list comprehension: 7.604 ms
apply(axis=1) : 64.885 ms
iterrows : 211.528 ms
iterrows is about 4800 times the vectorised version. itertuples is
about 170 times. apply(axis=1) sits between them and is not the optimisation
people often assume — it is a loop with extra machinery.
The reason is that the vectorised operation runs one typed loop in C over contiguous memory. Every row-wise form builds a Python object per row.
Why iterrows is the slowest
Each row comes back as a Series — a full pandas object with an index, built
fresh 20,000 times. Worse, it flattens your types:
df.dtypes : {'name': 'str', 'qty': 'int64', 'price': 'float64'}
a row from iterrows is a Series with dtype object
A row spanning columns of different dtypes has no single dtype, so pandas
upcasts to object. Your int64 becomes a Python int, and any float
precision assumptions go with it.
itertuples keeps them:
itertuples keeps them : qty is int, price is float
So if you must loop, loop with itertuples. It is faster, it preserves dtypes,
and the attribute access reads better than row["column"].
Writing to the row does nothing
This is the one that costs people an afternoon:
for idx, row in df.iterrows():
row["qty"] = 999
after assigning to row['qty'] in iterrows : [2, 3]
The DataFrame is unchanged. The row you were handed is a copy, so you modified something that was discarded at the end of the iteration. No error, no warning.
If you want to write, address the DataFrame:
df.loc[:, "qty"] = 999
df.loc[:, 'qty'] = 999 : [999, 999]
itertuples renames awkward columns
Namedtuple fields have to be valid Python identifiers, so anything that is not gets replaced positionally:
columns : ['my col', 'class', '3rd']
as a tuple : Pandas(Index=0, _1=1, _2=2, _3=3)
A space, a keyword and a leading digit — all three became _1, _2, _3. If
your column names come from a CSV somebody else produced, this will happen.
itertuples(name=None) : (0, 1, 2, 3) <- a plain tuple, no renaming
name=None gives plain tuples, which you then index positionally.
The error you will hit on the way
Putting a Series in an if:
bool(df['qty'] > 1) -> ValueError: The truth value of a Series is ambiguous.
Use a.empty, a.bool(), a.item(), a.any() or a.all().
df['qty'] > 1 is a Series of booleans, one per row, and pandas refuses to
guess whether you meant “any of them” or “all of them”. The message lists your
options.
What to do instead
Most row loops are one of two things. Arithmetic across columns:
df["total"] = df["qty"] * df["price"]
[3.0, 6.0]
Or a conditional column, which is
numpy.where:
df["band"] = np.where(df["qty"] > 2, "high", "low")
['low', 'high']
For more branches there is np.select, and for genuinely row-dependent logic
that will not vectorise, itertuples is the honest fallback. Just reach for it
knowing the price, rather than as a first instinct.
About Netcup (advertisement)
The German host Netcup offers, among other things, affordable and powerful web hosting packages, KVM-based root servers and dedicated servers. With our voucher codes you can save even more (6€ off your first order, 30% off all KVM-based root servers, ...).