How to rename columns in a Pandas DataFrame
Renaming columns is usually the first thing you do after reading somebody else’s CSV. There are two main ways, they behave differently when you get it wrong, and one of them fails without telling you.
Renaming some of them
df = df.rename(columns={"a": "alpha"})
before : ['a', 'b', 'c']
after : ['alpha', 'b', 'c']
original untouched : ['a', 'b', 'c'] <- rename returns a copy
rename
takes a mapping of old name to new name, leaves everything else alone, and
returns a new DataFrame.
Renaming all of them
df.columns = ["x", "y", "z"]
['x', 'y', 'z']
This one is positional and modifies the DataFrame directly. It insists on the right number:
wrong length -> ValueError: Length mismatch: Expected axis has 3 elements, new values have 2 elements
Which is a useful safety net, and the opposite of how rename behaves.
The silent failure
Here is the thing worth knowing about rename:
df.rename(columns={"typo": "alpha"})
['a', 'b', 'c']
Nothing happened. No error, no warning, no change. There is no column called
typo, so the mapping matched nothing and rename cheerfully returned a copy
of what you gave it.
That is fine for the intended use — you can pass a mapping covering columns
that may or may not be present — but it means a misspelled source name is
invisible. In a pipeline, the failure surfaces later as a KeyError on a
column you were sure you had renamed.
Ask for the check:
df.rename(columns={"typo": "alpha"}, errors="raise")
-> KeyError: "['typo'] not found in axis"
I would use errors="raise" by default and drop it only where a partial
mapping is deliberate.
inplace, and the None it hands back
result = df.rename(columns={"a": "alpha"}, inplace=True)
return value of inplace=True : None
df.columns after : ['alpha', 'b', 'c']
The rename happened, and the return value is None. So this common shape:
df = df.rename(columns={...}, inplace=True) # df is now None
replaces your DataFrame with None. Either use inplace=True as a statement,
or drop it and assign the result. Not both.
Note that pandas has been moving away from inplace generally — it rarely
saves memory and it breaks method chaining, so the assignment form is the one
to prefer.
Cleaning up messy names
rename accepts a callable, which is applied to every column:
df.rename(columns=str.lower)
before : ['First Name', 'Last Name']
after : ['first name', 'last name']
For a real tidy-up, a lambda:
df.rename(columns=lambda c: "_".join(c.lower().split()))
lower + collapse spaces : ['first_name', 'last_name']
.split() with no argument splits on runs of whitespace, so the double space
in Last Name collapses rather than becoming a double underscore.
The same thing through the index, which is often easier to read for a chain of string operations:
df.columns = df.columns.str.lower().str.replace(r"\s+", "_", regex=True)
['first_name', 'last_name']
Duplicates are allowed, and change what indexing returns
pandas will not stop you renaming one column to the name of another:
after renaming 'a' to 'b' : ['b', 'b', 'c']
Two columns called b. No warning. And now:
df['b'] now returns DataFrame with shape (1, 2)
df["b"] used to give you a Series and now gives you a DataFrame. Every piece
of code downstream that expected a Series — .str, .mean(), comparison
against a scalar — behaves differently or breaks.
If you are renaming from a mapping built at runtime, it is worth checking
df.columns.duplicated().any() afterwards.
The rest of the family
set_axis(['x','y','z'], axis=1) : ['x', 'y', 'z']
add_prefix('col_') : ['col_a', 'col_b', 'col_c']
add_suffix('_raw') : ['a_raw', 'b_raw', 'c_raw']
set_axis is the chainable version of assigning to .columns, which matters
if you are building a pipeline as one expression. add_prefix and
add_suffix are useful before a merge, to keep two sets of similarly-named
columns apart.
And rename renames the index in exactly the same way:
df.rename(index={'r1': 'first'}) : ['first', 'r2']
axis=1 is an alternative spelling of columns=, if you prefer symmetry with
the rest of the pandas API.
About Netcup (advertisement)
The German host Netcup offers, among other things, affordable and powerful web hosting packages, KVM-based root servers and dedicated servers. With our voucher codes you can save even more (6€ off your first order, 30% off all KVM-based root servers, ...).