How to split a list into equally-sized chunks in Python
Batching a list comes up constantly — API calls that take 100 IDs at a time, database inserts, rate limits. Since Python 3.12 the standard library has a function for it, and most of the answers you will find online predate it.
The answer, on Python 3.12 and newer
from itertools import batched
for batch in batched(data, 3):
...
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
list(batched(data, 3)) : [(1, 2, 3), (4, 5, 6), (7, 8, 9), (10,)]
itertools.batched
yields tuples, and the last one is short if the input does not divide
evenly. If you want lists, ask:
as lists : [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
It is lazy, so nothing beyond the batch you asked for has been computed:
type(batched(...)) : batched
first batch only : (1, 2, 3)
Python 3.13 added strict=, which refuses a short final batch instead of
returning it:
batched(data, 3, strict=True) -> ValueError: batched(): incomplete batch
batched(data, 5, strict=True) : [(1, 2, 3, 4, 5), (6, 7, 8, 9, 10)]
Useful when a short batch would mean your input was malformed.
On older Pythons: slicing
def chunks(seq, n):
for i in range(0, len(seq), n):
yield seq[i:i + n]
list(chunks(data, 3)) : [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
Two details make this shorter than you would expect. range with a step gives
exactly the start indices you need:
range(0, 10, 3) -> [0, 3, 6, 9]
And slices clamp rather than raising, so the tail needs no special case:
data[9:12] : [10] <- no IndexError, slices clamp
This version also keeps the type of the sequence. On a string you get strings
back, where batched would give you tuples of characters.
The catch is len(): this only works on something sliceable, so not on a
generator.
The old recipe gives a different answer
You will find this one in the pre-3.12 answers, from the itertools docs:
def grouper(iterable, n, fillvalue=None):
args = [iter(iterable)] * n
return zip_longest(*args, fillvalue=fillvalue)
grouper(data, 3) : [(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, None, None)]
grouper(data, 3, fillvalue=0) : [(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 0, 0)]
Look at the last group. It is padded to full length, where batched left
it short. That is a different result, not a different spelling — and if you
swap one for the other while modernising old code, you will start feeding two
None values into whatever consumes the batches.
The [iter(iterable)] * n trick is worth understanding, since it looks like it
should not work:
[iter(x)] * 2 makes two references to ONE iterator: True
Multiplying a list repeats the reference, so all n entries are the same
iterator. zip then pulls from it in turn, and each round of zip therefore
takes the next n items.
When you have a generator
len() is not available, so the slicing version is out:
len(a generator) -> TypeError: object of type 'generator' has no len()
batched handles it, because it only ever pulls forwards:
batched works anyway : [[1, 2, 3], [4, 5, 6], [7]]
Before 3.12, the equivalent uses
islice:
def chunks_islice(iterable, n):
it = iter(iterable)
while batch := list(islice(it, n)):
yield batch
islice version : [[1, 2, 3], [4, 5, 6], [7]]
Note that it = iter(iterable) outside the loop is load-bearing — a fresh
iterator each time round would return the first n items forever.
n parts, rather than parts of n
A different question that gets asked in the same thread. Splitting into a fixed number of pieces:
def into_n_parts(seq, parts):
k, m = divmod(len(seq), parts)
return [seq[i * k + min(i, m):(i + 1) * k + min(i + 1, m)] for i in range(parts)]
10 items into 3 parts : [[1, 2, 3, 4], [5, 6, 7], [8, 9, 10]]
10 items into 4 parts : [[1, 2, 3], [4, 5, 6], [7, 8], [9, 10]]
Sizes differ by at most one, and the larger pieces come first. This is what
numpy.array_split does, without needing numpy.
Edge cases
batched([], 3) : []
batched(data, 20) : [(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)]
batched(data, 0) -> ValueError: n must be at least one
Empty input gives no batches rather than one empty batch, and a chunk size larger than the input gives one short batch. Both are what you want; neither is obvious enough to guess.
About Netcup (advertisement)
The German host Netcup offers, among other things, affordable and powerful web hosting packages, KVM-based root servers and dedicated servers. With our voucher codes you can save even more (6€ off your first order, 30% off all KVM-based root servers, ...).