What the yield keyword does in Python
Python is a high level programming language with a large standard library. In
this article you will learn what the yield keyword does, why a function that
contains it behaves so differently from a normal one, and where the memory
savings people talk about actually come from.
The short version
A function that contains yield does not return a value. It returns a
generator, and the
body of the function does not run at all until you ask the generator for
something.
def counter():
print("...body starts running only now")
yield 1
yield 2
g = counter() # nothing printed
print(next(g)) # now the body starts, prints, then gives you 1
Running it:
$ python counter.py
calling counter() gave : generator
nothing printed yet -- now the first next():
...body starts running only now
next(g) -> 1
next(g) -> 2
next(g) -> StopIteration
That is the whole idea. yield hands a value back to the caller and freezes
the function where it stands. The next next() thaws it out again and carries
on from the line after the yield.
The generator is used up afterwards
This is the part that costs people an afternoon:
first list(g) : [1, 2]
second list(g) : []
The second pass is empty. Not an error, no warning — just nothing. A generator walks forwards once and that is it. If you need the values twice, either build a list from them or call the generator function again to get a fresh generator.
A list, by contrast, can be iterated as often as you like:
a list by contrast, twice : [0, 1, 4, 9, 16] / [0, 1, 4, 9, 16]
Where the memory goes
The usual claim is that generators save memory. Here is what that is worth,
measured with tracemalloc:
list of 1,000,000 squares : 38.57 MiB
the same as a generator : 0.00 MiB
sys.getsizeof(generator) : 208 bytes, whatever n is
The 208 bytes is the generator object itself, and it does not grow with n.
Whether you are about to produce ten items or ten billion, the object costs the
same, because it holds a paused stack frame and nothing else.
The same comparison for a modest 1000 items: the list comprehension is 8856 bytes, the generator expression 208.
Note that this only pays off if you consume the values one at a time. The
moment you write list(gen_squares(1_000_000)) you are back to 38.57 MiB — you
have just taken a detour to get there.
It really is lazy
Worth seeing rather than believing:
next() one at a time:
computing 0
got 0
(nothing more computed until we ask)
Only the first value was computed. This is what makes generators useful for reading a large file, or for a sequence that has no end. And it is why a generator can be faster even when memory is not a concern — if you break out of the loop after ten items, only ten items were ever computed.
yield remembers where it was
Every local variable in the function survives between calls, because the frame is paused rather than discarded:
def running_total():
total = 0
while True:
value = yield total
if value is None:
return
total += value
priming with next() : 0
r.send(10) : 10
r.send(5) : 15
r.send(1) : 16
send()
passes a value into the generator, where it becomes the result of the yield
expression. Note the priming: a fresh generator has not reached its first
yield yet, so you have to next() it once before the first send().
This is the mechanism that async/await was originally built on, and reading
it here is a good way to get a feel for how coroutines work underneath.
The mistake worth knowing about
Suppose you want a function to hand back three values. This does not do what it looks like:
def with_yield():
yield [1, 2, 3]
list(with_yield()) : [[1, 2, 3]] <- a list inside a list
yield somelist yields the list as one item. To yield its elements
individually you want
yield from,
introduced by PEP 380:
def with_yield_from():
yield from [1, 2, 3]
list(with_yield_from()) : [1, 2, 3]
yield from also delegates to another generator, which saves writing the
for item in inner(): yield item loop by hand:
manual loop : ['a', 'b']
yield from : ['a', 'b', 'c', 'd']
Things a generator cannot do
Because there is no sequence sitting in memory, there is nothing to measure or index:
len(g) -> TypeError: object of type 'generator' has no len()
g[0] -> TypeError: 'generator' object is not subscriptable
If you need either, you need a list, and you should ask yourself whether the
generator is buying you anything at that point.
itertools.islice
is the way to take a slice without materialising the whole thing.
The cheap version: generator expressions
You do not need a function for the simple cases. Swap the brackets of a list comprehension for parentheses:
sum(x*x for x in range(5)) : 30
type of (x*x for x in ...) : generator
Since it is the only argument, the parentheses of the call do double duty and you do not need a second pair.
Keep in mind that the same single-pass rule applies here. A generator expression assigned to a name is just as exhausted after one loop as anything else.
About Netcup (advertisement)
The German host Netcup offers, among other things, affordable and powerful web hosting packages, KVM-based root servers and dedicated servers. With our voucher codes you can save even more (6€ off your first order, 30% off all KVM-based root servers, ...).