Introduction to Python Generators

Generators make it easy to create iterations in Python and in return write less code. This tutorial will introduce you to Python generators, their benefits, and how they work.

Basics

A generator is a function that returns a generator object on which you can call the next() method, so that for every call it returns a value or the next value. A normal Python function uses the return keyword to return values, but generators use the keyword yield to return values. This means that any Python function containing a yield statement is a generator function.

The yield statement usually halts the function and saves the local state so that it can be resumed right where it left off. Generator functions can have one or more yield statements.

A generator is also an iterator, but what is an iterator? Before we dive into the details of generators, I think it’s important to know what iterators are because they form an integral part of this discussion.

Python Iterators

A Python iterator is simply a class that defines an __iter__() method. Most Python objects are iterable, which means you can loop over each and every element in the objects. Examples of iterables in Python include strings, lists, tuples, dictionaries, and ranges.

Let’s consider the example below, in which we are looping over a list of colors:

colors= [“red”,”blue”,”yellow”]

def my_funct():
    for color in colors:
        print color

Behind the scenes, the for statement will call iter() on the list object. The function will then return an iterator object that defines the method __next__(), which will then access each color, one at a time. When there are no more colors left, __next__ will raise a stopIteration exception, which will in turn inform the for loop to terminate.

Iterating Over a Dictionary

d = {'x': 10, 'y': 20, 'z': 30}
for k,v in d.items():
    print k, v

#result
# y 20
# x 10
# z 30

Iterating Over Rows in a CSV File

import csv

with open('file.csv', newline='') as File:  
    reader = csv.reader(File)
    for row in reader:
        yield row

Iterating Over a String

my_string = 'Generators'
for string in my_string:
    print (string)
    
#result

# G
# e
# n
# e
# r
# a
# t
# o
# r
# s

Benefits of Using Generators

Let’s discuss some of the benefits of using generators as opposed to iterators:

Easy to Implement

Building an iterator in Python will require you to implement a class with __iter__() and __next__() methods as well as taking care of any errors that may cause a stopIteration error.

class Reverse:
    """Iterator for looping over a sequence backwards."""
    def __init__(self, data):
        self.data = data
        self.index = len(data)

    def __iter__(self):
        return self

    def __next__(self):
        if self.index == 0:
            raise StopIteration
        self.index = self.index - 1
        return self.data[self.index]

As you can see above, the implementation is very lengthy. All this burden is automatically handled by generators.

Less Memory Consumption

Generators help to minimize memory consumption, especially when dealing with large data sets, because a generator will only return one item at a time.

Better Performance and Optimisation

Generators are lazy in nature. This means that they only generate values when required to do so. Unlike a normal iterator, where all values are generated regardless of whether they will be used or not, generators only generate the values needed. This will, in turn, lead to your program performing faster.

How to Create a Generator in Python

Creating a generator is very easy. All you need to do is write a normal function, but with a yield statement instead of a return statement, as shown below.

def gen_function():
    yield "python"

While a return statement terminates a function entirely, yield just pauses the function until it is called again by the next() method.

For example, the program below makes use of both the yield and next() statements.

def myGenerator(l):  
     total = 1
     for n in l:
       yield total
       total += n
     
newGenerator = myGenerator([10,3])

print(next(newGenerator))  
print(next(newGenerator))  

  

How Python Generators Work

Let’s  see how generators work. Consider the example below.

# generator_example.py

def myGenerator(l):  
     total = 0
     for n in l:
       total += n
       yield total
      
     
newGenerator = myGenerator([10,20,30])

print(next(newGenerator))  
print(next(newGenerator))  
print(next(newGenerator))  
  

In the function above, we define a generator named myGenerator, which takes a list l as an argument. We then define a variable total and assign to it a value of zero. In addition, we loop through each element in the list and subsequently add it to the total variable.

We then instantiate newGenerator and call the next() method on it. This will run the code until it yields the first value of total, which will be 0 in this case. The function then keeps the value of the total variable until the next time the function is called. Unlike a normal return statement, which will return all the values at once, the generator will pick up from where it left off.

Below are the remaining subsequent values.

# generator_example.py

def myGenerator(l):  
     total = 0
     for n in l:
       yield total
       total += n
      
     
newGenerator = myGenerator([10,20,30])

print(next(newGenerator))  
print(next(newGenerator))  
print(next(newGenerator))  
  
# result

# 0
# 10
# 30

If you try to call the function after it has completed the loop, you will get a StopIteration error.

A StopIteration is raised by the next() method to signal that there are no further items produced by the iterator.

0
10
30

Traceback (most recent call last):
  File "python", line 15, in 
StopIterationNormal function

Example 2

In this example, we show how to use multiple yield statements in a function.

# colors.py

def colors():
  yield "red"
  yield "blue"
  yield "green"
  
next_color =colors()
   
print(next(next_color))
print(next(next_color))
print(next(next_color))

# result

# red
# blue
# green

Whereas a normal function returns all the values when the function is a called, a generator waits until the next() method is called again. Once next() is called, the colors function resumes from where it had stopped.

Conclusion

Generators are more memory efficient, especially when working with very large lists or big objects. This is because you can use yields to work on smaller bits rather than having the whole data in memory all at once.

Additionally, don’t forget to see what we have available for sale and for study on Envato Market, and don’t hesitate to ask any questions and provide your valuable feedback using the feed below.

Furthermore, if you feel stuck, there is a very good course on Python generators in the course section.