Have you ever had to work with a dataset so large that it overwhelmed your machine’s memory? Or maybe we have a complex function that needs to maintain an internal state every time it’s called, but the function is too small to justify creating its own class. In these cases and more, generators and the Python yield statement are here to help.
Python Generator functions allow us to declare a function that behaves like an iterator, i.e. it can be used in a for loop.
This post will demonstrate how to run another powerful Python feature in Python4Delphi with RAD Studio: Generators in Python GUI and gets the output.
Prerequisites: Download and install the latest Python for your platform. Follow the Python4Delphi installation instructions mentioned here. Alternatively, you can check out this video Getting started with Python4Delphi.
Python4Delphi Demo1 Sample App shows how to run a Python Script by typing the python code in a Memo, execute and populate the result in another Memo. You can find the Demo1 source on GitHub.
First, run the Python GUI in the Demo1 App:
Table of Contents
1. Simplify Code
Python generator function and generator expression enable us to simplify our code.
To illustrate this, we will compare different implementations that implement a function, “first_n”, that represents the first n non-negative integers, where n is a really big number, and assume (for the sake of the examples in this section) that each integer takes up a lot of space, say 10 megabytes each.
Please note that in real life, integers do not take up that much space, unless they are really, really, really, big integers. For instance you can represent a 309 digit number with 128 bytes (add some overhead, it will still be less than 150 bytes).
First, let us consider the simple example of building a list and returning it, and try that in our Python GUI by Python4Delphi:
|
1 2 3 4 5 6 7 8 9 10 11 |
def first_n(n): '''Build and return a list''' num, nums = 0, [] while num < n: nums.append(num) num += 1 return nums # Calculate the sum of 0 + 1 + 2 + ... + 999999 + 1000000 sum_of_first_n = sum(first_n(1000000)) print(sum_of_first_n) |
Output:
The code above is quite simple and straightforward, but it builds the full list in memory. This is not acceptable in our case, because we cannot afford to keep all n “10-megabyte” integers in our memory.
So, we resort to the generator pattern. The following implements generator as an iterable object.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# Using the generator pattern (an iterable) class first_n(object): def __init__(self, n): self.n = n self.num = 0 def __iter__(self): return self # Python 3 compatibility def __next__(self): return self.next() def next(self): if self.num < self.n: cur, self.num = self.num, self.num + 1 return cur raise StopIteration() sum_of_first_n = sum(first_n(1000000)) print(sum_of_first_n) |
Output:
This code performs well, but we have the following issues:
- There are a lot of boilerplate
- The logic expressed in a somewhat convoluted way
Furthermore, this is a pattern that we will use over and over for many similar constructs. Imagine writing all that just to get an iterator.
Python provides generator functions as a convenient shortcut to building iterators. Our goal is to make our code shorter, not the opposite. Lets us rewrite the above iterator as a generator function:
|
1 2 3 4 5 6 7 8 9 |
# a generator that yields items instead of returning a list def firstn(n): num = 0 while num < n: yield num num += 1 sum_of_first_n = sum(firstn(1000000)) print(sum_of_first_n) |
Output:
Note that the expression of the number generation logic is clear and natural. It is very similar to the implementation that built a list in memory, but has the memory usage characteristic of the iterator implementation.
Generator expressions provide an additional shortcut to build generators out of expressions similar to that of list comprehensions.
In fact, we can turn a list comprehension into a generator expression by replacing the square brackets (“[ ]”) with parentheses. Alternately, we can think of list comprehensions as generator expressions wrapped in a list constructor.
2. Powerful Example
Shown below are some simple, powerful and useful implementation of Python generator:
Reading Large Files
First, we will try this without generator:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# Define a function to read a CSV file def csv_reader(file_name): file = open(file_name) result = file.read().split("n") return result csv_gen = csv_reader("C:/Users/ASUS/Documents/data.csv") row_count = 0 for row in csv_gen: row_count += 1 print(f"Row count is {row_count}") |
Output:
We successfully read a CSV file with 893 rows! But the code above would crash if we try to read larger files, with millions of rows. Now, we will use Python generator to read a file:
|
1 2 3 4 |
# Define a generator function to read a CSV file def csv_reader(file_name): for row in open(file_name, "r"): yield row |
Let’s try to read the larger file, and see the output with Python4Delphi:
What’s happens? Well, we’ve essentially turned csv_reader() into a generator function. This version opens a file, loops through each line, and yields each row, instead of returning it.
Check out Python4Delphi which easily allows you to build Python GUIs for Windows using Delphi.
