Table of Contents
What is code profiling?
Code profiling is a technique to figure out how time is spent in a program. For more details, a profile is a set of statistics that describes how often and for how long various parts of the program are executed. Using the right Python profiling tools – as well as using the right Python Desktop – can save you lots of time chasing down bottlenecks and slow downs in your Python apps.
With these statistics, we can find the “hot spot” of a program and think about ways of improvement. Sometimes, a hot spot in an unexpected location may give you the hint about bugs in your program.
Does Python have a built-in code profiler?
The Python standard library provides two different implementations of the same profiling interface:
1. cProfile
A C extension with reasonable overhead that makes it suitable for profiling long-running programs. cProfile is a recommended practice for most users. cProfile is built based on lsprof, contributed by Brett Rosen and Ted Czotter.
2. profile
A pure Python module whose interface is imitated by cProfile, but which adds significant overhead to profiled programs. If you’re trying to extend the profiler in some way, your task might be easier with this module. Originally designed and written by Jim Roskind.
How do I get the cProfile library?
As cProfile is a built-in Python library, no further installation is needed.
How do I get the snakeviz library, to visualize the Python profiling tools results?
Here is how you can get a stable release of snakeviz using pip:
|
1 |
pip install snakeviz |
How to profile a Python function with cProfile?
For example, run the following code, to profile a function that takes a single argument from the regex library:
|
1 2 3 4 |
import cProfile import re cProfile.run('re.compile("foo|bar")') |
The above action would run re.compile() and print profile results like the following (here is the output on PyScripter IDE):
Or you can use profile, instead of cProfile, like this:
|
1 2 3 4 |
import profile import re profile.run('re.compile("foo|bar")') |
And it would give you the similar output:
The first line indicates that 214 calls were monitored. 207 of the calls were primitive, meaning they were not induced by recursion. The next line: Ordered by: standard name, indicates that the text string in the far right column was used to sort the output.
The column headings include:
ncalls | The number of calls. |
tottime | The total time spent in the given function (and excluding time made in calls to sub-functions). |
percall | The quotient of tottime is divided by ncalls. |
cumtime | The cumulative time spent in this and all subfunctions (from invocation till exit). This figure is accurate even for recursive functions. |
percall | The quotient of cumtime is divided by primitive calls. |
filename:lineno(function) | Provides the respective data of each function. |
How to profile a Python script at a runtime?
Even more usefully, we can profile an existing Python script by running cProfile at a runtime.
Let’s test it to our web scraping script scrapyApp.py that is used in this post:
You can invoke the cProfile when running a Python script, using this command:
|
1 |
python -m cProfile scrapyApp.py |
The following is the excerpt of the profiling results:
How to save Python profiling tools results using cProfile?
Instead of only printing the profiling result on the command line, we can make it more useful to further results by exporting it into a file.
Here is how you can do it:
|
1 |
python -m cProfile -o stats.dump scrapyApp.py |
The above command would export the profiling results into a .dump file.
The .dump file would result in jumbles of weird characters. So, if you prefer the readability, you could save the result in .txt file, with this command:
|
1 |
python -m cProfile scrapyApp.py > stats.txt |
Here is the excerpt of the .txt file, when opened in the text editor:
How to visualize Python profiling tools results using snakeviz?
To visualize your Python code profiling results, call the .dump file with snakeviz, using this command:
|
1 |
snakeviz stats.dump |
It would start a snakeviz web server and would open the visualization results on your default browser. snakeviz web server started on 127.0.0.1:8080 by default.
You can set up the Style, Depth, and Cutoff of the visualization.
Visualize with Icicle style:
Visualize with Sunburst style:
Profiling results in tabular format
Excerpt of all the profiling results in tabular format:
Amazing isn’t it? Now you can easily find out the bottleneck in your program using cProfile, and visualize them professionally with snakeviz.
Congratulations, now you have learned about built-in Python profiling tools! Now you can implement it to find the bottleneck of your code, and improve it even more!
Click here to start using PyScripter, a free, feature-rich, and lightweight IDE for Python developers.
Download RAD Studio to build more powerful Python GUI Windows Apps 5x Faster with Less Code.
Check out Python4Delphi which easily allows you to build Python GUIs for Windows using Delphi.
Also, check out DelphiVCL which easily allows you to build GUIs for Windows using Python.
References & further readings
[1] Davis, M. (2021).
SnakeViz Documentation. GitHub. jiffyclub.github.io/snakeviz
[2] PyPI, Python Package Index. (2021).
snakeviz 2.1.1. Python Software Foundation. pypi.org/project/snakeviz
[3] Python Software Foundation. (2022).
The Python Profilers. Python 3.11.1 documentation. docs.python.org/3/library/profile.html
[4] Shrivarsheni. (2020).
cProfile – How to profile your python code. Machine Learning Plus. machinelearningplus.com/python/cprofile-how-to-profile-your-python-code
[5] Stack Overflow. (2009).
How do I profile a Python script? stackoverflow.com/questions/582336/how-do-i-profile-a-python-script
[6] Yegulalp, S. (2022).
9 fine libraries for profiling Python code. InfoWorld. infoworld.com/article/3600993/9-fine-libraries-for-profiling-python-code.html
