Python profiling tools enable you to understand how fast your code executes. Also, they help you identify the bottlenecks. They play a major role in optimizing your program. It gives you several advantages. For instance, the change in business needs may require you to run code faster. With Python profilers, you can identify the parts of code that are causing sluggish performance. In this article, you will find different ways to profile your Python code. Let’s dive in.
Table of Contents
What are Python profilers?
Python profilers are tools that help you to figure out how the time is spent on the program. It enables you to identify parts that are slowing down the performance. For instance, Application Performance Monitoring (APM) is one of the top profiling tools. You can use it to profile the entire life cycle of transactions for the web application. Hence, you can determine the bottlenecks in your code.
Python Profilers – How can I profile my Python code?
Python Profilers – Use timeit in Command-Line for Profiling Python Code
Python has a module, called timeit. It enables you to time small code snippets. It utilizes platform-specific time functions. As a result, you will get the most accurate timings.
Let’s take a look at an example. Start the terminal on your PC. Then run the following commands:
|
1 2 3 4 |
python -m timeit -s "[ord(x) for x in 'abcdfghi']" 100000000 loops, best of 3: 0.0115 usec per loop python -m timeit -s "[chr(int(x)) for x in '123456789']" 100000000 loops, best of 3: 0.0119 usec per loop |
Here, you are calling Python on the command line. Then you are passing the “-m” option to look up and use the timeit module as the main program. Next, you pass the “-s” option to tell the module to run the setup once. The command runs the code s 3 times. It returns the best average of the runs.
The -s option is usually used to import the libraries. For instance, you can utilize it to compare the square root function from Python’s math module from NumPy. Here is the code:
|
1 2 3 |
python -m timeit '[x**0.5 for x in range(1000)]' python -m timeit -s 'from math import sqrt' '[sqrt(x) for x in range(1000)]' python -m timeit -s 'from numpy import sqrt' '[sqrt(x) for x in range(1000)]' |
Now, let’s write a silly function. You can time it from the command line.
|
1 2 3 4 5 6 |
# simple_func.py def my_function(): try: 1 / 0 except ZeroDivisionError: pass |
The function will cause an error. However, it is promptly ignored.
Now, you have to run this command:
|
1 2 |
python -m timeit "import simple_func; simple_func.my_function()" 1000000 loops, best of 3: 1.77 usec per loop |
Here, you are importing and calling the function. Next, you will learn to use timeit inside an actual Python script.
Read: 5 Real-Life Lessons About The Best IDE for Python
Python Profilers – Import timeit for Testing
You can easily use the timeit module inside your code. Here is an example:
|
1 2 3 4 5 6 7 8 9 10 11 |
# simple_func2.py def my_function(): try: 1 / 0 except ZeroDivisionError: pass if __name__ == "__main__": import timeit setup = "from __main__ import my_function" print(timeit.timeit("my_function()", setup=setup)) |
Python Profilers – Use a Decorator
You can consider writing your own timer. However, it may not be as accurate as using timeit.
Let’s write the custom decorator:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
import random import time def timerfunc(func): """ A timer decorator """ def function_timer(*args, **kwargs): """ A nested function for timing other functions """ start = time.time() value = func(*args, **kwargs) end = time.time() runtime = end - start msg = "The runtime for {func} took {time} seconds to complete" print(msg.format(func=func.__name__, time=runtime)) return value return function_timer @timerfunc def long_runner(): for x in range(5): sleep_time = random.choice(range(1,5)) time.sleep(sleep_time) if __name__ == '__main__': long_runner() |
Here, you are importing the random and time modules from Python’s standard library. Then you are creating your decorator function. It accepts a function. Also, it has another function inside. Before calling the passed-in function, the nested function will grab the time. Next, it waits for the function to return and grabs the end time. Then you will know the time the function took to run. You can print it out. The last two statements return the result of the function call and the function itself.
Python Profilers – Create a Timing Context Manager
You can consider using context managers. It will enable you to time small pieces of code. Let’s create the timer context manager class. Here is the code:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
import random import time class MyTimer(): def __init__(self): self.start = time.time() def __enter__(self): return self def __exit__(self, exc_type, exc_val, exc_tb): end = time.time() runtime = end - self.start msg = 'The function took {time} seconds to complete' print(msg.format(time=runtime)) def long_runner(): for x in range(5): sleep_time = random.choice(range(1,5)) time.sleep(sleep_time) if __name__ == '__main__': with MyTimer(): long_runner() |
Python Profilers – Using cProfile
Python features built-in code profilers. They are known as the profile module and the cProfile module. The first one is pure Python. It will add a lot of overhead to anything you profile. That’s why you should go with cProfile. It has a similar interface. However, it is significantly faster.
Read: What is the best Python editor?
Let’s look at an example:
|
1 2 3 4 5 6 7 8 9 10 11 |
>>> import cProfile >>> cProfile.run("[x for x in range(1500)]") 4 function calls in 0.001 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.000 0.000 <string>:1(<listcomp>) 1 0.000 0.000 0.000 0.000 <string>:1(<module>) 1 0.001 0.001 0.001 0.001 {built-in method builtins.exec} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} |
What do these columns mean? Let’s find it out.
- ncalls – It’s the number of calls made.
- tottime – It refers to the total time spent in the given function.
- percall – It refers to the quotient of tottime divided by ncalls
- cumtime – It is the cumulative time spent in this function, along with all subfunctions. For recursive functions, you will find it to be very accurate
- The second percall column – It refers to the quotient of cumtime divided by primitive calls
- filename:lineno(function) – It provides the respective data of each function
You can call cProfile on the command line in almost the same way as the timeit module. However, there is a difference. You have to pass a Python script, rather than a snippet. Here’s an example call:
|
1 |
python -m cProfile test.py |
Python Profilers – Use line_profiler
You can use a 3rd party project, called line_profiler. It enables you to profile the time each line takes to execute. The installation process is very simple. You just need to use pip. Here is the command that you need to utilize:
|
1 |
pip install line_profiler |
To use line_profiler, you will need some code to profile. You just need to utilize this command:
|
1 |
kernprof -l silly_functions.py |
You will see this output
|
1 |
Wrote profile results to silly_functions.py.lprof |
Now, let’s write the script:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# silly_functions.py import time @profile def fast_function(): print("I'm a fast function!") @profile def slow_function(): time.sleep(2) print("I'm a slow function") if __name__ == '__main__': fast_function() slow_function() |
Now, you have to run this command:
|
1 |
python -m line_profiler silly_functions.py.lprof |
You will see this output:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
I'm a fast function! I'm a slow function Wrote profile results to silly_functions.py.lprof Timer unit: 1e-06 s Total time: 3.4e-05 s File: silly_functions.py Function: fast_function at line 3 Line # Hits Time Per Hit % Time Line Contents ============================================================== 3 @profile 4 def fast_function(): 5 1 34 34.0 100.0 print("I'm a fast function!") Total time: 2.001 s File: silly_functions.py Function: slow_function at line 7 Line # Hits Time Per Hit % Time Line Contents ============================================================== 7 @profile 8 def slow_function(): 9 1 2000942 2000942.0 100.0 time.sleep(2) 10 1 59 59.0 0.0 print("I'm a slow function") |
Python Profilers – Use memory_profiler
memory_profiler is an amazing 3rd party profiling package. You can use it to monitor memory consumption in a process. Also, you can use it for the line-by-line analysis of your code.
Read: 6 Best GUI in Python Frameworks
To install memory_profiler, you need to use pip:
|
1 |
pip install memory_profiler |
Once the installation is done, you have to run a few lines of code against it. Here’s a simple example:
|
1 2 3 4 5 6 7 8 9 10 |
# memo_prof.py @profile def mem_func(): lots_of_numbers = list(range(1500)) x = ['letters'] * (5 ** 10) del lots_of_numbers return None if __name__ == '__main__': mem_func() |
Now, you can run the code by using this command:
|
1 2 3 4 5 6 7 8 9 10 11 |
python -m memory_profiler memo_prof.py Filename: memo_prof.py Line # Mem usage Increment Line Contents ================================================ 1 16.672 MiB 0.000 MiB @profile 2 def mem_func(): 3 16.707 MiB 0.035 MiB lots_of_numbers = list(range(1500)) 4 91.215 MiB 74.508 MiB x = ['letters'] * (5 ** 10) 5 91.215 MiB 0.000 MiB del lots_of_numbers 6 91.215 MiB 0.000 MiB return None |
Python Profilers – Use profilehooks
Profilehooks is a collection of decorators. It is designed for profiling functions. To install profilehooks, you have to run this command:
|
1 |
pip install profilehooks |
Next, you have to write the script:
|
1 2 3 4 5 6 7 8 9 10 |
# memo_prof.py @profile def mem_func(): lots_of_numbers = list(range(1500)) x = ['letters'] * (5 ** 10) del lots_of_numbers return None if __name__ == '__main__': mem_func() |
Now, let’s run the code. You will see this output:
|
1 2 3 4 5 6 7 8 9 10 11 |
*** PROFILER RESULTS *** mem_func (profhooks.py:5) function called 1 times 2 function calls in 0.045 seconds Ordered by: cumulative time, internal time, call count ncalls tottime percall cumtime percall filename:lineno(function) 1 0.045 0.045 0.000 0.045 prohooks.py:5(mem_func) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 0 0.000 0.000 profile:0(profiler) |
Python Profilers – Additional Resources
The profiler of Python’s standard library is very powerful. However, there are plenty of other options to choose from. For example, you can consider using the Python Call Graph module. It visualizes how functions call each other using the GraphViz tool.
- Python Call Graph: https://pycallgraph.readthedocs.io/en/master/
Also, you can dig into the compiled code by using Valgrind.
- Valgrind: https://valgrind.org/
However, you may have to recompile the Python interpreter to enable debugging support.
Conclusion
In this article, you have learned to use timeit and cProfile modules to time and profile your code. Also, you have learned about using the decorator and the context manager. Finally, you explore different 3rd party packages, including line_profiler, memory_profilerand profilehooks. Now, you are ready to use Python profilers. Therefore, you can easily find the bottlenecks in your code.
FAQ
What is the Python standard library?
The Python Standard Library is a collection of script modules accessible to a Python program. It enables you to simplify the programming process. Also, it eliminates the need to rewrite commonly used commands.
How can I optimize the memory usage in my Python program?
You can optimize memory usage using profilers. It enables you to find memory leaks. Hence, you can efficiently optimize your Python programs.
What are function calls?
Function calls are expressions that pass control and arguments to a function.
What is a profiler in programming?
A profiler is a performance analysis tool. It measures the frequency and duration of function calls.
What is debugging in Python?
Debugging is the process of detecting and removing existing and potential errors. It enables you to flawlessly run your Python program.
