CodeLearn PythonPythonWindows

The Super Short Guide To Built-In Python Profiling Tools

builtin python profiling tools

What is code profiling?

Code profiling is a technique to figure out how time is spent in a program. For more details, a profile is a set of statistics that describes how often and for how long various parts of the program are executed. Using the right Python profiling tools – as well as using the right Python Desktop – can save you lots of time chasing down bottlenecks and slow downs in your Python apps.

With these statistics, we can find the “hot spot” of a program and think about ways of improvement. Sometimes, a hot spot in an unexpected location may give you the hint about bugs in your program.

 

Does Python have a built-in code profiler?

The Python standard library provides two different implementations of the same profiling interface:

  1. cProfile: A C extension with reasonable overhead that makes it suitable for profiling long-running programs. cProfile is a recommended practice for most users. cProfile is built based on lsprof, contributed by Brett Rosen and Ted Czotter.
  2. profile: A pure Python module whose interface is imitated by cProfile, but which adds significant overhead to profiled programs. If you’re trying to extend the profiler in some way, your task might be easier with this module. Originally designed and written by Jim Roskind.

 

How do I get the cProfile library?

As cProfile is a built-in Python library, no further installation is needed.

 

How do I get the snakeviz library, to visualize the Python profiling tools results?

Here is how you can get a stable release of snakeviz using pip:

 

How to profile a Python function with cProfile?

For example, run the following code, to profile a function  that takes a single argument from the regex library:

The above action would run re.compile() and print profile results like the following (here is the output on PyScripter IDE):

The Super Short Guide To Built-In Python Profiling Tools. A cProfile screenshot.

Or you can use profile, instead of cProfile, like this:

And it would give you the similar output:

The Super Short Guide To Built-In Python Profiling Tools. cProfile doing its thing.

The first line indicates that 214 calls were monitored. 207 of the calls were primitive, meaning they were not induced by recursion. The next line: Ordered by: standard name, indicates that the text string in the far right column was used to sort the output.

The column headings include:

ncallsThe number of calls.
tottimeThe total time spent in the given function (and excluding time made in calls to sub-functions).
percallThe quotient of tottime is divided by ncalls.
cumtimeThe cumulative time spent in this and all subfunctions (from invocation till exit). This figure is accurate even for recursive functions.
percallThe quotient of cumtime is divided by primitive calls.
filename:lineno(function)Provides the respective data of each function.

 

How to profile a Python script at a runtime?

Even more usefully, we can profile an existing Python script by running cProfile at a runtime. 

Let’s test it to our web scraping script scrapyApp.py that is used in this post:

You can invoke the cProfile when running a Python script, using this command:

The following is the excerpt of the profiling results:

The Super Short Guide To Built-In Python Profiling Tools. A screenshot of the results from profiling scrapyapp.py

 

How to save Python profiling tools results using cProfile?

Instead of only printing the profiling result on the command line, we can make it more useful to further results by exporting it into a file. 

Here is how you can do it:

The above command would export the profiling results into a .dump file.

The .dump file would result in jumbles of weird characters. So, if you prefer the readability, you could save the result in .txt file, with this command:

Here is the excerpt of the .txt file, when opened in the text editor:

The Super Short Guide To Built-In Python Profiling Tools. Saving the cProfile results.

 

How to visualize Python profiling tools results using snakeviz?

To visualize your Python code profiling results, call the .dump file with snakeviz, using this command:

It would start a snakeviz web server and would open the visualization results on your default browser. snakeviz web server started on 127.0.0.1:8080 by default.

You can set up the Style, Depth, and Cutoff of the visualization.

Visualize with Icicle style:

The Super Short Guide To Built-In Python Profiling Tools. A visualization of profiuling results.

Visualize with Sunburst style:

The Super Short Guide To Built-In Python Profiling Tools. A sunburst visualization.

Excerpt of all the profiling results in tabular format:

The Super Short Guide To Built-In Python Profiling Tools. Profiling results shown as a table.

Amazing isn’t it? Now you can easily find out the bottleneck in your program using cProfile, and visualize them professionally with snakeviz.

Congratulations, now you have learned about built-in Python profiling tools! Now you can implement it to find the bottleneck of your code, and improve it even more!

close
Related posts
Python

A Beginner's Guide To Python Tools

Learn PythonPythonPython GUIVideos

This Is How To Create A Python Gui With PyQt5

Learn PythonPythonPython GUIVideos

How To Make A Python GUI For An ATM System

Python

10 Resources That'll Make You Better At Python Programming Software

Leave a Reply

Your email address will not be published.

en_USEnglish