Memory Profiling Python : How to Perform

Welcome back, Python enthusiasts! In today’s blog post, we’ll be diving deep into the world of Python memory profiling. This technique is essential for optimizing your Python code, providing you with detailed insights into memory allocation and deallocation.

Understanding Memory Profiling in Python

Memory profiling is a process that helps you examine your Python program’s memory consumption. By doing so, you can understand better how your program allocates and deallocates memory. This information is vital, especially when it comes to optimizing your code for better performance.

Introducing Memory Profiler

To facilitate memory profiling in Python, we’ll be using a package named ‘Memory Profiler’. This external Python package needs install to your Python environment. You can accomplish this by running command bellow on your command line.

pip3 install memory-profiler

To use Memory Profiler, you will need to import two crucial components – ‘profile’ and ‘memory_usage’. The ‘profile’ is a decorator that you use above a function, while ‘memory_usage’ is a function you call within your code.

from memory_profiler import profile,

@profile
def my_function(list_size):
    my_list = ['hello'] * list_size
    my_list2 = ['world'] * list_size
    del my_list
    return my_list2

if name == "main":
    my_function(1000000)

Let’s start by creating an example function, which we’ll call ‘my_function’. This function will take ‘list_size’ as a parameter, which will define the size of two lists we’ll create.

We will then allocate memory space twice and deallocate it once, examining the changes in memory usage throughout this process.

Memory profiling doesn’t occur by default when you run your Python function. To enable it, you’ll need to use the ‘@profile’ decorator above your function. This action will print detailed information about your function’s memory usage when executed.

Once you run your code with the ‘@profile’ decorator, you’ll get a detailed output showcasing memory allocation and deallocation for different lines of your code.

Memory profiling in Python is a powerful tool for optimizing your code and ensuring efficient memory usage. By implementing the Memory Profiler package, you’re taking a significant step towards producing cleaner, more efficient Python code.

Example 02: Add Log

from memory_profiler import profile, 

log_file = open('memory.log', 'w+')

@profile(stream=log_file)
def my_function(list_size):
    """
    This function creates two lists of the given size and measures the memory usage.
    
    Parameters:
    - list_size (int): The size of the lists to be created.
    
    Returns:
    - my_list2 (list): The second list created.
    """
    my_list = ['hello'] * list_size
    my_list2 = ['world'] * list_size
    del my_list
    return my_list2
    

if __name__ == "__main__":
    my_function(10000)
    my_function(1000000)
    my_function(100000000)

In the above script, we’ve defined a function my_function that creates two lists of a given size, list_size. We’re using the @profile decorator from the memory_profiler module to profile this function’s memory usage. The profiling results will write into a log file named ‘memory.log’.

The my_function function will be call with three different list sizes (10,000, 1,000,000, and 100,000,000) in the script’s main execution block.

Adding to the tutorial, by running the command

python memory-profile02.py

in your terminal or command prompt will generate a file named ‘memory.log’. This file contains the results of the memory profiling performed by the Memory Profiler.

It provides a line-by-line breakdown of the memory usage of the my_function, allowing you to pinpoint which parts of your code consume the most memory. This output can be invaluable when optimizing your Python applications for better performance and efficiency.

Example 03: memory_usage

from memory_profiler import memory_usage

def my_function(list_size):
    my_list = ['hello'] * list_size
    my_list2 = ['world'] * list_size
    del my_list
    return my_list2
    

if __name__ == "__main__":
    mem_usage = memory_usage((my_function, (), {'list_size':10000000}))
    print(mem_usage)

In the provided Python code snippet, we are utilizing the Memory Profiler module to monitor the memory usage of a function called my_function. This function creates two lists, my_list and my_list2, each filled with repeated strings 'hello' and 'world' respectively. The size of these lists is determined by the list_size parameter. After creating these lists, my_list is deleted to free up memory. The remaining list, my_list2, is then returned by the function.

In the main execution block of the code, the memory_usage function from the Memory Profiler module is used to monitor the memory consumption of my_function when it is called with a list size of 10,000,000. The resulting memory usage will print to the console. This implementation allows developers to gauge the efficiency of their code by providing insight into how memory usage scales with increasing list size.