Parallel Pythoning
PART 004: One line game changer — MP decorator
Every now and then I manage to write a piece of code that really makes me smile for a long time. It happens about twice a year, sometimes more often. The shorter and more useful the code is, the better. Yesterday I felt enlightened again, and I cannot resist showing you the result of my internal brainstorm.
To start with, I read many bad opinions about multiprocessing (MP) in Python — useless, slow, badly designed. But to be fair, first steps with multiprocessing may be difficult in any programming language. Queues, locks, races, synchronization, shared memory — they all make our life more complicated. Debugging a bad multiprocessing implementation is a nightmare. Usually, it is better to start from scratch than to search for a hidden bug.
Personally, I do not remember any of my bigger projects in Python without MP. I honestly like it, and I think it does a good job — from data preprocessing to preparation of batches. It all needs to be done on a daily basis. I even wrote some wrappers of ready-to-use functions and objects based on MP concepts to speed up such frequent tasks. But you need to know how to use it properly, and in some cases it can be really complicated.
So… Here it is: few lines of code, probably the simplest and most useful MP I have ever seen :). A simple and yet powerful. The best thing about this function is that it can be reused as a decorator anywhere in your code to run parallel computing with just a single annotation.
With this decorator you may run your function as many times as you want — in loops, in objects, without waiting till any previously started processes finish. You can run functions in a separate process with just one decorator! (But be careful — your processors may explode!). Example of usage:
Soon after the first one, came the second — the alternative version that waits till the child process is finished. Both could be done with one decorator and parameters, but I decided that it is better like that — just two simple decorators.
You may ask: ‘Why do we need a parallel process if the main process has to wait?’ And here goes a simple answer. Think of how many times you had a case when running the same piece of code from the same process caused strange effects like runtime errors, memory overflow, locked GPU. Sometimes it just happens, and usually the problem is somewhere not exactly in ‘your’ code. Maybe you know the odd behaviour of libraries like TensorFlow, where you need some very deep knowledge to run models in parallel, and not get into trouble with Sessions, Graphs and other ‘globals’. Do you know that TensorFlow will not free the GPU memory until you close the Python process? Clearing the Graph and closing the Session does not help, believe me. Imagine now running a few models from one python script with just one decorator! There is no easier way!
Obviously, both decorators have some limitations. They do not enable communication with a parent process, and a function cannot return anything. Its simple form will not fulfill some bigger needs, but (for sure!) will do a good job many times. It is a simple solution for a fair number of simple cases. It is also a good starting point for custom modification.
Both decorators, examples and a bit more you can find in my ptools repo: https://github.com/piteren/ptools/blob/master/mpython/mpdecor.py
Please, feel free to share your thoughts in the comments.
This article goes beyond the topic of pypoks a little, but the presented feature appeared during the project development process. Next time, I will get back to the project curriculum.