python tips: multiprocessing and shared memory

With multiprocessing package, it is extremely easy to make programs which share tasks with multiple workers(processes). This post is memo for how to share object which initialized by parent process with child process.

General way for sharing object between parent process and child processes is using Queue or Pipe. However, it's the easiest way for me to use global valuable. Below code is just a example of how to share object between processes using global valuable.

#!/usr/bin/env python
# coding:utf-8

import os
import multiprocessing

x = 10
y = 20
z = []
a = None
b = [1,2,3]
c = None

c = 3

def main():
    a = 1
    print("###############################")
    print("x:: {0}".format(print_value(x)))
    print("y:: {0}".format(print_value(y)))
    print("z:: {0}".format(print_value(z)))
    print("a:: {0}".format(print_value(a)))
    print("b:: {0}".format(print_value(b)))
    print("c:: {0}".format(print_value(c)))
    print("###############################")
    pool = multiprocessing.Pool(3)
    pool.map(func, [1,2,3], chunksize=1)
    

def print_value(val):
    return "{0}, id:{1}".format(val, id(val))
    
def func(val):
    print("pid: {0}, process_number: {1}".format(os.getpid(), val))
    print("x:: {0}".format(print_value(x)))
    print("y:: {0}".format(print_value(y)))
    print("z:: {0}".format(print_value(z)))    
    print("a:: {0}".format(print_value(a)))
    print("b:: {0}".format(print_value(b)))
    print("c:: {0}".format(print_value(c)))    

if __name__ == "__main__":
    main()

Here is output from above code. 'id' means memory-address pointed by valuable.

yaboo@ubuntu:~$ python shared_memory.py 
###############################
x:: 10, id:24606432
y:: 20, id:24606192
z:: [], id:139931380327848
a:: 1, id:24606648
b:: [1, 2, 3], id:139931380327920
c:: 4, id:24606576
###############################
pid: 3747, process_number: 1
x:: 10, id:24606432
pid: 3748, process_number: 2
y:: 20, id:24606192
x:: 10, id:24606432
z:: [], id:139931380327848
pid: 3749, process_number: 3
y:: 20, id:24606192
a:: None, id:9071744
x:: 10, id:24606432
z:: [], id:139931380327848
y:: 20, id:24606192
b:: [1, 2, 3], id:139931380327920
a:: None, id:9071744
c:: 3, id:24606600
z:: [], id:139931380327848
a:: None, id:9071744
b:: [1, 2, 3], id:139931380327920
b:: [1, 2, 3], id:139931380327920
c:: 3, id:24606600
c:: 3, id:24606600
yaboo@ubuntu:~$