2022-03-21

Parallelization of loop with multiple arguments python

I have a loop that takes minimum 2 hours to run, so I decided to parallelize it. I never paralleled with python but heard multiprocessing is reliable therefore I tried it but when I try to use starmap it outputs ValueError: not enough values to unpack (expected 3, got 0)

The code is down below :

def main():
    EXR_2_JPG()
    arg1 = "validation/Church/Image_0.png"
    arg2 = "validation/Church/Image_1.png"
    with Pool() as pool:
        pool.starmap(compare,zip(arg1,arg2))
        pool.close()
        pool.join()

Basically the compare function is a disparity map calculator so compare(img1,img2) will take the path and turn them to arrays then will loop inside to calculate the disparity that's why it takes a long time. so i want to parallelize it

I also tried to use map but it only works with 1-argument also adding itertools.repeat and functools.partial .

The output is :

  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 771, in get
    raise self._value
ValueError: not enough values to unpack (expected 3, got 0)

The sequential main is :

def main():
    EXR_2_JPG()
    arg1 = "validation/Church/Image_0.png"
    arg2 = "validation/Church/Image_1.png"
    compare(arg1,arg2)

i tired to create another function just for the the double loop but now i get an error of TypeError: starmap() takes from 3 to 4 positional arguments but 10 were given

def loop_f(milieu,rows1,columns1,img1,img2,search_range,sub,out):
    for i in trange(milieu,rows1-milieu):
        for j in trange(milieu,columns1-milieu):
            
            kernel_img1 = img1[i-milieu:i+milieu-1,j-milieu:j+milieu-1,0:3]        
            for w in range (j,min([j+search_range,columns1-milieu])):
                kernel_img2 = img2[i-milieu:i+milieu-1,w-milieu:w+milieu-1,0:3]
                #print(np.sum(kernel_img1),np.sum(kernel_img2))
                testered = tester(kernel_img1)
                testered2 = tester(kernel_img2)
                sum1 = np.sum(kernel_img1)
                sum2 = np.sum(kernel_img2)
                sub[w-j] = np.abs(np.sum(kernel_img1 - kernel_img2))
            #print(sub)
            #print("///////////////////////////////////////////////////////////")
            indice_min = np.argmin(sub)
            out[i,j] = indice_min
    return out

and inside the compare function now i have this :

    with Pool() as pool:
        outx =pool.starmap(loop_f,milieu,rows1,columns1,img1,img2,search_range,sub,out)
        pool.close()
        pool.join()


No comments:

Post a Comment