Parallelization of loop with multiple arguments python
I have a loop that takes minimum 2 hours to run, so I decided to parallelize it. I never paralleled with python but heard multiprocessing
is reliable therefore I tried it but when I try to use starmap
it outputs ValueError: not enough values to unpack (expected 3, got 0)
The code is down below :
def main():
EXR_2_JPG()
arg1 = "validation/Church/Image_0.png"
arg2 = "validation/Church/Image_1.png"
with Pool() as pool:
pool.starmap(compare,zip(arg1,arg2))
pool.close()
pool.join()
Basically the compare
function is a disparity map calculator so compare(img1,img2) will take the path and turn them to arrays then will loop inside to calculate the disparity that's why it takes a long time. so i want to parallelize it
I also tried to use map
but it only works with 1-argument also adding itertools.repeat
and functools.partial
.
The output is :
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\multiprocessing\pool.py", line 771, in get
raise self._value
ValueError: not enough values to unpack (expected 3, got 0)
The sequential main is :
def main():
EXR_2_JPG()
arg1 = "validation/Church/Image_0.png"
arg2 = "validation/Church/Image_1.png"
compare(arg1,arg2)
i tired to create another function just for the the double loop but now i get an error of TypeError: starmap() takes from 3 to 4 positional arguments but 10 were given
def loop_f(milieu,rows1,columns1,img1,img2,search_range,sub,out):
for i in trange(milieu,rows1-milieu):
for j in trange(milieu,columns1-milieu):
kernel_img1 = img1[i-milieu:i+milieu-1,j-milieu:j+milieu-1,0:3]
for w in range (j,min([j+search_range,columns1-milieu])):
kernel_img2 = img2[i-milieu:i+milieu-1,w-milieu:w+milieu-1,0:3]
#print(np.sum(kernel_img1),np.sum(kernel_img2))
testered = tester(kernel_img1)
testered2 = tester(kernel_img2)
sum1 = np.sum(kernel_img1)
sum2 = np.sum(kernel_img2)
sub[w-j] = np.abs(np.sum(kernel_img1 - kernel_img2))
#print(sub)
#print("///////////////////////////////////////////////////////////")
indice_min = np.argmin(sub)
out[i,j] = indice_min
return out
and inside the compare
function now i have this :
with Pool() as pool:
outx =pool.starmap(loop_f,milieu,rows1,columns1,img1,img2,search_range,sub,out)
pool.close()
pool.join()
Comments
Post a Comment