Using Python Multiprocessing to find distances between points in array

I've been trying to speed up a practice project using python's multiprocessing library. In the project I have 2 arrays, points and weights, full of x, y coordinates. I'm trying to find the distance between each weight coordinate and point. When I run the program with multiprocessing, the program uses all the computers ram and CPU, and when looking at task manager, up to 20 instances of python are running. I know the program works because it works without multiprocessing but takes around 20 seconds to complete.

Here is the code, and at the bottom is the programming running with Pool.map and Process from the multiprocessing library.

import math
import random
import multiprocessing as mp

screenSize = 1000000

pointsLength = 2000
weightLength = 20000

weightBuffer = screenSize/weightLength
pointBuffer = screenSize/pointsLength

points = []
weights = []
weightPoints = []

counter = 0

for i in range(pointsLength):
    for j in range(pointsLength):
        points.append([random.randint(j * pointBuffer, j * pointBuffer * 2), 
        random.randint(i * pointBuffer, i * pointBuffer * 2)])


for i in range(pointsLength):
    for j in range(pointsLength):
        weightPoints.append([j * weightBuffer, i * weightBuffer])
        weights.append(0)


def FindDistance(i):
    row = math.floor((i / weightLength) / (weightLength / pointsLength))
    col = math.floor((i % weightLength) / (weightLength / pointsLength))
    points1d = (pointsLength * row) + col

    dist = math.dist(points[points1d], weightPoints[i])

    weights[i] = dist

# With Multiprocessing Pool
# sumthing = []
# for i in range(len(weights)):
#     sumthing.append(i)

# with mp.Pool(4) as p:
#     p.map(FindDistance, sumthing)


# With Multiproessing Process
processes = []
for i in range(len(weights)):
    p = mp.Process(target=FindDistance, args=[i])
    p.start()
    processes.append(p)

for process in processes:
    process.join()


# Without Multiprocessing
# for i in range(len(weights)):
#     FindDistance(i)

#     counter += 1

#     if (counter % 25000 == 0):
#         print(counter / 25000)

If anyone knows how I could get multiprocessing to work, where the program would use the 8 cores on my computer without crashing the program because of ram or cpu limitations.



from Recent Questions - Stack Overflow https://ift.tt/3aE3Ry0
https://ift.tt/eA8V8J

Comments

Popular posts from this blog

Today Walkin 14th-Sept

Spring Elasticsearch Operations

Hibernate Search - Elasticsearch with JSON manipulation