Short Answer

Pool's chunksize-algorithm is a heuristic. It provides a simple solution for all imaginable problem scenarios you are trying to stuff into Pool's methods. As a consequence, it cannot be optimized for any specific scenario.

The algorithm arbitrarily divides the iterable in approximately four times more chunks than the naive approach. More chunks mean more overhead, but increased scheduling flexibility. How this answer will show, this leads to a higher worker-utilization on average, but without the guarantee of a shorter overall computation time for every case.

"That's nice to know" you might think, "but how does knowing this help me with my concrete multiprocessing problems?" Well, it doesn't. The more honest short answer is, "there is no short answer", "multiprocessing is complex" and "it depends". An observed symptom can have different roots, even for similar scenarios.

This answer tries to provide you with basic concepts helping you to get a clearer picture of Pool's scheduling black box. It also tries to give you some basic tools at hand for recognizing and avoiding potential cliffs as far they are related to chunksize.

Table of Contents

Part I

1. Definitions
2. Parallelization Goals
3. Parallelization Scenarios
4. Risks of Chunksize > 1
5. Pool's Chunksize-Algorithm

6. Quantifying Algorithm Efficiency

   6.1 Models

   6.2 Parallel Schedule

   6.3 Efficiencies

   6.3.1 Absolute Distribution Efficiency (ADE)

   6.3.2 Relative Distribution Efficiency (RDE)

Part II

7. Naive vs. Pool's Chunksize-Algorithm
8. Reality Check
9. Conclusion

It is necessary to clarify some important terms first.

1. Definitions

Chunk

A chunk here is a share of the iterable-argument specified in a pool-method call. How the chunksize gets calculated and what effects this can have, is the topic of this answer.

Task

A task's physical representation in a worker-process in terms of data can be seen in the figure below.

pool.map()この図は、関数から取得したコード行に沿って表示されるの呼び出し例を示しています。このコードmultiprocessing.pool.worker行では、 から読み取られたタスクがinqueue展開されます。は、プール ワーカー プロセスのworkerの基になるメイン関数です。 プール メソッドで指定された 引数は、を使用した やなどの単一呼び出しメソッドの場合のみ、 関数内の 変数と一致します。 パラメータを持つその他のプール メソッドの場合、処理関数はマッパー関数 (または) になります。 この関数は、反復可能オブジェクトの送信されたチャンクのすべての要素に、ユーザーが指定したパラメータをマップします (-->「map-tasks」)。 この処理にかかる時間によって、MainThreadfuncfuncworkerapply_asyncimapchunksize=1chunksizefuncmapstarstarmapstarfuncタスクまた、作業単位。

タスクル

「タスク」という言葉の使用は全体1つのチャンクの処理は、コード内のコードによって一致しますがmultiprocessing.pool、単一通話funcチャンクの1つの要素を引数として持つ、ユーザー指定の を参照する必要があります。名前の競合による混乱を避けるため（ maxtasksperchildPoolの__init__-methodの -parameterを考えてみてください）、この回答ではタスク内の単一の作業単位を次のように参照します。タスク。

あタスク（からタスク + エルメントは、タスクこれは、メソッドfuncのパラメータで指定された関数Poolを、単一の要素送信されたかたまり.あタスク構成chunksize タスク。

並列化オーバーヘッド (PO)

POPython 内部のオーバーヘッドとプロセス間通信 (IPC) のオーバーヘッドで構成されます。Python 内のタスクごとのオーバーヘッドには、タスクとその結果のパッケージ化とアンパックに必要なコードが伴います。IPC オーバーヘッドには、必要なスレッドの同期と、異なるアドレス空間間でのデータのコピー (必要なコピー手順は 2 つ: 親 -> キュー -> 子) が伴います。IPC オーバーヘッドの量は OS、ハードウェア、およびデータ サイズに依存するため、影響について一般化することは困難です。

2. 並列化の目標

マルチプロセッシングを使用する場合、全体的な目標は（当然ですが）すべてのタスクの合計処理時間を最小化することです。この全体的な目標を達成するために、技術的目標必要となるハードウェアリソースの利用を最適化する。

技術目標を達成するための重要なサブ目標は次のとおりです。

• 並列化のオーバーヘッドを最小限に抑える（最も有名なのは、しかしそれだけではない：国際PC）
• すべてのCPUコアで高い使用率
• OSの過剰なページングを防ぐためにメモリ使用量を制限します（破壊する）

まず、タスクは計算的に重い（集中的）である必要があり、取り戻す並列化のために支払う必要があるPO。タスクあたりの絶対計算時間が長くなると、POの重要性は低下します。逆に言えば、絶対計算時間が長くなると、タスクあたり for your problem, the less relevant gets the need for reducing PO. If your computation will take hours per taskel, the IPC overhead will be negligible in comparison. The primary concern here is to prevent idling worker processes after all tasks have been distributed. Keeping all cores loaded means, we are parallelizing as much as possible.

3. Parallelization Scenarios

What factors determine an optimal chunksize argument to methods like multiprocessing.Pool.map()

The major factor in question is how much computation time may vary across our single taskels. To name it, the choice for an optimal chunksize is determined by the Coefficient of Variation (CV) for computation times per taskel.

The two extreme scenarios on a scale, following from the extent of this variation are:

1. All taskels need exactly the same computation time.
2. A taskel could take seconds or days to finish.

For better memorability, I will refer to these scenarios as:

1. Dense Scenario
2. Wide Scenario

Dense Scenario

In a Dense Scenario it would be desirable to distribute all taskels at once, to keep necessary IPC and context switching at a minimum. This means we want to create only as much chunks, as much worker processes there are. How already stated above, the weight of PO increases with shorter computation times per taskel.

For maximal throughput, we also want all worker processes busy until all tasks are processed (no idling workers). For this goal, the distributed chunks should be of equal size or close to.

Wide Scenario

The prime example for a Wide Scenario would be an optimization problem, where results either converge quickly or computation can take hours, if not days. Usually it is not predictable what mixture of "light taskels" and "heavy taskels" a task will contain in such a case, hence it's not advisable to distribute too many taskels in a task-batch at once. Distributing less taskels at once than possible, means increasing scheduling flexibility. This is needed here to reach our sub-goal of high utilization of all cores.

If Pool methods, by default, would be totally optimized for the Dense Scenario, they would increasingly create suboptimal timings for every problem located closer to the Wide Scenario.

4. Risks of Chunksize > 1

Consider this simplified pseudo-code example of a Wide Scenario-iterable, which we want to pass into a pool-method:

good_luck_iterable = [60, 60, 86400, 60, 86400, 60, 60, 84600]

Instead of the actual values, we pretend to see the needed computation time in seconds, for simplicity only 1 minute or 1 day. We assume the pool has four worker processes (on four cores) and chunksize is set to 2. Because the order will be kept, the chunks send to the workers will be these:

[(60, 60), (86400, 60), (86400, 60), (60, 84600)]

Since we have enough workers and the computation time is high enough, we can say, that every worker process will get a chunk to work on in the first place. (This does not have to be the case for fast completing tasks). Further we can say, the whole processing will take about 86400+60 seconds, because that's the highest total computation time for a chunk in this artificial scenario and we distribute chunks only once.

Now consider this iterable, which has only one element switching its position compared to the previous iterable:

bad_luck_iterable = [60, 60, 86400, 86400, 60, 60, 60, 84600]

...and the corresponding chunks:

[(60, 60), (86400, 86400), (60, 60), (60, 84600)]

Just bad luck with the sorting of our iterable nearly doubled (86400+86400) our total processing time! The worker getting the vicious (86400, 86400)-chunk is blocking the second heavy taskel in its task from getting distributed to one of the idling workers already finished with their (60, 60)-chunks. We obviously would not risk such an unpleasant outcome if we set chunksize=1.

This is the risk of bigger chunksizes. With higher chunksizes we trade scheduling flexibility for less overhead and in cases like above, that's a bad deal.

How we will see in chapter 6. Quantifying Algorithm Efficiency, bigger chunksizes can also lead to suboptimal results for Dense Scenarios.

5. Pool's Chunksize-Algorithm

Below you will find a slightly modified version of the algorithm inside the source code. As you can see, I cut off the lower part and wrapped it into a function for calculating the chunksize argument externally. I also replaced 4 with a factor parameter and outsourced the len() calls.

# mp_utils.py

def calc_chunksize(n_workers, len_iterable, factor=4):
    """Calculate chunksize argument for Pool-methods.

    Resembles source-code within `multiprocessing.pool.Pool._map_async`.
    """
    chunksize, extra = divmod(len_iterable, n_workers * factor)
    if extra:
        chunksize += 1
    return chunksize

def compare_chunksizes(len_iterable, n_workers=4):
    """Calculate naive chunksize, Pool's stage-1 chunksize and the chunksize
    for Pool's complete algorithm. Return chunksizes and the real factors by
    which naive chunksizes are bigger.
    """
    cs_naive = len_iterable // n_workers or 1  # naive approach
    cs_pool1 = len_iterable // (n_workers * 4) or 1  # incomplete pool algo.
    cs_pool2 = calc_chunksize(n_workers, len_iterable)

    real_factor_pool1 = cs_naive / cs_pool1
    real_factor_pool2 = cs_naive / cs_pool2

    return cs_naive, cs_pool1, cs_pool2, real_factor_pool1, real_factor_pool2

if extra:
    chunksize += 1

# mp_utils.py

from collections import namedtuple


Chunkinfo = namedtuple(
    'Chunkinfo', ['n_workers', 'len_iterable', 'n_chunks',
                  'chunksize', 'last_chunk']
)

def calc_chunksize_info(n_workers, len_iterable, factor=4):
    """Calculate chunksize numbers."""
    chunksize, extra = divmod(len_iterable, n_workers * factor)
    if extra:
        chunksize += 1
    # `+ (len_iterable % chunksize > 0)` exploits that `True == 1`
    n_chunks = len_iterable // chunksize + (len_iterable % chunksize > 0)
    # exploit `0 == False`
    last_chunk = len_iterable % chunksize or chunksize

    return Chunkinfo(
        n_workers, len_iterable, n_chunks, chunksize, last_chunk
    )

def calc_naive_chunksize_info(n_workers, len_iterable):
    """Calculate naive chunksize numbers."""
    chunksize, extra = divmod(len_iterable, n_workers)
    if chunksize == 0:
        chunksize = 1
        n_chunks = extra
        last_chunk = chunksize
    else:
        n_chunks = len_iterable // chunksize + (len_iterable % chunksize > 0)
        last_chunk = len_iterable % chunksize or chunksize

    return Chunkinfo(
        n_workers, len_iterable, n_chunks, chunksize, last_chunk
    )

# mp_utils.py

def calc_ade(n_workers, len_iterable, n_chunks, chunksize, last_chunk):
    """Calculate Absolute Distribution Efficiency (ADE).

    `len_iterable` is not used, but contained to keep a consistent signature
    with `calc_rde`.
    """
    if n_workers == 1:
        return 1

    potential = (
        ((n_chunks // n_workers + (n_chunks % n_workers > 1)) * chunksize)
        + (n_chunks % n_workers == 1) * last_chunk
    ) * n_workers

    n_full_chunks = n_chunks - (chunksize > last_chunk)
    taskels_in_regular_chunks = n_full_chunks * chunksize
    real = taskels_in_regular_chunks + (chunksize > last_chunk) * last_chunk
    ade = real / potential

    return ade

# mp_utils.py

def calc_rde(n_workers, len_iterable, n_chunks, chunksize, last_chunk):
    """Calculate Relative Distribution Efficiency (RDE)."""
    ade_cs1 = calc_ade(
        n_workers, len_iterable, n_chunks=len_iterable,
        chunksize=1, last_chunk=1
    )
    ade = calc_ade(n_workers, len_iterable, n_chunks, chunksize, last_chunk)
    rde = ade / ade_cs1

    return rde