Fix: Python multiprocessing Not Working (freeze_support, Pickle Errors, Zombie Processes)

The Error

You use Python’s multiprocessing module and get one of these errors:

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.
        ...
        if __name__ == '__main__':
            freeze_support()

Or:

_pickle.PicklingError: Can't pickle <function <lambda> at 0x...>: attribute lookup <lambda> on __main__ failed

Or the script hangs indefinitely with no output:

pool = multiprocessing.Pool(4)
results = pool.map(my_func, data)
# Hangs forever

Or processes finish but are left as zombie processes in ps aux.

Why This Happens

Python multiprocessing works by spawning new Python interpreter processes. On Windows and macOS (Python 3.8+), the default start method is spawn — it starts a fresh Python interpreter and re-imports your script in each worker. This has several consequences:

• The if __name__ == "__main__" guard is required on Windows/macOS — without it, spawned workers import the script and try to spawn more workers, causing the freeze_support error.
• Only picklable objects can be passed between processes — lambdas, closures, and locally defined functions cannot be pickled.
• The Pool must be properly closed — leaving a Pool open causes zombie processes or hangs.
• Shared state does not work like threads — each process has its own memory; modifying a global variable in a worker does not affect the parent.

Fix 1: Add the if name == “main” Guard

This is required on Windows and macOS when using spawn (the default start method):

Broken — no guard:

import multiprocessing

def worker(x):
    return x * x

pool = multiprocessing.Pool(4)
results = pool.map(worker, range(10))
print(results)

Fixed:

import multiprocessing

def worker(x):
    return x * x

if __name__ == "__main__":
    pool = multiprocessing.Pool(4)
    results = pool.map(worker, range(10))
    print(results)
    pool.close()
    pool.join()

On Linux, the default start method is fork (copies the parent process), which does not require this guard. But your code must run correctly on all platforms — always include the guard.

Pro Tip: When Python spawns a new process on Windows, it starts a fresh interpreter and imports your script from the top. Without if __name__ == "__main__", the spawn code at module level runs again inside the worker, creating another Pool, which tries to spawn more workers — an infinite recursive spawn that crashes immediately.

Fix 2: Fix Pickle Errors — Replace Lambdas with Named Functions

multiprocessing passes arguments and functions between processes using pickle. Lambdas, closures over local variables, and locally defined classes cannot be pickled:

Broken — lambda cannot be pickled:

from multiprocessing import Pool

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(lambda x: x * x, range(10))
        # PicklingError: Can't pickle <function <lambda>>

Fixed — use a named function defined at module level:

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(square, range(10))
    print(results)

Broken — closure captures local variable:

def make_multiplier(factor):
    def multiply(x):
        return x * factor  # Closes over `factor` — not picklable
    return multiply

if __name__ == "__main__":
    with Pool(4) as pool:
        double = make_multiplier(2)
        results = pool.map(double, range(10))  # PicklingError

Fixed — use functools.partial:

from multiprocessing import Pool
from functools import partial

def multiply(factor, x):
    return x * factor

if __name__ == "__main__":
    with Pool(4) as pool:
        double = partial(multiply, 2)  # partial is picklable
        results = pool.map(double, range(10))
    print(results)

Alternative — use pathos.multiprocessing which uses dill instead of pickle and supports lambdas:

pip install pathos

from pathos.multiprocessing import ProcessingPool as Pool

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(lambda x: x * x, range(10))
    print(results)

Fix 3: Always Close and Join the Pool

Not closing the Pool leaves worker processes running as zombies:

Broken — Pool not closed:

from multiprocessing import Pool

if __name__ == "__main__":
    pool = Pool(4)
    results = pool.map(my_func, data)
    # Pool never closed — worker processes become zombies

Fixed — use a context manager (recommended):

from multiprocessing import Pool

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(my_func, data)
    # Pool automatically closed and joined on exit
    print(results)

Fixed — manual close/join:

from multiprocessing import Pool

if __name__ == "__main__":
    pool = Pool(4)
    try:
        results = pool.map(my_func, data)
    finally:
        pool.close()  # No more tasks will be submitted
        pool.join()   # Wait for all workers to finish

pool.close() tells the pool no more work will be submitted. pool.join() blocks until all workers finish. Both are needed — close() alone does not wait for workers.

Fix 4: Fix Pool Hanging Indefinitely

If pool.map() hangs forever, common causes are:

A worker raised an exception that was swallowed:

def bad_worker(x):
    if x == 5:
        raise ValueError("Bad input!")  # Exception in worker
    return x * x

if __name__ == "__main__":
    with Pool(4) as pool:
        # pool.map re-raises worker exceptions in the parent
        try:
            results = pool.map(bad_worker, range(10))
        except ValueError as e:
            print(f"Worker failed: {e}")

pool.map() re-raises exceptions from workers. Use pool.map_async() with error callbacks to handle exceptions without blocking:

def handle_error(e):
    print(f"Worker error: {e}")

if __name__ == "__main__":
    with Pool(4) as pool:
        result = pool.map_async(bad_worker, range(10), error_callback=handle_error)
        result.wait(timeout=30)  # Don't hang forever
        if result.ready():
            print(result.get())

A worker is blocked on I/O or waiting for a lock:

# Broken — worker blocks on shared queue without timeout
def worker(queue):
    item = queue.get()  # Blocks if queue is empty
    return item

# Fixed — use timeout
def worker(queue):
    try:
        item = queue.get(timeout=5)  # Give up after 5 seconds
        return item
    except Exception:
        return None

Fix 5: Fix Shared State Between Processes

Unlike threads, processes do not share memory. Modifying a global variable in a worker has no effect on the parent:

Broken — expecting shared state:

from multiprocessing import Pool

results = []

def worker(x):
    results.append(x * x)  # Modifies the worker's own copy — parent never sees it

if __name__ == "__main__":
    with Pool(4) as pool:
        pool.map(worker, range(10))
    print(results)  # Always [] — workers modified their own copy

Fixed — return values instead:

from multiprocessing import Pool

def worker(x):
    return x * x  # Return the result

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(worker, range(10))
    print(results)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

For genuinely shared state, use multiprocessing.Manager or Value/Array:

from multiprocessing import Pool, Manager

def worker(shared_list, x):
    shared_list.append(x * x)

if __name__ == "__main__":
    with Manager() as manager:
        shared = manager.list()
        with Pool(4) as pool:
            pool.starmap(worker, [(shared, x) for x in range(10)])
        print(list(shared))

Common Mistake: Using a global list or dict to collect results from workers. Always return results from worker functions and let pool.map() collect them.

Fix 6: Fix multiprocessing with Classes

Methods of class instances are picklable in Python 3, but only if the class is defined at module level:

Broken — class defined locally:

def main():
    class Processor:
        def process(self, x):
            return x * 2

    p = Processor()
    with Pool(4) as pool:
        results = pool.map(p.process, range(10))  # PicklingError

Fixed — define class at module level:

class Processor:
    def process(self, x):
        return x * 2

if __name__ == "__main__":
    p = Processor()
    with Pool(4) as pool:
        results = pool.map(p.process, range(10))
    print(results)

Fix 7: Choose the Right Start Method

Python 3 supports three start methods:

import multiprocessing

# Check default
print(multiprocessing.get_start_method())  # 'fork' on Linux, 'spawn' on Windows/macOS

# Set explicitly (must be done before any Pool/Process creation)
if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")  # Safe, slow
    # or "fork"   — fast but unsafe with threads (default on Linux)
    # or "forkserver" — compromise

For PyTorch, CUDA, or multithreaded libraries, always use spawn or forkserver — fork after threading causes deadlocks:

if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")
    with Pool(4) as pool:
        results = pool.map(train_model, configs)

Still Not Working?

Check Python version differences. macOS changed the default start method from fork to spawn in Python 3.8. Code that worked on older Python or Linux may break on macOS 3.8+. Always use the if __name__ == "__main__" guard.

Check for recursive imports. If your worker function imports a module that imports __main__, it can trigger the spawn loop. Restructure imports so worker functions are in separate modules.

Use concurrent.futures.ProcessPoolExecutor as a higher-level alternative — it handles many of these edge cases more gracefully:

from concurrent.futures import ProcessPoolExecutor

def worker(x):
    return x * x

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=4) as executor:
        results = list(executor.map(worker, range(10)))
    print(results)

For async multiprocessing with asyncio, use loop.run_in_executor() as shown in Fix: Python asyncio not running.