multiprocessing throw's an error

So I am testing various aspect's of Python in Klayout
Below code is a very basic multiprocessing example found pretty much every where.
The code at p.start() throws the following error.
[WinError 2] The system cannot find the file specified
C:\Users\Tracy\AppData\Roaming\KLayout\lib\python3.7\multiprocessing\popen_spawn_win32.py:72
C:\Users\Tracy\AppData\Roaming\KLayout\lib\python3.7\multiprocessing\context.py:322
C:\Users\Tracy\AppData\Roaming\KLayout\lib\python3.7\multiprocessing\context.py:223
C:\Users\Tracy\AppData\Roaming\KLayout\lib\python3.7\multiprocessing\process.py:112
C:/Users/Tracy/KLayout/pymacros/new_python_file_35.py:21

Ideas , comments .
Tracy

# Enter your Python code here
from multiprocessing import Process
import os

def info(title):
    print(title)
    print('module name:', __name__)
    print('parent process:', os.getppid())
    print('process id:', os.getpid())

def f(name):
    info('function f')
    print('hello', name)

if __name__ == '__main__':
    info('main line')
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

Comments

  • edited March 26

    Don't do multiprocessing in KLayout ;). I fought through this until I realized Qt is safeguarding against it and I don't think there is an easy way out. Simply put Qt won't let you just spawn chilprocesses this way. Reason is simple: multiprocessing uses fork() in the background, so essentially cloning your program stack. Works wonderfully for small python scripts. Doesn't work well for much bigger things like KLayout (and in KLayout case I am sure it just wouldn't work due to the GUI part). I have found a way around it: use shared memory and have a KLayout spawn a daemon that processes your data (through subprocess). It's not pretty, it's not fast, but it did its job reasonably for what I wanted it to do.

    You could try to spawn a new thread through Qt to do it, but for me that didn't work when I tried.

    If you want inspiration, my old (and for some reason at the moment defunct) code is here:
    https://github.com/sebastian-goeldi/KLayoutPhotonicPCells-core/blob/master/python/kppc/drc/init.py#L90

    My advice: If you can somehow fit it into a DRC script, use that. There you have the Tilingprocessor that you can use to multithread.

  • Sebastian,

    Thanks you have saved my mouse much grief :D ;)

  • @sebastian Thanks from my side too. You saved me some research in this topic :)

    I never used multiprocessing myself. Is "Process" doing what I think it does? Then maybe it's equivalent to QProcess?

    Matthias

  • Yes and no. Multiprocessing in python came from the neverending story about the Global Interpreter Lock (GIL). Since python cannot do true multi-threading (since only every one thread can be active due to the lock).

    Warning the next part is my understanding of multiprocessing and not really accurate (probably):
    The way around this is multiprocessing. This will spawn new python processes. This new process has a cloned stack from the original process it forked from with just a flag about which process number it is. This process will the normally follow the stack and in the end the results are passed back to the original process. So this means you are copying the whole memory of the original process, which is probably what causes the grieve with Qt.

    I think QProcess is way smarter than this. But I haven't been able to actually start any python process with it (can't remember the exact problem, but the errors were very confusing), so at some point I resorted to spawn external helpers. If @Matthias thinks QProcess can be used to execute python code, I would be thrilled to use it (also QThread for that matter) :) , also from ruby if the python way shouldn't work.

  • The things we learn and the thing's we don't about know Klayout, it keeps getting
    interesting , down that rabbit hole I go ..

  • Well, I think QProcess can fork too, but forking is a special technique and with QProcess it's up to you to join the results again.

    I have not seen fork used productively in recent years. Today's solutions go for distributing a task over a number of network-connected services in a cloud arrangement. This will scale much further than the number of cores.

    Matthias

Sign In or Register to comment.