Installing Numpy on Windows KLayout.

edited December 2017 in Python scripting
Hi Matthias,
I was wondering if you could provide more detail regarding how one could download and install numpy on a windows machine.
I understand that a binary build is required. But the only way I know how to install a package in python is to use pip.
Thanks.

Comments

  • edited November -1
    Hi Matthias:

    In Windows version 0.25, setting KLAYOUT_PYTHONPATH results in a runtime error:
    Fatal Python error: Py_Initialize: unable to load the file system codec
    ImportError: No module named 'encodings'

    I can circumvent it by instead adding sys.path.append statements in the Python script, like:
    sys.path.append('C:\\Users\\nxa11113\\PycharmProjects\\Layout\\LayerMap')

    So this way, I can, e.g., install Python 3.5.2 (the same version used for KLayout 0.25), use its pip to install modules (e.g. ply which I need), and then add its Lib/site-packages to the sys.path.

    However, this does not work for modules which need to be compiled, I think. So trying to add numpy this way results in a loading error.

    I think this is a corner case of a corner case for your app, but maybe the KLAYOUTPATH issue should be fixed at some point.

    Happy Christmas !!

    Best regards, Erwin
  • edited November -1
    So I am wondering whether one possible fix would be to provide the full Python distribution compiled with the same compiler as KLayout in Windows. I suspect this may work, then one can use pip to install modules and even binary modules would have the same makeup.
  • edited December 2017

    Hi Erwin,

    KLAYOUT_PYTHONPATH needs to include the default installation location for the encoding libraries because otherwise Python won't start up (I have not invented this - I also wish CPython was easier to integrate).

    In my case KLayout is installed locally for my user, so KLAYOUT_PYTHONPATH needs to be

    c:/Users/matthias/AppData/Roaming/KLayout/lib/pyton3.5;... other paths of yours ...
    

    Regarding your second question: you're not seriously suggesting to maintain a full Python distribution, are you? I packaged KLayout with a minimum Python to save users the effort to install Python themselves. That's all. Hence this limitation.

    The KLayout 0.25 package provided for download is built against Python 3.5.2-4 from the MSYS2/MINGW32 or MINGW64 system for 32 or 64bit respectively. This system provides everything you probably wish for - including NumPy. You can make KLayout use a separate MSYS2/MINGW installation, including additional packages you installed there.

    Here is the recipe for 64bit:

    • Install MSYS2 including MINGW64 from www.msys2.org
    • On the MINGW64 shell install Python 3.5.2-4 with NumPy 1.11.2-1 (this is the latest version that comes with Python 3.5):

      pacman -U http://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-python3-3.5.2-4-any.pkg.tar.xz

      pacman -U http://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-python3-numpy-1.11.2-1-any.pkg.tar.xz

    • Install other packages as you like - just make sure it's compatible with Python 3.5. http://repo.msys2.org/mingw/x86_64/ gives you a list of all packages you can install.

    Unfortunately MSYS2 is a rolling release, so you can't just take the latest version - unless you want to build KLayout against it yourself.

    Finally, you have to point KLayout to the MSYS installation. You can set KLAYOUT_PYTHONPATH to mirror the path from MSYS2's Python3 installation. Or you can permanently configure KLayout to use the new installation by editing ".python-paths.txt" next to "klayout_app.exe" (in C:/Users/matthias/AppData/Roaming/KLayout in my case). This is how this file needs to look like after I have installed MSYS/MINGW64 in the default location (c:/msys2-x86_64):

    [
      'C:/msys2-x86_64/mingw64/lib/python35.zip',
      'C:/msys2-x86_64/mingw64/lib/python3.5',
      'C:/msys2-x86_64/mingw64/lib/python3.5/plat-win',
      'C:/msys2-x86_64/mingw64/lib/python3.5/lib-dynload',
      'C:/msys2-x86_64/mingw64/lib/python3.5/site-packages'
    ]
    

    And - voilà - numpy is available in KLayout.
    BTW: the same scheme works for Ruby packages too. In this case, the paths are kept in ".ruby-paths.txt".

    Matthias

  • edited November -1

    Wow, that looks very complicated. I sure like the "pip install numpy" method on Linux and OSX. Seems like there should be an easier way to solve this: a) get users to install a Python of their choice (such as Anaconda), and have KLayout use that like in Linux (perhaps similar to what you describe above with MSYS2), or b) could you add numpy to KLayout?

    thanks
    Lukas

  • edited December 2017

    Hi Matthias,

    So I tried your procedure.

    Downloaded MSYS2 from the web page (msys2-x86_64-20161025.exe) - you mention you can't use the latest version -- is this the wrong one to use? I found older ones here: http://repo.msys2.org/distrib/x86_64/

    Found the path from msys64 / python3.5 using sys.path and set the environment in a command shell:

     set KLAYOUT_PATH=C:\msys64\mingw64\lib\python35.zip;
     C:\msys64\mingw64\lib\python3.5;C:\msys64\mingw64\lib\python3.5\plat-win;
     C:\msys64\mingw64\lib\python3.5\lib-dynload;C:\building\msys64\mingw64;
     C:\msys64\mingw64\lib\python3.5\site-packages
    

    KLayout started, and Python seems to work, but still no numpy:

     DLL load failed: The specified module could not be found.
     C:\msys64\mingw64\lib\python3.5\site-packages\numpy\core\__init__.py:14
     C:\msys64\mingw64\lib\python3.5\site-packages\numpy\lib\type_check.py:11
     C:\msys64\mingw64\lib\python3.5\site-packages\numpy\lib\__init__.py:8
     C:\msys64\mingw64\lib\python3.5\site-packages\numpy\add_newdocs.py:13
     C:\msys64\mingw64\lib\python3.5\site-packages\numpy\__init__.py:142
     (eval):1
    

    I also tried the .python-paths.txt version, but that didn't work (sys.path doesn't change after I update the file). Two differences: there is no python35.zip file in my installation, and the root folder for the default msys2 is now msys64.

    Another option - the files that need to be added for numpy, perhaps we could just add them into the KLayout python installation? Can you share these files?

    thanks

  • edited January 2018

    Hi Lukas,

    unfortunately MSYS2 version is "rolling" - the packages will be the "latest greatest" only. MSYS2 itself is less critical, but when the package repo gets updated, "pacman -U ..." is the only option left to install older packages.

    Regarding your problem:

    The environment variable is "KLAYOUT_PYTHONPATH", not "KLAYOUT_PATH". From the error message I'd say the package is found (numpy/core/__init__.py is executed), but the DLL (which is in lib-dynload) cannot be loaded. Maybe KLayout is 32bit? That would explain, why a 64bit DLL can't be loaded.

    The python35.zip isn't there is my case too. Maybe you can send me the .python-paths.txt file, so I can take a look why it's not read.

    Best regards,

    Matthias

  • edited January 2018

    Matthias,
    I was successful in changing the .python-path.txt file so that KLayout is using the MSYS2 python interpreter. However, still no luck in importing numpy. Same error message as Lukas. Numpy does work when used in the MSYS2 environment. Is there a way to get a more informative error message as to which DLL is failing to load? I don't think it is the multiarray dll but some dependency issue. (there error comes from the command (line 14 of numpy/core/init.py): from . import multiarray)

    thoughts?

    thanks,
    Jonas

  • edited November -1

    Hi,

    There's a line shown in the error message (line 14 in init.py). Whatever is there should tell you what DLL wants to be loaded. I assume "lib_dynload" is . If you run a 32 bit KLayout, it can only load 32 bit DLL's. Same for 64 bit. So even if the DLL is there, it may not be possible to load it.

    I could give you more advise on Linux - but numpy isn't an issue there (just install the package. done).

    Matthias

  • edited November -1

    Hi Matthias,

    We have successfully added numpy for KLayout-Windows to SiEPIC-Tools (hosted in KLayout Package Manager).

    It's a never-ending battle though with Windows and the version of Python you chose to compile with as it doesn't come with pip. Next I can't get the Python XML module working (again works fine in Linux/OSX) but it seems to be missing important DLLs in Windows.

    I am thinking of creating a separate Package something like "KLayout-Windows missing packages", which my list presently includes: numpy, requests and several of its dependancies (urllib3, cerifi, charet, idna), xml (import expat isn't working).

  • edited November -1
    Hi LukasC, it may be easier to take a fully debugged Windows Python distribution (e.g. Anaconda) and just compile KLayout with the same compiler. Then you get all the Python packages and all you have to do is to compile KLayout. I haven't tried it (because I just need pure Python modules) but there is a Podcast at:
    https://talkpython.fm/episodes/show/94/guarenteed-packages-via-conda-and-conda-forge
    which describes the packaging issue in Python on Windows.
  • edited November -1

    Hi ejprinz,

    I agree that using a Windows Python distribution like Anaconda would be a good idea. I am not a Windows user however, so do not wish to take on the task of recompiling KLayout on Windows...

    My interest in Windows KLayout + Python is to support other users who need it. The solution needs to be easy (like presently) without requiring users to compile anything. It could in principle be a solution that requires people to install Anaconda first, then KLayout. Or KLayout can include Python from the latest stable Anaconda.

  • edited November -1

    Hi all,

    my scope is MSYS2. IMHO that's an equivalent solution to Anaconda. If I had infinite resources I'd support both.

    In general, building shouldn't be an issue on both systems as long as you have bash and a few packages (gcc/clang, qt5, python, zlib, make).

    Matthias

Sign In or Register to comment.