import numpy

ashkan · November 2023

Hi,

I recently tried to rerun some of my old codes (maybe from a year ago), and surprisingly, they didn’t compile! In some of the codes, I imported the numpy library, but Klayout returned this error:

ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

Did Klayout change anything that prevents numpy from being imported?
My Klayout version is 0.28.13.

To add to this problem, this issue appear only in Windows version and not in Ubuntu version.

Matthias · November 2023

Hi @ashkan,

I am regularly upgrading the build environment as including new components forces me to do so. This means that also numpy versions change.

In 0.28.12 numpy was 1.21.0. In 0.28.13, numpy is 1.26.1.

I am not really familiar with numpy and I have no idea what this message means. But plain "import numpy" on the console works, so I don't think there is a general issue. Maybe you did set a sys.path that numpy does not like? Or maybe the version makes a difference?

Regards,

Matthias

sekigawa · November 2023

I'm having the same problem in both the Windows binary distribution and the MINGW64 source build.

1. Comparison of Different Builds

2. Initial Survey
https://github.com/numpy/numpy/issues/24965#issuecomment-1772872708

Somehow you've ended up with an installation of numpy where from numpy.__config__ import show fails.
Normally that only happens if numpy is imported from inside the numpy source tree.
It sounds like you installed numpy via pip, so that shouldn't be happening, which makes me think
you have a broken install somehow.

I see some minGW strings in your debug output
 - minGW isn't supported by us and isn't tested in our CI so there may be issues we aren't aware of.
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
I doubt anyone will spend a lot of time working on this so you'll likely have to dig in a little
bit more into what's going wrong.

Despite the above comments,

3. Reinstallation
I uninstalled then reinstalled numpy and the packages that depend on it, but the problem remained unchanged.

4. Values of sys.path
I also checked the sys.path and found that the source build was weird.

5. KLayout 0.28.13 on different MacOS
No problems.

Best regards,
Kazzz-S

sekigawa · December 2023

6. Values of sys.path (again)

I edited the contents of bin-release/.python-paths.txt and got the expected path settings.

sekigawa · December 2023

7. Further investigation(1)

The original exception message seems inappropriate, so I have modified numpy/__init__.py and got different messages.

To be continued...

Matthias · December 2023

Hi @sekigawa,

thanks for these investigations.

I could not reproduce the problems myself yet, but I will try with different machines. The ones I tested had numpy installed on them.

I actually did not change anything between 0.28.12 and 0.28.13 except adding libgit2 and upgrading to latest MSYS. I did not a few times before and never ran in trouble with numpy.

Matthias

Matthias · December 2023

I could reproduce the problem after wiping my system entirely. So I am suspecting some essential DLL is missing.

sys.path looks normal to me.

I will try to find the missing pieces ...

Matthias

Matthias · December 2023

Looks like for some reason the essential DLLs are missing in the installation. I assume the packaging script has some trouble with the new MSYS version. I will try to fix that and provide new Windows packages.

Matthias

Matthias · December 2023

I stand corrected: the DLLs are now called ".pyd" - so they are there. It's a different problem then.

Matthias

Matthias · December 2023

Okay, found the issue: some DLLs that the .pyd depend on are not installed. I got it working by copying

/mingw64/bin/libopenblas.dll
/mingw64/bin/libgomp-1.dll
/mingw64/bin/libgfortran-5.dll
/mingw64/bin/libquadmath-0.dll

to the same place than "_multiarray_umath.cp311-mingw_x86_64.pyd".

The reason for the missing DLLs is that the build script collects dependent DLLs, but only analyzes ".dll" files, not ".pyd".

I think that is easy to fix. I'll provide a new Windows installer.

Matthias

Matthias · December 2023

I have created a ticket for this here: https://github.com/KLayout/klayout/issues/1553

Matthias

sekigawa · December 2023

Hi @Matthias,

Thanks for your effort

8. Further investigation(2)
After manually copying the four DLLs (in the source build env), I can import numpy normally.
I hope the other cases have the same root cause.

Best regards,
Kazzz-S

Matthias · December 2023

I fixed the deployment script and uploaded new installers. Version is still 0.28.13 because only DLLs got added. Please re-install using the Windows installers from https://www.klayout.de/build.html.

I can import numpy, scipy, pandas, matplotlib and other packages now.

Thanks for these investigations.

Matthias

sekigawa · December 2023

Hi @Matthias,

Thanks for the quick fix.
I've tested the new Windows ZIP package and found it OK!

Kazzz-S

Matthias · December 2023

Very good. I'm happy to hear it works on your side too!

Best regards,

Matthias

ashkan · December 2023

@Matthias It works on my device too.
Thanks.

ashkan · December 2023

@Matthias I'd like to report a possible bug in KLayout 0.28.13. I am not sure if it is related to my version, but I tried it on version 0.28.5 and it works fine. The problem occurs when I try to move an object with the Movements setting in Diagonal mode. I cannot move the object to any angle between 90 and 180 degrees or between 270 and 360 degrees. If I try to do so, the object’s movement will be mirrored on the X-axis. Could you please check on this issue and let me know if there is a solution?

Edit1:
Another issue that I have noticed is that when I try to move an array of objects by clicking on the desired location, they will suddenly jump to a different place.

Edit2:
I have found another strange behavior when I use a Pcell and then flatten it. If I modify anything on that specific Pcell in its layout, all other Pcells also change accordingly!

Matthias · December 2023

Please use the issue tracker on GitHub for bug reports (https://github.com/KLayout/klayout). This discussion is about a different topic.

But I guess you're not facing a bug. Instead you are editing objects inside the hierarchy and that will make all other places, where this object is used, change as well. This is the magic of hierarchical layout.

Turn on "Select Top Level Objects" in the "View" menu, so you can only select objects from the top cell. This will avoid these issues, but you have to enter subcells if you want to do edits there. "Descend in hierarchy" (Ctrl+D) is a way to do so and it is comparable with "Edit in Place" in Cadence.

Matthias

Long · March 19

@Matthias I just installed ST-klayout-0.29.12-macOS-Sequoia-1-qt5MP-RsysPsys in my Mac running Sequoia 15.3.2. I have no problem loading numpy in Terminal. When I try to load numpy in the KLayout macro environment, I get an error importing numpy. The error lead me to this forum post. Can you help?

Thanks, Long

sekigawa · March 21

Hi @Long,

Since you can successfully start the application, you installed the Apple Command Line Tools correctly.
However, some optional Python packages, including numpy, are not correctly installed.
I guess you have installed another Python3 through Anaconda3 (most likely), Homebrew, or MacPorts.
The Python3 you are running in your Terminal does not match the one KLayout uses.
See the command executions and their outputs below for details.

Sequoia1{kazzz-s} MacOS (1)% pwd
/Applications/klayout-std.app/Contents/MacOS (I renamed the application bundle.)

Sequoia1{kazzz-s} MacOS (2)% ll
total 200
drwxr-xr-x@  7 kazzz-s  admin    224 Mar  8 11:07 .
drwxr-xr-x@  9 kazzz-s  admin    288 Mar  8 11:06 ..
drwxr-xr-x@ 10 kazzz-s  admin    320 Mar  8 11:06 db_plugins
-rwxr-xr-x@  1 kazzz-s  admin  95288 Mar  8 11:07 klayout <=== KLayout's executable
drwxr-xr-x@ 16 kazzz-s  admin    512 Mar  8 11:06 lay_plugins
drwxr-xr-x@  4 kazzz-s  admin    128 Mar  8 11:00 pymod
-rwxr-xr-x@  1 kazzz-s  admin     87 Dec 14 11:03 qt.conf

To find the Python3 library, which KLayout depends on...
Sequoia1{kazzz-s} MacOS (3)% otool -L klayout | grep Python
    @rpath/Python3.framework/Versions/3.9/Python3 (compatibility version 3.9.0, current version 3.9.0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sequoia1{kazzz-s} 3.9 (4)% pwd
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9
                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sequoia1{kazzz-s} 3.9 (5)% ll
total 4944
drwxr-xr-x   10 root  wheel      320 Dec 13 19:51 .
drwxr-xr-x    4 root  wheel      128 Dec 13 19:51 ..
drwxr-xr-x  102 root  wheel     3264 Dec 13 19:51 Headers
-rwxr-xr-x    1 root  wheel  5815472 Dec  7 16:16 Python3 <=== symbolic link from lib/libpython3.9.dylib
drwxr-xr-x    5 root  wheel      160 Dec 13 19:51 Resources
drwxr-xr-x    3 root  wheel       96 Dec 13 19:51 _CodeSignature
drwxr-xr-x    8 root  wheel      256 Dec 13 19:51 bin
drwxr-xr-x    5 root  wheel      160 Dec 13 19:51 include
drwxr-xr-x    5 root  wheel      160 Dec 13 19:51 lib
drwxr-xr-x    3 root  wheel       96 Aug 10  2024 share

Sequoia1{kazzz-s} MacOS (6)% which python3 <=== the python3 command used below
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3

Sequoia1{kazzz-s} MacOS (7)% python3
Python 3.9.6 (default, Nov 11 2024, 03:15:39)
[Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__file__
'/Library/Python/3.9/site-packages/numpy/__init__.py' <=== numpy package required
>>>

How to use pip to install a new package using the python3 command above...
Use the full path if required.
Sequoia1{kazzz-s} temp (1)% [sudo] python3 -m pip install a_new_module

To newly install numpy...
Sequoia1{kazzz-s} temp (2)% [sudo] python3 -m pip install numpy

In my case...

Sequoia1{kazzz-s} temp (3)% python3 -m pip list --outdated (some packages are outdated)
Package             Version Latest  Type
------------------- ------- ------- -----
altgraph            0.17.2  0.17.4  wheel
fonttools           4.55.2  4.56.0  wheel
future              0.18.2  1.0.0   wheel
importlib_resources 6.4.5   6.5.2   wheel
klayout             0.29.10 0.29.12 wheel
macholib            1.15.2  1.16.3  wheel
matplotlib          3.9.3   3.9.4   wheel
pillow              11.0.0  11.1.0  wheel
pyparsing           3.2.0   3.2.1   wheel
pytz                2024.2  2025.1  wheel
setuptools          58.0.4  77.0.3  wheel
six                 1.15.0  1.17.0  wheel
tzdata              2024.2  2025.1  wheel
wheel               0.37.0  0.45.1  wheel

Sequoia1{kazzz-s} temp (4)% python3 -m pip install --upgrade altgraph fonttools .... tzdata wheel
:
:
Sequoia1{kazzz-s} temp (5)% python3 -m pip list --outdated
(empty)

Sequoia1{kazzz-s} temp (6)% python3 -m pip list
Package             Version  (the latest as of 2025-03-21)
------------------- -----------
altgraph            0.17.4
contourpy           1.3.0
cycler              0.12.1
fonttools           4.56.0
future              1.0.0
importlib_resources 6.5.2
kiwisolver          1.4.7
klayout             0.29.12
macholib            1.16.3
matplotlib          3.9.4
numpy               2.0.2 <===
packaging           24.2
pandas              2.2.3
pillow              11.1.0
pip                 25.0.1
pyparsing           3.2.1
python-dateutil     2.9.0.post0
pytz                2025.1
scipy               1.13.1
setuptools          77.0.3
six                 1.17.0
tzdata              2025.1
wheel               0.45.1
zipp                3.21.0

The test CSV file sampleCSV.csv is not included in the DMG files.
You can use any (UTF-8 encoded and "," separated) one.

Good luck.
Kazzz-S

sekigawa · March 21

Hi @Long,

The actual cause may be different from what I imagined initially.
Please let me know the results by comparing them with my environment.

Cont.

Where does @rpath point?

Sequoia1{kazzz-s} MacOS (1)% export DYLD_PRINT_RPATHS=1

Sequoia1{kazzz-s} MacOS (2)% otool -l klayout | grep -A2 LC_RPATH
          cmd LC_RPATH
      cmdsize 72
         path /Library/Developer/CommandLineTools/Library/Frameworks (offset 12) <===
--
          cmd LC_RPATH
      cmdsize 48
         path @executable_path/../Frameworks (offset 12)

Re-installation of the numpy package
I uninstalled the existing numpy package with sudo and reinstalled it to a different directory.
It still works as expected.

Sequoia1{kazzz-s} ~ (3)% python3
Python 3.9.6 (default, Nov 11 2024, 03:15:39)
[Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for p in sys.path:
...     print(p)
...

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python39.zip
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/lib-dynload
/Users/kazzz-s/Library/Python/3.9/lib/python/site-packages <===
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
/Library/Python/3.9/site-packages
>>>

sekigawa · March 22

Hi @Long,

I tried to use a virtual environment as follows.
This also works fine.

Setup the virtual environment

Sequoia1{kazzz-s} ~ (1)% which python3
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3

Sequoia1{kazzz-s} ~ (2)% python3 -m venv ~/opt/KLstd
                                         ^^^^^^^^^^^ the target directory

Sequoia1{kazzz-s} ~ (3)% source ~/opt/KLstd/bin/activate

(KLstd) Sequoia1{kazzz-s} ~ (4)% which python3
/Users/kazzz-s/opt/KLstd/bin/python3

(KLstd) Sequoia1{kazzz-s} ~ (5)% which pip3
/Users/kazzz-s/opt/KLstd/bin/pip3

(KLstd) Sequoia1{kazzz-s} ~ (6)% pip3 list
Package    Version
---------- -------
pip        21.2.4
setuptools 58.0.4
WARNING: You are using pip version 21.2.4; however, version 25.0.1 is available.
You should consider upgrading via the '/Users/kazzz-s/opt/KLstd/bin/python3 -m pip install --upgrade pip' command.

(KLstd) Sequoia1{kazzz-s} ~ (7)% pip3 install --upgrade pip
:

(KLstd) Sequoia1{kazzz-s} ~ (8)% pip3 list
Package    Version
---------- -------
pip        25.0.1
setuptools 58.0.4

(KLstd) Sequoia1{kazzz-s} ~ (9)% pip3 list --outdated
Package    Version Latest Type
---------- ------- ------ -----
setuptools 58.0.4  77.0.3 wheel

(KLstd) Sequoia1{kazzz-s} ~ (10)% pip3 install --upgrade setuptools
:

(KLstd) Sequoia1{kazzz-s} ~ (11)% pip3 list --outdated
(empty)

(KLstd) Sequoia1{kazzz-s} ~ (12)% pip3 install numpy scipy pandas matplotlib
:

(KLstd) Sequoia1{kazzz-s} ~ (13)% pip3 list
Package             Version
------------------- -----------
contourpy           1.3.0
cycler              0.12.1
fonttools           4.56.0
importlib_resources 6.5.2
kiwisolver          1.4.7
matplotlib          3.9.4
numpy               2.0.2
packaging           24.2
pandas              2.2.3
pillow              11.1.0
pip                 25.0.1
pyparsing           3.2.1
python-dateutil     2.9.0.post0
pytz                2025.1
scipy               1.13.1
setuptools          77.0.3
six                 1.17.0
tzdata              2025.1
zipp                3.21.0

(KLstd) Sequoia1{kazzz-s} ~ (14)% python3
Python 3.9.6 (default, Nov 11 2024, 03:15:39)
[Clang 16.0.0 (clang-1600.0.26.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__file__
'/Users/kazzz-s/opt/KLstd/lib/python3.9/site-packages/numpy/__init__.py'
>>>

Use the virtual environment in KLayout

Using the Automator.app application, create a launch service app as follows.

#---------------------------------------------------------------------------------------
# Bundle: KLayoutStdVenv.app
#
# Descriptions:
#  This script invokes "klayout.app" that shares the system Ruby and Python, where
#  Python (3.9) uses an virtual environment.
#---------------------------------------------------------------------------------------

#--------------------------------------------------------------------------
# Localizing the language
# Workaround discussed in https://github.com/KLayout/klayout/issues/1213
#--------------------------------------------------------------------------
export LANG=$(defaults read -g AppleLocale).UTF-8

#--------------------------------------------------------------------------
# Your setups
#--------------------------------------------------------------------------

#--------------------------------------------------------------------------
# Export the 'KLAYOUT_GIT_HTTP_PROXY' environment variable if you need to
# use the git-based Salt Package Manager through a proxy server.
#--------------------------------------------------------------------------
#export KLAYOUT_GIT_HTTP_PROXY="http://111.222.333.444:5678"

#--------------------------------------------------------------------------
# Downloading data from the package server might time out (default=10 sec).
# If so, set the 'KLAYOUT_HTTP_TIMEOUT' environment variable.
# For example, to make a timeout of 25 seconds,
#--------------------------------------------------------------------------
export KLAYOUT_HTTP_TIMEOUT=25.0

#--------------------------------------------------------------------------
# Export the 'KLAYOUT_PYTHONPATH' environment variable if required
# Ref. https://www.klayout.de/forum/discussion/2557/
#--------------------------------------------------------------------------
export VIRTUAL_ENV=$HOME/opt/KLstd
export KLAYOUT_PYTHONPATH=$VIRTUAL_ENV/lib/python3.9/site-packages

#--------------------------------------------------------------------------
# With "-n" option, you can invoke multiple instances
# Note that I changed the original application bundle name: 'klayout.app'
#--------------------------------------------------------------------------
myklayout=/Applications/klayout-std.app
myconfig=$HOME/.klayout/klayoutrc

open -n -a $myklayout --args -e -c $myconfig -style=fusion $@

The critical point is correctly setting the KLAYOUT_PYTHONPATH environment variable before invoking the KLayout application bundle.

Check your sys.path

Long · March 25

Hi @sekigawa, I thought about trying out the HW-klayout-0.29.12-macOS-Sequoia-1-qt5MP-RsysPhb311.dmg on my way to work. After installing it and then numpy (according to the ReadMeFirst), I was able to load it! Then I logged onto this forum to find your excellent guide to setup a virtual environment. Is this effectively what the HW version does?

I developed a package of macros and PCells (https://github.com/UUhy/LTK, https://uhnf.egr.uh.edu/nanofabrication/lithography/klayout/digitize-graph/) a few years ago and uploaded it to salt. It was difficult for me to help whenever someone installs my package and it doesn't work because I just don't know how application find dependencies it needs on other folk's computer. Perhaps telling them to install the HW version is the solution?

Thank you so much Kazzz-S,
Long

sekigawa · March 26

Hi @Long,

As you mentioned, the steps explained in Homebrew-User-ReadMe First.txt (HW-*.dmg) and the ones regarding the virtual environment setup (described above) are effectively equivalent.

Academic people originally conceived the idea for HW (all-in-one). I understand they wanted to make KLayout and its extensions easy for students unfamiliar with software development environment administration.

However, if you can set up a virtual environment as described above (I don't think it's that difficult), I strongly recommend it.
We must create a launch service for macOS, but this is not difficult either.
Using a virtual environment is clean, safe, and flexible. It is beneficial for projects that require trial and error compared to installing different packages globally.

Also, Homebrew's Python 3.12 and later prohibits installing packages globally via pip.
For this reason, HW-klayout-0.29.12-macOS-Sequoia-1-qt5MP-RsysPhb311.dmg embeds Python 3.11.
On the other hand, LW-klayout-0.29.12-macOS-Sequoia-1-qt6Brew-Rhb34Phb312.dmg is built to use Homebrew's newer Python 3.12 and a virtual environment.

Kazzz-S

Howdy, Stranger!

Categories

import numpy

Comments

Howdy, Stranger!

Quick Links

Categories

import numpy

Comments