klayout install on Linux - system python vs venv

fawqati

I believe, KLayout by default, points to the system python installation.

Is there a way of making KLayout use a python virtual environment (venv) ?

fawqati

Any feedback would be much appreciated ?!

Matthias

Yes, there is.

You can use "KLAYOUT_PYTHONPATH" to point KLayout to the virtual environment. This replaces the standard "PYTHONPATH" variable.

BUT: this will only switch the libraries. As KLayout is already built against a certain Python version (usually the system version), this will not switch the Python interpreter. So you cannot create a virtual environment with a different (non-system) Python interpreter and use that for KLayout. This combination is likely to crash.

If you want full control, you have to drop the UI and use the KLayout Standalone Python module from PyPI. Just do:

pip install klayout

inside your venv.

Matthias

fawqati

Hi Matthias,

We couldnt get the "KLAYOUT_PYTHONPATH" pointing to the venv to pick up the packages in the virtual environment.

We tried setting the env variable to the .venv, .venv/bin, .venv/lib/python3.11 and venv/lib/python3.11/site-packages directories and none of those worked either !

Any feedback / insights would be greatly appreciated.

Ps. In the meantime, we're trying to get you a small testcase.

Faisal

Matthias

Hi @fawqati,

this is how it works for me on Ubuntu-24 with Python 3.12 and klayout-0.29.6, built locally for Ubuntu-24:

# after activating the venv
KLAYOUT_PYTHONPATH=${VIRTUAL_ENV}/lib/python3.12/site-packages klayout

# batch mode
KLAYOUT_PYTHONPATH=${VIRTUAL_ENV}/lib/python3.12/site-packages klayout  -b -r script.py

I could install pandas locally and import it in KLayout.

Matthias

fawqati

Good morning Matthias,
That worked. We are so grateful. A BIG THANK YOU !!

sekigawa

Hello folks,

I have a question (suggestion) related to these posts.

My Environment

OS: Linux Mint 20.3
Python: 3.12.6+ (built from the source with "--enable-shared")
KLayout: 0.29.7 (built from the source)

I invoke the klayout executable via the below Bash script.

SCRIPT_DIR=$(cd $(dirname $(readlink -f $0 || echo $0));pwd -P)

# Python 3.12 virtual environment
export VIRTUAL_ENV=$HOME/opt/KLPy312
export KLAYOUT_PYTHONPATH=$VIRTUAL_ENV/lib/python3.12/site-packages
export PYTHONNOUSERSITE=1

KLAYOUT_EXE=$SCRIPT_DIR/klayout

unset LD_LIBRARY_PATH
eval "$KLAYOUT_EXE -style=fusion $*"

---

In the IDE

The below macro unexpectedly runs successfully in the IDE.


This is unexpected because I have not yet installed the scipy module into the virtual environment, as shown below.

(KLPy312) Mint{sekigawa}(1)$ pip3 list
Package            Version
------------------ -----------
:
matplotlib         3.9.2
:
numpy              2.1.1
:
pandas             2.2.3
:
pip                24.2
:
requests           2.32.3
setuptools         75.1.0
six                1.16.0
tqdm               4.66.5
:
XlsxWriter         3.2.0

In fact, scipy is retrieved from the system directory.

This is reasonable since sys.path contains the system site-packages directory.

---

In the Terminal

import scipy fails as expected.

Mint{sekigawa}(1)$ KLPy312

(KLPy312) Mint{sekigawa}(2)$ python3
Python 3.12.6+ (heads/3.12:c60d97805f7, Sep 25 2024, 12:47:26) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import scipy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'scipy'
:
>>> import sys
>>> for item in sys.path:
...   print(item)
...

/usr/local/lib/python312.zip
/usr/local/lib/python3.12
/usr/local/lib/python3.12/lib-dynload
/home/sekigawa/opt/KLPy312/lib/python3.12/site-packages
>>>

---

KLayout's current behavior provides a fail-safe mechanism when modules are not found in the virtual environment, but as seen above, it can also be a source of confusion.

My first idea is to change the contents of sys.path depending on the environment variables used.
1. KLAYOUT_PYTHONPATH: current behavior (the fail-safe mechanism persists)
2. KLAYOUT_PYTHONPATH_STRICT: eliminate the system site-packages directory from sys.path

Another idea is:
1. KLAYOUT_PYTHONPATH: eliminate the system site-packages directory from sys.path (always strict)

What do you think of it?

Best regards,
Kazzz-S

Matthias

Hi @sekigawa,

The virtual environment implementation of Python is not fully compatible with embedding Python. As I understand the implementation, it is based on the call path of the Python executable. If you create a virtual environment, a link to the system interpreter is created and is added to PATH. If you call Python from there, it will initialize sys.path differently.

That is a problem for embedded Python, as the binary is not the one of the original Python interpreter. So it will stay connected to the system Python.

To fully disconnect KLayout from the system Python, you can set KLAYOUT_PYTHONHOME to some non-existing location - i.e. "/dev/null". However, in that case, the system encoding libraries are no longer found. One needs to add them again, in addition to the venv-local site-packages folder.

This creates an isolated environment for me on Ubuntu 24:

KLAYOUT_PYTHONHOME=/dev/null KLAYOUT_PYTHONPATH=.venv/lib/python3.12/site-packages:/usr/lib/python3.12 klayout ...

Matthias

sekigawa

Hi @Matthias,

Thanks for your comments and clarification.

Kazzz-S