›

3D Viewer & Salt dependency managment

toktok
in Python scripting

Hi,
Recently I wrote a Python script for viewing layouts as 3D models in OpenSCAD. Initially it was just for fun but then I realized it's also quite helpful for debugging. Therefore I thought I'd best package it as a Salt module.

And here comes my question: Can Salt manage Python dependencies (PyPI/pip packages)? How do I tell Salt about them?

«1

Comments

  • MatthiasMatthias

    Hi tok,

    thanks for considering sharing your work!

    However, you cannot tell Salt about this currently. But you can document this dependency of course.

    An automated option (I have not tried myself) to take care of this is using an autorun script inside the module. Let this script run once (use your own configuration variable to mark it run). In this script you can try to pull the package from PyPI. But I suspect many side effects and error handling is crucial. Otherwise there is risk to damage a user's installation.

    But I'd be careful to require a specific Python module. Unless the module is a very common one, the dependency isn't easy to fulfil on all platforms.

    Regards,

    Matthias

  • toktok

    Thanks for answering! Then I best let the users manage the dependencies. They need to install OpenSCAD manually anyway.

  • toktok

    BTW: It is called 'gds3xtrude' and should be available from the package manager.
    However, the dependencies must be installed manually first:
    pip3 install --user gds3xtrude
    pip3 install --user klayout # Only if gds3xtrude is used as a standalone tool.

    More documentation is here: https://codeberg.org/tok/gds3xtrude

  • MatthiasMatthias

    Awesome! Thanks for providing this!

    Best regards,

    Matthias

  • toktok

    If somebody would like to use Blender instead of OpenSCAD, let me know. I started experimenting and I would be motivated to push the project if somebody is interested.
    Blender would be useful for good-looking renderings and illustrations.

  • allevaalleva
    edited May 2019

    Setting the correct path on windows(10) is a bit confusing. If you create a blank macro:

    import sys
print(sys.path)

    The interpreter returns the path set in:
    C:\Users[user-name]\AppData\Roaming\KLayout.python-paths.txt

    This is not the same as the 'path' environmental variable as returned by the python3 install:
    C:\Program Files (x86)\Python36-32\python.exe

    Does the macro environment use a different python interpreter? Why is the path variable different to sys.path?

    Note:
    The dependencies are correctly installed under C:\Program Files (x86)\Python36-32\ but the macro environment does not see this package location. Manually adding the dependencies to
    C:\Users[user-name]\AppData\Roaming\KLayout.python-paths.txt
    did not resolve the issue.

    Cheers
    Andrew

  • MatthiasMatthias

    Hi Andrew,

    Does this post belong to this discussion? If not, please open a new one.

    KLayout comes with it's own Python interpreter and does not take any interpreter provided by the system. This leads to manifold issues usually due to incompatibilities of the binary APIs. And KLayout also does not read PYTHON_PATH, because some programs set this variable to their own libraries on installation. Hence "python-paths.txt".

    I don't see what you're trying to achieve. What is the issue?

    Matthias

  • allevaalleva

    Hi Matthias,

    My comment relates to "Can Salt manage Python dependencies" and "dependency isn't easy to fulfil on all platforms.". Specifically, I was trying to get gds3xtrude up and running on a windows box.

    I use both Windows and Linux environments for development. Setting up gds3xtrude on Linux is straight forward (setting aside a permissions issue). The instructions (above) when run on a Windows box will install the packages on the system, but the KLayout interpreter does not see the install location:

    pip3 install --user gds3xtrude
pip3 install --user klayout # Only if gds3xtrude is used as a standalone tool.

More documentation is here: https://codeberg.org/tok/gds3xtrude

    Using "python-paths.txt" to point to the system package location may create a bit of a mess (especially if the system is running a different version of python).

    My solution was to switch back to Linux. This is now a heads-up to anyone trying to get gds3xtrude running on windows. It would be helpful if the install process was more self contained, but I understand the reasons why this isn't the case.

    Cheers
    Andrew

  • toktok

    Unfortunately I don't have a Windows installation to reproduce the issue.
    But I heard from other people who had difficulties because they had python3.7 installed but KLayout has python3.5 built-in. Therefore, when running pip3 the packages go to a directory explicitly for python3.7 and cannot be found by python3.5. The solution in this case is to install python3.5 if it is not already there and run pip3.5 instead of pip3.

  • allevaalleva

    Thanks Tok, and thanks for sharing your work!

    re. because they had python3.7 installed but KLayout has python3.5 built-in

    I'm also running python3.7, so likely the same issue. On a windows box, after installing python3.5, the dependencies can be installed using py and the -m switch:

    py -3.5 -m pip install --user gds3xtrude
https://docs.python.org/3/installing/index.html

    the equivalent for a Linux box would be:

    python3.5 -m pip install --user gds3xtrude
  • MatthiasMatthias

    Hi all,

    thanks for this discussion. Actually on Windows, the problem is much deeper than just Python versions. There is also an incompatibility between the binary APIs of different builds (based on gcc/MinGW, MSVC etc.). For example, KLayout builds with MinGW while Anaconda Python comes for MSVC 2017. As these versions are incompatible and must never by mixed, KLayout tries it's best to become independent from other installations. Therefore it will ignore PYTHONPATH and use it's own path (KLAYOUT_PYTHONPATH) and use python_path.txt to configure it's own environment.

    This makes mixing in other packages difficult.

    You can mix in Python packages that come with KLayout's build system (which is MSYS2). But as this is a rolling release, it's likely that you won't find a package for the version KLayout was built with.

    I established MSVC compatibility for the master branch which means you can mix with Anaconda packages. This should make things easier then.

    But frankly, my goal is to provide a good end user tool, not becoming an expert in Python installations in particular and the whole DevOps stuff in general. This consumes too much energy which is lost for real development. I'm setting priorities for myself.

    Matthias

  • sbharadwajsbharadwaj

    Hi,

    I am attempting to use gds3xtrude on either Mac or windows, and get the same error on both:

    gds3xtrude python module not installed (or not in the required version).
    Please run: pip3 install --upgrade --user gds3xtrude

    I used pip to install the package, but I guess that I'm running into an issue similar to the one described above. Is there any update on how to work around this? Sorry if this isn't the right place to leave this comment.

    Thanks!

  • sbharadwajsbharadwaj
    edited March 2021

    Update: I figured out a method for running gds3xtrude on Windows. I needed to:

    1) pip3 install gds3xtrude 0.0.9
    2) pip3 install an older version of regex (I used 2019.03.08)
    3) pip3 install an older version of solid (I used 0.2.0)
    4) Move gds3xtrude, regex, and solid python folders from pip install site-packages location to C\Users[username]\AppData\Roaming\Klayout\lib\python3.7\site-packages (default installation location for Klayout)
    5) Add openSCAD to system environment path. I used the latest version.

    Note: for debugging, the Klayout\salt\gds3xtrude\pymacros folder has the gds3xtrude.lym file, which can be modified to try/catch errors and display str(e) exceptions in pya.MessageBox.warning.

  • dick_freebirddick_freebird

    This looks really interesting. Especially to folks who
    might like to use the 3D view for the starting point
    of a TCAD mesh. I know some of those.

    So question that comes to mind is, can the displayed
    "3D displayed in 2D" image be exported to any common
    3D geometry format (not that I know what's common
    or even available)? Better still, some format that TCAD
    tools can translate into a starting mesh?

  • MatthiasMatthias

    Side note: I have tried to create 3d data with DXF but could not make it load into a specific 3d simulation tool (a 3d EM simulator in my case). DXF offers extrusion which should in general create 3d models, but all I got was thin shells if I got something at all.

    I was told by an expert that DXF is nowhere used for importing geometry into 3d modelling tools. The hottest candidate seems to be STEP (or more specifically a certain flavour of it). But I did not find time yet to implement it.

    In the master branch, KLayout comes with a "extruded 3d" view (sometimes called 2.5d view). Hence there has to be thickness information. I'm not happy with the navigation yet and it's kind of experimental, but a beginning has been made.

    Matthias

  • toktok

    Hi Matthias,
    So cool! I just compiled KLayout from the master branch to give it a try. Can you give me a hint how to set-up the 3D view? I could not find it in the GUI.
    Thanks!

  • MatthiasMatthias
    edited March 2021

    Better: there is a video on YouTube:

    No sound, but I hope it shows some details. I'm not sure why the quality isn't good consistently, but I understand YouTube is playing some tricks depending on the number of video views. So go ahead and take a look :)

    Disclaimer: I personally do not like the navigation. Some thing I'd like to improve before advertising the feature.

    Matthias

  • toktok

    Thanks for the video! I cannot see the 2.5D view entry in the Tools menu. I build KLayout from the master branch. Do I miss some dependency like OpenGL?

  • MatthiasMatthias

    Hi Thomas,

    If Qt is not built with OpenGL, then the 2.5d view is skipped :(

    Maybe that's the case here?

    Matthias

  • toktok

    Hi Matthias,

    That's actually the main suspect. On Debian 10 it looks like I can install Qt libraries with OpenGL, but I did not yet figure out how I can then build KLayout with those.

  • sebastiansebastian

    Hi Thomas,

    I have KLayout compiled from master-branch on debian 11 and I see the 2.5D viewer :wink: . I am also not entirely sure which libraries are necessary for it, but my suspects are libqt5opengl5{,-dev} qml-module-qtgraphicaleffects. Most likely the dev one. Lmk if that works, otherwise I can dig a bit more, but I installed a lot of libraries when I tried to get KLayout running on webgl-streaming ^^

  • toktok

    Hi Sebastian,
    Thanks for the hints! I found out that I always compiled with qt4, qt5 somehow fails with errors like 'error: ‘QtInfoMsg’ was not declared in this scope'. Think will have to try harder. Or maybe I'll just set up a Debian 11 and give it a try there.

  • sebastiansebastian

    Hey Thomas,
    It should work perfectly fine on Debian 10 (I did qt5 compile on Debian 10 before, also in a VM). I think the main problem was trying to force it to use qmake for qt5 propely (I printed qmake version to debug this in build.sh). So make sure to use the flag to force use a special qmake (I think -qmake path/to/binary) At the beginning I got lots of errors due to missing libraries and modules.

    I documented it at some point:

    # Compile KLayout
cd ~
sudo apt update
sudo apt install -y libqt5xmlpatterns5 libqt5xmlpatterns5-dev libqt5svg5-dev libqt5xml5 libqt5multimediawidgets5 libqt5multimediagsttools5 libqt5multimedia5-plugins libqt5multimedia5 qttools5-dev qttools5-dev-tools libqt5designer5 libqt5designercomponents5 libqt5multimediawidgets5 python3-dev qt5-default qt5-qmake ruby-dev libz-dev gcc g++ make cmake qtmultimedia5-dev qt5-default qtbase5-dev git

# go to your folder where you want klayout
# e.g. `cd $HOME/repos`
git clone https://github.com/KLayout/klayout
cd klayout
./build.sh -j32 -qt5 -prefix /usr/local/klayout/ -qmake /usr/bin/qmake #prefix so you can install it properly on debian and make sure that you are using qt5-qmake not the qt4, debian 10 jumped to qt4-qmake for me in the past even when using -qt5
cd build-release
sudo make install
sudo ln -s /usr/local/klayout/klayout /usr/local/bin/klayout

    Last part is only to create a folder and link it so it's available in shell and DE, so feel free to drop that parts of course

  • toktok

    Thanks for the hint! I will give it a try.

  • toktok
    edited April 2021

    Thank you Sebastian, this did the job!

    Now I have the 2.5d View option. However, I get an error:

    Shapes vertex shader compilation failed:
0:1(10): error: GLSL 1.50 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.00 ES, and 3.00 ES

    It's related to OpenGL. Possibly an incompatible version. Unfortunately I have zero knowledge about OpenGL.

  • sebastiansebastian

    This might be related to your graphics driver. Are you using the open source video driver? I am using the proprietary (I know, I know, I hate it to) Nvidea driver because some things just don't work on nouveau. For AMD there are open source drivers from AMD directly I think. Or are you in a VM? I would assume in a VM you would have to passthrough the GPU unfortunately. I don't think you can (easily) have 3D acceleration without passthrough.

    These are the things I can think of. If these options don't work, I suspect that it's a configuration error in the Qt part. But I have zero experience with that ^^

  • toktok

    I have some Intel graphics card on a bare-metal computer. Never installed any other graphics driver than supplied by Debian 10. I also think it is some version mismatch or configuration but I also have zero experience with OpenGL and Qt.

  • sebastiansebastian

    I would suggest taking a look at the debian wiki: https://wiki.debian.org/Mesa and https://wiki.debian.org/GraphicsCard#Intel

    And from there the glxinfo for my pc:

    $ glxinfo | grep OpenGL
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce GTX 1080/PCIe/SSE2
OpenGL core profile version string: 4.6.0 NVIDIA 460.56
OpenGL core profile shading language version string: 4.60 NVIDIA
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.6.0 NVIDIA 460.56
OpenGL shading language version string: 4.60 NVIDIA
OpenGL context flags: (none)
OpenGL profile mask: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 460.56
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:

    looking on other sites, for intel it might be helpful to install xserver-xorg-video-intel if you don't already have it.

  • cadnavcadnav

    still no solution, it seems :( ... was excited to get all my CAD tools up and running on AWS with Nvidia Tesla T4...nothing but issues and desperately need 2.5d/3d view

    sh Shapes vertex shader compilation failed: 0:1(10): error: GLSL 1.50 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.40, 1.00 ES, and 3.00 ES

  • MatthiasMatthias

    Hi @cadnav,

    GLSL 1.5 comes with OpenGL 3.2 which dates back to 2009. I intentionally used an old version to stay compatible with older platforms. What is glxinfo | grep "OpenGL version" saying on your system?

    I for myself never had issues on Linux except for cases where OpenGL wasn't available at all.

    Matthias

Sign In or Register to comment.
Powered by Vanilla