Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

3D Viewer & Salt dependency managment

Hi,
Recently I wrote a Python script for viewing layouts as 3D models in OpenSCAD. Initially it was just for fun but then I realized it's also quite helpful for debugging. Therefore I thought I'd best package it as a Salt module.

And here comes my question: Can Salt manage Python dependencies (PyPI/pip packages)? How do I tell Salt about them?

Comments

  • Hi tok,

    thanks for considering sharing your work!

    However, you cannot tell Salt about this currently. But you can document this dependency of course.

    An automated option (I have not tried myself) to take care of this is using an autorun script inside the module. Let this script run once (use your own configuration variable to mark it run). In this script you can try to pull the package from PyPI. But I suspect many side effects and error handling is crucial. Otherwise there is risk to damage a user's installation.

    But I'd be careful to require a specific Python module. Unless the module is a very common one, the dependency isn't easy to fulfil on all platforms.

    Regards,

    Matthias

  • Thanks for answering! Then I best let the users manage the dependencies. They need to install OpenSCAD manually anyway.

  • BTW: It is called 'gds3xtrude' and should be available from the package manager.
    However, the dependencies must be installed manually first:
    pip3 install --user gds3xtrude
    pip3 install --user klayout # Only if gds3xtrude is used as a standalone tool.

    More documentation is here: https://codeberg.org/tok/gds3xtrude

  • Awesome! Thanks for providing this!

    Best regards,

    Matthias

  • If somebody would like to use Blender instead of OpenSCAD, let me know. I started experimenting and I would be motivated to push the project if somebody is interested.
    Blender would be useful for good-looking renderings and illustrations.

  • edited May 24

    Setting the correct path on windows(10) is a bit confusing. If you create a blank macro:

    import sys
    print(sys.path)
    

    The interpreter returns the path set in:
    C:\Users[user-name]\AppData\Roaming\KLayout.python-paths.txt

    This is not the same as the 'path' environmental variable as returned by the python3 install:
    C:\Program Files (x86)\Python36-32\python.exe

    Does the macro environment use a different python interpreter? Why is the path variable different to sys.path?

    Note:
    The dependencies are correctly installed under C:\Program Files (x86)\Python36-32\ but the macro environment does not see this package location. Manually adding the dependencies to
    C:\Users[user-name]\AppData\Roaming\KLayout.python-paths.txt
    did not resolve the issue.

    Cheers
    Andrew

  • Hi Andrew,

    Does this post belong to this discussion? If not, please open a new one.

    KLayout comes with it's own Python interpreter and does not take any interpreter provided by the system. This leads to manifold issues usually due to incompatibilities of the binary APIs. And KLayout also does not read PYTHON_PATH, because some programs set this variable to their own libraries on installation. Hence "python-paths.txt".

    I don't see what you're trying to achieve. What is the issue?

    Matthias

  • Hi Matthias,

    My comment relates to "Can Salt manage Python dependencies" and "dependency isn't easy to fulfil on all platforms.". Specifically, I was trying to get gds3xtrude up and running on a windows box.

    I use both Windows and Linux environments for development. Setting up gds3xtrude on Linux is straight forward (setting aside a permissions issue). The instructions (above) when run on a Windows box will install the packages on the system, but the KLayout interpreter does not see the install location:

    pip3 install --user gds3xtrude
    pip3 install --user klayout # Only if gds3xtrude is used as a standalone tool.
    
    More documentation is here: https://codeberg.org/tok/gds3xtrude
    

    Using "python-paths.txt" to point to the system package location may create a bit of a mess (especially if the system is running a different version of python).

    My solution was to switch back to Linux. This is now a heads-up to anyone trying to get gds3xtrude running on windows. It would be helpful if the install process was more self contained, but I understand the reasons why this isn't the case.

    Cheers
    Andrew

  • Unfortunately I don't have a Windows installation to reproduce the issue.
    But I heard from other people who had difficulties because they had python3.7 installed but KLayout has python3.5 built-in. Therefore, when running pip3 the packages go to a directory explicitly for python3.7 and cannot be found by python3.5. The solution in this case is to install python3.5 if it is not already there and run pip3.5 instead of pip3.

  • Thanks Tok, and thanks for sharing your work!

    re. because they had python3.7 installed but KLayout has python3.5 built-in

    I'm also running python3.7, so likely the same issue. On a windows box, after installing python3.5, the dependencies can be installed using py and the -m switch:

    py -3.5 -m pip install --user gds3xtrude
    https://docs.python.org/3/installing/index.html
    

    the equivalent for a Linux box would be:

    python3.5 -m pip install --user gds3xtrude
    
  • Hi all,

    thanks for this discussion. Actually on Windows, the problem is much deeper than just Python versions. There is also an incompatibility between the binary APIs of different builds (based on gcc/MinGW, MSVC etc.). For example, KLayout builds with MinGW while Anaconda Python comes for MSVC 2017. As these versions are incompatible and must never by mixed, KLayout tries it's best to become independent from other installations. Therefore it will ignore PYTHONPATH and use it's own path (KLAYOUT_PYTHONPATH) and use python_path.txt to configure it's own environment.

    This makes mixing in other packages difficult.

    You can mix in Python packages that come with KLayout's build system (which is MSYS2). But as this is a rolling release, it's likely that you won't find a package for the version KLayout was built with.

    I established MSVC compatibility for the master branch which means you can mix with Anaconda packages. This should make things easier then.

    But frankly, my goal is to provide a good end user tool, not becoming an expert in Python installations in particular and the whole DevOps stuff in general. This consumes too much energy which is lost for real development. I'm setting priorities for myself.

    Matthias

Sign In or Register to comment.