Hi matthias, there was some progress with FreeCAD build on MSYS2-MINGW64 and MSYS2-UCRT64 (see https://github.com/FreeCAD/FreeCAD/issues/11523)
Now MSYS2-UCRT64 is the recommended environment by MSYS (over MSYS2-MINGW64) and I am switching to it.
I need a different FreeCAD version (the LinkMerge branch) and I am trying to build it in MSYS2-UCRT64.
Regarding compatibility with prebuilt KLayout why don't you ask MSYS to make a package for Klayout ?
You may do it opening an issue on msys2/MINGW-package repo.
I have done it asking for packaging FreeCAD LinkMerge (see https://github.com/msys2/MINGW-packages/issues/19175#issuecomment-1837066688) and I received a very constructive feedback from @MehdiChinoune who has already packaged the master FreeCAD branch.
If they accept to make the package I think that it will be regularly updated by MSYS maintainers assuring compatibility across different packages.
Hi Matthias,
Finally I could build LinkMerge branch of FreeCAD inside the MSYS2/ucrt64 environment. Python version is 3.11.
In the last commit to layoutDD package (at https://github.com/wsteffe/layoutDD) I have added .pyd files for modules FreeCAD and Part that I need to import.
But I don't know how to import such modules defined as binary files. "sys.path.append(pathToPydFiles)" doesn't work. Do you have any idea ?
basically this should work. Usually the .pyd files come with some Python wrapper (like __init__.py) that loads them internally and adds Python stuff like __all__. Also, if a package directory features a "python" folder, it is automatically added to sys.path, so explicitly adding it should not be required.
One reason for your problem may be, that these .pyd files depend on other DLLs that you did not package. That is what killed "numpy" recently in 0.28.13 (that is fixed). If that happens, you will need to put additional DLLs into the same location than the .pyd files.
I used "ldd" on MSYS to figure out the dependencies. "ldd" does not like ".pyd", so you have to copy them to ".dll" first.
In general, such issues are not easy to debug. Windows lacks decent ways of doing so (ProcMon is an option, but it is hard to use). On Linux, that is as simple as setting LD_DEBUG and checking the outputs.
Hi Matthias thanks for help.
Indeed there were some missing dlls preventing import also from python of environment MSY2/ucrt64.
Now I have added those dlls and I am able to import FreeCAD from python console in ucrt64.
But it still fails loading from other environemnts (Msys, Mingw64 and also from Klayout internal python).
I think I must rebuild FreeCAD in the same environment used for klayout. Is it Mingw64 ?
Hi Matthias,
I am able to build FreeCAD only in the ucrt64 environment but I get a lot of undefined symbols with Mingw64. Mingw64 seems to be a deprecated (or at least not the recommended) MSYS build environment (see https://github.com/microsoft/vscode-cpptools/issues/11063). Would you consider building klayout with ucrt64 ?
I would rather go for Anaconda or some other native solution than staying with MSYS. But I don't have the bandwidth to do that. Neither for another adventure towards another rolling release or a new CRT. There is hardly time left for functionality already, so I have to pull the brakes here.
Ok ,but which compiler/environment was used to build klayout-0.28.13-win64-install.exe ? I was thinking it is mingw-w64-x86_64-gcc but on your web pages I may read that for klayout Version >= 0.26 Visual Studio 2017 is also supported.
Is there a prebuilt release based on Visual Studio 2017 for klayout-0.28.13 ?
If so I could simply use FreeCAD-Link-Tip-Win-x86_64-py3.11-20231102.7z which was also made with Visual Studio 2017.
I have tried to build klayut in the ucrt64 environment. The build was done with the same script (build.sh) used for mingw. The only difference is that I started it fromm the uctr64 shell after havig installed all dependencies in the ucrt64 environment.
For the packaging I have used the annexed file which is derived (with small changes) from deploy-win-mingw.sh.
I could start klayout from windows also after adding following dlls:
Starting from windows it immediately gives the following error:
Problem reading config file C:/Users/walter/KLayout/klayoutrc: Read error on file: C:/Users/walter/KLayout/klayoutrc (errno=0)
Starting from shell ucrt64 it goes better. I could also open a layoutDD project (which includes a gds file) but it failed when I tried to install a python module (ezdxf). To do that I used <cmd "Install Python Dependencies" from my layoutDD package. Here is the log:
Loading file: C:/Users/walter/Documents/layoutDD/Test/printed_circuit_board_si_geom.gds with technology: mil
Loaded layer properties from C:/Users/walter/Documents/layoutDD/Test/printed_circuit_board_si_geom.lyp
Loading file: C:/Users/walter/Documents/layoutDD/Test/partition.gds with technology: mil
ImportError: DLL load failed while importing pyexpat: The specified module could not be found.
C:/msys64/home/walter/Software/klayout/bin-release-win64/lib/python3.11/xml/parsers/expat.py:4
C:/msys64/home/walter/Software/klayout/bin-release-win64/lib/python3.11/plistlib.py:70
C:/msys64/home/walter/Software/klayout/bin-release-win64/lib/python3.11/site-packages/pkg_resources/__init__.py:35
C:/Users\walter\KLayout\salt\layoutDD\python/layoutDD/setup.py:7
But "import pyexpat" is properly executed from a python console in the ucrt64 environment.
I have added other dlls and now ucrt64 based klayout starts directly from the windows GUI.
But the pyexpat import problem is not yet solved.
Hereafter the complete list of required dlls.
I have slightly changed my function installRequirements() removing pkg_resources which seems to be deprecated.
The update code is:
def installRequirements():
import importlib.util
import pip
import sys
import os
reqPath = os.path.realpath(os.path.join(os.path.dirname(__file__), "..", "requirements.txt"))
required = set()
with open(reqPath) as reqfile:
for line in reqfile:
line=line.split("#")[0]
line=line.rstrip()
if len(line)>0:
required.add(line)
for package in required:
if importlib.util.find_spec(package) is None:
pya.MessageBox.info("Information", "Install "+package+" ?", pya.MessageBox.Ok)
pip.main(['install', package])
The only requirement is ezdxf.
Using this function with klayout built with ucrt64 (but not using official klayout) I get following errors:
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\ctypes\__init__.py:8
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_vendor\platformdirs\windows.py:4
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_vendor\platformdirs\__init__.py:26
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_vendor\platformdirs\__init__.py:46
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\utils\appdirs.py:13
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\locations\base.py:9
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\locations\_sysconfig.py:11
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\locations\__init__.py:14
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\utils\misc.py:44
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\utils\logging.py:29
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\cli\spinners.py:9
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\build_env.py:19
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\cli\main_parser.py:9
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\cli\autocompletion.py:10
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\cli\main.py:10
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\_internal\utils\entrypoints.py:7
C:\Users\walter\AppData\Roaming\KLayout\lib\python3.11\site-packages\pip\__init__.py:11
C:\Users\walter\KLayout\salt\layoutDD\python\layoutDD\setup.py:22
Loading file: C:/Users/walter/Documents/layoutDD/Test/printed_circuit_board_si_geom.gds with technology: mil
Loaded layer properties from C:/Users/walter/Documents/layoutDD/Test/printed_circuit_board_si_geom.lyp
Loading file: C:/Users/walter/Documents/layoutDD/Test/partition.gds with technology: mil
ModuleNotFoundError: No module named 'ezdxf'
C:\Users\walter\KLayout\salt\layoutDD\python\layoutDD\subdomains.py:969
C:\Users\walter\KLayout\salt\layoutDD\python\layoutDD\subdomains.py:1032
The poblem with ezdxf install is solved. It was a connection problem from the VM in which I am developing the windows stuff.
I have still a problem of missing DLLs when I am importing FreeCAD. "import FreeCAD" is properly done when python is "/msys64/ucrt64/bin/python" but it doesn't work in klayout macro debugger with the following test file:
C:/Users/walter/KLayout/pymacros\test.py:8: ImportError: DLL load failed while importing FreeCAD: The specified module could not be found.
C:/Users/walter/KLayout/pymacros\test.py:8
The big problem is always the same: how may I trace DLLs dependencies of imports made from klayout python interpreter ?
Hi Matthias,
I have solved several FC problems using pyinstaller but there I is a last problem (I hope it is the last) which is described more in detail in the last post at https://github.com/msys2/MINGW-packages/issues/19175#issuecomment-1854058561.
I don't know if it is something specific to the klayout environment or a bad packaged FreeCAD module.
The klayout built in the ucrt64 environment now works well. Here annexed is the script that I have used to add all missing dlls into my install path.
Would it possible to retrieve automatically all these dependencies in the script deploy-ucrt64.sh ? How do you handle this problem in your release based on mingw64.
I would also like to make an installer that takes care of adding "C:\windows\system32\downlevel" to the user path (for several api-ms-win-crt-xxx.dll).
Once solved these last problems I would like make a pull request supporting klayout build in MSYS2/ucrt64.
I will not ask you to make realeases based on this crt even if I would be happy if you did that.
I will instead upload an istaller on my github repo to allow usage of my layoutDD package on windows10.
Hi Matthias,
I have uploaded the FreeCAD package at https://github.com/wsteffe/layoutDD_FreeCAD.
The FreeCAD folder includes the following test.py:
#! /ucrt64/bin/python
import sys
sys.path.append("bin")
import FreeCAD
import Part
import Import
from BOPTools.ShapeMerge import mergeShapes
from BOPTools.GeneralFuseResult import GeneralFuseResult
Import.readDXF("pippo.dxf")
If you run test.py in a msys shell (not necessarly a ucrt shell) from the FreeCAD folder (after having installed mingw-w64-ucrt-x86_64-python),
you will see that it is executed without problem. But a similar script fails in the klayout internal python even if klayout and FreeCAD are both built in MSYS2/ucrt64.
With uploaded file structure the macro debugger gives an error at the line "Import Part".
This errors disappears if I move into the bin folder all subfolders of Mod.
But then the debugger gives an error at line "from BOPTools.ShapeMerge import mergeShapes".
This error can be fixed moving up into the bin folder all Part subfolders (which include BOPTools) .
The last error occurs at the line "Import.readDXF("pippo.dxf")" and I am not able to fix it.
But my question is:
why does it the klayout python behaves so differetly than MSYS python with respect to finding or not finding modules in a folder tree?
Comments
Hi matthias, there was some progress with FreeCAD build on MSYS2-MINGW64 and MSYS2-UCRT64 (see https://github.com/FreeCAD/FreeCAD/issues/11523)
Now MSYS2-UCRT64 is the recommended environment by MSYS (over MSYS2-MINGW64) and I am switching to it.
I need a different FreeCAD version (the LinkMerge branch) and I am trying to build it in MSYS2-UCRT64.
Regarding compatibility with prebuilt KLayout why don't you ask MSYS to make a package for Klayout ?
You may do it opening an issue on msys2/MINGW-package repo.
I have done it asking for packaging FreeCAD LinkMerge (see https://github.com/msys2/MINGW-packages/issues/19175#issuecomment-1837066688) and I received a very constructive feedback from @MehdiChinoune who has already packaged the master FreeCAD branch.
If they accept to make the package I think that it will be regularly updated by MSYS maintainers assuring compatibility across different packages.
Hi Matthias,
Finally I could build LinkMerge branch of FreeCAD inside the MSYS2/ucrt64 environment. Python version is 3.11.
In the last commit to layoutDD package (at https://github.com/wsteffe/layoutDD) I have added .pyd files for modules FreeCAD and Part that I need to import.
But I don't know how to import such modules defined as binary files. "sys.path.append(pathToPydFiles)" doesn't work. Do you have any idea ?
Hi @wsteffe,
basically this should work. Usually the .pyd files come with some Python wrapper (like
__init__.py) that loads them internally and adds Python stuff like__all__. Also, if a package directory features a "python" folder, it is automatically added to sys.path, so explicitly adding it should not be required.One reason for your problem may be, that these .pyd files depend on other DLLs that you did not package. That is what killed "numpy" recently in 0.28.13 (that is fixed). If that happens, you will need to put additional DLLs into the same location than the .pyd files.
I used "ldd" on MSYS to figure out the dependencies. "ldd" does not like ".pyd", so you have to copy them to ".dll" first.
In general, such issues are not easy to debug. Windows lacks decent ways of doing so (ProcMon is an option, but it is hard to use). On Linux, that is as simple as setting LD_DEBUG and checking the outputs.
Matthias
Hi Matthias thanks for help.
Indeed there were some missing dlls preventing import also from python of environment MSY2/ucrt64.
Now I have added those dlls and I am able to import FreeCAD from python console in ucrt64.
But it still fails loading from other environemnts (Msys, Mingw64 and also from Klayout internal python).
I think I must rebuild FreeCAD in the same environment used for klayout. Is it Mingw64 ?
Hi Matthias,
I am able to build FreeCAD only in the ucrt64 environment but I get a lot of undefined symbols with Mingw64. Mingw64 seems to be a deprecated (or at least not the recommended) MSYS build environment (see https://github.com/microsoft/vscode-cpptools/issues/11063). Would you consider building klayout with ucrt64 ?
Hi @wsteffe,
I would rather go for Anaconda or some other native solution than staying with MSYS. But I don't have the bandwidth to do that. Neither for another adventure towards another rolling release or a new CRT. There is hardly time left for functionality already, so I have to pull the brakes here.
Matthias
Ok ,but which compiler/environment was used to build klayout-0.28.13-win64-install.exe ? I was thinking it is mingw-w64-x86_64-gcc but on your web pages I may read that for klayout Version >= 0.26 Visual Studio 2017 is also supported.
Is there a prebuilt release based on Visual Studio 2017 for klayout-0.28.13 ?
If so I could simply use FreeCAD-Link-Tip-Win-x86_64-py3.11-20231102.7z which was also made with Visual Studio 2017.
I have tried to build klayut in the ucrt64 environment. The build was done with the same script (build.sh) used for mingw. The only difference is that I started it fromm the uctr64 shell after havig installed all dependencies in the ucrt64 environment.
For the packaging I have used the annexed file which is derived (with small changes) from deploy-win-mingw.sh.
I could start klayout from windows also after adding following dlls:
But it gives errorrs:
Starting from windows it immediately gives the following error:
Starting from shell ucrt64 it goes better. I could also open a layoutDD project (which includes a gds file) but it failed when I tried to install a python module (ezdxf). To do that I used <cmd "Install Python Dependencies" from my layoutDD package. Here is the log:
But "import pyexpat" is properly executed from a python console in the ucrt64 environment.
I have added other dlls and now ucrt64 based klayout starts directly from the windows GUI.
But the pyexpat import problem is not yet solved.
Hereafter the complete list of required dlls.
Copied from /msys64/ucrt64/bin:
Copied from C:\windows\system32\downlevel:
I have slightly changed my function installRequirements() removing pkg_resources which seems to be deprecated.
The update code is:
The only requirement is ezdxf.
Using this function with klayout built with ucrt64 (but not using official klayout) I get following errors:
Any idea of a posseble reason and how to debug ?
Hi matthias,
I have restricted the problem a single line:
C:/Users/walter/KLayout/pymacros\test.pys:
Result in debug console:
import ctypes is properly done from a python console inside the ucrt64 environment.
I have no idea how to find out the missing dll for an import called from the klayout internal python interpeter.
Last update. I have found the last two missing DLLs:
Now the import cmake and also following:
is executed and no error is reported.
But ezxdf package is not really installed. Any idea about what could have been wrong ?
Understood:
pip.main(['install', "ezxdf"]) doesn't work even in the python shell of ucrt64. It fails with following error:
The poblem with ezdxf install is solved. It was a connection problem from the VM in which I am developing the windows stuff.
I have still a problem of missing DLLs when I am importing FreeCAD. "import FreeCAD" is properly done when python is "/msys64/ucrt64/bin/python" but it doesn't work in klayout macro debugger with the following test file:
The Error is:
The big problem is always the same: how may I trace DLLs dependencies of imports made from klayout python interpreter ?
Hi Matthias,
I have solved several FC problems using pyinstaller but there I is a last problem (I hope it is the last) which is described more in detail in the last post at https://github.com/msys2/MINGW-packages/issues/19175#issuecomment-1854058561.
I don't know if it is something specific to the klayout environment or a bad packaged FreeCAD module.
The klayout built in the ucrt64 environment now works well. Here annexed is the script that I have used to add all missing dlls into my install path.
Would it possible to retrieve automatically all these dependencies in the script deploy-ucrt64.sh ? How do you handle this problem in your release based on mingw64.
I would also like to make an installer that takes care of adding "C:\windows\system32\downlevel" to the user path (for several api-ms-win-crt-xxx.dll).
Once solved these last problems I would like make a pull request supporting klayout build in MSYS2/ucrt64.
I will not ask you to make realeases based on this crt even if I would be happy if you did that.
I will instead upload an istaller on my github repo to allow usage of my layoutDD package on windows10.
Hi Matthias,
I have uploaded the FreeCAD package at https://github.com/wsteffe/layoutDD_FreeCAD.
The FreeCAD folder includes the following test.py:
#! /ucrt64/bin/python
import sys
sys.path.append("bin")
import FreeCAD
import Part
import Import
from BOPTools.ShapeMerge import mergeShapes
from BOPTools.GeneralFuseResult import GeneralFuseResult
Import.readDXF("pippo.dxf")
If you run test.py in a msys shell (not necessarly a ucrt shell) from the FreeCAD folder (after having installed mingw-w64-ucrt-x86_64-python),
you will see that it is executed without problem. But a similar script fails in the klayout internal python even if klayout and FreeCAD are both built in MSYS2/ucrt64.
With uploaded file structure the macro debugger gives an error at the line "Import Part".
This errors disappears if I move into the bin folder all subfolders of Mod.
But then the debugger gives an error at line "from BOPTools.ShapeMerge import mergeShapes".
This error can be fixed moving up into the bin folder all Part subfolders (which include BOPTools) .
The last error occurs at the line "Import.readDXF("pippo.dxf")" and I am not able to fix it.
But my question is:
why does it the klayout python behaves so differetly than MSYS python with respect to finding or not finding modules in a folder tree?