Salt Package Manager timeout under Linux (proxy problem?)

sekigawa

Hello Matthias,

In my environment below, I got a timeout (after 60 sec) while fetching http://sami.klayout.org/repository.xml.

OS: Linux Mint 20.3 (5.4.0-166-generic)
g++: version (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
KLayout: 0.28.12 source code build
  * Q5: 5.12.8
  * Ruby: 3.1.3
  * Python: 3.8.10
  * built with:
      ./build.sh  \
           -release \
           -qmake         /opt/Qt5/bin/qmake \
           -ruby          /usr/local/bin/ruby31 \
           -python        /usr/bin/python3 \
           -build         ./qt5.build.linux-64-gcc-release \
           -bin           ./qt5.bin.linux-64-gcc-release/ \
           -option        -j8 \
           -with-qtbinding \
           -qt5 \
           -rblib         /usr/local/lib/libruby.so.3.1 \
           -rbinc         /usr/local/include/ruby-3.1.0 \
           -rbinc2        /usr/local/include/ruby-3.1.0/x86_64-linux \
           -pylib         /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0 \
           -pyinc         /usr/include/python3.8  2>&1 \
          | tee           qt5.build.linux-64-gcc-release.log

Both curl http://sami.klayout.org/repository.xml (with proxy settings in $HOME/.curlrc) and wget http://sami.klayout.org/repository.xml (with proxy settings in $HOME/.wgetrc) work fine (took about one second) in a terminal.

I have libcurl.so.4.6.0 installed, but it does not seem to be used.
Instead, libQt5Network.so.5 seems to be linked.
What am I missing?

Thanks and best regards,
Kazzz-S

Matthias

Hi @sekigawa,

I think QtNetwork does not recognize the proxy settings from .curlrc or .wgetrc. I understand that Qt uses the "system proxy" by default and I do not configure anything else. Honestly, I never had to configure proxy settings myself. Even in a company network, QtNetwork worked without changes.

I found a little documentation here: https://doc.qt.io/qt-5/qnetworkproxyfactory.html#systemProxyForQuery. I don't know if that helps.

If the solution is curl, you can try using libcurl by building with "-libcurl" in build.sh. That enables libcurl instead of QtNetwork. My understanding is that after this, KLayout should respect ".curlrc", but I have not tested that myself.

Best regards,

Matthias

sekigawa

Hi @Matthias,

Thanks for the URL, which was helpful.
It says:

:
On other systems, this function will pick up proxy settings from the "http_proxy" environment variable. 
This variable must be a URL using one of the following schemes: "http", "socks5" or "socks5h".

In my launching service Bash script, I set export http_proxy=http://111.222.333.444:5678.
Then, fetching was successful.

Warm regards,
Kazzz-S

sekigawa

Next, in relation to No.2399, using the same Linux environment:

sekigawa

Direct Internet connection from Mac

The unit test tlGitTests passed in the four environments.

Matthias

Very good ! Thanks for sharing this information!

Matthias

sekigawa

Connection from Linux via Proxy

I'm playing with libgit2

I have found a way to solve the timeout problem in the Linux environment with the following changes.
I'm sure there must be a more sophisticated way to handle a proxy.
But just for information.

Matthias

Hello @sekigawa,

thanks for sharing this code piece.

I'm sorry I missed that ... I am right now releasing 0.28.13 (for now without that patch). But I am sure there will be an update soon.

I just wonder whether it is not possible to configure a proxy in general through the system. I was somehow assuming libgit2 would be using curl and that curl would read "http_proxy" itself without need to configure it externally. But looks like I am wrong. I still feel it is a bit weird that proxies need to be configured in the code.

Thanks and best regards,

Matthias

sekigawa

Hello @Matthias,

I just wonder whether it is not possible to configure a proxy in general through the system.

I agree. Please take your time.

Warm regards,
Kazzz-S

sekigawa

Meanwhile, in the mingw64 environment, I got an error when starting ut_runner.exe, as shown below.
What could be the reasons?

sekig@KS01-Win10 MINGW64 /home/sekig/GitWork/klayout/build-release
$ echo $LD_LIBRARY_PATH
/home/sekig/GitWork/klayout/build-release: \
/home/sekig/GitWork/klayout/build-release/db_plugins: \
/home/sekig/GitWork/klayout/build-release/lay_plugins

sekig@KS01-Win10 MINGW64 /home/sekig/GitWork/klayout/build-release
$ ./ut_runner.exe
    Unable to load plugin tests: \
    C:\msys64\home\sekig\GitWork\klayout\build-release\cif_tests.ut with error message: 126

All test modules seem to be there...

sekig@KS01-Win10 MINGW64 /home/sekig/GitWork/klayout/build-release
$ ll *.ut
-rwxr-xr-x 1 sekig sekig   55296 2023-11-23 01:04 ant_tests.ut*
-rwxr-xr-x 1 sekig sekig  263168 2023-11-23 01:37 bd_tests.ut*
-rwxr-xr-x 1 sekig sekig   72192 2023-11-23 01:21 cif_tests.ut*
-rwxr-xr-x 1 sekig sekig 7833088 2023-11-22 23:06 db_tests.ut*
-rwxr-xr-x 1 sekig sekig  186880 2023-11-23 01:34 drc_tests.ut*
-rwxr-xr-x 1 sekig sekig  124928 2023-11-23 01:22 dxf_tests.ut*
-rwxr-xr-x 1 sekig sekig   77824 2023-11-23 01:09 edt_tests.ut*
-rwxr-xr-x 1 sekig sekig  255488 2023-11-23 01:24 gds2_tests.ut*
-rwxr-xr-x 1 sekig sekig  129536 2023-11-22 22:23 gsi_tests.ut*
-rwxr-xr-x 1 sekig sekig  115200 2023-11-23 01:06 img_tests.ut*
-rwxr-xr-x 1 sekig sekig   16896 2023-11-23 01:37 klayout_main_tests.ut*
-rwxr-xr-x 1 sekig sekig  186368 2023-11-23 01:20 lay_tests.ut*
-rwxr-xr-x 1 sekig sekig  526336 2023-11-23 00:52 laybasic_tests.ut*
-rwxr-xr-x 1 sekig sekig  207872 2023-11-23 01:02 layui_tests.ut*
-rwxr-xr-x 1 sekig sekig   38912 2023-11-23 01:03 layview_tests.ut*
-rwxr-xr-x 1 sekig sekig  211456 2023-11-23 01:25 lefdef_tests.ut*
-rwxr-xr-x 1 sekig sekig  145408 2023-11-22 23:08 lib_tests.ut*
-rwxr-xr-x 1 sekig sekig  114688 2023-11-23 01:35 lvs_tests.ut*
-rwxr-xr-x 1 sekig sekig   55296 2023-11-23 01:11 lym_tests.ut*
-rwxr-xr-x 1 sekig sekig   46592 2023-11-23 01:26 mag_tests.ut*
-rwxr-xr-x 1 sekig sekig  116224 2023-11-23 01:31 net_tracer_tests.ut*
-rwxr-xr-x 1 sekig sekig  294912 2023-11-23 01:28 oasis_tests.ut*
-rwxr-xr-x 1 sekig sekig   86016 2023-11-23 01:29 pcb_tests.ut*
-rwxr-xr-x 1 sekig sekig   43008 2023-11-23 01:11 pya_tests.ut*
-rwxr-xr-x 1 sekig sekig   46592 2023-11-23 01:38 pymod_tests.ut*
-rwxr-xr-x 1 sekig sekig   88064 2023-11-23 01:10 rba_tests.ut*
-rwxr-xr-x 1 sekig sekig  114176 2023-11-22 23:07 rdb_tests.ut*
-rwxr-xr-x 1 sekig sekig 2063872 2023-11-22 22:20 tl_tests.ut*
-rwxr-xr-x 1 sekig sekig  114688 2023-11-23 01:33 view_25d_tests.ut*

Thank you.
Kazzz-S

Matthias

Hi Kazzz,

Is $LD_LIBRARY_PATH actually working? To be frank I have not tried myself - I though there is not such thing on MSYS ..

Let me share with you my complete configuration to run the test suite on my Jenkins instance. Maybe this helps. I assume the interesting part is the comment about Python >= 3.8:

cd build-release-win64

export TESTSRC=..
export TESTTMP=testtmp 

rm -rf $TESTTMP

pwd=`pwd`
echo "#!/bin/env python3" >path.py
echo "import sys" >>path.py
echo "print(';'.join(sys.path))" >>path.py
chmod 755 path.py

export KLAYOUT_PYTHONPATH=`MSYSTEM=MINGW64 bash --login -c $pwd/path.py`

# Python >=3.8 on Windows does no longer respect PATH
# for DLL loading. We have to help it a little:
cp *.dll pymod/klayout 
rm -rf pymod/klayout/{db,lay}_plugins
cp -r db_plugins lay_plugins pymod/klayout

echo '#!/bin/env sh' >ut_runner.sh
echo "cd $pwd" >>ut_runner.sh
echo 'for f in plugins/* db_plugins lay_plugins; do' >>ut_runner.sh
echo '  if [ -d $f ]; then' >>ut_runner.sh
echo '    export PATH="$f:$PATH"' >>ut_runner.sh
echo '  fi' >>ut_runner.sh
echo 'done' >>ut_runner.sh
echo './ut_runner.exe -a >ut_runner.xml' >>ut_runner.sh
echo 'exit 0' >>ut_runner.sh

MSYSTEM=MINGW64 bash --login -c $pwd/ut_runner.sh

Best regards,

Matthias

sekigawa

Hi @Matthias,

Regarding LD_LIBRARY_PATH, your point is correct.

  (ChatGPT)
    [Q] Does MSYS2 or MINGW64 support the LD_LIBRARY_PATH environment variable?

    [A] In the context of MSYS2 and MINGW64, which are environments for developing and
    running software on Windows that use the GNU Compiler Collection (GCC),
    the concept of LD_LIBRARY_PATH is not directly applicable.
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    LD_LIBRARY_PATH is an environment variable used in Unix-like operating systems to
    specify directories where the system should look for dynamic libraries before searching
    the default locations. On Windows, the equivalent environment variable is PATH,
    which is used to specify directories for executable files.
    In the MSYS2 and MINGW64 environments, dynamic linking is typically handled by the
    Windows DLL (Dynamic Link Library) mechanism, and the PATH environment variable is
    used to locate DLLs.
    When you compile and link your programs with GCC in these environments, the resulting
    executables will look for DLLs in the directories specified in the PATH variable.
    If you need to specify additional directories for the loader to search for DLLs
    when running your program, you can modify the PATH variable accordingly.
    You can do this either temporarily in the command prompt where you run your program or
    by modifying the system or user environment variables.
    For example, in the command prompt:
    (bash)
    `
    set PATH=C:\path\to\your\dlls;%PATH%
    `
    This adds the specified directory to the beginning of the PATH variable for the current
    command prompt session.
    Keep in mind that if you distribute your software, users will need to ensure that
    the required DLLs are in directories listed in their PATH.

Regards,
Kazzz-S

sekigawa

Hi @Matthias,

Thank you for sharing the code I could run ut_runner.exe using the attached script in my mingw64 environment.

In my company's Windows environment, the proxy to be used is automatically assigned according to each PC's IP address, so setting the http_proxy environment variable is unnecessary. But the above changes in tlGit.cc are required.
<== My misunderstandings. Both are required.

Best regards,
Kazzz-S

Matthias

Hi @sekigawa,

I picked up your suggestion about the HTTP_PROXY variable, but I found it may interfere with curl's own mechanism when built against curl.

I decided to call the variable "KLAYOUT_GIT_HTTP_PROXY", so it is unique. I tested the implementation with squid and it seems to work. I will release that feature with the next minor release.

Best regards,

Matthias

sekigawa

Hi @Matthias,

Thank you for supporting the proxy. I like your idea.
I'll set the KLAYOUT_GIT_HTTP_PROXY environment variable in my launching service Bash script.

Best regards,
Kazzz-S