›

Can't publish KQCircuits as KLayout SALT package

PavelSmirnovPavelSmirnov
in KLayout Development

I've managed to go through the process of updating KQCircuits package using https://sami.klayout.org/index, but it doesn't show up on the global package index when I browse Tools > Manage Packages on KLayout app.

So I started by pushing a commit with tag v4.8.0. I understood that KLayout has relatively new feature to pull packages using GIT rather SVN, so I tested succesfully using KLAYOUT_SALT_MINE env var that the package gets pulled correctly if I specify <url> tag in grain.xml as git+https://github.com/iqm-finland/KQCircuits.git/tags/v4.8.0/klayout_package. Then I notice in the SALT package update form that the recommended url form is git+https://github.com/iqm-finland/KQCircuits.git+klayout_package[v4.8.0]. Changing in the form the Package URL value to either url format causes the package defintion link to redirect to the error page with attached message:


So I thought I would power through and submit the form anyway. I received confirmation email, updated KQCircuits is now listed in https://sami.klayout.org/index, but the package link for it doesn't work, and in KLayout app it is not listed.

I thought it fails because git+https://github.com/iqm-finland/KQCircuits.git/tags/v4.8.0/klayout_package format doesn't work, so I pushed a new commit tagged as v4.8.1 where I change the <url> tag in grain.xml to recommended format. Tested using KLAYOUT_SALT_MINE that KLayout pulls correct package. But now as I want to update the package again, the preview link again gives me an error page.

So I thought at this point I should stop and ask for help. Which url format is recommended to use, is the error caused by this or something else?

Thanks,
-Pavel

Comments

  • MatthiasMatthias

    Hi Pavel,

    I am sorry for this inconvenience.

    The URL is correct, the problem is with the grain.xml file.

    If I read the server logs correctly, the problem is with the "authored-time" field. Rendering the package information stops because the time value is invalid.

    If you use it, you should use a specific time, not a year interval. The supported formats are described here: https://api.rubyonrails.org/classes/String.html#method-i-to_time.

    I have not debugged why it works in some cases and I admit the message could be somewhat more specific.

    Matthias

  • MatthiasMatthias

    I just confirmed with a fork that changing the time field to

     <authored-time>2024-06-15</authored-time>

    makes the preview link work.

    I can, however, register packages also with the wrong field. So package registration does not seem to be affected by that.

    I observe some spurious internal server errors however, without a trace in the logs. Maybe there are some problems with the provider. If you see a plain-HTML 500 message (not the rendered message box above), please retry.

    Best regards,

    Matthias

  • PavelSmirnovPavelSmirnov

    Thanks for your debugging help! Seems that we have misunderstood the function of the <authored-time> field for a long time :D Starting now we will put a singular date there. Made version 4.8.2 now, and I have verified that the package is now accessible from KLayout's Manage packages dialogue and it installs correctly!

  • sekigawasekigawa

    Unfortunately, Manage Packages ends with errors on macOS, as shown below.
    This function previously worked fine. I ran klayout 0.29.2 with -d 100.
    Case-A happened most of the time, and Case-B happened only once for now.
    In Linux Mint 21.3 (source build), Case-A sometimes occurs; that is, unstable. My network problem?

    Image 1 Image 2

    Case-A Terminal Output

    Downloading package repository from http://sami.klayout.org/repository.xml
HTTP request URL: http://sami.klayout.org/repository.xml
HTTP request: GET
HTTP request creation failed
ERROR: Error 0: Request creation failed, fetching http://sami.klayout.org/repository.xml

    Case-B Terminal Output

    Downloading package repository from http://sami.klayout.org/repository.xml
HTTP request URL: http://sami.klayout.org/repository.xml
HTTP request: GET
HTTP response header: Date: Tue, 18 Jun 2024 10:21:06 GMT
HTTP response header: Server: Apache
HTTP response header: Content-Length: 728
HTTP response header: Connection: close
HTTP response header: Content-Type: text/html; charset=iso-8859-1
HTTP response error: Internal Server Error
ERROR: Error 500: Internal Server Error, fetching http://sami.klayout.org/repository.xml

Reply body:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<p>Additionally, a 500 Internal Server Error
error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache Server at sami.klayout.org Port 80</address>
</body></html>
  • sekigawasekigawa

    Cont.

    When accessing the URL with the Chrome browser on my Mac...

    Attempt-1

    Attempt-2

    Attempt-3

  • sekigawasekigawa
    edited June 2024

    Cont.



    My home network environment is suspicious.

  • sekigawasekigawa
    edited June 2024

    I rebuilt a MacOS binary by changing the timeout from 10 to 20 seconds.
    Now it works! But I encountered Case-B three times.
    I may have to complain to my ISP about the slow connection. :wink:
    Sorry for my noisy posts.

    • 2024-06-20: Case-B two times.
  • MatthiasMatthias
    Hi kazzz,

    Thanks for debugging that issue.

    I think there is a problem with my server. I will reboot and try if the problem persists. I am right on my way back from FSiC2024 and do not have access right now.


    Best regards,

    Matthias
  • MatthiasMatthias

    Hi @sekigawa,

    Currently there does not seem to be an issue with the server. I tried to find anything useful in the logs - but apart of the usual attacks (or rather: attempts), there is nothing specific.

    Error 500 usually does not point to a timeout problem. It indicates some script error - in my case, I found some "could not deliver" messages in the log. Maybe those are DOS attacks. After all, there is a war going on in Europe, and maybe those are the consequences.

    Anyway, I think it is quiet now. Could you try again with the 10s timeout?

    Thanks,

    Matthias

  • sekigawasekigawa
    edited June 2024

    Hi @Matthias,

    Thank you for looking into this problem.
    With the 10s timeout, I frequently encountered the Case-A error.
    I also got the Case-B problem thrice (in the last 3 hours; I do not remember exactly when).
    I believe my home Internet environment is problematic since I do not have such a problem from my office (one mile from my residence; the network route must be different). I show the test results with Chrome below.

    I've modified the source code as follows.


    I set export KLAYOUT_INPUT_HTTP_STREAM_PRIVATE_DATA_TIMEOUT=25.0 in my launch service script.
    25.0 [sec] seems long enough for me.

    With shorter timeout values...,


    If the above source code change is acceptable, I'll create a new ticket on the GitHub site.
    Then, I'll update the existing pull request.

    Thank you and best regards,
    Kazzz-S

  • MatthiasMatthias

    Hi @sekigawa,

    I see the same effects myself too. I do not think it is a problem with your ISP. Sporadically the connection timeouts after 10 seconds and the server reports error 500. Usual access time is less than a second.

    The way you can test that easily is with curl:

    time curl https://sami.klayout.org/repository.xml

    When the error happens, I do not see anything suspicious in the logs.

    I will try to reboot and if that does not help I am going to contact the support team.

    Sorry for the effort you are spending on that. I really think the problem is on my side. But I do not really know how to solve it yet.

    Matthias

  • MatthiasMatthias

    I just rebooted. I hope no one got kicked out of a session ...

    It looks like the problem is gone - at least for now.

    I have set up a cron job to monitor the performance over time. Maybe I get some better idea what is going on.

    Thanks again and best regards,

    Matthias

  • sekigawasekigawa

    Hi @Matthias,

    Thanks again for looking into this problem.
    Between 23:42 and 23:45 (Munich time) 2024-06-24, I attempted twice and got two timeouts.
    With the curl command,

    : 
  </salt-grain>
</salt-mine>
curl https://sami.klayout.org/repository.xml  0.05s user 0.09s system 0% cpu 20.066 total

    Regards,
    Kazzz-S

  • MatthiasMatthias

    Hi @sekigawa,

    So it actually looks to be slow, but there is no 500 error as I see. Here is my annotated monitoring log (MUC time):

    Mo 24. Jun 23:11:57 CEST 2024 - 0:00.47
Mo 24. Jun 23:12:01 CEST 2024 - 0:09.84  # error 500
Mo 24. Jun 23:14:01 CEST 2024 - 0:01.73
Mo 24. Jun 23:16:01 CEST 2024 - 0:00.48
Mo 24. Jun 23:18:01 CEST 2024 - 0:00.51
Mo 24. Jun 23:20:01 CEST 2024 - 0:09.99  # error 500
Mo 24. Jun 23:22:01 CEST 2024 - 0:00.50
Mo 24. Jun 23:24:01 CEST 2024 - 0:00.49
Mo 24. Jun 23:26:01 CEST 2024 - 0:20.79  # error 500
Mo 24. Jun 23:28:01 CEST 2024 - 0:00.49
--- reboot ---
Mo 24. Jun 23:30:01 CEST 2024 - 0:00.52
Mo 24. Jun 23:32:01 CEST 2024 - 0:00.50
Mo 24. Jun 23:34:01 CEST 2024 - 0:00.48
Mo 24. Jun 23:36:01 CEST 2024 - 0:00.66
Mo 24. Jun 23:38:01 CEST 2024 - 0:00.46
Mo 24. Jun 23:40:01 CEST 2024 - 0:00.52
Mo 24. Jun 23:42:01 CEST 2024 - 0:00.52
Mo 24. Jun 23:44:01 CEST 2024 - 0:00.49
Mo 24. Jun 23:46:01 CEST 2024 - 0:02.06
Mo 24. Jun 23:48:01 CEST 2024 - 0:00.47
Mo 24. Jun 23:50:01 CEST 2024 - 0:00.52
Mo 24. Jun 23:52:01 CEST 2024 - 0:00.50
Mo 24. Jun 23:54:01 CEST 2024 - 0:00.50
Mo 24. Jun 23:56:01 CEST 2024 - 0:01.23

    So there is some jitter, but the 500 problem is gone. My server is a vhost which means it shares CPU with other servers. Maybe there is a rogue one. This may explain the 500 errors.

    Since you still see timeouts maybe your ISP really is to blame.

    I noticed that the repository.xml is already more than 2MB in size - this is mainly due to the screenshot images attached. I should actually restrict the size of images allowed for the screenshots or not transmit them in the repository index.

    But to my understanding, the file size should not be responsible for a timeout - transmission may take longer, but a timeout only happens if the server is not responding at all within a certain time frame.

    Best regards,

    Matthias

  • sekigawasekigawa

    GitHub ticket: https://github.com/KLayout/klayout/issues/1756
    Proposed PR: https://github.com/KLayout/klayout/pull/1762
    Adopted PR: https://github.com/KLayout/klayout/pull/1764

  • MatthiasMatthias

    Thanks a lot :)

    I checked my monitoring log and response times for the repository index are typically at 0.5s with a few excursions, but with one exception below 10s.

    But no 500 errors :)

    Matthias

Sign In or Register to comment.
Powered by Vanilla