Red alert: KLayout Package Manager will stop working beginning of 2024 with GitHub-hosted packages

Matthias

GitHub will discontinue Subversion support, which is the basis of all the packages hosted in KLayout (https://github.blog/2023-01-20-sunsetting-subversion-support/). KLayout uses the SVN interface to download the package data.

As a quick mitigation, it is possible to host packages on SourceForge with Subversion selected as the code tool. This will mean manual effort upon releasing GitHub-hosted projects. This used to be very convenient with the subversion bridge.

To facilitate this migration I will add SourceForge to the package repositories and remove GitHub.

Existing packages will be transferred to a read-only KLayout-owned SVN repository so they will continue working.

As a long-term solution, I plan to add Git support for package download. That means:

• It will again be possible to deploy KLayout packages directly from GitHub
• KLayout packages have to reside at the top level of a Git repo (specifically: grain.xml needs to be located at top level). So far, it was possible to use a sub-folder as the package home. This will no longer work. A workaround is to use a Git submodule for the package part.
• This deployment path will only work for new KLayout versions (target is 0.29)

There is a ticket for this: https://github.com/KLayout/klayout/issues/1514

Matthias

Matthias

Here is a brief recipe for the transfer to SoureForge. The example is for the klayoutmatthias/xsection project.

First, create a project at SourceForge. Make sure you select "Subversion" for the Repository type. In my case the project was "klayoutmatthias-xsection". You should be able to navigate to the project and on the "Code" page and find SVN checkout instructions. In my case there are two ways:

SSH: via URL svn+ssh://koefferl@svn.code.sf.net/p/klayoutmatthias-xsection/code/

HTTPS: via URL https://svn.code.sf.net/p/klayoutmatthias-xsection/code/

We will use SSH for the import and HTTPS as the deployment path. For SSH you need to register your public SSH key in your User Profile.

I want to import tags only for the releases, so I created a "tags" folder in the Subversion repo:

svn mkdir -m "Initialization" svn+ssh://<user>@svn.code.sf.net/p/klayoutmatthias-xsection/code/tags
# (<user> is my SourceForge account name).

To transfer a GitHub release (tag) to a Subversion tag on SourceForge, I use these commands for the 1.7 tag:

# In some temporary directory:
git clone https://github.com/klayoutmatthias/xsection.git --depth=1 -b 1.7 
cd xsection
rm -rf .git
svn import svn+ssh://<user>@svn.code.sf.net/p/klayoutmatthias-xsection/code/tags/1.7 -m "tag 1.7 imported"
# (again, <user> is my SourceForge account name).

With the SourceForge mirror in place, I can substitute the original GitHub URL by the SourceForge URL:

https://github.com/klayoutmatthias/xsection.git/tags/1.7/src

becomes

https://svn.code.sf.net/p/klayoutmatthias-xsection/code/tags/1.7/src

Note, that the package data itself is not located directly in the top level of the module, but inside the "src" subfolder. Hence, we need to add "src" to the tag path.

Matthias

I have started duplicating the first project tags to a separate SVN server. This will keep the packages working after January 2024. But as these are snapshots take today, this will disconnect them from the evolution on GitHub.

The new package URLs will be formed by replacing https://github.com with https://www.klayout.org/svn-public/packages-migrated.
Technically, the SVN repo is a snapshot of selected pieces from the GitHub space.

Matthias

dick_freebird

I find GitHub baffling and don't think I've ever found
what I'm looking for, there. I always took the latest off
either SourceForge or the klayout site "Get" links. As
long as those have good latest / greatest, GitHub can
disappear for all I care.

lukasc

Hi Matthias,

I see that SourceForge provides an automated synchronization between GitHub and SourceForge. Could that be useful?
https://sourceforge.net/p/import_project/github/

When do you anticipate having v0.29 and a new Git integration released? If before Jan 2024, then perhaps we can wait for that instead of using the SVN workaround?

In your top-level approach, would there be mechanism to only include specific subfolders, or to exclude specific folders?

thank you
Lukas

Matthias

Hi Lukas,

I tried the sync feature of SourceForge, but as I understand it, it can only synchronize Git to Git and not Git to SVN.

I am currently implementing git support for download. git cannot efficiently download subtrees - it will always download the worktree for the full repository and that makes retrieving a single file like grain.xml rather inefficient. There is shallow and sparse checkout which helps mitigating the effects somewhat. Still things will be considerably slower with git.

Maybe it is possible to support sub-folders by pruning everything above. However, it still is required to download the whole tree first. In case of https://github.com/siepic/SiEPIC_EBeam_PDK.git for example that means downloading 277MB of data, just to retrieve grain.xml.

Matthias

lukasc

If it is a question of accessing all the grain.xml files, what about just queering GitHub for the single file? e.g.,

https://raw.githubusercontent.com/SiEPIC/SiEPIC_EBeam_PDK/master/klayout/grain.xml

Putting the grain.xml file at the top of the repo should be ok.

Matthias

Hi Lukas,

It's unfortunately the Git protocol is very bad in delivering single files only. But I found a solution that is acceptable. It involves caching all the repository description files on the Salt Mine server and using that cached information inside KLayout. So there is little need to download the full package except if someone installs it.

It took some rework of the server application plus I had to upgrade the Windows builds and now everything is breaking. But I am working on that. Basically a solution is at sight

Matthias

wsteffe

Hi Matthias, any news on this problem ?
I have just tried to update my layoutDD package (hosted at hithub) with the usual procedure but it didn't work.
Is there anything that I can do on my side to solve the issue ?

wsteffe

Now I have been able to register layoutDD v0.4.1 with a small change in the link.
Previously I had to enter the url of the specific release https://github.com/wsteffe/layoutDD/tags/v0.4.1
but it doesn't work any more. Instead it worked ponting directly to the main page https://github.com/wsteffe/layoutDD

wsteffe

But now the problem is that I can't register a new update because the link https://github.com/wsteffe/layoutDD can't be reused a second time (unless I delete the package and register it again).

wsteffe

To be more precise the problem is that the package version reported at Salt.Mine page is not updated without a registration update (which can't be done reusing same url).

Matthias

Hi @wsteffe,

well, yes. News is that GitHub now shut down their SVN bridge forever and the convenient "tags/xyz" thingy no longer works.

I assume you wanted to register tag v0.4.2. In that case the URL is:

git+https://github.com/wsteffe/layoutDD.git[v0.4.2]

Please note the ".git", which was not mandatory before, and the tag in square brackets.

Now, the package will be reachable through the Git integration in KLayout, but this also means you will need KLayout >= 0.28.13 for downloading it. Another option is to copy the package to SourceForge which still supports SVN, but I share security concerns about this provider.

I edited the package link and for me it works now. Before GitHub killed the SVN port, I migrated some packages to a private SVN server of mine to keep them functional, but this is not some scheme I want to keep up forever.

But package update should be possible with the same URL, but only from the original user account (here: mail address). Every package (by name) can only be updated or modified by the original user.

Matthias