Red alert: KLayout Package Manager will stop working beginning of 2024 with GitHub-hosted packages

Matthias

GitHub will discontinue Subversion support, which is the basis of all the packages hosted in KLayout (https://github.blog/2023-01-20-sunsetting-subversion-support/). KLayout uses the SVN interface to download the package data.

As a quick mitigation, it is possible to host packages on SourceForge with Subversion selected as the code tool. This will mean manual effort upon releasing GitHub-hosted projects. This used to be very convenient with the subversion bridge.

To facilitate this migration I will add SourceForge to the package repositories and remove GitHub.

Existing packages will be transferred to a read-only KLayout-owned SVN repository so they will continue working.

As a long-term solution, I plan to add Git support for package download. That means:

• It will again be possible to deploy KLayout packages directly from GitHub
• KLayout packages have to reside at the top level of a Git repo (specifically: grain.xml needs to be located at top level). So far, it was possible to use a sub-folder as the package home. This will no longer work. A workaround is to use a Git submodule for the package part.
• This deployment path will only work for new KLayout versions (target is 0.29)

There is a ticket for this: https://github.com/KLayout/klayout/issues/1514

Matthias

Matthias

Here is a brief recipe for the transfer to SoureForge. The example is for the klayoutmatthias/xsection project.

First, create a project at SourceForge. Make sure you select "Subversion" for the Repository type. In my case the project was "klayoutmatthias-xsection". You should be able to navigate to the project and on the "Code" page and find SVN checkout instructions. In my case there are two ways:

SSH: via URL svn+ssh://koefferl@svn.code.sf.net/p/klayoutmatthias-xsection/code/

HTTPS: via URL https://svn.code.sf.net/p/klayoutmatthias-xsection/code/

We will use SSH for the import and HTTPS as the deployment path. For SSH you need to register your public SSH key in your User Profile.

I want to import tags only for the releases, so I created a "tags" folder in the Subversion repo:

svn mkdir -m "Initialization" svn+ssh://<user>@svn.code.sf.net/p/klayoutmatthias-xsection/code/tags
# (<user> is my SourceForge account name).

To transfer a GitHub release (tag) to a Subversion tag on SourceForge, I use these commands for the 1.7 tag:

# In some temporary directory:
git clone https://github.com/klayoutmatthias/xsection.git --depth=1 -b 1.7 
cd xsection
rm -rf .git
svn import svn+ssh://<user>@svn.code.sf.net/p/klayoutmatthias-xsection/code/tags/1.7 -m "tag 1.7 imported"
# (again, <user> is my SourceForge account name).

With the SourceForge mirror in place, I can substitute the original GitHub URL by the SourceForge URL:

https://github.com/klayoutmatthias/xsection.git/tags/1.7/src

becomes

https://svn.code.sf.net/p/klayoutmatthias-xsection/code/tags/1.7/src

Note, that the package data itself is not located directly in the top level of the module, but inside the "src" subfolder. Hence, we need to add "src" to the tag path.

Matthias

I have started duplicating the first project tags to a separate SVN server. This will keep the packages working after January 2024. But as these are snapshots take today, this will disconnect them from the evolution on GitHub.

The new package URLs will be formed by replacing https://github.com with https://www.klayout.org/svn-public/packages-migrated.
Technically, the SVN repo is a snapshot of selected pieces from the GitHub space.

Matthias

dick_freebird

I find GitHub baffling and don't think I've ever found
what I'm looking for, there. I always took the latest off
either SourceForge or the klayout site "Get" links. As
long as those have good latest / greatest, GitHub can
disappear for all I care.

lukasc

Hi Matthias,

I see that SourceForge provides an automated synchronization between GitHub and SourceForge. Could that be useful?
https://sourceforge.net/p/import_project/github/

When do you anticipate having v0.29 and a new Git integration released? If before Jan 2024, then perhaps we can wait for that instead of using the SVN workaround?

In your top-level approach, would there be mechanism to only include specific subfolders, or to exclude specific folders?

thank you
Lukas

Matthias

Hi Lukas,

I tried the sync feature of SourceForge, but as I understand it, it can only synchronize Git to Git and not Git to SVN.

I am currently implementing git support for download. git cannot efficiently download subtrees - it will always download the worktree for the full repository and that makes retrieving a single file like grain.xml rather inefficient. There is shallow and sparse checkout which helps mitigating the effects somewhat. Still things will be considerably slower with git.

Maybe it is possible to support sub-folders by pruning everything above. However, it still is required to download the whole tree first. In case of https://github.com/siepic/SiEPIC_EBeam_PDK.git for example that means downloading 277MB of data, just to retrieve grain.xml.

Matthias

lukasc

If it is a question of accessing all the grain.xml files, what about just queering GitHub for the single file? e.g.,

https://raw.githubusercontent.com/SiEPIC/SiEPIC_EBeam_PDK/master/klayout/grain.xml

Putting the grain.xml file at the top of the repo should be ok.

Matthias

Hi Lukas,

It's unfortunately the Git protocol is very bad in delivering single files only. But I found a solution that is acceptable. It involves caching all the repository description files on the Salt Mine server and using that cached information inside KLayout. So there is little need to download the full package except if someone installs it.

It took some rework of the server application plus I had to upgrade the Windows builds and now everything is breaking. But I am working on that. Basically a solution is at sight

Matthias