Could you please extract all the Ruby code for the DRC engine outside of Klayout main build?

Hi @Matthias,

First of all, I would like to thank you for this amazing tool. I'm currently working on developing a DRC/LVS rule deck using Klayout. We are facing some memory capacity and performance limitations.

I'm investigating the possibility of extracting the verification code DRC/LVS written in ruby that you have in Klayout and making it as separate project and try to enhance the performance of that code allowing us to run it from the command line.

Here is what I have found, you store the entire DRC/LVS ruby code as resource inside the Qt application "Klayout". And run it from inside once needed. What I'm basically asking if you could take out that code for DRC/LVS and any required parts of klayout and make them as separate project/repo that basically has a build and allow us to run drc and lvs like this:

ruby run_drc.rb .....

ruby run_lvs.rb

All of that without all the Qt wrappers.

And it compiles all necessary klayout parts as shared libraries that is basically loaded in your ruby code to make it work. I'm struggling with Ruby as I'm not a ruby programmer and it takes me a lot of time to do that. I would imagine it would be much easier for you to do this. If you can do this in a separate repo that would be amazing.

BTW, I might make some major changes to your code once you do this to enhance the performance. Also, even if you feel that it would be hard to take out the underlying base shared libraries as it's tied hardly to the application code and you still need the core klayout code, I would only need the ruby code out first to be able to work even if it's not running and missing the shared libraries. I'm doing some investigations that I believe worth while.

Appreciate all your help.

Regards,
Amro

Comments

  • @Matthias FYI, we don't need a GUI for such tool. Only command line would be the best.

  • I can see value in a standalone, GDS-II DRC checker.
    However I imagine that the operation of DRC here, has
    to take advantage of the GDS-II layout loaded-database
    infrastructure, and to "bust out" the DRC you'd probably
    end up duplicating that - to what advantage, then?
    Expect memory usage goes mostly to that (data structure).

  • @dick_freebird We only need a limited number of libraries that are part of the base DRC check. It makes it easier for us to contribute in klayout code base. Yes, I know we would duplicate that code in separate repo. But we could resolve that to by using sub modules.

  • @atork have you tried using klayout with the cmd-line arguments? you can run klayout without loading the frontend with klayout -zz -r <script-name>? If you want to have klayout completely without qt, you can compile it without qt. I.e. with sh build.sh -without-qt. This will compile what is equivalent to what is in the python module. I am not sure that is what you are looking for. You can also just compile it without the UI part.

    Keep in mind that most of the code base is in C++ not Ruby. So I am not sure there is or whether there ever will be Ruby code for the DRC operations. I think these are all part of the core.db APIs.

  • edited June 2022

    @sebastian I tried all of that. Actually, we have a full rule deck developed that runs without GUI. But we are still seeing a possible improvements if we could isolate the Ruby code manually with all required libraries code outside of klayout main program. Klayout consumes massive amount of Memory even though we use "forget" command which we don't see any changes in memory usage using it. BTW, we track memory usage and add it to the log output to know how much memory we are using.

  • @sebastian we tried

    sh build.sh -without-qt 
    

    And it still required mutliple Qt libraries to exist and the build failed.

  • Of course it will still require certain qt libraries, the build still runs qmake, so I am quite sure you still need some basic qt libs to compile, that doesn't mean that the endproduct uses them. Build requirements vs used libs.

  • edited June 2022

    You're lucky, Qt-less build is on it's way :)

    The code for KLayout without Qt is in the "qtless-canvas2" branch: https://github.com/KLayout/klayout/tree/qtless-canvas2

    I will merge that soon to master. In my case I am able to build a "klayout" binary without Qt which then of course makes sense only with "klayout -b ...".

    The technical background for this activity is to provide a native Python module which - without need for Qt - still supports drawing of layouts. It can basically utilize a web page as a (rudimentary) user interface (see https://github.com/klayoutmatthias/canvas2canvas).

    It is correct that you will still need qmake for building that application, but there will not be dependencies on Qt libraries anymore.

    Matthias

  • Hi @Matthias ,

    That's great news. Looking forward to seeing this it would definitely help.

    But I think mine is different, just to clarify my request, I only want the DRC/LVS sections only from klayout without the rest at that point. Looking at the code, Here is what I want to isolate:

    https://github.com/KLayout/klayout/tree/master/src/drc
    https://github.com/KLayout/klayout/tree/master/src/lvs

    And based on this, here the dependencies mentioned in: https://github.com/KLayout/klayout/blob/master/src/drc/drc/drc.pro

    And for lvs:

    All of that is easy to take out and try to build. But the part that I'm struggling with is that I want to avoid the need to remove the need for DRC/LVS Ruby code to be a Qt Resource to Klayout. I basically need the base Ruby code to run directly and include the built libraries from the dependencies above. This will remove the Qt dependency entirely and make this a command line tool. Allowing us to modify and optimize and increase it's capacity if needed.

    BTW, I already tried and failed to isolate it as I'm not very familiar with both Qt and Ruby coding. That's why I sent my request here hoping that it would be much easier for you to do this isolation as you are the developer of the code and aware of all the details.

    @sebastian I understand that. And I'm hoping to remove this dependency as well.

  • edited June 2022

    BTW the reason behind my request is a trial to increase the capacity and speed of the current engine. We are struggling to use Klayout on medium sized designs. It takes a lot of Memory/time to complete.

  • edited June 2022

    The DRC/LVS engine is not entirely independent of the lay module and the Qt-less build will provide enough functionality for supporting DRC+LVS. What is left are exactly the modules you mention. So this definitely is what you ask for.

    But basically I think you're on the wrong track. Reducing DRC+LVS memory isn't a matter of Qt libraries.

    There are manifold other optimization options.

    By default, DRC+LVS runs flat and single threaded. That is the worst case scenario. Try tiled mode with threads or deep mode for hierarchical processing. The latter can be very efficient, but there is no warranty.

    Additionally, you need to make sure you free resources. DRC is single-pass and each layer that you create is a variable that will persist until the end of the script. Use "layer.forget" to release resources once you do no longer need them.

    There are many other ways to optimize. Often you can rewrite your checks to become much more efficient. Bad example: I have seen cases where users did something like

    a_inverted = boundary - a
    b_outside_a = b & a
    

    which is much slower than the equivalent

    b_outside_a = b - a
    

    As a general rule, less shapes mean less memory and faster execution. It's possible often to rearrange steps so only sparse layers are involved in expensive checks. Also try to avoid long-range checks. They are usually very slow.

    My advise is to track execution (verbose, -d levels) and focus on the high score steps. Output the involved layers and apply natural intelligence for optimizing those steps.

    I don't know what DRC and LVS script you work on, so that is mainly guesswork.

    Matthias

  • Thanks @Matthias for the detailed response.

    But here is our findings:

    • Tested .forget but didn't do much. It didn't clear memory that much anyhow.
    • We generate the ruby rule deck. And we analyze dependencies to make sure we have proper order but even that didn't help.
    • We tried deep and tiled options both had their problems. Tiled generated bad DRC results. It has to be carefully set. I'm working setting it properly on our end.
    • I know about the lower number of polygons. I have been writing rule decks my entire life. I believe we tried to avoid this. Even we report the memory usage while running.

    Extracting the code itself will not improve the run. I know that for sure. But will make my life much easier to start tackling this problem of optimizing the actual engine. I have some ideas but not able to test them given that I have to compile the entire Klayout to do that. Making it separate will help do this faster.

  • You can try current master with "-without-qt". That should give you a binary (klayout) which you can use with "klayout -b ..." as usual, but without the Qt overhead.

    There is not much that can be skipped without compromising vital parts of the engine (e.g. readers). So that is as lean as it can get.

    Plain DRC (without tiles or not deep) does not make sense even on medium-scale layouts. So either deep or tiles are mandatory to reduce memory. Tiled mode has a finite search range, so if long-range interactions matter, this may be a problem (e.g. latch-up or density rules). You can switch between modes during DRC execution, so basically you can work on short-range rules with tiles and switch to global more for long-range checks.

    There are manifold options to improve performance without touching the core algorithms. In general however it is very hard to give advise without knowing the actual implementation and having a test case.

    Matthias

  • @Matthias Could I use deep and tiling together?

  • edited June 2022

    @Matthias I have tried using "-without-qt" and I got this:

    In file included from ../../../src/tl/unit_tests/tlFileSystemWatcherTests.cc:24:
    ../../../src/tl/tl/tlFileSystemWatcher.h:32:10: fatal error: QObject: No such file or directory
     #include <QObject>
              ^~~~~~~~~
    compilation terminated.
    make[2]: *** [Makefile:535: tlFileSystemWatcherTests.o] Error 1
    make[2]: *** Waiting for unfinished jobs....
    ../../../src/tl/unit_tests/tlDeferredExecutionTests.cc:26:10: fatal error: QCoreApplication: No such file or directory
     #include <QCoreApplication>
              ^~~~~~~~~~~~~~~~~~
    compilation terminated.
    make[2]: *** [Makefile:443: tlDeferredExecutionTests.o] Error 1
    ../../../src/tl/unit_tests/tlExpressionTests.cc: In member function 'virtual void {anonymous}::TestImpl8::execute(tl::TestBase*)':
    ../../../src/tl/unit_tests/tlExpressionTests.cc:936:16: warning: catching polymorphic type 'class tl::EvalError' by value [-Wcatch-value=]
       } catch (tl::EvalError) {
                    ^~~~~~~~~
    ../../../src/tl/unit_tests/tlExpressionTests.cc:944:16: warning: catching polymorphic type 'class tl::EvalError' by value [-Wcatch-value=]
       } catch (tl::EvalError) {
                    ^~~~~~~~~
    ../../../src/tl/unit_tests/tlExpressionTests.cc:956:16: warning: catching polymorphic type 'class tl::EvalError' by value [-Wcatch-value=]
       } catch (tl::EvalError) {
                    ^~~~~~~~~
    At global scope:
    cc1plus: warning: unrecognized command line option '-Wno-reserved-user-defined-literal'
    make[2]: Leaving directory '../repos/klayout-0.27.10/build-release/tl/unit_tests'
    make[1]: *** [Makefile:62: sub-unit_tests-make_default] Error 2
    make[1]: Leaving directory '../repos/klayout-0.27.10/build-release/tl'
    make: *** [Makefile:48: sub-tl-make_default] Error 2
    
    
Sign In or Register to comment.