memory required to open oas file

edited April 2018 in General
Hi

I am trying to open an oasis file with ~700MB. The memory footprint is about 6GB. is this expected?

Appreciate any suggestions/comments on reducing memory usage

Steven

Comments

  • edited November -1

    Hi Steven,

    OASIS is a very compact format. 6GB out of 700MB is not entirely unexpected. Typical values are around 3..4x times the size of the OASIS file. But it's hard to predict. If CBLOCK compression is involved, much larger factors are possible.

    If you only want to view the data, make sure you run KLayout in viewer mode. That's reducing the memory footprint as far as possible.

    Apart from that there aren't many options. But memory is cheap.

    Matthias

  • edited November -1
    Thanks Matthias.

    Just out of curiosity, it appears that the oasis file is fully loaded and "expanded" before displaying. Is that the case?

    In addition, it appears by default only single thread is used, as the load time is roughly the same between my laptop and the 8-core desktop I have. I can not find if there is a multi-threading flag that can be used
  • edited November -1

    Hi Steven,

    What do you mean by "expanded"? Shape arrays are kept in viewer mode, but not in editor mode (you want to edit shapes, not the arrays). That makes a huge difference.

    Multi-Threaded reading is not necessarily applicable - only if the OASIS file support indexing (strict mode), such a thing can be implemented. But 700M of OASIS hardly loads more than a minute in viewer mode. If you're loading over a network, the file transfer times are still dominant.

    Matthias

  • edited November -1
    what I mean is that the hierachy is fully flattened before displaying. Is that the case? I can not figure out why the memory usage is 6GB for a 700MB file.

    also the load time is like 5 minutes and the oasis file is located in local SSD

    Thanks
    steven
  • edited November -1

    Hi Steven,

    no, no flattening happens.

    As I said, OASIS is basically pretty compact (relative coordinates, modal variables, deltas, etc.). What's a 32bit coordinate in memory can be less than a byte in OASIS. Plus OASIS features built-in compression. So when it is read, some expansion happens. The ratio is hard to predict.

    In some cases ("shallow" hierarchy like big stdcell designs), memory fragmentation can happen and the load times may get bigger due to the post-read sorting effort. This is known to be an issue in some cases, but I'm not able to reproduce this behaviour on synthetic test cases.

    I assume you're not allowed to send the test case, so apart from guessing, I can't do much more.

    Matthias

  • edited November -1
    Thanks.

    Just to be clarify, the memory increase is due to the "uncompress" the compressed stuff when loading oasis into memory, right? If that is case, if somehow we can keep the compressed stuff in the memory, the memory usage will be much reduced.

    Sorry I am allowed send test case I have - will try several files that are "publicly" available and see if the same problem happens. In the mean time, can you suggest how to tell if load time is due to this "post-read" sorting thing, or other things?

    Thanks
    Steven
  • edited November -1

    Hi Steven,

    keeping things "compressed" in memory is already happening to some extend. But you cannot keep modal variables (they depend on sequence, so no random access), CBLOCK compression (also no random access) and you have to attach data management information which also takes memory but is required to implement search trees for example.

    Unless there is a issue with memory fragmentation I don't believe there is much room for improvement without seriously compromising functionality or drawing performance. Unless I can debug the case, I can't tell what's going on or propose a solution.

    You can enable diagnostic output with

    klayout -d 31 -ne your_file.oas
    

    (-ne forces viewer mode).

    The output gets pretty noisy, but the interesting part is this:

    File read: 4.65 (user) 0.27 (sys) 5.188 (wall) 820.72M (mem)
    Updating relations: 0.61 (user) 0 (sys) 0.625 (wall) 850.80M (mem)
    Topological sort: 0.05 (user) 0 (sys) 0.051 (wall) 850.96M (mem)
    Updating bounding boxes: 0.57 (user) 0.02 (sys) 0.587 (wall) 869.27M (mem)
    Sorting shapes: 0.11 (user) 0 (sys) 0.115 (wall) 869.40M (mem)
    Sorting instances: 2.24 (user) 0.01 (sys) 2.362 (wall) 889.25M (mem)
    Sorting: 3.58 (user) 0.03 (sys) 3.746 (wall) 889.25M (mem)
    Memory usage per master category:
      Layout info    : 5307430 (used) 5501810 (reqd)
      Cell info      : 11513116 (used) 11513116 (reqd)
      Instances      : 118589888 (used) 118589888 (reqd)
      Shapes info    : 58582320 (used) 70019656 (reqd)
      Total          : 193992754 (used) 205624470 (reqd)
    Loading: 8.34 (user) 0.3 (sys) 9.061 (wall) 865.25M (mem)
    

    This sample is taken from a 64MB OASIS file which I consider representative for a product layout. Loading takes about 5s, sorting 4s (usually the sorting part is a small fraction). Memory is 206MB (Total required). Essentially, somewhat more is allocated due to fragmentation (in this case ~300MB), but depending on the nature of the layout, these "memory holes" will be filled by other parts.

    In contrast to this, loading in edit mode renders:

    File read: 3.83 (user) 0.22 (sys) 4.141 (wall) 1095.01M (mem)
    Updating relations: 0.61 (user) 0.01 (sys) 0.614 (wall) 1124.84M (mem)
    Topological sort: 0.03 (user) 0.01 (sys) 0.043 (wall) 1125.00M (mem)
    Updating bounding boxes: 0.88 (user) 0.02 (sys) 0.91 (wall) 1143.30M (mem)
    Sorting shapes: 1.87 (user) 0.06 (sys) 1.944 (wall) 1300.22M (mem)
    Sorting instances: 0.83 (user) 0.01 (sys) 0.866 (wall) 1334.90M (mem)
    Sorting: 4.23 (user) 0.11 (sys) 4.381 (wall) 1334.90M (mem)
    Memory usage per master category:
      Layout info    : 4676958 (used) 4871338 (reqd)
      Cell info      : 11513116 (used) 11513116 (reqd)
      Instances      : 147754280 (used) 147754280 (reqd)
      Shapes info    : 472188624 (used) 631341736 (reqd)
      Total          : 636132978 (used) 795480470 (reqd)
    Loading: 8.36 (user) 0.33 (sys) 8.827 (wall) 1310.89M (mem)
    

    In contrast to the previous sample, memory footprint is 795MB.

    Please note that both samples are obtained with the upcoming next release which is somewhat more accurate in collecting memory footprints.

    Maybe you can paste the corresponding numbers of yours.

    Thanks,

    Matthias

  • edited May 2018

    Matthias

    Here is the debug log

    File read: 301.35 (user) 3.11 (sys) 305.498 (wall) 5686.76M (mem)
    Updating relations: 0.86 (user) 0.03 (sys) 0.893 (wall) 5764.23M (mem)
    Topological sort: 0.42 (user) 0 (sys) 0.421 (wall) 5765.04M (mem)
    Updating bounding boxes: 8.1 (user) 0.06 (sys) 8.164 (wall) 5841.09M (mem)
    Sorting shapes: 14.03 (user) 0.02 (sys) 14.082 (wall) 5862.11M (mem)
    Sorting instances: 2.97 (user) 0 (sys) 2.982 (wall) 5866.17M (mem)
    Sorting: 26.39 (user) 0.11 (sys) 26.55 (wall) 5866.17M (mem)
    Memory usage:
       Layout info    6123623 (used) 5835361 (reqd)
       Cell info      88942156 (used) 83769542 (reqd) 
       Instances      80381766 (used) 80281545 (reqd) 
       Instance trees 176503936 (used) 169389366 (reqd) 
       Shapes info    2378646418 (used) 1936823579 (reqd) 
       Shapes cache   47679331 (used) 43211872 (reqd) 
    Loading: 328.7 (user) 3.22 (sys) 333.022 (wall) 5863.88M (mem)
    

    the oas file I am using is about 700MB. It seems the memory usage and individual time (like sorting) is not out of whack, comparing to the 64MB test case you use, but the total file read time is quite long

    Can you take a look and suggest if anything can be done to speed up the file loading?

    Thanks
    Steven

  • edited November -1
    Hi Matthias, Steven:

    I think the file loading time is inline with other layout tools. I get 1.6GB memory use for a 0.3GB compressed GDSII file which I think makes sense. The redraw times are pretty good once the file is loaded. To speed up things, I would go to a machine with say 5-10x the memory of a compressed GSDII or Oasis file.

    Best regards, Erwin
  • edited May 2018

    Hi Steven,

    Your numbers indeed show that there is an issue with the loading times. They should be about a tenth. The numbers are not

    Here are some questions:

    • If you save the file to OASIS and load it again - will the load times stay the same?
    • Does your layout contain user properties? There was a performance issue with those once. It should be solved, but maybe it still persists in certain configurations. User properties are more critical for performance if there any many different keys.
    • Maybe you can run the statistics with the most recent version. It features a more accurate memory footprint calculation. Are those numbers different?
    • In File/Layout Statistics there is a link called "Detailed layer statistics" (computation may take some time). Maybe you can paste this function's output here.

    Kind regards,

    Matthias

  • edited November -1
    Hi Matthias

    How to tell if the file contains user properties (and the amount)?

    I am using the 0.25.2.

    will try #1 and #4 later today and post the findings

    Thanks
    Steven
  • edited November -1

    Hi Steven,

    currently there is no statistics about user properties. But if you select a shape and use "Edit/Properties", a dialog with the shape's properties opens. After clicking at the "User properties" button, you'll see the user properties.

    User properties are sometimes used to attach net names to shapes for example. Their memory footprint is included in "Layout info". As there isn't much, there cannot be many different property sets in your case. So I'd rule out this as a potential cause here.

    Matthias

Sign In or Register to comment.