OASIS is a very compact format. 6GB out of 700MB is not entirely unexpected. Typical values are around 3..4x times the size of the OASIS file. But it's hard to predict. If CBLOCK compression is involved, much larger factors are possible.
If you only want to view the data, make sure you run KLayout in viewer mode. That's reducing the memory footprint as far as possible.
Apart from that there aren't many options. But memory is cheap.
Just out of curiosity, it appears that the oasis file is fully loaded and "expanded" before displaying. Is that the case?
In addition, it appears by default only single thread is used, as the load time is roughly the same between my laptop and the 8-core desktop I have. I can not find if there is a multi-threading flag that can be used
What do you mean by "expanded"? Shape arrays are kept in viewer mode, but not in editor mode (you want to edit shapes, not the arrays). That makes a huge difference.
Multi-Threaded reading is not necessarily applicable - only if the OASIS file support indexing (strict mode), such a thing can be implemented. But 700M of OASIS hardly loads more than a minute in viewer mode. If you're loading over a network, the file transfer times are still dominant.
what I mean is that the hierachy is fully flattened before displaying. Is that the case? I can not figure out why the memory usage is 6GB for a 700MB file.
also the load time is like 5 minutes and the oasis file is located in local SSD
As I said, OASIS is basically pretty compact (relative coordinates, modal variables, deltas, etc.). What's a 32bit coordinate in memory can be less than a byte in OASIS. Plus OASIS features built-in compression. So when it is read, some expansion happens. The ratio is hard to predict.
In some cases ("shallow" hierarchy like big stdcell designs), memory fragmentation can happen and the load times may get bigger due to the post-read sorting effort. This is known to be an issue in some cases, but I'm not able to reproduce this behaviour on synthetic test cases.
I assume you're not allowed to send the test case, so apart from guessing, I can't do much more.
Just to be clarify, the memory increase is due to the "uncompress" the compressed stuff when loading oasis into memory, right? If that is case, if somehow we can keep the compressed stuff in the memory, the memory usage will be much reduced.
Sorry I am allowed send test case I have - will try several files that are "publicly" available and see if the same problem happens. In the mean time, can you suggest how to tell if load time is due to this "post-read" sorting thing, or other things?
keeping things "compressed" in memory is already happening to some extend. But you cannot keep modal variables (they depend on sequence, so no random access), CBLOCK compression (also no random access) and you have to attach data management information which also takes memory but is required to implement search trees for example.
Unless there is a issue with memory fragmentation I don't believe there is much room for improvement without seriously compromising functionality or drawing performance. Unless I can debug the case, I can't tell what's going on or propose a solution.
You can enable diagnostic output with
klayout -d 31 -ne your_file.oas
(-ne forces viewer mode).
The output gets pretty noisy, but the interesting part is this:
This sample is taken from a 64MB OASIS file which I consider representative for a product layout. Loading takes about 5s, sorting 4s (usually the sorting part is a small fraction). Memory is 206MB (Total required). Essentially, somewhat more is allocated due to fragmentation (in this case ~300MB), but depending on the nature of the layout, these "memory holes" will be filled by other parts.
In contrast to this, loading in edit mode renders:
the oas file I am using is about 700MB. It seems the memory usage and individual time (like sorting) is not out of whack, comparing to the 64MB test case you use, but the total file read time is quite long
Can you take a look and suggest if anything can be done to speed up the file loading?
I think the file loading time is inline with other layout tools. I get 1.6GB memory use for a 0.3GB compressed GDSII file which I think makes sense. The redraw times are pretty good once the file is loaded. To speed up things, I would go to a machine with say 5-10x the memory of a compressed GSDII or Oasis file.
Your numbers indeed show that there is an issue with the loading times. They should be about a tenth. The numbers are not
Here are some questions:
If you save the file to OASIS and load it again - will the load times stay the same?
Does your layout contain user properties? There was a performance issue with those once. It should be solved, but maybe it still persists in certain configurations. User properties are more critical for performance if there any many different keys.
Maybe you can run the statistics with the most recent version. It features a more accurate memory footprint calculation. Are those numbers different?
In File/Layout Statistics there is a link called "Detailed layer statistics" (computation may take some time). Maybe you can paste this function's output here.
currently there is no statistics about user properties. But if you select a shape and use "Edit/Properties", a dialog with the shape's properties opens. After clicking at the "User properties" button, you'll see the user properties.
User properties are sometimes used to attach net names to shapes for example. Their memory footprint is included in "Layout info". As there isn't much, there cannot be many different property sets in your case. So I'd rule out this as a potential cause here.
Comments
Hi Steven,
OASIS is a very compact format. 6GB out of 700MB is not entirely unexpected. Typical values are around 3..4x times the size of the OASIS file. But it's hard to predict. If CBLOCK compression is involved, much larger factors are possible.
If you only want to view the data, make sure you run KLayout in viewer mode. That's reducing the memory footprint as far as possible.
Apart from that there aren't many options. But memory is cheap.
Matthias
Just out of curiosity, it appears that the oasis file is fully loaded and "expanded" before displaying. Is that the case?
In addition, it appears by default only single thread is used, as the load time is roughly the same between my laptop and the 8-core desktop I have. I can not find if there is a multi-threading flag that can be used
Hi Steven,
What do you mean by "expanded"? Shape arrays are kept in viewer mode, but not in editor mode (you want to edit shapes, not the arrays). That makes a huge difference.
Multi-Threaded reading is not necessarily applicable - only if the OASIS file support indexing (strict mode), such a thing can be implemented. But 700M of OASIS hardly loads more than a minute in viewer mode. If you're loading over a network, the file transfer times are still dominant.
Matthias
also the load time is like 5 minutes and the oasis file is located in local SSD
Thanks
steven
Hi Steven,
no, no flattening happens.
As I said, OASIS is basically pretty compact (relative coordinates, modal variables, deltas, etc.). What's a 32bit coordinate in memory can be less than a byte in OASIS. Plus OASIS features built-in compression. So when it is read, some expansion happens. The ratio is hard to predict.
In some cases ("shallow" hierarchy like big stdcell designs), memory fragmentation can happen and the load times may get bigger due to the post-read sorting effort. This is known to be an issue in some cases, but I'm not able to reproduce this behaviour on synthetic test cases.
I assume you're not allowed to send the test case, so apart from guessing, I can't do much more.
Matthias
Just to be clarify, the memory increase is due to the "uncompress" the compressed stuff when loading oasis into memory, right? If that is case, if somehow we can keep the compressed stuff in the memory, the memory usage will be much reduced.
Sorry I am allowed send test case I have - will try several files that are "publicly" available and see if the same problem happens. In the mean time, can you suggest how to tell if load time is due to this "post-read" sorting thing, or other things?
Thanks
Steven
Hi Steven,
keeping things "compressed" in memory is already happening to some extend. But you cannot keep modal variables (they depend on sequence, so no random access), CBLOCK compression (also no random access) and you have to attach data management information which also takes memory but is required to implement search trees for example.
Unless there is a issue with memory fragmentation I don't believe there is much room for improvement without seriously compromising functionality or drawing performance. Unless I can debug the case, I can't tell what's going on or propose a solution.
You can enable diagnostic output with
(-ne forces viewer mode).
The output gets pretty noisy, but the interesting part is this:
This sample is taken from a 64MB OASIS file which I consider representative for a product layout. Loading takes about 5s, sorting 4s (usually the sorting part is a small fraction). Memory is 206MB (Total required). Essentially, somewhat more is allocated due to fragmentation (in this case ~300MB), but depending on the nature of the layout, these "memory holes" will be filled by other parts.
In contrast to this, loading in edit mode renders:
In contrast to the previous sample, memory footprint is 795MB.
Please note that both samples are obtained with the upcoming next release which is somewhat more accurate in collecting memory footprints.
Maybe you can paste the corresponding numbers of yours.
Thanks,
Matthias
Matthias
Here is the debug log
the oas file I am using is about 700MB. It seems the memory usage and individual time (like sorting) is not out of whack, comparing to the 64MB test case you use, but the total file read time is quite long
Can you take a look and suggest if anything can be done to speed up the file loading?
Thanks
Steven
I think the file loading time is inline with other layout tools. I get 1.6GB memory use for a 0.3GB compressed GDSII file which I think makes sense. The redraw times are pretty good once the file is loaded. To speed up things, I would go to a machine with say 5-10x the memory of a compressed GSDII or Oasis file.
Best regards, Erwin
Hi Steven,
Your numbers indeed show that there is an issue with the loading times. They should be about a tenth. The numbers are not
Here are some questions:
Kind regards,
Matthias
How to tell if the file contains user properties (and the amount)?
I am using the 0.25.2.
will try #1 and #4 later today and post the findings
Thanks
Steven
Hi Steven,
currently there is no statistics about user properties. But if you select a shape and use "Edit/Properties", a dialog with the shape's properties opens. After clicking at the "User properties" button, you'll see the user properties.
User properties are sometimes used to attach net names to shapes for example. Their memory footprint is included in "Layout info". As there isn't much, there cannot be many different property sets in your case. So I'd rule out this as a potential cause here.
Matthias