It looks like you're new here. If you want to get involved, click one of these buttons!
Hi,
Can we store the relevant information into a disk cache file during the first read of GDS or OASIS?
This way, during the second reading of the file, the related file scanning process can be skipped, speeding up the reading process.
In my daily work, I often need to repeatedly open a file, so this approach can save a lot of time.
Thanks,
Dion
Comments
I'm leaving that to commerical tools. I hate file caching. It is very hard to detect when you need to invalidate it.
Take Virtuoso Express PCells - they drive me crazy. Never the right layout.
Matthias
a) I usually change files before opening them again, in this case a cache won’t help at all.
What’s your use case for opening the same file again and again ?
b) What kind of oasis file size and read times do you experience ?
To see klayout read times, use the log viewer.
My workflow is divided into several steps.Each step requires reading the original OAS as the basis for validation.Therefore, a single file will be opened multiple times.In some cases, the OAS file is approximately 2~3 GB and takes about 20 minutes to load.
What do you run klayout on ?
Windows/Linux/Mac, what kind of cpu/corecount, how much memory ?
What kind of design ?
P&R data ? Maybe with hierarchical fill ? Or fractured (mask) data ?
Please see this discussion on how to provide some information about your data that would be helpful to put your memory usage and runtimes into perspective:
https://www.klayout.de/forum/discussion/1061/memory-required-to-open-oas-file
klayout -d 31 -ne your_file.oas
Hi @dion,
When you say "load" - what kind of loading is that? Are you sure that you are opening the file in read mode? Edit mode is much slower and has a much higher memory footprint.
20 Minutes for 2..3GB OASIS appears too much. The largest OASIS file I have is 330MB (with CBLOCK compression) and loads in 16 seconds with "klayout -ne". So that would be a few minutes for 2..3GB OASIS.
Matthias
Created layout test.oas
Loading: started
Loading file: test.oas with technology:
Reading file: test.oas: started
Sorting: started
Sorting: 51.58 (user) 10.47 (sys) 62.051 (wall) 40717.39M (mem)
Reading file: test.oas: 1177.73 (user) 151.26 (sys) 1330.844 (wall) 40717.39M (mem)
Memory usage per master category:
Layout info : 4611776747 (used) 4613609827 (reqd)
Cell info : 84143932 (used) 84143932 (reqd)
Instances : 369221168 (used) 369221168 (reqd)
Shapes info : 12481865640 (used) 14733187232 (reqd)
Total : 17547007487 (used) 19800162159 (reqd)
Loading: 1195.78 (user) 157.31 (sys) 1354.946 (wall) 40719.16M (mem)
Recomputing layers with shapes in view: started
Recomputing layers with shapes in view: 0 (user) 0 (sys) 0.001 (wall) 40720.16M (mem)
This case is about 3.7G oas file with 'klayout -d 31 -ne'
I use centOS 7.9, the layout is post P&R data with dummy fill
For an OAS file, for example, if there are 10 layers, is it possible to use multiple processes to open this OAS file simultaneously, with each process reading one layer, and then combine them at the end? By using multiprocessing, can the loading speed be increased?
klayout -ne reads a 1GB oas using cblocks (1.4GB uncompressed) in < 60s using 1.5GB of memory on a Mac M1. Testcase is the 700k stdcell bsg_chip from the public ispd24 contest, routed/stdcell filled with OpenRoad, uniquely referenced 36 times, no fill.
Your runtime seems long.
Can you please provide more info about the machine you run on and the type of fill ?
What kind of cpu/corecount ?
How much memory does the machine have ?
Is your 3.7GB file on a local disk or an external disk ?
Is it correct to assume that the fill is hierarchical ?
I’ve seen klayout’s internal query trees be slow in the presence of hierarchical fill, but have no testcase.
Can you compare file size/runtimes/memory usage pre/post fill ?
With regard to your question about multithreaded oasis reading:
Klayout by default writes oasis with cellname offset property.
This enables a multithreaded reader to read cells in parallel.
The read speed increase will be limited by file io and memory bandwidth, so it depends on the type of machine, cpu count and file storage.
It takes considerable programming effort to implement a multithreaded reader, volunteers are always welcome.
16 cores intel cpu
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
stepping : 4
microcode : 0x2000050
cpu MHz : 1000.023
cache size : 19712 KB
256G memory
file located in NFS disk
Thank you for your suggestion. I will look into the OAS load code, but my programming skills are quite basic. I am not sure if I can manage it.
@dion,
from the statistics I see an obvious overhead of shape info, taking the majority of the memory (~15G). Also, the process size is much larger (40G) than the actual memory required (20G) which pioints to fragmentation. I assume that the dummy fill is very dense and consists of many different shape types.
Is that post-OPC data maybe?
In that case, OASIS compression is not very effective and memory management strategies that rely on that, will fail. It is also likely that there is a unknown bottleneck in the reader code that hits in that case.
I assume you cannot share data, not even a clip, right?
Matthias
This is the final data before the mask-making process, and I think it might contain several OPC layers. I work at Fudan University, and OPC-related tasks are handled by another team, so I am not very familiar with it. I apologize for not being able to provide the related layout OAS. KLayout is a great tool and has been very helpful for our research work. Once again, we express our gratitude.