Hi,
I have to build a large oas file from lots of small files.
For an example, read in geometry from one small oas, then do some polygon operations,
then save data as new cell in final oas.
Now, I have to read all small files to memory and then write them out.
It takes about 70~80G memory. (There are about 2000 small oas)
Could I partially write out cell data after one small oas processed?
Then writing out cell reference information in the end.
It could reduce memory usage.
Thanks,
Dion
Comments
Hi Dion,
this may be possible in some flavours of OASIS, but not in all of them: some OASIS flavours require tables to be built upfront, so you need to know the full file content before you can write it. One can overcome this by a two-pass approach and modify the file's header afterwards.
Nevertheless, KLayout OASIS write is single-pass and not optimized for this use case. In order to get the implementation you look for, you'd have to modify the OASIS writer code in a very specific way.
But maybe there are some other ways to optimize memory:
as soon as they are no longer required (use object._destroy
in this mode, but most are. It's worth a try.
read again and combine the layouts in non-editable mode. Advantage: the OASIS write will compress
the shapes and when using non-editable mode, the compression is retained.
Kind regards,
Matthias
I will try it.
Thanks,
Dion
Hi Matthias,
I tried your suggestion,
Do the operation first on all files while writing temporary files (2000 read + 2000 OASIS writes). Then read again and combine the layouts in non-editable mode.
For step 2, I open layout in editable mode, then open 2000 oasis one by one in non-editable mode, then using 'copy_shapes' in cell class to combine final output.
Is it correct? It seems not reduce memory usage.
Thanks,
Dion