›

how to partially write out oas?

diondion
edited May 2018 in File Formats
Hi,
I have to build a large oas file from lots of small files.

For an example, read in geometry from one small oas, then do some polygon operations,
then save data as new cell in final oas.

Now, I have to read all small files to memory and then write them out.
It takes about 70~80G memory. (There are about 2000 small oas)

Could I partially write out cell data after one small oas processed?
Then writing out cell reference information in the end.
It could reduce memory usage.

Thanks,
Dion

Comments

  • MatthiasMatthias
    edited November -1

    Hi Dion,

    this may be possible in some flavours of OASIS, but not in all of them: some OASIS flavours require tables to be built upfront, so you need to know the full file content before you can write it. One can overcome this by a two-pass approach and modify the file's header afterwards.

    Nevertheless, KLayout OASIS write is single-pass and not optimized for this use case. In order to get the implementation you look for, you'd have to modify the OASIS writer code in a very specific way.

    But maybe there are some other ways to optimize memory:

    • If you script the generation code, make sure the big objects (specifically Layout) are deleted
      as soon as they are no longer required (use object._destroy
    • Construct Layout objects as "non-editable" (use "Layout(false)"). Some operations are not supported
      in this mode, but most are. It's worth a try.
    • Do the operation first on all files while writing temporary files (2000 read + 2000 OASIS writes). Then
      read again and combine the layouts in non-editable mode. Advantage: the OASIS write will compress
      the shapes and when using non-editable mode, the compression is retained.

    Kind regards,

    Matthias

  • diondion
    edited November -1
    Thanks for your suggestion :)
    I will try it.

    Thanks,
    Dion
Sign In or Register to comment.
Powered by Vanilla