Multi-step OASIS stream file writer

dion

When a design is too large to be completely created in memory, gdstk provide the method of Multi-step stream writer, like,

writer = gdstk.GdsWriter()
cell1 = some_function_that_creates_a_huge_cell()
writer.write(cell1)
del cell1
cell2 = some_function_that_creates_a_huge_cell()
writer.write(cell2)
del cell2
writer.close()

How to do it in klayout ?

Matthias

Do you have an application that really needs that?

Basically, such a thing is scary. Stream writers, in particular OASIS, benefit a lot from having all the knowledge about the layout and hierarchy at hand. For example, if cell2 is a child cell of cell1 - how can you compute the bounding box of cell1 then?

If you really need to write huge layouts and come across a memory barrier, you can try writing each cell into separate OASIS. This benefits from the OASIS compression - specifically shape array formation.
In another step you combine these individual layouts. If you load them into a non-editable layout, it will keep the OASIS compression, hence require a lot less memory.

But I have been creating huge cells already myself without seeing the need to do the multi-step thing.

Here is an experiment:

import klayout.db as kdb

ly = kdb.Layout()

l1 = ly.layer(1, 0)
c1 = ly.create_cell("TOP")
shapes = c1.shapes(l1)

for i in range(0, 10000):
  for j in range (0, 10000):
    shapes.insert(kdb.Box(i * 10, j * 10, i * 10 + 9, j * 10 + 9))

ly.write("test.oas")

This script creates 100M of boxes. It runs about 5 minutes (C++: much, much faster) and consumes 1.6G memory (2.2G virtual) on Ubuntu 22. That is consistent with 16 bytes per box. During OASIS write, memory peaks at 3.2G because in that extreme single-cell and single-layer case, the OASIS writer produces a flat map for repetition extraction. Finally, the OASIS file is only 400 Bytes.

Matthias