Run out of memory while merging multiple oasis files

Ege_Bey

I have multiple oasis files that I want to combine and make a single oasis file that contains the layout for the whole reticle.
However when I try to do this, I run out of memory. (I run out of memory even if I try to do it on Azure Cloud HPC)
The total size of the oasis file reach around 30 GB.

The code I use to merge different oasis files is given below.

layout_paths_to_import = ['layout.oas','layout2.oas']
top_cell_names_to_import = ['top1','top2']
output_layout_path = 'merged_layout.oas'
output_top_cell_name = 'output_top_cell'

def make_cell_instance(in_cell, angle=0, mirror=0, x=0, y=0, scale=1):
    trans=pya.CplxTrans(scale, angle, mirror, x, y)
    tmp_inst = pya.CellInstArray(in_cell.cell_index(), trans)
    return tmp_inst

def insert_cell_to_layout(layout,layout_path_to_import,top_cell_name_to_import,rotation=0,mirror=0,offset_x=0,offset_y=0,scale=1):
    top_cell = layout.top_cell()
    layout.read(layout_path_to_import)
    tmp_cell = layout.cell(top_cell_name_to_import)
    tmp_cellinst = make_cell_instance(tmp_cell, rotation, mirror, offset_x, offset_y, scale)
    top_cell.insert(tmp_cellinst)
    return layout

 output_layout = pya.Layout()
 output_layout.create_cell(output_top_cell_name)
 for layout_path_to_import, top_cell_name in zip(layout_paths_to_import, top_cell_names_to_import):
     output_layout = insert_cell_to_layout(output_layout, layout_path_to_import, top_cell_name)
 output_layout.write(output_layout_path)

Is there a more memory efficient way of doing this ?

Matthias

Please try to use non-editable layouts:

output_layout = pya.Layout(False)

Non-editable layouts store shape arrays more efficiently than editable ones. On the other hand, non-editable Layout don't allow certain shape manipulations because they don't break the shape arrays. Still they can be modified. So unless you manipulate shapes in place, non-editable layouts should do the job as well, but with less memory.

If that still fails, I'm afraid there isn't much room for improvement here. The way you do the merging is already efficient. Just beware that there are some pitfalls, like missing conversion of database units if one of the files has a different one.

Matthias

cyrustabery

Matthias, can we use file merge concept? Maybe it is possible to merge layouts in klayout without reading the layout at all. Similar to the concept mentioned in this youtube video :

Matthias

No sorry, I won't comment on this. You're not serious about showing me a commercial feature and asking whether to get this for free.

Your putting me at risk of getting in trouble with Mentor. Are you aware of this?

Matthias

cyrustabery

Matthias, the last thing i want is to get you in trouble --> open source layout and layout verification is a revolution and of huge value to me and the world. Thank you so much for the work you do. The ability to use streaming formats for merge is compelling and not a mentor feature, i think you could consider adding a streaming disk based merge to klayout. If you did i would use it. I linked to the mentor feature since that was an easy way to communicate; but the capability of doing a stream based merge is not owned by mentor and i hope doesnt get you in trouble.