It looks like you're new here. If you want to get involved, click one of these buttons!
Hello,
I have created a script to merge 2 or more gds/oasis files with support of few options (rename all cells, rename only cells with naming conflicts, etc.). The script is based on the function copy_tree of klayout API.
It works fine for small gds files. But, it takes a lot of time for bigger GDS/OASIS files. For example, it takes 24minutes for 2 oasis file merging, while it takes 2/3 minutes with a commercial tool.
Is there any alternative to this function ?
Thanks .
Regards,
Comments
@ahmedo Sorry, but I can't provide support if you don't provide details.
Here are some details you should provide:
I think that just complaining this way isn't a fair benchmark.
I have a benchmark of similar functionality against a commercial tool which does not show a significant performance difference. And "copy_tree" isn't slow in particular.
Matthias
Hello Matthias,
Thank you for your answer.
The script takes some time in reading the oasis file (~1-2 minutes). The merge takes a lot of time (~20min) and the writing of the oasis takes almost 1 minute.
I didn't monitor the memory usage, but I am sure that I have enough memory on my server which is based on Linux RHEL7. I'm using Klayout 0.26.5.
Here is the code of the script mergeLayoutFiles.py :
Sorry, I have omitted a part of the program as I have exceeded the limit of the caracters I could write in a single post.
Example of usage:
time python ./mergeLayoutFiles.py -i input.gds -i input2.gds -m rename -t MERGE_MASK --force -o output.gds -tr "0,0;100,100" -v
As I said before, it works fine on small gds files but takes a lot of time on a relatively big oasis file (19Mb). I have made tests and found that the copy_tree used in the script is the cause of the run time issue.
The commercial tool that has been used for comparison is K2_Viewer for Cadence.
Thanks.
Regards
@ahmedo Thanks for providing this information
I think there is a lot of optimization potential in your script. 19MB isn't really a big file - I deal with GB-size OASIS files on a daily basis. But OASIS can have a hidden complexity: small files may hide a huge number of shapes. But KLayout can deal with this.
First of all, please switch to the latest version. There is a no particular reason for doing so here, but it saves me the effort of looking for differences.
There are too many loops over cells to my taste. For example this:
can be written much shorter and efficiently like this:
Second, you do not need to rename the cells. "copy_tree" takes care of creating unique cell names.
"copy_tree" will also take care of DBU translation which is another plus.
Third, OASIS works much better with "non-editable" layouts. Such layouts keep OASIS shape arrays without exploding them. This disables certain (few) operations associated with editing. Hence the name. But this does not mean the layout is immutable. Non-editable layouts can be created passing "False" to the constructor:
I some KLayout versions, the default is taken from the user settings, so explicitly asking for non-editable makes sense.
I have no performance experience with the Python packages. I cannot say if their performance is equivalent to the application. I usually use the "klayout" binary in batch mode:
It offers less options to access the command line, so I'll usually put a wrapper script around that and use "-rd var_name=value' to pass arguments to the script.
I have prepared a benchmark script which is the core essence of a merge script for two files:
For two files being 12MB each my numbers are the following when I run this script in batch mode ("klayout -b -r script.py"):
They are for 0.26.9 on a i7 with 3200MHz and reading the files from a local SSD. Not really bad numbers, I'd say. Over all it's 14 seconds.
You can improve these numbers by applying a trick: layouts can be put into "frozen" mode which does not mean they are frozen, but some internal properties are not updated automatically and wasting resources for information you don't need. This includes bounding boxes and hierarchy hints (such as child cells). As copy_tree does not need them you can use "start_changes" before the first copy and "end_changes" after the last one to disable this. This gains you a few seconds:
with this change, the times are these (again, two OASIS files with 12MB):
which sums to roughly 11 seconds.
I tried another test case which is a 100MB OASIS file and merged it two times into one file giving these numbers:
Which is an overall 40 seconds and shows that the times scale pretty well with OASIS size.
Matthias
Hello Matthias,
Thank you so much for your response and for the code examples that you have shared in this discussion. I know that it should take some of your time to help us resolve our layout related issues .
I have made tests with the latest release of Klayout (0.26.10) with the script I have shared with you and I've got the same runtime as with release 0.26.5.
But, when I have made the changes that you've suggested, the run time is now quite impressive. The run time changed from 23min to 2s and it's due to the command:
ly = pya.Layout(False)
With option 'False", copy_tree seems very fast. I have also added the freeze commands but I didn't change a lot for the run time, because I think finally that my oasis file is small (19MB) unlike what I have said in my initial discussion concerning this oasis file.
In addition, I've changed the script to use your simplified code for adding cells to the list all_cells. So now, I think that my script is now quite good for daily usage.
Concerning you question about using a lot of code for the merge while it could be done simply using copy_tree as in your examples. I have developed this script to give the user the ability to either rename all the cells or just the conflicting cells in the merged database. The user could sometimes want to add a specific suffix for the names of conflicting cells while copy_tree generates names automatically (without any control on that).
Thank you again for your help.
Ahmed
Very good!
Non-editable isn't a feature of copy_tree actually, but this means shape arrays are left compressed. This no only simplifies the work for copy_tree but also leads to a predicable and reduced memory footprint and will also make the OASIS writer's job easier. It does not need to establish shape arrays again and will rather reuse the ones it got on reading.
Matthias