Python Scripting for reading an Excel file and creating a polar array on Klayout Macro

Hi,
I used the following python script to read an excel file (attached) and generate a text script as input for CNST (Java based nanofab toolbox developed by NIST) toolbox. This CNST toolbox is supposed to generate GDS2 file. However, for large size files, CNST toolbox stalls.
My script first create a line array of circles where position and radii for individual circles are given in the Excel file. Then this script creates a polar array of this line array of circles; the final layout is a metalens of circular symmetry.
Can someone either modify this python script (copied below)or guide me how to do this on Klayout Macro?
Thanks a lot.
KS

import csv
import math

circles = []

Open csv file

with open('circles.csv') as csvfile:
circleReader = csv.reader(csvfile)
for num, row in enumerate(circleReader):
if num != 0:
circles.append((float(row[1]), float(row[2]), float(row[3])))

Open file for output

with open('metalens.cnst','w') as output:

# Header
output.write('# This file is created using metalens.py\n\n')
output.write('0.001 gdsReso\n')
output.write('0.001 shapeReso\n')
output.write('1 layer\n')


# create all circle structures
for num, circle in enumerate(circles, start=1):
    output.write(f'<circle_{num} struct>\n')
    output.write(f'0 0 {circle[1]/2} {circle[1]/2} 0 ellipseVector\n\n')

# create metalens
output.write('# Metalens structure comment\n')
output.write('<metalens struct>\n')

#create each ring of the metalens from the inside out
for num,circle in enumerate(circles, start=1):
    theta = 180*circle[2]/math.pi
    if num == 1:
        output.write(f'<circle_{num} 0 0 N 0 0 instance>\n')
    else:
        output.write(f'<circle_{num} 0 359 {theta} {circle[0]} {circle[0]} 1 1 arrayPolar>\n')

Comments

  • Hi Matthias, Any input on the above? Is this doable on Klayout Macro Development or modified python script can directly create the gds2 file? In either case, can you direct me as to how to modify the above code to achieve this? Thanks.
    KS

  • Hi,
    I'm a complete outsider of CNST but challenged your problem since I have solved a similar problem very recently.
    Does this look like what you want finally?

    Regards,
    Kazzz-S

  • Hi Kazzz-S,
    Thanks for response. Yes, this is how it should look like, but the issue is when the number of initial set of circles (whose positions and radii given in excel file) exceeds, lets say 1000 (or approx 500 micron diameter metalens), CNST crashes.

  • Kazzz-S, If you can share how you solved your problem, it will be a great help and i would appreciate it.
    Regards,
    KS

  • Hi,

    Please refer to the files in the attached ZIP file, where
    1. I have remade your original Excel file based on my guess and created a CSV file to fit my solution.
    2. Even if you have 1000 rows in your real-life problem, I hope this solution can work gracefully. KLayout is powerful:-)
    3. In my solution, I used "Pandas" to handle a CSV file.
    4. Also, I used the "PCell" of KLayout in my approach. Read the KLayout's manual regarding PCell.
    5. There are a few global control parameters you can try. One of them is to convert "PCells" to "Static (ordinary)" cells.

    I hope the main PYA script KazzzS-KL1642.py is self-explanatory.

    Regards,
    Kazzz-S

  • Kazzz-S,
    Heartening to see someone taking time to help (unknown) another. Thanks so much. let me take a look and get back with you. By the way, i have over 13,300 points in the excel file with position, radii values, as well as angular incremental values for each circle having upto 7 decimal places, but i agree Klayout should be able to handle this.

    KS

  • edited September 20

    Hi,

    I'm benefitting from KLaout a lot! My activity is a small token of thanks to it.

    I have found some typos and fixed them. Sorry about that. Please update with the second ZIP file.

    By the way, I got the reason why there are many digits below the decimal points in the sample data.
    Perhaps, you need to choose a more refined database unit other than 0.001 [um], which is the default.
    In my applications, I have never changed this value.

    Regards,
    Kazzz-S

  • Very nice comment about how you give back after benefiting a lot from Klayout.
    Yes, it makes sense to change the data values to make the smallest value correspond to Klayout resolution of 0.001 um. Will do.
    Also, i am new to Kalyout Macro, so it will take some time for me to report to you what happened. Will get back with you.
    Thanks again.

  • Hi KAZZZ-S,
    If you dont mind, help me with some stupid questions:
    1. I keep getting an error message "no module named pandas" as i run your script although i have installed pandas module on my computer
    2. When you say, "Load (drag & drop) "KL1642.lyp" layer property file" , you mean drag and dropping on the new layout file, right?
    3. The following done on file explorer on Windows, right? Basically leaving the .csv file and the output .gds file in the same folder as the .py file?
    MyRoot = os.environ["HOME"] + "/KLayout/Forum1642/"
    MyCSV = MyRoot + "KazzzS-circles.csv"
    OutGDS2 = MyRoot + "KL1642.gds"

    Thanks.

  • Hi dsenuka,

    what's the platform you're on? If it's Windows 64bit, you can manually install pandas using this recipe: https://www.klayout.de/forum/discussion/1645/pandas-library-for-windows#latest

    Matthias

  • edited September 24

    Hi dsenuka,

    1. Pandas issue: please follow Matthias' instruction. Thanks, Matthias :)
    2. LYP file: a layer property file is for customizing the layer color, stipple, etc. So, it does not work with an empty design.
      • In the ZIP file, you find "KL1642.gds" file for demonstration purposes. First, open it. The color and stipple settings are KLayout's default.
      • Next, File===>Load Layer Properties OR Drag&Drop "KL1642.lyp" then you can see the above image.
      • Once the PYA script run, you can do the same operation for changing the layer properties.
    3. File location: I've tested the script on Mac. So,
      • os.environ["HOME"] ==> "/Users/kazzzs"
      • then, the full path of MyRoot ==> "/Users/kazzzs/KLayout/Forun1642/", which is the directory where I stored the related files.
      • if you are using Windows and placed files under C:\KLayout\Forum1642\ you can change it to MyRoot="C:/KLayout/Forum1642/"
      • Similarly, you can specify any existing directory for GDS2 output. In the sample, I chose the same directory as the input CSV.

    Kazzz-S

  • THANK YOU, Matthias and Kazzz-S ! Let me try these.

  • Hi, i keep getting this message although i have the files in the correct folder.
    [Errno 2] File b'C:/Users/rabeysin/KLayout/Forum1642/KazzzS-circles.csv' does not exist: b'C:/Users/rabeysin/KLayout/Forum1642/KazzzS-circles.csv'

    Attached screenshot image file shows how i changed the global parameters in your script.

    sorry for keep bothering you.

  • Tried several variations of folder path, like double back slash etc. still without any luck.

  • edited September 24

    It looks you are in a small pitfall.

    MyRoot = C:\\Users\\rabeysin\\KLayout\\Forum 1642     # Wrong! It's not a string object! Syntax error.
                                                          # See the color of "KL1642.gds" in the text editor.
    
    MyRoot = "C:\\Users\\rabeysin\\KLayout\\Forum 1642"   # Nearly but still wrong! It should end with "\\"
                                                          # for succeeding concatenation with the file names.
    
    MyRoot = "C:\\Users\\rabeysin\\KLayout\\Forum 1642\\" # OK
    
    MyRoot = "C:/Users/rabeysin/KLayout/Forum 1642/"      # OK (Unix style)
    
    

    Kazzz-S

  • Thanks so much Kazzz-S !! Is Klayout Macro documentation the only thing to study for someone with an intermediate-level python experience to be conversant with Klayout Macro in Python? pl. let me know if there's anything else.if not, just ignore this message. Thanks again.

  • Still no luck, keep getting the same message.

  • i wonder how Repository Browser on the left side of Macro IDE look like (in the image i sent out earlier), anything to do there?

  • edited September 24

    Your left side pane of the IDE looks normal. It's more or less the same as mine. Notice that the path is in Unix style in the 2nd & 3rd lines.

    When I tested some "MyRoot" expressions mentioned above, I could see a space between Forum and 1642 in the image.
    That's why the above sample.
    However, the error message you got does not contain the space if you copied and pasted it.
    It seems, in your existing file system, there IS a space between Forum and 1642 but your script does not have it,
    which causes the mismatch. Am I correct?

    BTW, it would be better to choose a fixed-width font for the IDE text window. Then, the presence of a space character is easily visible.

    Kazzz-S

  • Dear Kazzz-S, You are right, as always. Sorry for not noticing such a thing and bothering you. I finally got your script running and got the dgs file produced. Now i am going to use my csv file (attached herewith) in your script, but it will be monumental it looks like. My file contains the position and radii info for an initial set of circles and their polar array parameters. I will give it a try.
    When you get a chance, pl. let me know if there's any other materials/tutorials other than those documentation in Klayout site if one wants to master Macro development in python.
    Thanks again.
    dsenuka

  • edited September 27

    Hi dsenuka,

    Even with the given real-data Excel sheet, I do not fully understand your real-life problem.
    However, if my assumptions are correct, you are trying to layout about 1.4 million circles in the quadrant-I.
    Then, my main interest has moved to the performance aspect of KLayout when handling such a big number, which I never come across in my daily work.
    I have modified the previous PYA (no significant change was required) and created a new CSV file with 1335 rows.
    Below is the summary output of the modified PYA:

     :
    ### <KazzzS-RealKL1642.py> processed <RealKL1642.csv> ###
        KLayout version: KLayout 0.26.8
        Machine info: Darwin, MacBookPro2.local, x86_64
        Using PCell?: Yes
        Total number of instances inserted: 1,399,419
        Timing info: Sys=2.420[sec]  User=27.920[sec]  Wall=30.414[sec]
    

    The output GDS2 file size is about 43MB.

    As you can see, your real-life problem is an easy win for KLayout!
    KLayout won't crash when handling such a large number of instances.
    So I believe it's worth learning PYA for you. Try it by yourself!

    Some images follow.
    Near the origin:

    Near 12 O'clock:

    Near 2 O'clock:

    Near 3 O'clock:

    Regarding additional materials to study PYA, I think,
    1. KLayout document set (either on-line web or built-in assistant) comes first
    2. Then, this forum
    Recently I have commented to this topic, which is related to your problem.

    Kazzz-S

  • Hey Kazzz-S,
    This is great !!! From what i see in the images, you seemed to have gotten the real life problem right. Basically, the columns B n C gives you the center coordinates (x and y) for an initial set of circles lying almost on the positive side of X axis. Column D gives the diameter for each one of these circles.
    Then, for each circle in this initial array of circles, i want to make an array of circles in a ring; this is where we need parameters for polar arraying, that is,
    1. angular increment (column E) and
    2. the radius of the ring (column B which is the same as X coordinate)
    We can disregard column F (no of pillars) if we do polar arraying using 1 and 2 above.

    I think this is how you have understood this problem, right? I know the file size coming out to be very large, how about we set the no of vertices for circles drop by half (no of vertices can ideally be 32 or even 16 should be ok).
    Hope you will share the modified script at some point. Many many thanks !!

  • edited September 28

    Hi dsenuka,

    I've attached the related files.
    You can make experiments changing different parameters.
    Good luck.

    Kazzz-S

  • Hi Kazzz-S, Great ! Thank you so much again !
    dsenuka

  • Kazzz-S, Sorry to bother you again. I keep getting the previous GDS layout as output when i run KazzzS-RealKL1642 with the relavent files in a separate folder from the previous one.
    dsenuka

  • Hi dsenuka,

    It's wired!
    For a quick testing, on my Linux, I have created a new directory $HOME/KLayout/Forum1642x/, which holds only RealKL1642.csv to start with.
    After the new script KazzzS-RealKL1642.py ran, I got RealKL1642.gds in the same directory.

    Line-27 to -35 are as below in the testing.

    Debug          = True   # set False not to print debug messages
    NumVertices    = None   # set an integer like 32 or 16 to fix the number of vertices
                            # of a polygon that approximates a circle
    Convert2Static = False  # set True to convert PCells to static cells
    MyDBU          = 0.001  # set an appropriate database unit in [um]
    NumDigits      = 8      # set an appropriate number of digits below the decimal point
    MyRoot         = os.environ["HOME"] + "/KLayout/Forum1642x/"
    MyCSV          = MyRoot + "RealKL1642.csv"
    OutGDS2        = MyRoot + "RealKL1642.gds"
    

    Check your code carefully.
    Kazzz-S

Sign In or Register to comment.