›

layout unit issue about float number

frostchufrostchu
edited June 2021 in Python scripting

Hi admins:
I' m using klayout python independent module, and get an issue about layout unit.
for example, I used script below to get the layout unit:

 from klayout import db
 gdsPath = 'xxxx.gds'
 layout = db.Layout()
 layout.read(gdsPath)
 dbu = layout.dbu
 print (dbu)

#
and I have 2 sample layout files: A.gds & B.gds,
for A.gds, the output is 0.001 which is correct, but for B.gds, the output is 0.000999999999...
I know the unit is stored in 8-byte format float numbers, so I use 'exdump -C' command to open both layout files and get unit records.
A.gds unit record: 00 14 03 05 3e 41 89 37 4b c6 a7 f0 39 44 b8 2f a0 9b 5a 54
B.gds unit record: 00 14 03 05 3e 41 89 37 4b c6 a7 f0 39 44 b8 2f a0 9b 5a 50

I wonder how does klayout process those float numbers.
In addtion, B.gds was generated by synopsys Laker, I don't know why synopsys has a different standard from other EDA tools.

thanks & best regards
frostchu

Comments

  • MatthiasMatthias
    edited June 2021

    @frostchu The GDSII format employs a 64 bit, base-16 floating point representation (see here: https://en.wikipedia.org/wiki/IBM_hexadecimal_floating-point). Neither the IEEE 754 format used in today's CPUs nor the IBM format can exactly represent a value like 0.001 - so some rounding is always involved.

    But I doubt that is really a problem. In your case, the difference happens at the 49th bit, so the actual error is small enough to accumulate to one nanometer over a distance of 563 meters (if I did the math correctly) :)

    Matthias

Sign In or Register to comment.
Powered by Vanilla