dbu changes when reading/writing

I noticed something about reading/writing files and dbu value changes. I don't think it makes a difference in the output, but it wasn't the behavior I expected. I'm sure this is related to floating point representation and rounding, but it doesn't seem to be exactly matching the issue dealt with in 2017: https://www.klayout.de/forum/comments.php?DiscussionID=994&page=1#Comment_4547

Basically, with script below, I get a dbu of 0.001 immediately after creating a new layout. However, if I read that file in, it reports 0.0009999999999999998.

I don't really have a question but people may want to be aware that this is the output of ly.dbu after reading in a file. (I was writing a script to check for errors, and this was one of the parameters I was checking)

import pya

ly = pya.Layout()
cell = ly.create_cell('top')

cell.shapes(ly.layer(1, 0)).insert(pya.Box(0, 0, 1000, 1000))

opt = pya.SaveLayoutOptions.new()
opt.format = 'OASIS'
ly.write('~/masks/test_new_python.oas', opt)

ly2 = pya.Layout()


  • I forgot to add: if you create a new layout using the GUI and then load it with python, it will report dbu=0.001 as expected. Subsequent writing and re-reading the file reports 0.0009999999999999998

  • edited June 15

    Hi @mjcich,

    Floating point representation of decimal fractions is never precise. So 0.001 is actually not exact already.

    Try this in Python:

    >>> print("%.25f" % 0.001)

    OASIS stores the value as "1/dbu" (called "resolution"). KLayout correctly stores a rounded integer value of "1000" here.

    On reading, the reader computes the DBU as "1 / 1000" and because the computation is done in meter units, it picks a different rounding - this time down instead of up. So it comes out as "0.0009999999999999998".

    I think I can basically drop the meter units, using micron instead. That would probably come closer to the Python value and maybe prints as 0.001 in Python. The important point is, however, that the relative error you introduce is 2e-16. So an error of 1nm at the distance of about 5000km. I do not think this is a problem.

    Furthermore, saving that layout again will restore the "1000" resolution in OASIS. So repeatedly reading and saving the file does not accumulate errors.

    And as a general rule, you should never trust floating-point values. If you compare them, allow for some slack, e.g.

    math.fabs(dbu_a - dbu_b) < 1e-15

    There are many tutorials explaining how and why.


  • Thanks! I knew about the floating point precision issue, I just didn't know the OASIS standard stored 1/dbu instead of dbu. It's funny because I have been using DBU=1000 in my scripts for convenience and decided I should correct it to 0.001 to be consistent.

Sign In or Register to comment.