self.layout.dbu = 0.0010000000000000002 instead of 0.001

carlsv4

Hi!

We're encountering an issue with the dbu value when working with some PCells. The expected dbu value is 0.001, but sometimes it shows up as 0.0010000000000000002, which then causes incorrect conversions when calculating dimensions in dbu.

Instead of the wanted result:

JJ
Dbu:  0.001
Width:  1.0
Length:  10.0
Overlap:  2.0
Width in dbu:  1000.0
Length in dbu:  10000.0
Overlap in dbu:  2000.0

We get this:

JJ
Dbu:  0.0010000000000000002
Width:  1.0
Length:  10.0
Overlap:  2.0
Width in dbu:  999.9999999999998
Length in dbu:  9999.999999999998
Overlap in dbu:  1999.9999999999995

It looks like the floating-point representation of 0.001 is slightly off in some cases, causing precision issues.

Has anyone else run into this? Could this be a typical Python float issue, and if so, how do you usually handle it?

tomas2004

Hello,

It has something to do with the internal behavior of float calculations (don't ask me why, I'm not an expert ;-). Just add ".round(8)" (Ruby) and it will round to the right value (8 decimals, much smaller than any DBU value)...

Cheers,

Tomas

Matthias

You're encounting a very basic issue of floating point representation of decimals. There are tons of articles explaining why. For example this one: https://stackoverflow.com/questions/588004/is-floating-point-math-broken

There is no fix. You need to be aware of rounding errors and handle them properly.

Matthias

carlsv4

Yes thank you Tomas and Matthias!