Dump inaccurate coordinate with ruby 1.9.3

edited September 2012 in General

Hi Matthias,

We are trying to upgrade ruby version from 1.8.7 to 1.9.3, and encounter a strange problem when running dump_texts_file.rbm .

With ruby 1.8.7, the coordinate could be output precisely, however in 1.9.3, it would output a special number.
For example:

1.8.7: TEXT1 (89.0,231.2)

1.9.3: TEXT1 (88.99999999999999,231.19999999999996)

Is it our ruby 1.9.3 issue ?

The partial code of dump_texts_file.rbm

  lv.each_object_selected do |sel|
    if !sel.is_cell_inst? && sel.shape.is_text?
      text = sel.shape.text
      p = sel.trans.trans(text.trans.disp)*lv.cellview(sel.cv_index).layout.dbu
      file.puts("#{text.string} #{p.x} #{p.y}")
    end
  end

Best Regards,

chhung

Comments

  • edited September 2012

    Hi chhung,

    apparently Ruby 1.9.3 uses a different precision when converting float's to string (or the underlying C library does it differently).

    Anyway, you can fix the problem using a specific format and the % operator:

    file.puts("#{text.string} #{'%.12g'%p.x} #{'%.12g'%p.y}")
    

    this example outputs the numbers with a maximum precision of 12 digits which is the number of digits I am using usually. It not too much so that rounding errors are suppressed and it's not too little, so we don't loose precision.

    Best regards,

    Matthias

  • edited November -1

    Hi Matthias,

    Thanks for the new code, it solved our problem and works fine. :)

    Thank you~

    Best Regards,

    chhung

Sign In or Register to comment.