Forum Discussion
hoolio
Cirrostratus
CirrostratusBest way of ensuring floating point precision?
Hi there,
I need to ensure I get floating point precision for division where the values I'm dividing could both be integers. The problem I'm seeing is if the two numbers are integers, the result isn't as accurate as I need it. I need at least precision to three decimal places. There are plenty of references to this in the expr man page and on DC (Click here). No one seems to have solid recommendations for forcing floating precision though.
There seem to be a few ways to do this. I'm wondering if anyone has experience in which would be the best. Here are a few options I've tested and the results:
Rule : This doesn't give floating point precision:
Rule : [expr {5 / 4}]: 1. Time: 17
Rule :
Rule : This gives the correct result and seems like the most thorough option (also the slowest):
Rule : [expr {[format %f 5] / [format %f 4]}]: 1.25. Time: 62
Rule :
Rule : This is tied for the fastest and seems like a valid approach:
Rule : [expr {[format %f 5] / 4}]: 1.25. Time: 42
Rule :
Rule : This is tied for the fastest but is a bit of a hack (by appending a . to one value before dividing):
Rule : [expr {5. / 4}]: 1.25. Time: 42
when RULE_INIT {
set iterations 10000
set a 5
set b 4
log local0. "==================================================================================== "
log local0. "[clock clicks -milliseconds]"
set start [clock clicks -milliseconds]
for {set i 0}{$i < $iterations}{incr i}{
set test [expr {$a / $b}]
}
log local0. "This doesn't give floating point precision:"
log local0. "\[expr {$a / $b}\]: [expr {$a / $b}]. Time: [expr {[clock clicks -milliseconds] - $start}]"
log local0. ""
set start [clock clicks -milliseconds]
for {set i 0}{$i < $iterations}{incr i}{
set test [expr {[format %f $a] / [format %f $b]}]
}
log local0. "This gives the correct result and seems like the most thorough option (also the slowest):"
log local0. "\[expr {\[format %f $a\] / \[format %f $b]}\]: [expr {[format %f $a] / [format %f $b]}].\
Time: [expr {[clock clicks -milliseconds] - $start}]"
log local0. ""
set start [clock clicks -milliseconds]
for {set i 0}{$i < $iterations}{incr i}{
set test [expr {[format %f $a] / $b}]
}
log local0. "This is tied for the fastest and seems like a valid approach:"
log local0. "\[expr {\[format %f $a\] / $b}\]: [expr {[format %f $a] / $b}]. Time: [expr {[clock clicks -milliseconds] - $start}]"
log local0. ""
set start [clock clicks -milliseconds]
append a .
for {set i 0}{$i < $iterations}{incr i}{
set test [expr {[format %f $a] / $b}]
}
log local0. "This is tied for the fastest but is a bit of a hack (by appending a . to one value before dividing):"
log local0. "\[expr {$a / $b}\]: [expr {$a / $b}]. Time: [expr {[clock clicks -milliseconds] - $start}]"
log local0. "[clock clicks -milliseconds]"
log local0. "===================================================================================="
}
Thanks,
Aaron
I need to ensure I get floating point precision for division where the values I'm dividing could both be integers. The problem I'm seeing is if the two numbers are integers, the result isn't as accurate as I need it. I need at least precision to three decimal places.
Ignoring the whole "wide integer" case for the moment, if Tcl sees both sides of the "/" operator as integers, it will do integer division. If either operand is a double, then it will (in a temporary place) promote the other operand to a double if it isn't already and perform division in the usual way. So, to get what you want, you just need to force one side to be a double...Here are a few options I've tested and the results:
And as you've found, there are several ways to do this, with varying speeds. Now, there is an important bug in your program that you posted. The last case you display ("5. / 4") is not the last case you test; you're testing "[format %f 5.] / 4" in the loop. You presumably copy/pasted and only changed the calculation once instead of twice. If you change all your log messages to log the actual value of $test, then you only have the actual calculation in one place (inside the for loop), which would avoid that sort of error. I've corrected this and run your program on my (low-end) box; results below.There are plenty of references to this in the expr man page and on DC (Click here). No one seems to have solid recommendations for forcing floating precision though.
That page actually does have the Tcl way to do what you want; the double() command. I've added it to your test program, also. Here are my results:
====================================================================================
568329753
Integer division:
[expr {5 / 4}]: 1. Time: 18
Forcing both operands via format:
[expr {[format %f 5] / [format %f 4]}]: 1.25. Time: 182
Forcing one operand via format:
[expr {[format %f 5] / 4}]: 1.25. Time: 119
Forcing one operand via ".":
[expr {5. / 4}]: 1.25. Time: 19
Forcing one operand the official way, using double():
[expr {double(5) / 4}]: 1.25. Time: 18
568330109
====================================================================================
Forcing the operand via "." is not a hack, by the way; the dot tells Tcl that it's not an integer, and so it next tries to parse it as a double, and succeeds. That said, the best way is probably using double(), as it's just as fast as anything else and is the most explicit so as to let future maintainers (like maybe you, six months from now) know exactly what you're doing.
hoolioHi Spark,
Thanks for the tips. I thought I was testing a double using [format %d 1], but that was a bad assumption as %d converts the input to a signed integer.
expr {[format %d 1] / 3}
0
Thanks again,
Aaron
