Forum Discussion
hoolio
Cirrostratus
Oct 21, 2008Best way of ensuring floating point precision?
Hi there,
I need to ensure I get floating point precision for division where the values I'm dividing could both be integers. The problem I'm seeing is if the two numbers are integers...
hoolio
Cirrostratus
Oct 22, 2008Hi Spark,
Thanks for the tips. I thought I was testing a double using [format %d 1], but that was a bad assumption as %d converts the input to a signed integer.
expr {[format %d 1] / 3}
0
Thanks again,
Aaron
Recent Discussions
Related Content
DevCentral Quicklinks
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
Discover DevCentral Connects