Number Representation
Computers store numbers in a fixed number of bits, so they cannot represent every value exactly. Integers have a limited range, and decimal fractions are usually approximated.
Integers have limits
A signed 32-bit integer holds values from -2,147,483,648 to 2,147,483,647. Go past the maximum and, depending on the language, the value wraps around to a negative number, or an error is raised, or (in Python) nothing happens, because Python integers grow as needed.
- Use a wider type (64-bit) for counters, sizes and ids that may grow.
- Take care when converting between types of different sizes, or between signed and unsigned.
Floating point is approximate
Most languages store non-integer numbers as binary floating point (IEEE 754). Many decimal fractions, such as 0.1, have no exact binary form.
# Python
0.1 + 0.2 # 0.30000000000000004
0.1 + 0.2 == 0.3 # False
// JavaScript: every number is a double
Number.MAX_SAFE_INTEGER; // 9007199254740991, above this integers lose exactness
Consequences:
- Do not compare floats with
==. Compare with a small tolerance. - Do not use floats for money. Use integers of the smallest unit (cents) or a decimal type (
Decimalin Python,decimalin C#). - Large ids lose precision in JavaScript. Send them as strings in JSON. See Data Serialization.
NaNis not equal to itself, and JSON cannot represent it.
Rounding
- Integer division differs between languages:
7 / 2is3.5in Python 3 and JavaScript, but3in C, Java and C#. - Halves round differently: Python’s
round(2.5)gives 2 (to the nearest even), JavaScript’sMath.round(2.5)gives 3. Check what your language does and what the business rule requires. - Round at the end of a calculation, not at every step.
Try it
In a Python or JavaScript console, evaluate 0.1 + 0.2, then add 19.99 one hundred times and compare with 1999. Redo the sum in whole cents. In JavaScript, also evaluate 9007199254740993 and see what comes back.
Common mistakes
- Comparing floats with == instead of using a tolerance.
- Storing money as floating point.
- Sending large ids as JSON numbers to JavaScript, where they lose precision.
- Rounding at every step instead of at the end.
Check yourself
- Why does 0.1 + 0.2 == 0.3 give False in Python?
- How should you represent prices in code?
- What is Number.MAX_SAFE_INTEGER and why does it matter for ids?