Number Representation

Expected level: Awareness

Computers store numbers in a fixed number of bits, so they cannot represent every value exactly. Integers have a limited range, and decimal fractions are usually approximated.

Integers have limits

A signed 32-bit integer holds values from -2,147,483,648 to 2,147,483,647. Go past the maximum and, depending on the language, the value wraps around to a negative number, or an error is raised, or (in Python) nothing happens, because Python integers grow as needed.

  • Use a wider type (64-bit) for counters, sizes and ids that may grow.
  • Take care when converting between types of different sizes, or between signed and unsigned.

Floating point is approximate

Most languages store non-integer numbers as binary floating point (IEEE 754). Many decimal fractions, such as 0.1, have no exact binary form.

# Python
0.1 + 0.2            # 0.30000000000000004
0.1 + 0.2 == 0.3     # False
// JavaScript: every number is a double
Number.MAX_SAFE_INTEGER;   // 9007199254740991, above this integers lose exactness

Consequences:

  • Do not compare floats with ==. Compare with a small tolerance.
  • Do not use floats for money. Use integers of the smallest unit (cents) or a decimal type (Decimal in Python, decimal in C#).
  • Large ids lose precision in JavaScript. Send them as strings in JSON. See Data Serialization.
  • NaN is not equal to itself, and JSON cannot represent it.

Rounding

  • Integer division differs between languages: 7 / 2 is 3.5 in Python 3 and JavaScript, but 3 in C, Java and C#.
  • Halves round differently: Python’s round(2.5) gives 2 (to the nearest even), JavaScript’s Math.round(2.5) gives 3. Check what your language does and what the business rule requires.
  • Round at the end of a calculation, not at every step.

Try it

In a Python or JavaScript console, evaluate 0.1 + 0.2, then add 19.99 one hundred times and compare with 1999. Redo the sum in whole cents. In JavaScript, also evaluate 9007199254740993 and see what comes back.

Common mistakes

  • Comparing floats with == instead of using a tolerance.
  • Storing money as floating point.
  • Sending large ids as JSON numbers to JavaScript, where they lose precision.
  • Rounding at every step instead of at the end.

Check yourself

  • Why does 0.1 + 0.2 == 0.3 give False in Python?
  • How should you represent prices in code?
  • What is Number.MAX_SAFE_INTEGER and why does it matter for ids?

Related topics

Further reading