Computer Arithmetic

Computers do arithmetic with finite representations. That constraint shapes every result. An integer type covers a bounded set of whole numbers. A floating-point type covers a finite, unevenly spaced set of real-number approximations.

You need this mental model whenever values can become large, small, fractional, or sensitive to rounding. It explains why an integer can wrap, why decimal 0.1 is usually approximate in binary floating point, and why two algebraically equivalent expressions can produce different machine results.

Start with representation

A numeral such as 1011 is a pattern. Its value depends on the radix and interpretation. In base two, each position has a power-of-two weight. Unsigned interpretation gives all bits nonnegative weights. Two's-complement interpretation gives the highest bit a negative weight and provides the usual signed range.

Width matters. An unsigned value with n bits ranges from zero through 2^n - 1. A two's-complement signed value with n bits ranges from -2^(n-1) through 2^(n-1) - 1. The same bits can therefore denote different values under different types.

Arithmetic must also define what happens outside that range. Some systems wrap modulo a power of two. Others trap, saturate, widen the result, or declare the operation invalid. Never infer overflow behavior from the bit pattern alone. Check the language, instruction set, or data-format contract.

Fixed point and floating point

Fixed-point arithmetic assigns an implied scale to an integer. If a stored value counts cents, 1234 represents 12.34 currency units. Addition is direct when scales match. Multiplication and division require deliberate rescaling and rounding.

Floating-point arithmetic stores a sign, a significand, and an exponent. The exponent moves the radix point, so one format covers very small and very large magnitudes. The tradeoff is variable spacing. Adjacent representable values grow farther apart as magnitude grows.

IEEE 754 defines binary and decimal floating-point formats, arithmetic, conversions, exceptions, and default handling. Its binary64 format uses a sign bit, an 11-bit exponent field, and a 52-bit fraction field. Normal values gain an implicit leading bit, giving 53 bits of precision.

Floating-point also includes signed zero, infinities, NaNs, and subnormal values. A NaN represents an invalid or indeterminate numerical result. A subnormal value extends the range near zero with reduced precision.

Exact value, rounded result

Most decimal fractions do not have finite binary expansions. The decimal fraction 0.1 therefore rounds to a nearby binary value in binary64. Printing 0.1 does not prove that the stored value equals one tenth.

Each basic floating-point operation conceptually produces an exact result and then rounds it to the destination format. Round-to-nearest with ties to even is a common IEEE 754 direction. Directed modes also round toward zero, positive infinity, or negative infinity.

Rounding breaks familiar real-number identities. Floating-point addition is not generally associative. Computing (a + b) + c can round at a different point from a + (b + c). Reordering a reduction can therefore change its result.

Range failures and numerical error

Integer overflow occurs when an exact integer result falls outside the destination range. Floating-point overflow occurs when a rounded finite result exceeds the format's finite range. Underflow concerns results near zero. IEEE subnormal values provide gradual underflow instead of forcing every tiny result directly to zero.

Rounding error is the difference introduced when an exact value is replaced by a representable value. Relative error measures that difference against the exact value's magnitude. A unit in the last place, or ulp, describes the spacing at a value's least significant represented position.

Cancellation occurs when you subtract nearly equal approximations. Leading digits disappear, exposing earlier input or rounding error. Cancellation does not create error by itself, but it can magnify error already present in the operands.

Choose arithmetic from the requirement

Use bounded integers for counts, indexes, masks, and modular arithmetic when you can prove range safety. Use a wider or arbitrary-precision integer when intermediate results may exceed the bounded range.

Use scaled integers or decimal arithmetic when decimal quantities and rounding rules are part of the contract. Finance is a common example. The scale, precision, rounding mode, and overflow policy must still be explicit.

Use binary floating point for measurements, simulation, graphics, and scientific work when bounded approximation is acceptable. Compare approximate results with a tolerance derived from the problem. A fixed universal epsilon is rarely meaningful across different magnitudes.

Treat arithmetic behavior as part of system design. Specify types at interfaces. Validate before narrowing conversions. Check exceptional results. Test boundaries, halfway rounding cases, cancellation, and different evaluation orders.

Limits of this course

This course gives you a practical map of machine arithmetic. It does not prove numerical algorithms stable or define one programming language's overflow rules. Those tasks require the relevant language specification, data model, and error analysis.