Computer Arithmetic
Computer arithmetic covers how processors represent and compute with numbers: integer and floating-point formats, overflow and precision limits, rounding behavior, and the hardware circuits that perform addition, multiplication, and division in binary.
itComputer fundamentals | OpenSkills.info
Intro
Computer Arithmetic
Computers do arithmetic with finite representations. That constraint shapes every result. An integer type covers a bounded set of whole numbers. A floating-point type covers a finite, unevenly spaced set of real-number approximations.
You need this mental model whenever values can become large, small, fractional, or sensitive to rounding. It explains why an integer can wrap, why decimal 0.1 is usually approximate in binary floating point, and why two algebraically equivalent expressions can produce different machine results.
Start with representation
A numeral such as 1011 is a pattern. Its value depends on the radix and interpretation. In base two, each position has a power-of-two weight. Unsigned interpretation gives all bits nonnegative weights. Two's-complement interpretation gives the highest bit a negative weight and provides the usual signed range.
Width matters. An unsigned value with n bits ranges from zero through 2^n - 1. A two's-complement signed value with n bits ranges from -2^(n-1) through 2^(n-1) - 1. The same bits can therefore denote different values under different types.
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Sources
- https://standards.ieee.org/ieee/754/6210/
Supports
- Binary and decimal floating-point formats and methods
- Operations, conversions, exception conditions, and default handling
- Dependence of results on inputs, operation order, and destination format
- Quiz answer about intermediate rounding and grouping
- https://docs.oracle.com/cd/E37069_01/html/E39019/z4000ac019127.html
Supports
- IEEE rounding directions
- Five IEEE floating-point exception classes
- Rounding precision and exception handling
- Boundary and halfway rounding tests
- https://docs.oracle.com/cd/E37069_01/html/E39019/z4000ac019269.html
Supports
- Binary64 sign, exponent, and fraction field widths
- Fifty-three bits of precision for normal values
- Normal values, subnormal values, signed zeros, infinities, and NaNs
- Quiz answer about binary64 precision
- https://docs.oracle.com/cd/E37069_01/html/E39019/z4000ac019677.html
Supports
- Normal and subnormal representations
- Gradual underflow and reduced subnormal precision
- Extended nonzero range near zero
- Quiz answer about subnormal values
- https://docs.oracle.com/cd/E77782_01/html/E77791/index.html
Supports
- Link rationale covering formats, rounding, underflow, exceptions, and reproducibility
- Ordered path into detailed numerical-computation material
- https://docs.python.org/3/tutorial/floatingpoint.html
Supports
- Binary fractions and representation error
- Inexact representation of decimal one tenth in binary64
- Approximate comparison and exact float inspection
- More accurate summation approaches
- Quiz answers about representation error and decimal arithmetic choice
- https://docs.python.org/3.11/library/decimal.html
Supports
- Decimal numbers as sign, coefficient, and exponent
- Exact unrounded decimal and rounded decimal floating-point arithmetic
- Precision, rounding, exponent limits, flags, and traps in an arithmetic context
- Fixed-point and decimal course guidance
- Quiz answers about fixed scale and exact decimal units
- https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2792.pdf
Supports
- Fixed-width integer ranges and out-of-range results
- Unsigned modular behavior and signed overflow risk
- Checked integer operations and overflow detection
- Quiz answers about width, unsigned range, and narrowing
- https://www.itl.nist.gov/div898/strd/general/related/jsm97wg/lin13.html
Supports
- Definition of cancellation
- Removal of identical leading digits during close subtraction
- Magnification of rounding error already present in operands
- Quiz answer about cancellation
