Computer Arithmetic
Computer arithmetic covers how processors represent and compute with numbers: integer and floating-point formats, overflow and precision limits, rounding behavior, and the hardware circuits that perform addition, multiplication, and division in binary.
itComputer fundamentals | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic - Computer Arithmetic
Computer arithmetic is arithmetic with finite representations. An integer type covers a bounded set of whole numbers. A floating-point type covers a finite, unevenly spaced set of real-number approximations. That constraint explains wraparound, approximate decimal fractions, and why algebraically equivalent expressions can disagree on a machine.
Start with representation. A bit pattern needs a radix and an interpretation. Unsigned and two's-complement signed ranges differ for the same width. Overflow behavior is part of the language or ISA contract: wrap, trap, saturate, widen, or invalid. Never infer it from the bits alone.
Fixed-point arithmetic stores an integer with an implied scale. Floating-point stores a sign, significand, and exponent so one format can cover tiny and huge magnitudes with variable spacing. IEEE 754 defines common binary and decimal formats, rounding, exceptions, and special values such as infinities, NaNs, and subnormals. Each basic operation conceptually produces an exact result, then rounds to the destination format.
Rounding breaks familiar identities. Addition is not generally associative. Cancellation can magnify earlier error when you subtract nearly equal values. Choose bounded integers when you can prove range safety, decimal or scaled integers when decimal rules are the contract, and binary floating point when bounded approximation is acceptable. Specify types at interfaces and test boundaries deliberately.
Read the Intro for representation, rounding, and error vocabulary. Use the Cheatsheet when you need ranges, IEEE binary64 fields, and exception classes. Updates stays false because no single open official tracker covers the combined numeric contracts this course teaches.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://standards.ieee.org/ieee/754/6210/
Supports
- Binary and decimal floating-point formats and methods
- Operations, conversions, exception conditions, and default handling
- Dependence of results on inputs, operation order, and destination format
- Quiz answer about intermediate rounding and grouping
- https://docs.oracle.com/cd/E37069_01/html/E39019/z4000ac019127.html
Supports
- IEEE rounding directions
- Five IEEE floating-point exception classes
- Rounding precision and exception handling
- Boundary and halfway rounding tests
- https://docs.oracle.com/cd/E37069_01/html/E39019/z4000ac019269.html
Supports
- Binary64 sign, exponent, and fraction field widths
- Fifty-three bits of precision for normal values
- Normal values, subnormal values, signed zeros, infinities, and NaNs
- Quiz answer about binary64 precision
- https://docs.oracle.com/cd/E37069_01/html/E39019/z4000ac019677.html
Supports
- Normal and subnormal representations
- Gradual underflow and reduced subnormal precision
- Extended nonzero range near zero
- Quiz answer about subnormal values
- https://docs.oracle.com/cd/E77782_01/html/E77791/index.html
Supports
- Link rationale covering formats, rounding, underflow, exceptions, and reproducibility
- Ordered path into detailed numerical-computation material
- https://docs.python.org/3/tutorial/floatingpoint.html
Supports
- Binary fractions and representation error
- Inexact representation of decimal one tenth in binary64
- Approximate comparison and exact float inspection
- More accurate summation approaches
- Quiz answers about representation error and decimal arithmetic choice
- https://docs.python.org/3.11/library/decimal.html
Supports
- Decimal numbers as sign, coefficient, and exponent
- Exact unrounded decimal and rounded decimal floating-point arithmetic
- Precision, rounding, exponent limits, flags, and traps in an arithmetic context
- Fixed-point and decimal course guidance
- Quiz answers about fixed scale and exact decimal units
- https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2792.pdf
Supports
- Fixed-width integer ranges and out-of-range results
- Unsigned modular behavior and signed overflow risk
- Checked integer operations and overflow detection
- Quiz answers about width, unsigned range, and narrowing
- https://www.itl.nist.gov/div898/strd/general/related/jsm97wg/lin13.html
Supports
- Definition of cancellation
- Removal of identical leading digits during close subtraction
- Magnification of rounding error already present in operands
- Quiz answer about cancellation
