IEEE 754 Floating Point
Computers must represent numbers like 3.14159, 6.022×10²³, and 0.0000001. They must do this using only finite binary storage. Fixed-point Two's Complement Integers representation cannot span both astronomical and microscopic magnitudes with one format. IEEE 754 solves this by borrowing an idea from scientific notation. It splits a number into a sign, an exponent, and a fraction. This lets a small number of bits represent an enormous dynamic range. This standard is not an academic curiosity. It is the arithmetic every processor, GPU, and machine learning accelerator performs billions of times per second. When a neural network trains, when a physics engine simulates gravity, when a spreadsheet sums a column of dollar amounts, IEEE 754 floating point is silently doing the work underneath. Nearly every general-purpose CPU built since the late 1980s implements this exact standard. That is why the same floating-point computation gives the same bit-for-bit result on an Intel laptop, an ARM phone, and a RISC-V microcontroller. The big picture is a tradeoff: floating point trades exactness for range. Integers represent every value in a bounded interval exactly. Floats represent an enormous range of magnitudes but only approximately. Their precision shrinks as numbers grow. Understanding the bit layout — sign, exponent, mantissa — matters. So do the rules for bias, normalization, and special values (infinity, NaN, zero, subnormals). These are what separate a programmer who merely uses floats from one who understands why `0.1 + 0.2 != 0.3`. That difference is not a bug. This article dissects the 32-bit and 64-bit IEEE 754 formats bit by bit. It walks through the encoding and decoding algorithm with worked hexadecimal traces. It explains the reasoning behind biased exponents and normalization. It also catalogs the special values that make floating-point arithmetic closed and well-defined even at its edges.
- IEEE 754 encodes a number as sign · 1.mantissa × 2^(exponent − bias), packed into 32 or 64 bits.
- The exponent is stored with a bias (127 for single precision) so it can represent negative exponents using only unsigned bit patterns.
- Normalization forces a unique representation: exactly one nonzero digit before the binary point, giving an implicit leading 1.
- Subnormal numbers (exponent field all zeros) fill the gap near zero, trading precision for smoother underflow.
- Special values — ±0, ±∞, and NaN — are reserved exponent/mantissa patterns that make operations like 1/0 and 0/0 well-defined.
- Floating-point arithmetic is not associative; rounding error accumulates, which is why numerical algorithms must be designed carefully.
IEEE 754 represents a real number as sign, biased exponent, and normalized mantissa — trading exact values for a vast, uniform dynamic range across all computers.
The idea
A floating-point number represents a real value in a format analogous to scientific notation: a sign, a significand (also called the mantissa or fraction), and an exponent that scales the significand by a power of two. Formally, IEEE 754 single precision (32 bits) encodes ``` value = (-1)^sign × 1.mantissa₂ × 2^(exponent - 127) ``` where `sign` is one bit, `exponent` is an 8-bit unsigned field, and `mantissa` is a 23-bit fraction. Double precision (64 bits) uses the same structure with an 11-bit exponent (bias 1023) and a 52-bit mantissa. The exponent field stores a biased value rather than a signed integer: the true exponent is `stored_exponent - bias`. This lets the hardware compare two floating-point numbers' magnitudes using ordinary unsigned integer comparison on the exponent field, without needing separate signed-exponent logic — a key reason the format is efficient in silicon.
Normalization is the rule that the significand is always written as `1.xxxxx` in binary, with exactly one nonzero bit before the radix point (except for subnormals and zero). Because that leading bit is always 1 for normalized numbers, it does not need to be stored at all — it is an implicit leading bit, giving 23 stored mantissa bits an effective 24 bits of precision for free. This is the single cleverest trick in the format: one bit of precision purchased at zero storage cost.
How it works
Encoding a real number into IEEE 754 single precision proceeds in four steps: determine the sign, convert the magnitude to binary, normalize it into the form `1.f × 2^e`, then pack sign, biased exponent, and fraction bits into the 32-bit word. ``` FUNCTION encode_ieee754_single(x: real number) -> 32-bit word: if x == 0: return sign_bit(x) followed by 31 zero bits // handles +0 and -0 if x is NaN: return sign=0, exponent=255, mantissa=nonzero (e.g., 0x400000) if x is +infinity or -infinity: return sign, exponent=255, mantissa=0 sign = 0 if x >= 0 else 1 x = abs(x) // Step 1: convert magnitude to binary scientific notation e = 0 while x >= 2: x = x / 2 e = e + 1 while x < 1: x = x * 2 e = e - 1 // invariant here: 1 <= x < 2 // Step 2: check exponent range against normalized limits if e < -126: return encode_subnormal(sign, x, e) // gradual underflow if e > 127: return sign, exponent=255, mantissa=0 // overflow to infinity // Step 3: bias the exponent biased_exponent = e + 127 // 1 <= biased_exponent <= 254 // Step 4: extract 23 mantissa bits from the fractional part of x fraction = x - 1.0 // drop implicit leading 1 mantissa_bits = round_to_23_bits(fraction) // round-to-nearest-even return pack(sign, biased_exponent, mantissa_bits) ``` Decoding reverses this exactly: ``` FUNCTION decode_ieee754_single(word: 32-bit word) -> real number: sign = word[31] exponent = word[30:23] // 8 bits, unsigned mantissa = word[22:0] // 23 bits if exponent == 255: if mantissa == 0: return sign ? -infinity : +infinity else: return NaN if exponent == 0: if mantissa == 0: return sign ? -0.0 : +0.0 // subnormal: no implicit leading 1, fixed exponent -126 value = mantissa / 2^23 × 2^(-126) return sign ? -value : value // normalized case: implicit leading 1 value = (1 + mantissa / 2^23) × 2^(exponent - 127) return sign ? -value : value ``` The exponent field's two extreme values (`00000000` and `11111111` in single precision) are reserved as flags rather than ordinary exponents — this is what makes special values possible, discussed below. ``` bit 31 bits 30–23 bits 22–0 ┌───────┬──────────────────────┬─────────────────────────┐ Single (32b) │ S │ Exponent (8) │ Mantissa (23) │ └───────┴──────────────────────┴─────────────────────────┘ 1 bit bias = 127 23 bits ```
The special-value zoo follows directly from reserving the exponent extremes:
| Exponent field | Mantissa | Meaning |
|---|---|---|
| all 0s | 0 | ±0 (sign bit distinguishes +0 and −0) |
| all 0s | nonzero | subnormal (denormalized) number, no implicit 1 |
| 1 to (max−1) | any | normalized number |
| all 1s | 0 | ±∞ |
| all 1s | nonzero | NaN (Not a Number) |
Subnormals exist so that the smallest representable numbers approach zero gradually rather than jumping abruptly from the smallest normal number to zero. Without them, `x - y == 0` could fail to imply `x == y` for tiny x, y — a property called gradual underflow, and losing it breaks numerical algorithms that rely on `a == b` iff `a - b == 0`.
NaN (Not a Number) results from indeterminate operations like `0/0`, `∞ - ∞`, or `sqrt(-1)`. NaN is defined to compare unequal to everything, including itself (`NaN != NaN`), which is precisely how software detects it. Infinity results from overflow or from `x/0` for nonzero x, and it participates correctly in further arithmetic (`1/∞ = 0`, `∞ + 1 = ∞`), keeping the arithmetic system closed under all four operations.
Complexity & correctness
Space. Single precision uses exactly 32 bits (4 bytes); double precision uses exactly 64 bits (8 bytes), fixed regardless of the value represented. This is O(1) space per number, unlike arbitrary-precision or rational representations whose size can grow with the input.
Time. Encoding and decoding are O(1) operations: a fixed number of shifts to normalize (bounded by exponent width, e.g., at most ~127 shifts in the worst case for single precision, though hardware does this in a single normalized-shift circuit, not a loop) followed by constant-time bit packing. Arithmetic operations (add, multiply) are also O(1) per operation in hardware, implemented as fixed-width combinational or pipelined circuits — this is why floating-point throughput on modern CPUs/GPUs is measured in FLOPS (floating point operations per second) at billions to trillions per second.
Correctness of the encoding. The core invariant is: > For any normalized encoding, exactly one representation exists for a given real value (up to rounding), because normalization fixes the leading bit to 1, eliminating the redundancy of unnormalized scientific notation (e.g., `1.0 × 2^3` and `0.5 × 2^4` represent the same number, but only the first is normalized). This uniqueness is why comparing two normalized floats can be done by comparing their bit patterns as if they were unsigned (or signed-magnitude) integers — the ordering of the bit pattern matches the ordering of the value, a property that hardware comparators exploit directly.
Correctness of rounding. IEEE 754 mandates round-to-nearest-even as the default rounding mode: when a true value falls exactly between two representable floats, round to the one whose mantissa's last bit is 0. This eliminates systematic bias that round-half-up would introduce over many operations, and it is provably the rounding mode that minimizes expected error for uniformly distributed inputs.
The honest tradeoff. Floating-point arithmetic is not associative and not distributive: `(a + b) + c` can differ from `a + (b + c)` because each addition rounds independently. This is a direct, unavoidable consequence of finite precision — not an implementation bug. The machine epsilon (2⁻²³ ≈ 1.19×10⁻⁷ for single precision, 2⁻⁵² ≈ 2.22×10⁻¹⁶ for double) bounds the relative rounding error of a single operation: ``` | fl(x op y) - (x op y) | ≤ ε × |x op y| ``` This relative-error bound is the theoretical foundation of numerical analysis: algorithms are analyzed for how they amplify this per-operation error, not for whether error exists at all (it always does, except for exactly representable results).
Worked example
Encode x = 13.375 into IEEE 754 single precision, then decode it back.
Step 1 — sign. x is positive, so `sign = 0`.
Step 2 — binary conversion. 13 in binary is `1101`. The fractional part 0.375 = 0.011₂ (since 0.375 = 1/4 + 1/8 = 0.01₂ + 0.001₂). So: ``` 13.375 = 1101.011₂ ```
Step 3 — normalize. Shift the binary point so exactly one digit precedes it: ``` 1101.011₂ = 1.101011₂ × 2^3 ``` True exponent e = 3.
Step 4 — bias the exponent. ``` biased_exponent = 3 + 127 = 130 = 10000010₂ ```
Step 5 — mantissa. Drop the implicit leading 1, pad to 23 bits: ``` fraction = 101011000000000000000000 (23 bits, zero-padded) ```
Step 6 — assemble the 32-bit word:
| Sign | Exponent (8b) | Mantissa (23b) |
|---|---|---|
| 0 | 10000010 | 10101100000000000000000 |
Packed word: `0 10000010 10101100000000000000000` In hexadecimal, grouping into nibbles: `0100 0001 0101 0110 0000 0000 0000 0000` = `0x41560000`.
Decoding it back, to verify round-trip correctness:
| Field | Bits | Value |
|---|---|---|
| sign | 0 | + |
| exponent | 10000010₂ = 130 | true exponent = 130 − 127 = 3 |
| mantissa | 10101100...₂ | fraction = 0.6875 |
``` value = (1 + 0.6875) × 2^3 = 1.6875 × 8 = 13.5 ``` Wait — that gives 13.5, not 13.375. Let's recheck the mantissa arithmetic carefully, since this is exactly the kind of error a careless trace produces. Mantissa bits `10101100000000000000000` (23 bits) interpreted as a fraction over 2²³: ``` bit values (left to right, weights 2^-1 ... 2^-23): 1·2⁻¹ + 0·2⁻² + 1·2⁻³ + 0·2⁻⁴ + 1·2⁻⁵ + 1·2⁻⁶ = 0.5 + 0.125 + 0.03125 + 0.015625 = 0.671875 ``` So the fraction is 0.671875, not 0.6875 — the earlier decode step made an arithmetic slip, exactly as a real reader might. Redo the final assembly: ``` value = (1 + 0.671875) × 2^3 = 1.671875 × 8 = 13.375 ✓ ``` This matches the original input exactly, confirming the encoding is exact (13.375 has a finite binary expansion, so no rounding was needed) and the round trip is lossless.
Common pitfalls & traps
| Pitfall | Why it happens | Fix | ||
|---|---|---|---|---|
| Assuming `0.1 + 0.2 == 0.3` | 0.1 and 0.2 have no finite binary expansion; both are rounded | Compare with a tolerance: `\ | a - b\ | < epsilon` |
| Forgetting the implicit leading 1 when decoding | Only subnormals and zero omit it; normalized numbers always have it | Always add 1 before multiplying by 2^(exp−bias) for normalized numbers | ||
| Treating NaN as equal to itself | IEEE 754 defines NaN ≠ NaN by specification | Use `isnan(x)` rather than `x == x` checks (though `x != x` is a valid legacy trick) | ||
| Comparing floats for exact equality after arithmetic | Every operation can introduce rounding error | Use relative/absolute epsilon comparisons | ||
| Ignoring subnormal performance cliffs | Many CPUs handle subnormals via slow microcode paths | Flush-to-zero mode in performance-critical numerical code, when correctness allows | ||
| Assuming floating-point addition is associative | Each partial sum rounds independently | Use compensated summation (e.g., Kahan summation) for long reductions |
Pioneers & history
The IEEE 754 standard was not the work of one person but of a committee formed to end decades of chaos, where every computer manufacturer had its own incompatible floating-point format, making numerical results non-portable across machines. The chief architect and driving intellectual force behind the standard was William "Velvel" Kahan, whose insistence on rigorous, provably correct rounding behavior — including the round-to-nearest-even rule and the design of NaN and infinity — shaped nearly every detail of the final specification.
Origins · 1 minWilliam Kahan and how this was discovered
Kahan, a numerical analyst, argued fiercely that floating-point arithmetic should have provable, predictable error bounds rather than ad hoc vendor-specific behavior. His analysis of catastrophic cancellation and his design of the Kahan summation algorithm for compensating rounding error in long sums remain foundational tools in numerical computing. He received the Turing Award in 1989 largely for his contributions to floating-point standardization.
Where this matters
Every general-purpose CPU since the late 1980s — x86, ARM, RISC-V, POWER — implements IEEE 754 arithmetic units directly in hardware, because software emulation of floating point is far too slow for graphics, physics, and scientific workloads. GPUs, which power modern machine learning training, are built around massively parallel IEEE 754 (and reduced-precision variants like FP16 and bfloat16) multiply-accumulate units, since the entire deep learning revolution runs on floating-point matrix multiplication at scale. Programming languages expose these formats directly: `float` and `double` in C, C++, Java, and Python's `float` are IEEE 754 single and double precision respectively. Databases, spreadsheets, and financial systems must be aware of floating-point rounding limitations — many financial applications deliberately avoid floating point for currency, using fixed-point or decimal types instead, precisely because of the pitfalls described above. Scientific computing libraries (LAPACK, BLAS, NumPy) are built entirely atop IEEE 754 semantics, and numerical analysis as a discipline exists largely to reason rigorously about the error these formats introduce. Understanding this standard is foundational to reasoning about Binary and Hexadecimal Representation more broadly, and it directly extends the fixed-width encoding ideas seen in Two's Complement Integers into the domain of real-valued computation.
Ask the tutor: How does fused multiply-add (FMA) reduce rounding error compared to separate multiply and add instructions?
Why can't floating point represent 0.1 exactly?
0.1 in binary is a repeating fraction: `0.0001100110011...₂`, just as 1/3 is a repeating decimal in base 10. IEEE 754 stores only a finite number of mantissa bits (23 for single, 52 for double), so this infinite repetition must be truncated and rounded to the nearest representable value. The stored value is extremely close to 0.1 but not exactly equal to it. This is why arithmetic like `0.1 + 0.2` yields `0.30000000000000004` in double precision rather than exactly `0.3` — both inputs were already rounded before the addition even happened.
What is the difference between +0 and -0 in IEEE 754?
Both represent the mathematical value zero and compare equal under `==`, but they have different sign bits and can produce different results in edge cases. For example, `1/+0` yields `+infinity` while `1/-0` yields `-infinity`, which matters in numerical code that tracks the sign of a quantity approaching zero from either direction. Most everyday code can ignore the distinction, but it exists specifically to preserve sign information through operations like division and to make limits well-defined, matching how signed zero behaves in real analysis and complex function branch cuts.
Why does the exponent use a biased representation instead of two's complement?
Biased representation lets hardware compare the magnitude of two positive floating-point numbers by comparing their raw bit patterns as unsigned integers, without separately decoding the exponent's sign. If the exponent used two's complement, a larger true exponent would not always correspond to a larger bit pattern, breaking this direct comparison trick. Biasing shifts the entire signed exponent range into an unsigned range (0 to 255 for 8 bits, representing exponents −127 to +128), so ordinary unsigned comparison hardware, already needed elsewhere in the CPU, can be reused for float comparison.
What is a subnormal number and why does it matter?
A subnormal (or denormalized) number has an exponent field of all zeros and a nonzero mantissa, representing values smaller in magnitude than the smallest normalized number. It uses a fixed exponent (−126 for single precision) and no implicit leading 1, sacrificing precision as the value shrinks toward zero. Without subnormals, there would be a sudden gap between the smallest normal number and zero, breaking the guarantee that `a - b == 0` implies `a == b`. This property, called gradual underflow, is essential for many iterative numerical algorithms to terminate correctly.
Why is NaN not equal to itself?
The IEEE 754 standard specifically defines all comparisons involving NaN, including `NaN == NaN`, to return false, and all ordered comparisons (`<`, `>`, `<=`, `>=`) to return false as well. This is a deliberate design choice, not an accident: NaN represents "the result of an operation that has no meaningful numeric answer," such as `0/0`, and treating it as equal to itself would falsely suggest that two independently computed NaNs represent the same underlying error. This unusual property is also the standard trick programmers use to detect NaN: `if (x != x) then x is NaN`.
How much precision does double precision actually provide?
Double precision uses a 52-bit stored mantissa plus the implicit leading bit, giving 53 bits of significand precision, roughly 15 to 17 significant decimal digits. Its exponent range spans roughly 10⁻³⁰⁸ to 10³⁰⁸ in magnitude, covering essentially every physically meaningful quantity from subatomic scales to cosmological ones. This is why double precision, despite costing twice the storage of single precision, is the default choice in most scientific and engineering software: the extra bits substantially reduce the accumulation of rounding error over long computations.
Is floating-point arithmetic deterministic across different machines?
Yes, for a fixed sequence of IEEE 754-compliant operations in the same order, the result is bit-for-bit identical on any conforming hardware, because the standard precisely specifies rounding behavior for every basic operation. Non-determinism in real programs usually comes from compilers reordering operations (e.g., auto-vectorization, fused multiply-add substitution) or parallel reduction order differing across runs, not from the arithmetic standard itself. This is why reproducible scientific computing and machine learning training often require disabling certain compiler optimizations or fixing reduction order explicitly.