The File Abstraction
Imagine writing a program that stores a user's high scores. Imagine every program had to know exactly which sectors on which disk cylinder hold that data. Application code would be tangled with disk geometry. A single firmware quirk would break every program on the machine. Operating systems solve this by inserting one clean abstraction between programs and physical storage: the file. A file lets a programmer say "open my save-game data." They never think about tracks, sectors, wear leveling. They also ignore the difference between a spinning platter and a flash chip underneath. This matters because persistent storage is fundamentally messy and heterogeneous. A modern machine may have an NVMe SSD, a USB flash drive, a network share, and a virtual disk image. These all have wildly different physical characteristics. Yet the exact same `open`, `read`, `write`, and `close` calls work on all of them. That uniformity is not an accident. It is deliberate engineering built on top of the I/O Devices and the Device Driver layer. That layer already hides individual device quirks behind block-level read/write operations. The file abstraction goes one layer higher. It hides the very existence of blocks. Instead, it substitutes a named, typed, logically contiguous byte stream. The big picture is that the operating system's storage stack is a tower of abstractions. Each layer hides detail from the one above it. Device drivers turn physical media into arrays of fixed-size blocks. The file system turns blocks into named byte streams with attributes such as owner, size, and timestamps. Above the file system, directories organize files into a navigable namespace. Application libraries add further structure such as records or database pages. This article is about the middle layer of that tower. It covers what a file is, and what attributes and structure it carries. It also covers what types of files exist. Finally, it explains why treating "persistent storage" as "a set of named byte streams" is one of the most durable design decisions in computing history. By the end of this article, you will be able to define a file precisely. You will do so in terms of its attributes and operations. You will explain the tradeoffs between structured and unstructured (byte-stream) file models. You will trace how a file's metadata is represented internally. You will also understand why nearly every modern OS converged on the same minimal abstraction. This sets up the machinery covered in File-System Interface and Operations and Directory Structure. Those articles describe how files are opened, located, and manipulated.
- A file is the OS's abstraction for persistent storage: a named collection of related information, usually modeled as a linear sequence of bytes.
- Every file has attributes (metadata) — name, identifier, type, size, permissions, timestamps — stored separately from the file's actual data.
- Most modern systems use the unstructured byte-stream model; interpretation of structure (records, formats) is left to applications, not the OS.
- File types are often signaled by name extension or magic numbers, but the OS itself may or may not enforce type semantics.
- The OS exposes files through a small set of operations — create, open, read, write, seek, close, delete — that together define the file's interface, independent of physical storage layout.
- The abstraction succeeds because it achieves device independence and persistence while adding only a thin, uniform layer over block storage.
A file turns the messy physical reality of blocks, sectors, and device timing into one clean, named, persistent byte stream that every program can use the same way.
The idea
A file is a named, logical unit of secondary storage that the operating system presents to programs as a persistent, addressable sequence of information. The critical word is logical: a file need not correspond to a physically contiguous region of a disk at all. It is a mapping the file system maintains between a name (and an internal identifier) and some set of storage blocks, wherever they happen to live. This indirection is precisely what allows files to grow, shrink, move, and survive reboots without the program that uses them ever noticing. Two properties distinguish a file from an ordinary in-memory data structure. First, persistence: a file's contents outlive the process that created them, and typically outlive reboots and power loss. Second, naming: a file is identified by a symbolic name (and, beneath that, a system-internal identifier such as an inode number) rather than by a memory address. Everything else — size, type, permissions, timestamps — is metadata, information about the file that the OS tracks on the file's behalf but that is conceptually separate from the file's own byte content. This metadata is what the OS calls the file's attributes. The most consequential design choice in the file abstraction is the structure question: does the OS impose internal structure on a file's bytes, or does it treat the file as an undifferentiated stream and let applications impose whatever structure they like? Early operating systems (and OSes tied closely to specific applications, like some mainframe and record-oriented systems) supported structured files: sequences of fixed- or variable-length records, sometimes with built-in indexing. Unix popularized the opposite extreme — the unstructured byte-stream model — in which the OS knows nothing about "records" or "lines"; a file is simply `N` bytes, and any structure (a CSV row, a JSON object, a JPEG header) is a convention understood only by the application that wrote it. Virtually every general-purpose OS today (Linux, Windows, macOS, BSD) follows the byte-stream model, because it is simpler for the kernel, more flexible for applications, and does not force every program into one record format.
How it works
Conceptually, the OS maintains, for every file, a file control block (FCB) — sometimes called an inode (Unix-family systems) or a master file table entry (NTFS) — that stores the file's attributes and the information needed to locate its data blocks. The file itself, from the program's point of view, is manipulated through a small, uniform set of operations regardless of what device or file system actually stores it.
Core file attributes typically tracked by the OS:
| Attribute | Meaning |
|---|---|
| Name | Human-readable symbolic identifier (kept in the directory, not the FCB itself, in most designs) |
| Identifier | Unique internal number (e.g., inode number) that the file system uses instead of the name |
| Type | Kind of file, if the system enforces types (data, executable, directory, device, etc.) |
| Size | Current length in bytes (and sometimes size in allocated blocks) |
| Location | Pointer(s) to where the data blocks live on the device |
| Protection | Access-control information: owner, group, permission bits or ACL |
| Time stamps | Creation, last-modified, last-accessed times, and the identity of the last modifier |
Core file operations, the minimal interface that any file system must expose: ``` create(name) -- allocate a new FCB, insert entry into directory, size = 0 delete(name) -- remove directory entry, free FCB, release data blocks open(name, mode) -- locate FCB via directory lookup, load/cache it, allocate an in-memory "open file" descriptor, initialize a file position pointer read(fd, buf, n) -- copy n bytes starting at current position into buf, advance position by (bytes actually read) write(fd, buf, n) -- copy n bytes from buf into the file at current position, extend file/allocate blocks if needed, advance position by n seek(fd, offset) -- reposition the file pointer without I/O close(fd) -- flush buffers, release the open-file descriptor, decrement reference count on the FCB ``` The key mechanism that makes this interface efficient is the separation between the on-disk FCB (persistent, one per file, describing attributes and block locations) and the in-memory open-file table entry (transient, one per open, tracking the current byte offset and access mode). Because `open` is the expensive step — it requires a directory search and may require reading the FCB from the storage device — the OS does it once and then lets `read`/`write`/`seek` operate cheaply against cached, in-memory state. ```pseudocode // Simplified model of the open-file mechanism. // Two-level structure: per-process descriptor table -> system-wide open-file table -> FCB. structure FCB: id, type, size, block_pointers[], permissions, timestamps structure OpenFileEntry: fcb_ref // pointer to the file's FCB (cached in memory) offset // current byte position, starts at 0 (or EOF for append mode) access_mode // READ, WRITE, READ_WRITE ref_count // number of descriptors referencing this open instance function OPEN(path, mode): fcb <- DIRECTORY_LOOKUP(path) // resolve name -> FCB identifier if fcb is null: error "no such file" entry <- FIND_OR_CREATE(OpenFileTable, fcb) entry.ref_count += 1 fd <- ALLOCATE_DESCRIPTOR(process, entry) if mode == APPEND: entry.offset <- fcb.size else: entry.offset <- 0 return fd function READ(fd, buffer, n): entry <- DESCRIPTOR_TABLE[process][fd] bytes_available <- entry.fcb_ref.size - entry.offset to_read <- min(n, bytes_available) if to_read <= 0: return 0 // end of file COPY_BYTES(entry.fcb_ref, entry.offset, buffer, to_read) entry.offset += to_read return to_read function WRITE(fd, buffer, n): entry <- DESCRIPTOR_TABLE[process][fd] ENSURE_CAPACITY(entry.fcb_ref, entry.offset + n) // allocate new blocks if needed STORE_BYTES(entry.fcb_ref, entry.offset, buffer, n) entry.offset += n entry.fcb_ref.size <- max(entry.fcb_ref.size, entry.offset) return n function CLOSE(fd): entry <- DESCRIPTOR_TABLE[process][fd] entry.ref_count -= 1 if entry.ref_count == 0: FLUSH_TO_DISK(entry.fcb_ref) REMOVE_FROM_TABLE(OpenFileTable, entry) FREE_DESCRIPTOR(process, fd) ``` Notice what this pseudocode does not do: it never mentions cylinders, sectors, controllers, or DMA. All of that lives beneath `COPY_BYTES`/`STORE_BYTES`, delegated to the block layer and ultimately the device driver. The file abstraction's entire job is to make `read`/`write`/`seek` behave identically whether the underlying medium is an SSD, a spinning disk, or a RAM-backed tmpfs.
Complexity & correctness
Time complexity of the core operations. Let a file system organize its directory as a hash table or B-tree (as most modern systems do) and represent block locations via an index structure such as a multi-level index (classic Unix inode) or an extent list (modern journaling file systems).
- `open(name)`: dominated by directory lookup. With a hashed or tree-indexed directory, this is `O(log n)` or expected `O(1)`, where `n` is the number of entries in the directory (not the whole file system). A naive linear-scan directory (some very old or simple systems) costs `O(n)`.
- `read`/`write` of `k` bytes at a given offset: requires translating a byte offset to a block number. With a multi-level index (direct blocks, single/double/triple indirect blocks, as in classic Unix), this is `O(1)` for small files (direct blocks) and `O(log_B(size))`-ish in the worst case for very large files needing indirect-block traversal, where `B` is the branching factor of index blocks — in practice bounded by a small constant (2–3 indirection hops) because block pointers are wide relative to block size. Actual data transfer of `k` bytes costs `O(k / block_size)` I/O operations, i.e., linear in the amount of data moved, which is the best possible since every byte must be touched to be read or written.
- `seek(offset)`: `O(1)` — it only updates the in-memory offset field; no I/O occurs.
- `close(fd)`: `O(1)` bookkeeping, plus the cost of flushing any dirty cached blocks, which is proportional to the number of modified blocks, not the file size.
Space complexity. Each open file costs `O(1)` additional memory (one open-file table entry, one descriptor). The FCB itself is `O(1)` per file for attributes plus `O(size / block_size)` for the block-pointer structure in the worst case (though indirect blocks amortize this to a small constant number of extra blocks). The total metadata overhead of a file system is therefore linear in the number of files and the total data size, which is asymptotically optimal — you cannot address `m` blocks using less than `Ω(log m)` bits per pointer, and you need at least one pointer per block region.
Correctness argument. The abstraction's correctness rests on one invariant that every implementation must preserve: > Byte-addressing invariant: for a file of size `S`, at every moment between `open` and `close`, a call to `read` (respectively `write`) at logical offset `o` must return (respectively modify) exactly the bytes that were most recently written at that offset, for all `0 ≤ o < S`, regardless of how those bytes are physically distributed across blocks or devices. The FCB's block-mapping structure is precisely the data structure that must maintain this invariant under mutation: whenever `WRITE` extends the file (`ENSURE_CAPACITY`), it must allocate new blocks and update the mapping before advertising the new size, so no reader ever sees a size that exceeds the actual mapped and initialized data (this is why many systems zero-fill or explicitly track "holes" in sparse files rather than exposing uninitialized disk contents — a real security concern). The open-file table's offset field must be updated atomically with respect to the corresponding I/O so that concurrent `read`/`write` calls on the same descriptor observe a consistent, monotonically-progressing position (single-threaded correctness is immediate from the pseudocode; multi-threaded correctness requires locking the offset field, which real systems do).
Tradeoffs. The byte-stream model is simple and general but pushes all structure-parsing cost to user space; a structured/record-oriented file system can offer `O(1)` "get me record `i`" access built into the OS, at the cost of forcing one record model on every application and substantially complicating the kernel. Nearly universal industry convergence on the unstructured model reflects a bet, borne out over five decades, that flexibility at the OS layer is worth more than built-in structure.
Worked example
Consider a small text file `notes.txt`, created and then appended to, tracked with 4 KB blocks and a simple direct-block index (first 10 pointers direct, an 11th indirect pointer for larger files — the classic simplified inode layout).
Step 1 — `create("notes.txt")`. The file system allocates a fresh FCB (say, inode #402), sets `size = 0`, `type = regular`, all block pointers null, and inserts a directory entry mapping the name `"notes.txt"` to inode `402`.
Step 2 — `open("notes.txt", WRITE)`. Directory lookup resolves `"notes.txt" → 402` in `O(log n)`. The FCB is loaded into memory; an open-file table entry is created with `offset = 0`, `ref_count = 1`.
Step 3 — `write(fd, "Meeting at 10am\n", 16)`. `ENSURE_CAPACITY` sees the file needs 16 bytes, which fits inside one 4 KB block. A free block, say physical block 8801, is allocated and recorded as direct-pointer #0 in the FCB. The 16 bytes are written into block 8801 at offset 0. `entry.offset → 16`, `fcb.size → 16`.
Step 4 — `write(fd, "Buy milk\n", 9)` (append continues from current offset). Still within block 8801 (16 + 9 = 25 bytes, well under 4096). Bytes are written at offset 16 within that block. `entry.offset → 25`, `fcb.size → 25`.
Step 5 — a large `write` of 5000 more bytes (simulating pasting a big log). `ENSURE_CAPACITY(25 + 5000 = 5025)` now exceeds one block (4096 bytes), so a second block, say 8802, is allocated as direct-pointer #1. The write is split: the remaining `4096 − 25 = 4071` bytes of the first block are filled, and the remaining `5000 − 4071 = 929` bytes go into block 8802. `entry.offset → 5025`, `fcb.size → 5025`.
Step 6 — `seek(fd, 20)` then `read(fd, buf, 5)`. `seek` sets `entry.offset ← 20` with zero I/O. `READ` computes `bytes_available = 5025 − 20 = 5005`, so `to_read = min(5, 5005) = 5`. It translates logical offset 20 into block 8801 (since `20 < 4096`), copies 5 bytes ("milk\n" minus the newline handling aside), and advances `entry.offset → 25`.
Step 7 — `close(fd)`. `ref_count` drops to 0; the FCB (now with two block pointers, size 5025, updated modification time) is flushed to persistent storage; the open-file entry is destroyed.
| Step | Operation | fcb.size | entry.offset | Blocks allocated | I/O cost |
|---|---|---|---|---|---|
| 1 | create | 0 | — | none | metadata write, O(1) |
| 2 | open | 0 | 0 | none | directory lookup, O(log n) |
| 3 | write 16B | 16 | 16 | {8801} | 1 block write |
| 4 | write 9B | 25 | 25 | {8801} | 1 block write (same block) |
| 5 | write 5000B | 5025 | 5025 | {8801, 8802} | 2 block writes |
| 6 | seek+read 5B | 5025 | 25 | {8801, 8802} | 1 block read |
| 7 | close | 5025 | (freed) | {8801, 8802} | metadata flush |
Every intermediate step preserves the byte-addressing invariant: at Step 6, reading logical offset 20 correctly retrieves data written in Step 3, even though two more writes and one block allocation happened in between — because the FCB's block-mapping structure, not the physical layout, is the single source of truth for "what byte lives where."
Common pitfalls & traps
| Pitfall | Why it happens | Consequence |
|---|---|---|
| Confusing the file name with the file identity | Beginners assume the name is the file | Hard links / renames break intuitions; a file can have many names, one identity (inode) |
| Treating `size` as always accurate for sparse files | Some writes create "holes" (seek past end, then write) | Reading unwritten regions may return zero-fill or, in buggy systems, stale physical data — a real security bug class |
| Assuming `close` guarantees data is on physical media | `close` may only flush to the OS cache, not the device | Power loss after `close` can still lose data unless `fsync`/`flush`-to-device is called explicitly |
| Off-by-one in offset/size after `write` | Forgetting that `size` is `max(size, offset)`, not `size += n`, when writes aren't purely sequential | Overwrites in the middle of a file silently shrink the reported size, or size fails to grow correctly |
| Ignoring concurrent access to a shared file descriptor | Two threads share one `fd`/offset without locking | Interleaved reads/writes corrupt the logical position, violating the byte-addressing invariant |
| Assuming all file systems enforce "type" | Type enforcement (e.g., Windows extensions vs. Unix magic numbers) is a convention, not a kernel guarantee on most Unix systems | Renaming `virus.exe` to `virus.txt` doesn't change what it does if executed directly; type checks must happen elsewhere |
Pioneers & history
The file abstraction did not appear fully formed; it emerged from decades of tension between record-oriented mainframe storage and the desire for a simpler, uniform model.
Origins · 1 minKen Thompson and how this was discovered
Ken Thompson, working with Dennis Ritchie at Bell Labs in the early 1970s, helped crystallize the idea that "everything is a file" — that devices, pipes, and ordinary data could all be manipulated through the same small set of byte-stream operations (`open`, `read`, `write`, `close`). This unification, radical for its time compared to the record- and device-specific I/O of contemporaneous systems like OS/360, became the dominant model for nearly all subsequent operating systems.
Where this matters
The file abstraction is the load-bearing wall underneath nearly all persistent-storage software. Databases (PostgreSQL, SQLite, MySQL's InnoDB) build their own structured records on top of plain OS files, deliberately choosing to reinterpret the byte stream rather than rely on any OS-level record support — a direct validation of the unstructured model's flexibility. Version control systems like Git store objects as ordinary files, with all "structure" (trees, commits) encoded purely as byte-stream content plus naming convention. Container and virtualization technologies (Docker overlay filesystems, virtual machine disk images) manipulate entire file systems as single large files, relying on the guarantee that a file is just an addressable byte sequence regardless of what's inside it. Even network file protocols (NFS, SMB) and cloud object stores (S3) preserve variants of this same interface — named, byte-addressable, attribute-bearing storage units — because decades of application software assume it. Device files (`/dev/sda`, `/dev/null`) in Unix-like systems are the abstraction pushed to its logical extreme: hardware itself presented as a file, closing the loop back to the I/O Devices and the Device Driver layer this article builds upon.
Ask the tutor: How does a file system decide where to place new blocks on disk to minimize fragmentation, and how does that interact with the file abstraction described here?
Why don't operating systems just let programs address disk blocks directly?
Direct block addressing would tie every program to the physical geometry and allocation policy of one specific storage device, breaking portability the moment hardware changed. It would also make it impossible to safely share storage between programs, since nothing would prevent one program's raw block writes from clobbering another's data. The file abstraction solves both problems: it introduces indirection (a name and an FCB) so the OS can freely relocate, grow, or shrink the physical blocks behind a file, and it centralizes access control so the kernel — not each application — enforces who may read or write which bytes. This indirection costs only a small, bounded lookup overhead per operation, which is a worthwhile price for portability, safety, and simplicity.
What exactly is stored in an FCB, and where does it live?
An FCB (file control block, called an inode in Unix-family systems) stores everything the file system needs to know about a file except its name: a unique identifier, type, size, permission bits, timestamps, and pointers to the data blocks holding the file's content. Critically, the FCB is stored separately from the directory entry that maps a human-readable name to that identifier — this separation is what allows multiple names (hard links) to refer to the same underlying file. FCBs live in a reserved metadata region of the storage device (an inode table in ext-family file systems, the Master File Table in NTFS) and are cached in memory while a file is open, to avoid repeated slow reads from the device.
Does the OS enforce file "type," like .exe versus .txt?
It depends on the system. Windows and many desktop environments rely heavily on filename extensions and sometimes an internal type field, using them to decide which application handles a double-click and, in some contexts, to restrict execution. Unix-family systems generally treat the extension as pure convention with no kernel-enforced meaning; the kernel instead inspects the file's actual leading bytes ("magic numbers") when execution is attempted, and utilities like `file` do the same for identification. In neither model does renaming a file change its underlying bytes or genuinely alter what it "is" — type enforcement, where it exists, is a policy layered on top of the byte-stream abstraction, not a property the file format itself guarantees.
Why does the file abstraction separate the on-disk FCB from the in-memory open-file table entry?
The FCB holds information that must persist and be shared correctly across every process that might ever open the file — its size, permissions, and block map. The open-file table entry, in contrast, holds information specific to one particular act of opening the file: the current read/write position and the mode it was opened in. If these were merged, two processes opening the same file simultaneously would be forced to share one read/write cursor, which is almost never the desired behavior — each process typically expects its own independent position within the file. Separating them lets multiple opens of the same file proceed independently while still sharing one consistent view of the file's actual content and attributes.
How does the file abstraction relate to what comes next, like directories and the file-system interface?
This article focuses on what a file is in isolation: its attributes, its byte-stream (or structured) content model, and the minimal operation set that defines its interface. It deliberately stops short of explaining how names are organized into hierarchies, how multiple files are located efficiently among millions of others, or how access permissions cascade through a tree — those are the concerns of Directory Structure. Likewise, the exact system-call semantics, error handling, and richer operations like file locking or memory-mapped files belong to File-System Interface and Operations. Think of this article as defining the noun ("a file"), while the following material defines the verbs and the namespace around it.
Is the unstructured byte-stream model strictly better than structured/record-oriented files?
Not strictly — it is a tradeoff that happened to win in general-purpose computing. Structured files, where the OS understands records and can offer built-in indexed or keyed access, can be more efficient for workloads the OS was specifically designed around, and they were common in earlier mainframe and business-data-processing systems. The unstructured model sacrifices that built-in efficiency for generality: any application can impose any structure it wants (text lines, database pages, image formats) without the kernel needing to understand or validate it. History shows this flexibility mattered more in practice, since applications like databases simply reimplement structure on top of plain files rather than needing the OS to provide it, and the resulting kernel is simpler and more broadly reusable.
What happens, mechanically, when a file grows beyond what its current blocks can hold?
The file system's block-allocation logic detects, during a write, that the requested offset plus length exceeds the space already reserved by the file's existing block pointers. It then requests one or more free blocks from its free-space management structure (a bitmap or free list), records the new block's location in the file's index structure — a direct pointer if there is room, or an indirect block if the direct pointers are exhausted — and only then performs the actual byte copy into the newly mapped region. The file's `size` attribute is updated to reflect the new highest byte written, and this updated FCB is eventually flushed to persistent storage so the growth survives a crash between allocation and the next flush.