File-System Interface and Operations
Every program that saves a document, reads a config file, or appends to a log depends on one deceptively simple question. How does an operating system let thousands of unrelated processes safely share a handful of physical storage devices? The answer is a carefully layered interface. It is built from a small number of system calls — `open`, `read`, `write`, `lseek`, `close`. It also relies on a handful of kernel data structures. These structures track what each process is doing with each file at every instant. This article is about that interface: the contract between user programs and the kernel that makes files usable. It builds directly on The File Abstraction, which explains what a file is (a named, persistent sequence of bytes or records). Here we explain how a process gets hold of one, manipulates it, and lets it go. We also explain how the kernel keeps many such interactions consistent. This matters even when several processes touch the same file at once. The central design problem is indirection with bookkeeping. A process should never manipulate a file directly by name on every operation. Names are slow to resolve, and permissions must be rechecked. Position within the file must also be remembered somewhere. The solution is file descriptors backed by an open-file table. That table is itself backed by a system-wide open-file description table. This is one of the most durable ideas in operating systems. It has been essentially unchanged since it appeared in Unix in the early 1970s. By the end of this article you will understand precisely what a file descriptor is. You will see how the three-level table structure works and why it exists. You will learn how read/write/seek cooperate through a shared file offset. You will also understand how access modes are enforced. Finally, you will see how locking is layered on top. This gives processes control over concurrent access. This sets up the next natural questions. These are how names map to files (Directory Structure) and how the kernel actually finds the bytes on disk (File-System Implementation and Layout).
- `open()` translates a pathname into a small integer, the file descriptor, that names an open file within a process.
- Three levels of indirection — per-process fd table, system-wide open-file table, and the in-memory inode/vnode — let file state be shared correctly across `fork` and `dup`.
- `read`/`write` operate on an implicit file offset kept in the open-file table entry, which is why sequential I/O "just works" without explicit position arguments.
- `lseek` repositions that offset without touching data, enabling random access and sparse files.
- Access modes (`O_RDONLY`, `O_WRONLY`, `O_RDWR`, plus flags like `O_APPEND`, `O_CREAT`, `O_TRUNC`) are checked once at `open` time and cached, not rechecked on every I/O call.
- File locking (advisory `flock`/`fcntl` locks, or mandatory locks in some systems) is a cooperative mechanism layered above this interface to serialize concurrent access.
A file descriptor is not the file — it is a handle into a chain of kernel tables (per-process → system-wide → in-memory file object) that together remember which file, at what offset, with what permissions, a process is currently interacting.
The idea
A file descriptor (fd) is a small non-negative integer that a process uses to refer to an open file, pipe, socket, or other I/O channel. It is returned by `open()` and consumed by `read`, `write`, `lseek`, `close`, and dozens of other calls. The fd itself carries no information about the file — it is purely an index into a table maintained by the kernel on behalf of the process. This indirection is what allows the same small integer space (typically 0–1023 by default, extensible) to refer to files, network connections, and devices interchangeably: the kernel, not the calling code, decides what operations mean for that particular fd. Three cooperating data structures implement this indirection, and understanding why there are three, not one, is the crux of the topic. The per-process file descriptor table maps small integers to entries in a system-wide open-file table (sometimes called the open-file description table or file table). Each open-file table entry holds the file offset (current read/write position), the status flags from `open` (such as `O_APPEND`), and a pointer into the in-memory inode table (or vnode table in systems supporting multiple file-system types), which holds file-system-level metadata: size, permissions, block pointers, reference counts. Two independent `open()` calls on the same file produce two different open-file table entries — each with its own offset — while a single `open()` shared across `fork()` or `dup()` produces descriptors that point to the same open-file table entry, and therefore share an offset. This distinction — shared vs. independent offsets — is precisely what the three-level structure is built to express, and it cannot be expressed with only two levels. An access mode is the permission with which a file was opened: read-only, write-only, or read-write, established once at `open()` time via flags and validated against the file's permission bits and the process's credentials. Because the mode is fixed for the life of the descriptor, every subsequent `read`/`write` call is a cheap check against a cached flag rather than a full permission re-evaluation. File locking sits above this layer entirely: it is a voluntary (or, in some systems, kernel-enforced) protocol by which processes coordinate concurrent access to the same file, using either whole-file locks (`flock`) or byte-range locks (`fcntl` `F_SETLK`), independent of the descriptor/offset machinery underneath.
How it works
Opening a file. `open(path, flags, mode)` performs pathname resolution (traversing Directory Structure to locate the file, or creating it if `O_CREAT` is set), checks the requesting process's permissions against the file's owner/group/mode bits, and — if all checks pass — allocates the three-level chain of table entries described above. It returns the smallest unused non-negative integer as the new file descriptor (POSIX guarantees this "lowest available fd" rule, which matters for correctness of `dup2`-based redirection idioms).
Reading and writing. `read(fd, buffer, n)` and `write(fd, buffer, n)` look up the fd in the process table to find the open-file table entry, use its stored offset to determine where to transfer bytes, perform the transfer (via the File Abstraction's block/record model down into the file-system implementation), and then advance the offset by the number of bytes actually transferred. Both calls return the count of bytes actually moved, which may be less than requested (a "short read/write") near end-of-file or due to interrupted system calls — correct client code must always check this return value and loop if necessary.
Seeking. `lseek(fd, offset, whence)` changes the stored offset without transferring any data, where `whence` is one of `SEEK_SET` (absolute), `SEEK_CUR` (relative to current offset), or `SEEK_END` (relative to end-of-file). Seeking past the current end of file is legal; a subsequent write there creates a sparse file, with the gap logically read back as zero bytes without necessarily consuming physical storage.
Closing. `close(fd)` removes the process's fd table entry and decrements the reference count on the open-file table entry; only when that count reaches zero is the entry actually freed (running any final flush) and the in-memory inode's own reference count decremented in turn.
Locking. A process wanting exclusive or shared access calls `flock(fd, operation)` for whole-file advisory locks, or `fcntl(fd, F_SETLK/F_SETLKW, lockdesc)` for POSIX byte-range locks specifying a start offset, length, and type (`F_RDLCK`, `F_WRLCK`, `F_UNLCK`). These locks are checked cooperatively — a well-behaved reader/writer takes the lock before touching data — and are typically advisory: the kernel does not stop an uncooperative process from ignoring them, though a few systems support mandatory locking where the kernel itself enforces them on every I/O call. ```pseudocode // --- Kernel-side state, conceptually --- struct OpenFileTableEntry { inode_ptr : pointer to in-memory inode/vnode offset : integer // shared file position status_flags : bitmask // O_APPEND, O_RDONLY, O_WRONLY, O_RDWR, ... refcount : integer // number of fd-table entries pointing here } struct Process { fd_table : array of pointer to OpenFileTableEntry // indexed 0, 1, 2, ... } // --- open() --- function OPEN(path, flags, mode): inode = RESOLVE_PATH(path) // see Directory Structure if inode == NULL: if flags has O_CREAT: inode = CREATE_FILE(path, mode) else: return ERROR(ENOENT) if not PERMITTED(current_process, inode, flags): return ERROR(EACCES) if flags has O_TRUNC and inode.size > 0: TRUNCATE(inode) entry = new OpenFileTableEntry( inode_ptr = inode, offset = (flags has O_APPEND) ? inode.size : 0, status_flags = flags, refcount = 1) fd = SMALLEST_UNUSED_INDEX(current_process.fd_table) current_process.fd_table[fd] = pointer to entry inode.open_refcount += 1 return fd // --- read() --- function READ(fd, buffer, n): entry = current_process.fd_table[fd] if entry == NULL: return ERROR(EBADF) if entry.status_flags does not permit reading: return ERROR(EBADF) bytes_available = entry.inode_ptr.size - entry.offset to_read = min(n, max(bytes_available, 0)) if to_read == 0: return 0 // end of file TRANSFER_BYTES(entry.inode_ptr, entry.offset, buffer, to_read) // may be a short transfer entry.offset += to_read return to_read // --- write() --- function WRITE(fd, buffer, n): entry = current_process.fd_table[fd] if entry == NULL: return ERROR(EBADF) if entry.status_flags does not permit writing: return ERROR(EBADF) if entry.status_flags has O_APPEND: entry.offset = entry.inode_ptr.size // re-sync to current end, atomically // (kernel serializes this against other appenders) STORE_BYTES(entry.inode_ptr, entry.offset, buffer, n) entry.offset += n entry.inode_ptr.size = max(entry.inode_ptr.size, entry.offset) return n // --- lseek() --- function LSEEK(fd, offset, whence): entry = current_process.fd_table[fd] if entry == NULL: return ERROR(EBADF) base = 0 if whence == SEEK_SET: base = 0 if whence == SEEK_CUR: base = entry.offset if whence == SEEK_END: base = entry.inode_ptr.size new_offset = base + offset if new_offset < 0: return ERROR(EINVAL) entry.offset = new_offset // no I/O performed return new_offset // --- close() --- function CLOSE(fd): entry = current_process.fd_table[fd] if entry == NULL: return ERROR(EBADF) current_process.fd_table[fd] = NULL entry.refcount -= 1 if entry.refcount == 0: FLUSH_IF_NEEDED(entry.inode_ptr) entry.inode_ptr.open_refcount -= 1 if entry.inode_ptr.open_refcount == 0 and entry.inode_ptr.link_count == 0: RECLAIM_STORAGE(entry.inode_ptr) // deferred delete: "unlink after last close" FREE(entry) return SUCCESS // --- dup(): make a second fd alias the same open-file entry --- function DUP(fd): entry = current_process.fd_table[fd] if entry == NULL: return ERROR(EBADF) new_fd = SMALLEST_UNUSED_INDEX(current_process.fd_table) current_process.fd_table[new_fd] = entry entry.refcount += 1 return new_fd ```
Complexity & correctness
Time complexity. `open()` costs `O(d)` where `d` is the depth of the pathname, since each path component requires a directory lookup (Directory Structure typically implements each lookup in `O(1)` amortized via hashing, or `O(log k)` with a balanced structure over `k` entries, per directory). Once open, `read`, `write`, and `lseek` are `O(1)` for the descriptor-table/offset bookkeeping itself; the data transfer cost is `O(n)` in the number of bytes moved, dominated by the underlying storage access pattern (sequential vs. random), which is the concern of File-System Implementation and Layout rather than of the interface layer. `close()` is `O(1)` amortized, with an occasional `O(b)` flush cost proportional to dirty buffers `b` if a synchronous flush policy is used.
Space complexity. Each process's fd table is `O(k)` in the number of concurrently open descriptors `k` for that process (bounded by a per-process and system-wide limit, e.g. `RLIMIT_NOFILE`). The system-wide open-file table is `O(m)` for `m` total open-file descriptions across all processes, and the in-memory inode table is `O(f)` for `f` distinct files currently referenced by any open description, deduplicated because two descriptions on the same file share one inode entry.
Correctness argument — why three levels, provably. Consider the invariant the interface must maintain: two descriptors created by independent `open()` calls on the same file must never share an offset, while two descriptors created by `dup`/`dup2`/`fork` from a common ancestor descriptor must always share an offset. A two-level design (fd table → inode table directly) cannot express this: if the offset were stored in the inode, all descriptors on that file — related or not — would incorrectly share position; if the offset were stored in the fd table, no descriptors — even `dup`-ed ones — would share position, breaking the POSIX semantics that `write` after `fork` advances the shared offset seen by both parent and child. Only a middle table, whose entries are the unit of sharing for `dup`/`fork` and whose reference count governs lifetime, satisfies both halves of the invariant simultaneously. This is a structural necessity, not an implementation convenience.
Correctness of read/write bookkeeping. The offset-update rule `entry.offset += bytes_transferred` (rather than `+= n_requested`) is what guarantees that a loop of short reads correctly reconstructs the full file with no gaps or overlaps: by induction, if the offset accurately reflects "bytes consumed so far" before a call, updating it by exactly the transferred count preserves that invariant afterward, regardless of how the kernel chooses to fragment the transfer internally (e.g., across multiple disk blocks).
Tradeoffs. The indirection costs one or two extra pointer dereferences per I/O call versus a hypothetical flat design, which is negligible next to actual storage latency, so the added safety and sharing semantics are essentially free in practice. The advisory nature of most locking APIs is a genuine tradeoff: it is cheap and flexible (readers that don't care about coordination pay nothing) but provides no protection against a misbehaving or buggy process, unlike mandatory locking, which is safer but adds overhead to every I/O call and complicates deadlock avoidance.
Worked example
Trace a shell running `cmd > log.txt 2>&1`, followed by a child process that also opens the same file independently for reading.
| Step | Call | fd table (parent) | Open-file entry created/used | Offset after call | ||
|---|---|---|---|---|---|---|
| 1 | `fd1 = open("log.txt", O_WRONLY\ | O_CREAT\ | O_TRUNC)` | fd 3 → Entry A | A: inode=log.txt, flags=O_WRONLY, offset=0 | 0 |
| 2 | `dup2(fd1, 1)` (redirect stdout) | fd 1 → Entry A also | A.refcount = 2 | 0 | ||
| 3 | `dup2(1, 2)` (`2>&1`, redirect stderr to same place) | fd 2 → Entry A also | A.refcount = 3 | 0 | ||
| 4 | child writes `"build ok\n"` (9 bytes) to fd 1 | — | A.offset: 0 → 9 | 9 | ||
| 5 | child writes `"warn: x\n"` (8 bytes) to fd 2 | — | A.offset: 9 → 17 | 17 | ||
| 6 | separate process: `fd2 = open("log.txt", O_RDONLY)` | (own process) fd 3 → Entry B | B: inode=log.txt (same inode as A), offset=0 | 0 | ||
| 7 | that process: `read(fd2, buf, 100)` | — | reads bytes 0–16 (all 17 written so far) | B.offset → 17 | ||
| 8 | original process: `close(2)` | fd 2 cleared | A.refcount: 3 → 2 | (A unaffected) | ||
| 9 | original process: `close(1)` | fd 1 cleared | A.refcount: 2 → 1 | (A unaffected) | ||
| 10 | original process: `close(3)` (fd1) | fd 3 cleared | A.refcount: 1 → 0 → entry freed, flush | — |
Two facts fall directly out of the three-level design. First, steps 4 and 5 show stdout and stderr, despite being different descriptors (1 and 2), sharing one offset because both trace back to the same `open()` call via `dup2` — this is exactly why interleaved writes to redirected stdout/stderr don't overwrite each other. Second, step 6 shows a second, fully independent `open()` on the same underlying file produces its own offset (Entry B), which is why the reading process in step 7 correctly sees bytes from the start of the file rather than inheriting the writer's position — the sharing is a property of the descriptor's ancestry (via `dup`/`fork`), never of the filename.
Common pitfalls & traps
| Pitfall | Why it happens | Fix |
|---|---|---|
| Assuming `read`/`write` always transfer the full requested count | Short reads/writes are legal and common (pipes, interrupted syscalls, EOF near the boundary) | Always loop, checking the return value, until `n` bytes are moved or 0/EOF/error is returned |
| Forgetting that `fork()` duplicates descriptors, not files | Child and parent get separate fd-table entries but the same open-file entry | Understand that writes from either process advance a shared offset — coordinate explicitly if that's unwanted |
| Treating fd numbers as stable identities across the process's lifetime | The kernel reuses the lowest free fd; closing fd 3 then opening a new file may reissue fd 3 | Never hard-code fd numbers for meaning beyond 0/1/2; track semantics via variables, not integer values |
| Using `O_APPEND` incorrectly for "exclusive append" | `O_APPEND` guarantees the offset is reset to end-of-file atomically before each write, but does not itself serialize the content across processes without kernel-atomic append support | Rely on `O_APPEND`'s atomic seek-to-end guarantee for interleaving-safety of small writes; use locks for larger multi-write transactions |
| Assuming advisory locks stop concurrent writers | `flock`/`fcntl` locks are opt-in: a process that never checks the lock can still `write()` freely | Use mandatory locking where the OS supports it, or ensure every accessing program honors the advisory protocol |
| Leaking descriptors across `exec()` | By default, descriptors survive `exec` unless marked `close-on-exec` | Set `FD_CLOEXEC` (or the equivalent flag) on descriptors that a child program should not inherit |
Pioneers & history
Origins · 1 minDennis Ritchie and how this was discovered
Ritchie, together with Ken Thompson, designed the Unix file model in which `open`, `read`, `write`, `close`, and small-integer file descriptors form the entire interface to files, devices, and (later) pipes and sockets alike — "everything is a file." This uniform interface, radically simple for its era, is the direct ancestor of the descriptor/table design described in this article and remains essentially unchanged in Linux, BSD, and POSIX-compliant systems today.
Where this matters
Every Unix-like kernel (Linux, BSD, macOS/XNU) and every language runtime built on top of them implements this interface almost verbatim: Python's `open()`, Java's `FileInputStream`, and Node.js's `fs` module all ultimately call `open`/`read`/`write`/`close` with the same descriptor and offset semantics described here. Shell pipelines (`cmd1 | cmd2`) and I/O redirection (`>`, `>>`, `2>&1`) are implemented entirely through `dup`/`dup2` manipulation of the fd table, which is why understanding descriptor sharing explains behavior that otherwise looks like magic. Database systems and mail servers rely on `fcntl` byte-range locks or equivalent mechanisms to implement safe concurrent record updates without redesigning their own locking protocol from scratch. Network servers historically multiplexed thousands of client connections by treating sockets as file descriptors and using `select`/`poll`/`epoll` over the same table structure, a design choice that traces directly back to "everything is a file." Container runtimes and sandboxing systems manipulate the fd table (close-on-exec flags, descriptor passing over Unix domain sockets) as a primary tool for controlling what resources a process can reach, making this interface a load-bearing part of modern security architecture, not merely a historical curiosity.
Ask the tutor: How would this three-level table design need to change to support memory-mapped files (`mmap`) sharing state with regular `read`/`write` access to the same file?
What exactly is stored in a file descriptor, versus what is stored in the open-file table entry?
The file descriptor itself is just a small integer — an index into the calling process's private table. It stores nothing about the file. The corresponding open-file table entry, which the descriptor points to, holds the actual mutable state: the current byte offset, the access-mode flags established at `open()` time, and a pointer to the in-memory inode carrying file-system metadata like size and permissions. This separation is why two descriptors can point at the same underlying file yet behave differently (independent offsets) or identically (shared offsets), depending entirely on whether they trace back to the same `open()` call.
Why does `fork()` cause the parent and child to share a file offset, but two separate `open()` calls on the same path do not?
`fork()` duplicates the parent's file descriptor table, so each child descriptor is a copy that still points at the same open-file table entries the parent was using — sharing is inherited, not recreated. A fresh `open()` call, by contrast, always allocates a brand-new open-file table entry with offset zero (or end-of-file for `O_APPEND`), regardless of whether some other process already has the file open. The rule is simple: sharing tracks descriptor ancestry (fork/dup/dup2), never filename equality.
What happens if a process tries to `write()` using a descriptor opened with `O_RDONLY`?
The kernel checks the cached status flags in the open-file table entry before performing the transfer and immediately returns an error (`EBADF`, "bad file descriptor," on POSIX systems) without touching any data. This check is cheap precisely because the access mode was validated once at `open()` time and stored, rather than requiring a full permission re-evaluation against the file's owner/group/mode bits on every single `read`/`write` call.
Why can `lseek` move the offset past the current end of the file, and what happens if you then write there?
POSIX explicitly permits seeking beyond end-of-file because the offset is just a number tracked in the open-file table entry — no data structure prevents it from exceeding the file's current size. If a subsequent `write()` occurs at that offset, the file's size grows to cover it, and the gap in between is defined to read back as zero bytes. Many file systems implement this gap as a genuine hole, consuming no physical disk blocks, producing what is called a sparse file.
Is a file lock the same thing as an access-mode restriction from `open`?
No — they operate at different layers and serve different purposes. The access mode (`O_RDONLY`/`O_WRONLY`/`O_RDWR`) is checked once, fixes what operations a specific descriptor may perform, and is enforced unconditionally by the kernel on every `read`/`write` call. A lock (`flock` or `fcntl`) is a separate, optional protocol that processes voluntarily consult to coordinate concurrent access; it says nothing about whether a call is permitted, only whether some other cooperating process currently claims exclusive or shared use of the file or a byte range within it.
Why does closing a file sometimes trigger a flush, and why is that tied to a reference count rather than to `close()` itself?
Because a single open-file table entry can be referenced by multiple descriptors (via `dup` or `fork`), closing any one of them should not discard buffered data that another descriptor still relies on. The kernel decrements a reference count on `close()` and performs cleanup — flushing dirty buffers, releasing the in-memory inode reference — only when that count reaches zero, i.e., when the very last descriptor referring to that open-file entry has been closed. This mirrors the same invariant that motivates the three-level table structure in the first place: sharing state must be released only when no one still depends on it.
How does this interface relate to what comes next — directories and file-system implementation?
`open()` needs a way to turn a human-readable pathname into a concrete file, which is exactly the problem Directory Structure solves: mapping names to inodes (or equivalent metadata records) through a hierarchy of directory entries. Once a file is open, `read`/`write` need to know how the file's logical bytes actually map onto physical blocks on a storage device, allocation strategies, and caching — the subject of File-System Implementation and Layout. This article deliberately treats both as opaque services, so its own indirection layer (descriptors, tables, offsets) stays visible and easy to reason about on its own terms.