Streaming large files

1. When reading the whole file into String is no longer funny

If you are just starting out with programming, it feels natural: “I want text, so I read the file into String and work with it.” On small files, that is exactly what you should do. Problems begin when the file becomes large: tens of megabytes, hundreds of megabytes, a gigabyte. Then the strategy “read everything” can suddenly turn into the strategy “read everything, then the computer reads you.”

Let us start with an honest admission: reading a file into String is convenient. It is one line of code, easy to debug, easy to print the first characters, easy to use split. That is exactly why so many utilities in the world start with “let us just load everything into memory.” And that is exactly why so many utilities end with “why did it crash on the production file?”

The problem is not only the size of the text itself. The problem is that on top of the text you often create intermediate structures too: an array of lines, arrays of fields, dictionaries, temporary String values from Substring, and all of this can multiply memory usage many times over.

Why split can become very expensive

When we see multi-line text, our hand reaches for split: “I will split by lines, and then split each line again by \t.” On small data, that is perfectly fine. But it is important to remember: the standard split is an eager operation. It immediately creates a collection of parts instead of yielding them one by one on demand. For huge text, that means: you asked to look at the lines, but in fact you created an entire warehouse of lines in memory.

To feel this with your hands, a small thought experiment is enough. Imagine a file with 500 000 lines. Even if the average line is short, an array with 500 000 elements is already not “nothing.” And if each line then turns into an array of fields, you easily build a “matrix” of millions of text fragments.

A tiny demo on an example: we split text into lines. The code is innocent, but the meaning is important.

import Foundation

let text = """
a
b
c
"""

let lines = text.split(whereSeparator: { $0.isNewline })
print(lines.count) // 3

On three lines, it is perfect. On millions of lines, lines is no longer just a “result” but also a serious memory load, plus time spent creating all those pieces.

Here it is important not to get scared and start “being afraid of split.” split is an excellent tool. But once files become large, you must learn to ask yourself an adult question: “Do I really need an array of all lines at once, or is it enough to process them one by one?”

2. Line-by-line processing: enumerateLines without an array of lines

When you realize that an array of lines is expensive, you want something more “lazy,” where lines come one by one. And in Swift, there is a very pleasant compromise for text: String.enumerateLines. It does not create an array of all lines, but calls your closure for each line in turn. This is still working with text already loaded into memory, but at least you do not spawn an extra “bag of lines.”

Let us start simple: read the file into String, then go through the lines and count them. Yes, the file is still entirely in memory, but there is no array of lines anymore.

import Foundation

let url = URL(fileURLWithPath: "/tmp/big.txt")

do {
    let text = try String(contentsOf: url, encoding: .utf8)

    var linesCount = 0
    text.enumerateLines { _, _ in
        linesCount += 1
    }

    print("lines:", linesCount) // for example: lines: 1200
} catch {
    print("Read failed:", error)
}

Pay attention to the style: inside enumerateLines we try to do the minimum amount of work. This is an important habit: line-by-line processing is about the discipline of “do not drag everything with you.”

Another typical scenario is aggregating numbers from a file. Suppose the file contains one number per line, and we want the sum while ignoring garbage.

import Foundation

let url = URL(fileURLWithPath: "/tmp/numbers.txt")

do {
    let text = try String(contentsOf: url, encoding: .utf8)

    var sum = 0
    text.enumerateLines { line, _ in
        if let x = Int(line) {
            sum += x
        }
    }

    print("sum:", sum) // sum: 42
} catch {
    print("Read failed:", error)
}

This approach is already much more “grown-up” than split → map → map → …, because you are not multiplying collections. But there is a fundamental limitation here, which we will talk about a little lower: the source String still has to fit into memory entirely.

4. Importing TSV on the fly in LibraryCLI

To keep the topic from staying abstract, let us imagine a practical situation in our console application LibraryCLI. Suppose we were given a huge TSV export file: each line is a book, fields are separated by tabs. Our goal is to walk through the file, count valid lines, and gracefully skip broken ones without turning everything into a gigantic [[String]].

Let us make a small parser for one line. We deliberately keep the format simple: exactly 3 columns, with no quotes or escaping.

import Foundation

func parseBookTSVLine(_ line: String) -> (title: String, author: String, year: Int)? {
    let parts = line.split(separator: "\t", omittingEmptySubsequences: false)
    guard parts.count == 3 else { return nil }

    let title = parts[0].trimmingCharacters(in: .whitespaces)
    let author = parts[1].trimmingCharacters(in: .whitespaces)
    let year = Int(parts[2].trimmingCharacters(in: .whitespaces)) ?? 0

    return (title, author, year)
}

Now let us apply this line by line. Notice: we do not collect an array of all lines, and even an array of all books is not required either — we can write or update storage immediately. For simplicity, here we will count statistics.

import Foundation

let url = URL(fileURLWithPath: "/tmp/books.tsv")

do {
    let text = try String(contentsOf: url, encoding: .utf8)

    var ok = 0
    var bad = 0

    text.enumerateLines { line, _ in
        if parseBookTSVLine(line) != nil {
            ok += 1
        } else {
            bad += 1
        }
    }

    print("ok:", ok, "bad:", bad) // ok: 120 bad: 3
} catch {
    print("Import failed:", error)
}

If you want to add a bit of “humanity,” you can set a limit on the number of errors so that you do not print 50 000 messages to the console, but stop and say: “this file is clearly the wrong one.”

import Foundation

var bad = 0
let badLimit = 10

func registerBadLine() -> Bool {
    bad += 1
    return bad >= badLimit
}

This small detail saves nerves in practice: logging every error in a huge file is another way to stage a “DDoS from your own app.”

5. The limitation of enumerateLines: the whole text is still in memory

At this point, people usually feel victorious: “Hooray! I am processing lines one by one!” And that is true... half true. Because enumerateLines saves intermediate arrays, but it does not cancel the basic fact: to call enumerateLines, you have already created a String, which means you have already loaded the whole file into memory.

To see this clearly, it helps to separate two stages: “how we read the file” and “how we process the text.” enumerateLines is about processing text. And reading through String(contentsOf:encoding:) is reading “all at once.”

Here is a table that helps avoid confusion:

That is exactly why we are discussing this now as an idea and direction: the next step is real streaming reading, where the file does not have to fit into RAM entirely.

6. Streaming reading: read the file in Data chunks

Real streaming reading is when we take a file from disk not all at once, but in portions: for example, 64 KB at a time. It is like eating pizza: a normal person eats it in slices, not by trying to shove the entire box into their mouth.

The technical idea is simple: we read a chunk of bytes, add it to a buffer, look for line separators (\n), extract ready-made lines, and leave the “tail” (an unfinished line) in the buffer until the next chunk. This allows us to keep only a small working buffer in memory, not the whole file.

Pipeline diagram

flowchart TD
    A[Opened the file] --> B[Read N-byte chunk]
    B --> C[Added chunk to buffer]
    C --> D{Is there a newline?}
    D -- yes --> E[Extracted line]
    E --> F[Processed line]
    F --> D
    D -- no --> G{EOF?}
    G -- no --> B
    G -- yes --> H[Processed buffer remainder as the last line]

The key “brain” here is the remainder buffer. Without it, you either lose pieces of lines at chunk boundaries or start doing incredible tricks with “rewinding” the read backwards.

Counting lines without decoding UTF-8

Sometimes you do not need to turn bytes into String at all. For example, you want to quickly estimate the file size, count the number of lines, find the number of empty lines, or roughly count the number of records. In such tasks, you can work at the byte level and simply count \n (byte 0x0A). This is not broken by Unicode and requires almost no memory.

Here is a small (and fairly practical) example: count lines through FileHandle by reading chunks. This is already closer to real streaming reading.

import Foundation

func countLinesStreaming(url: URL) throws -> Int {
    let handle = try FileHandle(forReadingFrom: url)
    defer { try? handle.close() }

    var lines = 0
    while let chunk = try handle.read(upToCount: 64 * 1024), !chunk.isEmpty {
        for b in chunk where b == 0x0A { lines += 1 }
    }
    return lines
}

And usage:

import Foundation

do {
    let url = URL(fileURLWithPath: "/tmp/big.tsv")
    let n = try countLinesStreaming(url: url)
    print("lines:", n) // lines: 500000
} catch {
    print("Failed:", error)
}

Notice how “adult” the memory usage looks: we have only chunk (64 KB) and a few Int values. Against the background of a gigabyte file, that is almost free.

Of course, this counts exactly \n. If the file ends without \n at the end, the last line will not be counted. This is not a Swift bug; it is a matter of the format contract, and you need to handle it explicitly: either require a trailing newline or adjust the logic (for example, add + 1 when the remainder is non-empty).

Getting lines: remainder buffer and parsing by \n

When you need lines as text (for example, you are really importing books from TSV), you will have to not only count bytes but also decode them into String. Here there is a subtlety: UTF-8 is variable-length, and a character can be split at a chunk boundary. Therefore, a fully robust streaming decoder is a topic you can dive into deeply.

But right now, we care about the idea and the basic mental framework: “read a chunk → accumulate in a buffer → pull out complete lines → leave the remainder.” Below is exactly an outline of the algorithm. It shows the structure, but does not promise that tomorrow you will write a universal tail -f in Swift.

import Foundation

// Pseudocode/outline: shows the idea "buffer + split by \n".
var buffer = Data()

while let chunk = try handle.read(upToCount: 64 * 1024), !chunk.isEmpty {
    buffer.append(chunk)

    while let nl = buffer.firstIndex(of: 0x0A) { // '\n'
        let lineData = buffer[..<nl]
        buffer.removeSubrange(...nl) // including '\n'

        let line = String(data: lineData, encoding: .utf8) ?? ""
        processLine(line)
    }
}

if !buffer.isEmpty {
    let lastLine = String(data: buffer, encoding: .utf8) ?? ""
    processLine(lastLine)
}

There are two important conceptual points here. The first is that we look for the line separator at the byte level, which is simple. The second is that we decode only the ready line, not the whole file. Even if there are millions of lines, memory will hold at most the buffer and one line.

If you read carefully, you noticed a potential trap: “what if a UTF-8 character got broken at the chunk boundary and broken bytes ended up in lineData?” In practice, this is handled more carefully than ?? "": most often you either fail with a format error or do strict diagnostics. But as a starting mental model, this framework is useful: it shows exactly where stream processing is born.

How to choose an approach

After all these techniques, it is easy to swing to the opposite extreme: “From today on, I will read everything only in a streaming way!” This is a bit like deciding after the first workout that you will now always walk everywhere, even to another city. Sometimes it is really useful, but more often it is unnecessary.

The choice of approach is usually determined by two things: the size of the file and what you want to do with the data. If you need the whole text at once (for example, you really want to show the user the whole file), streaming reading will not give you magic, because the final result will still be a large object. If you need aggregation, filtering, or record import, streaming often gives you huge memory savings.

A practical rule for our CLI sounds like this: as long as you can honestly say “the file is expected to be small,” reading into String remains the simplest and most maintainable option. As soon as files from the outside world appear and they can be huge, it is better to switch either to line-by-line processing without arrays (enumerateLines) or to chunked reading when memory becomes critical.

A good developer differs from a beginner not by always choosing the “smartest” technique, but by choosing the technique that fits the task and does not turn the code into a museum of complicated solutions.

7. Typical mistakes

When you start optimizing file reading, it is easy to step on rakes that are not visible on small test data. The irony is that these rakes often look like “clean code,” just clean in the wrong place. Let us go through the most common mistakes I see in beginners (and sometimes in myself, to be honest).

Mistake #1: “I use enumerateLines, so I am reading in a streaming way.”
enumerateLines really does process lines one by one and saves the line array, but the source text is still already in memory entirely, because you first did String(contentsOf:). This is an excellent step toward saving memory, but it is not reading the file “in parts.”

Mistake #2: building a chain of split → map → split → map on large data “for beauty.”
Such code looks functional and modern, but it easily creates several large intermediate collections. On a small file, you will not notice anything. On a large one, you will get a memory spike or a sharp slowdown. Sometimes a “plain loop” with minimal allocations is even easier to read.

Mistake #3: using try? in a critical import and losing the cause.
In tasks like data import, “failed to read the file” is an important event. If you do let text = (try? String(...)) ?? "", you turn a serious error into an “empty file,” and then spend a long time figuring out why the import “just does nothing.” For imports, do/catch and a clear message to the user or the log are usually better.

Mistake #4: logging every broken line endlessly.
When the file is large and slightly broken, logging every problem turns into an even bigger problem: you flood the disk, the console, and the user's brain with millions of identical messages. It is better to count errors, show the first N examples, and then say “and 12 345 more like these.”

Mistake #5: forgetting about the last line without \n.
This is a classic boundary case: many files end with a newline, but not all of them do. If you count lines by \n, the last one may “fall out.” This is not a disaster, but it is the kind of case where a careful developer defines the contract explicitly: either “a trailing \n is required,” or “we process the remainder.”