Optimizing work with large files

1. Introduction

When you work with small files, you often don't notice anything: write, read — and forget. But once file size starts exceeding at least 100–500 MB, and especially gigabytes, interesting effects pop up:

• Operations get slower, especially if you work "straight on" — e.g. File.ReadAllBytes() or File.WriteAllText() can suddenly slow down the whole program.
• You may run out of RAM, causing OutOfMemoryException.
• The system may start using swap, which can slow down the entire machine.
• Parallel operations can create excessive load on the disk.

In real-world tasks this happens often:

• Server logs (gigabytes per day).
• Processing large CSV or XML files for export/import.
• Working with video, audio, archives, binary files.
• Copying large assemblies or backups.

2. Optimization strategies when working with large files

Before rushing to optimize — let's decide what exactly we want to speed up or improve. Here are the most common goals:

• "Read/write the file as fast as possible without killing the system."
• "Process the file in chunks so we don't overload memory."
• "Avoid creating unnecessary copies of data in memory."
• "Process large files in parallel (when possible)."

General approaches:

• Streaming read and write: use streams and buffers, read/write in chunks (FileStream, BufferedStream).
• Operate on files directly on disk — without intermediate copies in memory.
• Carefully manage buffers and memory: don't keep the whole file in RAM.
• Asynchronous operations — if you need to avoid blocking the main thread (more in the next lecture).

3. Streaming read and write: basic pattern

Main principle: Work with files in parts! In modern C# this is easy with classes like FileStream, BufferedStream, or generally any stream-based approach.

// Open stream for reading:
using FileStream fs = new FileStream("bigfile.bin", FileMode.Open, FileAccess.Read);
byte[] buffer = new byte[1024 * 1024]; // 1 MB
int bytesRead;

// Read until end of file
while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0)
{
    // Handle the read data here!
    // For example, compute the sum of all bytes (just as an exercise)
    long sum = 0;
    for (int i = 0; i < bytesRead; i++)
        sum += buffer[i];

    Console.WriteLine($"Read {bytesRead} bytes, sum: {sum}");
}

Tip: Choose buffer size (e.g. 64 KB, 128 KB, 1 MB) experimentally. Too small — many disk calls. Too large — not always better and uses more memory.

Why? Classic reading with File.ReadAllBytes() or File.ReadAllText() without chunking tries to load the entire content into memory. If the file is huge — the result is obvious: OutOfMemoryException is likely.

4. BufferedStream: why and when

We met this class in the previous lecture, but a reminder: BufferedStream is a wrapper over any other stream that lets you read/write not one byte at a time but in blocks.

Usage example:

using var fileStream = new FileStream("bigfile.bin", FileMode.Open, FileAccess.Read);
using var bufferedStream = new BufferedStream(fileStream, 1024 * 128);

byte[] buffer = new byte[1024 * 128];
int bytesRead;
while ((bytesRead = bufferedStream.Read(buffer, 0, buffer.Length)) > 0)
{
    // Data processing
}

In some cases using BufferedStream gives a speedup, especially if you read a few bytes at a time and the file system prefers block access.

Interesting fact: Newer .NET versions have buffering inside FileStream, so manual use of BufferedStream gives the most benefit with "raw" streams, e.g. network or nonstandard devices.

5. Reading and writing large text files

Binary files are straightforward: read and write in chunks. But what about text files, especially large CSVs, logs, JSON files?

Here StreamReader for reading and StreamWriter for writing come to the rescue.

Line-by-line reading:

using var reader = new StreamReader("biglog.txt");
string? line;
while ((line = reader.ReadLine()) != null)
{
    // Process the line
    if (line.Contains("ERROR"))
        Console.WriteLine("Detected ERROR: " + line);
}

Why this is good?

• We don't keep the whole file in memory.
• Buffering inside StreamReader is already tuned well.

Writing line by line:

using var writer = new StreamWriter("output.txt");
for (int i = 0; i < 1000000; i++)
    writer.WriteLine($"This is line number {i}");

Streaming read architecture

   [File on disk]
          |
     [FileStream]
          |
  [BufferedStream (optional)]
          |
   [StreamReader/StreamWriter (for text)]
          |
   [Your code: data processing]

6. Applying in practice

Let's continue building our log-processing app. Suppose archiving old logs (older than 7 days) is done not by moving files but compressing them into one archive. If a file is huge — we'll read and write it in chunks so as not to "crash" memory.

For simplicity use standard copying via streams:

void CopyLargeFile(string sourcePath, string destPath)
{
    using var sourceStream = new FileStream(sourcePath, FileMode.Open, FileAccess.Read);
    using var destStream = new FileStream(destPath, FileMode.Create, FileAccess.Write);
    byte[] buffer = new byte[1024 * 256]; // 256 KB
    int bytesRead;
    while ((bytesRead = sourceStream.Read(buffer, 0, buffer.Length)) > 0)
    {
        destStream.Write(buffer, 0, bytesRead);
        // We can add a progress bar here!
    }
}

Where this is used?

• Copying backups
• Merging logs by day or month
• Preprocessing files (e.g. filtering lines)

7. How to measure buffer effectiveness?

Sometimes you want to know: "How much faster did it get?" It's easy to check by timing operations:

var watch = System.Diagnostics.Stopwatch.StartNew();
CopyLargeFile("source.bin", "dest.bin");
watch.Stop();
Console.WriteLine($"Copy time: {watch.Elapsed.TotalSeconds} seconds");

Typical buffer sizes:

• 4 KB — minimal filesystem block.
• 64 KB/128 KB — works well in practice almost always.
• 1 MB and above — justified only on very fast SSDs and very large files.

Try different buffer sizes yourself!

8. Searching and processing data in large files

What if you need not just copy but search for a certain line, number, phrase? In large files it's most efficient to do this in chunks.

Example: find all lines with errors in a huge log

using var reader = new StreamReader("server.log");
using var writer = new StreamWriter("errors.txt");
string? line;
while ((line = reader.ReadLine()) != null)
{
    if (line.Contains("ERROR"))
        writer.WriteLine(line); // Only needed lines go to the result
}

This approach lets you process gigabyte-scale logs without wasting lots of memory.

9. Peculiarities of working with huge files (>2 GB)

.NET and Windows handle files of any size well (even tens of terabytes) if you use streaming. But there are nuances!

• 32-bit apps are limited to 2 GB of addressable memory — use x64!
• For large files always target 64-bit platform (AnyCPU or x64).
• For files larger than 4 GB the FAT32 filesystem won't work — use NTFS/exFAT.

Iterative processing of large files

+------------------+
|    Start         |
+------------------+
        |
        v
+------------------------------+
| Open stream for reading      |
+------------------------------+
        |
        v
+------------------------------+
| While not end of file        |
+------------------------------+
        |
        v
+---------------------------+
| Read a block of data      |
+---------------------------+
        |
        v
+---------------------------+
| Process the block         |
+---------------------------+
        |
        v
+--------------------------+
| Next block               |
+--------------------------+
        |
        v
+--------------------+
| Close stream       |
+--------------------+

10. Useful nuances

File as a stream: key FileStream methods

Read

Write

Seek

Length

Position

Example of using Seek():

using var stream = new FileStream("bigfile.bin", FileMode.Open);
// Move 1 GB forward!
stream.Seek(1024L * 1024 * 1024, SeekOrigin.Begin);

byte[] buffer = new byte[1024];
int bytesRead = stream.Read(buffer, 0, buffer.Length);
// Now reading data from the middle of the file!

Where useful?

• File indexing
• Fast access to needed blocks (e.g. in large databases)

Working with files in multiple threads (multi-threading)

If the task allows, large files can be processed in parallel: for example, split into blocks and read/write different chunks concurrently. But understand that HDDs perform poorly with random access, while SSDs are much faster.

In simple home tasks, if unsure, work sequentially. Parallelism helps when processing different files at once, not necessarily for a single file (otherwise the gain is questionable).

11. Common mistakes when working with large files

Mistake #1: reading the whole file into memory.
Beginners often use File.ReadAllBytes() or File.ReadAllText() even for huge files. If the file is gigabytes, the app will just crash with an out-of-memory error. Use streaming.

Mistake #2: using too small a buffer.
Reading with a tiny buffer is like drinking soup with a teaspoon. The program will spend a lot of time on disk calls and just wait instead of doing work. Pick a reasonable buffer size.

Mistake #3: forgetting to close the stream.
If you don't close a stream after work, the file descriptor remains occupied. This prevents other programs and can cause OS-level errors. Always use using — it's safer and cleaner.

Mistake #4: simultaneous access to the same file.
Trying to read and write the same file from different parts of the program is a sure way to get IOException. Even if it sometimes "works", you won't get stability.

2

Task

C# SELF, level 41, lesson 4

Locked

Copying a Large File

Copying a Large File

Open

1

Survey/quiz

Data Buffering Principle, level 41, lesson 4

Unavailable

Data Buffering Principle

I/O Optimization

Open