1. Introduction
Why buffering is needed and a performance comparison
We already know that buffering is a strategy that lets you group data into bigger "chunks" so you don't hit the disk hundreds of thousands of times, but much less often, sending lots of data at once. That gives a significant performance boost, especially with large amounts of data.
When to think about I/O performance
- If you work with large files (gigabytes, terabytes — like that beloved collection of logs accumulated over your company's lifetime).
- If you need minimal latency (for example, real-time log processing).
- If the number of file accesses is huge (for example, bulk renaming/copying a photo archive).
- To show in an interview that you understand modern optimization approaches (and that you like measuring speed, not just "making it work somehow").
In .NET most file streams have buffering by default, but sometimes you need fine tuning or a special strategy.
Main "players" — what kinds of buffers exist
| Class | Default buffer | Can change size | Applicability |
|---|---|---|---|
|
Exists (4096 bytes) | Yes (via constructor) | Base file stream |
|
Yes (4096 bytes) | Yes (via constructor) | "Wrapper" around a stream |
|
Yes (1024/1024 bytes) | Yes (constructor) | Working with text |
- BufferedStream can "wrap" another stream to boost performance (for example, when the underlying stream has poor buffering or you need a bigger buffer).
- Buffer size is a trade-off between speed and memory usage.
2. Examples:
Let's compare three approaches for copying a large file:
- No buffering — one byte at a time (bad idea, but illustrative!)
- Default buffer — the standard FileStream with CopyTo
- Manual buffer control — we provide the buffer ourselves, optimize its size
For experiments let's create a simple file copy utility to add to our app. Let the file be named BigFile.bin.
class FileCopyBenchmarks
{
// Copying one byte at a time (anti-example — don't do this!)
public static void CopyOneByte(string source, string dest)
{
using var input = new FileStream(source, FileMode.Open, FileAccess.Read);
using var output = new FileStream(dest, FileMode.Create, FileAccess.Write);
int b;
while ((b = input.ReadByte()) != -1)
{
output.WriteByte((byte)b);
}
}
// Copying using FileStream's default buffer
public static void CopyWithDefaultBuffer(string source, string dest)
{
using var input = new FileStream(source, FileMode.Open, FileAccess.Read);
using var output = new FileStream(dest, FileMode.Create, FileAccess.Write);
input.CopyTo(output); // Uses internal buffer (typically 81920 bytes)
}
// Copying with manual buffer control
public static void CopyWithCustomBuffer(string source, string dest, int bufferSize = 1024 * 1024)
{
using var input = new FileStream(source, FileMode.Open, FileAccess.Read);
using var output = new FileStream(dest, FileMode.Create, FileAccess.Write);
byte[] buffer = new byte[bufferSize];
int bytesRead;
while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
}
Which function is faster? Let's measure execution time.
How to measure performance correctly
In .NET the easiest way to measure time is using Stopwatch:
static void Measure(Action action, string description)
{
var sw = Stopwatch.StartNew();
action();
sw.Stop();
Console.WriteLine($"{description}: {sw.ElapsedMilliseconds} ms");
}
Now let's try copying the same file in different ways:
string source = "BigFile.bin";
string dest1 = "copy1.bin";
string dest2 = "copy2.bin";
string dest3 = "copy3.bin";
// Create BigFile.bin in advance (for example, 100-500 MB) or use any large file.
Measure(() => FileCopyBenchmarks.CopyOneByte(source, dest1), "CopyOneByte (1 byte at a time)");
Measure(() => FileCopyBenchmarks.CopyWithDefaultBuffer(source, dest2), "CopyWithDefaultBuffer (default)");
Measure(() => FileCopyBenchmarks.CopyWithCustomBuffer(source, dest3, 1024 * 1024), "CopyWithCustomBuffer (1 MB)");
Dangerous points and pitfalls
- If you run it many times in a row, the OS cache can "warm up" the disk and later measurements will be faster — for a real assessment it's better to restart the program and clear the cache.
- If the file is small (10–20 KB), buffering benefits will be unnoticeable — the larger the file, the bigger the difference.
- If you set buffer size too large (for example, 100 MB), memory consumption can spike and the rest of the system will suffer.
Visualizing results: a table
| Method | Time (ms) for a 500 MB file |
|---|---|
| One byte at a time | 100,000+ |
| Standard FileStream / CopyTo | 1,000 — 5,000 |
| Manual buffer 1 MB | 700 — 1,200 |
Numbers are approximate, but the trend is clear — the bigger the buffer, the fewer disk accesses and the higher the speed.
3. Anatomy of manual buffer control
Why would you still want to tune buffer size sometimes? Here's a simple analogy: you're moving to a new apartment. You can carry stuff one cup at a time, or take a huge box. But the box has a limit too, otherwise you can't lift it!
How reading with a manual buffer works
// Example of manual buffer size control
int bufferSize = 1024 * 1024; // 1 MB
byte[] buffer = new byte[bufferSize];
int read;
while ((read = inputStream.Read(buffer, 0, buffer.Length)) > 0)
{
outputStream.Write(buffer, 0, read);
}
- The Read method tries to fill the whole buffer but may return less if the file ends.
- Buffer size is often chosen in the range from 32 KB to 4–8 MB — larger often gives little extra benefit.
- Don't forget about RAM, especially if there are many such operations or threads.
Experimenting with buffer size
Try changing the buffer size in our example (32 KB, 128 KB, 1 MB, 4 MB) and see where performance peaks. Usually the "sweet spot" is around 1 MB.
Scenarios where manual buffer is more useful
- When you need to control the amount of used memory (for example, the program runs on a weak server).
- When the stream isn't automatically buffered (NetworkStream, custom streams).
- When working with many parallel operations — you can allocate an optimal buffer for each thread.
- When you want to squeeze max speed out of processing a very large file (for example, converting a huge CSV file).
4. Best practices and common mistakes
Can you just make the buffer huge? Like, "there's tons of memory"? You can, but why? Too big a buffer can sometimes even reduce performance: some memory will sit unused and the system may start "lagging" due to caching issues.
Manual buffering doesn't speed up everything. For small files or already optimized streams (for example, FileStream with a large internal buffer) the gain will be minimal and the code becomes more complex.
Typical trap: forgetting to close a stream or not handling exceptions — the file will remain locked. Use using and error handling (try-catch) when working with files.
GO TO FULL VERSION