Working with large files: chunking, memory mapping

1. Chunking — reading a file in chunks

As we already discussed in the previous lecture, chunking lets you work with files piece by piece without loading them entirely into memory. This is especially important when dealing with large volumes of data. If a file is 10 MB, it’s usually fine — you can load it and work with it any way you like. But what if the file reaches 10 GB, you only have 8 GB of RAM, and you also have a browser with dozens of tabs and an IDE open? Attempting to read such a file in full usually ends badly: OutOfMemoryError, a frozen program, and developer tears.

Real examples of such large files appear constantly: server logs for a month can take up tens of gigabytes, large CSV files contain millions of lines, and videos, archives, and database dumps are even larger.

The main idea remains the same: don’t try to “eat the elephant whole,” but work in pieces. It is exactly chunking that allows you to process such data safely and efficiently by splitting the file into manageable parts.

A quick refresher on chunking

Chunk (a piece/block) is simply a part of a file of a certain size. Instead of reading everything at once, we read, for example, 4 MB (or 64 KB, or 1 MB — depending on the situation).

Principle:

• Open a stream to read the file.
• Create a buffer — a fixed-size byte array.
• In a loop, read from the file into the buffer until you reach the end.
• Process each “piece” separately.

Example: copying a large file in chunks

Suppose we have a huge file that needs to be copied. Let’s write a program that does it the proper way.

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;

public class BigFileCopy {
    public static void main(String[] args) throws IOException {
        String source = "bigfile.dat";
        String dest = "bigfile_copy.dat";
        int bufferSize = 4 * 1024 * 1024; // 4 MB

        try (FileInputStream in = new FileInputStream(source);
             FileOutputStream out = new FileOutputStream(dest)) {

            byte[] buffer = new byte[bufferSize];
            int bytesRead;
            while ((bytesRead = in.read(buffer)) != -1) {
                out.write(buffer, 0, bytesRead);
                // You can add progress output or data processing
            }
        }
        System.out.println("Copy complete!");
    }
}

To work with files in Java, you typically use the standard streams FileInputStream and FileOutputStream. A good practice is to use a buffer of about 4 MB — that’s enough for modern disks to read and write efficiently. In the loop, the program reads chunks of the file and immediately writes them to a new file, without trying to keep the entire file in memory.

This approach saves RAM, avoids errors like OutOfMemoryError, and works with files of virtually any size, even 100 GB and more.

2. Chunking for data processing

Often the task is not just to copy a file but, for example, to find a specific line, count occurrences, replace something, etc.

Example: searching for a line in a large text file

If the file is text, it’s more convenient to use character streams and read line by line:

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;

public class BigFileSearch {
    public static void main(String[] args) throws IOException {
        String file = "biglog.txt";
        String keyword = "ERROR";
        int count = 0;

        try (BufferedReader reader = new BufferedReader(new FileReader(file))) {
            String line;
            while ((line = reader.readLine()) != null) {
                if (line.contains(keyword)) {
                    count++;
                }
            }
        }
        System.out.println("Found " + count + " lines with ERROR");
    }
}

Why does this work even for gigabyte-size files?

• BufferedReader reads the file in chunks (the default buffer is 8 KB, but you can set it larger).
• At any moment, only a single line is stored in memory.

Buffer size: which one to choose?

Golden rule: a buffer that’s too small means many disk accesses; too large — wasted memory.

• For modern HDDs/SSDs, a buffer of 64 KB–4 MB usually works well.
• For network or very fast SSDs — you can go larger (8–16 MB).
• For text files — you can increase the buffer in BufferedReader.

Experiment! Measure your program’s runtime with different buffers. Sometimes increasing the buffer gives a 2–3× speed-up; sometimes it barely matters.

3. Memory-mapped files (mapping a file into memory)

What is it, anyway?

Memory mapping is a way to “map” a file directly into a process’s memory using operating system mechanisms. In Java, this is done via the MappedByteBuffer class from the java.nio package. The file effectively becomes a large byte array that you can work with directly, without explicitly reading and writing each piece.

This approach is especially useful for working with very large files. The operating system itself loads the required parts of the file into memory, and you can access any place in the file as if it were a regular array. Memory-mapped files provide high-speed random access. For example, when you need to quickly read chunks from different places in the file without loading it entirely.

What does it look like in code?

import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;

public class MemoryMappedRead {
    public static void main(String[] args) throws Exception {
        String fileName = "bigfile.dat";
        try (RandomAccessFile file = new RandomAccessFile(fileName, "r");
             FileChannel channel = file.getChannel()) {

            long fileSize = channel.size();
            int chunkSize = 1024 * 1024 * 128; // 128 MB — size of one mapping

            long position = 0;
            while (position < fileSize) {
                long size = Math.min(chunkSize, fileSize - position);
                MappedByteBuffer buffer = channel.map(FileChannel.MapMode.READ_ONLY, position, size);

                // Read data from buffer as from an array
                for (int i = 0; i < size; i++) {
                    byte b = buffer.get(i);
                    // Process the byte (for example, look for a specific value)
                }

                position += size;
            }
        }
        System.out.println("Reading via memory mapping completed!");
    }
}

RandomAccessFile and FileChannel provide low-level access to the file. The call to channel.map maps a region of the file into memory. Access to the data is through the MappedByteBuffer buffer.

What are the advantages of memory mapping?

• Very fast for random access to different parts of a file.
• You can work with files larger than available RAM (the OS loads needed pages itself).
• Used in modern databases, indexes, large logs.

What are the drawbacks?

• Not always suitable for writing (especially on networked file systems).
• Mapping size limits (typically up to 2 GB per mapping on 32-bit JVMs).
• If you forget to close the file, it can become “stuck”/locked (especially on Windows).
• Not all file operations are accelerated — if you just need to read sequentially, a regular buffer often keeps up.

4. Practical examples

Example 1: Searching for a substring in a large file via memory mapping

Suppose we have a 10 GB file and we want to find a specific byte sequence in it (for example, the string "SECRET").

import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;

public class MemoryMappedSearch {
    public static void main(String[] args) throws Exception {
        String fileName = "hugefile.bin";
        byte[] target = "SECRET".getBytes(StandardCharsets.UTF_8);

        try (RandomAccessFile file = new RandomAccessFile(fileName, "r");
             FileChannel channel = file.getChannel()) {

            long fileSize = channel.size();
            int chunkSize = 128 * 1024 * 1024; // 128 MB

            long position = 0;
            while (position < fileSize) {
                long size = Math.min(chunkSize, fileSize - position);
                MappedByteBuffer buffer = channel.map(FileChannel.MapMode.READ_ONLY, position, size);

                for (int i = 0; i < size - target.length; i++) {
                    boolean found = true;
                    for (int j = 0; j < target.length; j++) {
                        if (buffer.get(i + j) != target[j]) {
                            found = false;
                            break;
                        }
                    }
                    if (found) {
                        System.out.println("Found at position " + (position + i));
                        // You can stop the search or continue
                    }
                }
                position += size;
            }
        }
    }
}

Note:
If the substring can “split” across two chunks, you need to add overlap between chunks equal to the length of the target sequence.

5. Useful details

When to use chunking, and when to use memory mapping?

• Chunking is a universal approach for any files (text, binary, logs, archives). Works well for sequential processing.
• Memory mapping is super-efficient for random access, working with large indexes and databases, and fast searches across huge files.

If you’re not sure what to choose — start with chunking! Memory mapping is a powerful but more low-level tool that requires care.

Recommendations

• Use try-with-resources to automatically close streams and channels.
• Don’t open too many files at once: the OS has limits on the number of open descriptors.
• Don’t map overly large regions — this can lead to errors (especially on 32-bit JVMs).
• For parallel processing, you can split the file into chunks and process them in separate threads (but be careful not to saturate the disk or exceed memory limits).

6. Common mistakes when working with large files

Mistake #1: trying to load the entire large file into memory.
A very common problem — especially for beginners. If the file is larger than 1–2 GB, use chunking or line-by-line reading; otherwise the program will crash with OutOfMemoryError.

Mistake #2: the buffer is too small.
A 512-byte buffer is not an optimization — it’s a recipe for terrible performance. Use buffers of 64 KB and above.

Mistake #3: forgot to close a stream or channel.
The file descriptor will remain open; the file may not be deleted or released until the JVM restarts. Use try-with-resources.

Mistake #4: incorrect use of memory mapping.
If the file is changed by another process during mapping, you can get inconsistent data or an error. Don’t use memory mapping for files that change frequently.

Mistake #5: not accounting for chunk overlap when searching for substrings.
If the target string can end up “at the boundary” of two chunks, be sure to add overlap equal to the string’s length between chunks.