Parallel FS traversals: Files.walk + parallel() and ForkJoin

1. Problem: how to efficiently process many files in a directory

In modern applications there’s often a task: process a large number of files in a folder and its subdirectories. For example:

• Count the total number of lines in all project ".java" files.
• Find all files modified in the last month.
• Copy or delete files based on a certain criterion.

If there are few files, a regular loop is enough. But with thousands and tens of thousands, especially when each file requires a "heavy" operation (reading, parsing, analysis), the time grows significantly.

Question: how to speed up processing a large number of files?
Answer: use parallelism — process files concurrently across multiple threads.

2. Tools for traversing the file system

Files.walk()

Starting with Java 8, there’s a convenient way to traverse a directory tree — the Files.walk() method from the java.nio.file package. It returns a Stream<Path> — all files and directories starting from the specified directory.

Example:

import java.nio.file.*;
import java.util.stream.Stream;

Path start = Paths.get("src");
try (Stream<Path> stream = Files.walk(start)) {
    stream.forEach(System.out::println);
}

• Files.walk(start) — returns a stream of all files and directories, including subdirectories.
• You can set a maximum traversal depth: Files.walk(start, 3).

Files.find()

If you need to filter right away (for example, only ".java" files), use Files.find():

import java.nio.file.*;
import java.util.stream.Stream;

Path start = Paths.get("src");

try (Stream<Path> stream = Files.find(
        start,
        Integer.MAX_VALUE,
        (path, attr) -> path.toString().endsWith(".java"))) {
    stream.forEach(System.out::println);
}

• Files.find() takes a filter (BiPredicate<Path, BasicFileAttributes>) that receives the path and file attributes.

3. Parallel processing: parallel() and ForkJoinPool

Parallel streams: .parallel()

Any Stream has a parallel() method. If you call it, element processing will proceed in multiple threads.

Files.walk(start)
    .parallel()
    .forEach(path -> processFile(path));

Each file will be processed in parallel (where possible), which is especially effective for "heavy" operations: reading, parsing, computations.

How does it work under the hood? ForkJoinPool

Parallel streams use a common thread pool — ForkJoinPool.commonPool(). This is a "smart" pool that distributes tasks among threads.

• By default, the number of threads equals the number of available processors: Runtime.getRuntime().availableProcessors().
• The fork/join model fits independent tasks well — such as processing individual files.

When to use .parallel()?

• When processing of each file is independent of others.
• When the operation is "heavy" (CPU-intensive or spends time waiting on IO).
• When there are many files (hundreds, thousands).

Avoid using parallel streams:

• If there are few files (overheads of parallelization may outweigh the benefits).
• If strict ordering is required or there are dependencies between elements.

4. Alternatives and tuning parallelism

When is ExecutorService a better choice?

Parallel streams are good for simple cases. But if you need to:

• Control the exact number of threads (for IO-bound tasks it’s often beneficial to have more threads than cores).
• Manage queues, cancellation, retries, error handling.
• Build more complex task pipelines.

Then use ExecutorService:

import java.nio.file.*;
import java.util.concurrent.*;

ExecutorService executor = Executors.newFixedThreadPool(8);
Files.walk(start)
    .filter(Files::isRegularFile)
    .forEach(path -> executor.submit(() -> processFile(path)));
executor.shutdown();

Tuning ForkJoinPool

By default, the common pool uses the number of threads equal to the number of processors. You can change this via a system property (before the first use of parallel streams):

System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "16");

• After this call, all parallel streams will use up to 16 threads.

CPU-bound vs IO-bound tasks

• CPU-bound: heavily load the CPU (math, parsing, compression). Number of threads ≈ number of cores.
• IO-bound: lots of disk/network waiting. It’s often beneficial to have more threads than cores.

Parallel streams are not always optimal for IO-bound tasks — a dedicated ExecutorService with an enlarged pool often wins.

5. Example: parallel file search and processing

Let’s count the total number of lines in all project ".java" files using a parallel traversal.

import java.nio.file.*;
import java.util.stream.*;
import java.io.IOException;

public class LineCounter {
    public static void main(String[] args) throws IOException {
        Path start = Paths.get("src");

        long totalLines = Files.walk(start)
            .parallel() // parallel processing!
            .filter(p -> p.toString().endsWith(".java"))
            .mapToLong(LineCounter::countLines)
            .sum();

        System.out.println("Total lines of code: " + totalLines);
    }

    // Method to count lines in a file
    private static long countLines(Path path) {
        try (Stream<String> lines = Files.lines(path)) {
            return lines.count();
        } catch (IOException e) {
            System.err.println("File read error: " + path);
            return 0;
        }
    }
}

What’s happening:

• Files.walk(start) — traverse all paths.
• parallel() — enable parallel processing.
• filter(...) — keep only ".java" files.
• mapToLong(...) — count the lines in each file.
• sum() — sum up the result.

Pros: multiple threads are engaged while the code remains concise.

6. Important nuances and common mistakes

• Not all tasks speed up with parallelism. For small sets of files or very fast operations, overhead can slow the program down.
• Close resources. When working with files, use try-with-resources so descriptors don’t leak. For example, Files.lines(path) in a try(...).
• Nested parallelism. Running parallel streams inside other parallel tasks (nested parallelism) is rarely effective and can degrade performance.
• Side effects. Avoid writing to shared structures/files without synchronization. Prefer pure operations on elements.

7. Diagram: how a parallel file traversal works

flowchart TD
    A["Files.walk(start)"] --> B["Stream<Path>"]
    B --> C{".parallel()?"}
    C -- No --> D[Regular forEach]
    C -- Yes --> E["Parallel forEach (ForkJoinPool)"]
    E --> F[Processing files across multiple threads]

8. Common mistakes in parallel file processing

Error #1: Using parallel streams for small tasks — overhead is higher than the gain.

Error #2: Expecting parallel streams to speed up IO-bound tasks the same way as CPU-bound. For IO you typically need an ExecutorService with a larger pool.

Error #3: Unhandled exceptions in lambdas — without handling IOException, the stream may break and the result may be incomplete.

Error #4: Races when writing to shared variables or files — synchronize access or avoid side effects.

Error #5: Forgetting to close resources — use try-with-resources for all file operations.

Error #6: Attempting to change ForkJoinPool.commonPool() after first use — set System.setProperty(...) early.

Error #7: Using parallel streams inside other parallel streams — often leads to performance degradation.