Problems with encoding mismatches and BOM

1. Why are encodings so messy?

You already know text files are just sequences of bytes. And C# (and .NET in general) is a platform that expects characters to be where they belong. You'd think specifying the encoding when reading or writing is enough — and usually it is. But real life is messier.

Reasons for encoding confusion:

• Historical baggage: Files can be created on different OSes and editors, each picking its own default encoding.
• Cross-platform issues: A file made on Windows might be opened on Linux or Mac, where the defaults are different.
• BOM (Byte Order Mark): A special “header” that sometimes exists and sometimes doesn't. It affects how programs interpret the file.

2. What is BOM and why does it exist

Quick intro to BOM

BOM (Byte Order Mark) is a special byte sequence at the start of a file that tells the program: "Hey! I'm in this encoding, read my bytes like this."

• BOM is most often found in files encoded as UTF-8, UTF-16, and UTF-32.
• In UTF-8 the BOM is optional. Its presence or absence can change how different programs read the file.

EF BB BF

239 187 191

FF FE

255 254

FE FF

254 255

FF FE 00 00

255 254 0 0

00 00 FE FF

0 0 254 255

Fun fact from the encoding courtroom: In ASCII and ANSI encodings the BOM is not used. If it shows up there, it will surprise old programs.

Illustration: where the BOM sits

+--------------------------+
| BOM |     TEXT BYTES     |
+--------------------------+
|EFBBBF|  48 65 6C 6C 6F   |  // "Hello" in UTF-8 with BOM
+--------------------------+

EF BB BF is the BOM for UTF-8. It's at the very start of the file.
48 65 6C 6C 6F are the normal bytes for the text "Hello" in UTF-8 (without the BOM they look exactly the same).
So: the file starts with EF BB BF, then the text follows.

3. Encoding mismatch: where mojibake comes from

Typical scenario

1. You write a file in UTF-8 but without a BOM.
2. You open it in a Windows editor that expected Windows-1251 or UTF-8 with a BOM.
3. Result — instead of "Hello, world!" you see mojibake like "Привет, РјРёСЂ!".

Why this happens

• The program thinks the file is in one encoding, but the bytes are actually in another.
• BOM helps guess the encoding. But if the BOM is missing, guessing becomes a game of trial-and-error. The result — mojibake.

Examples of mismatch scenarios:

• You read a UTF-8 file as Windows-1251 — all non-ASCII chars turn into garbage.
• You read a UTF-8 file with a BOM as "plain" UTF-8 — often it's fine, but some old programs will render the first characters wrong.
• You write a file with a BOM but the consumer expects none — external software that doesn't like BOM may choke.

4. How encoding and BOM affect stream I/O

Example: writing and reading files with different encodings

// Write file in UTF-8 with BOM
using var writer = new StreamWriter("test_utf8_bom.txt", false, new UTF8Encoding(true));
writer.WriteLine("Hello, world!");

new UTF8Encoding(true) — enables the BOM.

// Write file in UTF-8 without BOM
using var writer = new StreamWriter("test_utf8_no_bom.txt", false, new UTF8Encoding(false));
writer.WriteLine("Hello, world!");

new UTF8Encoding(false) — no BOM.

// Read file with explicit encoding
using var reader = new StreamReader("test_utf8_no_bom.txt", new UTF8Encoding(false));
string line = reader.ReadLine();
Console.WriteLine(line);

If the file is UTF-8 without a BOM, explicitly specifying the encoding guarantees correct reading.

Typical mistake

When you don't specify an encoding while reading, StreamReader will try to guess — it checks for a BOM first; if there's none — it falls back to the system default (on Windows that's often Windows-1251 for Russian locales, UTF-8 on Linux/Mac).

5. What to do when encodings don't match

You see mojibake. Steps to take:

1. Check which encoding the file was created with.
   Open the file in an editor that shows encoding (for example, Notepad++).
2. Specify the encoding explicitly when reading/writing.
   Don't trust defaults, even if it seemed to "always work":

using var reader = new StreamReader("data.txt", Encoding.UTF8);

Mind the BOM.

• If other software requires a BOM — add it (see new UTF8Encoding(true)).
• If it doesn't — write without the BOM (see new UTF8Encoding(false)).

Example: wrong encoding when reading

// File was created in UTF-8, reading as Windows-1251
using var reader = new StreamReader("test_utf8_no_bom.txt", Encoding.GetEncoding(1251));
var text = reader.ReadToEnd();
Console.WriteLine(text); // "Hello, world!" will be corrupted

6. Helpful nuances

BOM in real projects and interviews

If you work on a project that saves configs, logs, or exports data, decide on an encoding up front and lock it in docs. It might look like a small detail, but when other programs or people start reading those files, encoding mismatches cause a ton of issues.

On interviews, the topic of BOM and "weird characters at the start of a file" is almost a guaranteed question. It's important not only to know what a BOM is, but to explain why its presence (or absence) matters for integrations and how that ties to explicit encoding.

Be extra careful when exchanging files with external systems on different OSes or implemented in different languages. In some places BOM is required, in others it breaks parsing. If you don't plan for it, these issues are hard to track down.

Recommendations and best practices

• Always specify the encoding when working with files (never rely on "defaults").
• If you need a BOM — use new UTF8Encoding(true), if not — new UTF8Encoding(false).
• Check files in editors that support multiple encodings (for example Notepad++, Visual Studio Code).
• If you receive a file from outside — ask colleagues or check documentation which encoding was used.
• If you need to convert encoding or remove a BOM — do it explicitly.

What different programs see

UTF-8 with BOM

UTF-8 + BOM

UTF-8

UTF-8 without BOM

UTF-8

Windows-1251

Win-1251

Win-1251

UTF-8

UTF-8 with BOM

UTF-8 + BOM

ASCII

7. How to check for and remove BOM

Example: check for BOM

byte[] bytes = File.ReadAllBytes("test_utf8_bom.txt");
// Check the first 3 bytes
if (bytes.Length >= 3 && bytes[0] == 0xEF && bytes[1] == 0xBB && bytes[2] == 0xBF)
{
    Console.WriteLine("BOM found! This is UTF-8 with BOM.");
}
else
{
    Console.WriteLine("BOM is absent.");
}

Example: remove BOM (if it's getting in the way)

if (bytes.Length >= 3 && bytes[0] == 0xEF && bytes[1] == 0xBB && bytes[2] == 0xBF)
{
    // Write file without the first 3 bytes
    File.WriteAllBytes("no_bom.txt", bytes.Skip(3).ToArray());
}

8. Common mistakes and how to fix them

Beginners often get surprised when their program starts "misbehaving" while working with text files. Usually the issue is that the writer used one encoding (or BOM mode) and the reader used another. For example, if a file was written in UTF-8 without a BOM, and you read it on Windows where the system default encoding is Windows-1251, it's no wonder characters turn into garbage. It's crucial to always specify the encoding explicitly. If a file is meant to be shared across programs or platforms, use a universal format — UTF-8 (or UTF-8 with BOM if external software requires it).