Specifying the encoding when reading and writing ( Encoding)

1. Default encoding

When you work with StreamReader or StreamWriter, they use a default encoding.

If you write:

using var reader = new StreamReader("myfile.txt");

or

using var writer = new StreamWriter("output.txt");

.NET picks the default encoding. On Windows this is usually UTF-8 (often without a BOM), but older apps or runs on other systems might behave differently.

In most cases UTF-8 is a good choice: it supports all languages and stores English compactly (and Russian pretty well too). But if a file was created in another encoding (for example, Windows-1251), or other programs expect a specific encoding, you should explicitly specify the correct one.

2. Specifying encoding for StreamReader and StreamWriter

Constructor options

Both StreamReader and StreamWriter have constructors that accept a System.Text.Encoding object.

Example:

using var reader = new StreamReader("myfile.txt", Encoding.UTF8);
using var writer = new StreamWriter("output.txt", false, Encoding.UTF8);

If you want to work with ASCII, Windows-1251 or UTF-16, just plug in a different encoding:

using var reader = new StreamReader("myfile.txt", Encoding.Unicode); // UTF-16
using var reader2 = new StreamReader("rus.txt", Encoding.GetEncoding("windows-1251"));

Quick table of common encodings:

Encoding.UTF8

Encoding.Unicode

Encoding.ASCII

Encoding.GetEncoding("windows-1251")

Heads up! For non-standard encodings like Windows-1251 use Encoding.GetEncoding("windows-1251").

3. Read and write with explicit encoding

Let's add a feature to our little app: it should read a file in a specified encoding and write the result to a new file with an explicit encoding. This is how you can convert text between different systems!

Example 1: Read a file in Windows-1251 and print to console

// Suppose the file was created in old Windows-1251
using var reader = new StreamReader("ru_text.txt", Encoding.GetEncoding("windows-1251"));

string line;
while ((line = reader.ReadLine()) != null)
{
    Console.WriteLine(line);
}

Why this matters: If we hadn't specified the encoding, instead of Russian letters we'd get horrible characters like "УчебнРиРє".

Example 2: Write a file in UTF-16 (Unicode)

using var writer = new StreamWriter("utf16text.txt", false, Encoding.Unicode);
writer.WriteLine("Hello, world!");
writer.WriteLine("Hello, world!");
writer.WriteLine("こんにちは世界");

Result: Opening the file, for example in Notepad, you'll see that all characters display correctly.

4. Small utility to convert encodings

A common interview task: Convert a text file from Windows-1251 to UTF-8.

string sourcePath = "ru_text_1251.txt";
string destPath = "ru_text_utf8.txt";

// Read in Windows-1251 encoding
using var reader = new StreamReader(sourcePath, Encoding.GetEncoding("windows-1251"));
// Write in UTF-8
using var writer = new StreamWriter(destPath, false, Encoding.UTF8);

string? line;
while ((line = reader.ReadLine()) != null)
    writer.WriteLine(line);

Now ru_text_utf8.txt can be opened in VSCode, Linux, Mac and you'll see readable text.

5. Useful tips

How to know what encoding a file uses?

This is a bit tricky! A plain file usually doesn't store its encoding (except files with a BOM, more on that in the next lecture). If you open a file with the wrong encoding, you'll get gibberish.

Typical practical approaches:

• If the file was created by your program — always use the same encoding to read that you used to write it.
• If the file came from an unknown source — try different encodings or use third-party tools to detect it.

Working with emoji and unusual characters

UTF-8 and UTF-16 let you freely use emoji and characters from many languages. Try saving a file with emoji in UTF-8, then open it in an old encoding — it can be funny (or not)!

Emoji code:

using var writer = new StreamWriter("emoji.txt", false, Encoding.UTF8);
writer.WriteLine("Hello 👋😀🌍");

StreamReader with auto-detect

The StreamReader constructor has an overload with the detectEncodingFromByteOrderMarks parameter:

new StreamReader(path, encoding, detectEncodingFromByteOrderMarks: true)

If this parameter is true (that's the default!), StreamReader will try to guess the file encoding based on the presence of a BOM (a special "marker" at the start of the file). But if the file was created without a BOM or uses an exotic encoding, the auto-detect won't help.

When and why to explicitly specify the encoding?

• Multilingual data: support for Arabic, Chinese, emoji, etc.
• Integrations: the file is read/written not only by your program but by another system that expects a specific encoding.
• File size matters: ASCII/UTF-8 are more compact for Latin text, UTF-16 can be better for ideographs.
• Compatibility: logs/exports for other applications.

6. Common mistakes when working with encodings

Mistake #1: not explicitly specifying encoding when reading a file.
Reading a file with Russian (or other non-Latin) text without specifying encoding often results in unreadable "gibberish".

Mistake #2: mismatch between encoding used to write and read the file.
The file was written in one encoding but opened in another — you get corrupted text. This often happens when moving files between OSes or programs.

Mistake #3: unsupported or partially supported encodings.
Some editors don't handle UTF-8 without a BOM well, or they choke on old single-byte encodings like Windows-1251. Sometimes you need to manually switch encodings and experiment.

2

Task

C# SELF, level 37, lesson 2

Locked

Writing Emoji Text to a File

Writing Emoji Text to a File

Open