What encoding is and why it matters

1. Introduction

Imagine you're a diplomat at an international summit: everyone speaks their own language and uses their own script. To understand each other you need a universal translator — a shared set of rules for mapping characters. In computers that job is done by encodings.

The computer understands only one "language" — sequences of zeros and ones. 0 and 1 are its "alphabet". All information is stored and transmitted as bytes. One byte is 8 bits (for example, 01000001).

So how do we link our letters and symbols to bytes? How does a computer know that the letter "A" is not just some combo of 0/1, but the actual character shown on the screen?

Encoding

An encoding is a set of rules (a mapping table) that defines how each character (letter, digit, punctuation, ideograph, emoji) is converted into a sequence of bytes and how those bytes are then interpreted back into characters.

Analogy — Morse code: you translate text into dots and dashes, send the message, and the receiver reconstructs the characters using the same rules. In computers the agreement "bytes ↔ characters" is the encoding.

2. So why do we deal with this encoding headache?

• Translation between human and machine worlds: without encoding text is just bytes; with encoding it becomes meaningful characters.
• Interoperability and compatibility: different programs and OSes need to agree on rules. If a file is declared as UTF-8, you should read it as UTF-8 too.
• Support for many languages and symbols: Cyrillic, Arabic script, ideographs, math symbols, emoji — the wider the character set, the more complex and flexible the encoding has to be.

3. ASCII — the "primitive" encoding

One of the oldest and most basic encodings is ASCII (American Standard Code for Information Interchange). It uses 7 bits per character, so it can represent 128 different symbols: the Latin alphabet (A-Z, a-z), digits (0-9), punctuation and control codes (for example, newline, tab).

A

65

1000001

B

66

1000010

a

97

1100001

b

98

1100010

0

48

0110000

1

49

0110001

!

33

0100001

32

0100000

Historically the eighth bit was often used as a parity bit, and later people started using it in "extensions" of ASCII — that's how different single-byte sets for locales appeared, leading to a lot of inconsistency.

If you write "Hello", on disk it would look roughly like this (1 byte per character; for 7-bit ASCII the high bit is zero):

H (01001000) e (01100101) l (01101100) l (01101100) o (01101111)

Simple. But where are Cyrillic letters or ideographs? They aren't in ASCII — it's a "single-language dictionary" fit only for basic Latin.

4. Garbled text — why encoding is no joke

Sometimes when you open a file you see something like Привет instead of "Privet". People call this garbled text (or mojibake) — the result of reading bytes with the wrong encoding.

Say you saved "Privet, mir!" in Windows-1251, where the bytes for the letters of "Privet" might look like this (simplified):

• P → 207
• r → 240
• i → 232
• v → 226
• e → 229
• t → 242

Then your colleague opens the file in an editor expecting ISO-8859-1 (Latin-1), or you used StreamReader without specifying an encoding and it didn't match the file. The result: the byte 207 gets interpreted as a character from another table — and the text breaks.

207

240

232

226

229

242

In the end you get Çàèâåò instead of "Privet". If a character doesn't exist at all in the expected encoding you'll see squares or question marks.

The practical takeaway: when reading/writing text you should explicitly specify the encoding, especially if you don't control the data source. In .NET this is done via StreamReader/StreamWriter with the right Encoding (for example, UTF-8 or Encoding.GetEncoding("windows-1251")). That helps avoid mojibake and ensures correct data exchange between systems.

In the next lectures we'll learn how to confidently pick and specify encoding when working with files and streams so your code is international, robust, and reliable.

2

Task

C# SELF, level 37, lesson 0

Locked

Decoding Text from ASCII Codes

Decoding Text from ASCII Codes

Open