character encoding
The rule for turning text characters into bytes and back, such as UTF-8.
In more detail
Text is stored and sent as bytes, and an encoding such as UTF-8 says which bytes mean which characters. Decoding with the wrong encoding produces garbled text or an error, so formats and protocols declare the encoding they use. Naming the encoding explicitly, usually UTF-8, avoids depending on an operating system default.
Examples
msg.value.decode()In the browser: a network response is only bytes until it is decoded, and declaring the encoding early with a meta charset tag or a response header avoids guessing and re-parsing.Node.js Buffers support encodings such as "utf8", "hex", "base64" and "latin1" in Buffer.from() and buf.toString().open("cafe.txt", "w", encoding="utf-8")Related terms
- string — A sequence of characters, that is, a piece of text.
- bytes object — A Python value holding raw bytes, numbers from 0 to 255, written like b"abc", as opposed to a string of characters.
- ASCII — An old character encoding covering 128 characters: English letters, digits, punctuation and control codes.
- serialization — Turning an in-memory object into bytes, such as JSON text, that can be sent or stored, and parsing it back again.
- HTML parser — The part of the browser that reads HTML text and builds the DOM tree from it, one token at a time.
- bytes, kilobytes, megabytes — Units of data size: a kilobyte is about a thousand bytes, a megabyte about a million, a gigabyte about a billion.
- TextDecoderStream — A stream transformer that turns incoming bytes into text using a character encoding such as UTF-8, even when a character is split across chunks.
- UTF-8 — The most widely used character encoding, which can represent every Unicode character and matches ASCII for plain English letters.
- Unicode — The standard that gives every character in every writing system its own number, called a code point.
- Base64 — A way to write any bytes using only 64 safe text characters, so binary data can travel in places that expect text.
- hexadecimal — A way of writing numbers with sixteen digits, 0 to 9 and then a to f, so each character stands for four bits.
- blocklist — A list of values, such as addresses, domains or file hashes, that a control refuses outright.
- escaping — Rewriting special characters in a value so an interpreter reads them as plain text, for example turning < into < in HTML.
- pathlib — Python's standard library module for working with file paths as objects rather than plain strings.
Taught in
- Retention Deletes Files, Not Records — Confluent Apache Kafka Certifications (CCDAK and CCAAK) Exam Prep
- HTML Parsing, Tokenization And The Document Object Model (DOM) — Frontend Architecture & Web Platform
- The Fetch API: Streaming Responses, HTTP Caching And Keepalive — Frontend Architecture & Web Platform
- A Producer In Python — Apache Kafka & Event Streaming
- A Consumer In Python — Apache Kafka & Event Streaming
- Headers, Envelopes And Event Metadata — Apache Kafka & Event Streaming
- Buffer Fundamentals: Slab Allocator & V8 Memory — Node.js Internals & Backend Services
- Buffer Encodings: UTF-8, Hex, Base64 & Slicing — Node.js Internals & Backend Services
- The Print Function And String Literals — Hands-on Python: Complete Video Course & Book
- Floats, Division And Mixed Types — Hands-on Python: Complete Video Course & Book
- Objects, Methods And String Case — Hands-on Python: Complete Video Course & Book
- Paths And Files The Modern Way — Hands-on Python: Complete Video Course & Book
- JSON And CSV — Hands-on Python: Complete Video Course & Book
- Injection In All Its Forms — Application Security & Threat Modeling for Engineers
- Testing The Filesystem — Automated Testing, TDD & Quality Engineering
References
- Python Unicode HOWTO
- WHATWG Encoding Standard
- MDN glossary: Character encoding
- codecs: standard encodings
- PEP 686: make UTF-8 mode default