A customer list goes into WordPress overnight. In the morning half the names read Müller, Straße, Köln, and the other half are perfectly fine. Nothing is corrupt in the usual sense. Every one of those characters is valid, well formed Unicode that any editor will happily show you. That is exactly the problem.
The letter ü is one code point, U+00FC, and UTF-8 writes it as two bytes: C3 BC. Somewhere in the chain a program was told those bytes were Windows-1252, where C3 is à and BC is ¼. It printed two characters where there had been one. Then something saved that result as UTF-8, and the two wrong characters became four new bytes. The original C3 BC is gone. The file is not damaged, it just says something else now.
This is why the find and replace lists you can copy off a forum half work. A table that turns ü back into ü repairs text that went through the wrong decoder once, and does nothing at all to text that went through twice, where ü now reads ü, four characters, none of them in the list.
Start with the text in front of you
To fix mojibake you have to know which wrong turn the text took, so the tool below does not consult a substitution table. It re-runs the damage as real encode and decode round trips, 36 candidate chains, scores each result against your target language, and names the winner in plain words, for example “UTF-8 read as Windows-1252, twice”. Everything happens in your browser: no upload, no request, nothing leaves the page.
Encoding forensics
Paste text that has come back wrong from a database, a CSV or a feed. This tries every plausible wrong turn the bytes could have taken and scores what comes back. Every round trip is a real one, made with the browser's own TextEncoder and TextDecoder rather than a lookup table of broken pairs, because a lookup table gets double encoding wrong. It ranks the readings, names the chain in plain words, marks the runs that were damaged, and then lists every non-ASCII character in the result with its code point, its name and its category, so a non-breaking space, a soft hyphen or a zero width joiner becomes something you can see and count. It also says when the text is not damaged at all. Everything happens in this browser tab: nothing is uploaded, nothing is stored, nothing leaves the page.
The score counts how many characters land where that language actually lives, and how few replacement characters, C1 controls and mojibake pairs are left over. A mojibake pair is a UTF-8 lead byte followed by a continuation byte, each read as a character of its own, which is the signature of the damage rather than a guess at it. Picking a language sharpens the ranking; any language accepts every letter and only punishes the wreckage.
Nothing pasted yet.
Each chain turns the characters back into the bytes of the encoding that misread them and decodes those bytes again, with the UTF-8 decoder set to fatal, so a chain that cannot be true is dropped rather than papered over with question marks. Click a reading to take it instead of the winner.
Nothing to inspect yet.
| Char | Code point | Name | Category | Count |
|---|
Names and categories are exact for the control, Latin, punctuation, format and currency ranges, which is where encoding damage lives. Outside those the block is named instead of the character, and the category comes from the browser's own case mapping.
Two bytes and a wrong table
Take the word Müller. Stored as UTF-8 the file holds 4D C3 BC 6C 6C 65 72. A program that assumes a single byte encoding reads one byte per character, and in Windows-1252 that gives M, then U+00C3 LATIN CAPITAL LETTER A WITH TILDE, then U+00BC VULGAR FRACTION ONE QUARTER, then ller. Nothing has failed. Every byte decoded, no exception was raised, no log line was written. There is no error to catch, which is why this survives so many pipelines.
The damage happens on the way out. When that string is written back as UTF-8, à encodes as C3 83 and ¼ encodes as C2 BC. Two bytes for one letter have become four bytes for two letters, and the wrong reading is now the content. Read it correctly and you see ü. Read it wrongly a second time and you see ü: Ã, then U+0192 LATIN SMALL LETTER F WITH HOOK, then Â, then ¼.
That  is the single most useful signal in the whole subject. Code points from U+00A0 to U+00BF encode as C2 plus one byte, and C2 is  in Windows-1252. So one wrong pass puts  in front of the symbols (£, ©, °) and à in front of the accented letters. Two wrong passes put  in front of almost everything, because the intermediate characters nearly all sit in that range.
Windows-1252 is not the only wrong reader, and the differences are diagnostic. Latin-1 is identical to it except in one region, the bytes 80 to 9F, which Windows-1252 fills with printable characters (the curly quotes, the bullet, the euro sign, Ÿ) and Latin-1 leaves as invisible C1 controls. So ß, C3 9F, comes out as ß through Windows-1252 and as à plus an invisible U+009F through Latin-1. Mac Roman maps C3 to the square root sign, so ü becomes √º and ä becomes √§, the signature of an old Mac export.
One trap sits under all of this. In the WHATWG encoding standard that browsers implement, the labels iso-8859-1 and latin1 are aliases for windows-1252: ask a browser for Latin-1 and you get Windows-1252. The tool builds true Latin-1 itself, and inverts each reverse byte map out of the decoder’s own 256 byte output rather than trusting a hand written table, which is what makes the double encoded chains come out right.
Why the replacement list half works
A substitution list is a map of outcomes. Damage is a function, and functions compose. The outcome of two passes is not in a list of the outcomes of one pass, and the intermediate step contains characters like ƒ and Ÿ that no such list bothers to include. Worse, the lists are order sensitive: replace à before you have tried ü and you have destroyed the longer match.
Running the chain as a real round trip fixes composition and creates a new risk: over-peeling. Some text genuinely contains à or ¼, and a greedy repairer will “fix” a correct string into nonsense. Two mechanisms hold that in check. The UTF-8 decoder runs with fatal:true, so a chain whose bytes are not valid UTF-8 is dropped rather than papered over with replacement characters. And nothing is applied unless a reading beats the text as it stands: with positive evidence of damage (a mojibake pair, a C1 control, a replacement character, or an entity that still decodes after one pass) a tie is enough, otherwise a clear margin is required.
Scoring is the other half. Each candidate is rated on how many characters land in the ranges the chosen language actually uses, minus replacement characters, C1 and C0 controls, stray surrogates, and mojibake pairs, where a pair is a UTF-8 lead byte followed by a continuation byte. Both byte sets are derived from the tables rather than written down. That is why the target language selector is not decoration.
Real exports are rarely damaged uniformly. Half a line survives an import and half does not, so you get something like “Käse und äpfel”. When the whole text will not decode as one piece, the chain is applied island by island and the reading is labelled “run by run”. Each damaged run is then listed with its position, the text before and the text after.
What MySQL means by utf8
MySQL’s utf8 is not UTF-8. It is an alias for utf8mb3: at most three bytes per character, which covers the Basic Multilingual Plane and nothing above U+FFFF. No emoji, no astral plane CJK, no mathematical alphabets. utf8mb4 is the real thing, up to four bytes per character. WordPress gained utf8mb4 support in 4.2; before that, a four byte character reaching a three byte column was stripped.
Two constants decide what your install asks for. Current WordPress ships them like this in wp-config-sample.php:
define( 'DB_CHARSET', 'utf8mb4' );
define( 'DB_COLLATE', '' );Installs older than that still carry 'utf8' in the file, and it mostly does not matter, because wpdb::determine_charset() rewrites utf8 to utf8mb4 as soon as there is a live mysqli handle, turns an empty collation (or utf8_general_ci) into utf8mb4_unicode_ci, and upgrades that to utf8mb4_unicode_520_ci when the server reports the capability. The rest of what belongs in that file is in wp-config.php explained.
Here is the part that generates mojibake. DB_CHARSET sets the charset of the connection, the wire between PHP and MySQL. It does not set the charset of your columns, which keep whatever the CREATE TABLE gave them, often latin1_swedish_ci on a database created a decade ago. Mojibake is what happens when the wire and the column disagree. Get those constants wrong in a cruder way and you do not get mojibake at all, you get error establishing a database connection.
Two queries tell you which world you are in. The first says what the columns claim to be, the second says what is actually stored:
wp db query "SHOW FULL COLUMNS FROM wp_posts LIKE 'post_title'"
wp db query "SELECT ID, post_title, HEX(post_title)
FROM wp_posts WHERE post_title LIKE '%Ã%' LIMIT 5"HEX() is the only honest witness here: everything else in the chain can lie in both directions. If a title that should read Müller comes back as 4DC3BC6C6C6572, the stored bytes are correct and something in the display path is misreading them. If it comes back as 4DC383C2BC6C6C6572, the damage is in the data and no configuration change will undo it.
The dump that bakes it in
The classic setup: the columns are latin1, the connection is latin1, and the application sends UTF-8 bytes anyway. MySQL believes each byte is one latin1 character and stores them verbatim. On the way out it tells the same lie in reverse, the browser receives the original UTF-8 bytes, and every page is perfect. A site can run like this for eight years and nobody finds out.
Then somebody migrates. mysqldump reads the column as latin1 characters and writes them out in the client charset, and modern clients default to utf8mb4. Every stored byte is converted to its UTF-8 encoding: C3 becomes C3 83, BC becomes C2 BC. The dump file now literally contains the damaged text. Import it into a utf8mb4 database and every ü on the site reads ü. The display lie has become data.
The dump file tells you which happened: mysqldump writes the charset it used into the header as a SET NAMES statement, and the table definitions carry their own DEFAULT CHARSET=. Dumping with --default-character-set=latin1 hands back the raw bytes untouched, and those bytes are honest UTF-8, so changing the charset declarations in the file and importing into utf8mb4 lands the text correctly. The same trap exists inside one database:
-- Wrong when the column holds UTF-8 bytes labelled latin1.
-- CONVERT TO transcodes the values and bakes the mojibake in.
ALTER TABLE wp_posts CONVERT TO CHARACTER SET utf8mb4;
-- Right: relabel through a binary type, which moves the bytes
-- without touching them, then declare what they always were.
ALTER TABLE wp_posts MODIFY post_content LONGBLOB;
ALTER TABLE wp_posts MODIFY post_content LONGTEXT
CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;The binary round trip only works if every value in the column really is UTF-8. On a half converted table it turns the clean half into damage, so do it on a copy first. And repairing mojibake changes the byte length of strings, while anything WordPress stored as a serialized array carries its string lengths inline, so a naive UPDATE ... REPLACE() across wp_options breaks the option instead of fixing it. That mechanism is in serialized data repair and search replace.
Reading the damage without any tool at all
Most of the time you can name the wrong turn from the text alone, which is worth doing first, because it tells you where in the pipeline to look.
- Ã before accented letters, Â before symbols. One wrong pass, single byte table, damage now stored as text.
-  before almost everything, with ƒ, Å and Œ turning up. Two passes. A one pass replacement list will not touch it.
- †before quotes and long dashes. The general punctuation block, three bytes beginning
E2 80, read one byte at a time. - √ where you expected Ã. Mac Roman, so the wrong turn happened on a desktop, not on the server.
- Invisible controls between U+0080 and U+009F. Strict Latin-1, not Windows-1252, where those bytes are printable.
- U+FFFD or literal question marks. Bytes were dropped, not misread. Nothing recovers them, and the tool says so instead of inventing something plausible.
- An entity in the visible text. Escaping applied twice:
üon the page whereüwas meant. - Only some rows damaged. Two imports, or a search and replace that ran halfway.
Entities deserve a note. A singly escaped entity is ordinary HTML, not damage, so the tool offers it as a candidate reading and does not apply it. Only when a reference still decodes after the first pass, the doubly escaped case, does it count as evidence. When two chains tie the longer one wins, so a doubly escaped entity is never left half undone.
The half you cannot see
Fix the visible damage and a second population is left behind: characters that are correct Unicode, that no decoder complains about, and that are invisible to every visual check. They arrived with the same paste from Word or the same legacy import, and they break things quietly.
U+00A0 NO-BREAK SPACE, C2 A0, is the worst of them because it looks exactly like a space. PHP’s trim() strips " tnr x0B" and nothing else, so a no-break space at the end of a title survives trimming and then fails every comparison and every LIKE you write against it. U+00AD SOFT HYPHEN is invisible until a line wraps, at which point a hyphen appears mid word and nobody can find it. U+200B ZERO WIDTH SPACE splits a word for search while leaving it intact for the eye.
U+FEFF has two lives. Inside text it is a zero width no-break space; at the very start of a file, written as EF BB BF, it is a byte order mark, and in a PHP file it is output before any header can be sent. U+2028 LINE SEPARATOR is legal inside a JSON string and was illegal inside a JavaScript string literal until ES2019, which is why JSON pasted into a script tag used to break for no visible reason. When a parser rejects a payload and your eyes cannot find why, what the parser actually saw is the companion piece.
The second half of the tool is built for exactly this. The character inspector lists every non-ASCII code point in the result or the original, with its hex value, Unicode name, general category and count, and shows the invisible ones as named chips rather than blank cells. The “only the invisible and the suspect” checkbox cuts a German page down to the rows worth looking at. For the routine version of this job, stripping junk out of pasted copy before it reaches a post, see getting the junk out of pasted text.
Where this tool stops
It covers Windows-1252, Latin-1 and Mac Roman, which between them account for nearly all Western European mojibake, and it does not cover Shift-JIS, KOI8 or the other national tables. Japanese or Cyrillic damage needs a different set of byte maps, and a chain built from the wrong maps will confidently produce a wrong answer, so it is better to have none.
It inspects text that is already in the page: there is no file input, and the input is capped at 40000 characters. Highlighting is skipped above 20000, and the listings stop at 8 readings, 14 runs and 200 inspector rows, each with a line saying how many were left out. Character names and categories are exact for the control, Latin-1, punctuation, format and currency ranges and a chosen set of Latin Extended-A; outside those you get the block name and a category from the browser’s own case mapping, because the full Unicode database is not shipped to fix an umlaut.
Most of all, it repairs text, not a site. It is the right instrument for a column sample, a CSV header, a feed item or one post you are about to fix by hand, and the wrong one for 40000 rows. For those, use it on a sample to identify the chain with certainty, then run the equivalent transformation once at the database level, with the serialization caveat above in mind.
Mojibake is not corruption. It is a correct answer to the wrong question, which is why it is both so persistent and so recoverable: the information is still there, it has simply been relabelled. Recovering it is not guesswork either. You run the wrong turn backwards as a real byte level round trip and check whether what comes out is a language.
What that will not do is stop it happening again. Clean text going back into a database whose columns and connection still disagree is dirty again on the next import, so the repair order is always connection charset, column charset, then content. Verify with HEX() rather than with your eyes, and make the dump honest before you make the text pretty.
Then run the inspector once over a page you were confident about. Almost every long lived site carries a small population of no-break spaces, soft hyphens and stray zero width characters that arrived years ago in a paste and have been failing string comparisons ever since. They cost nothing to remove, and you cannot see them until you list them.
