Chapter 12 Character encoding

12.2 Translating two blog posts from Ruby to R

For now, this page walks through these two mini-tutorials (written for Ruby), but translated to R:

Don’t expect much creativity from me here. My goal is faithful translation.

12.3 What is an encoding?

Look at the string "hello!" in bytes. Ruby

The base function charToRaw() “converts a length-one character string to raw bytes. It does so without taking into account any declared encoding”. It displays bytes in hexadecimal. Use as.integer() to convert to decimal, which is more intuitive and allows us to compare against the Ruby results.

Use a character less common in English: Ruby

irb(main):002:0> "hellṏ!".bytes
=> [104, 101, 108, 108, 225, 185, 143, 33]

Now we see that it takes more than one byte to represent "ṏ". Three in fact: [225, 185, 143]. The encoding of a string defines this relationship: encoding is a map between one or more bytes and a displayable character.

Take a look at what a single set of bytes looks like when you try different encodings.

Here’s, a string encoded as ISO-8859-1 (also known as “Latin1”) with a special character. Ruby

irb(main):003:0> str = "hellÔ!".encode("ISO-8859-1"); str.encode("UTF-8")
=> "hellÔ!"

irb(main):004:0> str.bytes
=> [104, 101, 108, 108, 212, 33]

We’ve confirmed that we have the correct bytes (meaning the same as the Ruby example). What would that string look like interpreted as ISO-8859-5 instead? Ruby

irb(main):005:0> str.force_encoding("ISO-8859-5"); str.encode("UTF-8")
=> "hellд!"

It’s garbled, which is your first tip-off to an encoding problem.

Also not all strings can be represented in all encodings: Ruby

irb(main):006:0> "hi∑".encode("Windows-1252")
Encoding::UndefinedConversionError: U+2211 to WINDOWS-1252 in conversion from UTF-8 to WINDOWS-1252
 from (irb):61:in `encode'
 from (irb):61
 from /usr/local/bin/irb:11:in `<main>'

In Ruby, apparently that is an error. In R, we just get NA. Alternatively, and somewhat like Ruby, you can specify a substitution for non-convertible bytes.

In the Ruby post, we’ve seen 3 string functions so far. Review and note which R function was used in the translation.

  • encode translates a string to another encoding. We’ve used iconv(x, from = "UTF-8", to = <DIFFERENT_ENCODING>) here.
  • bytes shows the bytes that make up a string. We’ve used charToRaw(), which returns hexadecimal in R. For the sake of comparison to the Ruby post, I’ve converted to decimal with as.integer().
  • force_encoding shows what the input bytes would look like if interpreted by a different encoding. We’ve used iconv(x, from = <DIFFERENT_ENCODING>, to = "UTF-8").

12.4 A three-step process for fixing encoding bugs

12.4.1 Discover which encoding your string is actually in.

Shhh. Secret: this is encoded as Windows-1252. \x99 should be the trademark symbol ™. Ruby can guess at the encoding. Ruby

Ruby’s guess is bad. This is not encoded as UTF-8. R admits it doesn’t know and stringi’s guess is not good.

Advice given in post is to sleuth it out based on where the data came from. With larger amounts of text, each language’s guessing facilities presumably do better than they do here. In real life, all of this advice can prove to be … overly optimistic?

I find it helpful to scrutinize debugging charts and look for the weird stuff showing up in my text. Here’s one that shows what UTF-8 bytes look like when erroneously interpreted under Windows-1252 encoding. This phenomenon is known as mojibake, which is a delightful word for a super-annoying phenomenon. If it helps, know that the most common encodings are UTF-8, ISO-8859-1 (or Latin1), and Windows-1252, so that really narrows things down.

12.4.2 Decide which encoding you want the string to be

That’s easy. UTF-8. Done.

12.5 How to Get From They’re to They’re

Moving on to the second blog post now.

12.5.1 Multi-byte characters

Since we need to represent more than 256 characters, not all can be represented by a single byte. Let’s look at the curly single quote. Ruby

irb(main):001:0> "they’re".bytes
=> [116, 104, 101, 121, 226, 128, 153, 114, 101]

The string has 7 characters, but 9 bytes, because we’re using 3 bytes to represent the curly single quote. Let’s focus just on that. Ruby

irb(main):002:0> "’".bytes
=> [226, 128, 153]

One of the most common encoding fiascos you’ll see is this: they’re. Note that the curly single quote has been turned into a 3 character monstrosity. This is no coincidence. Remember those 3 bytes?

This is what happens when you interpret bytes that represent text in the UTF-8 encoding as if it’s encoded as Windows-1252. Learn to recognize it. Ruby

irb(main):003:0> "they’re".force_encoding("Windows-1252").encode("UTF-8")
=> "they’re"

Let’s assume this little gem is buried in some large file and you don’t immediately notice. So this string is interpreted with the wrong encoding, i.e. stored as the wrong bytes, either in an R object or in a file on disk. Now what?

Let’s review the original, correct bytes vs. the current, incorrect bytes and print the associated strings.

12.5.2 Encoding repair

How do you fix this? You have to reverse your steps. You have a UTF-8 encoded string. Convert it back to Windows-1252, to get the original bytes. Then re-encode that as UTF-8. Ruby

irb(main):006:0> "they’re".encode("Windows-1252").force_encoding("UTF-8")
=> "they’re"