Has a UnicodeEncodeError ever got on your nerves? I think it has happened to pretty much everyone. I thought it’s a good time to start demystifying this mystery, and to have fun while doing it.
You are probably familiar with the famous hello program, which does nothing but print “Hello, world!” Now we are going to create the bon program, which prints “Bon appétit!” Here it is, but don’t type it in or copy/paste it:
import sys sys.stdout.write("Bon appétit!\n")
We could have written it with a single
Now, I told you to not type in this program or copy/paste it. Instead, we are going to create it by writing down the bytes that comprise it. The file
bon.py, like any file, is a series of bytes. These bytes represent the characters in the file. The first character is “i”, which is represented by the byte 105; the second one is “m”, represented by 109; and so on. You can find the table for all characters (except é) at Wikipedia.
In order to create the file, we will use another Python program. Here it is. You can copy and paste that one; name it
byte_values = [ # i m p o r t s y s 105, 109, 112, 111, 114, 116, 32, 115, 121, 115, 13, 10, 13, 10, # s y s . s t d o u t . 115, 121, 115, 46, 115, 116, 100, 111, 117, 116, 46, # w r i t e ( " B o n 119, 114, 105, 116, 101, 40, 34, 66, 111, 110, 32, # a p p +------+ t i t ! \ n " ) 97, 112, 112, 195, 169, 116, 105, 116, 33, 92, 110, 34, 41, 13, 10, ] with open('bon.py', 'wb') as f: f.write(bytearray(byte_values))
python createbon.py and it will create
bon.py. Don’t try to run it yet; instead, open
bon.py in a UTF-8 editor to see it. By default, on GNU/Linux systems any editor should work properly. On Windows I experimented with Notepad and it worked, but you might need to tell it it’s UTF-8.
You may have noticed that 32 is the space, and that I’m using the sequence 13 10 for a new line. 13 stands for “carriage return”, and 10 for “line feed”. On Windows both characters are needed to change line, whereas in Unixes only line feed is used. Here I used the Windows version. Python programs in GNU/Linux will rarely use 13 10 as the newline, they will normally use only 10, but if it happens that a file uses 13 10 it will work without problem.
bon.py is a different story. First of all, you can only run it in Python 3. Python 3 assumes that the encoding of the input file is UTF-8, which it is.
If you try to run
bon.py with Python 3 on GNU/Linux, most likely it will work; however, on several environments, it might not work and instead give you a UnicodeEncodeError:
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 7: ordinal not in range(128)
Whether it succeeds or not, here is what happens: Python reads
bon.py. When it reaches the “Bon appétit!\n” part, it reads the bytes and decodes them into a string of characters. Python assumes these bytes are the string encoded in UTF-8, which it is, so it is able to decode these bytes into an internal representation of the string.
When the time comes to print the string to the standard output, Python must convert the string from the internal representation to an encoding that will be understood by the terminal inside which Python is being run. The way Python determines that encoding depends on the system on which it is running.
Let’s see some examples only for the word “appétit”. If the terminal uses UTF-8, the bytes produced by the encoding will be 97, 112, 112, 195, 169, 116, 105, 116 (I’ve bolded é just to make it easier for you to read the whole sequence). If it uses ISO-8859-1, the bytes will be 97, 112, 112, 233, 116, 105, 116. If it uses UTF-16, the bytes will be 97, 0, 112, 0, 112, 0, 233, 0, 116, 0, 105, 0, 116, 0. In all cases, Python will throw these bytes to the terminal, and the terminal will decode them into its internal representation, and in co-operation with the operating system and the windowing system it will paint the corresponding glyphs on the screen.
If the terminal uses ASCII, Python will attempt to encode the string into ASCII. In addition, if the encoding used by the terminal is unknown to Python, Python will also use ASCII. But é cannot be encoded into ASCII, because the ASCII character set does not contain that character, so Python will throw the UnicodeEncodeError.
Python 3 is much better than Python 2 when it comes to all this stuff. In Python 2 there are two complications.
First, Python 2 assumes that the encoding of the input file is ASCII, and in ASCII all bytes are a number between 0 and 127, and here we have two bytes that are more than 127—it’s the two bytes that represent é, 195 and 169. So Python 2 will stop and complain (“SyntaxError: Non-ASCII character ‘\xc3’ in file /tmp/bon.py on line 3, but no encoding declared”). You can solve this problem by adding a comment on top of the file that tells Python that the encoding of the file is UTF-8, but we will not bother with that here; read PEP 263 if you are interested in that. (Python 3 always works with UTF-8, whereas Python 2 needs a PEP 263 comment, and if you use one it should always be UTF-8; don’t use anything else, otherwise you will be in trouble when porting your code to Python 3).
Second, in Python 2 the literal
"Bon appétit!\n" is (speaking in the much cleaner Python 3 terminology) a sequence of bytes, not characters. While the program will work as it is, Python 2 treats the string as a sequence of bytes and just throws these bytes to the terminal. It will not attempt to translate them to the terminal’s character set, as they’re just bytes. So it might work in this case but the program is actually buggy, and it will fail in different circumstances, such as when you try to run it on a UTF-16 terminal. To make Python 2 treat this as a string of characters, you’d need to prefix the string with a
u and make it
u"Bon appétit!\n"; or, better, to use
from __future__ import unicode_literals. This talking about “unicode” is quite confusing and the reason for it is Python 2’s history and backwards compatibility. Your best bet is to forget about it and think in terms of Python 3’s strings and bytes. So the
u prefix tells Python 2 that what follows is a sequence of characters, not a sequence of bytes. In other words, Python 2 (if you use the
u prefix) will internally know that
é is the character
é, whereas, if you don’t use the
u prefix, Python 2 will understand it as the otherwise meaningless sequence of numbers (195, 169). In Python 2,
len(u"Bon appétit!\n")is 13 whereas
len("Bon appétit!\n") is 14.