Character sets

When you type the letter A, the computer does not store a letter: it stores a number, and a character set says which number means A. This note covers ASCII, extended ASCII and Unicode, how text is held in binary, and why the order of character codes matters for comparing and sorting strings. Paper 1 asks you to describe and compare character sets; Paper 2 relies on the same ideas whenever you use ASC, CHR or compare characters in pseudocode.

What a character set is

Every character, including letters, digits, punctuation, the space and invisible control characters such as carriage return or tab, gets a code. Text is stored as the sequence of codes for its characters. For two computers to exchange text correctly, they must use the same character set, which is why standards exist.

The number of bits used per character fixes how many characters the set can hold: $n$ bits allow $2^n$ different codes.

ASCII

ASCII (American Standard Code for Information Interchange) uses 7 bits per character, giving $2^7 = 128$ codes (0 to 127).

• Codes 0 to 31 (and 127) are control characters: non-printing instructions such as line feed, carriage return and backspace.
• Code 32 is the space.
• The rest are the printable English characters: digits, upper- and lower-case letters and punctuation.

In practice each ASCII character is stored in a full byte, with the extra bit set to 0 (or historically used as a parity bit).

The codes are arranged in a very deliberate order:

• the digits '0' to '9' are consecutive (48 to 57);
• the upper-case letters 'A' to 'Z' are consecutive (65 to 90);
• the lower-case letters 'a' to 'z' are consecutive (97 to 122);
• each lower-case letter is exactly 32 more than its upper-case partner, which in binary means only bit 5 differs (A is 01000001, a is 01100001).

You do not need to memorise any codes: the syllabus says they will be given. But you should know that the groups are consecutive, because questions rely on it. Given that 'A' is 65, you can work out that 'H' is $65 + 7 = 72$.

Extended ASCII

Extended ASCII uses all 8 bits of a byte, giving $2^8 = 256$ codes. Codes 0 to 127 are identical to ASCII, so plain ASCII text is still valid. Codes 128 to 255 add accented letters (é, ü), currency symbols such as £, and line-drawing symbols.

The problem: there is no single extended ASCII. Different code pages assign different characters to codes 128 to 255 (Western European, Cyrillic, Greek...). A document written with one code page and opened with another shows the wrong symbols, and 256 codes can never cover Chinese, Japanese, Arabic or Hindi, each of which needs thousands of characters.

Unicode

Unicode was created to give every character in every writing system a single, unique code, so that text can be exchanged worldwide without ambiguity. It includes all modern scripts, historic scripts, mathematical symbols and emoji.

• Each character has a code point, written U+ followed by hex digits: A is U+0041, € is U+20AC.
• There is room for 1 114 112 code points (U+0000 to U+10FFFF); about 150 000 are currently assigned.
• The first 128 code points are the same as ASCII, so Unicode is backward compatible with ASCII.

Code points are stored using an encoding:

So the string Café €5 (7 characters) takes 10 bytes in UTF-8 (é needs 2, € needs 3), 14 bytes in UTF-16 and 28 bytes in UTF-32.

Benefits and drawbacks

Unicode's benefit is universality: one standard for all languages, so multilingual documents, websites and messages display correctly everywhere. Its drawback is size: depending on the encoding, each character may need two or four bytes, so files and memory use can be larger than with ASCII. ASCII is compact and simple but limited to English characters.

Comparing and sorting text

Because characters are numbers, a computer compares characters by comparing their codes. Strings are compared character by character from the left until two differ.

This explains some results that surprise students:

• 'Z' (90) comes before 'a' (97), so "Zebra" < "apple" is TRUE in a straightforward comparison.
• Digits come before letters, so "42" < "Banana".
• "10" < "9", because '1' (49) is less than '9' (57) at the first character.

Sorting ["apple", "Banana", "cherry", "Zebra", "42"] by character codes gives "42", "Banana", "Zebra", "apple", "cherry". To sort alphabetically regardless of case, programs convert both strings to the same case before comparing.

Working with codes in pseudocode

The pseudocode guide provides LCASE and UCASE for single characters. Exam papers usually also give ASC(ThisChar : CHAR) RETURNS INTEGER (the character code of a character) and CHR(x : INTEGER) RETURNS CHAR (the character with code x) in the insert. Because the codes of a group are consecutive, you can do arithmetic on them.

FUNCTION ToUpper(InString : STRING) RETURNS STRING
   DECLARE OutString : STRING
   DECLARE Index : INTEGER
   DECLARE ThisChar : CHAR

   OutString ← ""
   FOR Index ← 1 TO LENGTH(InString)
      ThisChar ← MID(InString, Index, 1)
      IF ThisChar >= 'a' AND ThisChar <= 'z' THEN
         ThisChar ← CHR(ASC(ThisChar) - 32)
      ENDIF
      OutString ← OutString & ThisChar
   NEXT Index
   RETURN OutString
ENDFUNCTION

FUNCTION Encrypt(PlainText : STRING, Key : INTEGER) RETURNS STRING
   DECLARE CipherText : STRING
   DECLARE Index : INTEGER
   DECLARE Position : INTEGER
   DECLARE ThisChar : CHAR

   CipherText ← ""
   FOR Index ← 1 TO LENGTH(PlainText)
      ThisChar ← MID(PlainText, Index, 1)
      IF ThisChar >= 'A' AND ThisChar <= 'Z' THEN
         Position ← ASC(ThisChar) - ASC('A')
         Position ← (Position + Key) MOD 26
         ThisChar ← CHR(Position + ASC('A'))
      ENDIF
      CipherText ← CipherText & ThisChar
   NEXT Index
   RETURN CipherText
ENDFUNCTION

Practice

FUNCTION CountDigits(Text : STRING) RETURNS INTEGER
   DECLARE Count : INTEGER
   DECLARE Index : INTEGER
   DECLARE ThisChar : CHAR
   Count ← 0
   FOR Index ← 1 TO LENGTH(Text)
      ThisChar ← MID(Text, Index, 1)
      IF ThisChar >= '0' AND ThisChar <= '9' THEN
         Count ← Count + 1
      ENDIF
   NEXT Index
   RETURN Count
ENDFUNCTION