CS520
Spring 2012
Programming Assignment 1
Due Tuesday February 7

You are to write two programs to convert between Unicode BMP and the UTF-8 encoding.

Unicode BMP is a 16-bit character encoding that is "designed to support the interchange, processing, and display of the written texts of the diverse languages of the modern world." UTF-8 is a "transformation format" for efficiently storing or transmitting Unicode text.

UTF-8 encodes the Unicode BMP 16-bit characters in the following manner:

• \u0000 - \u007F ( 7 data bits): 0XXXXXXX
• \u0080 - \u07FF (11 data bits): 110XXXXX 10XXXXXX
• \u0800 - \uFFFF (16 data bits): 1110XXXX 10XXXXXX 10XXXXXX

Some examples in binary (Unicode BMP : UTF-8):

• 0000000000001111 : 00001111
• 0000011001110111 : 11011001 10110111
• 0111010111010101 : 11100111 10010111 10010101

A reference for the UTF-8 encoding is section 4.4.7 of the The Java Virtual Machine Specification by Lindholm and Yellin.

Write a program, uni2utf, that will read a stream of Unicode BMP characters from stdin and will write the corresponding stream of UTF-8 bytes to stdout. Be sure to handle the error cases where the input is empty or has an odd number of bytes by emitting to stderr an appropriate error message, which includes the byte offset in the file where the error was detected.

Write a program, utf2uni, that will read a stream of UTF-8 bytes from stdin and will write the corresponding Unicode BMP characters to stdout. Be sure to check that the UTF-8 byte sequences are valid. If you encounter an invalid UTF-8 byte sequence, then emit an appropriate error message (that includes the byte offset in the file at which the error was detected) to stderr, then skip to the next valid character, and continue processing:

• If the error is detected in what appears to be a one-byte UTF-8 encoding (first byte has its topmost bit set to 0), then re-start processing on the next (second) byte.
• If the error is detected in what appears to be a two-byte UTF-8 encoding (first byte has topmost three bits set to 110), then re-start processing on the third byte.
• If the error is detected in what appears to be a three-byte UTF-8 encoding (first byte has its topmost four bits set to 1110), then re-start processing on the fourth byte.
• If the header byte itself is in error (none of the above cases applies), then re-start processing on the next byte.

When reading a stream of Unicode BMP characters, assume that the high byte of a character will be read first and the low byte will be next. When writing a stream of Unicode BMP characters, write the high byte first, followed by the low byte.

Similarly, write multi-byte UTF-8 encoded characters with the high byte first and read multi-byte UTF-8 encoded characters assuming the high byte will be first.

The two programs will be worth equal credit: each is worth 50% of the points for the assignment.

Your program will be graded primarily by testing it for correct functionality. However, you may lose points if your program is not properly structured or adequately documented. See the mandatory guidelines given in the course overview webpage.

You may find using the od command helpful for analyzing the test files. In particular, using the -tx1 flag will display the bytes of a file, one byte at a time, in hexadecimal.

You must write your programs in C.

Put your two programs in two separate files: uni2utf.c and utf2uni.c. Your programs will be graded using agate.cs.unh.edu, so be sure to test in that environment.

There are two labs associated with this program: Lab 1 on January 27 and Lab 2 on February 3.

Your assignment should be submitted for grading from agate.cs.unh.edu. To turn in this assignment, type:
~cs520/bin/submit prog1 uni2utf.c utf2uni.c

Please submit only your C source files. Do not turn in any other files!

Submissions can be checked from agate.cs.unh.edu by typing:
~cs520/bin/scheck prog1

This assignment is due Tuesday February 8. The standard late policy concerning late submissions will be in effect. See the course overview webpage.

Remember: as always you are expected to do your own work on this assignment. Copying code from another student or from sites on the internet is explicitly forbidden!

If you developed your code on a DOS/Windows system, be sure to appropriately transfer your files to a CIS Linux system before submitting them. You need to convert the DOS ASCII file format to UNIX format. If you need help with this, please see me.

Comments and questions should be directed to hatcher@unh.edu