The easiest way to understand the current state of encodings is to look at the history of how we got to where we are today. The pre-ASCII days are not particularly relevant, but they are interesting, so I have run through the various encodings that preceded ASCII from a technical perspective here if you are interested. The story of Unicode really starts with ASCII. Note that since Windows and UNIX are the only two types of operating systems of interest to most modern programmers, I’m only going to discuss them. (Linux, iOS, Android, etc. are all UNIX-based). First I’m going to go through all the problems of the various pre-Unicode ASCII based encodings, and then I’ll provide some programming pointers for cross-platform programming. You can skip the pre-Unicode stuff if you’re not interested in the history
While various flag and signal based encoding systems have existed since pre-history (signal fires, smoke signals), true digital encodings were not needed until attempts to communicate via electrical signals in the form of the electromagnetic telegraph. The early telegraph systems in the 1830s and 1840s were extremely limited in terms of electric components, with basically only batteries and electromagnets available. In another running theme throughout technology, there was also a desire to make the systems usable without the operators having to learn complex codes, and this placed further constraints on the systems. With these components, three basic systems were invented.
It’s 1887. The only electrical components you have access to are electromagnets. And you want to build a digital to analog converter. What do you do?
This device is somewhat of a revolution in telegraph technology from the 1870s. It uses a fixed-length 5-bit binary encoding, the kind of encoding that we would expect to see in silicon-based electronics equipment. Baudot created a receiver that resembles a mechanical computer to perform the decoding and printing. Looked at in modern terms, it uses a 5-bit latching transfer register and a second 5-bit decode register for actually printing the received codes as human readable characters instead of squiggles or dots on paper. The encoding used by the device is named after its inventor Emile Baudot. I’ll run through the entire device, although it is the receiver where things get interesting.
Here is one of the early patents for electroacoustic multiplexing of telegraph signals. (Note that this is not the same as electroharmonic multiplexing, which used multiple frequencies as carrier waves to achieve multiplexing.) I like this one because it really shows the acoustic component of the system. Here is a picture of a quadruplex system.
This is an early attempt at an isochronous distributor-based multiplexing telegraph from 1873. It demonstrates the problematic nature of using Morse code with this kind of multiplexing system. First, let’s have a look at the top view of the device.
If you hear the word Morse Code, you probably imagine people tapping on a key and listening to the code through headphones or perhaps on old-style buzzer. However, this was not how Morse originally conceived his telegraph. Here is the first patent Morse filed for his invention (filed in the 1830s and granted in the 1840s).
The Wheatstone needle telegraph operated based on 5 needles that could tilt to the left or right. Owing to the simplicity of the device, there’s not much detail to show. However, the design is interesting in how it differs from how we would build a similar device using modern ideas. First, here is a picture of an actual telegraph machine (from Wikipedia):
The letter-printing telegraph patented by Royal Earl House in the 1830s was one of the first devices to operate under virtually the same principles as modern computer terminals. They had a keyboard for typing in letters, which were then transmitted down a wire. They also had a printer for printing out the signal they received from the wire as letters on paper. (So theoretically, you should be able to hook one of these up to a modern Linux box as a /dev/tty device.) However, electronics in the 1800s were not particularly advanced, and it turns out that the biggest improvements that could be made to the device was the addition of steam-power. Here is a picture of the steam-assisted device from 1852 (from nadcomm.com. To get an idea of the size, the keyboard is basically a re-purposed piano keyboard):
For comparison, here is the steampunk keyboard of today that this kind of device inspired:
This article runs through the patent from 1852 and shows how steam was incorporated into the instrument to give a steam/electromagnetic/mechanical hybrid device, the kind of thing that steampunk idealizes. However, spoiler alert to all the steampunks out there, although the device was originally designed to run off steam, the inventor soon realized that compressed air worked much better, and the device was actually used as a pneumatic device in practice.
This page shows the detail of the hydraulic regulator in the letter-printing telegraph described here.