Inside Siri's brain: The challenges of extending Apple's virtual assistant

Siri is, by far, my favorite recent addition to iOS. In an age when electronic devices keep getting smaller, faster, and thinner, the humble keyboard feels increasingly like a relic of a bygone era—an era when computers were designed to occupy entire rooms instead of the palm of a hand. Apple’s virtual assistant has changed the ways in which I interact with my mobile devices in a significant, if small, way.

Alas, Siri is frustratingly limited. It’s integrated well enough with Apple’s own apps, as well as with services that the company decides to support, but it’s otherwise nearly impossible to use with third-party software of any kind. That’s disappointing, because in such a context it could be a game changer—especially for people who have difficulty interacting with a normal keyboard because of a disability.


There and back again

The extra microphones on the iPhone help reduce noise.


When you issue Siri a command, your device is mainly responsible for collecting the sound of your voice and converting it into an audio file, which it then sends to Apple’s data center for processing. This is not as trivial a task as it sounds—you’d be surprised at how much noise a microphone picks up, even when you’re in what seems like a quiet environment. For this reason, Apple has been investing heavily in technologies that make that sound as clear as possible: Most recent iOS devices feature multiple microphones, along with sophisticated hardware that analyzes the mics’ input to produce a signal scrubbed of most of its noise—a cleaner signal requires less data to transmit and is easier to process.

Once it reaches Apple’s servers, your audio file goes through a series of steps that progressively transform it into an action that a computer program can perform—such as figuring out what the weather looks like. The output of that action is then transformed back into text that can be read to you in a natural way.

Recognition and context

Siri's voice recognition works very well (this was dictated in a room full of screaming children).

The first of these steps consists of converting your spoken words into text—a task that Apple reportedly delegates to voice recognition powerhouse Nuance. Siri does a remarkable job here: Even with my Italian accent—“Noticeable,” as a friend once told me, “but you’ll never be picked as the next voice of Mario”—I find myself only rarely having to repeat a command.

However, Siri’s success at understanding me is possible only because it already “knows” the words I am likely to speak: The service uploads your contacts and other data about you so that it can recognize the information later on with a good degree of accuracy. Apple has programmed Siri to understand all the terms that are required to fulfill the tasks it supports, based on the context in which they are presented.



From words to concepts

Teaching a computer to understand idioms can be very hard. Luckily, here Siri pulled up a list of restaurants instead of calling the police.



Assuming that the user has a passable command of their chosen language, this is usually a relatively easy problem to handle. The hard part comes when all those words have to be turned into some sort of actionable content that an app can process; to do this well, the system must have what is called domain knowledge—in other words, it must know the subject area you’re talking about.

You’ve likely encountered a similar problem when asked to deal with a body of knowledge you’re unfamiliar with: Your doctor, for example, may tell you that you need to be treated for dyspepsia, but unless you are a medical professional, you probably won’t know that you just have indigestion and need an antacid or two. Apple would have to come up with a way for developers to explain to Siri what their apps can do, and provide all the appropriate terminology for those actions.

Of all the parts that make up Siri, this natural language analysis is probably the toughest for developers to tackle, because apps differ greatly, and it’s hard to come up with a magic solution that can easily be applied to every possible situation. To make things worse, natural language analysis is not a familiar field for most programmers—who, until now, have mainly been concerned with point-and-click (or point-and-tap) interfaces.

Putting results to text

Siri's personality may feel gimmicky, but it helps the user feel like they are part of a conversation.

Once a request has been processed, Siri must convert the result back into text that can be spoken to the user. While not as hard as processing a user’s commands, this task, known as natural language generation, still presents some challenges.



Siri for everyone

Allowing third-party apps to integrate with Siri would be a boon for both developers and users, but it’s going to require a lot of effort for everyone involved, in large part because it would represent a significant departure from the way we are used to designing and interacting with our software.

Still, it’s fair to say that the company is quietly laying the groundwork for putting more and more Siri-like capabilities within the reach of every programmer, starting with Apple’s ever-increasing investment in the back-end facilities that it needs to run Siri’s complex infrastructure.

For example, dictation is now built into both of Apple’s operating systems, though developers are not currently allowed to add their own specialized jargon to the vocabulary. Similarly, both OS X and iOS have recently acquired several programming interfaces that can be used to analyze the syntax of a text document, although they do not help much in the much harder task of interpreting its meaning. Finally, Apple’s software has long been adept at speech synthesis; right now this capability is used primarily by system tools like VoiceOver and is off-limits to developers (at least on iOS), but it wouldn’t take much work, from a technical perspective, to turn speech synthesis into a general-purpose tool that everyone could use.


Shop Tech Products at Amazon