One type of Business Consulting project I frequently participate in is called an IVR Optimization.  During these engagements, we do our best to identify barriers to an application's success...however that may be defined.  Our research includes:

• interviews with company stakeholders
• reviewing system logs & reports
• placing test calls
• monitoring caller interactions (when possible)  

I want to talk a little about the last bullet.  On the whole, most of the companies we've done Optimizations for have had easy access to caller/agent recordings.  This is a great resource for identifying automatable tasks, or for understanding why callers reach an agent for tasks that are already automated.  We'll hear comments like "I just want to make sure the payment went through" (design flaw) or "I tried to use the system but it sent me to you" (business rule issue) or "I hate those automated phone applications" (unfortunate reality). 

What's surprising, though, is how rarely we get access to whole-call recordings.  Instead, we're left with logs and reports to figure out what's going on in the IVR.  But even when those logs include speech data, which obviously doesn't happen when we're optimizing a DTMF application, there are so many stones left uncovered with this kind of information.  

To be honest, it's kind of fun to listen to whole-call recordings because you can hear callers talking in the background, such as while they're on hold.  A few gems that come to mind include the caller who was overheard saying "I'll bet I can convince this agent to give me a refund if I start crying..." and the one who swore up a storm when he thought nobody was listening but never had the courage to tell the agent he was even a little bit upset.  But this important research tool is about much more than eavesdropping.  Here is a sampling of important caller experience issues that can only be discovered through whole-call recordings:

Misroutes
It's very enlightening to compare what callers tell the IVR they want to do with what they tell agents they want to do.  Consider the caller who chooses the "question about my bill" option and gets a line-by-line description of what each item in the bill represents, but what she really wanted was to dispute a charge.  Or what about the caller who chose "discuss a lost or stolen phone" when he actually wanted to complain that his replacement phone wasn't working right?  Both of these situations could be fixed by subtle changes to the menu options, and would never have been identified without whole-call recordings.

Audio issues
If you rely on speech recognizer logs and utterance capture alone, you'll never know if there are underlying platform issues that are preventing speech from reaching the recognition engine in the first place.  That's what we found in one engagement with whole-call recordings available.  Callers were talking and talking, and the system kept replying "Sorry, I didn't hear anything."  Based on traditional data, this looked like a working system that was appropriately throwing no-speech errors...but not when we turned to whole-call recordings.

Non-players
Not all failures to cooperate with an IVR are because callers don't like automation.  For example, listening to whole-call recordings into a healthcare provider line demonstrated that many failures to respond to a prompt correlated to another phone ringing in the doctor's office.  Rather than listening to the quick snippets of background noise that make it into the processed audio the speech recognizer gets, whole-call recordings give a more complete picture of the kinds of distractions that accompany typical customer calls.

If you want your IVR to truly be "optimized," then please adopt a whole-call recording solution...or let us craft one for you as part of a project with Nuance Business Consulting.  I can't tell you what we'll find, but rest assured we've never before walked away empty handed.