John French
Author and Musician
March 2023
On AR and ChatGPT
29/03/23 10:24
Two weeks ago I bought an Apple Watch and after four days returned it without a second thought. It was a really interesting example of sprinkling a little augmented reality on top of everyday life.
On day one I mostly played with with the watch; On day two I understood that its appeal was not actually in what features it brought to the table, but that it let me have those features available at all times with no friction. The interaction with technology was, for a few specific features, virtually frictionless in an awesome way. If I wanted music on a drive, I could get into my car, and as I was pulling away just quickly ask my wrist to start playing something by Santana. Before you know it, Evil Ways is coming through the car stereo - and all it took was a lazy request that was forgotten as soon as it was spoken.
It gamified exercise, forwarded notifications to my wrist with a little tap against my skin, let me handle contactless payments, and provided a means to raise my wrist and request a song or dictate a message or whatever other thing from Siri. In fact, Siri was the key to the whole thing's appeal; here was a thing with all the casual presence of a Star Trek comm badge that could respond to my whims. But, like Siri has always done, the limits of it quickly emerge when you have to know in advance how to phrase each request, and decide what your request needs to be. "Play me something chilled, erm, no, actually instrumental and relaxing" is understandable to any human, but Siri is challenged by it. Similarly, while I could ask Siri to send a chat message on my behalf, the phrasing was odd: I had to start the command by addressing Siri, but halfway through switch the command to addressing the message recipient. “Tell my wife I’ll be at her mother’s house at three” ends up sending my wife, “I’ll be at her mother’s place at three”. It took a bit of learning and never quite felt right; being conversational with the request to Siri collided with Siri being extremely literal in the dictation.
On day three I decided that, compared to just pulling my phone out of my pocket, removing friction for a few things and introducing new friction for other things wasn't quite worth £300. On day four I went back to the Apple store and got the money back.
I think Augmented Reality, that sibling technology to Virtual Reality, has intoxicating potential to make features we take for granted in our devices today seem that much more magical. I came across this recently: AR Ski Goggles, which are already bulky (good to hide the tech) and have pretty obvious use cases (good to work out useful features): Augmented Reality Ski Goggles (kottke.org)
It got me thinking: If you're gonna chase augmented reality, you need to make the technology an ergonomically and mentally weightless addition to the 'dumb' world.
AR is not strictly about putting a virtual world over the current one, but about making virtual interactions accessible from non-virtual contexts. The Watch was almost great at this. And that I think is where ChatGPT's conversational capabilities will contribute. Ten years ago "smart" assistants were enormous piles of stateless if() statements trying to parse your speech, match it to a database of pre-written statements, and spitting back a formulaic response or action in return. An Amazon Echo you buy today is still working that way.
Arguably we’ve all faced challenges talking to somebody from a different cultural background; we might both be speaking English but that doesn’t mean we’re able to understand each other. I would put interactions with voice assistants currently in this same category of frustration. The need to learn a particular vocabulary to interact with your technology is ultimately frustrating; what seems weightless at a glance doesn’t stay that way for long. We've seen more public progress in solving that challenge from OpenAI in the last three months than in the entirety of the proceeding decade, and I’m looking forward to familiar voice assistants incorporating it. We might not notice when it happens - but when it does, I might go back and try out that Watch again.
On day one I mostly played with with the watch; On day two I understood that its appeal was not actually in what features it brought to the table, but that it let me have those features available at all times with no friction. The interaction with technology was, for a few specific features, virtually frictionless in an awesome way. If I wanted music on a drive, I could get into my car, and as I was pulling away just quickly ask my wrist to start playing something by Santana. Before you know it, Evil Ways is coming through the car stereo - and all it took was a lazy request that was forgotten as soon as it was spoken.
It gamified exercise, forwarded notifications to my wrist with a little tap against my skin, let me handle contactless payments, and provided a means to raise my wrist and request a song or dictate a message or whatever other thing from Siri. In fact, Siri was the key to the whole thing's appeal; here was a thing with all the casual presence of a Star Trek comm badge that could respond to my whims. But, like Siri has always done, the limits of it quickly emerge when you have to know in advance how to phrase each request, and decide what your request needs to be. "Play me something chilled, erm, no, actually instrumental and relaxing" is understandable to any human, but Siri is challenged by it. Similarly, while I could ask Siri to send a chat message on my behalf, the phrasing was odd: I had to start the command by addressing Siri, but halfway through switch the command to addressing the message recipient. “Tell my wife I’ll be at her mother’s house at three” ends up sending my wife, “I’ll be at her mother’s place at three”. It took a bit of learning and never quite felt right; being conversational with the request to Siri collided with Siri being extremely literal in the dictation.
On day three I decided that, compared to just pulling my phone out of my pocket, removing friction for a few things and introducing new friction for other things wasn't quite worth £300. On day four I went back to the Apple store and got the money back.
I think Augmented Reality, that sibling technology to Virtual Reality, has intoxicating potential to make features we take for granted in our devices today seem that much more magical. I came across this recently: AR Ski Goggles, which are already bulky (good to hide the tech) and have pretty obvious use cases (good to work out useful features): Augmented Reality Ski Goggles (kottke.org)
It got me thinking: If you're gonna chase augmented reality, you need to make the technology an ergonomically and mentally weightless addition to the 'dumb' world.
AR is not strictly about putting a virtual world over the current one, but about making virtual interactions accessible from non-virtual contexts. The Watch was almost great at this. And that I think is where ChatGPT's conversational capabilities will contribute. Ten years ago "smart" assistants were enormous piles of stateless if() statements trying to parse your speech, match it to a database of pre-written statements, and spitting back a formulaic response or action in return. An Amazon Echo you buy today is still working that way.
Arguably we’ve all faced challenges talking to somebody from a different cultural background; we might both be speaking English but that doesn’t mean we’re able to understand each other. I would put interactions with voice assistants currently in this same category of frustration. The need to learn a particular vocabulary to interact with your technology is ultimately frustrating; what seems weightless at a glance doesn’t stay that way for long. We've seen more public progress in solving that challenge from OpenAI in the last three months than in the entirety of the proceeding decade, and I’m looking forward to familiar voice assistants incorporating it. We might not notice when it happens - but when it does, I might go back and try out that Watch again.
