Facebook CEO Mark Zuckerberg has completed his personal challenge for 2016: he’s built his own personal assistant bot. In his latest blog post, Zuck describes how his AI creation, which he likens to Jarvis in Iron Man, can:
- talk on his phone and computer
- understand text messages
- control his home
- learn his tastes and patterns
- learn new words and concepts
- and even entertain his daughter Max.
Zuck’s short video below shows Jarvis in action – briefing Zuck on appointments for the day, describing the weather outside, teaching Max Mandarin, firing over a clean shirt, setting up the video conferencing room, playing music, joining in games and more.
Jarvis can even recognize people at the gate and let them in, just letting Mark know about it. Voiced by Morgan Freeman, listen out for him calling out:
Mark, your parents are coming in!
How did Zuck do it?
As well as 100 hours of coding in Python, PHP and Objective C, behind Jarvis lies a whole wealth of emerging technologies including natural language processing, speech recognition, face recognition and reinforcement learning.
But before Zuck could bring Jarvis to life, he had to write code to connect all the systems, which all speak different languages and use different protocols. In a note to the connected industry, in is blog Zuck pleads:
We need more devices to be connected and the industry needs to develop common APIs and standards for the devices to talk to each other.
Beyond connecting the systems, Zuck also had to find appliances that would work the way he wanted:
It’s hard to find a toaster that will let you push the bread down while it’s powered off so you can automatically start toasting when the power goes on.
Communicating with Jarvis
Jarvis’ communications interface started out as just text. Later it on Zuck evolved it to recognize speech and translate that into text for Jarvis to act on.
To enable voice, Zuck build a dedicated Jarvis app that could listen continuously to what he was saying. While the project started out on a computer, the end solutions needed to be ubiquitous. Zuck deployed a Jarvis app to multiple mobile devices planted around the house so Jarvis could continue to listen in as he and his family moved from room to room.
But interpreting a word and grasping its meaning are two different things, so Zuck needed a way to teach Jarvis new words and concepts. He also reveals:
Understanding context is important for any AI… The more context an AI has, the better it can handle open-ended requests.
And we shouldn’t forget vision, after all it also plays an important role in how we communicate. Zuck notes some of the challenges bots face in understanding what is happening in images and videos include:
- Tracking – is Max awake and moving around
- Object recognition – is that Beast the dog, or a rug
- Face recognition – who’s at the door
Zuck plans to continue to improve Jarvis. In fact, he calls for suggestions at the end of his video.
Jarvis needs an Android app (it’s currently only iOS) so Zuck can set up more Jarvis voice terminals and connect more appliances:
I’d love to have Jarvis control my Big Green Egg and help me cook, but that will take even more serious hacking.
Zuck also wants Jarvis to be able to learn by himself and hopes to, one day, open-source his code.
I personally expect some huge steps forward during 2017, not just in AI and machine learning technologies themselves, but in how we apply them.
We will have to work hard to ensure they are being applied for good and for the good of all. There will be many lessons to learn along the way…