Facebook CEO Mark Zuckerberg has completed his personal challenge for 2016: he’s built his own personal assistant bot. In his latest blog post, Zuck describes how his AI creation, which he likens to Jarvis in Iron Man, can:
- talk on his phone and computer
- understand text messages
- control his home
- learn his tastes and patterns
- learn new words and concepts
- and even entertain his daughter Max.
Zuck’s short video below shows Jarvis in action – briefing Zuck on appointments for the day, describing the weather outside, teaching Max Mandarin, firing over a clean shirt, setting up the video conferencing room, playing music, joining in games and more.
Jarvis can even recognize people at the gate and let them in, just letting Mark know about it. Voiced by Morgan Freeman, listen out for him calling out:
Mark, your parents are coming in!
How did Zuck do it?
As well as 100 hours of coding in Python, PHP and Objective C, behind Jarvis lies a whole wealth of emerging technologies including natural language processing, speech recognition, face recognition and reinforcement learning.
But before Zuck could bring Jarvis to life, he had to write code to connect all the systems, which all speak different languages and use different protocols. In a note to the connected industry, in is blog Zuck pleads:
We need more devices to be connected and the industry needs to develop common APIs and standards for the devices to talk to each other.
Beyond connecting the systems, Zuck also had to find appliances that would work the way he wanted:
It’s hard to find a toaster that will let you push the bread down while it’s powered off so you can automatically start toasting when the power goes on.
Communicating with Jarvis
Jarvis’ communications interface started out as just text. Later it on Zuck evolved it to recognize speech and translate that into text for Jarvis to act on.
To enable voice, Zuck build a dedicated Jarvis app that could listen continuously to what he was saying. While the project started out on a computer, the end solutions needed to be ubiquitous. Zuck deployed a Jarvis app to multiple mobile devices planted around the house so Jarvis could continue to listen in as he and his family moved from room to room.
But interpreting a word and grasping its meaning are two different things, so Zuck needed a way to teach Jarvis new words and concepts. He also reveals:
Understanding context is important for any AI… The more context an AI has, the better it can handle open-ended requests.
And we shouldn’t forget vision, after all it also plays an important role in how we communicate. Zuck notes some of the challenges bots face in understanding what is happening in images and videos include:
- Tracking – is Max awake and moving around
- Object recognition – is that Beast the dog, or a rug
- Face recognition – who’s at the door
What next?
Zuck plans to continue to improve Jarvis. In fact, he calls for suggestions at the end of his video.
Jarvis needs an Android app (it’s currently only iOS) so Zuck can set up more Jarvis voice terminals and connect more appliances:
I’d love to have Jarvis control my Big Green Egg and help me cook, but that will take even more serious hacking.
Zuck also wants Jarvis to be able to learn by himself and hopes to, one day, open-source his code.
I personally expect some huge steps forward during 2017, not just in AI and machine learning technologies themselves, but in how we apply them.
We will have to work hard to ensure they are being applied for good and for the good of all. There will be many lessons to learn along the way…
N Wheeler
This from the guys who has tape over his webcam
Sylvester Johnson
Maybe Jarvis can determine the source of the equipment by the usage of the electricity it use, the power consumption, and the wattage. Maybe Zuck can try it. I never heard of this tried but it can open doors.
Igor
Is this all a Billion dollar man can do? It looks like a kids presentation. What’s new on thisAI? A toaster?? A device that can continuously listen to what you say? No Thanks NSA!
Matt Parkes
I wander if Jarvis has met Mirai yet, I suspect it could after all this is a man whose notoriously bad at choosing passwords.
Oscar Doogie
just saw that sugarmountain made fun of NICKLEBACK with jarvis – guess im going to amazon.
Kirk Gee
Home AI certainly has great application. My biggest gripe at the moment is it’s inability to really interpret what is spoken in natural language and also the inability to intelligently link comments made previously that are obviously in context of a single conversation. Also, what about more useful things like circumstantial awareness etc. A great deal more thought needs to go into these things to really be useful. With AI like Watson, we know that all of this is possible but still too expensive for broad application. It’s still too much like the rumba that doesn’t really work because its waste container is too small, it doesn’t intelligently navigate simple everyday obstacles and doesn’t have in depth sensory feedback. The whole development effort reminds me of a movie that has a great subject, excellent cast and direction but falls short because the story-line is so incredibly impractical or stupid that you just cant get into it.
This doesn’t mean give up, just means put some practical thought and logical reality into it!