At F8 today, Facebook is announcing a bunch of utterly crazy shit that we'll soon be able to do to the pictures we take. That includes Facebook, WhatsApp, Facebook Messenger, and Instagram and affects, oh, somewhere approaching 2 billion people. But while the company is talking a lot about cameras, it would be a mistake to look at what it is rolling out as a mere photography tool. Yes, there are cool picture effects. But what Facebook is really trying to do is to fully insert itself in the real world. Facebook’s augmented reality camera effects are an early attempt to let the digital infiltrate the physical, a way for the company to become the conduit between everything you see in the world around you, and all the information that exists, via your smartphone.
“Facebook is so much about marrying the physical world with online,” the company’s CEO Mark Zuckerberg told BuzzFeed News in an interview late last week. “When you can make it so that you can intermix digital and physical parts of the world, that's going to make a lot of our experiences better and our lives richer.”
"Facebook is so much about marrying the physical world with online."
It is certainly going to make life weirder. At an earlier demo, when a group of 18 Facebook engineers gathered to show their work to an outsider for the first time, they were clearly nervous. One pointed his phone at a table, and a 3D propeller plane popped up on screen, circling around a water bottle that rested on the tabletop. Another used his phone’s camera to turn the room into a planetarium, with planets and stars hewing across the ceiling as shooting stars fired from side to side. Still another took a normal photo of a face — and then made it smile, frown, and gape with the push of a button. Little wonder they seemed on edge: The stuff they were showing off was wild and largely unprecedented.
This new Camera Platform, as the company calls it, is a major bet that the camera isn’t simply a tool used to capture images. It’s something you’ll use when you want to share photos and videos, sure, but also when you want to overlay digital experiences on the real world. Imagine, Zuckerberg urged, using Facebook’s camera to view pieces of digital art affixed to a wall. Or to play a digital game overlaid on a tabletop. Or to leave a digital object in a room for someone to later discover — perhaps even future generations. Imagine using your phone to take a 2D photo, and then transform that photo into a 3D space. Imagine manipulating a friend’s expression to make them smile, or frown, or, well, whatever. Imagine changing your home into Hogwarts for a Harry Potter-obsessed daughter. That’s what Facebook is doing. “We see the beginning of an important platform,” Zuckerberg said.
Onstage at F8 Tuesday morning, Zuckerberg doubled down on this point: “The camera needs to be more central than the text box in all of our apps," he said. "We’re making the camera the first augmented reality platform."
And you thought this was just about Snapchat.
AI at War
It’s easy to draw comparisons to Snapchat. And certainly the camera platform’s Snapchat-like effects are likely to grab the most attention early on. But the more interesting stuff that Facebook is trying to pull off involves layering the digital and physical worlds on top of each other — bringing the former into the latter, and vice versa. There will be three big augmented reality areas Facebook is pushing into. The ability to display information on top of the world in front of you, the ability to add new digital objects to your environment (think: Pokemon Go), and the ability to enhance existing objects.
For example, Facebook’s Camera can map out two-dimensional photographs in 3D. The company hopes developers will someday build digital products that behave and interact in those formerly 2D spaces, just as they would in the rich three-dimensional world we live in. Picture this: In one demo, Facebook showed off various 3D scenes created entirely from a handful of 2D photos. The scenes had real depth to them — you could peer around a tree in a forest, or tilt your head to see behind a bed in a room. With a few clicks, the lights went down in the room. The forest flooded with water. It was magic.
The demo was on an Oculus headset, but Facebook’s ambition is to bring these kind of scenes directly into the News Feed itself, no Oculus required. It wants people to be able to create and interact with them directly on their phones.
The ultimate idea here is to turn the real world into an extension of Facebook itself. “There's all these different random effects which are fun, but also foundational to a platform where people can create 3D objects and put them into the world,” Zuckerberg explained.
To pull off these radical camera effects, the company turned to an unexpected source: its AI team. When Zuckerberg began setting plans in motion for his company’s camera platform more than a year ago, he tapped Facebook’s Applied Machine Learning group (AML) to lead it. That put the technology in the hands of team artificial intelligence geeks, not the graphic designers or 3D artists you might otherwise expect.
While not a traditional imaging team, Facebook’s AML group does work extensively in visuals. Much of what the team does is in the AI discipline of computer vision, the science of training computers to analyze and extract information from images, the same way humans do (Think about the way Facebook or Google can identify a face or a landmark in pictures uploaded to them). The group’s computer vision expertise made it an ideal fit for a project predicated on understanding what’s appearing before and beyond a camera lens.
As Facebook’s AML group went to work on Camera last summer, it waded into a thicket of wildly popular rival camera products. Snapchat’s beloved selfie filters, for instance, had inspired hundreds of millions of shares and put the company on the fast track to a multibillion-dollar IPO. Meanwhile, Prisma, a photo app for iOS and Android, was using AI-powered effects to break down images and redraw them in the style of famous paintings.
Facebook promptly put its AML group on lockdown, a drop-everything-and-work-on-only-this measure the company sometimes uses when developing products it sees as highly competitive. Facebook famously went into lockdown to improve its site performance and user experience in 2011 following the debut of Google+.
Yet by the end of lockdown, the camera team had pulled off a significant feat: It had neural net–powered AI software working directly on people’s phones — not remotely on servers where this kind of stuff has traditionally operated. That meant Facebook now had the ability to read and manipulate images very quickly, and could create powerful camera effects that were previously infeasible due to computing limitations.
The first effect the team developed was one called “style transfers.” Like Prisma, it redrew photos as artwork, but unlike the app, it could do so almost instantly. The AML team created a green-screen effect that could pick out a person’s body and put all sorts of backgrounds behind it live in camera. It built filters that automatically identified common objects that might appear in images and created specialized effects for more than 100 of them: a heart-shaped cloud of steam that rises from a cup of coffee, a propeller plane that circles household objects, starscapes that transform a bedroom into a planetarium, and more.
The centralized camera team quickly became the de facto hub for camera effects across Messenger, Instagram, WhatsApp, and Facebook proper. Build once; deploy everywhere. “This is heaven,” Joaquin Candela, the head of AML, told BuzzFeed News. “We have this massive release channel and we’re just going to keep putting stuff in there.”
And Facebook won’t be alone in “putting stuff in there” — at least not if things go the way it hopes. Over the coming months, Zuckerberg said, the company plans to give developers (and to a more minor extent the public at large) a chance to use its tools to create their own filters and effects for Facebook’s cameras. Developers who want to build their own apps, games, and art will be able to do so, opening up a wide array of creative possibilities that Zuckerberg himself admits — and perhaps even hopes — will take Camera in unanticipated directions.
And in opening its platform, Facebook will give developers access not only to AML’s tools, but also to its multi-app, billion-plus-person release channel. “Even though they'll feel a little bit different in terms of features between Instagram and WhatsApp and Messenger, all the stuff that developers are going to build is going to be fundamentally compatible with cameras in all of these,” Zuckerberg said.
Snatch That
But, okay, remember when we said it’s not about Snapchat? Well, it’s also more than a little bit about Snapchat. Or at least, it’s certainly heavily Snapchat-influenced.
In the past few months, Facebook has gone hard at its neighbor in Southern California, adding Snapchat-style ephemeral stories to Facebook, WhatsApp, Instagram, and Messenger. Snapchat, for its part, isn’t standing still, today releasing its own set of augmented reality effects, albeit underwhelming compared to Facebook’s. When BuzzFeed News asked Zuckerberg if he was happy with Stories’ performance on Facebook, and showed him an utterly barren Stories section on an account with more than 700 friends, the Facebook CEO swallowed, paused, and replied, “It’s still early.”
True!
While Zuckerberg may urge patience, it’s likely his new camera platform will be judged in the early going by whether it can help Stories take off inside all Facebook products — not just Instagram, but Messenger, Facebook, and WhatsApp as well. And the seeming failure of stories to gain traction inside places like the main Facebook app, or Messenger, raises the question of what truly belongs there. Because Facebook’s real power is in its network.
The same social graph Mark Zuckerberg talked about at F8 some 10 years ago — the one that connects you to your old friends, new acquaintances, high school teachers, and probably a lot of co-workers — remains its defining characteristic. The lesson of Snapchat seems to be that some things make sense on the big social graph, and some things don’t. And what will that mean for all this augmented reality? Are we really going to want to see flooded forests in our feeds?
And yet there is also this: A year ago, the social giant was in the midst of a small crisis, fending off a challenge from Snapchat which seemed to now own the fun, raw moments that originally gave social media its charm. Meanwhile Facebook proper was experiencing a decline in orignal sharing. In response, Facebook ruthlessly copied Snapchat Stories into all its products. And while Stories may seem like a wasteland in the main Facebook App, last week, daily users of Instagram Stories surpassed Snapchat as a whole (at least based on the latest numbers Snapchat provided). There are a lot of ways Facebook can use its network to win.
So, yes it’s still early. And yes, this may be a shot at Snapchat. But the war is for something much bigger. It’s about using the thing in your hand to analyze, interpret, explain, and fundamentally alter the way you experience the world around you. “We just view this as part of the first round of what a modern camera is,” Zuckerberg said.
Zuckerberg recalled telling his team a year ago that the path ahead of them wouldn’t necessarily be smooth. That they’d ship products missing many of the capabilities the company intended to develop down the road. And that they’d have to deal with whatever criticism came at them. “We're going to go through a period where people don't understand what we're doing. And don't understand the full vision,” Zuckerberg explained. “But, hey, that's the cost of entry to doing anything interesting.”