Facebook has submitted a patent application for technology that would predict who your family and other household members are, based on images and captions posted to Facebook, as well as your device information, like shared IP addresses. The application, titled “Predicting household demographics based on image data,” was originally filed May 10, 2017, and made public today.
Facebook did not immediately respond to a request for comment, but the patent suggests that the company is interested in exploring the technology, which is intended to help Facebook target advertising more effectively. After the story's publication, a Facebook spokesperson said, "We often seek patents for technology we never implement, and patents should not be taken as an indication of future plans.”
Facebook submitted the application before this year’s security and privacy scandals — Cambridge Analytica, a massive hack, and backlash against its recent hardware product — but the patent’s publication comes at time when the social media giant is grappling with the public’s growing distrust.
The system Facebook proposes in its patent application would use facial recognition and learning models trained to understand text to help Facebook better understand whom you live with and interact with most. The technology described in the patent looks for clues in your profile pictures on Facebook and Instagram, as well as photos of you that you or your friends post.
It would note the people identified in a photo, and how frequently the people are included in your pictures. Then, it would assess information from comments on the photos, captions, or tags (#family, #mom, #kids) — anything that indicates whether someone is a husband, daughter, cousin, etc. — to predict what your family/household actually looks like.
According to the patent application, Facebook’s prediction models would also analyze “messaging history, past tagging history, [and] web browsing history” to see if multiple people share IP addresses (a unique identifier for every internet network).
In one specific example, the model looked at an image a user posted with two females tagged, “#my_boss_at_home,” and another image with a young girl marked “my angel.” Facebook predicted that there were three people in the household, including the male user, and two females, “who are likely the male user’s wife and daughter.”
The social media giant didn’t specify exactly what kind of demographic information its proposed system would predict, other than the number of people in a household, but its data policy specifies that demographics could include gender and age.
The application makes clear that the information is intended to help Facebook target advertising more effectively: “Existing solutions of content delivery to a target household are not effective ... Without such knowledge of a user’s household features, most of the content items that are sent to the user are poorly tailored to the user and are likely ignored by the user.”
In a flowchart showing how the data would be collected and used, the final step for the household demographic data would be to “provide for display content items targeting the user based on the predictions.”
In June, Facebook added an ad-targeting option that allows businesses to target an entire household at once; it determines that data using shared last names, home locations, check-ins, life events, and where people connect to the internet, a Facebook spokesperson told Marketing Land.
Facebook collects a lot of data about you to feed its personalization algorithms, which serve you content on your news feed and ads it thinks you’ll be most likely to click on. Considering recent reports about Facebook’s struggles to protect the massive amounts of personal data it has collected from its users, that revelation is probably not a surprise.