Here's something you probably didn't know: Facebook has a team of employees who read your private messages if they have been flagged by an automated tool. The tool searches for content that appears to violate their terms of service, namely malicious (infected) URLs or child pornography. It's imperfect, of course — that's where humans come in.
If a private message is flagged, actual people will jump in and read it. If there is something that could be illegal — particularly regarding child exploitation — those people contact law enforcement. The intent here is clear and defensible, yet the fact remains: All that stands between your "private" messages and the eyes of a stranger is the snap judgment of an algorithm.
"There are strong internal controls around the use of these tools to prevent misuse and abuse," Facebook told BuzzFeed, "and stringent guidelines for the way we cooperate with law enforcement."
It's not just Facebook. Dating site OkCupid has humans read private messages that have been flagged by its users, as reported by San Francisco Public Press.
Even more troublesome, OkCupid's readers are sometimes regular users, not employees. These users are deputized to help hand the large volume of "flagged" messages and can view ongoing conversations that may contain private information, whether or not the flagging users' claims are legitimate.
Some moderators also share them publicly on blogs such as ThatsNotOkCupid.com and NotSoNiceGuys.tumblr.com.
While some aspects of content moderation can be easily automated, many can't. Google notoriously uses freelancers to comb through inappropriate public content, as does Facebook. But the issue of moderating privately shared content is a more complicated one; users should know that a Gmail message could easily be exposed by subpoena and that it is scanned for keywords to serve advertising, but would be mortified if Google employees were known to read "flagged" images. (Google did not respond to a request for comment on this story; neither did OkCupid nor its parent company Match.com.)
"I wouldn't characterize it as 'access,'" OkCupid told the SF Public Press. "It's not that OKCupid is doing something. It's the recipient of the message. The recipient of the message has said, 'OK, this is noncompliant,' and has chosen to share it with OKCupid."
Twitter tells BuzzFeed that it doesn't monitor direct messages either through automated tools or humans.