Social media wants to know. They collect data indiscriminately and ceaselessly. Facebook, Google+ and other social networks have employed various tactics to get to know their users – us – better over the years. While Google+ is comparably harmless, Facebook has always been rather aggressive.

Facebook’s methods have included, but were not limited to, flat out asking what our favourite movies or bands or restaurants were, then asking us if we’ve seen a particular movie, rolling out plugins for other social networks such as GoodReads and TripAdvisor as well as asking variations of the question «You liked $thing, you might like $relatedThing, too». All this in order to target advertising better and sell our data to advertisers and whoever else pays the money.

Correlation such as asking if we like something related because we liked something else is a thing of the past. Antiquated, even. With the rise of increasingly faster internet connections, new technological means become available. And we’re at the point where user input is not necessary anymore, perhaps even unwanted. The latest in this regard is a Facebook feature that is currently being rolled out in the United States: Audio discovery.

Audio Discovery – Spying on You For No Good Reason

Audio discovery is not a new concept, nor is it unique to Facebook. But what sets Facebook’s implementation apart is the general use and details of it. In Google’s Android, there’s Google Now and in Apple’s products, Siri is embedded. They operate on the same principle. They record what users are saying and perform useful tasks such as finding the best route to a nearby place or playing music on a streaming app. In recent months, the effectiveness and intelligence of these features has progressed insanely quickly, so it’s no wonder others want to have a piece of the cake.

At Facebook, the audio discovery feature has no name. It’s simply part of the mobile applications in the US for now. The international launch is expected soon. What the feature does is simple. When a user types a status update, Facebook turns on the phone’s microphone and listens to ambient sound, identifying songs, TV shows or movies playing in the background. When the status is posted, a sentence like «is listening to Hol Baumann – [Human] is appended to the user’s status. Their friends can then listen to a 30 second segment of the song or watch a trailer and are then linked to purchasing options.

Facebook’s Listening – A Long History

The audio discovery feature by Facebook is by no means a new invention. The earliest public mention of it date back to early 2014 when it was officially announced. However, the feature was apparently not rolled out immediately and went back into obscurity. It was only recently that it resurfaced and caused the exact same waves it did when it was first announced.

Facebook says the feature will be used for harmless things, like identifying the song or TV show playing in the background, but by using the phone’s microphone every time you write a status update, it has the ability to listen to everything. Not only is this move just downright creepy, it’s also a massive threat to our privacy. The feature is opt-in, but many won’t even read the warnings. – SumOfUs.org Petition

In addition to that, many a user has suspected that Facebook listens in even if the recording function is off. That is a theoretical possibility, but gets denied by Facebook themselves.

Myth: The feature listens to and stores your conversations. Fact: Nope, no matter how interesting your conversation, this feature does not store sound or recordings. Facebook isn’t listening to or storing your conversations. Here’s how it works: if you choose to turn the feature on, when you write a status update, the app converts any sound into an audio fingerprint on your phone. This fingerprint is sent to our servers to try and match it against our database of audio and TV fingerprints. By design, we do not store fingerprints from your device for any amount of time. And in any event, the fingerprints can’t be reversed into the original audio because they don’t contain enough information. Myth: Facebook is always listening using your microphone. Fact: Nope, if you choose to turn this feature on, it will only use your microphone (for 15 seconds) when you’re actually writing a status update to try and match music and TV. – newsroom.fb.com

Still, almost immediately after the new feature was announced in 2014, users flocked to online activism to force Facebook’s hand to have the feature removed. They called the feature creepy and didn’t like the idea at all.

One year later, society is far more used to sounds triggering events on our devices and with the advancement of technology, these commands can be increasingly complex and useful. The public opinion on voice control is changing rapidly with the advancement of the technology and the comfort it brings.

Attacks on Audio

Supplying commands to a device using nothing but a human voice carries a lot of vulnerabilities. The big one, of course, is surveillance.

If an application gets unrestricted access to the devices microphone, it’s essentially bugging the room the device is in. Hijacking the application or making a malicious app that listens in can lead to massive amounts of information disclosure. Every place the smartphone or tablet is at can theoretically be put under surveillance.

This is not just interesting for law enforcement, but also advertisers and spies. The latter are not the James Bond type, but people after confidential data of a company, for example. Many managers carry their phone with them at all times seeing as they need to be able to respond to incidents above a certain threshold immediately. These phones are present during a lot of highly confidential discussion. Being able to access the microphone and eavesdrop from a remote location could yield quite the competitive edge.

Thinking more along the lines of James Bond, government spies could also benefit from this kind of surveillance, not just during times of conflict. In fact, there has been a case of a government spying on a friendly nation: The whole affair surrounding the Merkelphone. Edward Snowden’s leaks have revealed that the NSA was spying on German Chancellor Angela Merkel’s phone conversation on her private mobile phone that she also used for political business. While this strained the relationship between the two countries, the investigation was dropped and Angela Merkel as well as the German government use encrypted phones built by the Deutsche Telekom subsidiary T-Systems.

The other obvious vector of attack is that of interference. So far, all commercial grade devices with audio features do not distinguish between users. This means that while user Alice is giving her phone commands, attacker Mallory can yell things from a distance away and the phone picks it up as if it was a legitimate command. While this most likely won’t lead to much damage other than general annoyance, it will become a bigger threat once voice activated devices are able to carry out more sensitive tasks.

Mount Your Defenses

There are a number of suggested defenses against the attacks, the majority of them from the field of Operations Security, also referred to as OPSEC. The most obvious seems to be the most effective: If a meeting deals with confidential materials above a certain threshold, there shall be no phones in the room. This low-tech solution is the most convenient and the most reliable.

The next best alternative would be censoring language. During meetings, certain terms such as names of clients or interest groups are not to be mentioned. Alternatively, they could be referred to by a code name. This can be very cumbersome in certain cases and lead to a lot of misunderstandings. Leaving the phones outside of the room might be easier and is more secure.

Creating a distinct voiceprint of the main user’s voice as well as that of a few trusted people can block interference by other users. In a household, this could mean that only the residents can give the appliance commands. In an office, the only authorized users would be defined by their importance to the operation of the appliance. This would require users to pre-register their voices though, using certain markers that would most likely be discovered by repeating a sentence or several.

In Facebook We Trust?

The question here isn’t whether or not Facebook can listen to our conversation outside of the constraints of only listening when typing a status but whether or not they do it. As it stands right now, it’s a reasonable assumption that they do not. Or do they?

There are quite a few reasons for this:

• It’s illegal in most countries
• It’s morally and ethically so questionable that Facebook can’t ignore the concerns by people
• It’s morally and ethically so questionable that no advertiser is likely to pay money for the use of such data without proper marketing research data, promising healthy profits.

While Facebook generally seems to have a rather weird concept of moral and ethics – allowing hate speech but not nudity, indicating a very American understanding of the concepts – they have their occasional bouts of privacy concerns. That makes sense, as it does not pay off to antagonize your customer base that has to be loyal to the company and not wander off to use one of the other big social media networks. After all, Facebook is aware that the users are not their clients, but the product being sold. But it doesn’t mean that the users can be treated too badly. A certain degree of bad treatment and indifference towards user needs is okay, but drastic changes and too much controversy drive their product to the competition.

So while Facebook might not be willing to listen to everything that is said in the vicinity of a device capable of features that carry functionality by audio interception, it is far from impossible for them to implement that.

Just in case Facebook can’t be trusted, which it probably shouldn’t be, developer Daniel Velazco has developed an Android application that isolates Facebook from the rest of an Android Device.

How it Works

There are two different ways audio is being intercepted by modern smartphones and tablets.

Facebook’s method is the more passive one, where the microphone is presumably turned off unless a very specific number of criteria are met. In Facebook’s case, the criteria look like this:

• Option for listening feature enabled
• User must be typing a status update
• User must be typing

In addition to that, Facebook claims that they’re only recording for 15 seconds and that these recordings are not stored anywhere. So they’re not actively analyzing what users are saying and storing it away for later use, despite rumours to the contrary.

The other method is the one that can be activated in Google Now’s settings or that is the default with the Amazon Echo or Apple’s Siri. Devices with these settings are always listening but only react to you and presumably open an Internet uplink only then when they hear a trigger word. In case of Google, it’s Okay Google and the names of the products. Alexa in case of Amazon Echo and Hey, Siri on Apple’s devices. Once the devices have heard their trigger word, they start to not only upload your voice print to the cloud servers of their owners but also to analyze massive amounts of data that will hopefully produce an answer that is satisfactory to users.

Privacy Concerns Galore

As it stands, users have no way of verifying that devices and services using Facebook’s method actually do what they’re told. There are numerous cases where the users believe that the Facebook app is listening at all time.

We found a roach in our apartment (very common in my part of the world). We were complaining about it and the need for pest control. Within 10 minutes I had an ad on my Facebook feed for a local pest control place. Never ever ever have a seen a similar ad on Facebook. The rationalist in me wanted to say that it must have to do with the time of year and the frequency of roach problems in my area, but the coincidence was enough to give me pause. – User NewHoustonian on Reddit’s Jailbreak subreddit.

This is apparently not an isolated incident. People in the same thread report that their Facebook app has picked up on all kinds of things it shouldn’t have picked up on, such as a movie with a specific quote, a special kind of almonds and diarrhea. All these users swear that they did not give any other indication of the issues Facebook picked up on other than the spoken word, so it must have listened. However, data correlation does wonders and with enough data, the predictions can be fairly accurate. That is not to say that the Facebook app wasn’t listening, but it is merely a reminder that there are other ways. Alternatively, the Baader-Meinhof Phenomenon, also known as Frequency Illusion, could be responsible for these strange occurrences. The Phenomenon describes the mind’s trick that once a person cognitively notices something – in this case a conversation with certain topics – terms related to it appear seemingly everywhere.

Still, even those 15 seconds the app admittedly is listening to could theoretically lead to a public nightmare. Let’s assume you’re talking to your mother about an illness. Or to your girlfriend about what you want to do under the sheets later. Generally, imagine yourself in any situation where you’ve typed a Facebook status and try to remember what you were talking about. At least theoretically, Facebook has the ability to analyze this conversation. So do Google, Amazon and Apple with their devices.

Protection Might Be Avoidance

It is hard to protect users against snooping, mainly for two reasons. The main reason might be that users simply do not care and think of Facebook and their competitors as inherently benevolent, despite there being numerous instances where it’s proven that this is not the case. One such case is the fall of Swiss local politician Sarah Bösch who, after being detained by police on a DUI charge, took to Facebook to vent. Her provocative statements caused the media to investigate other aspects of her life. Ultimately, this led to her being thrown out of her party. Running as an independent politician with a scandal in her background, Bösch is highly unlikely to have any chances at being elected for anything.

The other one is that humans as a species are a curious bunch. The reason why we as a humanity have come so far is because people couldn’t resist the urge of What happens when I do…?

The solution, however, just might be in the constellation of Facebook not wanting to treat its users too badly and the users resisting the urge to wonder about new features as well as forgoing a bit of convenience. In simpler terms: Just uninstall the app. No online petition or news report will have the effect that a sharp drop in users has. Luckily, at this point, the feature is still opt-in and not available outside of the United States so – right now – we’re still safe.

About the Author

Dominik Bärlocher has been working with IT subjects since 2006. The journalist relied on his affinity for all things IT during his tenures at news papers and benefited from it. At scip, he conducts OSINT researches and is an expert at information gathering.

Links

• http://action.sumofus.org/a/Facebook-app-taps-phones/
• http://en.wikipedia.org/wiki/Edward_Snowden
• http://newsroom.fb.com/news/2014/05/a-new-optional-way-to-share-and-discover-music-tv-and-movies/
• http://onforb.es/1mbkC38
• http://read.bi/1ow9SfI
• http://wat.is/-yYldzAzA
• http://www.amazon.com/Amazon-SK705DI-Echo/dp/B00×4WHP5E
• http://www.bbc.com/news/world-europe-33106044
• http://www.tripadvisor.com
• https://play.google.com/store/apps/details?id=com.danvelazco.fbwrapper&hl=en
• https://plus.google.com
• https://redd.it/2×0qm9??
• https://redd.it/3nxjwt
• https://www.facebook.com
• https://www.goodreads.com
• https://www.reddit.com/user/NewHoustonian
• https://www.telekom.com/media/media-kits/105252
• https://www.youtube.com/watch?v=1m4-i9t_ffk»