When Data Models and Rights Collide

My first position as a data scientist, was in the israeli intelligence, in a unit which is the equivalent of the American National Security Agency (NSA). I do not know anything about PRISM, NSA’s surveillance program, but my experience both in government positions and afterwards working for two big data companies helps me understand what are the drivers of the people who work on it.

The first thing to understand about this problem, is that it is not a big data problem. It is a huge data problem. Words cannot describe the amounts of data we are talking about. Storing all the communication information from Google, Facebook and the wireless carriers would require multiple data centers each the size of several football fields. Such amounts of data cannot be processed by people, instead, intelligence organizations rely on sophisticated algorithms to do most of the work to find this needle in a haystack they are looking for.

There are a lot of different challenges intelligence agencies are trying to solve. For this post I will focus on a single challenge – which individuals present a threat to national security. This problem is referred to in the machine learning world as a classification problem. Classification is the problem of identifying to which category an observation belongs, for example: given the level of force applied to a car’s door handle determine whether the alarm should go off, given an image determine whether it contains a human face, or given a person’s communication records determine whether she poses a threat.

The way to build a good classification model is to have a good training set. A training set is a set of records which you already know the answer for. If we use the face recognition challenge, the training will have some images that have faces and some that do not. It is very easy to come up with an extensive training set for the facial recognition challenge. Not so much for terrorists. There are just not so many cases of real threats to national security to learn from, and those that who exist, differ considerably from each other. A very small training set leads to a very inaccurate classification model.

In the case of national security, this is simply not good enough. Every threat that the model fails to detect might result in a very bad outcome. After 9/11 no one wants to be the person who let a terrorism event happen on his watch. This concern drives data scientist to constantly try to improve their models. Unfortunately, it is not always possible to improve models using the same training data. So when data scientists exhaust their modeling capabilities, they turn to get more data. But in the case of identifying people who don’t want to be identified, the most useful data is not one you would find in public records.

At a certain point, the fear of being responsible for people’s lives numbs the sense of right and wrong. People start to see the model and forget about its consequences. When every percentage point of improvement is another terrorist that will be detected before committing something terrible, privacy takes a backseat.

But privacy is not necessarily less important than safety. In some sense, privacy is safety and “National Security” is not a magic term that allows the government to trample over basic rights. In a democratic society, the government is required to do better than paternalize its people. Drastic times call for drastic measures, but these are not drastic times; so drastic measures call for drastic explanations.

disclaimer: This post does not represent the official opinion of anyone but myself. I have no knowledge whatsoever on PRISM or equivalent programs. This post is just my educated guess on what is driving the people who work on it.

You should follow me on twitter


Would Steve Jobs Pass Your Interview?

People are the most important resource a tech company has. It is the employees who come up with brilliant new products, develop them efficiently and find customers to use them. The best employees can be far more valuable than the average employee, and there is no better example than Steve Jobs – the man who transformed Apple into the largest company in the world.

It is very hard to assess the impact an employee has on an organization. We cannot perform an A/B test to compare the outcomes of a company with or without the same employee. But in the cases of some CEOs, sometimes, it is very hard to argue with their success. In 1997, when Jobs returned to Apple, the company he co-founded and was fired from 12 years earlier, it was worth a bit more than $3B. Just before his death, Apple’s worth reached an all time high, over $650B. In his 15 year tenure as the CEO, Apple’s market cap has increased by more than half a trillion dollars – making Apple the most valuable company in the world.

Steve Jobs didn’t start his career as the icon he is today. He graduated from high school with a 2.65 GPA and unlike Bill Gates and Mark Zuckerberg, who dropped out of Harvard to start their companies, Jobs dropped out of the much less prestigious Reed College. A candidate with his credentials would find it hard to get an interview at a fast food chain, let alone Apple, the company he co-founded and led.

Great companies require great employees across the entire organization – not only at the CEO position. This is why hiring is the top priority on many tech companies’ lists. Valve, the company that developed the Half-Life series, even writes in its handbook for new employees

“Hiring well is the most important thing in the universe. Nothing else comes close. It’s more important than breathing. So when you’re working on hiring—participating in an interview loop or innovating in the general area of recruiting—everything else you could be doing is stupid and should be ignored!”.

So why do most hiring processes look more like an assembly line?

Truth is, hiring is a very time consuming task. Successful companies will hire only about 1% of all the candidates who apply. Since interviewing 99% of the candidates results in a waste of time for the employees who interview them, companies are doing everything to optimize this process. These optimizations include filtering out all resumes that don’t have the right keywords, conducting automated coding screens that don’t require people and front-loading the most difficult interview to reduce the average amount of interviews per candidate. The other problem with hiring, is the asymmetry of the mistakes. Hiring the wrong person is much more noticeable than not hiring the right one. This puts the hiring priorities as follows:

  1. Go through as many candidates as possible for a given unit of time

  2. Avoid hiring the wrong people

  3. Hire the right people

This prioritization causes people involved in a hiring process to look for any reason to reject a candidate instead of looking for reasons to accept a candidate. This is very unfortunate. While a great employee can increase the value of a company by orders of magnitude, the risk of very bad employees can be mitigated by termination.

Much digital ink has been spilled on describing how a 45 minute interview should be used to assess certain skills. But this shouldn’t be the main discussion. The best employees are not necessarily the best at any particular skill. The most valuable employees usually stand out for their passion, enthusiasm and creativity – traits that are very hard to pick up in an interview.

In this information age, gathering information about candidates is easier than ever. My second degree network has more than 200,000 people. For everyone of these people, I can find at least one person I know that can tell me something about them. If you include other colleagues at your company who you are not connected to, but can give an honest opinion about a candidate, that network grows even larger. One of the best people I’ve worked with, was a DBA in a different part of the company, but his name kept coming up as this great person to work with. When I met him I found out that one of his hobbies is data science and while putting crazy hours on his understaffed team, he still managed to find some time to hack cool data products. If he would just pass his resume back then, it wouldn’t even get an interview, and I would be without a great employee.

At my current company, LinkedIn, we have a tradition to devote one day each month to a hackday. People work one day on a project and then have 2 minutes to present it to a panel of judges. Each month about 30 projects submitted, and some are just amazing for a day worth of work. If a candidate came to an interview and showed me one of those projects, I would probably hire her on the spot. At the very least, I would assign much more weight to this showcase of talent than any interview question I can think of. The average about hiring process at LinkedIn takes, in total, roughly the same time that people spend on their hackday projects. Allow people an opportunity to amaze you by hacking a project at the comfort of their own home and showing the results afterwards. With some creativity this method can be applied not only to engineers, but also to designers, marketers, salespeople and many other professions.

If the candidate is new to the area so he doesn’t have a network to vouch for her and the circumstances do not allow for her to showcase her skills. Another approach might be – “Try Before You Buy”. This is very similar to an internship, the employee works for a predetermined period and at the end, either gets a full time offer or not. This method is the best in terms of the amount of information gathered on the candidate, but can be tricky to perform. First, there is the problem of not many employees will agree to this conditions, but since the alternative is usually a flat no, the employer doesn’t have a lot to lose. The added benefit here, that the people who will agree are probably the most passionate about your company. Second, even the best employees might find it hard to prove themselves in a short period of time. This requires a very careful planning and to be honest and open with the candidate about what is expected from her.

Great employees are like great startups – a handful of them create most of the value. Steve Jobs, Larry Page or Mark Zuckerberg did not have the best credentials for entrepreneurs, but it didn’t stop them from building some of the most exciting companies in the world. Same goes for employees. The best people I have worked with did not have the shiniest resumes, but their value to their companies was a great deal higher than their average colleague. Avoiding hiring the wrong people does not result in hiring the right people, but in passing over the next Steve Jobs.

You should follow me on twitter