The Algorithm Excuse – an Exploit in the Wild

A few days ago I saw DHH’s tweet storm about how he and his wife have different credit limits on their Apple credit cards. When they tried to find out the reason, the answer they gave them was “sorry, it’s the algorithm.” The problem with this response is that it reveals no information regarding the real reasons, and this highlights a relatively recent legal loophole.

If you actively enact a discriminatory policy, that is illegal. Someone can point at your policy and sue you for discrimination. However, there is nothing stopping us from encoding discrimination into an algorithm. This is in fact fairly easy to do, as Cathy O’Neil explains in this Ted Talk.

If an organization does this and they get caught, all they need to do is act surprised and promise to look into it. Nobody is going to scrutinize their data or code. Unlike a public policy disclosure, data and algorithms are opaque and often secret. This needs to change. We do not need to know the data or the code, but we can feed a variety of cases to an algorithm (for example, a million artificial person profiles) and see what it does.

Until the laws force organizations to prove that their algorithms are fair, we will live in this legal loophole. If you run into this situation yourself, remember this post. Make the situation public, ask for transparency. Ping me on Twitter if you need help!

Nobody Knows Anything About Startup Ideas

In the world of entrepreneurship, ideas occupy a mystical place. Is it possible to have a unique idea? What makes a good idea? There is a common trope about how ideas don’t matter, it’s all about execution. Some people believe that great ideas look like bad ideas at first.

If we try to formalize this, a startup idea is simply a hypothesis about the following:

  1. I can build a certain thing.
  2. By the time this thing reaches the market, people will want it.
  3. The world will evolve in a direction that will favor this thing.

The first one is the easiest to prove. You either can built something or you can’t. The second one is a little harder because it’s contingent on the first having happened. Now, the third one is the really difficult one, because of Chaos Theory. Let me illustrate with an example.

It’s the year 1500, and two scientists are arguing about when humans will explore space. They make a bet. One of them believes it occur before the year 2000, the other believes it will be after. Fast forward to the 17th century, a young man named Isaac Newton is sitting under a tree, contemplating the clouds on the horizon. Suddenly an apple falls on his head, and he has a great idea regarding gravity. Excited, he gets up and starts walking home to write it down. Unfortunately he gets hit by a lightning bolt and dies. This sets physics back by several decades. In the year 2019, a human being rides a rocket outside the Earth atmosphere for the first time. “Space is no longer the final frontier,” someone tweets.

Was Webvan a great idea? Was it a better idea than AirBnb? Was WeWork a great idea? Were these ideas better than something I cannot name because it was quickly killed by metaphorical lightning?

We humans are proud creatures, and we want to believe we are good at predicting trends. The truth is that we are not. The best we can do is bet on a scenario that seems plausible to us, and then hope it unfolds. Survivor bias makes it look like prophets exist, but it is reasonable to ask ourselves: for every winner who gets to tell me their story how many of them are back at the drawing board, hoping to catch the right wave the next time?

The News Is a Liability – How to Be Informed in 2019

How easy it is to make people believe a lie, and [how] hard it is to undo that work again!” –Mark Twain

Only thirty years ago, it was normal to go for hours or days without being exposed to breaking news. One time in the early 90s I went on a week-long hike in the Andes. I remember that when I came back I was curious about what might have happened in the world while I was gone. I did not know it at the, but I would never experience this feeling again.

Now, every time I check my phone I find a story that a person I trust believes is newsworthy. Most often this comes accompanied with an emotional reaction, and an implicit assertion that the story is true. Here’s an example, a tweet mentioning Taylor Swift’s dispute with a music label:

By her account, Taylor Swift has been wronged by unscrupulous businessmen. She is urging her fan base to help her fight injustice. It is a very good story, and well told. I want to believe it.

I immediately resort to my usual modus operandi: if I care enough about something (it could be a military coup in Latin America instead of the tribulations of Tay, it does not matter), I do my own research. A few minutes and keystrokes later, I have a different opinion. The situation is a mess, and you could argue either side of the dispute. If I were selected as a civil juror for a potential court case, I would be able to reserve my judgement until seeing all the evidence. I have little doubt that both lawyers would be able to make a compelling case for their clients.

The ease with which anyone on the internet can manipulate my feelings has made me cynical. I do not want to help information spread unless I have made an effort to:

  1. verify that it has a reasonable chance to be true.
  2. convince myself that this information is useful signal in an extremely noisy world (I know this is subjective, but still better than not considering the issue at all).

I believe the world would be less noisy if everyone tried to apply similar principles. Wouldn’t you agree?

Technologies of Distraction

What do these three technologies have in common?

  • Amazon’s drone delivery prototypes.
  • Boston Dynamics humanoid robots.
  • Facial recognition for tracking people.

Answer: they are all flashy, impressive, easy to understand. Also, they are possibly not the most effective way of accomplishing a certain function.

  • Drone delivery is really expensive, risky and inefficient. It’s potentially useful in cases of extreme urgency where there are no roads. It really does not make sense for mass delivery of consumer goods. For this, a hybrid delivery network using all sorts of transportation technologies suited to individual scenarios is the answer.
  • Seeing a robot perform athletic feats is the equivalent of a computer beating us at chess. However, a robot can have any shape and any size. It can be a gun with wings, it can be a chainsaw on wheels, it can be both. A robot that can jump, dance and fight like a human is really just flexing.
  • Facial recognition is one of countless technologies to track humans. Off the top of my head you can track someone’s gait, body type/shape, style of clothes, voice, devices they carry, fingerprints, body temperature, modes of transportation, heartbeat, breathing rate, etc. We don’t hear many people complain about the prevalence of those other forms because humans just can’t relate to them as well.

The point: for every one of these marketing-friendly technologies that catch the public eye (and maybe infuriate activists) there are countless others that are more powerful, unnoticed and inevitable.

The New Order of Silicon Valley: Forget the Silicon Part

I moved to the San Francisco Bay Area over twenty years ago. Back then I could think of no other place to be; it was the kitchen of the internet. Obviously in hindsight it was a good call. I was fortunate to live through several waves of crazy growth: first, internet infrastructure and basic applications that needed to exist. Then social networks, and finally mobile computing.

Right now we are not in the middle of an innovation wave as meaningful as those, at least not in the areas that I know about. Sure, machine learning is advancing every day. Hardware keeps getting better. Autonomous vehicles will happen eventually. However, from the point of view of an investor or entrepreneur these are not the insane land grabs that we experienced in past decades.

Here are the trends I see in Silicon Valley today:

  • The “Silicon” part of the name no longer applies. Hardware is not made here anymore. The bulk of it is not even designed here. If you are interested in hardware innovation, you want to be in Asia. That is where the kitchen is, you cannot really design consumer-grade devices if you are not immersed in the manufacturing ecosystem.
  • VCs do not invest in hardware. These days they generally do not invest in technology per se, even. The typical company VCs adore these days is a service provider. They take some aspect of daily life (education, transportation, health, food). They use commodity hardware, and reuse as much software as they can. The role of most developers at these companies is to glue cloud services together. A startup that is building a scalable service should not be inventing new technology unless there is no other option.

So here is an obvious corollary. Like I just mentioned, if you have an idea for an interesting gadget you will move to a place like Shenzhen or Hong Kong or Taipei. You will build a prototype, prove your concept, work with a manufacturer to iterate your design until it’s mature enough to mass-produce. Either you will bootstrap the business or you will partner with someone local to fund it, because VCs won’t give you the time of the day. Now, let’s say hardware is not your cup of tea and you want to build services. Why be in Silicon Valley at all?

The number one advantage of Silicon Valley is access to money. We have an ecosystem of investors that does not exist anywhere else yet. So of course you want to be here to pitch your idea. Once you have VCs on board, what advantage is there to be here in terms of building a service oriented business?

There are many services for which California is a good starting point (e.g. marijuana delivery). However, labor in California is particularly expensive. In terms of hiring software developers, it’s the most expensive location I know of; you’re competing for the best and brightest with the infinite pockets of Facebook, Google, and friends. If I were building a service company today, I would:

  1. pitch my idea to Silicon Valley VCs.
  2. build an execution team wherever convenient (could be Asia, Europe, India, anywhere in the US, who knows).
  3. if applicable, establish a Bay Area presence for market development or sales.

I hope that I get to ride another crazy technology wave here in the Bay Area, but I am betting that the next one will take shape very far from here. This is why these days I am investing significant time into learning Mandarin and attending foreign startup pitches.

Startups and Privilege

One fine day, the NBA executives have a realization: there are not enough players trying to join the league (we must have entered a bizarre dimension but stay with me for analogy’s sake). What do these marketing geniuses do? Increase the funnel, of course. They tell everyone who will listen that the NBA is awesome, and that anyone who tries hard enough has a chance.

Of course, it is not true that anyone who tries hard has a chance. You can predict at birth that the odds of a given person making it to the NBA are 1/10000th of someone else’s. If both your parents are short, it is extremely unlikely that you will be tall enough. As of this writing, no player in the NBA is shorter than average.  The odds of making it to the NBA if you’re short are ridiculously small; if you’re seven feet tall they are pretty good (about one in ten).

What does this have to do with startups? It is pretty clear that the odds of startup success are dependent on factors that are outside of everyone’s control. Some just do not have what it takes to succeed, no matter how hard they try. Others have it relatively easy. I suspect that the sweet spot in terms of predicting first-time startup success (all else being equal) is some sector of the middle class that has:

  • enough desire. If you worry about paying the rent, the prospect of working ridiculously hard for a few years and becoming rich is more appealing than if you are already financially independent.
  • enough education. You cannot be successful in business without knowledge that privileged people (like yours truly) sometimes take for granted. You do not necessarily need a college education, but you do need to spend thousands of hours educating yourself somehow.

Someone who is really poor may indeed have a lot of motivation to get out of that situation. When I say “really poor” I do not mean someone like myself but penniless. That is not what I have in mind. I think of those born in slums, in poor countries without safety or basic infrastructure. Even in the US there are people like that.  It is very hard for me to imagine such a person spending hours every day reading or researching online.

Paul Graham gives the example of Brian Chesky as someone who did not come from money, and that is true. That does not mean that he was not born to a certain privilege. I do not know him personally, but his Wikipedia page says his parents were social workers in New York, and he was able to go to college. He is also male and white. Those things are sufficient privilege to me. Had he been black, or born in a poor country, his chances would have been substantially lower.

Paul proposes a statement to debunk:  “startups are for the rich.” Without further formalization, it is not possible to prove it or disprove it. It is obvious that you do not need to be rich in order to become a successful startup founder. However, it is also easy to prove (by sampling of successful founders) that privileged backgrounds are over-represented.

I have one final beef with this whole thing. At the beginning of the post I imagined a universe in which the NBA believes there are not enough applicants, and this sounds absurd. Why do people believe this may be true of startups? Why do we need to motivate potential entrepreneurs with “you can do it too”-style pep tweets?

I believe the opposite is true. What I see in Silicon Valley is way too many people trying to become rich, and not enough trying to fix the problems that are not permeable to capitalism. Too many people tell themselves a story of trickle-down economics and wealth creation that benefits everyone. All you need to do is take a look at the crazy degree of homelessness in the Bay Area (or the state of public health in the US) to know that it is simply false. Some social problems take decades to solve. Some of them may not generate returns on investment that would satisfy a venture fund. I would like to see more smart and well-meaning individuals get into politics instead of playing with technology and venture funding to “make the world a better place.”

The Pitfalls of Software Complexity, Part [NaN]

Let’s say you’re writing code to print out three 3D shapes: a cube, a cylinder and a cone. You test it. Each one takes a few minutes to print, all look good. Ship it! A few days later, you get a bug report. Once every so often, a rounded cube comes out.

Well, you know more or less where to look. The problem must be in the cube code, right? You find the issue, fix it, go home. Leave the program printing cubes all night. The next morning look fine and sharp. You build a nice stack of cubes in your cubicle, all is good again.

A week later you get a change request: now objects need to come out in three sizes: small, medium and large. No problem, you implement it. Now you can produce nine different objects. Everything works fine, the customer is happy. Obviously the next thing they want is colors. Oh, and densities. Let’s say you compromise on providing four colors: blue, red, yellow, black. And density can be 1 to 5. Fantastic, now your code can create 3 x 3 x 4 x 5 = 180 different objects. It could be a large yellow cylinder of density 5, or a small black cube of density 2. Lots of things!

A couple of weeks go by. Your manager comes in with a puzzled look on his face, and tells you that some people have been reporting donut shaped objects once in a while. Sometimes they are green. Out of hundreds of thousands of objects printed the vast majority have been fine, but there are at least twenty of these (mostly green but sometimes orange) donuts out there. Of course you cannot reproduce this bug, because it would take you weeks of nonstop printing to encounter one of these cases on average. You have to dive into the code.

You start finding some weird edge cases. Sometimes a cylinder could become a donut, but only if the density happens to be in a certain range. But green? What’s up with that? How do densities change colors?

After two weeks and -14 pots of coffee [uint8_t coffee_pot_counter treated as signed] you think you’ve found the issue but you can’t be 100% sure. You test as much as you can, after one thousand prints no weird objects show up. Of course that doesn’t mean much. By the time you’re reading to ship what you believe is the fix, there is a new feature request: multicolor patterns!

You ragequit. A new guy gets your job. He looks at your code and has no idea what’s going on. He starts rewriting it from scratch.

A new requirement comes in. Now the customer wants different materials. And also a “shape morph feature.” Your ex-manager and your replacement can’t agree on how to implement the feature, or what exactly it should do, so they have a conference call with the client. The client says “well, the math is super simple. Take two shapes, for every cubic pixel in the intersection compute the average of the properties of the two and print that!”

In the meantime, you went off and started your own company. You now print exactly one object: a ball. It’s orange, and it has a unique texture. Customers love it; it’s happy, it’s fun. You sell a million of them. Some people want different colors and sizes, you say no. Some of them buy yours as is anyway, some wait for your former employer’s announcement of a bug-free orange ball any day now.

Do not taunt happy fun software complexity!

From Zero to Building and Flying Drones in a Month

On October 21st I bought this at a Toys “R” Us because it was on sale for $25. If you have never flown a quadcopter and have the money to spare, I suggest that you buy one right now (today I’d recommend this one instead, it’s more stable and much better outdoors). At first I could barely get it to hover in place. I crashed it countless times until it stopped working, so I bought another one. I was fascinated by the fact that this tiny thing can take off vertically (even from my hand), hold its position in space, and also move really fast in any direction.

I had an intuition for how quad rotors fly (watch this video for a beginner-friendly explanation) but I wanted to learn more about the technology. As a teenager I’d had a brief period of playing with remote-controlled planes and boats, which back then used combustion engines. Batteries had terrible power/weight ratios so electric motors were rare for flying things. Having precise control over four combustion engines pointing upward would have been difficult, so that’s perhaps why quadcopters didn’t get popular until Lithium polymer batteries (LiPo) became affordable to hobbyists. I started checking out the multiple subreddits dedicated to multicopters, quadcopter specifically, and diy drones. I quickly learned the following things:

  • There are two kinds of quadcopters: toys and hobby/pro ones. The first kind are very cheap (typically $20 to $100 ready to fly), come with their own proprietary transmitters, often have a camera. They are generally fragile and mostly harmless because they are very light.
  • Non-toy quadcopters can be further split into two categories: ready-to-fly and DIY. The most popular one in the first category is probably the DJI Phantom in its various models starting at around $500. These copters are for people who are mostly interested in aerial cinematography. They are very easy to fly: they come with GPS and know to hold their position in the sky like a cursor on a screen. You don’t get turned around flying them, even if they rotate the sticks continue to move them in directions relative to their original heading by default. They often have a function to come home when they are running low on battery.
  • DIY multicopters are significantly more involved. First you have to decide what kind of aircraft you’d like to build. One of the most popular kinds is the 250 (distance in millimeters between diagonal motor shafts). It’s very fast, so typically people use them for FPV racing/freestyle or acrobatics (watch those videos).
  • Most people start flying LOS (line-of-sight) as opposed to FPV (first-person-view) because FPV gear is not cheap. Some pilots who are used to the videogame-like dynamics of FPV don’t know how to fly LOS.

Having said all that, an actual drone is something that flies itself. If it has a human pilot, technically it’s not a drone. This week I’m in the process of building a proper drone; I started with a cheap 360mm quadcopter using parts from HobbyKing as described here. I’m replacing the controller by an APM with a GPS/compass module ($70 for both). This hardware supports Mission Planner, which lets you program the aircraft to go do something you want. My next goal is to have it take off, move a short distance, hover for a few seconds, and come back to land where it started.

Also, last week I finished building a 250 “racer” quad with FPV capability that I’ve flown a few times. I’m relatively confident at LOS flying, but FPV seems much harder to me.

This is me flying LOS:

And FPV:


Why We Hate Ads

If you are aware that you want to find something, you search for it. There was a time when searching wasn’t easy, so you had to rely on your memory cache of known things. You could argue that ads were useful in filling that cache, particularly for things that you had not needed or experienced up to that point. Now that we have instant online search, that problem is no more.

Sometimes you’re unaware that you want to find something, but you’d be pleasantly surprised if a genie told you “here’s what you didn’t realize you wanted.” That’s what predictive search is about. A perfect predictive search agent would either shut up (perhaps most of the time) or give you extremely relevant results in anticipation of a query that you haven’t articulated yet. Unfortunately, online ads are not that genie. If that were their purpose, we could say that irrelevant ads are a poor implementation of predictive search.

Some argue that with better ad technologies there will be fewer mismatched ads: it will become easier to match products to consumers at the right time. That assumes that every product for sale can be matched to a consumer at a given point in time. Why should that be true? One of the uses of advertising is discovering whether people are interested in a product or service. Suppose a chef from a different galaxy came to our planet and opened a restaurant. Not being sure of the tastes of humans, he starts with his three favorite items on the menu: fish, steak, and uranium. It won’t take our alien restaurateur very long to figure out which item he should stop offering. There may be large numbers of things advertised that nobody cares for; a product genie would immediately tell the advertiser “don’t waste your money, I’ve analyzed humanity and nobody in their right mind wants this.”

Because that product genie does not exist, advertising will continue to be a mechanism for (in)validation of products for the foreseeable future. As a result, we will continue to see ads that we’ll perceive as annoying or irrelevant. No product-consumer matching technology should try to find hungry people interested in ingesting uranium.

Startups Compete Against Background Noise

The traditional competition between companies has been about the price and quality of their products. For most startups that haven’t reached product-market fit, that shouldn’t be a real concern. As Justin Kan puts it:

But why and how do startups compete against no one giving a shit? For one, starting a company has never been easier so there are lots of them. “I am the founder of a startup” says little about a random person you meet in Silicon Valley. Imagine that Wimbledon expanded from 128 to 65,536 players. More people would be able to say “I played at Wimbledon” but the average participant would not have a sponsor or be able to make a living as a tennis player. As this “Wimbledon expansion” slowly took place, the traditional competition moved to a later stage. In the beginning, startups compete against other businesses in areas such as:

  • attention. Because there are more products and services that ever before, there is a significant chance that nobody will learn about what you’re doing. In that sense they don’t compete just against other startups or large companies; they compete with the background noise of the online world that all of us generate.
  • employees. If your startup makes backup cameras for bicycles (“parallel parking your bike is now easier than ever with BikeSeatCam!“), you want to hire the people lots of other companies in completely unrelated markets also look after.
  • funding. That’s perhaps the closest thing to a startup league competition. Investors must decide where to put their money, and often choose among startups that are unaware of each other’s existence. Still, this is the fiercest competition most startups will face. You can study the weaknesses of a specific opponent and perhaps exploit them, but when competing against the faceless crowd there are no shortcuts: you must be significantly better than most.
  • customers. That comes after surviving the other types of competition long enough to have a product out in a validated market. Most startups will never have the luxury of reaching this stage.

Many founders I’ve met have a hard time understanding the nature of this competition, and adjusting their thinking to compete in those terms. For example:

  • Getting attention is an art, and many underestimate how important it is. If you’re doing something wacky (“our drones keep your cellphone in front of your face so you can watch Netflix while you jog outside!”) or controversial (e.g. Peeple this week), attention comes more easily. If your product is relatively boring, you’ll have to figure out how to get the attention you need.
  • Attracting employees is a multidimensional task: some people are lured by the promise of working with known industry figures, some want to take on a sexy project, others value the potential of the business. What you have to do to attract the right employees may be unrelated with the nature of your business. For example, your customers may not care at all about your software stack but your potential employees will obviously prefer some tools over others. Is there anything that makes your company a unique place to work?
  • Fundraising is literally selling. A startup is selling a part of itself in exchange for funds to operate. Investors are people who buy investments in the same way that consumers buy daily necessities. Even though this is obvious, I keep meeting startup founders who believe that investors are like their rich uncles. In reality, investors are more like shoppers in a giant grocery store filled with unknown stuff: should I buy some gluten-free Twitter substitute or try the “I can’t believe it’s not Uber” instead?

If you’re reading this post you’ve probably heard the YCombinator mantra “make something people want.” Even though it has a zen-like air of wisdom, I find it as actionable as “score more points than your opponent.” There are countless reasons making something people want is really hard. One of them is that if you startup cannot rise above the background noise and get the right amount of attention, it will never find out if it’s on the right track. For startups, loudness is a virtue.