Should your startup to do remote development?

Many people ask me what I think about running a startup in which the founder(s) are in Silicon Valley and the development team works from another city / country. This is what IndexTank did, and we were successful. The answer is easy: this setup helped us as much as doing hard drugs helps a rockstar. It’s a handicap, you can succeed in spite of it but I recommend avoiding it if you can. I defer to the words of Steve Jobs, when he was explaining the design of the Pixar building to encourage random meetings:

Despite being a denizen of the digital world, or maybe because he knew all too well its isolating potential, Jobs was a strong believer in face-to-face meetings. “There’s a temptation in our networked age to think that ideas can be developed by email and iChat,” he said. “That’s crazy. Creativity comes from spontaneous meetings, from random discussions. You run into someone, you ask what they’re doing, you say ‘Wow,’ and soon you’re cooking up all sorts of ideas.”

Youtube please

Youtube is one of those websites I can’t live without. A few years ago it used to be hip to declare that you didn’t own a TV. I do own one but I rarely use it; I watch most of my movies / TV shows / music videos on my computer. Not owning a TV today simply means “I couldn’t care less for cable content or a big screen.” I would guess that at least two thirds of the video content I consume on a daily basis comes from Youtube.

Naturally I share some of this content on Twitter. Over the past couple of years I’ve become increasingly irritated as my friends in different countries tell me that a particular video cannot be seen in their country / region. The explanation given by Youtube is as succint as it gets:


Of course I’m fully aware of the reasons for this, and as much as I disagree with this policy of segmenting the internet by country borders that is not the point of this post. Rather, I’m asking for something much more straightforward:

Youtube: I want to know at a glance whether a video has no global viewing restrictions of any kind, or is available only in certain countries. In the latter case, please let me know what countries it’s available / not available in. At the very least let me check if it can be seen in a given country (e.g. Argentina, Spain, the US or the UK where most of my friends live).

Make it part of the api if you must, I will build an app on AppEngine if necessary. I don’t care.

If you enjoyed this rant, follow me on Twitter so I can share awesome videos with you 🙂

Edit: Thanks to walrus from hacker news for this tip:

curl -s | xmllint –format – | grep media:restriction



If you’re not famous, joining Twitter is painful

Twitter is great if you are a celebrity. Jerry Seinfeld joined a few days ago and already has 180k followers. But what if you are a random person who joins Twitter today? What’s in it for you?

Let’s say I’m Charlie Bucket and I see Seinfeld’s page. Awesome, I want to join Twitter and start interacting with people. I create an account and I’m shown a screen like this:


Ok, I know Chris Rock but I don’t want to follow him. Who are these other people? Take me out of this screen. Skip. Next step, Interests:


Ok. Chocolate is not there. Golden Tickets? Nintendo? Boring. I have an option to search by interest so I try chocolate. I see a number of accounts that supposedly will talk about chocolate. But, can I chat with them? Will they care about me? I click on the first suggestion (@divinechocolate) and see that he/she/it has 8k followers and doesn’t follow a lot of people. Ok, I’ll skip this section too.

Next step, friends. I don’t want Twitter to start looking into my Gmail and follow people who are on Twitter. Let’s jump back to reality for a moment. I’m not really Charlie Bucket, I’m creating this account for fun. My friends won’t know what’s going on it if Charlie Bucket follows them. Ok, back to Charlie Bucket. I click on “Search Contacts” for Gmail and I get a Fail Whale. Whatever, I’ll find my friends later.

So, after confirming my account this is what @charlucket’s home page looks like (after he unfollowed Seinfeld because he cluttered his timeline):


Charlucket’s avatar is also an egg, which is appropriate because he has 0 followers, 0 tweets and 0 listed.

So much for telling the world what’s happening! Nobody’s listening. So Charlie drops out of Twitter never to be seen again. Opening random Wonka bars is a better way to spend his time.

What I would do if I were Twitter

First, let’s admit that part of what makes Twitter interesting is egos and follower counts. There is clearly a pecking order on Twitter that doesn’t exist in other social networks. If you have 10k followers you can tell off some loser with only 500. When you join Twitter you are the new dorky kid in school. This is a direct result of how Twitter chose number of followers as an indicator of karma, along with number of tweets. That’s ok and shouldn’t change, but it does create the problem that new users need a way to build karma, and currently that doesn’t exist.

As a new Twitter user, it’s extremely hard to gain followers. I know, I’ve tried. People who happily respond to my tweets with my real account ignore witty comments from my “nobody” ones. People I’ve never heard of come and follow my official account. Nobody follows @charlucket.

I don’t know what Twitter’s business objectives are, but let’s assume that they want to create network connections and incentivize people to remain engaged. Here’s what I would do for a new user:

1) Don’t expose them to celebrities, or care so much about showing them people to follow. It’s ok to suggest a few people, but two screens of strangers who won’t give me the time of the day is too much.

2) Force me to introduce myself to the world. Do an ice-breaker. Let me put something out there about me that may help others find me and follow me.

3) Help me find newbies like me, who want to follow and be followed! It’s not hard to suggest pairs of newbies who should follow each other based on location, language or a few things they just said. Help me bootstrap my audience.

Remember Twitter, people want to be heard. There’s enough broadcast crap in the world. Celebrities don’t need another soapbox. This is why you should be very afraid of Google+, they give people an instant audience. That’s very powerful.

Speaking of, go follow @dbasch on Twitter now 🙂

Survived an AWS outage? Don’t gloat!

Remember a few weeks ago when the Great EBS Outage took place? Everyone became a software scalability architect overnight. Lots of posts were written about how most startups didn't know what they were doing by using ebs. Netflix in particular was the talk of the town. Their service wasn't affected at all because they didn't rely on EBS. While sites like Foursquare, Reddit and Quora were down, Netflix boasted that:

For Netflix, the short answer is that our systems are designed explicitly for these sorts of failures. When we re-designed for the cloud this Amazon failure was exactly the sort of issue that we wanted to be resilient to. Our architecture avoids using EBS as our main data storage service, and the SimpleDB, S3 and Cassandra services that we do depend upon were not affected by the outage.

Awesome. So last night, all those other sites were up but Netflix was down. What happened? SimpleDB was out cold for three hours



To Netflix's credit, their tone of their post back then was measured. Other companies which essentially lucked out to different extents were not so restrained. Lesson learned: if your service survives a major outage, don't rub it in the faces of startups who make their best out of their scrappy budgets!

Marketing for Hackers: a 5-min intro, 97% BS-free

1) Marketing is not necessarily evil.

If you come from the hard and logical side of coding, the word marketing has an aura of evil. It conjures images of blabbering corporate droids that make you want to play buzzword-bingo during all-hand meetings. On Hacker News I see comments such as:

Barf. Sad, to see Hacker News being used as a marketing channel.

The parent post of that link talks about how to use channels such as HN to maximize the launch of your startup. It’s a start, but marketing for a startup is about much more than just the launch. Is an ongoing tactical and strategic process. I’ll share a few quick things that have worked well for IndexTank so far. Disclaimer: I’m not a marketing expert, although I play one as the C*O of IndexTank. What I describe here is the result of trial/error plus valuable help from some experts (hi Martina).

2) You are already doing marketing. Might as well do it right.

First off, you are already doing marketing whether you like it or not. Are you trying to get page views? Do you talk about your product online? Do you go to meetups? Do you tweet/blog? All those are marketing activities. As the logical being you are, you do all that for a reason. Why not roll it all into a focused strategy that you can measure?

Ok, so you want to have a strategy. Before you have a strategy you need goals. I assume you know what you want: become the leading social network for Nepalese zombies, sell for one hundred billion dollars, whatever. You have assumptions about where you need to be in order to declare victory: you want to be making X dollars per month, have Y users, Z visitors to your site, etc. Let me introduce you to AARRR: Dave McClure’s startup metrics for pirates. Check it out and come back.

3) Measure, learn, iterate.

So you’ve been doing a lot of [A]cquisition activities. Are you measuring the results in terms of cost/effectiveness? How does writing a blog post compare to a tweet, running a contest, etc? For us, contests and some posts that did well on HN have been very cost-effective ways to increase the funnel. IRC/chat is not that cost/effective but I do it anyways because I enjoy it. Same with going to events. There are lots of other ways (ping me if you want ideas). I’m a big fan of guerrilla marketing, but in some cases you need the big guns of professional PR. Essentially experiment, measure, and go deep when you find out what works.

Other activities/metrics we use: for activation, we set specific goals on Google Analytics. For retention, we see what percentage of our users have made api calls in the past X days. For referrals, we measure tweets, blog posts and ask people how they heard about us. I’ll stop here because this is just a brief intro, but this is just the beginning.

TLDR version: don’t be afraid of marketing, it’s just a tool that you can use efficiently to your advantage. Also, it’s only as evil as you want it to be. If you truly believe your product is great, you are doing a good thing by making the world know about it. Have a strategy and use tactics that work.

This is the part where I ask you to follow me on Twitter for the hell of it. It’s free 🙂

Going Full Stack

One of our investors told me that one of the reasons he invested in IndexTank was because he’d seen me “go full stack.” I like that phrase, and I take it as a compliment. He was referring to the time back in 1998 when I built an mp3 search engine in C, hosted it on my desktop computer, promoted it until it reached 200k daily queries, and made six figures out of it (all in a matter of months).

Being able to “go full stack” is a very good ability to have for an entrepreneur. It’s not necessarily good for other life endeavors (e.g. being an academic researcher, or the CEO of a large corporation). I’m sure it can be very counterproductive in some cases, because it means that you won’t be able to have a multi-year focus on improving one single ability. Even for an entrepreneur, it’s neither necessary not sufficient. My case is that it’s very helpful, and I’ll elaborate a little bit on what “going full stack” means from beginning to end.

1) You have an itch to scratch. It’s possible to start a company building stuff you wouldn’t use but other people want. I prefer to build things I want because it’s easier. If I’m building something I wanted and I’m not using it myself, that’s a red flag. It makes me happy to find myself hacking quick apps with IndexTank once in a while.

2) You can code. This doesn’t mean you are a guru who would make it to the finals of Google Code Jam, or even that you could impress a technical interviewer with your deep knowledge of encryption protocols or graph traversal algorithms. However, you can read a “Learning X Language” book and manage to write a program from scratch. It runs and it does something. It may be ugly and inefficient. You may shudder at the thought that a seasoned programmer might see your code and post a snippet to TheDailyWTF. It doesn’t matter, it works and you are ready to try to build a business.

3) You understand people. It’s not enough to create something you want, other people have to want it too if it has a chance to become a business. You can sense frustration from people who are trying to do something and get on with their lives, even though they don’t care about technology. You can empathize with them. I don’t care if a bottle opener is well crafted or pretty, I just want to open the damn beer! Do you have one? I’ll pay a couple bucks or whatever.

4) You can make things “not too ugly.” A designer may cringe when seeing Craiglist, but it has been successful for many years despite not being beautiful. There is a threshold of ugliness that makes people ignore a product because they can’t stand to look at it, Craiglist is clearly on the right side of it (perhaps barely). Our brains evolved that way. Do you use any sites made in Comic Sans?

5) You can put the pieces together. Coding is not enough, you need to be able to set up your product or service somewhere and make it available for people to use. Today that’s easy, in the case of web apps all you need to do is know how to deploy to the cloud and make sure your stuff doesn’t crash too much.

6) You can promote your stuff. Some young hacker guns believe that creating something awesome is good enough. Sometimes that’s the case, but more often it is the squeaky wheel that gets the grease. There are plenty of stories of Silicon Valley startups that had great technology but could not figure out how to get it in front of the right people at the right time. For example… um, well, you get my point. Because you don’t have deep pockets you’ll have to resort to guerrilla marketing tactics. This is a lot of fun (who doesn’t like the thrill of seeing a blog post get lots of retweets or upvotes on social news sites), but you can’t do it willy-nilly. This leads to the next point.

7) You can communicate. You don’t need to be a motivational speaker, but you have to be able to present your ideas to an audience. You must be able to tell a story both orally and in writing. You have to put yourself in the place of the listener. Are you still reading this? Are you bored? If so, I’m failing. OMG ZOMBIES! MICHAEL BAY!!! Ok, moving on.

8) You can sell. Obviously this depends on being able to communicate, but it’s more. You must not be shy about asking for money. You must think of your product not in terms of what it cost you to produce but in terms of the value it gives to your customers. What alternatives do they have? Will they do it themselves? Why did they come to you? Why did they listen to you? This seems daunting to a young hacker but it isn’t. If you are building something good, selling it is just a matter of spending time paying attention and listening. Read “How to Win Friends & Influence People.”

9) AND MOST IMPORTANTLY: You enjoy the whole process. None of the above feels like a horrible burden. Some things may not be as interesting as others (OH NOES ANOTHER SALES CALL!!), but the prospect of doing *whatever it takes* to be successful is exciting to you.

Are you like this? Close this window and go build stuff. Investors are waiting!

BitCoin will be made illegal, but cryptocash is here to stay

In case you haven’t heard of BitCoin, it’s one of those concepts straight out of science fiction books from late last century. Untraceable digital cash that’s not controlled by a central agency such as a government. Jason Calacanis thinks it’s the most dangerous project he’s ever seen.

Bitcoin is a P2P currency that could topple governments, destabilize economies and create uncontrollable global bazaars for contraband.

Hyperbole much? I don’t think BitCoin is dangerous, at least when compared to unmanned drones, sharks with frickin’ lasers or BigDog. Sure, people can and will use BitCoin for buying drugs and other illegal services just like cash today. Of course, there’s the added advantage that you don’t need to be physically together in order to exchange payment. Spies will no longer have to collect suitcases from trashcans in exchange for lists of double agents, but for physical goods it’s not so important.

What’s interesting to me is that BitCoin is now in a legal grey area. The BitCoin economy is small enough that it’s still flying under the radar of governments (the total value of all existing BitCoin currency is on the order of $50M USD). It reminds me quite a bit of mp3 files in 1997, when they were just a novelty and record companies didn’t know what to make of them. It was obvious to me that digital music was going to be huge, so I spent six weeks hacking nonstop on my own mp3 search engine. It turned out to be a good investment, as I made some interesting cash operating the system and later sold the software to Inktomi. But I digress.

The parallel between mp3 files and digital currency is that both addressed a very specific need/want. In the case of music it was about convenience. There’s no need to explain to someone under 20 how ridiculous it would be spend time choosing the 10 or 15 CDs you’d want to take on a road trip, let alone packing huge boxes of them when going off to college. For digital currency it’s more than that. Most people I know don’t like the fact that all their purchases are being tracked by marketers, governments or their spouses. Cryptocash is not only convenient, but also untraceable. You can call that dangerous as much as alcohol, a hammer or bittorrent. It’s just a tool, and there are legitimate uses for exchanging cash anonymously.

Of course, governments don’t like to be left out of overseeing financial transactions. More importantly, they have the power to make these transactions illegal AND enforce this law. I agree with Calacanis in that if BitCoin takes off, it’s only a matter of time until the US government decides to throw serious resources to combat the evil of anonymous currency. I’m sure they’ll come up with a creative name, such as the War on Digital Evil Terror Drug Laundering Money. The difference with mp3 files is that this time the government will be the principal, not just a proxy for the movie/record industries.

Still, Pandora’s box has been opened. Because there are such compelling use cases for digital currency, it won’t go away. Maybe BitCoin will be thwarted, but ByteBill or NibbleNickel will take over. It would be sad if something as useful as anonymous, untraceable digital cash was only used by criminals. I for one am going to be watching this one unfold with popcorn in hand.

If you’ve read this far, you may want to follow me on Twitter.

The Siren Song of AWS

I have a deep respect for Amazon as a company. They have been consistently kicking ass for over a decade. They had a long-term vision and stuck to it while everyone else was out chasing the next buzzwordy fad. It worked. Moreover, they are not a one-trick-pony like Google so far (to be fair, the No Evil company's pony is more like a galloping flame-breathing dinosaur but still).

Amazon's second trick is of course AWS. The brilliance of AWS is not in that they figured out how to productize and make money with their existing private cloud infrastructure. Sure, they were the first serious player in the IaaS business but Google could have blown them out of the water had they chosen to. The genius move is this:

– Google App Engine forces you to do everything their way from day one. AWS lures you in with metal (servers, disk) and slowly creeps in on you with their multiple value-added services.

So it goes like this: let's check out AWS. Create an account, set up a Linux box, run your stack. You're the boss, you could switch anytime. So far so good. Say one day you want to use a key-value store and SimpleDB is there. It's simple, oh wait, it isn't. There are many subtleties that you discover as you scale. Next thing you know, your software is tightly coupled with SimpleDB. Migrating to, say, Rackspace becomes a serious project. Not only that, but SimpleDB is not cheap. Your bill starts going up, and so goes Amazon's stock.

TL;DR: AWS is doing an awesome job at luring you in and making you stay like a seductive siren. Use AWS as long as it works for you, but be ready to escape before you follow the fate of the unsuspecting sailors!

And now some music. 


Las Vegas is not known as a beacon of honesty and trust in the world, let’s put it that way. I went there last weekend to attend a rock climbing festival, and I stayed at one of the well-known casino/hotel combos. I checked in to my room and I found this little card on the night stand:


If you have ever waited tables, you know the marketing trick: if you say your name and create a “personal relationship” with patrons, you will likely receive bigger tips. Nobody agrees whether hotel room maids should be tipped or not, but clearly this is the same deal. This card is standard for the hotel, so the idea doesn’t come from the maids themselves. What called my attention here is the name: Peggy.

According to the awesome Baby Name Voyager tool, Peggy peaked as a name in the 1930s. I don’t know anyone young with that name. It’s hard to imagine a Peggy being a hotel room attendant in Las Vegas in 2011. Perhaps some market research shows that elderly couples are the most likely to leave tips, and they will be most sympathetic to someone with a name that suggests someone of their generation.


Sunday morning I checked out electronically and I was ready to leave the room. “Peggy” knocked on the door to see if she could come in and start cleaning the room. I immediately noticed her Spanish accent. Her name was Ana. I didn’t ask her how well the nom de guerre chosen by her employer is working out for her 🙂

How did you become a hardcore back-end developer?

I just saw this poston Hacker News, and I thought I’d write my answer here. Disclaimer: I don’t know if I am (or ever was) a hardcore back-end developer. I did make money for a long time doing web-scale development work for a bunch of companies.


I started coding in the 80s. No web, no connectivity for the most part, just simple games, random programs, hardware drivers, random utilities. Basic, Pascal, Assembly, C.


Fast forward to 1997. I’m doing my masters at Carnegie Mellon, enjoying the most awesome network connection. I download my first mp3, a novelty back then. It’s not even illegal, that will come later. A few months later the first mp3 search engines start to pop up. I think I can create one that’s better, and it becomes my personal summer project in 1998. After six weeks of nonstop hacking, I have running (check out the old snapshot, the domain hasn’t been mine since 2001).


So what was 2look4? A few things.


  • A CGI program written in C running under Apache that would process queries and call system(“/bin/grep”) over a file containing ftp links to mp3 files.
  • An ftp spider called by a cron job that would go through a list of known sites, list the mp3 files there and guess if the site had “unlimited access” or if it required an upload/download/ratio.
  • A submission form for new sites, a very basic poll, some javascript.
One day I put it out there and told some friends about it. The first day it had 100 queries. A week later, 10,000. My linear search with grep held up pretty well for the first few days, but as soon as I started getting concurrent queries it slowed down to a crawl. That’s the first time I saw a web-scale problem. Over the course of three nights I wrote a very simple inverted indexer and searcher to replace grep. It had no tf-idf or word positions. All you could do was an AND search (intersection between the sets of documents containing words), but it reduced the complexity from linear to almost logarithmic (binary search, basically). This was enough for the load of the server to drop and for the machine (a Pentium server) to run pretty much unattended for the next year. It would peak at around 200k queries per day sometime in 1999, before Napster came into the picture and killed ftp servers that shared mp3 files.


In the fall of ’98 I showed 2look4 to a former Carnegie Mellon professor who worked at Inktomi. Long story short, Inktomi acqui-hired me and my puny search engine software (not the site, which earned me some nice cash on the side from ads). I joined the crawling and indexing team and battled Google fiercely for the next four years and ultimately lost. We did a lot of cool stuff though. We grew from indexing 30M documents to 500M in a couple of years for example.


To answer the title question: I don’t think you “become a hardcore back-end developer”, you simply dip your toes in the water of terabytes and millions of requests per day. Try not to commit any algorithms with orders of complexity that don’t make sense for what you are doing. Pierce through abstraction layers and try to understand what the hardware is doing. Are we killing the network? The cpu? The disks? Do lots of back-of-the-envelope calculations. Write prototypes. Use load generator tools. Instrument code. Optimize carefully and only when necessary. Ask others who know more. It never ends, because technology changes fast. Game-changing technologies become affordable (cloud computing, SSDs, insane amounts of memory). It’s very humbling, because you think your system is as fast as it can be until someone comes along and makes it 10x faster.


If you are a software developer, try to work for a while on a web-scale problem. To me, it never gets old.