A relative newcomer to the search market, DuckDuckGo isn't shying away from the monumental task, however. In fact, with a simple, straightforward interface and clean results they've come up with one of the most appealing Google alternatives to date. Even if they are still far from changing the status quo, their no nonsense approach to privacy and instant answers are worth taking note of.
We had a chance to talk with the site's creator Gabriel Weinberg a couple of weeks ago. Here's what he had to say about his four-year-old search engine startup and his competitors.
TechSpot: Tell us a little bit about your professional background and how the idea of a building search engine (of all things) started.
Gabriel: I got into internet startups right out of school from MIT in 2000. I started a series of companies that went sideways and then hit upon one that was successful, which was sold in 2006 (Ed. note: Gabriel sold NamesDatabase.com to Classmates.com for $10 million). I was in Boston at the time. I moved down to Philadelphia and started a number of projects again, but had a lot more leeway into what I could do, so I really started messing around with things I was interested in. I didn't set out to do a search engine, but experimented a bit with some projects that I thought would be useful in search, things like more aggressively identifying spam and removing that from search results, and then using structured content like Wikipedia to improve results.
About a year into that I figured "hey, I could turn this into a search engine." I decided to package it up into something I could launch and was just surprised at the response of people, so I decided to really focus on it.
TS: So, initially this was a one-man project?
GW: Yeah. I self-funded it and worked on it essentially by myself for the first three years.
TS: And how big is the team right now?
GW: We are five full-time employees and then probably about 10 part-time people, and then about 10 other contributors... So, it depends how you want to count it, but you could count us like 10 to 15.
TS: What do you think DuckDuckGo's big break was?
GW: We really started growing since the beginning and if you look back, we've actually had a similar rate of growth of about 500% a year over the past several years, so I've been encouraged along the way. We really focused on things we thought the bigger search engines weren't doing very well for various reasons.
That includes the removal of spam and structured content stuff I mentioned earlier, as well as real privacy and a cleaner search experience – getting rid of all the clutter and other stuff. Incrementally we improved these four areas and at the end of 2010 it just seemed like they all kind of came together in a way that people started sticking more. So, I don't think it was one thing, it was just kind of incrementally improving the product to be in such a good place where people would want to use it as their primary search engine.
TS: Looking at the traffic statistics on your site I noticed there was a huge surge in traffic in January this year, right around around Data Privacy Day. What happened that day?
GW: We had made these two microsites, DontTrack.us and DontBubble.us, which were actually up a few months prior to that. We just wanted to spread them around because they are educational regarding search privacy. Someone posted it on Reddit around that day and it made it to the front page. It stayed there for a while so it probably brought in about 250,000 people to that page and spiked our traffic.
TS: What kind of traffic are you seeing right now?
GW: We've been bouncing around 40-45 million searches the last few months. You can find that information in our traffic page, which is publicly available for anyone to see.
TS: Tell us a little bit about the technology behind DuckDuckGo.
GW: We are a hybrid engine so we have our own crawler but we also use a bunch of APIs from other sites, both for link results and what we like to focus on which is instant answers.
We use about 100 different sources including probably five different indexes on our own site – indexes of different types for different things. When you type in a query what we really try to do is figure out what category it is (name query, technology product, etc.) and route it to that best source and return an instant answer from that best source. The thesis is that when you type in a query, most of the time you'd be better served if you had gotten an answer or some piece of data from a vertical site, say like TechSpot for certain things. But people often don't know what those sites are, so it's our job to figure out that site and format it at the top.
TS: And among these sources you use, do you use Google at all?
GW: Not currently.
TS: Why?
GW: Mainly because of privacy. They make it very difficult to use their data through our privacy policy, whereas with other providers we can route everything through us so we're not sharing any personal information when we call these APIs. Google is way more restrictive about how to call their APIs, and also they're branded, they don't usually like third parties to change things around.
By default, if you use DuckDuckGo we're not storing or sharing your personal information when you conduct a search. That's the kind of easy message that we try to stick to.
TS: Privacy-wise, what can users expect from DuckDuckGo?
GW: It's very simple. By default, if you use DuckDuckGo we're not storing or sharing your personal information when you conduct a search. That's the kind of easy message that we try to stick to.
We try to go beyond that some cases. We try to not share a search when you click on organic links, we operate a Tor exit enclave, among other things. The bottom line is our goal is not to collect or share your personal information by default.
TS: Is there anything related to user searches that you do log, even if it's anonymous?
GW: We log the actual query in aggregate so we can improve spelling and things like that. But it's not tied to a given user in any way.
TS: The decision not to keep records that could be traced back to users, was that something you picked up along the way realizing users valued this, or was that the idea from the start?
GW: It wasn't the idea from the start... actually it was not either of those choices. It was mainly because when I launched I didn't really know much about it. I immediately got a lot of questions about it and after looking into it I decided it was what I wanted to do for a couple of reasons. One, as a user myself I thought it was kind of creepy. And two, I didn't want to have to hand over personal information to government and law enforcement via subpoenas, which is what all these tech companies are faced with. Even if they don't want to, they often have to via a court order, so I thought we could take it out of our hands by just not collecting anything.
TS: Google says they need to log these large amounts of data in order to improve search results. What are your thoughts on this? Do you feel DuckDuckGo is at a disadvantage for not storing user data when it comes to providing relevant search results?
You can still get a lot of useful data without tying it back to users.
GW: There are a couple of things to note there. The first is we've taken a hybrid approach by using other people's APIs, like Bing's for example, and they already have a lot of useful data. So we can take some of that advantage without having to get users' data ourselves. So, there is some truth in that. Having data helps.
However, you can still get a lot of useful data without tying it back to users. You can see what people search for and you may have to guess a little more, but you can still get a lot out of it.
Also, Google has reached a stage of diminishing returns with the amount of data they have. You can see that in Bing's success. There's been a lot of studies that show that they're about as relevant or more relevant in some cases than Google. And they have a lot less data than Google. So, what that says is you need a certain amount of user data to get useful results out of it, but the difference in user data stored by Google compared to Bing – which is pretty big – is not seemingly mattering that much.
TS: Given it's market dominance, Google obviously receives a lot of heat from privacy advocates, and with reason: wifi snooping, safari cookies, etc. But often (and this is just conjecture on my side, I might be wrong) criticism comes from a vocal (more informed) minority and government regulators, while a majority of users may not know or care enough about privacy to look for an alternative search engine. Do you think this means DuckDuckGo will be relegated to a niche market?
GW: Our argument has been we're producing better search results, actually. So, that's the real argument to use us. Instant answers, less spam, less clutter. Privacy has been a focus, but it's more of a motivator to get you to try us out rather than the reason to stay with us.
But I do think privacy is a mainstream motivator. Once people are educated about privacy issues, they really do care, but there's a big education barrier. It's just most people have no idea of what's going on, and after they do and they've been surveyed about it they actually do care. I think the problem has been that in a lot of areas Google is in there's no compelling alternative. We're trying to offer an alternative in search.