The Moz Blog

Director of Bing Discusses Holistic Search and Clickstream Data - Whiteboard Friday

- Posted by to Whiteboard Friday

 A little over a couple weeks ago, Search Engine Land broke the story that Bing uses clickstream data from Google searches as a ranking factor in its results. The article was entitled, "Google: Bing Is Cheating, Copying Our Search Results," and as you can imagine, there was a lot of response from the SEO community. After the flames cooled down and the information in the article was discussed further, SELand wrote up a counter piece, "Bing: Why Google’s Wrong In Its Accusations," entailing a walk-through of what exactly Bing was doing and why it's not "copying" per se but instead mere consideration of a valuable search signal.

This week on Whiteboard Friday, Rand talks to the Director of Bing, Stefan Weitz, and discusses some of the implications of using clickstream data in Bing's search algorithm. As Stefan discusses, Bing is trying to take a more holistic approach to search results calculation and delivery to ensure searchers get the information most relevant to them, which means using a variety of data sources and experimenting with innovative ways to display results to users. He also gives us some insight into the future of Bing Webmaster Tools. There's also a ridiculous map of Manhattan with some fun-looking satellites pointing at it - check it out below!


Video Transcription

Rand:    Howdy, SEOmoz fans. Welcome to another edition of Whiteboard Friday. This week we have a very special guest with us, Stefan Weitz, director of Bing.

Stefan:    How are you doing?

Rand:    Stefan, thank you so much for joining us.

Stefan:    Sure.

Rand:    Really appreciate it.

Stefan:    It's on the way home. It's actually better for me this way.

Rand:    So, Stefan has two great distinctions. Number one, you were recently honored best hair at Bing. Is that right?

Stefan:    That was actually true and it was by mistake. One day I woke up and I had long hair and then I was afraid of getting scissors near my head and so I won't cut it now.

Rand:    That's phenomenal. And second, you are tasked with dealing with messy cleanup from incorrect PR that happens on the Internet and in major media sources.

Stefan:    Sometimes PR that I actually create. It's like job security over and over again.

Rand:    That's great. Well, let's see if we can't create some job security for you here today on Whiteboard Friday.

Stefan:    I'm sure we can.

Rand:    So we are going to get into some of the clickstream topics, but the first thing I actually wanted to chat about is something that you and I were talking about before this session. I think it will be of big interest to a lot of webmasters. So Bing obviously is gaining some nice market share, having a significant impact. Clearly Google is thinking you guys are more and more of a threat, which has got to feel a little flattering. It's got to be a nice feeling.

Stefan:    Well, I mean certainly it's good to know that. I mean, no one's really gained share against Google since Google launched. And look, they do a phenomenal job and they have for years and they've served us well for over a decade. So, yeah, it's nice to see us actually, between us and our Yahoo-powered search or the search that we power for Yahoo as well, I mean, you're talking between 26 and 30 percent of queries are now served by a Bing engine, which is great. The engineers love to see their work being used. There's nothing worse as engineers, as we all know, than writing something that never gets used.

Rand:    Oh boy. I can tell you some frustrations we've had on that front. Now, in terms of Bing and Google, one of the things that is really interesting is this New York Times story that came out this past weekend around JCPenney. So a search consultant was using Open Site Explorer, digging through the backlinks . . .

Stefan:    Nice plug.

Rand:    Yeah, it was a nice plug. Oh yeah, I have to plug SEOmoz on Whiteboard Friday. That's very important.

Stefan:    Okay. Make sure we're clear on that.

Rand:    So they're digging through these backlinks and they're finding a lot of what look like manipulative and paid links that apparently an agency that JCPenney had hired to do some search marketing work for them had purchased to help inflate their ranks. And for a year plus, or nine months plus, they'd been ranking number one, number two, number three at Google and fairly well, reasonably well on Bing, for most of this time for a lot of big searches. So one of the big ones we talked about was "dresses" and "bedding" and these kinds of generic terms. So the New York Times writes this article. That afternoon, Google sort of responds and says, "Hey, we're taking some punitive action against them." In the article they talked about this.

Stefan:    A penalty basically, yeah.

Rand:    Now, we did a search right before this Whiteboard started, and we searched for "JCPenney dresses."

Stefan:    That's right.

Rand:    And JCPenney is not in the top three results.

Stefan:    That's where it got weird for me, actually. I mean, look, I get . . . there are certainly, it's well within Google's rights to say, hey, we want to manually penalize a domain for any number of reasons. That happens. We generally don't like to use a lot of manual re-ranking. We like to try it algorithmically if we can because that just seems like it's actually a more scalable fashion. But you're right. What's happening here, I think, is they've kind of gone a little too far down the path of penalization. Because literally as we looked at, "JCPenney dresses," probably the best P1 result would be JCPenney's dress site, I would think. Right?

Rand:    Right.

Stefan:    And now it doesn't appear anywhere on that P1, page 1.

Rand:    And in fact, I think the fourth result is like their mobile page, which is sort of a weird result to have in the web search.

Stefan:    Yeah.

Rand:    And this is happening because of this penalization system. One of the things that is really curious, I think, for a lot of webmasters, how are Bing and Google ever going to beat this problem on a macro scale?

Stefan:    That's an excellent question. So, there's a couple ways. There is the kind of short-term thing we're looking at from the ranker itself. Like, are there particular classes of link farms that we know are of low quality? Can we kind of apply some kind of discount to links that are coming in from what appear to be low-quality link farms? There's stuff we can do there, which we'll do too. We'll all look at these methods to figure out . . .

Rand:    There's a lot of machine learning that goes into that process, to say like here's a big set of what we think are spam links.

Stefan:    Yes.

Rand:    So as a webmaster, you better not look like spam, either the spam that exists today or the spam that's going to exist tomorrow because a classifier is going to catch that.

Stefan:    That's right. And this is a machine, right? And so you could get ostensibly swept up into one of these categories, right? We actually had someone mail us, a couple weeks ago, who had a site that actually looked like it may have been a spam site. It actually wasn't, but it had a lot of characteristics of what we consider them to be and so she was getting a penalty it looked like on her site. And so we said, "Okay, we can't tell you exactly what to fix, obviously, but here's some things you ought to look at." So there are some things that are happening there.

But, I think, if we go a little bit further, there are two things. We, Google, everyone, Yahoo, Bing, we all use human judges to kind of measure relevancy.

Rand:    Right. Google has this big worldwide distributed team of search quality raters, and you guys have a team as well.

Stefan:    Same thing. HRS, yeah, same thing, and so human raters actually look at this. The challenge we have, we were looking over the weekend, all the engineers, we were all in threads over the weekend looking at this problem and we said, gosh, you know the problem is for bedding, the JCPenney result wasn't bad to a human. Maybe it shouldn't have been P1, but P4? I don't know. Actually if you ask a human to say, "Is this a good result?" They go, "Yeah, it's actually pretty good."

Rand:    So this is why you're not going to get spam quality raters who are saying, oh, JCPenney ranks first for bedding, complain and spam report.

Stefan:    Exactly.

Rand:    No. JCPenney seems perfectly reasonable.

Stefan:    That's right. And so if we took it all the way out, we actually realize that you likely would decrease any CG or the kind of cumulative gain that we get as we measure relevance. So if you yank it out, you actually then, people go, "Wait, where did JCPenney go? They sell bedding." And so it's one of those weird things actually to try to correct for when you have a situation like this.

But in the longer term, this really points to the bigger challenge with search, which is stopping thinking of search as just a way to navigate the collection of links across the Web. Essentially, we've been doing the same kind of search now for over a decade, right? It's basically anchor text and page rank and inbound links, and that's how we've kind of decided what page is best for a particular term.

Rand:    And you think that this model of anchor text, page rank, or in Bing's case static rank and diversity of incoming domains, that's going to fade out to something else? Or it's going to be bolstered by other things?

Stefan:    Yeah. I think that'll be around for a long time. So don't worry. You don't have to like go and sell off. Don't call up Rand and say, "I want out." That's all fine. I don't know. I always think of you as . . .

Rand:    That I sell links? Really?

Stefan:    No. You know orange is Bing's color. That's why I took the color.

But there are signals beyond this. That'll be for a long time. But what we're looking at, and the example I was using today with some engineers was, you do a query – not to pimp Bing here – but if you do a query for, say, shoes on Bing, we actually . . . the first answer now, we actually fire off what's called a visual search gallery, which basically is a way to navigate 3,000 pairs of shoes using a more familiar visual metaphor. So you can say, "Okay, I actually want high heels. I want patent leather. I want black."

It's kind of like applying metadata to your query, and it actually rejiggers the results in real time based on what it is you're trying to do. Because in many cases, people do come to engines and they put in one word, they put in two words, very ambiguous. "Shoes" is a horrible query, right? But we get a lot of that. So the question really is do we just take "shoes" and just try to throw back everything we can against that word, which of course is going to be fraught with problems.

Rand:    Right. Online shoes. Shoe store.

Stefan:    Yeah, exactly. Is that the right thing? Really, is something that ambiguous actually calling out for a more reasonable approach where we say, "What do you mean?" Kind of like if you and I were talking and I would say, "Shoes." And you go, "What? What, like dress shoes? What are you asking? Do you want to buy shoes?" So we have conversations. And today, engines, they kind of fail. They're very autonomic. They're kind of an in and out type transaction. And we think, with things like Visual Search, we're able to actually start to say, "Okay, great. What do you really mean with that query," and try to pivot and help people refine.

Rand:    So Google's got kind of on the left side those related searches or search suggestions. You guys have search suggestions.

Stefan:    I think the left and the top. On the top. We actually do an answer bar at the top. Those are interesting ways of conversation modes, but we're even looking at ways to go a little further than that without getting into Clippy, which I love Clippy, but not everyone loves Clippy. So something like that without being annoying.

Rand:    I always worry that people are going to think Roger, our mozBot, is Clippy-like.

Stefan:    Oh, you know what? Can we borrow him for our . . .

Rand:    You totally can. We have a current asking price. I'll let you know after this Whiteboard Friday.

Stefan:    So there's that. Longer term, we're looking at how we think of the Web really as a representation of the physical world itself. So we understand that, the weird example I was giving you earlier, "Inception" is a movie. You and I understand that "Inception" is a movie. We understand "Inception" as a movie has a certain number of characteristics. Movies have reviews. They have show times. They have previews. They have trailers. They have pictures. They have conversations on Twitter.

Rand:    So are you saying, from a marketer's standpoint, let me imagine that this is the knowledge that I've got about "Inception." It's a movie. Do I then want to say, "Oh, I should make sure that a page about 'Inception' of mine has things like information about who is acting in it and maybe a video of the preview or the trailer and reviews and data like that and that it's in the movie realm rather than just being the page on the Internet with the most inbound links that say 'Inception.'"

Stefan:    Yeah, because honestly, what's going to happen is . . . we just did this, actually, with our last release. We have become much smarter about these objects on the Web. So we do actually know "Inception," and that's a weird example. Think of like, "Casablanca," which actually has multiple editions that have been re-released over the years and there are different release dates.

Rand:    And there's an actual city.

Stefan:    Exactly, right that too. But now we can begin to say, okay, this "Casablanca" on Netflix is the same "Casablanca" you can buy on Amazon or you can rent on iTunes. That's actually a lot harder than you think because the movie domain is not as clear.

Rand:    So this is like an entity association type of algorithm.

Stefan:    Yeah, exactly. And so that's what we're going to see more and more search heading towards, we believe, is that us understanding the Web again literally as a representation of the world and not just a bunch of links and pages and static text. That offers up an entirely new way to think about ranking.

Rand:    I guess that would be my last question on this before we move on to the clickstream stuff is, as a marketer, what should I be thinking about to be a step ahead as this is coming forward? It sounds like there's naturally going to be some brands who are doing these types of things already. I'm a small website. I'm starting out. I review movies. I want to make sure that Bing and Google know that my stuff is good. What should I be thinking about?

Stefan:    Today, all the classification is done in a very machine learned process. Ideally, there are defined microformats you have for a lot of these things. I can see a lot of that coming down the road. Even like Facebook's Open Graph system has a limited amount of RDF.

Rand:    Right. I can say, "This is me. Rel=me."

Stefan:    Or you can say something like, I forget the actual phrasing, but basically you can say, "This page is about a movie," for example. I would be watching very carefully what standards or even what proposed standards begin to evolve to help describe things that you're working on in a more concrete manner. So if you are selling, let's say your business is selling sheet music of 1930s swing songs, as an example. There will be a time in the not too distant future, in most people's opinion actually, not just mine anymore, where you're able to actually mark up this piece of sheet music that I'm selling. There will be kind of an ontology or some kind of taxonomy which lets the marketers say, this is from, I can't think of composers from the 30s, but some composer – composer=foo, decade=foo, year=foo – and actually begin to describe this thing, this object as an object and not as simply a web page, hoping that we crawl and parse it correctly. Because, frankly, crawling and parsing is a very messy, expensive, and inaccurate science.

That's the brave new world. But again, for now, just keep doing great SEO work, but don't buy links.

Rand:    Don't buy links. Let's move into another really interesting story around this clickstream stuff. So a couple weeks ago . . . no, I'm sorry. I guess it was about a month and a half ago. So Google built a little honeypot, right? Essentially they say, we think that . . . well, because Google and Bing are both using these signals of clickstream data, Google says we think we can engineer this clever system to catch Bing by using a nonsense word.

Stefan:    Yes.

Rand:    And essentially what they caught you guys doing is not building a system that was robust enough to recognize, oh, when we have very few signals about some random nonsense word, maybe we should be tossing those out because otherwise we could be manipulated in a way that would make us look bad.

Stefan:    Right. Yeah, let's do it.

Rand:    So let's go with an analogy here. This is a terrible representation of the island of Manhattan, right?

Stefan:    Wow.

Rand:    So here's maybe here's Central Park. Does that sound . . .

Stefan:    That's better. Now I . . .

Rand:    Now you're there. Washington Heights right here.

Stefan:    My mom grew up in Harlem right over there.

Rand:    My sister lived right here for a while. This is kind of fun.

Stefan:    Yeah.

Rand:    So, navigating the island of Manhattan are tens of thousands, hundreds of thousands of vehicles.

Stefan:    Yes.

Rand:    Maybe even a million vehicles. I'm not exactly sure. There's like 11 million people in the city. So if I get a Bing navigation system, Bing is going to tell me based on traffic patterns and the weather and the time of day which way I should go in New York City. Hang with me, this analogy works.

Stefan:    It does work, I think.

Rand:    So Bing has this nice satellite that's orbiting up here.

Stefan:    This is hypothetical, by the way. There is no Bing satellite watching traffic.

Rand:    Right. This is a pure analogy.

Stefan:    Everyone is going to be freaking out.

Rand:    So there's a satellite that's sort of watching and saying, "Oh here, look at traffic patterns." And Google's got their nice satellite.

Stefan:    Yeah. They actually do have one though, watching everything you do.

Rand:    That's good to know.

Stefan:    I'm sorry. Was that out loud?

Rand:    Oh boy, you're going to get into big trouble.

Stefan:    There goes that sponsorship.

Rand:    So they're both watching all the traffic patterns and what they see is, oh, you know what? The best traffic pattern at 4:00 p.m. on a Tuesday for a midsize car is to go down the West Side. The East Side is just a mess. Park Avenue, nobody wants to be there, but the West Side, that looks quite good. So they're directing cars this way.

Now, what Google did in this instance, taking this analogy to search is essentially say, "Oh, you know what? It's 2:00 a.m. on Christmas Eve. There's virtually nobody on the road and we're actually going to grab . . . you know what? We're going to make this weird, new three-wheeled blue vehicle that no one has ever seen before and we're going to send them this weird circuitous route. In fact, they're going to have to cross over Central Park and around there." Oh my gosh, just to get down to Wall Street.

And so Bing is monitoring this and they see, all right, well there's one blue car traveling on Christmas Eve at 2:00 a.m. This is the route it takes. And then next Christmas Eve, when a blue car shows up, some Google engineer goes home and gets into his blue car and checks, "Oh look, that's the route that Bing is sending me. They must be . . ." What do they call it?

Stefan:    Copying.

Rand:    Right, cheating off my test.

Stefan:    Yes, copying, cheating.

Rand:    It doesn't seem like an entirely accurate analogy, but it's pretty good. Well, the cheating off the test results seems . . . it seems like the kind of way that things are going. So Google and Bing are both looking at where people are going on the Web, what they're clicking on, when they search for something, what do they get to next, what do they seem to be happy with, did they refine those search queries and do other types of searches, can we learn based on that behavior? Seems like a very smart way to go. And you saw, there was an ex-Google engineer who went on Quora and said, "Oh yeah, when I worked at Google, we did exactly the same thing."

Stefan:    Oh, yeah. That's kind of what I was so perplexed about is that it's not . . . (a) we've said we've been doing this for years. We've actually, since '07, been doing clickstream analysis. It's an opt-in thing. Again, it's anonymous. People opt-in to it.

Rand:    And the Google engineers who checked this, they went home and they installed Internet Explorer with the toolbar and opted in.

Stefan:    And opted in, correct.

Rand:    So that they would show you guys this data.

Stefan:    Exactly. And so what? We made no secret of this at all. This is actually, we think, a very valuable signal to have. Now, that being said, I can tell you, without getting into all the details, it's not a huge signal. It's one of thousands of signals we use to actually calculate PR.

Rand:    I bought thousands of Mechanical Turkers and had them all click your results for SEO, and I didn't move one position. So I'm really upset about that.

Stefan:    Aw, dammit.

Rand:    Because based on what I saw here I thought . . .

Stefan:    You thought you could nail them, right?

Rand:    I thought I could just get it.

Stefan:    So that's the first thing is that we, and Google even themselves uses this data. They are kind of going back and forth if they do or not. But look, we know for deep links, likely they use toolbar plus . . .

Rand:    Right, sure. Yeah. If I click that JCPenney search result, there's these other links that pop up there, and they're usually the same ones that people search for most often and click on most often.

Stefan:    Exactly.

Rand:    That's a good customer experience. You actually want to encourage that.

Stefan:    Oh, totally.

Rand:    They have, I think, they've got a site speed ranking factor as well. How fast is the site? And they'll show you that data in Webmaster Tools and that comes from the toolbar.

Stefan:    That's right.

Rand:    I think they say it comes from the toolbar inside Webmaster.

Stefan:    Oh, well, there you go. I'm not throwing mud at all. I think it's actually a very valuable thing. What was frustrating to me, I think, is just the fact that it was . . . they successfully proved something we've said we've done now for three years, which is great. They've given me a proof.

Rand:    Well, to be fair, you guys probably should have written something that said, hey, if someone builds a honeypot and uses these small signals . . . so they've got a little gotcha.

Stefan:    Absolutely.

Rand:    But I think maybe it was the press who blew it into a pretty big gotcha.

Stefan:    Yeah. At the end, it's really, you're right. And we actually were talking to Harry, who kind of runs all engineering for Bing and he's like, "If anything, they helped us kind of find a bug. Maybe we should be throwing out if we only have one signal." Because really, the reason that 7 of the 100 that actually tried the beta with . . . so we actually didn't fire. They gave 100 terms and 93 we didn't actually do anything.

Rand:    So they had 100 blue cars.

Stefan:    That's right.

Rand:    You only tracked a few of them.

Stefan:    We tracked all of them because they have clickstream, of course, but we just didn't use the clickstream in our ranker. I would have to look at the probe. Something triggered something somewhere and said, "Even though we have this signal from toolbar, something doesn't feel right. We're not going to fire anything at all." So seven of the hundred actually did, and that was, probably where we had . . . maybe the engineers got a little happy at home and they had a couple gin and tonics and started clicking like crazy on the link and that gave us more data than the other terms. Who knows. But the point is that we think it's a valuable thing to use. We think the behaviors of customers on the Web who have said they want to help improve the product through their usage, we think it's the customer's behavior that they're giving to us to use and refine.

Rand:    Well, this would be like in the SEOmoz web app, for example, us saying, "Oh, you know what? When people go from their on page to their ranking report, lots of people are doing that, let's put those tabs right next to each other."

Stefan:    Exactly.

Rand:    Let's do it. It seem pretty natural.

Stefan:    One of the arguments that I had heard from people on the Web was, "You just shouldn't be using it. It's just not your data." And I think that the Web was built on this notion of collective intelligence. And frankly, Google has a lot of search data. We know this, right? So to ignore it just because it comes from a competitor, it seems . . .

Rand:    Well, you're not just doing this on Google, right? You do it on Yahoo.

Stefan:    Oh, yeah. The clickstream, right.

Rand:    If somebody searches SEOmoz's website, you'll see that if I use Internet Explorer.

Stefan:    Yes.

Rand:    So it doesn't matter the specific source. Right?

Stefan:    That was the big brouhaha.

Rand:    So let me ask, from a marketer's perspective. I think a lot of people in search observed this behavior and then said to themselves, "Wait a minute. I can drive blue cars. Can I influence things through the clickstream?" Is that something where you've already seen an uptick in manipulation?

Stefan:    There have been attempts, I'm sure. But the reason this was so successful, if you call 7 out 100 successful, was because they were nonsense words, words that no one would ever use.

Rand:    Literally zero search volume ever.

Stefan:    Precisely. They were made up words. And then Google did a manual re-rank of those in their index. So when you searched for them on Google, this one site would pop up. It was a completely . . . and even that, even with that explicit signal, only 7 out of 100 actually worked. So I would tell you that it probably isn't a good use of your time to be doing a lot of clicking on clickstreams to try to rank higher.

Rand:    Gotcha. All right. Well, good to know. We'll try something else.

Stefan:    Exactly. But what it does say is that if you rank organically high generally . . . do what you generally do to rank high organically, because that is a factor. In other words, if you do rank at P2 on Google or P2 on Bing and people are clicking through, they search for SEOmoz and click on that link. It's just good practice.

Rand:    So you're saying one of the things that SEOs maybe need to be thinking a little bit more about is not just getting that position, but making sure that once you have that position, people want to click on it.

Stefan:    Are you maximizing? That's right. Your caption, your title, all this stuff.

Rand:    Right, that title and that meta description.

Stefan:    Exactly. Make sure . . . you want that click after they search for you if it is the right site for that particular term. So yeah, that was "Copygate" a couple weeks ago.

Rand:    Well, I'm glad we're done with that.

Stefan:    It was fun.

Rand:    Yeah, I'm sure you want to do that all the time.

Stefan:    One hair is grey now because of that.

Rand:    I have more than that. So, third question, and last question, but I think a lot of people are interested. What's some new stuff that's coming out at Bing? Either for searchers or for webmasters, because I know Webmaster Tools has sort of been in a little bit of a stagnant state for a little while.

Stefan:    It's catch-up, I'd say. We just actually brought in Duane Forrester who you guys might know out in the community.

Rand:    Yes, absolutely.

Stefan:    Duane now works on the Bing team, and he is in charge of a lot of stuff. He'll be out there in the forums with you guys a lot more and across the entire ecosystem for Bing. Duane will be our guy out there in the space. We updated the Webmaster Tools, I think it was a couple months ago now. It's all Silverlight-based. There's still some areas I know we need to catch up in.

Rand:    I won't give you too hard a time, but the Silverlight . . .

Stefan:    I know.

Rand:    When you install Silverlight functionality, there are a few things you lose that you have on the HTML side.

Stefan:    Yeah, that's frustrating. But the good news is that we've staffed a team of actually more than we had before. So those guys are cranking on the new functionalities. We know that we have to do a better job there. We know it. No one's hiding like, "No, we're hot. Let's just keep going." We think we do a good job. We can do a lot better.

And then as far as new stuff for consumers is concerned, I think one of the most interesting things that we'll have rolled out this week – since this is Friday, I can talk about it now – is this new thing called tiles. You'll see them on the page. Think of it now as really a user experience enhancement. In essence, what we found was that people are able, when they see basically a page, if you look at the page and you do a query for something like, I don't know, let's just do one. What's that new Natalie Portman film?

Rand:    Oh, that looks terrible. Not "Black Swan."

Stefan:    Not "Black Swan," the other one.

Rand:    Oh, with Ashton Kutcher.

Stefan:    So we'll let's call it "new Natalie Portman film." I don't what it's called, but whatever it's called.

Rand:    That's probably a good search. Since we can't remember, lots of people are querying that.

Stefan:    That's probably true. I can see the billboard in my head, but I can't . . .6

Rand:    I can see the preview. Clearly bad branding.

Stefan:    God, no kidding. What the hell? Anyway, so you do a search for this on Bing and you get all the results here, all of the organic results. And what you're seeing now with the introduction of tiles is a little visual indicator here on the side for a couple of these results that come from what we call authoritative sites. So you might have one from IMDB here, one from Rotten Tomatoes, one from, I don't know, Flixster, etc. We're actually going to be pulling in metadata from those sites. So you have the average 77% fresh rating here. Flixster might have their rating, IMDB can have their rating. But what we're doing now is we 're actually . . . because we know people actually are able to figure out the results they're looking for if you append some kind of visual cue onto the page. So by adding on these visual cues for these kind of authoritative sites or high-quality sites. . .

Rand:    Are they sending data specifically to you, or are you guys pulling that from their page without them even having to do anything?

Stefan:    No. In this case, there's been some agreement with these guys.

Rand:    So it's like when Google was testing some of their rich snippets, what they call recipes and yeah, yeah, yeah.

Stefan:    So that I think . . . we'll see. It's just a test now, but we'll see how it works. Internal flighting have been pretty successful with it, and it really provides a way for people to actually find what they're looking for much faster, because you know the logo for IMDB, you know the logo for Flixster, maybe.

Rand:    And if I think I'm a trusted website, hopefully in a few months I'll be able to submit something to you, give you some data, and you could potentially give me this kind of result.

Stefan:    Yeah. Nothing to announce yet, but you're getting the idea. Exactly. This notion of how do you kind of ingest third party content more successfully and make it more discoverable to people. So that's kind of fun.

And then, of course, a lot of mobile stuff has come out in the past couple months. You'll see more of this going forward. But really focusing on the scenarios as you're on the go. You know, unfortunately, it's not out yet, so I can't talk about it. Just let me think for a second. It's so awesome. You're going to love it. Just wait.

Rand:    Is it on all Nokia devices now?

Stefan:    It should be. We'll do a follow up. Next time, I'll actually bring a machine and we can actually demo stuff.

Rand:    Oh, that sounds awesome. I love it. Well, Stefan, this has been phenomenal stuff. I think people really appreciate you and Duane particularly being out in the community talking to webmasters about this type of stuff. I hope you'll join us maybe in the comments if folks have a couple questions.

Stefan:    Absolutely. Just don't make fun of my hair. That's all I ask.

Rand:    It was windy outside before he got up here.

Stefan:    Sure.

Rand:    Thanks so much for joining us. Thanks, everyone. Take care. We'll see you again next week for another edition of Whiteboard Friday.

Video transcription by


Do you like this post?

Get your social on


  • Pro
    Adam Thompson

    I just wanted to point out an error in what Rand said. According to Google's official releases, the manual search quality monitoring is not done by a team of humans, but rather a team of pigeons:


    • Tim Soulo

      hahaha.. have never seen that before! thanks a lot, will go & share it now :)

    • Bryce Bertola

      You've gotta love google's easter eggs.

  • Pro
    George Bounacos

    I'm very curious why there is a Bing backlash when Alexa used clickstream data for its toolbar for years.

    I can remember having to explain to clients or inhouse folks that simply installing the Alexa toolbar at that time would cause sites to have a higher traffic rank.  Everyone in the industry dismissed Alexa, but there's perception and reality and Alexa never really went away.  

    Sure it's gotten better, but ultimately the predator/prey relationship between engines and companies that want rankings are going to create gamesmanship.  

    So if Amazon and Alexa got a pass on puffing up a site's traffic rank certainly from the public and sometimes from the search marketing industry, how is that different than Bing using clickstream data when people use its browser?  Certainly Google is accepting clickstream data from all of its products, so why shouldn't Microsoft?  And if Google isn't processing clickstream---I can't even finish that sentence, of course they are...

    So why hammer Bing--because it was obvious?

  • Bill Slawski

    Nice Interview.

    Interesting tht Stefan mentions an object approach to search.  I know that Microsoft Asia has been publishing papers on an object-based search since 2004, and Microsoft has a couple of working models online that use such a system, including Microsoft Academic Search ( 

    One of the more recent papers on the topic is 2007's Web Object Retrieval, which covers the topic pretty well, but is more than 3 years old at this point.  Not sure what kinds of strides Microsoft has taken since then except possibly to scale the approach to a much wider range of object types.

    Interesting too that Stefan mentions the possibility of increased use of microformats in the future to aid in something like this object information extraction approach.  Some of Google's efforts involving extracted structured data from unstructured pages on the Web (such as the Webtables Project) use probabilities to come up with labels (or schema) for information that they extract.  I suspect that we may see something similar from Microsoft as well, though a standard microformat approach might make things easier for site owners.

    • Donnie Cooper

      Hey Bill,



      • Bill Slawski

        Hi Donnie,

        I'm not sure if we will see Microsoft come out with a publically editable knowledge base like Freebase or even a crawl-based knowledge base derived from specific databases like YAGO, but it does appear that they have created databases from crawls of the Web to power their Academic Search and (possibly) their product search, both driven by their object level vertical search technology.  Not sure if "Bing Shopping" is the direct successor to the Windows Live Product Search that used this technology, but I would guess that it's likely that it is.

        Microsoft Academic Search (Formerly Libra Academic Search) pulls in information from the ACM Digital Library, DBLP, CiteSeer, and other sources, as well as PDFs that it finds on the Web.  Microsoft's paper on Object Level Vertical Search  tells us that in their first month using an object level ranking approach for products, they had incorporated into the search, "more than 100,000 sellers, 31,627,416 commercial pages, and 800 million automatically extracted product records."

        So, the sources for the knowledge bases behind these types of searches may be from specifically selected sites, or from focused crawls on the web (such as the academic search focusing upon PDFs, or the product search looking for ecommerce sites).

  • James Norquay

    Good too see bing comming in for a video white board Friday, really enjoyed it. I agree with the comments about jcpenny, I feel they have been hit very hard, sure Google should take down their Generic terms, yet hitting brand terms is quite harsh imo.  I think it will be interesting too see the future of Bing I am sure we will see a large future in it especially with the strong connection with Facebook like data =)

    • Tim Soulo

      Definitely agree to that!

      Google should change their "sandbox" algorithm to allow all the branded terms to perform well even if the website is heavily penalized. Why should third parties benefit from a search demand for your branded term?

      • James Norquay


        Agree 100% Sure JC Panny has done the wrong thing but hitting them hard for brand terms is hurting the customers in my opinion (sure take down their generic terms). If a user searches for "JC Penny Bedding" JC Penny should rank first becuase it is the customers search intent to find the website. If the customer search's for "Bedding" then really they have no brand in mind they just want to see what all business is offering in the SERPs. 

    • Danny Sullivan

      So Stefan says don't buy links. He suggests Google has gone into overkill by pulling out JCP (and suggests a "manual" reranking that you know Google isn't going to like, since it is adamant that it doesn't manuall rerank -- and wouldn't consider a manual penalty to be a reranking.

      But when I search on Bing for "bedding" or "area rugs" or "comforter sets" or "tableclothes" -- all terms mentioned in the NYT article -- JCP still ranks tops on Bing. I conclude from this that buying links still seems to be pretty effective on Bing. Nearly a week after this particular incident was outed -- Bing didn't catch it before, and still hasn't caught it.

    • Francisco Meza

      Yeah, I think that may have been a bit too much punishment for JCPenny not coming up in SERPs when their own name + keyword is queried. 

  • Iulian Lita

    Very nice WBF. It's cool that you could get Stefan to talk about all of these things and clarify them. He's a funny guy, I like him. It's good that now we have another useful info, confirmed directly from the source, regarding how Bing operates, at least in gros lines.

    • Peter Bennett

      agree about Stefan! And he's doing great stuff for the 'wear glasses' community!

  • Associate
    Kate Morris

    Awesome WBF. Wish I could get in my question of "When is Bing/Microsoft going to build an analytics product?" :)

    • Bryce Bertola

      I'd be interested to see how they handle that. I really love Google Analytics, but a fresh take on that might not be a bad thing.

  • Pro

    Great idea to invite Stefan Weitz - you just outmatched yourself!First your talk about JCPenney and Google was a little bit too long .... but then he presented Bing very well.

    As we all know Europe is a Google continent (apart from small exceptions). Last year Bas van den Beld wrote in Searchengineland a very interesting article about Bing and Europe: "Europe Needs Bing, But Does Bing Need Europe?".Perhaps Stefan can give us a short statement about that.

    • Gianluca Fiorelli

      I am on the paths of Petra... I would sincerely love to see my job as an SEO not reduced to Google.

      Google owns the 98% of the searches in Spain and a little less in Italy. Surely a monopoly that is not wealthy for any business relying on Internet. Surely, a campaign to promote Bing in Europe would be somehow epic but, hint hint, now the trend in Europe is to start considering Google the new Microsoft, so that per absurd, Bing/Microsoft could take advantages from this perception.

      So, some of the questions I would have loved to make, would have been? Are you Bing really interested in the Europe Search Market? If you are really interested in Europe, when we finally will be able to see Bing shining all its features also is its regional version? 

      gfiorelli1 edited 2011-02-18T01:26:38-08:00
    • Thomas Høgenhaven

      Yes, Bing is a joke in Europe. We can't even access all parts of the US website - e.g. Bing News. I find this strategy very weird, as most Europeans partly rely on US news sources anyway.

      I think it has some very poor long term consequences for Bing, when 250 million Europeans get such a poor impression of Bing, which is somewhat undeserved - at least when looking at their US SERPs.

    • stefanweitz

      I just did an interview with Bas this week on Webmasterradio where he asked me the same question.  Short answer - we absolutely want to do better in Europe.  We're taking a more deliberate approach in Europe - rather than just taking the US site and implementing all the features in the euro markets, we've staffed teams in Munich, UK, and Paris (plus India, Israel and China) to actually build engines that are more "British" or "German".  We want to build the experience for the differences in how people in those countries actually use search (for example, people in the UK seem to overindex on News consumption).  That means we might be slower bringing the great Bing experience to all the markets but when we do take off the beta tag by market, its because we think we have more than just a copy of the US site.  Thanks for watching!

      • Gianluca Fiorelli

        Thanks... even though I'm little sad (no mention to South Europe: Spain, Italy...)... and you don't know how much we need to have more search options :), and how the SEO community would love to promote Bing.

        gfiorelli1 edited 2011-02-18T12:16:23-08:00
      • Peter Bennett

        keep wearing the glasses dude, you're doing great stuff for the 'wear glasses' community ;o) Fab vid!!

        PeterBennett edited 2011-02-18T13:21:45-08:00
  • Ryan Anderson

    Really great whiteboard Friday!  Full of great information!

  • Cody Jorgensen






    cojorgensen edited 2011-02-18T13:56:46-08:00
  • Peter Bennett

    That was a great half hour telly prog! Quite in-depth from SEO headquarters, learned a lot and feverishly tweaking. Rand should do a 'check shirt' type search and hit us with a new plaid next week tho ;o)

  • StyleTread

    Very interesting Whiteboard Friday.

    Would have preferred something slightly more hard-hitting, but in saying that Stefan certainly gave us some interesting insights into the way the search world is slowly moving.

  • mypctechs





  • Brian Reynolds

    Excellent interview. It is sad that JCP got hit so hard, maybe trying to send a message. I never realized that Google is so predominant in European countries as well.

  • Mark Keller

    I am really liking Bing's tile feature. The equivallent to Google's Rich Snippets, this adds a whole new demension to Search.


  • Remus Zoica

    Bing has a lot of catching up to do because at the moment it's quite unplasent to work in all the tools they provide for SEO or SEM.

  • Cooking.Com


    Thanks for the video I am happy Bing is finally putting a face on their search engine just like how Google did with Matt Cutts a while back.


    I wanted to learn more about Bing's web spam process and search quality control process. Does Bing have a web spam team similar to one of Google? or is it all part of the main team?

  • Pro
    Rodney Riley


    I thought that this whiteboard Friday was very illuminating, a lot of hints were discussed that I’ve wondered about for years. For example, if you look at the search engine ranking factors that are normally listed user behavior rarely features but this is clearly a ranking factor.

    I for one hope to hear much more from Bing in the future. Google may not be open but if Bing is then that helps everyone in our industry.

    Edit, oh yeah forgot to say the guy was amusing as well!


    Rodney Riley edited 2011-02-21T01:27:19-08:00
  • Umit Hassan

    I hope to see more of Stefan whether it’s elsewhere or on SEOmoz, but in future I would like you to keep any negative comments about Google to yourself. IMHO it kind of felt like you only wanted to retaliate.

    It’s great to see a face to Bing though.

    • stefanweitz

      To be clear, lots of my friends work at Google.  I just like to give them a hard time  when I get the chance.  They have many more opps than me to do that :)

  • Joshua Hedlund

    Interesting stuff. I'd heard about both the Google accusation and the Bing response, but is it pretty much accepted in the SEO community now that Bing was correct in their rebuttal? Like, did Google ever come out and say "Oops, our bad," or, "Actually, you're wrong because of....." or was it more of a grudging, "OK, fine, you're not technically copying but we still don't think it's very nice," or have they just been too embarrassed to say anything else?

    • Danny Sullivan

      Google's view is that Bing is still copying its results. Not "technically" copying, either. That because of the monitoring, in some cases -- and not just the test case -- Bing will have results that it solely had gained from Google. And that in general, Bing is copying the number one results at Google by watching that many people at Google are selecting that number one result for a particular search at Google and then using that data -- which will be one of the biggest data chunks in Bing's "search signal" to make Bing's results match.

      Google isn't embarrassed, nor has it changed its tune in any wait, as far as I can see -- and I spoke with them again about this last Friday for my follow-up article on what the Google Toolbar monitors.

      Bing, of course, sees things differently.

  • Tim Soulo

    "They actually DO have that one, watching everything you do" - LOL

    Stefan is a really nice guy, thanks for inviting him to WhiteBoard Friday

  • Ben Davies


    Great WBF overall!

  • Robb Young

    I'm in the UK and for me 'JC Penny Dresses' returns for the top 4 results. (I wanted to upload an image but don't see that option any more. What am I doing wrong?).

    I wondered if this is a very very recent change where Google have decided they went a little far, or whether it's simply a different result based on geography. Although why it should push those results to a UK searcher and not a US one is beyond me. Thoughts?

    experiencedays edited 2011-02-18T04:28:42-08:00
    • Jenni Brown

      I would think it's because a) Google recognises it as a brand search and b) there is no UK equivalent (e.g. if you were searching for 'Amazon dresses' it would default to the

      • Robb Young

        Right. I understand that. But I'm specifically referring to the section of the video that states Google have pushed JC Penny (and the specific phrase JC Penny Dresses) out of the first page altogether (See 3 mins 15).
        My point is that, I'm not seeing the same behavior in the UK.

        • Jenni Brown

          Seems like they did a pretty sloppy job - the mobile version of JC Penney is still listed in as well.

  • crerem

    Something is wrong with your RSS. I use google reader and the last article i have is the keyword one on 8th. I try to install/unintall the feed but with the same results. 

  • Ian Williams

    Nice interview!





    • Frank Wickers

      I think it's entirely reasonable to assume that the search engine's would do certain quality checks against the microformat declarations. 

      Like if you say that one page on your site is dedicated to the movie 'Inception', the engines are smart enough to scan your content - figure out if it really is relevant to Inception, and then further figure out if your spamming the same EXACT thing on the rest of your site or if this certaing page is indeed unique and fresh.

      So the engines get the benefit of you being able to tell them exactly what your intent is, and all they need to do (to oversimplify it) is quality check what you're saying and verify it.


    Mr. Weitz, I've been trying to get a response from many, many people at Bing, but I cannot.

    I have two questions. The first you should be able to answer, the second you will probably answer with a vague "It's coming soon!" type answer--which I respect, though a time frame would be nicer.


    1) What happened to "Wikipedia on Bing" or Bing research? I really loved that feature, but it seems to have up and disappeared with nary an explanation. Is this temporary? Will the feautre return, revamped, or did it see so little usage that it was canned? I miss reading Wikipedia without all the blue links (as it appeared on Bing).

    2) When, oh when will we see the Bing HTML5 update? There was supposed to be a preview site going up months ago. It never showed up. Neither was there any news. Nor any "It's coming soon!" or anything. 


    I have one suggestion: Can we get the Bing blog to post a bit more? On both of these queries, searching around the blog gave me nothing (and when it did, it was over a year old). Neither did any other website care enough to cover the death of Bing Research, nor about "Where is HTML5 Bing?" or anything. It's like a barren wasteland, and I can find no information. I know you guys are working hard on improving Bing, but it'd be nice to hear more about things you've promised--even if it's as simple as "it's taking a bit longer than we imagined, hang tight!" or something.


    I hope you'll take the time to respond, and thank you in advance.

    • stefanweitz

      Sorry for the delay - thanks for the question!  On Wikipedia, we removed the vertical due to low usage unfortunately.  We still index Wikipedia, of course, and took a lot of the natural language stuff into the core product but the actual 'vertical' that we had for Wikipedia is no longer.  On HTML5 - stay tuned :).  Part of the UX goals behind Bing are make it more visually organized - but that doesnt mean flashy.  We're working internally on figuring out what parts of "HTML5" really contribute to the power of search versus just making it look more awesome...


      HTH! thanks!


  • Pro
    Mark Stegman

     Quality and timely interview.  Thanks Rand.

    FoundationSEO edited 2011-02-18T07:29:42-08:00
  • Bureau-24

    It seems the Bing team has Google on the run. With Google having cried Bing is cheating I had flash backs of the whole Steve Jobs whining about Windows, Netscape whining and Java whining of the 90's. Bing is innovating and I believe it only improves search for their to be a generally competitive environment in the market place. 

    Now, what is facebook copying?

  • Pro
    TME Solutions

    I always read Whiteboard Friday, I can't do that without the text transcript - when's it coming? I appreciate the feature as a whole, but for those who prefer reading, the transcript could do with being available at the same time as the video. Is this possible in the future?

    TME Solutions edited 2011-02-18T09:02:32-08:00
    • Aaron Wheeler

      Hello Andy! We almost always have the transcript up at the same time as the video, but this week I submitted it a little late in the game for transcription so I had to put it up this morning. Sorry for the delay! You can usually expect it to be up when I make the post.

      • TME Solutions

        Cool, cheers for the reply Aaron - and for the transcript :P

  • ImagetecLP

    Great WBF. It will be very interesting to see how search engines such as Google and Bing do in the next few years to better define search quaries and how pages rank.

  • JourneyMexicoChase

    Really interesting WBF. I love the idea of a little visual representation with each result. Even just a snapshot of the page that would expand when moused over would save me from clicking on, I'd bet, 70% of sites I regretfully click on.

    • Gianluca Fiorelli

      Mmm... since almost 3/4 months you can do it with Google (click on the magnifing mirror close to the Title of the links).

    • stefanweitz

      Or just hover on the right of the link on Bing and we'll show you a rich caption that we mine from the page.  For sites like YouTube, we even will often let you play a portion of the clip in the caption. :)

  • Ehsan A H Rahmatulla

    I clearly understand that search engines pick up signals from users experience via toolbar and understand how it's important. My question to Stefan or Rand would be; why use the relevance & importance from Google's search query?

    One other question; when I do a search for 'SEO' on Bing or Google; I do not get too many informational websites. Most of the websites are marketing websites that have either big links or following some un-ethical techniques?

    Search engines were build with the intention of providing information to the users. It's become more of a marketing channel as opposed to information. The only reason Google became a success was that Google was providing information to its users and collection of websites.

    • Gianluca Fiorelli

      Let me defend the guild...

      so... when I look at the SERPs for SEO I see:

      Wikipedia (twice), a support page of, News about SEO... pure informational SERPs. 

      Then I see SEOchat, SEO Book, SEOmoz... and we cannot say they are playing dirty to stay there. And three SEO agencies (, and, and I would be very very cautious before saying they rank there because of un-ethical techniques without a deep review.

      Finally, and this very curious and weird, a website that doesn't have nothing to do with Search Engine Optimization (SEO) but has SEO as acronym of its name (and mission):, where SEO stands for Sponsors for Educational Opportunity. It's ranking for the exceptional high value domains linking to it (practically every university).


      • Ehsan A H Rahmatulla

        Thanks for the reply.

        Check out the UK search results on Bing and Google! 3 informational sites and 7 marketing sites.

        I don't say the search results are totally bad; but I would like to see more information sites like seobook; seomoz who talk a lot about seo; instead of 2 wikipedia results!!

      • pbhj


        Sounds like an opportunity ...

    • stefanweitz

      Good q!  To be clear - we dont use the relevance and importance of google's search query.  We use what the users have told us is the most helpful link for their query through their actions (i.e. clicking on the position 4 link for a particular keyword).  And like I said, it's one of well over 1000 signals we use for ranking.  Keep binging!

    • firstconversion

      Google is also a success for a shed load of reasons outside of providing decent search result - lots to do with good culture and using their money wisely

  • Woolly

    Nice presentation - Thanks Guys :) With regards to doing searches on generic terms, such as shoes, why not create a visible "keyword cloud". So, for example, when the user enters "shoes" all related keywords might show up in a cloud to the right of the results - a bit like Google's "Wonder wheel". Hmm, I have just seen the error of my ways... this would block out PPC advertising space :P