[User] Please transcribe this text

[Assistant] It was a big blue tickhound. Around his neck was a piece of rope about three feet long. One could see that the rope had been gnawed in two. The frayed end had become entangled in a fair-sized dead limb. Dragging the limb was what made the dog look so odd. I felt much better when I found out what it was.


[User] Please transcribe this text

[Assistant] The blue-tick hound was like the Pritchards, mean and ugly. He was a big dog, tall and heavy. His chest was thick and solid. He came up growling. The hair on his back was standing straight up. He walked stiff-legged around old Dan, showing his teeth.


[User] Please transcribe this text

[Assistant] I told Rainy he had better get hold of his dog or there was sure to be a fight. You better get hold of your dog, he said. I'm not worried about old Blue. He can take care of himself. I said no more.


[User] Please transcribe this text

[Assistant] I knew the killing of the coon was out of my control, but I didn't want to see him die. I said to Reuben,


[User] Please transcribe this text

[Assistant] I told him my dogs had treed the ghost coon and that was the bet to tree the ghost coon. Now it wasn't, Ruben said. You said you would kill him.


[User] Please transcribe this text

[Assistant] It was no such thing, I said. I'd done all I said I would.


[User] Please transcribe this text

[Assistant] Ruben walked up in front of me. He said, I ain't gonna give you the money. You didn't win it fair. Now, what are you gonna do about it?


[User] Please transcribe this text

[Assistant] You better get your dogs and get outta here before you get whipped.


[User] Please transcribe this text

[Assistant] In a loud voice, Rainey said, Bloody his nose, Reuben.


[User] Please transcribe this text

[Assistant] I was scared. I couldn't whip Reuben, he was too big for me. I started to turn and leave when I thought of what my grandfather had told him.


[User] Please transcribe this text

[Assistant] You had better remember what my grandpa said. I reminded them. He'll do just what he said he would.


[User] Please transcribe this text

[Assistant] Ruben didn't hit me, he just grabbed me, and with his brute strength threw me down on the ground. He had me on my back, with my arms outspread, he had a knee on each arm. I made no effort to fight back. I was scared.


[User] Please transcribe this text

[Assistant] If you say one word to your grandpa about this, Ruben said, I'll catch you hunting some night and take my knife to you.


[User] Please transcribe this text

[Assistant] Looking up into his ugly face, I knew he would do just what he said.


[User] Please transcribe this text

[Assistant] I told him to let me up and I would go and not say anything to anyone.


[User] Please transcribe this text

[Assistant] Just then I heard growling and a commotion off to one side. The blue hound had finally gotten a fight out of old Dan. Turning my head sideways, I could see them standing on their hind legs, tearing and slashing at each other. The weight of the big hound pushed old Dan over.


[User] Please transcribe this text

[Assistant] I told Ruben to let me up so he could stop the fight. He laughed. Haha! While my dog is whippin' yours, I think I'll just work you over a little.


[User] Please transcribe this text

[Assistant] So saying, he jerked my cap off and started whipping me in the face with it. I heard Rainy yell, Ruben, they're killing old Blue. Ruben jumped up off me. I clambered up and looked over to the fight. What I saw thrilled me. Faithful little Anne, bitch though she was, had gone to the assistance of old Nan.


[User] Please transcribe this text

[Assistant] I knew my dogs were very close to each other. Everything they did was done as a combination, but I never expected this. It is a very rare occasion for a bitch dog to fight another dog.


[User] Please transcribe this text

[Assistant] Reaching the dogfight, I saw the big hound was almost gone. He had long since ceased fighting. His body lay stretched full length on the ground. I grabbed old Dan's collar and pulled him back.


[User] Please transcribe this text

[Assistant] It was different with little Anne Pull, as I might she wouldn't let go of the hound's throat. Her jaws were locked.


[User] Please transcribe this text

[Assistant] I turned old Dan loose, and getting a straddle of little Ann, I pried her jaws apart with my hands. Old Dan had darted back in, grabbing his collar again, I pulled them off to one side. The blue hound lay where he was.


[User] Please transcribe this text

[Assistant] I thought perhaps he was already dead and then I saw him move a little. Still holding my dogs by their collar as I looked back. I couldn't understand what I saw.


[User] Please transcribe this text

[Assistant] Ruben was laying where he had fallen. His back was toward me and his body was bent in a U-shape.


[User] Please transcribe this text

[Assistant] Rainy was standing on the other side of him, staring down. I hollered and asked Rainy, what's the matter? He didn't answer.


[User] Please transcribe this text

[Assistant] He just stood as though in a trance, staring down at Reuben. I hollered again. He still didn't answer.


[User] Please transcribe this text

[Assistant] I didn't know what to do. I couldn't turn my dogs loose. They would go for the hound again.


[User] Please transcribe this text

[Assistant] Again I hollered at Rainy asking him to come and help me. He neither moved nor answered. I had to do something.


[User] Please transcribe this text

[Assistant] Looking around, my glance fell on the old barbed wire fence. I led my dogs to it, holding onto their collars. With one hand, I worked a rusty barbed wire backwards and forwards against a staple until it broke. Running the end of it under their collars, I tied them up. They made two or three lunges toward the hound, but the wire held.


[User] Please transcribe this text

[Assistant] I walked over and stopped at Rainy Side. I again asked, What's the matter? He said not a word.


[User] Please transcribe this text

[Assistant] I could see that Rainy was paralysed with fright. His mouth and eyes were open wide, and his face was his widest chalk. I laid my hand on his shoulder. At the touch of my hand he jumped and screamed. Still screaming, he turned and started running. I watched him until he disappeared in the darkness. Looking down at Reuben, I saw what had paralysed Rainy.


[User] Please transcribe this text

[Assistant] When Ruben had tripped, he had fallen on the axe. As it entered his stomach, the sharp blade had sunk to the eye of the double-bitted axe.


[User] Please transcribe this text

[Assistant] I felt a lot of pain in my stomach and I coughed up blood for about a month.


[User] Please transcribe this text

[Assistant] Uhm, so before we get into what I want to talk to you about today, I want to give you a little bit of context. Uhm, so, uh, planet Earth, about 7 billion people, this data is actually from March 2012, so a little bit more than the figure that you see up there.


[User] Please transcribe this text

[Assistant] Uhm, the number of Wikipedia readers on a monthly basis is about 500 million. So it's about 1 in 10. About 1 in 10 people planet Earth read Wikipedia at least once a month.


[User] Please transcribe this text

[Assistant] Uhm, but the number of people who edit Wikipedia is about 110,000. Uhm, and, uh, that's about a 1 to 10,000 relationship.


[User] Please transcribe this text

[Assistant] And I want you to know there actually is a dot right there. It's kind of hard to maybe see from the back of the room. And you should know that that dot is actually way too big. It's really hard to make a dot that's smaller than a pixel on a pixelated surface. And so you should just know that's a really tiny dot for a reason.


[User] Please transcribe this text

[Assistant] Uhm, so I wanna zoom into this little area quick and show you a little bit about what's going on right here.


[User] Please transcribe this text

[Assistant] So, the number of people who edit Wikipedia, I showed you before, is about 100,000 editors. The number of people who work at the Wikimedia Foundation is about 250 people. This was 225 back in 2012, but it's grown very slowly over time.


[User] Please transcribe this text

[Assistant] So the research at the, uh, the research team at the Wikimedia Foundation is actually about six people. And you can sort of compare this to like where about the same size at least as far as traffic and interest from the internet as Google. Google has, I actually have no idea, but I would guess that it's more on the scale of maybe 600 people in their research group.


[User] Please transcribe this text

[Assistant] First, I'm gonna talk to you about Wikipedia's socio-technical system. I'm gonna bring in a little bit of systems thinking and biological metaphors.


[User] Please transcribe this text

[Assistant] Uhm, but I'm gonna pull in a lot of the stuff that wasn't in my thesis to talk about like how I think Wikipedia got to this place. I'm gonna bring in a little bit of feminist theory with standpoint epistemology and I wanna talk about the way that we encode our ideologies and the technologies that we build.


[User] Please transcribe this text

[Assistant] Okay, so part one, the socio-technical of Wikipedia.


[User] Please transcribe this text

[Assistant] So, let's start off with, what is Wikipedia anyway?


[User] Please transcribe this text

[Assistant] Obviously, it's, uh, an enormous encyclopedia. It's the world's largest, arguably, uhm, with about five and a half million articles in English Wikipedia, and I was just, uh, in Rio giving a talk about this, and so I wanted to include that there are about one million articles in Portuguese. Uhm, you know, it's just enormous. This is something that you probably already know. Uhm,


[User] Please transcribe this text

[Assistant] So, it's obviously also a wiki. And a wiki is an interesting thing because it sort of, it flips the publication model. Where you publish first and review later. This is totally strange compared to how every other type of publishing system works.


[User] Please transcribe this text

[Assistant] Uhm, and beyond that, uh, in Wikipedia, anybody can edit and we share authorship. And so, uh, there's actually a policy in Wikipedia against stating or behaving as though you own an article. Uh, you have to share authorship with other people. They have to be allowed to edit the article too.


[User] Please transcribe this text

[Assistant] Uhm, it's also an online community. I was telling you before, there's about a hundred thousand, uh, active editors who are working in this space. They do a lot of the things that other online communities do, like set up forums. So in English Wikipedia, they call their forums village pumps.


[User] Please transcribe this text

[Assistant] Uhm, and so for example, they have one for discussing policies, one for discussing technical issues, proposals, etc.


[User] Please transcribe this text

[Assistant] They also have subject matter interest groups. Uhm, so for example, wiki project video games focuses on making sure that the articles in Wikipedia about video games are high quality. Same thing with wiki project medicine for the articles about medicine. There's lots of these. So.


[User] Please transcribe this text

[Assistant] But I'm a computer scientist, and so when I look at Wikipedia, I wanna put it in terms of a system. A system with inputs and outputs. In this case, the input is available human attention, and the output is encyclopedia material.


[User] Please transcribe this text

[Assistant] This is really cool for me because I can talk about the efficiency. How much output are we getting per unit input? So how much encyclopedia material are we getting based on the same level of available human attention?


[User] Please transcribe this text

[Assistant] Uhm, and so I want to make the argument that Wikipedia is a socio-technical system. And in order to do this, I want to define this term.


[User] Please transcribe this text

[Assistant] So, uh, to define this term, I want to talk to you about the history of sociotechnical systems from my point of view.


[User] Please transcribe this text

[Assistant] Uhm, but after a while we realized that we could actually start doing some of those social things with the technical things. So, at the very bottom of the screen here I have some very old, uh, calendar app and a very old mail app.


[User] Please transcribe this text

[Assistant] Uhm, and so these were examples of where we could help out communication, we could help out scheduling, getting people to the same place at the same time, using technologies. And so we realized that technical can support social. Interesting.


[User] Please transcribe this text

[Assistant] When I say socio-technical, I'm talking about something that's much more tightly integrated than that. Uhm, and that's the space that I think that Wikipedia and a lot of other, uh, online communities and technical systems actually operate. And so, I'm a technologist. But for just a minute, to make this point, I'm gonna pretend to be a techno-biologist. Uh, any biologists in the room?


[User] Please transcribe this text

[Assistant] Alright, I can get away with murder. Uhm, okay, so, uh, on the screen I have, uh, for you a couple sketches that I've made. Uhm, these are actually up on Commons, open license if you like them. Uhm, so of a bacterium and a paramecium. So, uhm,


[User] Please transcribe this text

[Assistant] This bacterium is, uh, way too big. It's really important that I explain that to you. If you were a biologist, you would look at this and go, well, that's not to scale. Uhm, because a bacterium would be so smaller, it would actually be smaller than those little bits of poop that are coming out of the paramecium right there. You would barely be able to see the bacterium next to the paramecium. And what I really want to convey to you right now is that


[User] Please transcribe this text

[Assistant] Uh, bacterium is really, really simple. It's basically just like a little sack of salt water. It's got some DNA floating around in it. There's some little specks in it, uh, these ribosomes that help transcribe DNA to proteins.


[User] Please transcribe this text

[Assistant] Uhm, and then I guess there's, there's a few vacuoles, which are these sort of like little fat bubbles, but uhm, really nothing very complicated there. Really just a little, little ball of, uh, salt water.


[User] Please transcribe this text

[Assistant] Uhm, it's got really cool things like two different nuclei. It has this really cool thing called a contractile vacuole, which is how a paramecium can survive in freshwater environments.


[User] Please transcribe this text

[Assistant] If you know anything about osmosis, you'll know that if you take a freshwater sack of water and you put it into, or sorry, a saltwater sack of water, and you put it into a freshwater lake, it will explode because all the water wants to come rushing into the salty environment. The way that the paramecium deals with this is it has a pump


[User] Please transcribe this text

[Assistant] There's actually that, that contractile vacuole is constantly pumping whenever the paramecium is in fresh water, and that's how it survives if that pump ever shuts off, it'll explode. Uhm, so.


[User] Please transcribe this text

[Assistant] When I think about a paramecium and a bacterium, I think about, uh, social systems and the scales in which they can operate.


[User] Please transcribe this text

[Assistant] And so at the bacterium end of the scale, we have a small fishing village, you know, less than 100 people, maybe 150 people, and at the scale of Paramecium, we have something that's like a city, hundreds of thousands, maybe millions of people, or something like Wikipedia, which operates at that kind of scale.


[User] Please transcribe this text

[Assistant] Yeaah, right. I don't usually get that many people when I give this kind of talk who have heard of Dunbar's number, but I appreciate that you have because I think it's really cool. So Dunbar was a social scientist.


[User] Please transcribe this text

[Assistant] I don't know if that's fair to call him that, but anyway, he was a researcher who was studying populations of people.


[User] Please transcribe this text

[Assistant] And he was looking at fishing villages like the photo that I have on the screen right now. And one of the things that he noticed is that a fishing village generally doesn't get above 150 people. Whenever it does get above 150 people, it'll have sort of like a civil war and a group of people will leave the fishing village and go start a new fishing village.


[User] Please transcribe this text

[Assistant] Uhm, and so he was, he was wondering, like, why this upper bound of about 150 people?


[User] Please transcribe this text

[Assistant] And I think what, uh, Dunbar was running into is that if you have a group of people without the infrastructure to allow them to operate at a larger scale, the largest scale that they can operate just using their relationships with each other is about 150.


[User] Please transcribe this text

[Assistant] We're on Wikipedia's end of the scale. That's where we're getting into these interesting substructures.


[User] Please transcribe this text

[Assistant] And so, before I move on, I wanna make this point really clear about like, when I say socio-technical.


[User] Please transcribe this text

[Assistant] I'm talking about how the social, which I would put in the case of this paramecium, which is basically, if you tried to imagine a paramecium that was just a sack of salt water, uhm, that's sort of like saying a large group of people with nothing to support them. In this case, I have a photo of Wikipedia, one of the, the yearly conferences that we have for Wikipedia stuff.


[User] Please transcribe this text

[Assistant] Uhm, and then in the critical subsystems that allow it to operate at scale, in the case of the paramecium, these organelles, but in the case of Wikipedia, technologies like MediaWiki, policies, guidelines and rules, and robots and tools that people build in order to make the system come together and function.


[User] Please transcribe this text

[Assistant] And so when I say socio-technical and I'm talking about the merge between social and technical, I'm really actually talking about this paramecium. That if you were a biologist and you were studying this paramecium, you couldn't just study the saltwater sac separate from the organelles and understand what's going on with the paramecium. You have to study them together.


[User] Please transcribe this text

[Assistant] Uhm, in the same way that if you're studying Wikipedia, you have to study social processes and the technologies that enact those social processes together in order to understand what's going on at all.


[User] Please transcribe this text

[Assistant] So that's what I mean with socio-technical. And now I wanna talk to you about the specialized subsystems of Wikipedia. Because I think Wikipedia only operates at the scale because it has these specialized subsystems. So, I'm gonna split these into five. Uhm, these are just the way that I view things. They're not any, there, there's no truth to this. I just think it's a useful lens.


[User] Please transcribe this text

[Assistant] And so, what, uh, Eric was talking about is that in an open source context, if you have a lot of people look at a problem, like maybe a bug in a piece of software, for one person, they're gonna be able to see, oh, I can fix that bug really easily. I know exactly what's causing this. They're gonna be able to see into the system and be able to fix it.


[User] Please transcribe this text

[Assistant] And so the critical insight in here is that visibility is critical to open collaboration. That when you, when you have a bunch of people who can work on something and they can see what's happening, they can see the problem, see the potential contributions, then you can sort of optimize the situation where the right person will find the thing that's easiest for them to contribute to.


[User] Please transcribe this text

[Assistant] And so my core layer for Wikipedia is that given enough, uh, people see an incomplete article, all potential contributions to that article will be easy for somebody.


[User] Please transcribe this text

[Assistant] Turns out that this, uh, works in theory. And by the way, these are citations in the right hand side. Just a little bit of, uh, uh, what I think is useful literature for understanding this.


[User] Please transcribe this text

[Assistant] Uh, it works in theory, it's part of the process of becoming a Wikipedia, and uh, we can actually support this kind of visibility with technological interventions like recommender systems, and really bad things happen when we take it away.


[User] Please transcribe this text

[Assistant] Uhm, okay, on to the next subsystem. You're gonna have to regulate behavior. The entire internet can add at Wikipedia. You don't even have to register an account. We're gonna have to form some norms. We're gonna have to propagate them and we're gonna have to enforce them.


[User] Please transcribe this text

[Assistant] So, if you search Wikipedia governance in Google Scholar, you're gonna get a crap load of results.


[User] Please transcribe this text

[Assistant] Uhm, Google thinks that it's gonna give you 37,000 results. Uh, however, that's not exactly true because it's an estimate. But if you scroll down to page 10, uh, which I encourage you to do right now if you wanna check me on this, you'll find out that there's actually, there's so much literature about Wikipedia's governance that you get to page 10, there's still, uh, articles about Wikipedia and governance structures. So, I just wanna tell you about how Wikipedians do this.


[User] Please transcribe this text

[Assistant] It might be that you're prescribing something, like, we don't do this and we should do this. Or maybe you're just describing something that already happens. We've been doing this thing this particular way, we should make a rule about that so people know the ways that we do things. And so those are two different types of norms that Wikipedians like to talk about.


[User] Please transcribe this text

[Assistant] So take your idea and write an essay about it. Explain what it is. Make a wiki page. Put it on, put it on Wikipedia. Not in the article space, there's other spaces for policies and that sort of stuff.


[User] Please transcribe this text

[Assistant] And once you think it's ready, then you file a request for comments and people will discuss the merits of this, this idea, whether it should be enacted as a rule or not. If it passes that threshold, it becomes what we refer to as a policy or a guideline. If it doesn't cross that threshold, then it just goes back to being an essay. You could do some work on it, file another request for comments, or just leave it as an essay.


[User] Please transcribe this text

[Assistant] So this sort of forms a divide between informal and formal policies in Wikipedia. Formal policies being the actual rules and informal policies being things that people talk about and share with each other but aren't exactly formal rules. And so I want to give you a couple examples of some informal and formal rules, my two favorites.


[User] Please transcribe this text

[Assistant] So my favorite informal rule is don't stuff beans up your nose. Which is actually not about putting beans up your nose, it's about not telling other people not to put beans up their nose because maybe they haven't thought about that yet.


[User] Please transcribe this text

[Assistant] Okay, so uh, example of a formalized rule is verifiability. Wikipedia is not truth. Wikipedia is verifiable. I'm quoting Jimmy Wales in this, but I think it's a really good summary of what this says.


[User] Please transcribe this text

[Assistant] Uh, if Wikipedia tried to be truth, the end all of truth, the debates would never stop. And so instead what Wikipedia is, is information that can be verified with an external reference.


[User] Please transcribe this text

[Assistant] And, uh, this is, I believe, what allowed Wikipedia to grow at the rate that it has because arguments stop on what you can cite.


[User] Please transcribe this text

[Assistant] So, fun story, you can look at these, uh, growth and regulations and see some really interesting patterns. Uh, this, this dark line on the top is the rate of growth for new essays. And, uh, the, the dashed lines of two different types of those formalized policies, those guidelines and guidelines and policies, those formalized norms.


[User] Please transcribe this text

[Assistant] Uhm, and the fun thing is that you can see that in 2006, Wikipedia stopped formalizing new rules at the rate that it used to be formalizing new rules.


[User] Please transcribe this text

[Assistant] The rate of growth of essay writing went through the roof. And so there was definitely a switch that was happening here where people were doing a lot of reflecting on what's going on in Wikipedia, but maybe not turning that into new rules.


[User] Please transcribe this text

[Assistant] So now I wanna get to the quality control system. If you're gonna open the webpage to the internet, you're gonna have to identify and remove damage. Obviously vandalism is a big deal in Wikipedia.


[User] Please transcribe this text

[Assistant] I think that mostly this system, like it gets a lot for free from Linus' law that given enough people see an article, somebody's gonna be able to see the damage in that article and be able to fix it.


[User] Please transcribe this text

[Assistant] Uhm, but in order to operate at the massive scale that Wikipedia has to operate, Wikipedia's have built two types of automated systems to help them manage quality control at scale.


[User] Please transcribe this text

[Assistant] I like to separate these into fully automated quality control systems like Cluebot and English Wikipedia, which uses a machine learning model to detect vandalism in Wikipedia. It only works for obvious vandalism, but it's really fast. It'll revert the vandalism generally about five seconds after it's saved. Uhm, for anything that, uh, makes it past Cluebot,


[User] Please transcribe this text

[Assistant] That'll get picked up when in one of the semi-automated quality control systems. They also use machine learning models so that they can sort.


[User] Please transcribe this text

[Assistant] Uhm, and most vandalism can be caught at a glance, so this works really well.


[User] Please transcribe this text

[Assistant] It turns out that Wikipedia is a bunch of tools that builds like distributed documentation structure so that nobody has to talk to anybody to find out that, oh, this person's been vandalizing Wikipedia for a while and should be reported to the Administrator's Notice Board.


[User] Please transcribe this text

[Assistant] Nobody actually makes the decision, I'm going to report this person to the Administrator's Notice Board. That just happens automatically through people using the tools in Wikipedia.


[User] Please transcribe this text

[Assistant] And the fun thing is that together these two strategies of reverting vandalism when you see it and reporting vandals to be banned from Wikipedia, uh, forms something that I think looks like a lot like an animal immune system with innate characteristics which are


[User] Please transcribe this text

[Assistant] Fast, general, and local, that's, you know, you get an infection, say on your arm, it'll swell up because your immune system is trying to attack the local problem. It's really fast, it'll happen right away, it works anywhere on your body. It's general.


[User] Please transcribe this text

[Assistant] Uhm, however, this adaptive side, and this is usually what we mean when we say immunity,


[User] Please transcribe this text

[Assistant] Is, it's, it's slow. It takes a little while to build up an immunity. It's specific to a particular pathogen. You've got to get this strain of the cold. A new strain of the cold won't be picked up.


[User] Please transcribe this text

[Assistant] Uhm, and it's global. It works across the entire system. You're not gonna get a cold in your hand anymore because you've dealt with the cold all over your body. Uhm, and this is like banning vandals in Wikipedia. I don't think it's a coincidence that Wikipedia's strategy for dealing with pathogens looks a lot like, uh, Animal Biology's strategy for working, dealing with pathogens.


[User] Please transcribe this text

[Assistant] Okay, so on to community management. We're gonna get a bunch of newcomers, arguably the most newcomers that any organization has ever gotten on a daily basis. We're gonna have to socialize them, we're gonna have to train them, and we're gonna have to deal with disputes that people have. Uhm, so Wikipedia gets 6,000 newcomers a day.


[User] Please transcribe this text

[Assistant] I, I don't know, like it depends on how you define an organization, right? So we've gotta route them.


[User] Please transcribe this text

[Assistant] So we have a tool called HostBot in English Wikipedia. The job of this bot is to identify newcomers who are obviously vandals and basically just leave them alone because we already have ways of dealing with them, but find the people who are obviously not vandals.


[User] Please transcribe this text

[Assistant] And route them to newcomer help spaces like the Tea House, which is a question and answer space for new editors, uh, to get help with dealing with the complexities of Wikipedia when they first show up.


[User] Please transcribe this text

[Assistant] Finally, I want to talk about a system that you might not have guessed, which is the reflection and adaptation system of Wikipedia. Wikipedia is no centralized governance system, so Wikipedians, as a collective, needs to ask and answer questions like, where are we going? Where do we want to go? And how do we want to get there?


[User] Please transcribe this text

[Assistant] Uhm, so I was showing you before about how Wikipedia's governance system works, and I was showing you the rate at which new policies were formalized. I think that formalizing policies are very clearly an adaptive process, whereas writing essays are very clearly a reflective process where people are asking questions and thinking about what should happen.


[User] Please transcribe this text

[Assistant] And I think it's maybe concerning that it looks like Wikipedia is not adapting as fast as it used to be, despite the fact that it still does have some problems, which I'll be telling you about in a little bit.


[User] Please transcribe this text

[Assistant] So, coming back to why I gave you this biological metaphor in the first place, is that, I mean, if you were to look at a paramecium and see it as, you know, a sack of salt water, or maybe a collection of chemicals just running into each other, you wouldn't get the whole picture.


[User] Please transcribe this text

[Assistant] And it's really important that you actually look at the whole thing in order to understand what's going on with the paramecium. In the same way, when you look at what's going on in Wikipedia, the kind of problems that it has, you shouldn't just look at the immediate context of that problem. You should ask yourself, how does this whole system enact this problem? And how might the whole system enact a solution? How can we get the people and their culture


[User] Please transcribe this text

[Assistant] Excuse me. More output per input. More encyclopedia material per human attention.


[User] Please transcribe this text

[Assistant] So on to the critique of algorithmic quality control systems. Let's dig into how one of these sub- one of these subsystems is going a little wonky.


[User] Please transcribe this text

[Assistant] So in the early days of Wikipedia leading up to about 2004, most of the infrastructure of Wikipedia, most of the subsystems were formed by humans and practices. They didn't have a way of formalizing rules. They didn't have very many bots, uh, that ran on the system. They didn't have advanced algorithmic tools to help them with quality control.


[User] Please transcribe this text

[Assistant] It was really just people forming the basic infrastructure of Wikipedia. Wikipedia was a lot like a fishing village at this time.


[User] Please transcribe this text

[Assistant] But between 2004 and 2007, I like to refer to this as the exponential growth period, everything turned into a fire hose.


[User] Please transcribe this text

[Assistant] Uh, suddenly there is, there's thousands of newcomers showing up, thousands of new pages being created every, uh, every day and hundreds of thousands of new edits to articles that we already have. We have to deal with this and deal with vandalism and damage and that sort of stuff.


[User] Please transcribe this text

[Assistant] And so, Wikipedia has made quality control tools that I told you about before, like Huggle, which is one of those semi-automated tools, and Cluebot, which is one of the fully automated tools, to help them deal with this issue. That all happens during the end of the exponential growth period.


[User] Please transcribe this text

[Assistant] Uhm, and I want to talk to you a little bit about like the key technologies that are inside of these tools. So the key technology that I see is the machine classifier. Essentially what this does is it takes a bunch of statistics about what happened in an edit.


[User] Please transcribe this text

[Assistant] Uhm, but it'll just like fill an entire line with the letter K. Uhm, and so these statistics will go into the machine learning model. The machine learning model will figure out some correlations between those statistics and if an edit is vandalism or not. And then it'll be able to make a prediction using some hidden strategy for doing that.


[User] Please transcribe this text

[Assistant] Uhm, so just to put this into some numbers and people for you. So, without machine prediction, reviewing 160,000 edits per day, if you stayed up all night, and you drank a lot of caffeine, and you didn't go to the bathroom, uh, would take 100 or 267 hours. And that's about 33 people working 8 hours a day without a break.


[User] Please transcribe this text

[Assistant] Uhm, but with machine prediction, uh, we can cut that down to 16,000 edits per day. Still a large number, but that would only take about 27 hours. Somewhere between three and four people spending eight hours a day looking at this. And so this was a massively useful technology for allowing Wikipedia to operate at the scale during this exponential period.


[User] Please transcribe this text

[Assistant] Uhm, so, moving on in the story, something went wrong. Uhm, during this decline phase, we see an abrupt change from an exponential growth period to a relatively steady decline.


[User] Please transcribe this text

[Assistant] So, the big question that I was dealing with in this work was, what the heck happened there? Like, how, how did this happen? There were a bunch of hypotheses at the time. We might get to, into some of those in the questions. Uhm, but I want to tell you about the hypothesis that I think has still failed to be refuted many years later.


[User] Please transcribe this text

[Assistant] But in order to tell you about, like, what I think was going on here, I want to talk to you about what Wikipedia were experiencing at the time and how we can see them encoding their experience and view of the world into the technologies that they built here. In order to do that, I want to talk to you about these terms, standpoint and objectivity.


[User] Please transcribe this text

[Assistant] In order to do this, I'm gonna be channeling one of my favorite feminist scholars, Donna Haraway. Donna Haraway was a researcher who studied scientists who were studying apes. Uh, an ape studying apes studying apes.


[User] Please transcribe this text

[Assistant] Uhm, and, uh, so, so what she did is she split, uh, the scientists, the, the research lab that she was looking at into two groups.


[User] Please transcribe this text

[Assistant] Uhm, the, the male scientists and female scientists and I put a star next to female scientists because really what she was looking at were, uh, research groups that were very patriarchal in their structure and research groups that were sort of informed by the feminism critique of patriarchal structures at the time. Uhm, so these are two


[User] Please transcribe this text

[Assistant] Two, uh, discrete groups of scientists who are looking at the exact same thing, the exact same, uh, observations of a behavior.


[User] Please transcribe this text

[Assistant] Uhm, yet they asked really different questions, uh, learned different things about what was going on with ape behavior. The patriarchal scientists were really interested in evolutionary patterns, so they asked questions about reproductive, uh, competition and dominance. You know, which apes had sex with which other apes and how was that organized within their social structure.


[User] Please transcribe this text

[Assistant] Uhm, whereas the female or feminist scientists instead ask questions about communication and social grooming, like how do social circles form in a behavior?


[User] Please transcribe this text

[Assistant] Uhm, so now back to defining these terms. So a standpoint gives you a view of what's important and valuable in the world. Do you wanna, uh, answer questions about communication patterns? Or do you wanna look at, uh, dominance patterns in sexual selection?


[User] Please transcribe this text

[Assistant] Uhm, your objectivity is what you construct in order to reify this value. In the case of the scientists, these were the methods. Were they measuring grooming patterns, like literally recording in a notebook, like when did this ape groom this other ape? And how often does that happen and when does it happen? Or were you looking at sex partners?


[User] Please transcribe this text

[Assistant] Same notebook, write down like, oh, this ape had sex with this other ape. Okay, interesting, let's put that into the data set. So that's the objectivity, how you reify this value.


[User] Please transcribe this text

[Assistant] But the really critical insight that Donna Haraway had, uh, at the time, and I think that we benefit from now, is that standpoints and objectivities can be merged. Neither one of these questions or views on what was valuable is inherently better than the other. We actually benefit from merging the two together.


[User] Please transcribe this text

[Assistant] And so, if you were to look at a, uh, a textbook on ape behavior right now, you would learn something about both redup- reproductive competition and communication and social grooming patterns.


[User] Please transcribe this text

[Assistant] Uhm, the fact that we had people looking from different point of views and thinking about different things is valuable. Turning those into different methodologies to answer different questions benefits us with a broader understanding of a behavior.


[User] Please transcribe this text

[Assistant] Okay, so let's pull that back to Wikipedia and talk about the standpoint that people in Wikipedia had during the exponential growth period.


[User] Please transcribe this text

[Assistant] So at the time they see holy cow Wikipedia has become a fire hose. We need to revert the bad edits as they're coming in Wikipedia, into Wikipedia. And we wanna minimize the effort that we waste doing this quality control work. We don't wanna spend 267 hours every day wasting, you know, reverting vandalism.


[User] Please transcribe this text

[Assistant] So the objectivity, the thing that they brought about in order to, uh, reify this value were the technologies that they put in place in the time based on machine learning models that were able to reduce their workload dramatically.


[User] Please transcribe this text

[Assistant] And this was a massive success. And so right now, you can sort of look at, uh, like, the internet has a firehose going into Wikipedia, that firehose gets filtered through these quality control tools that reduce the workload by about 90%, and then it actually makes it to Wikipedia. That's not quite right, because it's published first, but it's really hard to draw a diagram with a loop, so think about it that way.


[User] Please transcribe this text

[Assistant] Uhm, so, uh, it wasn't until 2009 that somebody was looking at a graph that was very similar to this and said, whoa, holy cow, uh, we're not growing exponentially anymore, we're actually declining now.


[User] Please transcribe this text

[Assistant] And it's not the experienced editors that are leaving and getting tired, you know, they edit Wikipedia for a while, now they're bored, they go do something else, it's the newcomers, the ones who are showing up really excited, want to edit Wikipedia, they're not sticking around anymore. This was, uh, Balwant San, his collaborators at Palo Alto Research Center, Singularity is Not Near, one of the critical, uh, bits of literature about Wikipedia.


[User] Please transcribe this text

[Assistant] So it took us a while to figure out what the heck was going on because there were a lot of hypotheses that were going on in this space like maybe Wikipedia is running out of stuff to write about or maybe Wikipedia shouldn't be that big and it's actually just shrinking down to the size that it really ought to be. Uhm, it took us until about 2012 for us to realize, holy cow,


[User] Please transcribe this text

[Assistant] In, in the standpoint that we brought to developing the technologies and practices that we put in place, we forgot about the value of socializing the newcomers.


[User] Please transcribe this text

[Assistant] Uhm, it turns out that when you split the 160,000 edits that are coming into Wikipedia,


[User] Please transcribe this text

[Assistant] And so, by making the subsystem for quality control boosted and much bigger and much more powerful, it actually ended up sort of squashing the community management subsystem of Wikipedia. And, uh, Wikipedians ended up throwing out the baby with the bathwater. Quite literally.


[User] Please transcribe this text

[Assistant] So hey, now that we know this, maybe we should also value newcomers having a good experience in design for that. Great.


[User] Please transcribe this text

[Assistant] Uhm, so let's review our standpoint. We're gonna extend it. So, so, Wikipedia's still a firehose, that's still true. Uh, bad edits need to be reverted. Mm-hm.


[User] Please transcribe this text

[Assistant] Uhm, but yeaah, now we actually also want to socialize and train the newcomers. Great!


[User] Please transcribe this text

[Assistant] Uhm, so, in the time period between 2012 and 2015, and I'll explain why I'm talking about 2015 in just a moment, uhm, the conversation around Wikipedia incorporates this new standpoint.


[User] Please transcribe this text

[Assistant] So, when, right? You know, like we extended our standpoint, we got new objectivities, this is great. Well...


[User] Please transcribe this text

[Assistant] So it turns out that this, this part of the system, the filtering part of the system that stands between the Internet and Wikipedia didn't really change that much. And I wanted, I'm gonna pick on one of the quality control tools here, but I want you to know that this story actually applies to all of, all of the quality control tools that are in this space. So I'm gonna pick on Huggle.


[User] Please transcribe this text

[Assistant] Before I get into the critique, I want to start with the sugar. Uhm, I give this kind of talk a lot to people who develop these tools. And it's important that you don't tell them that everything is bad. And I want you to know that everything about huggle is not bad.


[User] Please transcribe this text

[Assistant] Huggle is actually an amazing piece of software, and I have no doubt that it represents the state of the art in distributed quality control. I talked to people at Facebook, Google, Quora, about how they do quality control, and they're ten years behind us. Like, Huggle is what they have, it's not Huggle, but it looks exactly like Huggle.


[User] Please transcribe this text

[Assistant] This software and its users are responsible for critical work. Without them, Wikipedia would have totally tanked a long time ago. Uhm, its developers and its users are wonderful people. I know a lot of them personally. And we owe a lot to them, like we do, because we benefit from Wikipedia. So, let me take this opportunity to say thank you to them. Now the, now the medicine.


[User] Please transcribe this text

[Assistant] So, in 2009, when we first figured out that Wikipedia was declining, this is what, uh, Huggle looked like.


[User] Please transcribe this text

[Assistant] Uh, there's no like, hey thanks for a good edit or anything like that that comes out of clicking the good button.


[User] Please transcribe this text

[Assistant] Uhm, on the left hand side here we have edits sorted by the probability that they're damaging based on that machine learning model so that you review the stuff that's probably bad first and review the probably less bad stuff later.


[User] Please transcribe this text

[Assistant] So, uh, and in 2009 we had this big revelation that newcomers were getting hammered by these kind of tools.


[User] Please transcribe this text

[Assistant] In 2015, there was a major redesign of the Huggle software. And I talked to the developer of Huggle quite a lot about, like, the problems with the software, with how people use it, how it wasn't designed for these sort of things. And so I want to show you how the software changed. So you can see there's still this sort of diff pane where it's showing two versions of the article with the changes. Uhm, there's still these good and bad buttons at the top.


[User] Please transcribe this text

[Assistant] I would say this tool isn't really designed to use that welcome button, and so as far as I can tell, nobody's actually clicked on that button except for when they were running a test. Uhm, so the tool isn't really designed with newcomer socialization in mind, but it does have a button with newcomer socialization in mind.


[User] Please transcribe this text

[Assistant] So, the state of the matter is that quality control in Wikipedia is still not designed with this in mind. Uhm, newcomers and especially those who don't look like the people who joined during the first big wave into Wikipedia remain marginalized. You know, if you don't speak like them, if you don't value the same things that they do, then they'll see you as sketchy, at least different and needing review.


[User] Please transcribe this text

[Assistant] Uhm, and so we're still not seeing gains in the retention of good faith newcomers. Uhm, despite the fact that we've, we've incorporated a lot of conversations into this space, quality control is just the first thing to hit you when you show up at Wikipedia, way before you get invited to the D-House by HostBot.


[User] Please transcribe this text

[Assistant] Uhm, so why did this happen? Why, why did we adjust our objectivities in other places but not in the technologies that were developed to do quality control work?


[User] Please transcribe this text

[Assistant] So, why do we see changes in some of the subsystems of Wikipedia but not others?


[User] Please transcribe this text

[Assistant] So this is what brings me into the third part of my talk, the infrastructure for socio-technical change. What I think is missing from our ability to change in this space.


[User] Please transcribe this text

[Assistant] So, like I was telling you before, I think the machine classifier is really critical to these systems. Uhm, it uses statistics to make predictions about which edits are good and bad. Uhm, so, the three main quality control tools in Wikipedia are Huggle, Cluebot, and Sticky. Huggle and Sticky solve a very similar problem.


[User] Please transcribe this text

[Assistant] Uhm, but they're both used at about an even rate. Uhm, so each of these quality control tools has its own machine learning model.


[User] Please transcribe this text

[Assistant] Uhm, let's say that you had an idea, like you were inspired by my talk and you're gonna go out and you're gonna write the next piece of quality control software for Wikipedia that's going to account for newcomer socialization. So, you're gonna take this extended standpoint and do something with it. The first thing that you're gonna need to do is to build a machine learning model that works in real time in Wikipedia.


[User] Please transcribe this text

[Assistant] So, in order to do that, you should probably read about 20 research papers about Wikipedia damage detection, uhm, and if you're a volunteer developer for Wikipedia.


[User] Please transcribe this text

[Assistant] Even if you had all those things, building one of these things and making it work is extremely labor intensive, they're massive performance considerations, and so you'd probably just pull your hair out and give up.


[User] Please transcribe this text

[Assistant] So, it should be no surprise that all of these tools, all these three tools are authored by computer scientists with extensive skills in machine learning and distributed systems. Sticky, uh, for example, was authored by one of the most prolific authors of Wikipedia damage detection literature, Andrew West.


[User] Please transcribe this text

[Assistant] So imagine frying an egg. Denaturing a protein isn't exactly a chemical reaction, but stick with me. So if you wanted the transparent, uh, egg whites to turn into actual white egg stuff, then you generally have to get it up to about 150 degrees Fahrenheit.


[User] Please transcribe this text

[Assistant] Uhm, but if you had a catalyst, you know, some chemical that would make that reaction happen easier, then, uh, you could maybe fry an egg at room temperature. You wouldn't have to put so much energy into the system in order to get that reaction to take place. That's what's going on in this graph.


[User] Please transcribe this text

[Assistant] So now I want to show you my version of this graph, which is a progress pathway graph. And so, uh, I think that there is essentially a reaction energy problem. If you want to get to the side of better quality control and socialization, you first have to cross this mountain of building a machine learning model that works in real time.


[User] Please transcribe this text

[Assistant] And, but if we had some sort of progress catalyst, some sort of way to make this progress without putting so much energy into it, maybe we would actually get a lot more people crossing that mountain. Uhm.


[User] Please transcribe this text

[Assistant] What we do is we take this idea of the machine learning model and we centralize it. Because all these machine learning models that these different tools are doing, they're roughly doing the same thing in different languages, using different algorithms, but honestly they're making the exact same prediction.


[User] Please transcribe this text

[Assistant] And so they can all use the same central machine prediction system and new tools like Hostbot, the one that routes newcomers, Snuggles, another one that routes newcomers to help spaces, can use that prediction model too. And so if you have an idea, you can also take advantage of this centralized resource of a prediction model to build your tool on top of.


[User] Please transcribe this text

[Assistant] And so I want to show you a little bit about how this works. We call this system ORS. Uhm, it's a data mining metaphor.


[User] Please transcribe this text

[Assistant] Uhm, and so in this case, this is an edit to an article where they're fixing a link. Uh, Wikipedia's called this link disambiguation. They're making sure that when somebody clicks on the old woman in the page, it goes to the old woman, the play, as opposed to the, the page that says, hey, there's ten things called the old woman, which one do you want?


[User] Please transcribe this text

[Assistant] Uhm, so this is a good edit. We ask Oris to make a prediction about it. Oris says, oh, probably not damaging, false. And I'm about, uh, 94% likely that it's not damaging.


[User] Please transcribe this text

[Assistant] Uhm, the whole system is scalable and redundant. If we get a bunch of researchers who are using this for research, you should use this for your research. Uh, then we'll add machines so that we can handle the new capacity. Uhm, and because I've read those research papers, we can make these models comparable to the state of the art.


[User] Please transcribe this text

[Assistant] So, essentially what I hope that the system is, is a progress catalyst that reduces the act, the necessary activation energy to getting to better quality control by about 20,000 lines of code and one advanced degree in computer science. And honestly, I've been working on the system for about three years now, so probably a lot of hours on top of that.


[User] Please transcribe this text

[Assistant] Uhm, and so my hope is that then we'll get a lot more people experimenting in this space using this extended standpoint, experimenting with new objectivities that reify this value of better quality control, better support for newcomers. Uhm, and fun story, Huggles already crossed that gap.


[User] Please transcribe this text

[Assistant] Uh, there's been a lot of tools that have already been developed in this space, about 30 of them. Uh, there's been three major Wikimedia Foundation product initiatives, and there's been a bunch of really cool data science projects. By the way, you should do research on this, it's really cool.


[User] Please transcribe this text

[Assistant] Uhm, okay, so I told you that there were three things, but I wanted to have this, uh, part have its own section, uhm, and so it's three and a half, uh, feminist inspiration, like how we can use critiques of machine learning and social spaces in order to, uh, change the design of these systems.


[User] Please transcribe this text

[Assistant] Uhm, so I wanna use this term subjective algorithms, and I'm borrowing this from Zineb Tufeky, uh, that subjective algorithms are algorithms often aided by big data and now make decisions in subjective realms where there's no right decision and no anchor with which to judge outcomes. Like, what's good, what's relevant, what's important, what's desirable, and what's valuable.


[User] Please transcribe this text

[Assistant] And, uh, actually I don't want to talk about perspective API. Maybe we can come back to that. But in Wikimedia space, we have to deal with these questions all the time. Who's in, and depending on how we answer these questions, we can, uh, we're deciding who's allowed to participate. Who gets labeled bad faith? Who has their edits reverted?


[User] Please transcribe this text

[Assistant] You know, who gets to contribute to Wikipedia? The machine can really have a dramatic effect on these sort of things, so we have to be careful.


[User] Please transcribe this text

[Assistant] Uhm, so this machine learning model, it's of course not perfect. In the case where it says that something's good, it's actually gonna include some important bad stuff that it's gonna make a mistake about. And when it says bad, it's actually gonna include some important good stuff that it makes mistakes about.


[User] Please transcribe this text

[Assistant] And so essentially what WNT is talking about is like, let's say we built a model for harassment in several discourse, but for some reason our, the harassment side of our model picked up messages from South Africans because the way that they use English is strange and so therefore it's more likely to be harassment than we would be silencing a group of people who are already underrepresented on Wikipedia.


[User] Please transcribe this text

[Assistant] Uhm, so I wanna tell you, actually I'm gonna tell you one story because I don't have time, uh, about how we, uh, detected some of this bias in Wikipedia and how we dealt with it.


[User] Please transcribe this text

[Assistant] Uhm, so, uh, in this case, I'm gonna talk to you about how we use the Italian, or how, uh, we picked up the Italian word, ah, and actually we might have a little bit of time, so I might talk to you about the, uh, how we're biased against anonymous editors.


[User] Please transcribe this text

[Assistant] So, uh, the Italian word, ah, which is literally not a laughing matter, um,


[User] Please transcribe this text

[Assistant] So it turns out that we deploy these things for Wikipedians, uhm, and they, they realize like, oh wow, this makes mistakes sometimes, we should start recording those mistakes.


[User] Please transcribe this text

[Assistant] On the very top, you can see RotPunked telling me like, hey, by the way, we have a bunch of false positives related to the Italian verb for have.


[User] Please transcribe this text

[Assistant] Uhm, so what was going on here was that we don't just have bad words like racial slurs and that sort of stuff that we put into the model, we also have informals, such as, uh, and informals are casual speak that would not be welcome, uh, on the article, but would be welcome in a discussion, such as hello, or ha ha ha, you can see where I'm going with this. Uhm, so ha ha, it turns out is a problem.


[User] Please transcribe this text

[Assistant] People vandalize in English on all of the Wikis. And so we use our English language, bad words, and informalists everywhere. Uhm, it turns out that in our informalists from English, we have a regular expression that matches ha and ha ha ha. And that was catching ah in Italian Wikipedia.


[User] Please transcribe this text

[Assistant] And so it turns out that ha is laughing in English and ah is not laughing in Italian. So it turns out that once this was reported to us, it was a really easy problem to fix. We could just remove that one informal word from use in Italian Wikipedia, keep all of the rest of the English informal words, and dramatically improve the model. We would have never seen that unless people were auditing the system for us.


[User] Please transcribe this text

[Assistant] Yeah, I think I can make it through anonymous editors. I'm gonna be a little bit quick about this. Uhm, so apologies for going really fast.


[User] Please transcribe this text

[Assistant] So it turns out we were getting a bunch of reports that when people aren't logged in and they're making edits, the system is flagging that as vandalism all the time. We should probably, probably do something about that. But maybe anons are really bad. Maybe they're actually just vandalizing all the time.


[User] Please transcribe this text

[Assistant] Generally anonymous editors are twice as likely to be vandalizing Wikipedia. However, 90% of their edits are good, so we should probably not flag all of them as bad.


[User] Please transcribe this text

[Assistant] Uh, it turns out that, uh, you can use ORS and inject, uh, features into it. And so you can ask ORS something like, if this edit was saved by an anonymous editor, what would the score be? If it was saved by not an anonymous editor, everything else held constant, what would the score be?


[User] Please transcribe this text

[Assistant] And so you can find out that this particular edit is 11.5% more likely to be considered damaging if it was saved by an anonymous editor, but we changed nothing else about the edit. Aha! Ours has a bias.


[User] Please transcribe this text

[Assistant] Uhm, so you actually have to change a few more of the features in order to put this in place because, you know, anonymous editors don't have a registration date and all that sort of stuff. So I built three user classes and tested out two different modeling strategies to see if I could figure out what was going on here. One of the user classes is an anonymous editor, one is a new editor who just registered, and the other one's me. Uh, epoch fail.


[User] Please transcribe this text

[Assistant] So, uh, the two modeling strategies are a gradient boosting strategy and a linear SVM strategy. So this is not, not trying to affect the data at all, just looking at our test set. Um, so what we wanna see is that the not damaging edits have low probability or squished up against the left, and the damage, uh, the edits that are damaging have high probability or squished up against the right, yeah, the right.


[User] Please transcribe this text

[Assistant] Uhm, and, uh, you know, but we, we see a little bit of a weird thing under the linear SVM model. There's this weird bump of edits that aren't damaging with a high probability. What the heck is going on there? Well, we're gonna figure that out in just a second. So, now let's ask the system, how would these graphs look if every single edit was saved by an anonymous editor?


[User] Please transcribe this text

[Assistant] Well, we can see something quite dramatic here. So under the gradient boosting model, we see that most of the damaging or not damaging edits are up against the left. Most of the damaging edits are up against the right.


[User] Please transcribe this text

[Assistant] For the linear SVM model, all of the edits are up against the right with this super big spike at about 80% likely to be damaging. What the heck?


[User] Please transcribe this text

[Assistant] Uhm, well, okay, well, okay, that's for anons, what about for newcomers?


[User] Please transcribe this text

[Assistant] Alright, now let's ask, what if every edit was saved by an experienced editor like me? Remember, I'm not changing anything about the edit, just who saved it.


[User] Please transcribe this text

[Assistant] Uhm, it turns out that everything is squished up against the left. We generally draw our thresholds at about, uh, 0.5 for likely to be needing review. And so that means that I can get away with vandalizing Wikipedia. Muahahaha, this was part of my design the whole time.


[User] Please transcribe this text

[Assistant] Really, this was a great problem to understand and we didn't quite fix it entirely as you can see in that last slide. However, we realized that a gradient boosting strategy was just far better at discriminating in these cases and it wasn't so biased against anonymous editors.


[User] Please transcribe this text

[Assistant] So we deployed that around December of 2015. Um, but it still needs work. There's still a bias against anonymous editors and newcomers. It's not gone. And so we've been digging into sources of signal that have nothing to do with the person who's making the edit and more to do with the changes that they're making, such as hashing vectorization and probabilistic context for grammars and natural language processing strategy. Okay.


[User] Please transcribe this text

[Assistant] So, I just want to summarize the stuff that I've told you today. So, in part one, we talked about Wikipedia's socio-technical system, brought in that systems thinking biological metaphors. Part two, talked to you about the quality control system. We talked about standpoint epistemology and how we're throwing out the baby with bath water.


[User] Please transcribe this text

[Assistant] Uhm, and then finally in part three talked about how, uh, I see this new system, this new machine learning system as infrastructure for socio-technical change. This idea of a progress catalyst and talk to you about how we can address some of the dangers of subjective algorithms by having an open auditing space where people can talk to us about them. Thank you.


[User] Please transcribe this text

[Assistant] And I actually tried that. It was part of my thesis. It was that snuggle tool I was talking about and mostly didn't work. It, it, it, people thought about things differently because I made a tool that was focused on newcomer socialization. Uhm, but it didn't really get adopted, didn't really fix the problem.


[User] Please transcribe this text

[Assistant] Uhm, so a more, a more, uh, empowerment approach would be, uh, to say like, well, I'm gonna try and make it so that we're reducing barriers so more people can participate in this conversation. And I, I really do see like a technical innovation space as a conversation.


[User] Please transcribe this text

[Assistant] Like, for example, like, take, take the app market. Uhm, when somebody makes an app that does something new, there's suddenly ten more apps that are, are taking advantage of that thing. Well, assuming that it was good. There are a lot of apps that do something new and it was bad and nobody cared about it and nobody does it anymore.


[User] Please transcribe this text

[Assistant] But the really cool thing is that, you know, just like a meme, uhm, the, the concepts, the design, uh, and affordances of these apps will start to proliferate, and suddenly, if it's such a good idea, uh, maybe all the apps suddenly do it, like sharing things on Facebook, we can debate about how good of an idea that it is, but it'll certainly make you add revenue.


[User] Please transcribe this text

[Assistant] And so the, my hope is that people who couldn't enter the conversation before will now enter the conversation. But I'm not able to say what they'll do when they enter the conversation. And so I'm sort of making a bet.


[User] Please transcribe this text

[Assistant] That, I think that this problem is serious, and I think that talking about it openly brings, uh, brings, uh, will help people add that to their standpoint, their notion of value.


[User] Please transcribe this text

[Assistant] And in a way this is not great, because they could not solve the problem. But in another way, like maybe I'm actually wrong about what the real problem is, and somebody else will have an insight into what the real problem is, and now I've, I've still opened the door for them and they're able to operate in the space, the space that they couldn't act, uh, operate in before.


[User] Please transcribe this text

[Assistant] Uhm, and so what we're seeing a lot of is most of these tools that I was talking about are not developed by, uh, people with extensive computer science backgrounds. It turns out that Huggle is about, it's about 20,000 lines of code. And as soon as we deployed the service, somebody made a thing that does roughly what Huggle does in about 80 lines of JavaScript.


[User] Please transcribe this text

[Assistant] Uhm, and so, like, it's phenomenal, like, the difference that this makes for, like, people experimenting in this space. And so we've seen, like, the conversation definitely has expanded beyond the people that were involved in it before, or the types of people who were involved in it before, but it's still a bet.


[User] Please transcribe this text

[Assistant] for what they're gonna do with it. Uhm, my hope is that either they're going to do what I hope that they'll do with it, or they're going to do something that's actually better than that that I was too stupid to see.


[User] Please transcribe this text

[Assistant] You know, I struggle with that too, like, like it's much easier for me to look at shifting the technology than it is to shifting culture because like, I mean that's, that's been an open area of study for forever, uhm, as long as social science. Uhm, but my, my bet is by like getting new people in the door, that's, that's a strong way to shift culture, but uhm,


[User] Please transcribe this text

[Assistant] I guess eh, like one thing that this benefits from is at the end of the day we want encyclopedia content. And it turns out like, let's say that we just banned anonymous editors. I've looked into this. We would lose about 20% of our productivity. And so then you get to ask the question is, is doing quality control worth


[User] Please transcribe this text

[Assistant] 20% of productivity, like doing, doing 27 hours of quality control a day. Uhm, and, uh, the answer is, like, if I talk to people about that in Wikipedia, they're like, oh yeah, obviously.


[User] Please transcribe this text

[Assistant] And, and I think it's that way for newcomers too, like people see that, but the problem is like, you can't-


[User] Please transcribe this text

[Assistant] Have a personal interaction with everybody who shows up in Wikipedia.


[User] Please transcribe this text

[Assistant] But, we can do things with oars, and I've already been experimenting with this, for targeting newcomers who are most in need of support. The most likely to be productive, just based on their initial excitement about Wikipedia, not the quality of their work.


[User] Please transcribe this text

[Assistant] And the least likely to actually be vandals who will just waste your time if you try and go have an interaction with them. Right now we don't have that, and so we're not even taking advantage of those really, those few but key places where we can have a big effect.


[User] Please transcribe this text

[Assistant] And so that's where I think technology could help. That's where I think like the innovation space might move. But you're right, we can't do everything. But we can optimize for the things that we've never been able to optimize for before.


[User] Please transcribe this text

[Assistant] And so I would say, like, this is much more paving the way than it is, uh, just opening the door. You know, anybody could build a machine learning model and run a bot on Wikipedia before.


[User] Please transcribe this text

[Assistant] In order to, to do that sort of thing. And so, like, take in, like, considering the, the rest of the open source world, I see something like Jupyter Notebooks, a lot like how I see ORS. Jupyter Notebooks paves the way for people to learn data science strategies. Among other things that you can do with them, but I think that that's a really killer use case.


[User] Please transcribe this text

[Assistant] Uhm, if you're not familiar with Jupyter Notebooks, they're, they're like a web-based interface that allows you to write, say, Python or R, statistical languages, programming languages, without having to install anything.


[User] Please transcribe this text

[Assistant] Like Jupiter Hub, for example, will let you run the entire thing on a remote server. You don't have to learn about command line interfaces or installation dependencies. You can just open a web browser and start writing Python.


[User] Please transcribe this text

[Assistant] You're gonna have to learn Python, you're gonna have to learn statistical methods, but there's a huge mountain of other crap that you don't need to even start with before, before you get involved in, in data science. And so, so yea, that's where I would draw the line between like ORS and Jupiter versus just open source and open bug trackers and pull requests.


[User] Please transcribe this text

[Assistant] So a lot of other Wikipedias look exactly like English Wikipedia, especially German Wikipedia. German Wikipedia arguably has a more deletionist, quality focused culture and they use these tools much more aggressively towards new editors and they have an even more aggressive decline.


[User] Please transcribe this text

[Assistant] Uhm, whereas, uh, Japanese Wikipedia is much more open, much less focused on quality, much more like, uhm, you know, any sort of pop culture topic is welcome to be written about in the encyclopedia, and they don't have a decline that looks like this. They have, they have something that looks much more like an early late adopter curve, where it's like logistic growth that flattens out, doesn't, you know, doesn't collapse.


[User] Please transcribe this text

[Assistant] Uhm, uh, over time. And so yeaah, we definitely see this in different languages. And there's, there's a study that I know is coming out soon, uhm, that I don't think is out yet.


[User] Please transcribe this text

[Assistant] Which are, uhm, wiki is sort of like a space for hosting hobby wikis. Uhm, so like the Battlestar Galactica wiki and the Star Wars wiki and that sort of stuff are in this space. And they exhibit very similar patterns. So it seems like this pattern of focusing on quality and forgetting about newcomer socialization is really common.


[User] Please transcribe this text

[Assistant] You know, what my hope is, is with this work we can maybe stop doing that or at least make that shift lesser and lesser. We can still figure out how we can optimize for all of it. Optimize for quality control and still


[User] Please transcribe this text

[Assistant] Uh, have newcomer socialization and so maybe like new communities won't go through such a, uh, a hard shift, uhm, where they have this problem or maybe even the old communities like English Wikipedia might be able to recover substantially.


[User] Please transcribe this text

[Assistant] So I think Mohammed bin Salman is doing the right things by introducing Vision 2030. Now he just has to do it right.


[User] Please transcribe this text

[Assistant] You go and go and never give up. Against the wind, you keep on running. Even when you stumble with four of the rocks, you never stop fighting.


[User] Please transcribe this text

[Assistant] Yeah, yeah, Young Rob in the building, 1207 to the top, boy. It was good.


[User] Please transcribe this text

[Assistant] Yeaah, man. That shit was just crazy. I mean, it grew. That concert, Welcome to the Woods. We did that shit with the Healthy Boys and Xavier Wolf. That shit was just crazy, man.


[User] Please transcribe this text

[Assistant] Yeaah, it was like, last summer I would say? 14?


[User] Please transcribe this text

[Assistant] As soon as it hit the first, like, 50, 100,000, you know what I mean? I was like, woah, like, you know what I mean?


[User] Please transcribe this text

[Assistant] Just everybody that saw it, pretty much everybody that just pretty much came across it, just like was hella fucking with it and just pushed it.


[User] Please transcribe this text

[Assistant] Like it's just, and it was word of mouth, you know what I mean? It just going around like, that shit just blew up, I don't even know how.


[User] Please transcribe this text

[Assistant] Uh, nah, not at all, like, I was just like, getting back into it, you know, I had started going to the studio, maybe a few months before that, and I had moved back to San Diego, cause I went to Atlanta for a little bit.


[User] Please transcribe this text

[Assistant] And then, uh, yeaah, we just went into the studio, made that shit, and that shit was just like...


[User] Please transcribe this text

[Assistant] Nah, uh, Jay Davis had found the beat and, uh, he just loaded it up one day and, uh, but yeah, like, yeah, I grew, uh, fucking started blowing up and shit. So, you know, I had to holler at a perp dog eventually. I mean, then we chopped it up and, you know, we did business. All is well with it.


[User] Please transcribe this text

[Assistant] Nah, it's just love and support, you know what I mean? Like straight up like that, you know what I mean? I support his craft. Okay.


[User] Please transcribe this text

[Assistant] Uh, predominantly my boy Dubz. He's, uh, straight from New Jersey. And, uh, my boy Jepz. And, uh, right now we've been working with this dude, Flame Alcahes. Yeah, that's who that bangers, man. Yeah, man, he made Suspects and shit. That's like the next song that's like growing up a little bit. Right. But yeah, man, uh, those are like the main producers right now, man.


[User] Please transcribe this text

[Assistant] It's just crazy out there, like, you just gotta really be from the grove, you know what I mean? To really understand it, like, so Lemon Grove is like,


[User] Please transcribe this text

[Assistant] Damn near the heart of San Diego, you know, it's central to everywhere, East County, downtown, the Southeast, it's central, you know what I mean? So it's like a.


[User] Please transcribe this text

[Assistant] Oh man, I went to Helix, you know what I mean, like, like I said, Lemon Grove is a big ass melting pot, so Helix was in La Mesa, which is like right next to Lemon Grove, but that's a, no, everybody from everywhere went to Helix, it's a charter school, so like, you know, everybody from everywhere, even, I know people that lived in Temecula went to Helix.


[User] Please transcribe this text

[Assistant] Uh, I don't know man, I can't even say. Maybe, probably just like the big ass events they would throw like.


[User] Please transcribe this text

[Assistant] No, there was some fights and all that, but that's just everything. But it didn't get that crazy?


[User] Please transcribe this text

[Assistant] I would say like the parties maybe is the craziest, the craziest, craziest, cause yeaah, the police would show up, you know, 12 cars thick, you know, everybody running down the street. I didn't have to carry my homies in the bushes, like, just hella faded, like, man, the parties definitely get the wild, like, Project X, but not as many people.


[User] Please transcribe this text

[Assistant] Just that, like, when I get into, like, an artist, like, I, like, I find something that I like and I just, like, stick with that. Like, sometimes I'll dig into it depending who it is, but a lot of times I just stick with what I like.


[User] Please transcribe this text

[Assistant] When I, when I first started, you know what I mean, I was trying to be all lyrical, but then that was just not, that was just not me, you know what I mean, and then I just had to find me, find my voice, and then, uh, nah.


[User] Please transcribe this text

[Assistant] Yeah man, it's trash man, but it's crazy man, just being a father, that shit's life changing.


[User] Please transcribe this text

[Assistant] Yeah man, she hold me down man, you know what I mean? She just like, she keep me, she balance, she keep me balanced.


[User] Please transcribe this text

[Assistant] Nah, man. You know what I mean? Girls like that, you gotta stay away from girls like that. You gotta have them girls that, you know, they live their own life, but you gotta know they down for you. You know what I mean? You can't have a girl that's all up in your shit, man.


[User] Please transcribe this text

[Assistant] You shouldn't have to pay money to see your child, period.


[User] Please transcribe this text

[Assistant] 12 is cause the L from lemon is the 12th letter in the alphabet and then G, the 7th. Okay. Or it's lemon grove, basically. Yeah. Straight up.


[User] Please transcribe this text

[Assistant] And nah, like I put out the song, but I was working on my first mixtape, Straight Bummin'. You know what I mean? So, really Straight Bummin' took me like...


[User] Please transcribe this text

[Assistant] Uh, most of the time Chill Bill was like the first song I finished. Okay. From the tape. So, you know, I was just working on Straight Bum and like, and I know I can't ride the wave. I ain't ever tried to ride the wave. I just stay working. Yeah, yeah. Cause all the time like, shit, I'm going to studio ride after this.


[User] Please transcribe this text

[Assistant] February? February? Yeah, it came out February. Like early February. I don't remember the exact day. I want to say like the 12th. Right.


[User] Please transcribe this text

[Assistant] Yeah, man, but that shit just like, the whole mixtape, it like picked up.


[User] Please transcribe this text

[Assistant] A lot of attention as a whole, but of course Chill Bill was just like...


[User] Please transcribe this text

[Assistant] I heard it's a twisted nerve, that's the actual sample, but I first heard it in Kill Bill, the movie.


[User] Please transcribe this text

[Assistant] Oh yea, man, I'm a big fan of you. I'm not really good.


[User] Please transcribe this text

[Assistant] Excuse me. Yeah, damn. Uh, but yeah, I'm working on my first album right now actually. It's called Seven Gates to Twelve. I'm gonna drop it February 12th again.


[User] Please transcribe this text

[Assistant] Yeaah, about a month. Yeaah, it's almost done, man. But, uh.


[User] Please transcribe this text

[Assistant] I got some Daigo collabs too, uh, Rossi, and uh, I'm work- I'ma definitely get a song with, uh, my boy B-Wat. Okay. From Daigo, uh.


[User] Please transcribe this text

[Assistant] Yeaah man, uh, but yeaah. Really I do mostly my features with my squad, you know, Spooks, Malik Burgers, Tom Cruise, Jay Davis, uh, Jay Davis, I was on Chill Bill, uh.


[User] Please transcribe this text

[Assistant] Keely, fuckin' Adam, he's a producer, he DJs, uh, our shows and all that, but yeaah, man. Most of my collabs come from The Squad, like my whole first tape was strictly The Squad, my whole first album was The Squad.


[User] Please transcribe this text

[Assistant] Swash it. Swash it. Yeah, man, I'm definitely doing, uh, hella videos, man. Hella videos, uh, we just dropped our latest video, Rudecy. Uh huh. It's Boog's song, uh, featuring myself. Shout out to Alex Vibe. Yeah, shout out to my boy Alex Vibe and the cat, you know what I'm saying? Yeah, man.


[User] Please transcribe this text

[Assistant] Real shit man. Real shit. I pay a lot of attention to videos and like movies.


[User] Please transcribe this text

[Assistant] How was it? Uh, LSD changed my life. Really? How long was this? Uh, the first time I did LSD was in December. Uh, man, my mom's gonna see this, man. Mom understands. What's your mom's name? I'm out here. I'm out here. What's her name? Marlin.


[User] Please transcribe this text

[Assistant] Dude, I would love to fucking see, uh, fucking Kid Cudi live, like, that's like a life goal of mine, no doubt.


[User] Please transcribe this text

[Assistant] I do that too, but I write, definitely. Yeaah. Yeaah, man. I think writing is important. I got fucking A's and B's in English, my niggas. Really? Okay. English is important, I don't care what nobody says.


[User] Please transcribe this text

[Assistant] I mean, like, I think, uh, not writing and just going out and feeling is good because it's just natural, you know what I mean? It's what comes out, you know what I mean?


[User] Please transcribe this text

[Assistant] I think, uh, like, I fuck with Drake, but like, I don't know, I don't, I wouldn't have nobody write him.


[User] Please transcribe this text

[Assistant] I actually, I had tweeted her and told her that I want her to host my next mixtape.


[User] Please transcribe this text

[Assistant] I wanna learn how to skate though, you know, I be trying to do my little, my little whoop and all that, but, but I get.


[User] Please transcribe this text

[Assistant] I don't, I don't really like feel the fake using it, you know what I mean? Oh, you don't feel that, yeaah? Let me just say that.


[User] Please transcribe this text

[Assistant] Goddamn. L.A.'s crazy with the gangbang, you mean?


[User] Please transcribe this text

[Assistant] That shit, alright, one morning man, me, Spookz, J Davis, Dom Cruise, we linked up and, uh, yea, we was about to go hit a little house leak, you know what I mean, checked out the spot. As soon as we hopped out the car, fucking, the police come.


[User] Please transcribe this text

[Assistant] Rolling right down the street, fucking, hemmed us up, all the shit, put, uh, me and Dom Cruise in the back, of course, cause we're the black guys. Uh huh. You know, the profile and stuff. Wait, so they arrested the black guys? No, no, no, they didn't arrest us. Cause they didn't have nothing on us. Okay. I mean, they just.


[User] Please transcribe this text

[Assistant] Yeah man, it was just a rough time, grimy shit like.


[User] Please transcribe this text

[Assistant] But, uh, anyways, yeaah, but they let us go and shit and then, fuck it, we just went to, uh, uh, to the homie pad and then, uh,


[User] Please transcribe this text

[Assistant] And then we just fucking booked some hours and went and made chill bill. And so it's like, if the police would have never came down that street and stopped us, there would be no chill bill.


[User] Please transcribe this text

[Assistant] Like I said, two damn phones, cause I had two damn phones at the time, you know what I mean? And then they had found, I had lost my little trap phone and then they had found that shit, so they were calling the number and shit. But fucking, uh, they were calling, fucking, I don't know what number I gave them, but they were calling one of the phones.


[User] Please transcribe this text

[Assistant] And fucking, uh, but I found that tri-phone, but, so I had to answer that shit.


[User] Please transcribe this text

[Assistant] I'll just be doing my shit, you know what I mean? I'm a smart little young nigga out here. I mean a lot of people are dumb.


[User] Please transcribe this text

[Assistant] I mean, just pay attention, just be on your toes, man. Just pay attention to everything, you know what I mean? But, I stay out of trouble because I was in the football and all that shit, you know what I mean? Like, football was like, I was hella in it as a kid, but like, in high school, I played for Helix, man, and I just fell off, man. That's when I just, just like, started running the streets hella, man.


[User] Please transcribe this text

[Assistant] I went to Atlanta and then fucking started doing music and then came back.


[User] Please transcribe this text

[Assistant] And then I was just there, like, I felt like every class was a waste of my time. Like, everything I was learning in English I already knew. Like, I know how to write a good ass essay already, man. I know how to read a fucking book, man. I know how to talk to people, I understand what the fuck they're saying.


[User] Please transcribe this text

[Assistant] Yeah, man, like real shit, like I saw how fucking chill bill was progressing, like honestly when I first started I knew like some shit was gonna take off, like I don't know how, like. You just have faith in yourself. Exactly. I always believed in myself. People be getting that fucked up. You gotta believe in yourself, real shit.


[User] Please transcribe this text

[Assistant] If you're gonna do this shit, do this shit. Straight up, like, just dedicate yourself to that shit, man. Like, believe in yourself, don't give a fuck what nobody said, man.


[User] Please transcribe this text

[Assistant] Like for real. And that's what anything you do, it's not even just rap.


[User] Please transcribe this text

[Assistant] I just had a kid and I'm all the way up in LA for the week. You know what I mean? Sacrifice is main, but you gotta decide what you want.


[User] Please transcribe this text

[Assistant] I need to get out, I need to get out in Houston. Real shit I need to meet all the motherfuckers in Houston.


[User] Please transcribe this text

[Assistant] For the Saints, they fall out of it. The offense undercar wasn't fantastic. The Saints.


[User] Please transcribe this text

[Assistant] Get our closest what they are because of a special teams kick return that we don't usually get. Last time we had actually a punt return.


[User] Please transcribe this text

[Assistant] And the offense couldn't even put up two touchdowns in the game that could have won it for you. The offense is not even good enough.


[User] Please transcribe this text

[Assistant] To kick, to get you two touchdowns on the game. You got two veteran quarterbacks. Couldn't get it done.


[User] Please transcribe this text

[Assistant] A lot of answers to what needs to happen here for the Saints going forward, but this is going to be the final here. The Saints are going to lose this game.


[User] Please transcribe this text

[Assistant] Epic collapse in Green Bay about a New Orleans Saints man, they go down and burning non-glory. Saints go down and burning non-glory. All right, let's hit the, uh, and y'all stay right with us, fam. We're going to have


[User] Please transcribe this text

[Assistant] It's awful, it's awful, but this stinks. But in the end, man, this, my goodness. The Green Bay Packers came from behind and took the game from the Saints.


[User] Please transcribe this text

[Assistant] And a saint's offense is to blame, man. You won't please blame.


[User] Please transcribe this text

[Assistant] Who else do you put it on? The Saints offense gave one touchdown a day, the special teams gave the other one. You know? I mean, good grief.


[User] Please transcribe this text

[Assistant] And that's it. Saints lose, man. 18. So that's it. This team statistics here and the matchup, man. The saints go down.


[User] Please transcribe this text

[Assistant] In a match up 21 to 15 with the first down, Green Bay came from behind.


[User] Please transcribe this text

[Assistant] Third down, they went from, they were at eight of 18 on third down, the Saints imploded four of 14 on third down.


[User] Please transcribe this text

[Assistant] They just took it in the second half and blew up. They did everything in the fourth quarter to undermine the Saints. Saints defense was super tired because the offense was pathetic.


[User] Please transcribe this text

[Assistant] Advantage Green Bay 340 to 252 on total yards. Advantage Green Bay 12 to 10 on the drives. Advantage Green Bay.


[User] Please transcribe this text

[Assistant] Almost five yards per play advantage Green Bay over New Orleans 4.2, 245 versus 175 on the pass. The running attack 95 to 77 advantage Green Bay, everything advantage Green Bay in the end.


[User] Please transcribe this text

[Assistant] Despite the fact Green Bay had 11 penalties for 90 yards, the Saints had 7 penalties for 102 yards.


[User] Please transcribe this text

[Assistant] The Packers went inside St. Territory with two or three. All that happened in the second half.


[User] Please transcribe this text

[Assistant] They scored as much as they needed to win to beat the saints who are one and two in the red zone. They even took the time possession 32 to 10, 32, 10 to 27, 25 and the matchup going into the box score. You look at what happened with Derek Carr. He got knocked out of the game 13.


[User] Please transcribe this text

[Assistant] He was sacked three times. His rating was 104.6. He didn't look all that good either. While he was in there, Jameis Winston 10 of 16.


[User] Please transcribe this text

[Assistant] Uh, for 101, average 6 over 6 shots per throw, he was sacked once. His rating was 80.5.


[User] Please transcribe this text

[Assistant] Kendra Miller led all rushers, nine carries for 34 yards, Tony Jones eight for 31, Taysom's four.


[User] Please transcribe this text

[Assistant] Carries for 12 yards. Chris Olavi was the top receiver.


[User] Please transcribe this text

[Assistant] On a day eight catches for a buck old four off 11 targets.


[User] Please transcribe this text

[Assistant] Michael Thomas, six catches off of nine targets for 50 yards. Tony Jones, four for 21 off of four. Jawan Johnson, two.


[User] Please transcribe this text

[Assistant] Uh, for uh, two catches for 12 yards over four targets and a matchup and Michael Thomas had no catches.


[User] Please transcribe this text

[Assistant] And the first half of the game had six for 50 after the fake. Jimmy Graham had one target for one touchdown in the game. Uh, that's the only offensive score that the, the, uh, Saints could muster.


[User] Please transcribe this text

[Assistant] Sports Rashid Shahid was the, gave him the punt return for a touchdown. Pete Warner led all tacklers. He had 11 total tackles in the game. Ladi Dadi was second. Alonti Taylor had five tackles and a sack.


[User] Please transcribe this text

[Assistant] The Green Bay Packers only allowed the Saints to sack Jordan Love one time in a game, and that's when Alonti Taylor came in there and got him.


[User] Please transcribe this text

[Assistant] Saints defense couldn't get the Jordan love today, despite the fact that two of their best offensive linemen were missing. Outside of that, Lonnie Johnson Jr. did get an interception. Rashid Shahid did his thing.


[User] Please transcribe this text

[Assistant] Blake group a one for two, he missed the 46 or whatever it was field goal.


[User] Please transcribe this text

[Assistant] They could have won the game for the Saints, but that's what I was saying about family members. You got a rookie in there. You got to be patient with them. The rookie miss a game winning field goal. It happens every year.


[User] Please transcribe this text

[Assistant] It always, it doesn't happen too many punts for Luke Hadley at seven for 293, two of the seven inside the 20 average almost 42 yards per point. And you look at the Green Bay Packers play.


[User] Please transcribe this text

[Assistant] Of, uh, Jordan Love, 22, a 44, boy did he really step it up.


[User] Please transcribe this text

[Assistant] 259 yards, throwing the ball, one touchdown and the pick is rating with 66.4.


[User] Please transcribe this text

[Assistant] Romeo Dubs had five catches for 73 yards in a touchdown. Reed called three for 63. Musgrave had six for 49 and Deontay Vion. Wicks had four catches for 45 yards for the Green Bay club.


[User] Please transcribe this text

[Assistant] Outside at Rudy Ford, Darnell Savage and Quay Walker all tied with tackles.


[User] Please transcribe this text

[Assistant] For them and they were only, the Saints only gave up, they gave up four sacks there. Three of, three of the sacks was by Rashawn Gary. He had three sacks. Kenny Clark had one in the matchup for the Green Bay Packers. So shout out to Green Bay. They win their two and one. The Saints have pushed into two and they are two and one as well. They had an opportunity to go ahead in the division.


[User] Please transcribe this text

[Assistant] Unfortunately, it is what it is. Uh, the Saints, uh, gave up 18 points in the fourth quarter, an epic collapse offense for, since Derek Harle went down, they had


[User] Please transcribe this text

[Assistant] I want to say five points, four or five points straight and then that last miss field goal, totally, totally terrible in a game. So anyway, I don't know what to say about this one, man.


[User] Please transcribe this text

[Assistant] I don't know, this was one of the games where you look at and you say to yourself that the Saints should win this game.


[User] Please transcribe this text

[Assistant] This should be a game in which the Saints win. It should be a contest in which the Saints win. And it simply wasn't. The Saints allow 18 points in the fourth quarter and they lose. They gotta put their heads down and leave out of Green Bay.


[User] Please transcribe this text

[Assistant] A two in one club. So next up is Tampa Bay down the line, but.


[User] Please transcribe this text

[Assistant] This is just one of those ones that just, just gonna stink. It's gonna stink. It's gonna stink. Cause they had the game and gave it up. We've seen this last year happen too, but, uh, whatever.


[User] Please transcribe this text

[Assistant] Like I said, we're gonna have the post game momentarily. Let me get y'all guys' thoughts on the game, y'all. Please put it in the comments section.


[User] Please transcribe this text

[Assistant] Let me know how y'all feel. Bruce says Taylor did not interfere. The refs gave Packer six. Corey says, uh, why can't we lose the right way? Why has always got to be something embarrassing.


[User] Please transcribe this text

[Assistant] Uh, Hemisphere says, you can't keep putting your defense out there without an offense and expect to win games. You can't do it. That's, that is absolutely right. A lot of people are going to say, well, the defense did this, that and the third. The defense was tired, man. The defense didn't play like bad because they ain't, they were, they were, they were tired.


[User] Please transcribe this text

[Assistant] They were tied because the offense kept going three and out. The offense had four straight.


[User] Please transcribe this text

[Assistant] I want to say five drives where they were just like, since car got knocked out of the game, where they were just this straight punting. And then the last drive, they took it down there and the best they can do.


[User] Please transcribe this text

[Assistant] It said a rookie punter, a rookie kicker up for a 46 yard kill, field goal. That's the best we could do. And we got Michael Thomas, Chris Olavi.


[User] Please transcribe this text

[Assistant] James Winston, who's a veteran, had been here for three years. That's the, running the same offense, that's the best that we could do.


[User] Please transcribe this text

[Assistant] In essence, James was supposed to get in there and operate that offense better than Derek Carr did. You know, he's supposed to be able to take that team down there and four or five drives and get a touchdown. You know, do something, don't just be punting.


[User] Please transcribe this text

[Assistant] So I mean, it's just, it's already a done deal, man. It's over with.


[User] Please transcribe this text

[Assistant] Alright, Lemuel says, yeah Q, we not doing nothing. This brother Fincher says, let's shift to the shortest and available for any.


[User] Please transcribe this text

[Assistant] Have decade. Thomas says Carmichael got conservative when car went down and that was a wrap. Paul says, hope we get full team next week. AK switch them. Damn tackles. Switch them up. That ain't working.


[User] Please transcribe this text

[Assistant] Backwoods says Wowsers. Yeah, it was a major collapse by the Saints today, man. Big collapse. They had that game.


[User] Please transcribe this text

[Assistant] There were about 17 points and let them people score 18 in the back end to take the game. And we knew what time it was when it was.


[User] Please transcribe this text

[Assistant] Running like that. Chris says, if you wasn't going to be aggressive, you should have put the ball where the kicker needed it. Yeah, that's right. At least.


[User] Please transcribe this text

[Assistant] If you're going to line it up, let, let, straighten it up. You know, put it directly in the center to make them easy. A lot of stuff, man. King Chozma says saddle, O-line, captain, no show up. Wallace native says, I believe James did great in spite of the play calls that Carmichael gave. Mr. Barnes says, I knew we would lose this one. We will bounce back. Joshua says, where are all of y'all calling for Winston now? Y'all quiet. All right. Jody says car trash. So it's Carmichael.


[User] Please transcribe this text

[Assistant] Alright, Alvin says Carl looked no different than Carl, yet we played scared. James, I mean, James looked no...


[User] Please transcribe this text

[Assistant] I guess as you say in Jamus, Joshua says yeaah, 100 on Winston.