[User] Please transcribe this text

[Assistant] If these people wanted this algorithm, they were


[User] Please transcribe this text

[Assistant] How becoming aware of this algorithm can change their behavior. So what we did, it wasn't a control experiment, so we could not actually see how this changed, but what we did was a follow-up with users two to six months after this study. We asked them if after this study they changed anything.


[User] Please transcribe this text

[Assistant] And we found that many of these users changed their engagement with the system in different and sometimes interesting ways.


[User] Please transcribe this text

[Assistant] For example, they started to manipulate the manipulator, meaning that they said that, for example, now I know if you click on like, it's going to show more of this person to me or more of this topic, so I'm going to be selective.


[User] Please transcribe this text

[Assistant] And some of these people may not change their behavior, but they change their emotions. For example, the person said I feel less snug when I post something and...


[User] Please transcribe this text

[Assistant] No one likes or comments or just a few people. I think this is the algorithm that doesn't show it to enough people. It's not my friends.


[User] Please transcribe this text

[Assistant] This shows that other than the extreme cases that kind of their trust was broken with the system. This shows that if you provide some visibility to the system, it still can help users to be satisfied with the system but become more engaged and intelligent users.


[User] Please transcribe this text

[Assistant] And I'm happy that a few months ago, Facebook actually...


[User] Please transcribe this text

[Assistant] Added this, why am I seeing this post? I encourage you to check and see you are on that list of users who get that or not. I'm still not getting this. But it gives you some information about why you are seeing this and shows you that not everything is there. So giving you some hints about that.


[User] Please transcribe this text

[Assistant] But the question here is that we know that simple design could be helpful and we have transparency on the other side. That everything is transparent. But how much transparency do we need? Like do we want a system to be fully transparent?


[User] Please transcribe this text

[Assistant] And transparency is actually a complex problem. Transparency is a sweet word, but it comes with a lot of, uh, costs. For example,


[User] Please transcribe this text

[Assistant] Explaining an algorithm in a simple way is really hard, even for the developers of the system.


[User] Please transcribe this text

[Assistant] Users interaction with the system will become cumbersome if you want to overwhelm them with a machine learning algorithm with hundreds of thousands of factors.


[User] Please transcribe this text

[Assistant] And even if it's possible, people are gonna use that to game the system, no matter, uh,


[User] Please transcribe this text

[Assistant] To understand, like if there is a practical way.


[User] Please transcribe this text

[Assistant] To help users to understand what's going on in the system without interfering the interaction with the system. In the last part of my talk, I'm going to talk about designing for algorithm transparency.


[User] Please transcribe this text

[Assistant] And it's about how we can add visibility to the operation of the algorithm, not just the presence of an algorithm.


[User] Please transcribe this text

[Assistant] And I'm gonna turn my attention to online behavioral advertising algorithms.


[User] Please transcribe this text

[Assistant] Because they received a lot of attention during previous years.


[User] Please transcribe this text

[Assistant] As you see here, this is my Facebook page, but these are the ads I get. And as you see...


[User] Please transcribe this text

[Assistant] It's because I searched the boss baby for my new, my niece.


[User] Please transcribe this text

[Assistant] And my husband's looking for all of us over the internet and our accounts are connected.


[User] Please transcribe this text

[Assistant] So I kind of have an idea of what's going on in this system, like why I got this ad. But there are many times that you get an ad and you just wonder why I got this ad. Is this anything related to me? Is it random?


[User] Please transcribe this text

[Assistant] Turn off your ad blocker, send you about 11,000 online ads per month. Which is not something that you can just, trivial that you can just ignore. But the question is that


[User] Please transcribe this text

[Assistant] How you get these ads? You distill, some of them are obvious, like the ones I showed you, but there are many of them that know how you use your data, and you don't know how these, uh, algorithms work. And this has caused privacy concerns, mistrust, and a lot of dissatisfaction among users.


[User] Please transcribe this text

[Assistant] So, resulting in calls for algorithmic transparency,


[User] Please transcribe this text

[Assistant] Because of these, uh, opacities, many users ask that for a firmer understanding what's going on with the system in advertising.


[User] Please transcribe this text

[Assistant] Researchers also argue that users need openness with these marketers about how an ad is targeted to a user.


[User] Please transcribe this text

[Assistant] And if you heard about GDPR, it's a couple years that European regulations actually


[User] Please transcribe this text

[Assistant] Uh, make these systems to give users an explanation about an algorithmic decision, including advertisement.


[User] Please transcribe this text

[Assistant] So, what we did in this study to understand what we need to give users in terms of transparency, we evaluated some existing transparency mechanisms in online behavioral advertising, helping users


[User] Please transcribe this text

[Assistant] Trying to understand users, uh, in terms of their needs about transparency.


[User] Please transcribe this text

[Assistant] We did the participatory and speculative design task meaning that we also asked users to design their desired level of transparency.


[User] Please transcribe this text

[Assistant] We expose users to three views in an advertising system, which is what a user sees, what an advertising algorithm sees, and what an advertiser sees. In terms of what a user sees,


[User] Please transcribe this text

[Assistant] We show them their real ads, we ask them to bring their personal device.


[User] Please transcribe this text

[Assistant] Uh, like browse their real ads and look if there is an explanation about why they are seeing the ad and then explore the explanation.


[User] Please transcribe this text

[Assistant] Then we expose them to what an advertising algorithm knows about them. And with that, we actually showed them


[User] Please transcribe this text

[Assistant] A few views that some advertising companies


[User] Please transcribe this text

[Assistant] Choose to show to users in terms of the interests or, uh, like, uh, hobbies or interests they have.


[User] Please transcribe this text

[Assistant] I'm so famous in my family with liking the tuflas and overusing them. I have backup for backups in my backpack. You can see two of them.


[User] Please transcribe this text

[Assistant] But I've never searched online for dental floss. I've never even purchased one dental floss and I have no idea.


[User] Please transcribe this text

[Assistant] How Facebook found it and why it just categories it as an interest.


[User] Please transcribe this text

[Assistant] So, I encourage you to explore this. This has funny or interesting, eh, interests that it's gonna show you.


[User] Please transcribe this text

[Assistant] And finally, we showed them the advertiser view, meaning that we told them how an advertiser gets to target an audience. And here on Facebook, because Facebook has the most complex


[User] Please transcribe this text

[Assistant] Uh, advertising with like, uh, interface with more than 29,000 categories that they can use to target users.


[User] Please transcribe this text

[Assistant] And like, we walk them through some of these, like show them what kind of details they can target.


[User] Please transcribe this text

[Assistant] Now, choose a product of interest you want to buy in the next month.


[User] Please transcribe this text

[Assistant] Use the advertising interface to target an audience for that product. But make sure that when you target the audience, they are people like you. Meaning that if you post this ad on Facebook, you are gonna get this ad.


[User] Please transcribe this text

[Assistant] And then, we ask them to write an explanation for this ad as well.


[User] Please transcribe this text

[Assistant] Now users, because they are the advertiser themselves, they know the exact reasons why this ad was targeted. They have full transparency. So we wanted to know how much they want to actually use in the explanation. So I'm going to start talking about first


[User] Please transcribe this text

[Assistant] Uh, the evaluation of the current existing, uh, current existing transparency mechanisms and how they reflected those in their design of their own desired transparency.


[User] Please transcribe this text

[Assistant] So first, we found that users like interpretable explanation, which is no surprise.


[User] Please transcribe this text

[Assistant] So if it's an explanation, it's like that. This ad has been matched to your interest. It was based on your browsing activity. As users say, it's a black box. It doesn't give you anything other than just...


[User] Please transcribe this text

[Assistant] On the other side, there were some, uh, specific explanations that actually people liked. For example, this says Facebook for developers wants to reach people who have visited this page.


[User] Please transcribe this text

[Assistant] So, kind of giving you some specific explanation of what's going on, or if you're in this demographic.


[User] Please transcribe this text

[Assistant] But we found that there is something about having enough transparency, not full transparency.


[User] Please transcribe this text

[Assistant] And it's about having a linkage to identity. I'm going to explain you here.


[User] Please transcribe this text

[Assistant] So when we ask users to design their desired level of transparency about the ad that has got the target to themselves,


[User] Please transcribe this text

[Assistant] We saw that when they choose for example here like eight or nine categories, they choose just a few of this.


[User] Please transcribe this text

[Assistant] And when we ask, and this is just an example, most of the participants are like this, when we ask them why you don't have the complete explanation, why you just do not list everything here,


[User] Please transcribe this text

[Assistant] This does not mean that we don't want to provide users with full transparency. It means that if you want to have a design that doesn't get in the way of users, you might need


[User] Please transcribe this text

[Assistant] I'm gonna also show you here in more detail what this linkage to identity mean in advertising.


[User] Please transcribe this text

[Assistant] So when users explore their inferred interest by the algorithm,


[User] Please transcribe this text

[Assistant] When they saw a feature that they were proud of, they were, they kind of were happy. They were satisfied with the algorithm, like this person. But if they saw


[User] Please transcribe this text

[Assistant] A feature that they were not happy with, like this late technology adopter label.


[User] Please transcribe this text

[Assistant] That every time something new comes out, I get that. Why it labeled me as a late technology adopter? We have all the newest stuff at home.


[User] Please transcribe this text

[Assistant] So, it sounds like if the algorithm inferred something wrong about you, that you cared about, people did not like that.


[User] Please transcribe this text

[Assistant] If the explanation like this one had a link.


[User] Please transcribe this text

[Assistant] To the user's identity, they liked it because they said that the algorithm inferred it.


[User] Please transcribe this text

[Assistant] And actually went an extra mile to show it that to me.


[User] Please transcribe this text

[Assistant] If it did not this or showed something was wrong, people did not, uh, were not happy with the explanation and even a full transparency would not make them happy.


[User] Please transcribe this text

[Assistant] And this concept of enough transparency, something in between is not new. In another research study a few years ago,


[User] Please transcribe this text

[Assistant] It was shown for another type of algorithm that if students are getting exposed to too much or too little transparency about how a grading algorithm


[User] Please transcribe this text

[Assistant] Understand the system without getting in the way. And with that, I hope that I could show you


[User] Please transcribe this text

[Assistant] That if we have a simple design, adding some visibility to the system, but with enough transparency, we can actually help users to become more informed and intelligent when they are interacting with the system.


[User] Please transcribe this text

[Assistant] And with that I want to encourage us, like as researchers, as developers, or future developers of the systems, that when you write a code,


[User] Please transcribe this text

[Assistant] When you design a system, consider this, that the user who are sitting on the other side of the computer and using these algorithms, it's gonna affect their life. Small or large, it's gonna affect their life. And they have the right to know what's going on, but we have to design and build these systems in a way to help them, not interfere with them.


[User] Please transcribe this text

[Assistant] And I'm gonna finish with this example. A few years ago, like not a few, it's kind of 6 years ago.


[User] Please transcribe this text

[Assistant] When I started my PhD, I got this question a lot that we use cars, our cars have engines, and we never need to know what's going on with our cars and engines. And we are fine.


[User] Please transcribe this text

[Assistant] And what's the difference here? Why we need to get some visibility to these opaque algorithmic systems? We are just working with them, it's like our cars, we get the information and just move on with our lives. It took me actually many years to get a good answer to that.


[User] Please transcribe this text

[Assistant] But there are two differences. One is that when your car breaks, you know it's broken.


[User] Please transcribe this text

[Assistant] But when the algorithm, an algorithm in a system breaks, you don't know. It might be biased. It might be biased for many years. It might affect people's life if they know, even if they know or not. But you don't know it's broken.


[User] Please transcribe this text

[Assistant] I, I gave you at the beginning of the talk that users give, users have the confidence to work with the system and change the decision if it's necessary.


[User] Please transcribe this text

[Assistant] And with that, I want to thank you all for listening to me and I'll be happy to take questions.


[User] Please transcribe this text

[Assistant] And that's actually only in the context of advertising. I even don't know what other contexts might look like. I think the best thing is not, I mean, this is kind of a controversial because some people might say maybe we show something that is a lie, but people are happy with that.


[User] Please transcribe this text

[Assistant] So I'm not advocating for that. I'm just saying that if the algorithm makes a mistake.


[User] Please transcribe this text

[Assistant] And it's about the feature you like. They're not gonna like it. So one thing is definitely increasing the algorithms.


[User] Please transcribe this text

[Assistant] Accuracy, but one other thing we found is that showing people that algorithms make mistakes, that if it is hap- because people in the studies we've done, we found that people kind of, uh, trust the algorithms objectively. They think that what they decide is just all true.


[User] Please transcribe this text

[Assistant] And that makes them sad or upset. But when they started seeing more and more mistakes in the system, it didn't make them, uh, that dissatisfied.


[User] Please transcribe this text

[Assistant] Like for example, many of them did not know there is an algorithm, so they never even tried to give feedback to the algorithm. For example, if they did not, the algorithm did not show a friend for a while.


[User] Please transcribe this text

[Assistant] They give feedback and I think that's because also I don't know if you noticed there's a most recent and top stories on the side of your Facebook which is most recently supposed to be non-filtered or at least less filtered top stories.


[User] Please transcribe this text

[Assistant] I think it helps, like, users get some idea of the presence of an algorithm, which is the first step. I think the complexity goes where we want to explain the operation of the algorithm, which is, like, the existence is not really hard, like, you can add things here and there, telling them that there is an algorithm at work. But for the operation,


[User] Please transcribe this text

[Assistant] I think there is a rich line of literature in terms of explainable AI, like how. I think most of these work for expert users.


[User] Please transcribe this text

[Assistant] For delay users, as you said, if it goes a bit higher than basic, they don't understand that. So I think the solution, I think, I don't know if it's a solution or just mitigating the problem, is that


[User] Please transcribe this text

[Assistant] We always try to provide, like there should be some, and that goes, that actually can happen only with regulations, I don't think it's good.


[User] Please transcribe this text

[Assistant] There should be a way for full transparency, at least if there is, and if there is a gaming problem with the system, it could be like maybe visible, fully transparent to those who regulate the system. But for basic users, I think,


[User] Please transcribe this text

[Assistant] This is a fundamental issue. You can't teach users, like, how a neural network works. And it's not because of those early users, even a developer of a system. Like, I write a code and I don't know what's going on in the code.


[User] Please transcribe this text

[Assistant] It just goes to loop functions and return functions and it's just hard to explain. So I think...


[User] Please transcribe this text

[Assistant] There might be some help to make it easier, but I think at the end of the day, users need to build, and whatever users build as folk theories, they're gonna act on that. So I think what we need to make sure is that we give them some basic explanations, and make sure they are not building folk theories that are wrong, or at least if they are wrong,


[User] Please transcribe this text

[Assistant] Like, uh, like, kind of, algorithms are all powerful or all knowing.


[User] Please transcribe this text

[Assistant] They consider like Facebook as a spy or things like that. It's just going to interfere with the interaction with the system.


[User] Please transcribe this text

[Assistant] One answer for all. Like, I think it really depends on the system, on the context. But I think it's...


[User] Please transcribe this text

[Assistant] And see which of these work. And try to, and I didn't get to talk about folk theories much, but extract the folk theories, extract how people think about the system, how much they trust, and then revise the explanations.


[User] Please transcribe this text

[Assistant] But I honestly don't think it's gonna solve the problem. As you said, it's...


[User] Please transcribe this text

[Assistant] It's prob- also it's a problem of justice of the system that needs maybe regulations on the other side. So, but that's the part on the user side, you can never get everyone happy. But you can get maybe some information and more people, yeaah.


[User] Please transcribe this text

[Assistant] Trying to give feedback and change. I think they should work together. So just to transparency without giving users control.


[User] Please transcribe this text

[Assistant] Oh, I think for example, currently, like, in advertising, it says like, we inferred these interests for you, are you happy or not? And you can click and change them. So that goes as a feedback to the system.


[User] Please transcribe this text

[Assistant] And change, I don't know how much they changed the algorithm for that, but giving users a sense of control.


[User] Please transcribe this text

[Assistant] Explanation without contestability or appeal is not enough. People are not happy, particularly the decision you're not- the decision that you're not- you don't agree with.


[User] Please transcribe this text

[Assistant] But they found that if you give contestability, but they don't think that the contestability actually is real or helpful, it doesn't help. For example, if they, yeah, that would be worse because they think like,


[User] Please transcribe this text

[Assistant] It says, it's gonna do this whatever it wants. Nobody hears me, it's just...


[User] Please transcribe this text

[Assistant] If they don't get the real sense of control, no, it doesn't help. But yeah, definitely if I, if someone told me that why I'm on the list,


[User] Please transcribe this text

[Assistant] These are two ha- these are hand to hand. So you can't have explanation.


[User] Please transcribe this text

[Assistant] Without appeals or like contestability. It's gonna give, it might even make people less satisfied with the system.


[User] Please transcribe this text

[Assistant] Pearl sat close to her mother and her sister on the beach at the top of Fort Hill in Chenkiang.


[User] Please transcribe this text

[Assistant] They often watched the country market bustling with people below them. A little boy playing with a dragon kite caused Pearl to think about her little brother, Clyde. She remembered so little of him. When he died, she'd been six years old. Both of them had gotten sick with diphtheria, their throats sore, their faces flushed. Caroline realized they were very ill.


[User] Please transcribe this text

[Assistant] This doctor had a small supply of medicine that could cure diphtheria. Pearl was saved, but it was too late for four year old Clyde.


[User] Please transcribe this text

[Assistant] The disease must be treated fast or it can damage the heart and stop the lungs from working.


[User] Please transcribe this text

[Assistant] Pearl had prayed every day for her mother to have a baby boy with blue eyes and brown hair just like her brother.


[User] Please transcribe this text

[Assistant] Instead, her mother had given birth to a baby sister.


[User] Please transcribe this text

[Assistant] Pearl couldn't be disappointed when she saw the tiny infant. From that time on, she loved Grace very much. She took her little sister's hand. Soon a train would arrive at Fort Hill chugging through the tunnel.


[User] Please transcribe this text

[Assistant] Many peasants had toiled beneath the hill using hammers and shovels.


[User] Please transcribe this text

[Assistant] The Shanghai Nanking Railway was Chinese, but had been backed by British money.


[User] Please transcribe this text

[Assistant] The railway meant that travelers could get to places in a few hours rather than taking many days on river steamers.


[User] Please transcribe this text

[Assistant] The small country marketplace that so delighted Pearl was going to open up to many more people from all over the world. This did not make her happy, but change must be faced and she was facing personal changes too.


[User] Please transcribe this text

[Assistant] She was now 14 years old and was being sent to an Anglo-American school in Kuling.


[User] Please transcribe this text

[Assistant] Pearl loved cooling where she'd spent so many happy summers with her mother and sister, but going to a new school was different. She didn't expect her summer vacation to change much.


[User] Please transcribe this text

[Assistant] Yet she felt uneasy at the idea of being left at the school. She made the most of the hot days in Kuling though. She even learned to swim in Duck Pond.


[User] Please transcribe this text

[Assistant] A pool that was the result of a little stream being dammed.


[User] Please transcribe this text

[Assistant] She loved to sit near the water, dangling her long legs from the rocks. She enjoyed hikes and picnics with friends.


[User] Please transcribe this text

[Assistant] She'd gotten as tall as her mother, and so thin that she looked like a bamboo pole. All too soon summer ended.


[User] Please transcribe this text

[Assistant] Pearl hated to see her mother, her sister, and Wong leave for home.


[User] Please transcribe this text

[Assistant] It was good that her friends My Fun and Isabelle were at the school with her to keep her company. They had fun together.


[User] Please transcribe this text

[Assistant] In autumn, they collected chestnuts and roasted them over a fire.


[User] Please transcribe this text

[Assistant] As the months went by, the girls bought sweets and fruits and other odds and ends at the foreign shop in Kuling.


[User] Please transcribe this text

[Assistant] They felt like grown-ups able to run up a bill.


[User] Please transcribe this text

[Assistant] When Mr. Duff, the shopkeeper, asked them to settle their account, they discovered that they owed $12. Pearl didn't have enough money to pay her share.


[User] Please transcribe this text

[Assistant] Her friends got mad at her, but they took care of the bill anyway. During this time Pearl wrote some poetry.


[User] Please transcribe this text

[Assistant] She thought she would surely become a famous poet and sent her poems to her mother. Caroline didn't encourage her. Two poems, she said. Don't create a poet. Pearl was hurt. She felt bored in school, too. Nothing challenged her mind.


[User] Please transcribe this text

[Assistant] She continued to do her best, but preferred jokes to study.


[User] Please transcribe this text

[Assistant] At Christmas, when Pearl went home to Chen Qiang,


[User] Please transcribe this text

[Assistant] Her mother decided that since she didn't seem to be learning much,


[User] Please transcribe this text

[Assistant] There was no point in sending her back to the Kuhling School.


[User] Please transcribe this text

[Assistant] Pearl made friends with a new girl called Mary Longdon. They were together constantly.


[User] Please transcribe this text

[Assistant] Pearl stayed at Mary's house overnight quite often.


[User] Please transcribe this text

[Assistant] It was Mary who showed her a new hairstyle called, uh, forever parting pompadour.


[User] Please transcribe this text

[Assistant] It was styled by combing the sides of the hair back and then fanning the hair forward and curling it over itself.


[User] Please transcribe this text

[Assistant] Pompadour's later became a man's hairstyle in the 1950s in the United States.


[User] Please transcribe this text

[Assistant] The men's pompadours were formed by combing the front of the hair up high.


[User] Please transcribe this text

[Assistant] And curving it in a wave toward the back of the head.


[User] Please transcribe this text

[Assistant] This hairstyle was favored by singer Elvis Presley.


[User] Please transcribe this text

[Assistant] Whose music Pearl certainly heard after she came back to the states to live.


[User] Please transcribe this text

[Assistant] Pearl's mother disliked the pompadour on her daughter and her friends. She probably would have disliked it even more on men. Pearl kept it until it was no longer chic. She began to love pretty clothes, too.


[User] Please transcribe this text

[Assistant] Before long, Mary seemed more interested in Alcy. Pearl said very little.


[User] Please transcribe this text

[Assistant] And at the Methodist School, she had other friends, including her Chinese friends, Dottie Wei and Sui Wang. That year, 1907, there was another terrible famine.


[User] Please transcribe this text

[Assistant] Pearl watched groups of people in an endless procession carrying their belongings in bundles.


[User] Please transcribe this text

[Assistant] They were going south in the hope of finding food. Many people died.


[User] Please transcribe this text

[Assistant] That winter, it became dangerous for Pearl to go out on the streets to the Methodist School.


[User] Please transcribe this text

[Assistant] She had to study at home under her father's stern watch.


[User] Please transcribe this text

[Assistant] He taught her the two subjects, mathematics and Latin, which she most disliked. When he went away to help with famine relief, Pearl's lessons did not end. She continued to read. And a Texan man, boarding with her family, helped her with Latin. This man, Mr. Hancock, didn't understand the Chinese language very well, but Pearl helped him by becoming his translator.


[User] Please transcribe this text

[Assistant] Mr. Hancock also didn't understand Chinese culture or Chinese tradition. On one occasion, a snake got into the house.


[User] Please transcribe this text

[Assistant] Mr. Hancock waved his cane in the air and chased the poor creature outside, planning to kill it.


[User] Please transcribe this text

[Assistant] Pearl stopped him by telling him that it might be someone's ancestor.


[User] Please transcribe this text

[Assistant] In Hillsboro, Pearl's ancestral home, there are many snakes, including poisonous copperheads.


[User] Please transcribe this text

[Assistant] Poisonous snakes would certainly be killed if they came around the house.


[User] Please transcribe this text

[Assistant] There are also six feet long black snakes, which are harmless and eat mice and rats.


[User] Please transcribe this text

[Assistant] Pearl, back at the Methodist School, joined her friends Sue E. and Dottie. They often played the piano together.


[User] Please transcribe this text

[Assistant] Many of the girls getting ready to graduate were worried that they would be married off in the old Chinese way. One girl, Li An, cried because she wanted to be a teacher, but was not being given the chance.


[User] Please transcribe this text

[Assistant] Pearl told them that she could do whatever she wanted back in America.


[User] Please transcribe this text

[Assistant] She wouldn't have to get married unless she wanted to.


[User] Please transcribe this text

[Assistant] Pearl urged her friends to do whatever they wanted with their lives.


[User] Please transcribe this text

[Assistant] One girl who was the best student in the class said that she wanted to be a doctor. Another said she might work with little children, but she didn't want to have babies.


[User] Please transcribe this text

[Assistant] Since it would be another year before Pearl would turn 18 and go to college in America.


[User] Please transcribe this text

[Assistant] Caroline thought it wise to separate her from her Chinese friends.


[User] Please transcribe this text

[Assistant] She wanted to encourage Pearl to act more like an American.


[User] Please transcribe this text

[Assistant] She also wanted Pearl to be ready for her college entrance exams.


[User] Please transcribe this text

[Assistant] Thus Pearl was enrolled at an American boarding school in Shanghai. Her feelings about the city were mixed.


[User] Please transcribe this text

[Assistant] And that the food was, according to her ama, no good.


[User] Please transcribe this text

[Assistant] Still, there were parks along the river where she might get to have picnics and watch the foreign man of warships, ferries, and other boats.


[User] Please transcribe this text

[Assistant] Pearl had mixed feelings, too, about going to a boarding school that was only for Westerners.


[User] Please transcribe this text

[Assistant] Was exactly the type of woman Pearl's mother disliked and didn't want her daughter to become. The woman principal was handsome rather than pretty and she was controlling rather than yielding. She grasped Pearl's hand and gave her a firm handshake.


[User] Please transcribe this text

[Assistant] Pearl, who had enjoyed so much freedom with her Chinese friends roaming the hills and mountains, now found herself in a strict school. Rules were the order of the day. Even the school building seemed gloomy.


[User] Please transcribe this text

[Assistant] It was dark in the parlor and the windows were heavily draped so that very little light came in to brighten the big heavy furniture.


[User] Please transcribe this text

[Assistant] A few plants near the windows had grown tall trying to reach the light.


[User] Please transcribe this text

[Assistant] Up a plain staircase, Pearl shared a small dormitory room with two other girls, Ruth and Florence. They were the sisters of her old friend Mary Longdon. Their room was simple and bare.


[User] Please transcribe this text

[Assistant] At first Pearl missed the homey atmosphere her mother always managed to create, but she settled down quietly. She no longer played practical jokes on people.


[User] Please transcribe this text

[Assistant] She knew her stay here was temporary and was determined to do her best.


[User] Please transcribe this text

[Assistant] She wanted to be more than ready for American college.


[User] Please transcribe this text

[Assistant] Her memories of the states filled her with longing that was in part her mother's.


[User] Please transcribe this text

[Assistant] She'd heard so many stories about how great her country was, and she still recalled seeing the big houses with wide stretches of lawn that even ordinary people could afford.


[User] Please transcribe this text

[Assistant] In the boarding school, Pearl's only educational struggle was with mathematics.


[User] Please transcribe this text

[Assistant] She didn't try very hard because she didn't much care for the subject.


[User] Please transcribe this text

[Assistant] Her teachers realized that she was exceptional in her ability with literature. They did their best to encourage her. Every Friday at the school's literary club, Pearl read her own stories. Everyone loved to listen.


[User] Please transcribe this text

[Assistant] The teachers soon realized that she didn't need any coaching for her college entrance exams.


[User] Please transcribe this text

[Assistant] They gave her special coursework that they hoped would make her think.


[User] Please transcribe this text

[Assistant] Pearl used much of her time to study her fellow students, trying to understand how their personalities had developed.


[User] Please transcribe this text

[Assistant] She began to see how a person's background affected who they became. Mei Fan, who'd been an only child raised by her widowed mother, had become quiet and religious.


[User] Please transcribe this text

[Assistant] Another girl who looked Chinese but had an American father acted very much like a westerner.


[User] Please transcribe this text

[Assistant] However, this girl despised bread puddings and the overcooked vegetables that were often on the menu. She longed for her mother's tasty Chinese food.


[User] Please transcribe this text

[Assistant] She invited Pearl to her house so that Pearl could get some decent meals.


[User] Please transcribe this text

[Assistant] Pearl went, but she already knew about good Chinese food. She'd been eating it all of her life. During this period.


[User] Please transcribe this text

[Assistant] Pearl was in the habit of rereading Lewis Carroll's Alice in Wonderland.


[User] Please transcribe this text

[Assistant] Considered literary nonsense is actually full of meaning and logic that appears silly, but contains deep ideas.


[User] Please transcribe this text

[Assistant] His real name was Charles Ludwig Dodson.


[User] Please transcribe this text

[Assistant] To entertain the little girls, whom he'd taken boating on the River Thames in England, he told them a story.


[User] Please transcribe this text

[Assistant] One of the girls, Alice, asked Dodson to write it down for her.


[User] Please transcribe this text

[Assistant] He expanded the story that he'd originally recited and the book became famous. It is still required reading in many schools.


[User] Please transcribe this text

[Assistant] Pearl kept to herself much of the time, reading and writing. It wasn't that she was unfriendly.


[User] Please transcribe this text

[Assistant] She simply preferred solitude and didn't like the religious rites of this school.


[User] Please transcribe this text

[Assistant] Their long prayers and calls to be saved weren't part of her Presbyterian upbringing. She was intellectual.


[User] Please transcribe this text

[Assistant] Pearl liked Chinese sacred ways that she'd learned from her tutor and her friends. Even when she was a little girl.


[User] Please transcribe this text

[Assistant] She discovered that the Chinese people seemed to view death differently than did Christians.


[User] Please transcribe this text

[Assistant] She'd often wondered in the sanitary and gotten to visit people's houses where a coffin might be waiting for the matriarch or patriarch. The elderly did not view death as fearful, but seemed to accept life's rhythms.


[User] Please transcribe this text

[Assistant] Pearl saw Chinese funeral rituals as celebrating life as well as mourning loss.


[User] Please transcribe this text

[Assistant] The Shanghai Boarding School surely did not give Pearl a feeling of celebration.


[User] Please transcribe this text

[Assistant] Shanghai itself was a lively city with as many people as any urban center in the world. It was divided into three sectors.


[User] Please transcribe this text

[Assistant] The European part was full of cafes and even had Parisian hat shops.


[User] Please transcribe this text

[Assistant] Another part, that was called the international settlement, was dominated by British control.


[User] Please transcribe this text

[Assistant] The Chinese area of the city was walled in and rarely visited by white people.


[User] Please transcribe this text

[Assistant] The only chance Pearl had to explore the town was when she and the other girls went out to walk or when they rode on trolleys to parks.


[User] Please transcribe this text

[Assistant] What Pearl liked most was to see the narrow streets filled with banners and colorful lanterns.


[User] Please transcribe this text

[Assistant] She loved to watch vendors standing behind fish barrows, fruit stands, and cart loads of bright goods. They sold their wares to all kinds of people.


[User] Please transcribe this text

[Assistant] Proper Englishmen wearing suit jackets and stovepipe trousers, and Chinese gentlemen wearing dignified robes that flowed to the ground.


[User] Please transcribe this text

[Assistant] ...mingled with the elegant ladies who stepped from their rickshaws amongst the local people, many of whom were very poor.


[User] Please transcribe this text

[Assistant] Pearl enjoyed all the people, seeing the mixture of wealth and poverty, education and ignorance as simply a part of life.


[User] Please transcribe this text

[Assistant] I'll have that slug out of you in a minute. Here, bite on this.


[User] Please transcribe this text

[Assistant] How did she look? Just like you said, Joe. That's how she looked.


[User] Please transcribe this text

[Assistant] And that was the end. Uh, what, what happened to the Greek? Just what he said would happen. He delivered the stuff, then he joined the Merchant Marine. Coming back, the Briney Marlin drew a dead heat with a German torpedo.


[User] Please transcribe this text

[Assistant] Come on, Joe, the dinghy's at the end of the pier. Let's get back to ship.


[User] Please transcribe this text

[Assistant] Early on, uh, Salton of Sanctu was the best falco on the west coast, and he'd go online and talk mad trash. Mad, mad trash.


[User] Please transcribe this text

[Assistant] I actually went through pages and pages of Smash boards to the point where I'm like, dude, let's money match. Once I get to TG4, let's money match, 50 bucks. Everybody cleared the stages and Matt Dizzy had the big screen TV and I played Salt and the Sanity.


[User] Please transcribe this text

[Assistant] I actually found out before that that uhm, you can toss Falco and Fox up in the air and then re-grab him and toss him like a hacky sack and just keep on doing it over and over.


[User] Please transcribe this text

[Assistant] After TG4, MetDG released a DVD about it, and most of the East Coast players actually bought a copy or two, and they were just, they were laughing at us. They're like, oh look at him, he's an idiot, all he does is move.


[User] Please transcribe this text

[Assistant] You know, why would he dash back and forth when he can wave dash? You know, I was still new in the community. I didn't wave dash at the time. All I knew how to do was dash back and forth and that eventually became uh, dash dancing.


[User] Please transcribe this text

[Assistant] People started catching on to that and calling it out every single time I would do that.


[User] Please transcribe this text

[Assistant] I don't think the newlyweds or plant- oh, I don't think they were that great.


[User] Please transcribe this text

[Assistant] Honestly, they're all talking mostly- mostly-


[User] Please transcribe this text

[Assistant] The west coast players are more calm. When she gets to the east coast players, they're loud, they'll scream.


[User] Please transcribe this text

[Assistant] Captain Jack said that he was hosting a tournament in west coast Japan. He wanted like the top players from America to come over. Isaiah and I were just the only two like top players that decided to go.


[User] Please transcribe this text

[Assistant] For me it was kinda more like I wanted to travel to Japan and enjoy the culture over there and also get better at Smash and learn their style.


[User] Please transcribe this text

[Assistant] Captain Jack's mom was very accommodating, she took care of us, you know, fed us, things like that.


[User] Please transcribe this text

[Assistant] We're in Japan, we should eat Japanese food. Everyday, like, a smasher, like, come over, like, Captain Jack is like, oh, Masashi's coming over today.


[User] Please transcribe this text

[Assistant] When they play Smash, they play Smash. It's like they play all day, all night, all day, all night.


[User] Please transcribe this text

[Assistant] The weakest Japanese player was definitely like, our strongest player in America.


[User] Please transcribe this text

[Assistant] I think there was like around 65 to 80 Japanese players there. And I just wanted to learn. And it was to learn and take back, uh, what, what I saw at the Jack Orange tournament.


[User] Please transcribe this text

[Assistant] Bonkler just came out of nowhere and just started beating all these top Japanese players.


[User] Please transcribe this text

[Assistant] I ended up fighting Bomb Soldier in the finals and that was, that was probably one of the toughest Falcos I've ever fought in my life.


[User] Please transcribe this text

[Assistant] After a bot was able to bring back some of the videos from Japan, I think that inspired more Falco players to take Falco to the next level.


[User] Please transcribe this text

[Assistant] If we knew who he was, he might be willing to accept his responsibility.


[User] Please transcribe this text

[Assistant] But you haven't the means to take care of the child. We'll get along.


[User] Please transcribe this text

[Assistant] It's your duty to tell us when a man does this sort of thing.


[User] Please transcribe this text

[Assistant] Keep your guard up. He's going to use ambush.


[User] Please transcribe this text

[Assistant] He's only tough with weaker opponents. He won't bother you again.


[User] Please transcribe this text

[Assistant] In a comfortable clean little bed, in a pleasant airy room at the top of Miss Betsy Trotwood's house, I floated away into the world of my dreams.


[User] Please transcribe this text

[Assistant] At breakfast next morning, I bravely asked my aunt, are you, have you, what's going to happen to me? I've written to your stepfather, she replied. Oh, are you going to send me back to the Merdstones? Please don't, aunt. Please let me stay here, I cried, trembling. I don't know what I'll do yet. We'll have to wait and see, she answered firmly.


[User] Please transcribe this text

[Assistant] This news made me very depressed. But there was nothing I could do about it. And my aunt soon began to talk about something else.


[User] Please transcribe this text

[Assistant] Yo, what's going on swolebodies? Welcome back to another BORTO reaction today with Lil' Shane!


[User] Please transcribe this text

[Assistant] Today it is another great day to be a Swolebody because we are reacting to some more Boruto episode 38. Hopefully guys you keep along with the series but if you are not or you are new here


[User] Please transcribe this text

[Assistant] Last time we finished up the graduation exams and they were officially all Geni. They all officially got their headbands and we saw the class try to attack Kakashi. We saw, I think her name was Namira's annoying ass Jutsu with the screaming.


[User] Please transcribe this text

[Assistant] Sarda and Boruto had a massive brain play against Kotohamaru. It was a really interesting episode, and I'm interested to see what direction the anime is going to take now because


[User] Please transcribe this text

[Assistant] You've only ever seen them in the academy or going on a field trip as the academy. So I'm interested to see what teams are going to be made and how they're going to stay together and be friends and what they're going to do in their own teams and who's going to be their sensei and stuff. So we are to get to this. If you are new and you subscribe to get plus 10 games, Jessica will check the proof is there.


[User] Please transcribe this text

[Assistant] Patreon in the description down below if you guys do want to support the channel, become a Dio proti powder, get early reactions, also check out lil chink down there too, and my social medias of course. AND LET'S GET IT!


[User] Please transcribe this text

[Assistant] Oh, there it is! The Uzumaki baby! So I guess this is the new Boruto that we saw in the intro. He has his resolve now, and he's a different Boruto. How is that uncool? Get out of here, Brad.


[User] Please transcribe this text

[Assistant] Damn, so the boy Shikimaru shifted his Naruto schedule around just so Naruto could see Boruto. Damn. I wonder if Boruto even recognized that. Probably not. Yeaah. Aww, pretty good start to the episode so far. Back to the academy.


[User] Please transcribe this text

[Assistant] Or if they even had to in the beginning, that's just how it was in Naruto. Uh, but I see Boruto, Saruda, and Mitsuki, Inoshika-cho,


[User] Please transcribe this text

[Assistant] Denki and Iwabe for sure, and maybe Rock Lee. I don't really know. I see Wasabi and Nambi together too. That's just like what it feels like.


[User] Please transcribe this text

[Assistant] Boy scout lookin' ass. Hey, you know what, she doesn't have the overalls anymore.


[User] Please transcribe this text

[Assistant] That is literally Kakashi. I bet he's a Kakashi fanboy.


[User] Please transcribe this text

[Assistant] What is the use of that? Is it supposed to like help protect your head against the kunai and stuff?


[User] Please transcribe this text

[Assistant] TEAM FIVE LET'S GO! Okay. That is so sick. Cause I was kinda worried that they'd throw metal in with some side characters or something. If they had either two boys and one girl.


[User] Please transcribe this text

[Assistant] But they've got the boy squad. They don't even have any chicks, it's just the boys. I wonder if there's even gonna be a chick squad.


[User] Please transcribe this text

[Assistant] The girl squad. Wasabi. Oh yeaah, and Namida. We got the girl squad. Wait, Hokki.


[User] Please transcribe this text

[Assistant] I actually really like that team. I'm really gonna like-


[User] Please transcribe this text

[Assistant] Yeah, and the second generation of Sonic was Team 7.


[User] Please transcribe this text

[Assistant] Gets even stronger than the second, cause the second generation is stronger than the first generation. Boruto, Naruto is stronger than Jiraiya. And Mitsuki!


[User] Please transcribe this text

[Assistant] He is 12 years old and he already beat a Joni. Obviously, we know Shino wasn't going all out, but that's still very impressive, so he may be even stronger than the Orochimaru. I'm ready to see what that power was, cause that power, dude.


[User] Please transcribe this text

[Assistant] Kakashi and Obito and Marine. Was Jiraiya's team? Team 7?


[User] Please transcribe this text

[Assistant] The sonding? Well, I looked it up and I can't find team nate- team numbers, so. But team 7 would've been dope, bro. I wish.


[User] Please transcribe this text

[Assistant] Oh. Your compatibility is great. She's trippin bro. She just, she just don't like Boruto, but she is definitely compatible with the boy because...


[User] Please transcribe this text

[Assistant] Kagemasa, that was great. That was just like Naruto in Sasuke.


[User] Please transcribe this text

[Assistant] They had a great team play, and then last episode, what we saw with the shotting on and the transformations, that was so sick! They are good! So is Beatsky.


[User] Please transcribe this text

[Assistant] Submit this to Naruto. So Naruto did those. Or Sakiruzen did. Naruto is not gonna accept that.


[User] Please transcribe this text

[Assistant] And he said it's happening again this year. Does that mean Naruto creates a lot of teams where there's conflict? Yea, I guess. Maybe that's why Naruto purposely does that, cause he knows how good it was for him and Sasuke to be that way. Yea, I get it. Cause they didn't like each other, but we saw at the end of Shippuden like what they really thought when they were kids.


[User] Please transcribe this text

[Assistant] Let me see. Oh yeaah, they both have dojutsu too! We haven't seen more of those dojutsu in a while.


[User] Please transcribe this text

[Assistant] Yeaah. He's so funny, bro. I love- I love that Beatsky's on their team, cause he's just such a creepo. He's just always lurking. Yeaah.


[User] Please transcribe this text

[Assistant] I definitely see Sarada and Boruto on the same team, and I hope they- I know they're gonna work really well together, especially with Mitsuki. They- they already have against Shizuma too. They work really well. Even if they don't want to accept it.


[User] Please transcribe this text

[Assistant] What the hell is going on here? What the hell is going on here? WHAT?! What is he doing? What are they doing? Are they trying to overthrow him?


[User] Please transcribe this text

[Assistant] And he said when there's no orange on the eyes from sage, like the frog sage. But he's, but he has the...


[User] Please transcribe this text

[Assistant] The sage eyes? He's in sage or six paths mode.


[User] Please transcribe this text

[Assistant] Is this like a training thing, or is this to make sure it keeps its touch? Or is this a- trying to be a coup or something?


[User] Please transcribe this text

[Assistant] If those are the Joani- If those are like the strongest Joani in the village...


[User] Please transcribe this text

[Assistant] Mmhm. Naruto is so strong, that that's a pretty high bar to set, but it has to be set that high or about higher because of the Otsutuki.


[User] Please transcribe this text

[Assistant] There should be a reason why Boruto or why Naruto got those people in particular, unless...


[User] Please transcribe this text

[Assistant] He's just going through all the joining or something throughout the village. Hopefully, hopefully those weren't the strongest.


[User] Please transcribe this text

[Assistant] Oh, there's that dude! Now he was at the end of the- he was in Chikamaru's story.


[User] Please transcribe this text

[Assistant] Who the hell's that? OH!


[User] Please transcribe this text

[Assistant] Yeah, she's always had brown hair. Okay, uh.


[User] Please transcribe this text

[Assistant] I kinda wish she had current eyes hair a little bit. Uhm, first impressions. It's dope she's a shinobi. How old is she now? Like, I guess she would be like late teens. It's been like 15 years since the war.


[User] Please transcribe this text

[Assistant] I don't know about the hair choice, brother. Boruto's got longer hair than she does. Like, I'm not against short hair, shorter hair.


[User] Please transcribe this text

[Assistant] For the baddies, uhm, but usually, like, my, my personal, don't get offended, ugh, I can't believe this. My personal preference, okay, is like, the shortest is like ear level.


[User] Please transcribe this text

[Assistant] Anything like higher than that? Not my particular cup of tea. Does that mean it's bad? Absolutely not. But you ain't gonna hear me calling her bad, brother. I'm sorry. I'm sorry. It's just my preference.


[User] Please transcribe this text

[Assistant] It's just what I like, bro. If you like short hair, there ain't more power to you, dude. I don't care. Me personally.


[User] Please transcribe this text

[Assistant] Yes. Is she just had slightly longer hair? She kinda looks like she's sweet.


[User] Please transcribe this text

[Assistant] Okay, I like the- I like her eyes, her and my eyes. What the hell?


[User] Please transcribe this text

[Assistant] Oh, she do got the chakra blades though, that's kinda dope. Okay, Mirai, I like her voice and I like her character design. She's like a badass. But she ain't bad. I'm being honest.


[User] Please transcribe this text

[Assistant] But I was like, am I discussing how hot she is when she's like 15?


[User] Please transcribe this text

[Assistant] But what the hell was this? This was so strange. Somebody's into Genjutsu. I don't know if it's Kuronei or Sarada. Or if I say Kuronei, I meant Mirai. This is kinda like when Kuronei tried to use Genjutsu on Itachi. Let's see who's gonna win this time though. I don't know who got the Genjutsu off. What is that? It looks like Woodstyle. Uuuh.


[User] Please transcribe this text

[Assistant] Okay, let's go! So sorry to then get her getting Jutsu off. Damn! They're getting murked.


[User] Please transcribe this text

[Assistant] Mirai is strong though. And she is way older. Yeaah, yeaah. Uhm, so Mirai is a badass. She ain't bad. Unfortunately.


[User] Please transcribe this text

[Assistant] This is like the first girl in Naruto, or in the Naruto universe, so I said it isn't bad at 18 years old. Can't believe Bitsy's holding up that entire rage. OH!


[User] Please transcribe this text

[Assistant] Oh? The grandchild of the third Okage, yeah! She would be, just like, just like Konohama is.


[User] Please transcribe this text

[Assistant] Ayyyy. I was hoping they would've solved Naruto's- Okay!


[User] Please transcribe this text

[Assistant] The new T7, hell yes! I'm so happy it is actually...


[User] Please transcribe this text

[Assistant] They're actually team 7 now, and not team 3. That was kinda lame. I don't know why Naruto made them team 3. Maybe he was trying to actually push them to make them team 7 or something? I don't know, but seeing me die for the first time was definitely a shock. Uh,


[User] Please transcribe this text

[Assistant] I'm sure a lot of people will get mad, but short hair is okay, like I like soccer's short hair more than her long hair w- more than her long hair. That's just a bit too short for me, like I said.


[User] Please transcribe this text

[Assistant] Asada, Boruto, Omitsuki, I kind of expected that a little bit, I was hoping they'd be the team. And they are the new Team 7, the new Sonin, I guess you could say too. But yea, this is super dope, I really wanna know what's gonna happen from now on, like are they gonna go on missions, or what the hell is gonna happen with the teams, cause it's not like anything's, there's any war or anything too dangerous going on. Unless they're just some rogue ninja or something.


[User] Please transcribe this text

[Assistant] But I'm happy we got a boy squad too and a girl squad, that's dope. I didn't think that Metal Leaf fit anywhere besides with Iwabe and Denki. I don't know if Iwabe and Denki fit together.


[User] Please transcribe this text

[Assistant] Hopefully, Metal Week can break his nervousness too. I know dude. This is a really interesting episode. We're getting somewhere with teams in Genny. Hopefully they start establishing some Jutsus and maybe some Summons later. And whatever else the hell they do nowadays is to become a Ninja. Chooning and then Joaning.


[User] Please transcribe this text

[Assistant] But anyway boys, thank you all for watching this Bortel reaction. Make sure to check out Lil Cheek in the description down below.


[User] Please transcribe this text

[Assistant] Hi Ren, is everything ready? Hi Ron, yes, everything's ready. Good, let's go.


[User] Please transcribe this text

[Assistant] So the rats have had their drugs for about 20 minutes now. As you can see, the marijuana rat is mellowed out. The cocaine rat's gotten quite mad and is trying to escape out this cage here.


[User] Please transcribe this text

[Assistant] So overall we're seeing a modulation of the stopwatch where we can speed up the stopwatch and slow down the stopwatch or we can maintain normal speed under control conditions.


[User] Please transcribe this text

[Assistant] And the idea is that numbers flash very rapidly on this LED screen. They flash so rapidly that a normal brain under normal circumstances can't see what's being flashed here. But if time were running in slow motion, then you should be able to distinguish the numbers.


[User] Please transcribe this text

[Assistant] That is very difficult to read what the numbers are. Yes, very difficult. Okay, so are you able to read what's on the screen now?


[User] Please transcribe this text

[Assistant] Okay, the number that was actually presented was a 5 and a 0, 50. The 0 happens to look a lot like a 6. So what this means is that...


[User] Please transcribe this text

[Assistant] At least mostly, he was able to see a presentation rate that he was not able to see under normal circumstances. Maybe what we can do is try again and see if, see if you get both the numbers right this time.


[User] Please transcribe this text

[Assistant] I would have loved it if he saw both numbers exactly correctly, but this at least suggests to me that he's able to take in information faster than he was before.


[User] Please transcribe this text

[Assistant] And why should I forget you? And this grown up young man, in his soldier suit.


[User] Please transcribe this text

[Assistant] I'll hold you to that now. As for the room, I trusted you and Millie a long time ago, Jeff. And I'll trust you and the boy now.


[User] Please transcribe this text

[Assistant] Well, it's time you were getting home. Don't tell me you didn't see Pacelli today either. After two weeks? Why, what's the matter with that man?


[User] Please transcribe this text

[Assistant] Put this in your pocket. The time and place for pride. But this ain't it. Here.


[User] Please transcribe this text

[Assistant] She would, and she did. Poor little fella. All he gets for his birthday is a broken heart.


[User] Please transcribe this text

[Assistant] Oh, little fella, he sat out there on the steps for hours, all dressed up, and his shoes polished so you could see your face in them. And then this note came.


[User] Please transcribe this text

[Assistant] Jeff, don't you ever let that woman get near the child again after what she did to him today. It's something he'll never forget as long as he lives.


[User] Please transcribe this text

[Assistant] Yeah, we're here at the Etihad for the second episode of Offscript, a new podcast thing which allows us to get


[User] Please transcribe this text

[Assistant] Uh, just depends, I've had a very busy week. Is this your podcast then? No, so it was debuted last week by Shrees and Kara, and we need to top that. That won't be difficult. That won't be difficult at all.


[User] Please transcribe this text

[Assistant] Different bits. Anecdotal. So what I thought we might talk about this week with the Tunnel Club behind us, which if people don't know, people get warmed up seats, they get to through the glass, watch the interview happening.


[User] Please transcribe this text

[Assistant] Uh, see players warming up, stuff like that. So I wondered, with you, other bits of football clubs, behind the scenes, things we don't see. Always remember, Swrykes Ferguson's last game, I actually went into his office, and just sat there, terrified. But, see, that would have been a bit, and you used to sit next to Solskjaer in the changing room, right?


[User] Please transcribe this text

[Assistant] It feels like a big deal, even now you'll get kids come round and they'll hold them out in a match award and sit there with a microphone. It's massive and you forget if you're there every week how big it can be I think.


[User] Please transcribe this text

[Assistant] You see, even in pre-match interviews, Gary, you know, the team will be, so say today, uhm,


[User] Please transcribe this text

[Assistant] So you talked there about a fan's matchday experience and how that's changed. What about, I want to know your matchday, actually from the night before. How, how did you sleep? Now? Or back then? I'm not interested now.


[User] Please transcribe this text

[Assistant] As a player, what would you like? Big game. Say like today, the players there. How would you have slept? What would you have eaten the next day? What would you like?


[User] Please transcribe this text

[Assistant] Okay, right, we need to finish, cause soccer saturday, dude. There's more important people than us. Yes. And off the scripts, well, out of time.


[User] Please transcribe this text

[Assistant] That's what I thought. And if you go, I'll go. And if we go?


[User] Please transcribe this text

[Assistant] So this work is part of a larger thread of work centered around algorithmic governance, that is, how do we audit and measure problematic content like missed disinformation, uh, online extremism, and how do we do that on online platforms, and then specifically the search and recommendation algorithms driving these platforms.


[User] Please transcribe this text

[Assistant] And more recently this discussion has shifted to the harms posed by generative AI and large language models.


[User] Please transcribe this text

[Assistant] So he goes on to say that IBM was self griped by a special immoral corporate mantra. If it can be done, it should be done.


[User] Please transcribe this text

[Assistant] So we should take a pause and, and kind of re-read this phrase, if it can be done, it should be done, which is parallels, uh, very, very resemblance to the motives and culture practiced in modern day technology companies. And I'm sure,


[User] Please transcribe this text

[Assistant] And I'll come back to these ideas later on towards the end of the talk, but let me first dive into the meat of this presentation, uhm, a couple of studies which we did, uh, in, in this realm of algorithmic governance. So here I want to focus on two types of algorithms, uh, search and recommendations, and two specific platforms, YouTube and Amazon.


[User] Please transcribe this text

[Assistant] Now why audit for misinformation in particular, right? So currently there is this disproportionate focus on AI bias and fairness and tech journalist Karen Howe captures this notion very well in her article where she mentions how often responsible AI teams in companies are pigeonholed into targeting AI bias.


[User] Please transcribe this text

[Assistant] Now don't get me wrong, so bias and fairness are indeed important topics that one should pursue, but tackling just AI bias, uh, draws away attention from fixing much bigger problems, uh, of other types of harmful, uh, information such as misinformation, extremism, conspiratorial content. So this is what is underlying motivation behind the set of studies that we did.


[User] Please transcribe this text

[Assistant] And with that, let's dive into this first study, the YouTube audit work.


[User] Please transcribe this text

[Assistant] What really is an audit, right? I have talked about mentioned audit a couple of times so far, but how do you audit algorithms?


[User] Please transcribe this text

[Assistant] I'm sure some of you in the audience might be familiar with the concept of audit. For those of you who don't know, I'm going to give you a quick definition through an example and also say that this is a thriving field of research. Lots of work has been done in this space.


[User] Please transcribe this text

[Assistant] Where researchers conducted this very clever field experiment to investigate employment discrimination. That is, they audited the labor market for racial discrimination. So what they did was they responded with fictitious resumes to help wanted ads in the Boston and Chicago newspapers.


[User] Please transcribe this text

[Assistant] To manipulate the perception of race, uh, what they did was, they kept everything else in each of these resumes constant, only they changed the names. And the names were either, uh, very African American sounding names, such as Lakisha or Jamal, or a very white sounding name, such as Emily or Greg, and hence the name of this paper.


[User] Please transcribe this text

[Assistant] The results showed that there was significant discrimination against African American names, while white names, uh, received 50% more callbacks for interviews, uh, compared to, uh, the African American names despite everything else being constant in those resumes.


[User] Please transcribe this text

[Assistant] You would manipulate a variable to determine whether the search and recommendation algorithm returns different results. Say when someone's age, their other demographic attributes, gender, their watch history, where they are searching from the geolocation. If that differs, what happens with the research results?


[User] Please transcribe this text

[Assistant] Running queries on the search platform while at the same time a script at the backend were collecting whatever search and recommendation results the platform was, uh, returning.


[User] Please transcribe this text

[Assistant] So we started with selecting search topics and our goal here was to make sure that the topics are indeed high impact. That is lots of people are searching for these topics so they should be popular, uh, and they should be topics which also have the potential to return false conspiracies.


[User] Please transcribe this text

[Assistant] Uh, so we did some, uh, more background work, referring to Wikipedia, comparing with Google Trends, and we came up with this, uh, list of five different topics, 9-11 conspiracies, vaccine controversies, moon landing conspiracies, chemtrail, and then Flat Earth.


[User] Please transcribe this text

[Assistant] And then, uh, for geolocation, uh, we found this hot and cold regions. That is, uh, we call these hot and cold regions because these are the regions which have the highest or the lowest interest for that particular topic.


[User] Please transcribe this text

[Assistant] And we found these hot and cold regions comparing with Google's, uh, Google Trends interest over time graph. Uh, so this is how it looked like for all the topics. So for example, if you pick the flat earth theories, Montana was a hot region or a high interest, uh, showing region while New Jersey is a low interest or cold region.


[User] Please transcribe this text

[Assistant] So once we had all these parameters, the demography, uh, geolocation and all these parameters, we essentially created bot accounts or sock puppets and programmed these accounts to keep firing queries on YouTube. For geolocation, these bots fired queries, uh, from IP addresses of these locations.


[User] Please transcribe this text

[Assistant] Now one very important thing that, uh, we had to do for running these audits is, uh, throughout these audit experiments, you would have to control for noise to ensure that the effect that you're observing is actually from, uh, the algorithm is not because of the noise, uh, that might have, uh, been introduced while running the experiment.


[User] Please transcribe this text

[Assistant] Uhm, we controlled for browser noise by selecting one single version of Firefox browser. Uhm, we made sure that the YouTube searches are happening, uh, simultaneously to control for temporal effects and so on.


[User] Please transcribe this text

[Assistant] Uhm, and then we perform statistical comparison tests to essentially find out what, what's the result of these audits. So let's look at some of these.


[User] Please transcribe this text

[Assistant] So what did we find? We found that for brand new accounts, uh, demography and geolocation do not really have an effect on the amount of misinformation or the type of conspiracy theories that these platforms are, or YouTube is, um, is, uh, returning.


[User] Please transcribe this text

[Assistant] This is encouraging. This is what we want the platform to do, right? So it tells us that unlike those, uhm, uh, reports which were blaming YouTube for returning conspiracy theories when it's a brand new account, uh, turns out the demography anti-location do not really have an effect.


[User] Please transcribe this text

[Assistant] But once a count spills the history, uh, by watching both demography and geolocation starts exerting an effect on the recommendation, uh, for a certain combination of topics, stances, and component.


[User] Please transcribe this text

[Assistant] Now you might be thinking, okay, this is expected, right? This is what we, what we, how we think the platform would behave. But turns out there are a little bit more nuanced results when we dig deeper into these, the actual audit outcomes.


[User] Please transcribe this text

[Assistant] So for example, for the 9-11 topic, if the sock puppets watched YouTube videos promoting line event conspiracy, you would end up getting more of these promoting videos in the recommendations.


[User] Please transcribe this text

[Assistant] So we know that there was a lot of pushback for, for vaccine related misinformation and it appears that they have gone and fixed that, but they have not done that universally for other, uh, problematic topics.


[User] Please transcribe this text

[Assistant] We also found that certain demographics were prone to conspiracy video recommendations. So for example, among eight of those demographic cases, in all but one case, men accounts, that is bot accounts who had gender set at, set as male, were recommended more misinformative videos.


[User] Please transcribe this text

[Assistant] Now this is really problematic. It implies that the algorithm was actually recommending pro-conspiracy videos even when, uh, the user, or in this case sockpuppet, was watching neutral videos on the topic.


[User] Please transcribe this text

[Assistant] Not yet developed pro-conspiracy beliefs, but now has a higher chance of developing that because the platform is, is returning, uhm, these, uh, uh, promoting conspiratorial, uh, videos.


[User] Please transcribe this text

[Assistant] So, uh, wrapping up this work, uhm, the key contribution of, of, uh, this study was that, in some senses, this work developed a methodology to audit search engines for misinformation, and we were also able to statistically prove that YouTube's behavior varies across different misinformation topics.


[User] Please transcribe this text

[Assistant] And our study also identified, um, uh, certain populations that could be potentially targets of, uh, certain types of misinformation. Uh, so this tells us that audit itself could be a useful way, uh, for, um, studying how algorithms might have differential impacts on certain marginalized, uh, populations. Yes.


[User] Please transcribe this text

[Assistant] And, and so if they do more of that proactively, uh, of course there is, you know, some hope in, in kind of changing, uh,


[User] Please transcribe this text

[Assistant] Maybe it was one of your students who did this work with crowd audits, right? Like where the crowd itself is reporting, because it's not really possible for the red team to find all possible scenarios under which, uh, uh,


[User] Please transcribe this text

[Assistant] Uh, there were several media reports at the time coming out suggesting that Amazon's algorithm were putting health and vaccine misinformation at the top of your reading list. But, uh, there was very little research to fall back, uh, to either verify or even, you know, kind of disprove these, uh, these reports.


[User] Please transcribe this text

[Assistant] Uhm, so again, we, we, this was the same question, uh, as before. How bad is this scenario? Uh, so we essentially wanted to conduct the systematic audits on Amazon search and recommendation algorithm. And we, here we picked only one type of problematic content, vaccine misinformation.


[User] Please transcribe this text

[Assistant] Uhm, and we conducted two sets of audits, un-personalized one and then the personalized audits. The personalized audits goal was to assess whether users, uh, account history built progressively by a user performing certain actions such as clicking on a product, adding the product to the cart.


[User] Please transcribe this text

[Assistant] And here the user built these account history progressively, uh, by performing a particular action for seven consecutive days. Uh, so these were again, when I say users, these were sock puppets, searching, searching plus clicking, searching plus clicking plus adding the product to the card. So all these were different actions that were performed.


[User] Please transcribe this text

[Assistant] Uhm, and then we also controlled for noise, very similar setup as before, uh, just with the caveat that this whole audit experiment setup was a big software engineering feat, uh, considering, you know, the amount of different, uh, combinations of recommendations, uh, possible on Amazon.


[User] Please transcribe this text

[Assistant] Uhm, and so if, if the user clicks on this, that algorithm next serves the user three other anti-vaxing books in the product recommendation page.


[User] Please transcribe this text

[Assistant] And then once the user, uh, adds a product or book to the card that it shows their intention to buy, both the pre-purchase as well as a homepage recommendation also rapidly changes with many more anti-vaccine book recommendations.


[User] Please transcribe this text

[Assistant] So this was not just a one-off case study. We found that more than 10% of Amazon products, uh, during the time period of our study for search terms like vaccine, autism, immunization, uh, resulted in misinformative, uh, book contain anti-containing active and vaccination content.


[User] Please transcribe this text

[Assistant] And so our audit experiment, just to give you the scale of this, uh, experiment, this was ran for a little bit over three weeks, resulted in 36,000 search results, several 16,000 recommendations, and then,


[User] Please transcribe this text

[Assistant] Uh, worked with several search filters, like featured, sponsored, different recommendation types, user actions, and so all of that resulted in that number of more than 10%.


[User] Please transcribe this text

[Assistant] Note size here is proportional to the number of times the product was recommended and the color corresponds to whether if it's a red, if it denotes a product annotated as misinformative, green, neutral and then blue, uh, debunking.


[User] Please transcribe this text

[Assistant] There are these large red-sized nodes attached to other red nodes, which are almost completely separated. There are like two separate components, right?


[User] Please transcribe this text

[Assistant] Uh, this just shows how strong of a filter bubble effect there is for this particular recommendation. People who are recommended misinformation products, they keep getting recommended those products super hard for them to break out from that red zone to get to the blue or the green zone.


[User] Please transcribe this text

[Assistant] I think, I think it took us almost a month to even come up with the whole annotation scheme and then, you know, five or six of our experts, including me, we kind of coded.


[User] Please transcribe this text

[Assistant] So it's a lot more qualitative process and then all the markers that we took, we also put it into the classifier to get the final annotations.


[User] Please transcribe this text

[Assistant] Uhm, very similar trend. And so the key takeaway here was our goal was to bring the focus, uh, to e-commerce platforms and show how their algorithms could be pushing, uhm, anti-vaccine content to users.


[User] Please transcribe this text

[Assistant] Recommendation algorithms, uh, should not be blindly applied to all topics equally. If it's a health topic, uh, perhaps companies need to pay a little bit more attention, um, and to ensure that there is higher quality content coming out of, uh, in their, uh, in their platform.


[User] Please transcribe this text

[Assistant] So this work of ours, uh, uh, intentionally and unintentionally was kind of rightly timed during the COVID pandemic. And so this was widely covered by several news channels. And in fact, Congressman Adam Schiff and Elizabeth Warren actually cited this research of ours, uh, in their letter to Amazon to control vaccine misinformation.


[User] Please transcribe this text

[Assistant] Uh, in the interest of time, let me talk a little bit more, uh, on, uh, one other type of study that we did, uh, looking at another type of audit method.


[User] Please transcribe this text

[Assistant] While these audits provide great control over your experimental design, uhm, you can pinpoint exactly which variable might be affecting the output of the algorithm, but one criticism of these audits is that the bot's behavior


[User] Please transcribe this text

[Assistant] Uh, are usually built in a very conservative way, right? So the bot, eh, in this YouTube case was essentially going and watching all pro-conspiracy videos or all debunking videos. Real users do not really act exactly that or that way, right? So these are these very extreme bot behaviors or user behaviors.


[User] Please transcribe this text

[Assistant] So we conducted this audit for a nine day duration on YouTube, and our goal was to assess, uh, to the extent in which YouTube was regulating, uh, U.S. based election misinformation on their platform.


[User] Please transcribe this text

[Assistant] And they quickly responded to those criticisms by introducing these content moderation policies to remove videos that spread election-related falsehoods and claim that misinformation videos would not be surfaced on their platform.


[User] Please transcribe this text

[Assistant] But then again, during the midterm 2020 elections, there were reports saying that YouTube has, still has misinformation blind spots, right? So uhm, they have not been very effective.


[User] Please transcribe this text

[Assistant] And so we did this through this post-hoc crowd-sourced audit. Why is post-hoc? Because it's conducted after the fact the event has happened, the elections of 2020, and we were conducting this study in 2022. And it's crowd-sourced audit since we investigated YouTube's algorithm collecting data from real-world users.


[User] Please transcribe this text

[Assistant] Uhm, and so our crowdsourced investigation, uhm, I think we started with like recruiting 600 to 500 users and we ended up, uhm, slightly lower than 100 users, so 99 particularly. So all these 99 users first filled out a pre-survey.


[User] Please transcribe this text

[Assistant] And about the beliefs on personalization on YouTube, how they trust YouTube search and recommendation algorithm, and then they installed this browser extension, uh, which allowed us to ca- collect users' personalized data. We also had all these ethical considerations, uh, uh, which I can go on, um, into more detail if anyone is interested.


[User] Please transcribe this text

[Assistant] But what this extension was doing, it was collecting search results of search queries, uh, related to the 2020 US presidential election, as well as voter fraud claims surrounding the 2020 elections. So two kinds of, uh, collection was happening. One was with respect to search results.


[User] Please transcribe this text

[Assistant] Uh, in the standard and incognito window and by comparing these search results in both these windows, our goal was to tell the extent in which YouTube was personalizing search results.


[User] Please transcribe this text

[Assistant] And then we were also collecting recommendation results. And the way we were doing this, we were collecting these UpNext recommendation trails after a user has watched a list of preselected videos with different stances on election misinformation.


[User] Please transcribe this text

[Assistant] And the extension would start by first, first watching a preselected seed video and then collecting up next, uh, videos up to five different levels.


[User] Please transcribe this text

[Assistant] But this, uh, through our audit, we found that YouTube actually, that their YouTube stop search results have little to no personalization. So this also tells you how users believe, uh, in algorithms, like the way they behave, uh, is, is different from actually the way, uh, the, the platform might be behaving.


[User] Please transcribe this text

[Assistant] Uh, but when we asked how much YouTube personalizes their UpNext recommendation, that perception actually aligned with how actually the audit results showed. Like, 51% of participants believe that YouTube personalizes UpNext recommendation to a great extent, which is in line with what our audit results found.


[User] Please transcribe this text

[Assistant] So this misinformation bias score varies from minus one to one. What the score does is it captures the amount of misinformation, election related misinformation while taking into account the ranking of the search results. So a positive score indicates that search results contain videos that support election misinformation while negative


[User] Please transcribe this text

[Assistant] So if you look at the entire distribution of scores for our collected results, we found that the misinformation score, if you look at the x-axis, it's mostly negative, which indicates that YouTube presents more debunking or opposing videos in the search results.


[User] Please transcribe this text

[Assistant] A couple of other key things also jumps off, right? So you could see, uhm, there are, uh, this distribution is bimodal.


[User] Please transcribe this text

[Assistant] Uh, so essentially there are two different clusters and each of these clusters corresponds to two types of search queries, uh, uh, I mean we didn't, uh, cluster it ahead of time, right? This, this, uh,


[User] Please transcribe this text

[Assistant] So the first cluster corresponds to voter fraud, uhm, uh, basically anything related to fraud in conjunction with keywords related to election, while cluster two is more generic election related searches, presidential election, mail-in ballots, and so on.


[User] Please transcribe this text

[Assistant] Uh, what, what's interesting here is that the cluster one, uh, has these mission information bias score which are more negative, which indicates that


[User] Please transcribe this text

[Assistant] Uh, which in some sense tells that YouTubes pay more attention to queries about election fraud and ensures that when users are searching for them, they are, in fact, being exposed to opposing misinformation videos.


[User] Please transcribe this text

[Assistant] So key takeaway here is uhm, in some way, YouTube is in fact successful in enacting election misinformation policies. So things that we wanted to test, as it turns out.


[User] Please transcribe this text

[Assistant] It's actually, uh, you know, aligning with how they wanted to enforce these policies, but it is indeed paying special attention to certain queries about voter fraud.


[User] Please transcribe this text

[Assistant] Uhm, and then finally as a byproduct of this, uh, audit, uh, we also found that there was some mismatch in participants' beliefs and the algorithmic reality, uhm, that happens, right, uh, which indicates a lack, some lack of awareness of algorithmic.


[User] Please transcribe this text

[Assistant] How algorithms behave. And I think, uhm, there has been, uh, other researchers who have been working in this space, uhm, looking at algorithmic folk theories, uhm, and, and how people's perception differ from the way, uh, these, these platforms work.


[User] Please transcribe this text

[Assistant] Now, uh, wrapping up, so these three, these are all the other, you know, the core studies that I wanted to present, but obviously coming back to how I started the talk, where do we go from here, right? How do we do meaningful algorithmic governance in the first place? That is, how do we set the path towards


[User] Please transcribe this text

[Assistant] The first is algorithmic, uh, uh, audits. And so governance via audits. And there could be many layers to this, right? So one of the layer is, uh, conducting external audits. And I presented some of these external audit studies, uh, through the three, uh, uh, research work that we have done in the past.


[User] Please transcribe this text

[Assistant] And also these audits could identify different types of risks. So misinformation is one risk, but you could also do the same for bias, discrimination, accountability, accessibility, uh, accessibility, fairness, and so on. And, and there are many, uh, uh, researchers who have worked in this space. Uh, I've listed some of these citations here.


[User] Please transcribe this text

[Assistant] But obviously our question is, so we as academic community, third party researchers, we are doing all this audits, is it really making any difference?


[User] Please transcribe this text

[Assistant] Plus the external auditors do not really have access to the models, to the training data, you know, which are obviously protected as trade secrets.


[User] Please transcribe this text

[Assistant] So as an alternative, another layer to, uh, governance via audits is you could do, uh, internal audits as proactive governance. And so, uh, at the time, uh, researchers from Google who are no longer at Google at the, uh, right now, but they released this paper.


[User] Please transcribe this text

[Assistant] Uhm, making a case for internal audits where audit would be part of, a core part of product development at every step of the way.


[User] Please transcribe this text

[Assistant] Uh, you could also do the best of both worlds, right? You could do something called cooperative audits, which is a fairly newer concept where, uh, while external audits answers what problems the platform has, uh, and internal audit says why that's happening, you could have a combination of both and you could do cooperative audits as shared governance.


[User] Please transcribe this text

[Assistant] So I've done a little bit of this, uh, with, uh, uh, Spotify where, uh, working with their engineers, uh, in the, within the company, uh, figuring out how gender representations might be biased or not for their taste onboarding and listen action on podcasts.


[User] Please transcribe this text

[Assistant] And then finally you also need to do these audits multiple times, right? Longitudinally. That is we need to conduct these continuous audits, monitoring platforms multiple times instead of that single snapshot audit. Many of my studies that I presented today are all single snapshot and we do need that kind of longitudinal effort.


[User] Please transcribe this text

[Assistant] So here is where I want to highlight one of the quotes from Debraji's earlier internal audit paper where they mention the audit process is necessarily boring, it is slow, it is meticulous and methodological.


[User] Please transcribe this text

[Assistant] Which is, which stands in stark contrast to the, uh, what I had started my earlier slides with. Move fast and break things. Done is better than perfect. So very much in contrast with the rushed culture of technology development.


[User] Please transcribe this text

[Assistant] And then also what are the key challenges, right? Uh, so one of the key challenges that is it's difficult to assess the risks that AI systems and large language models in particular pose independent of the context in which they are deployed. So we do need application specific audits for large language models.


[User] Please transcribe this text

[Assistant] Uh, so it's really hard to keep up, uh, the pace. And so doing ethical, legal, and technically robust targets makes it super challenging for such a rapidly developing, uh, technology. And so it must be complemented probably with much more newer forms of supervision and control. So here is one possible framework, one possible blueprint.


[User] Please transcribe this text

[Assistant] For auditing large language models, which is kind of three layers. So the first one is a model audit. Uhm, so as the name suggests, it focuses on assessing the technical properties of the pre-attained language models. So this is very similar flavor to the internal audit that I mentioned earlier. So that sort of proactive governance before you deploy the model.


[User] Please transcribe this text

[Assistant] But then there is the application audit, which focuses on the assessing the applications built on top of the LLMs, uh, so which is these flavors of post-hoc audit, right, uhm, and, and it should be done longitudinally, right, multiple times over a long period of time so as to capture any sort of new properties that might be emerging.


[User] Please transcribe this text

[Assistant] Uhm, and so for auditors, if we bring back the conversation for a second back to misinformation, one of, one instance where auditors did not really feel perceived fair treatment, uh, was this scenario where, you know, fact checkers are one of the key auditors of misinformation on online platforms like Facebook, Twitter.


[User] Please transcribe this text

[Assistant] So the fact checking organization Snopes a couple of years ago actually backed out of their partnership with Facebook because they didn't feel their values were being respected.


[User] Please transcribe this text

[Assistant] I think to delve into this question of fair treatment of auditors, we need more effort. And so one way in which my group, uh, has, has started a few initiatives, uh, we have launched a research endeavor with the fact checking organization based in Kenya called PesaCheck. And this has also expanded to 16 other fact checking organization across four different continents.


[User] Please transcribe this text

[Assistant] And we released our first report, uh, called the human and technological infrastructures of, uh, fact checking.


[User] Please transcribe this text

[Assistant] Finally, how do we ensure actionable audits, that is audits that result in real change?


[User] Please transcribe this text

[Assistant] Uh, release new API versions that reduced, uh, accuracy disparities, uh, in, with gender, male and female, as well as race, darker and lighter skinned, uh, subgroups. So in other words, a gender-shade study is a classic example of commercial actual impact.


[User] Please transcribe this text

[Assistant] To highlight how much we had failed in doing the actionable auditing, right? So despite widespread media coverage, despite a letter to from congressional, uh, you know, Congressman Adam Schiff, Amazon did not really act much. All they did was add that banner of, uh, COVID-19, uh, information directing to CDC's webpage.


[User] Please transcribe this text

[Assistant] So hopefully, uh, this kind of summarizes the challenges as well as the opportunities in setting the path for algorithmic governance and hoping, uh, with the new regulations coming in, in, in place, uh, maybe if I were to give this talk, uh, next year, I would have a little bit more hopeful slide than how I'm ending this talk.


[User] Please transcribe this text

[Assistant] Uhm, uh, so, so that's it. So this is all I talked about today. Uh, most of this work, uh, was done with my PhDs, then PhD student Prerna Juneja, who is now a faculty at Seattle University.


[User] Please transcribe this text

[Assistant] Uh, and then I, uh, my group, I would also stick in three other threads of work which I obviously don't have time to talk about.


[User] Please transcribe this text

[Assistant] Uh, but these are like, uh, a couple of other amazing students. So, uh, Shruti Farkesh, she did a bunch of work on computational social science. I was earlier meeting a...


[User] Please transcribe this text

[Assistant] A little bit of design intervention and social system design work, um, you know, with another student who is also a faculty now.


[User] Please transcribe this text

[Assistant] Uh, this is more of an HCI flavor where essentially questions like, how do you design a system to nudge users towards meaningful credibility assessment? How do you design a system, uh, to, uh, uh, to allow users to break out of their filter bubble? Uh, something called OtherTube that we built on YouTube.


[User] Please transcribe this text

[Assistant] And then finally, the last and the least fleshed out thread is some of the work that is currently ongoing with two of my students. We are looking at challenges and opportunities of generative AI in fact checking work.


[User] Please transcribe this text

[Assistant] And then what are some cultural misalignment that might happen with language models? Uh, especially, uh, you know, with, with roots in the global south, we are looking at cultural implications of these language models in, in countries like India and other countries in, in Southeast Asia.


[User] Please transcribe this text

[Assistant] Monetarily, financially, health or your life, then they're going to be more proactive and act on it.


[User] Please transcribe this text

[Assistant] So you're right, like moon landing is probably not, uh, to that extent, versus if it's vaccine information that has direct life consequences, right? But obviously then there are all these, all these other questions that when it's very well known that if you are drawn to one conspiracy, you're likely to get other conspiracy theories, right?


[User] Please transcribe this text

[Assistant] What you are describing those very entrenched conspiracy users, we term this as veteran users. And so they are one of the big drivers of bringing other people, like what we call joiners, into the community of conspiracy group. So I hope YouTube never does that.


[User] Please transcribe this text

[Assistant] Well, what you're suggesting, but, but that's a classic marker of how these social dynamics can actually bring people into these conspiratorial world views.


[User] Please transcribe this text

[Assistant] And, you know, empirically, uhm, we have seen that. And there is also social science theory proving that that definitely happens.


[User] Please transcribe this text

[Assistant] Uh, do these sorts of long term audits. It's fine to kind of develop the methods and uhm,


[User] Please transcribe this text

[Assistant] And you know, kind of say, okay, this one should pay attention, for example, to Amazon or, you know, this is the method to do it. But then you need like separate third party companies to continuously do these audits, right? So sort of like consumer reports. How does, what's the equivalent of that for audits?


[User] Please transcribe this text

[Assistant] I think at CSCW, the closing keynote, uh, RoomRoom was mentioning that kind of red teaming, I think their company or whichever NGO she's working with, they are doing something like that. I think, so academics with that peer review process, I don't think we should be responsible for doing those sorts of continuous audits because we are always going to play catch up with, with companies. Yeah.


[User] Please transcribe this text

[Assistant] If Arpaio does not comply with the court ruling, Arpaio could ultimately be held in contempt of court and face jail time.


[User] Please transcribe this text

[Assistant] No one forced you in on this deal, Cribbins. You accepted because you liked the idea.


[User] Please transcribe this text

[Assistant] Take it easy, friend. You're coming with me. Paul! You didn't pay for your drink.


[User] Please transcribe this text

[Assistant] I'm pleased. He's mute. I think you got yourself a good worker there.


[User] Please transcribe this text

[Assistant] So this is x1, y1, where those are the two first components. And then I do the same thing with the second components and all the way down the line until I take the


[User] Please transcribe this text

[Assistant] I can take any one of these and I can rearrange it the other way around.


[User] Please transcribe this text

[Assistant] And so then I can do the same thing all the way along, and I can write it as ynxn, and finally I can say that this is just the vector y dotted with the vector x.


[User] Please transcribe this text

[Assistant] So in other words, I can commute my doc product or I can change the order in which it's written.


[User] Please transcribe this text

[Assistant] The dot product also is going to play nicely with vector addition.


[User] Please transcribe this text

[Assistant] Double absolute value, the length of vector x, sometimes called norm of x, uhm, the length of x by the length of x equals the square root of the vector x dotted with the vector x. So I've taken a


[User] Please transcribe this text

[Assistant] As I stared, I heard a voice speaking to me, but it was not Doris. It was her companion.


[User] Please transcribe this text

[Assistant] But it seemed to me