[User] Please transcribe this text

[Assistant] So this is Mrs. Hamilton, uh, she's 62 years old, she's one of the user survey respondents. So she's currently working as a document assistant in a law firm, so she's essentially a typist.


[User] Please transcribe this text

[Assistant] And, uh, she's been learning online using various resources such as mine. And, uh, she writes, I wanted to learn computer programming so as to establish a new career and have some independence. So this is somebody over 60 who is in a job and wants to start pivoting into something, uh, perhaps more technical or perhaps more, um, more lucrative.


[User] Please transcribe this text

[Assistant] But, uh, the most frustrating aspect of learning online is I do not have a teacher on a face-to-face basis, such as in a classroom environment, and this has impeded me greatly.


[User] Please transcribe this text

[Assistant] Kind of survey responses like hers are representative of some of the frustrations people are feeling when they have to learn online. That they don't have access to all these great things that we take for granted in a formal educational setting.


[User] Please transcribe this text

[Assistant] So the central challenge that motivates the work I'm presenting today is the following question.


[User] Please transcribe this text

[Assistant] It's how can we scale up the benefits of face to face learning, uh, to the rest of the world.


[User] Please transcribe this text

[Assistant] And I want to start with programming as an exemplar because it's an example of a really cognitively complex and also a very in-demand skill set. But these ideas that I talk about may generalize to learning other sorts of complex skills at scale as well. So throughout the talk I'm going to touch on this theme of what about going beyond just programming.


[User] Please transcribe this text

[Assistant] Uh, the way I wanna, uh, I wanna think about this, this journey throughout the hour is this design space with two dimensions, with scale on one axis and with fidelity on the other. So on one extreme there's face to face, which is extremely high fidelity, but very low scale because you have to be there co-located in person.


[User] Please transcribe this text

[Assistant] And on the other extreme, we have static online material. This is learning from reading a PDF or reading somebody's lecture notes they put online or PowerPoint or YouTube video.


[User] Please transcribe this text

[Assistant] And the way I approach my work in the past few years is to first get inspired by the best practices in face-to-face learning, by things that we know to work effectively face-to-face.


[User] Please transcribe this text

[Assistant] Uhm, and then think about designing systems for scale from day one, uh, to overcome these limitations of face-to-face, and then finally leveraging the scale to make discoveries and then to drive the next steps and the next projects.


[User] Please transcribe this text

[Assistant] So the outline of this talk is following is I'm gonna first present a visualization platform and then talk about two specific, uh, social learning systems built on top of that and then conclude with some, uh, discussion at future work.


[User] Please transcribe this text

[Assistant] Before I start talking about the systems, which comprise most of this talk, I want to do, uh, kind of get into a little bit of theory of learning. So, uh, there's a lot of literature out there that shows that learning programming is really, really hard. And for those of you in the audience who have been programming for years, it's easy to forget how difficult of a task this is for, uh, for many novices.


[User] Please transcribe this text

[Assistant] This input-output pair, so there's three print statements here, and there's three lines printed. And you have to think, what just happened in the computer to, uh, bring this input that I wrote to this output? You know, why did this happen?


[User] Please transcribe this text

[Assistant] And this is something that's so basic that those of you here who are expert programmers don't even think of it. This is subconscious. Uhm, and this is the mental model of, of what happened in the computer. Uhm, there's three global variables and there are five lists. They each point to a list. This is a nested list. So this fully explains, uh, this input-output pair. This is what people do all the time. Uhm, the problem here is that this is completely wrong.


[User] Please transcribe this text

[Assistant] Uhm, this completely incorrect mental model and it turns out the novices come with all these misconceptions about how programming works based on prior knowledge, mostly about prior knowledge of math or prior knowledge of,


[User] Please transcribe this text

[Assistant] One of my colleagues in his dissertation, part of his dissertation, documented 162 of these misconceptions in this giant table.


[User] Please transcribe this text

[Assistant] The good news is that, uh, visualizations, uh, and diagrams are really effective at clarifying and, uh, and, and remedying these misconceptions. So if there's only somebody that drew a diagram for you and showed you how this worked inside the computer, uhm, then the act of studying these visualizations, uhm,


[User] Please transcribe this text

[Assistant] Uh, for novices, what happens is it reduces extraneous cognitive loads. And when you're learning, and you just, you don't have pictures, then there's...


[User] Please transcribe this text

[Assistant] So all this cognitive load and think and try to comprise these diagrams yourself and, uh, not enough, uh, uh, extra load to actually learn. So if you see diagrams in front of you, uhm, it's a, it's an effective way to kind of start solidifying these, uhm, these mental models in your long-term memory.


[User] Please transcribe this text

[Assistant] So in fact, in this case, uh, this is the right picture here. Uhm, is that there are actually only two lists and they're shared. Uhm, so this is the Python semantics. It's not obvious if you haven't learned Python before.


[User] Please transcribe this text

[Assistant] So the underpinnings of my work kind of, uh, basically the scope of the project I'm going to present, um, are of the following.


[User] Please transcribe this text

[Assistant] Is that first, is that building these proper mental models are foundational. They're prerequisites for learning anything more advanced. So before you can get to understanding how to do software engineering or work on teams or do idioms or design patterns, you have to really know, know nuts and bolts. What is the computer even doing with my code?


[User] Please transcribe this text

[Assistant] Or else you're just basically scrambling and having those misconceptions.


[User] Please transcribe this text

[Assistant] Second one is that studying these pre-made visualizations is an effective first step before, say, the teacher makes you draw your own. And this is, uhm,


[User] Please transcribe this text

[Assistant] This is an effect called a worked example effect in, uh, Educational Psychology which states that having examples shown to you first with good explanations will, uh, is a lower cognitive load activity. Again, that will allow you to spend more of your attention on learning rather than on trying to construct yourself first. So there's value in seeing these diagrams for, for novices.


[User] Please transcribe this text

[Assistant] And finally, these visualizations are most effective for early stage novices. People are just learning. And that basically what happens is you wean off them over time. So as expert programmers here, you don't draw visualizations for every single thing. This stuff happens in your mind. The schemas are quite different. Uhm, but it turns out that this is a really important foundation for novices.


[User] Please transcribe this text

[Assistant] Okay, so we know that visualizations are important. Uhm, so how do we generate them? So the traditional way to generate those is to have humans draw them. So humans draw these visualizations all the time when you're tutoring on a blackboard or uhm, just on paper. And the question here that I started with is how do we scale this up?


[User] Please transcribe this text

[Assistant] Specifically, here, is how can computers automatically generate these visualizations, and uhm, and how will people end up, uhm, using them.


[User] Please transcribe this text

[Assistant] So to start investing this question, I've been working for the past few years on this visualization platform called Rosetta that automatically generates these diagrams. So this site is more commonly known as the Python Tutor because it's the domain name I registered years ago.


[User] Please transcribe this text

[Assistant] Uhm, and, uh, but now it has grown to many languages, uh, and some more are coming soon. So this is kind of a language-independent site by this point.


[User] Please transcribe this text

[Assistant] You hit the visualize button, and then that sends the code to a server. The server has some, uh, machinery to instrument, uh, your code, analyze it, run it in a sandbox, and then it produces a value trace of everything that happened when your code was running.


[User] Please transcribe this text

[Assistant] And that brings you back to an interactive visualization on the web. So you write or copy code here and then it goes, makes a round trip and you see something like this. So it looks like a visual debugger. There's an arrow here.


[User] Please transcribe this text

[Assistant] There's a slider, there are buttons, and when you click the forward button, or when you, uh, when you slide, it will run the first line of code. So there's x equals a, b, c, and in Python the model is, there's a global frame here, there's x, and it points to a list. And then second line, y equals x, this is the line that's not obvious if you haven't learned Python before, because


[User] Please transcribe this text

[Assistant] Several things might happen, but if you actually run it, you'll see that they both point to the same thing. And then as you keep running, the diagram starts evolving, and uh, and there's the printout down here like the terminal.


[User] Please transcribe this text

[Assistant] You can also go backwards. There's actually no trick here because all the code has already been run on the server and processed, so this is just a thing we can slide back and forth.


[User] Please transcribe this text

[Assistant] So this thing has been deployed for a few years, but I'm hardly the first person to think about the idea of...


[User] Please transcribe this text

[Assistant] So uhm, my colleague has this great survey paper serving dozens of these systems from the past few decades. This is for Modula 2, which is a language I've never used, but uh, you know, it shows code, it shows the arrow, it shows these things. They all look, kind of feel like, uh, same flavor.


[User] Please transcribe this text

[Assistant] What we found in the survey was that, from a systems perspective, these are all custom built with one language in mind, a modular two, and one vocabulary, one static set of diagrams.


[User] Please transcribe this text

[Assistant] The practical implications of that is that they get really obsolete really fast over time. So I don't think anyone teaches in this language anymore. Uhm, so as a result, these things are really prototypes made for demonstrating certain visualization ideas or certain pedagogical ideas. And none of them have scaled beyond


[User] Please transcribe this text

[Assistant] So knowing that, I wanted to, you know, I didn't want to just make another thing that's just, you know, the same idea as before. I really wanted to design for scale. So that's the theme throughout the talk. Yes, James?


[User] Please transcribe this text

[Assistant] Preemptive, my first little point was the fact that this came at a time when the open web was starting to mature enough that you could do everything without software, without plugin installations. Uhm, and the big design decision here that is different than what people are doing in MOOCs and other things to not have user accounts is that I wanted to have minimal friction to using this, even if that means we don't track as much data. We can't.


[User] Please transcribe this text

[Assistant] Really simple feature to turn out to be effective is to generate a snapshot URL. So all the state can be encapsulated in just the, you know, in just the get request here. So you can generate a URL.


[User] Please transcribe this text

[Assistant] Uh, amenified or whatever, and you can end up spreading this virally. So you're on a MOOC, and you want help in something, and you say, hey look, check out this visualization I'm starting a play with, can you help me with it? Or you IM this to a friend.


[User] Please transcribe this text

[Assistant] And this is a way that it spreads not only, you can get help, but if someone is in a MOOC forum, or one of these Stack Overflow and these online courses, they see a link to PythonTutor.com, blah, blah, blah, some visualization, they think, oh, what is this Python Tutor site? And they go and use it in book market and spread it.


[User] Please transcribe this text

[Assistant] Other way it spreads, again, very standard for the, uh, in the web era is embedding. So you can generate JavaScript code or iframes to easily embed these live visualizations and code editors and anything else. So people have embedded this in MOOCs, embedded this in digital textbooks, and again spread it really organically.


[User] Please transcribe this text

[Assistant] The last thing about scale is more about longevity, uhm, is the fact that, uhm, I made this, I started this with Python as the site shows, as the site name shows, but I started thinking about how do we make this into a more general platform for visualizing things in other languages because


[User] Please transcribe this text

[Assistant] You know, I, there's no guarantee that Python's gonna be a popular teaching language in five or ten years or so, but I want this platform to keep living on and evolving. Uh, and the way to do so is to, uh, to have a more modular graphics, uh, vocabulary engine.


[User] Please transcribe this text

[Assistant] So uhm, I implemented this grammar graphics which is a classic idea from digitalization and adapted it for the purposes of visualizing what goes on inside of computers while code is running.


[User] Please transcribe this text

[Assistant] For some design inspiration, uh, I flip through many, many books. So if you look through textbooks and programming books that have diagrams in them in all sorts of languages, you'll find that the...


[User] Please transcribe this text

[Assistant] Graphical vocabulary is actually reasonably limited. So they have these box and pointer diagrams, they have these things stacked, they have nested boxes and layouts and such. And I came up with a fairly parsimonious grammar. Uhm, there's more details inside, but basically what it has is it has the idea of a memory block.


[User] Please transcribe this text

[Assistant] A primitive is what you normally think of in a language, a number, a string. Also, an image can be primitive because we're on the open web, so you can imagine if your programming language or your environment is manipulating pictures, you can have those be elements in your visualization.


[User] Please transcribe this text

[Assistant] There's no reason why we have to strick the strings. A collection is simply a list of blocks along with a layout that shows the, uh, tells the engine how to lay them out. So these are horizontal, vertical, grid, tree, graph. These cover the kinds of data structures you want to render. And because collections have blocks and blocks have collections, you can get this nesting. You also have an idea of a pointer to join things together.


[User] Please transcribe this text

[Assistant] So how do we use this, uh, uh, grammar to, to implement stuff for a new language? So at a super high level view of the graphics engine is, uh, from the user's perspective, you're writing code on the site.


[User] Please transcribe this text

[Assistant] You are running it on the server using some kind of tracer. This tracer has to be written for each language because you have to be able to parse and execute and process that language. So here are some examples. Uh, Python uses a debugger. Ruby has some kind of API for tracing code and see, uh, and uses this thing called Valgrind, which instruments binaries.


[User] Please transcribe this text

[Assistant] So once you have the trace, this actually runs the code and produces some data structure that's everything that happened when your code is running. The stack, the heap, the values, the types.


[User] Please transcribe this text

[Assistant] These are all specific to the language, so the other thing you need, another part of this equation, is some kind of spec that the creator of the tool writes once. And this spec roughly says how do you map Python's data structures into Rosetta's grammar that you saw in the previous slide, or how do you map the C data structures in the grammar. You put those two together in a step and you create a scene graph.


[User] Please transcribe this text

[Assistant] That, the vocabulary in the scene graph is the grammar that I showed in the previous slide. And then you can run it through rendering paths as usual. So HTML for webpage, you can, if you want to render a book, you can do PDF. If you want to do audio rendering, which I think will be a cool, uhm, thing for accessibility, you can render this representation in any way you want.


[User] Please transcribe this text

[Assistant] So why go through all this indirection and abstraction is because we want to separate out the language specific from the language agnostic part. So everything below is language agnostic so we can reuse. So in effect that makes it easy to add new languages because all you have to do is add a new tracer and add a new spec.


[User] Please transcribe this text

[Assistant] Seven languages so far, and it's, at this point, it's fairly routine. It's just routine engineering work to add more. And you can be really flexible in choosing abstractions. I don't have time to go through all the pictures, but you really, as the creator of this tool, for your language, you get to choose the level of abstraction you want and how you show diagrams.


[User] Please transcribe this text

[Assistant] So as a result of this, uh, kind of combination of engineering design work and, and timing, uh, that coincided with online education coming up about, um, there's been a tremendous amount of usage for this project. So the Y-axis shows, uh, pieces of visualized code. So every time someone is running the code and hitting visualize, there's, it goes on the Y-axis. So it's been out.


[User] Please transcribe this text

[Assistant] About three years now, so every day there are tens of thousands of pieces of code visualized by 5,000 plus users. What's more interesting to me beyond this


[User] Please transcribe this text

[Assistant] That these are really are people around the world of all age groups who are learning in various ways, taking online courses, maybe self teaching, and using this as a tool to supplement their learning.


[User] Please transcribe this text

[Assistant] So uhm, some user survey excerpts of how people use this. There's plenty more.


[User] Please transcribe this text

[Assistant] Uhm, on the web, as you know, it allows me to visualize the code step by step and to see what, uh, what are different components and often code get triggered. I always wonder if this tool has helped me understand this better than any other resource information. This idea of building mental models by surfacing.


[User] Please transcribe this text

[Assistant] With underneath. Debugging use case, I love it because it draws up the scope of things going on making concepts like mutability easier to understand. I also like for general troubleshooting with trace through the code and what is actually going on compared to what I think should happen.


[User] Please transcribe this text

[Assistant] So what have we learned from work on this project? First, the kind of key idea here is that surfacing what I call procedural state, what is going on inside the machine at some appropriate level of abstraction, is useful for both building mental models and also for debugging code and correcting misunderstandings.


[User] Please transcribe this text

[Assistant] Uhm, and that the obvious kind of next step for this platform is to start deploying more controlled experiments to measure these learning effects more concretely. So, so in these lab studies, people have shown kind of learning effects in the lab that these visualizations are an effective thing, but online we have not seen the test question.


[User] Please transcribe this text

[Assistant] And an abstract way of the kind of features I showed you about, you know, not requiring accounts, about being able to be modular, extend to different languages, having the URL, sharing everything. I mean, this is not Facebook scale, right? So the backend doesn't need to, we can do very standard things. So that, that isn't really too innovative. Yeah, it's more about thinking about longer term usage.


[User] Please transcribe this text

[Assistant] So the next really obvious thing to do is think about really controlled experiments. We have this platform with thousands of people on it every day to really start measuring learning effects and retention effects and such. And we have a platform by which to launch that now.


[User] Please transcribe this text

[Assistant] So in this design space here, where I would put this is as an improvement over the static online materials. That ability to visualize procedural state, uh, gives you a bit of more fidelity, at least in reading, say, a PDF or watching a video.


[User] Please transcribe this text

[Assistant] And thinking beyond programming, uhm, can surfacing this step-by-step procedural state help people learn other soft subjects, right? Learn math or physics. So you can imagine math and physics being amenable to this similar idea of decomposing things into steps or doing proofs or those things. What about law? So, you know, legal documents are these,


[User] Please transcribe this text

[Assistant] Dozens of pages of dense, uh, dense text, uh, what about visualizing that? So, so some people have started manually doing this, of kind of encouraging design of, uh, more visual sorts of legal briefs, and I think there's opportunities to try to think about using NLP or things to automatically extract


[User] Please transcribe this text

[Assistant] Logical sort of statements from legal documents and try to, uh, surface those as well.


[User] Please transcribe this text

[Assistant] But there's still a problem. I mean, so far, there's still no human in the loop, right? So Mrs. Hamilton's original frustrations have not been met yet. That there's still no person helping her along. She can see these visualizations just fine, but uh, she needs someone to explain them to her.


[User] Please transcribe this text

[Assistant] So to start addressing this issue, I've created this tool called Coachella and I had two of my master's students help me run a case study on it last year.


[User] Please transcribe this text

[Assistant] For inspiration, I went to the computer lab. So this is a typical computer lab in any school. You see a lot of pair programming debugging going on. You see a lot of pointing to the screen and talking about shared context. And people draw diagrams as well, because the computer by default doesn't draw them.


[User] Please transcribe this text

[Assistant] So again, how can we scale this up? Uhm, and in particular here the question is,


[User] Please transcribe this text

[Assistant] Uhm, how can we bring these in-lab interactions that are really high fidelity, high touch, kind of pair programming, collaborative learning interactions to people who can't actually meet up face to face? Uhm, and to, uh, start addressing this question, I built this tool called Coachella, which is the first multi-user visualization system. It's an extension on the Rosetta platform, so.


[User] Please transcribe this text

[Assistant] Here's a demo. Uh, this is the Rosetta site. Python.jr.com. You can write code, visualize, you can...


[User] Please transcribe this text

[Assistant] Uh, visualize everything, and if you need help, click this button up here, and if your tutor is in a MOOC halfway around the world, you send them this URL, and then they join your session, so they're, you know, somewhere in another part of the world, and you see multiple cursors, you see their cursor, uh, all your actions are synchronized as well, and you can just chat in an embedded box like that.


[User] Please transcribe this text

[Assistant] And then you can edit code, there's a collaborative editor, uhm, all built in the site.


[User] Please transcribe this text

[Assistant] So what this is trying to do is start to emulate the sort of interaction that people have in lab, except you can be anywhere doing this.


[User] Please transcribe this text

[Assistant] So, ideas about designing for scale. So first of all, it's built on top of the Rosetta platform, which has already been scalable by this time, and the main things that I added were just the machinery at the backend for synchronizing cursors and state and chat and everything.


[User] Please transcribe this text

[Assistant] Uhm, just like using Rosetta, joining the session now is as simple as joining a web URL. Uhm, no installation, no screen sharing software, it's really lightweight. Uhm, you're just doing it right in your browser. So you can just send it to someone in a form.


[User] Please transcribe this text

[Assistant] The multiple cursors are a really lightweight way to emulate, uh, some sort of presence. So you can actually see people waving around saying, look, look here.


[User] Please transcribe this text

[Assistant] And finally, text chat is more scalable than, say, doing something like video, um, both for bandwidth reasons, if you're halfway around the world, and also it's sort of a lower-barreled entry thing, so that if you're Mrs. Hamilton in a MOOC, you might be kind of shy and not want to video chat with a tutor around the world you don't know, but text chatting or, you know, writing for them may be okay.


[User] Please transcribe this text

[Assistant] The design point of this tool is really for one-on-one tutoring like this, or small groups. You can imagine five people entering a room together with five cursors and chatting. But you don't want to scale this up too much more because, uh, you're all sharing one session.


[User] Please transcribe this text

[Assistant] So some questions about this is one, can it bring these in-lab interactions to people who cannot meet up face to face? And can people actually learn and feel some sort of human connection using this tool?


[User] Please transcribe this text

[Assistant] So to investigate these questions, we, uh, we deployed it to the website for about a year and a half, uh, simply a button up here, there's a little FAQ, but we did not do a, uh, we didn't have any task or paid recruitment. We wanted to see, we have all these thousand people on the site in a naturalistic way who are already learning online, uh, would they make use of this resource?


[User] Please transcribe this text

[Assistant] This additional resource we put here in their naturalistic learning, and if so, how would they do it?


[User] Please transcribe this text

[Assistant] So, uh, the data shows that in fact it did connect people who are remote, uh, which is good.


[User] Please transcribe this text

[Assistant] So there are 740 sessions after we filtered for junk sessions that weren't anything of substantive. So these are actually sessions where people were actually chatting and interacting.


[User] Please transcribe this text

[Assistant] Uh, they came from hundreds of cities, dozens of countries. Uh, the important data point here is that almost half the sessions are multiple-city ones. So these are actually people who found each other somehow, maybe on their MOOC, maybe on some forum, and were working with each other. And one-sixth are multiple kinds. So obviously these people could not have


[User] Please transcribe this text

[Assistant] Gotten together next door, uhm, and got over to each other's, uhm, dorms to chat. So this, uh, graph is connecting countries if there's a session, uh, where people participate in those countries. Again, because this data was all anonymous, no accounts, we don't know the detailed demographics of these people, but I suspect that these were taking, people taking online courses together. Yes, Michael?


[User] Please transcribe this text

[Assistant] Yo, yo, how does that relate? Uhm, it is a bit on the low end. I think that, uhm, because I think all we did was we just put this button on and just said, you know, there's like an FAQ here.


[User] Please transcribe this text

[Assistant] We didn't advertise this in any particular way. There was no structure around him and this is exactly the site. So uhm, yeaah. I mean, I think these numbers are a bit low.


[User] Please transcribe this text

[Assistant] Again, uh, we didn't, you know, we didn't purposely try to pump up those numbers or encourage in a structured way. Like, imagine if, if we're deployed as part of a course and there was a curriculum around it, then we would have more controlled usage. Yeaah, so the numbers are what they are.


[User] Please transcribe this text

[Assistant] Uhm, alright, so you know, this is kind of the naturalistic, as kind of Michael question alluded to, this is just in the wild, this is sort of the amount of usage we get. And I think part of the reason for the percentage


[User] Please transcribe this text

[Assistant] Of sessions created over total users being low is that we actually don't have a way of connecting people together currently. You actually have to know somebody to do this so not everyone is going on this site. So this kind of naturally previews some future work.


[User] Please transcribe this text

[Assistant] Uh, it turns out that people interacted like they would in lab. Uh, so by coding, manually coding the chat logs for, uh, for activity intent, there are two things that just, two kinds of activities, judges, popped up, really apparently from the logs. So one-on-one tutoring, which is one person is clearly the expert, and working with someone else, and then peer learning, which is


[User] Please transcribe this text

[Assistant] Socratically narrowing down the space and helping the learner debug.


[User] Please transcribe this text

[Assistant] And having learners answer questions. So we saw some really good tutors on this. We saw some tutors that weren't as good. It varied a lot. And future work is thinking about how do we structure the tutoring sessions.


[User] Please transcribe this text

[Assistant] Peer learning was, you know, there's two people here, so imagine two or three people hashing something out together like you're working on homework.


[User] Please transcribe this text

[Assistant] And you know, peer tutoring sessions, peer learning sessions are a lot more tentative and back and forth.


[User] Please transcribe this text

[Assistant] Um, you know, about a chat every minute, and there's way more visualization reaction than chat. So this is stepping, running, uh, going backwards and such. Um, so the, by analyzing the chat log, they're both declarative and procedural knowledge was transferred. Declarative is what is the syntax thing? What does this mean? Procedural is let's step-by-step debug this problem.


[User] Please transcribe this text

[Assistant] So here's an example here you can read that's pretty dense and it doesn't make much sense if you don't see the picture, right? This is the sort of conversation you can only have if you're seeing a picture in front of you and going back and forth and pointing to it. So that's kind of the value added having a picture and the text at the same time.


[User] Please transcribe this text

[Assistant] Tutor says, not a problem at all. You've written little code yet. It'll take a while for it to come more comfortable. So we definitely saw these technical, technical, technical talk and then there's some emotional bonding thing and the more technical stuff. So this is, you know, very different than say if you were just getting automated hints, uhm, from, from a, from a system.


[User] Please transcribe this text

[Assistant] So what have you learned? Uhm, that first, this real-time substrate for discussing procedural state, uhm, can be an effective substitute for people who otherwise can't get together face-to-face, right? That you probably want to get together face-to-face if you can, but if you can't, then this starts emulating some of these benefits.


[User] Please transcribe this text

[Assistant] But more interestingly than the half the population who were in different cities, actually in a third of the sessions it was being used even by people who knew each other in real life. The way we code it for this is there are indicators that they


[User] Please transcribe this text

[Assistant] New each other, they're like, we're gonna go to this party tonight, or, uh, this class is, you know, terrible, or whatever, or, you know, when is study hall? And they were using this tool, they're electing to use this tool, even when they could obviously been together, and this is, um,


[User] Please transcribe this text

[Assistant] And this classic HCI concept, design concept beyond being there, which is that, you know, thinking about how do we design systems that people would prefer to use even when they could just physically go to each other in real life. And I think this idea of discussing what's underneath contributes to that. Yes, Michael?


[User] Please transcribe this text

[Assistant] Part of it is that it is this lightweight thing that you could kind of integrate into your, I'm already sitting in my dorm, computering, and then this is just a lightweight thing rather than physically going somewhere. But, uh, but I should think about this more offline. But I was thinking more of it in terms of not having, you know, of the base case of, of this. Yeaah. But I think you're, I think this is a really good point.


[User] Please transcribe this text

[Assistant] Yeah, that's a great point. I mean, this is something we could definitely do in the lab. Like, we have the, like, both your points. We have this thing and we just put them in lab co-located using it. Um, there are people who did this a few years ago in person with some kind of simpler visualization system and one of their findings, I can dig up the paper, one of their findings is that, uh,


[User] Please transcribe this text

[Assistant] Being able to talk about the visualization, at least in person, gave people that sort of teamwork camaraderie sense. They gave people that the benefits were more social of having them feel closer to classmates and perhaps we don't get that as much in the online setting.


[User] Please transcribe this text

[Assistant] So uhm, the final thing is that these spontaneous effective exchanges, uhm, show this interface can humanize online learning. This is not just about, let's just exchange facts of this, this sort of, even if we have a low bandwidth chat, uhm, we could, people actually are engaging in ways that are non, non-technical.


[User] Please transcribe this text

[Assistant] So, uh, what Coachella is trying to do is to push out this, uh, this scale factor, right? So we have lower fidelity, uh, than face-to-face, but we're scaling a bit better because we don't need a, uh, a person there, you know, in person. We can be online.


[User] Please transcribe this text

[Assistant] Uhm, thinking beyond programming, so can this idea of synchronized procedural state foster real-time critique in say a virtual art or a design studio where people are working on these procedural tasks. So if you're sculpting, uh, or doing some kind of visual art, if we could say instrument the room or instrument these, uh, you know,


[User] Please transcribe this text

[Assistant] Metal scaffolding or everything and be able to project your real time state can, can someone remotely tutor you and critique you doing this. Whereas right now this sort of visual art and design stuff has to be very much in person.


[User] Please transcribe this text

[Assistant] But there is a problem that came up when we were doing this project, which is pretty obvious in retrospect, which is that the tutor, there aren't that many tutors available, right? So if you're in a MOOC, uh, or an online setting, you're Mrs. Hamilton, you can use this to summon a tutor, you ask for help.


[User] Please transcribe this text

[Assistant] This is great for you, uh, you have a personal tutor there, uh, but the tutor, it's, it's a lot more limited because they can only help one person at once because they are, uh, they are bound to this one-on-one tutoring situation and what if the tutor wants to help five students or ten students or such? They have to open up five or ten tabs and switch between them.


[User] Please transcribe this text

[Assistant] And that the tutor attention is really the bottleneck to further scaling up, uhm, being able to provide real-time help online. Is that because there are way fewer tutors than learners. Uhm, so how can a single tutor help more learners at once? Uhm, and that leads into, uh, the system I created called CodeOpticon, which has nothing to do with TurkOpticon.


[User] Please transcribe this text

[Assistant] Once again, for design inspiration, I went back to the computer lab and this time instead of looking at the students working, I looked at the instructors or their lab tutors.


[User] Please transcribe this text

[Assistant] Uhm, the ideal thing for them is to provide timely and targeted help, which is someone raises their hand and they go up to them and they provide a target assistance right away when the context is fresh. Proactive help is also really important because what a really good tutor would be able to do is to scan the room and be situationally aware of


[User] Please transcribe this text

[Assistant] How people are doing and if someone looks like they're really stuck or frustrated, go up to them and offer assistance. Many learners are either shy and they don't want to ask for help or they're too fixated on debugging something and they don't know when they should step back.


[User] Please transcribe this text

[Assistant] That's the ideal. Reality doesn't really match that, right? You have 20 to 1 ratio, 30 to 1, 50 to 1. There are just way more, uh, students in the lab than tutors. So it's a best effort scramble. It's physically exhausting to run around the lab and, you know, if you have lab hours for an hour, it's actually really tiring to run around.


[User] Please transcribe this text

[Assistant] I wanted to address this question not only for the in-lab case, so this is great if you're building a tool that can help tutors.


[User] Please transcribe this text

[Assistant] Uh, but I want to think about scales. I want to design something that not only works in person, but also works in this online case. If you're in a MOOC or in Khan Academy or in some online learning platform, you may have a hundred or so people online at once, uh, working in various ways and you want to be able to help them in real time.


[User] Please transcribe this text

[Assistant] So I made a system called CodeOpticon, uh, which is the first one to many tutoring interface. So how it works, roughly, they have a tutor on their computer, and they're monitoring a group of learners in real time as they're coding.


[User] Please transcribe this text

[Assistant] Uh, on here, on the left side is the learner's code editor. So this can be implemented in any, say, web-based editor. I put it on the Rosetta site because that's the platform that I, that I have. And the right side is the tutor's view. So this video will play twice. First, uh, look at the left side for the learner, and then the whole thing will play again. We're gonna look at the right side for the tutor.


[User] Please transcribe this text

[Assistant] So learner is on the site, uh, trying to write a program for maybe homework. So product function, x, y is arguments. And I know they change their mind, a and b is arguments.


[User] Please transcribe this text

[Assistant] And when you try to compile and run it, there's some bizarre error that is not helpful. So maybe it's a lowercase x.


[User] Please transcribe this text

[Assistant] The tutor sees the learner's actions streaming in, uh, in chunks, uh, nearly in real time. So green is a diff for adding and a flashing red shows deletion. So this way the tutor can see what the learner is doing.


[User] Please transcribe this text

[Assistant] And then when they run their code, uh, they see the error right away.


[User] Please transcribe this text

[Assistant] Okay, so now they can actually go in a history slider and just go backwards and see the chunks of what the learner was doing and compiling and running, uhm, just back and forth, uh, to try to get some contacts and then they can just start a chat.


[User] Please transcribe this text

[Assistant] So in sum, the tutor's view of one learner is to summarize in this tile of their compiling and edit activities with history and chat. So how does this scale up to more than one learner? The interface is pretty straightforward as we just put it in a grid. So this is, on a laptop you have about 10 spots, if you have a big computer you may have 15 on a big monitor.


[User] Please transcribe this text

[Assistant] The first thing here is that this is easily deployable. We have modules of engineering work is that the learner can use their existing interface. So if they're programming Rosetta or they're on Codecademy or on some MOOC, we have to instrument that


[User] Please transcribe this text

[Assistant] Working in real time, because there's so many people, you can only attend to probably a few at once, so if you miss something, you can just quickly slide back, uhm, and get the context. You can start a text chat with anybody you want and just open up these multiple text chats, so each one of these are private one-on-one chats, as though you're helping someone privately.


[User] Please transcribe this text

[Assistant] And of course the obvious question is what if there are more than 10 learners? So you can imagine more people joining in down here and piling down. So if you have 100 people and people coming in and out, then you're going to have to scroll a lot.


[User] Please transcribe this text

[Assistant] Uh, but the main observation is, especially in an online setting, people have very wildly different levels of activity. Someone may be watching a video in another tab or going on Facebook, and someone may be frantically trying to edit and, and debug and getting errors. So we have a pretty simple heuristic that,


[User] Please transcribe this text

[Assistant] Periodically reshuffles around some tiles to bubble up the learners who are most recently active, and then when they become inactive they get pushed down. And the idea is that


[User] Please transcribe this text

[Assistant] The tutor's main field of view, they're seeing the people who are most recently, uh, doing things. And this is a very simple heuristic, uhm, and it's worked reasonably well in the first pass, but there's many ways to improve this heuristic right here.


[User] Please transcribe this text

[Assistant] So the questions that everyone is thinking about is, one, the most obvious one is how many people can actually someone handle using this interface? Uhm, two is can they successfully help learners? And three is how does, how do they feel this compares to a real life tutoring situation?


[User] Please transcribe this text

[Assistant] To find out, I did a study that was both, uh, in the lab and online. So the tutors were in lab, physically in lab eight subjects, one at a time. Just, uh, one person sitting in front of the computer at a time. This was the staff from the intro course that all held these lab hours. This is exactly the target population. These are the people who would walk around doing lab hours.


[User] Please transcribe this text

[Assistant] First time using the tool, uh, as you said, spend 30, give them tutorials that spend 30 minutes helping anonymous learners on the Rosetta website in whatever way you feel most comfortable. So again, we wanted to see, you know, to emulate what they would have done in lab normally. Uh, there was no kind of tasks or quotas that you must help five people in half an hour or whatever.


[User] Please transcribe this text

[Assistant] The people online knew during the session that they were being monitored and their actions were all being monitored, but they weren't expecting the proactive help. So this is akin to the customer service case where you're browsing a website and someone pops up and says, it looks like you're buying shoes. And not surprisingly, most people close the tab.


[User] Please transcribe this text

[Assistant] Or they didn't speak English because it came from hundreds of other countries that didn't understand it. So uhm, this really, this simple sort of first study is really a worst case scenario.


[User] Please transcribe this text

[Assistant] If you were really to use this tool in a real way, the tutors would know how to use the tool and the learners know about it and we have say, virtual office hours. So let's see how they did even given this anonymous scenario.


[User] Please transcribe this text

[Assistant] Uh, what each squiggly line is one session, so there are eight of them. This is how many learners are online at once. So they hover around 50 at once.


[User] Please transcribe this text

[Assistant] Every 30 minute session about 200 people came in and out. So this is because this is an online site. People are going coding and then closing the tab or leaving or stuff. Uh, so this is around the range of people being monitored. So a few dozen. Uh, in a MOOC maybe there'll be a few more. Maybe a hundred or so. Uh, notice that most people went over the 30 minute time limit.


[User] Please transcribe this text

[Assistant] And that gets into this question here of how do the tutors feel while they're using this interface?


[User] Please transcribe this text

[Assistant] Uh, the biggest feeling they reported, both, uh, from their actions in their, their think-aloud, was that they, they felt this great feeling of immersion. That they felt that, first, this kind of, this foreign kind of interface, but they really got into it pretty fast, after about the first few minutes of looking around. Everybody pretty much went over the time limit.


[User] Please transcribe this text

[Assistant] They didn't show any visible signs of fatigue or overload even though I didn't like strap an ADG on their head, uh, but they-


[User] Please transcribe this text

[Assistant] They felt this was enough that they could just keep going, just like a lab hour. Also, they said this was way less tiring than having to physically run around. They're sitting at their desk. It turns out, as expected, they selectively attended to things. I didn't do eye tracking, but the mouse tracking is reasonable because you can kind of see people's mice going to certain places.


[User] Please transcribe this text

[Assistant] Authenticity was a big thing they reported as well. That they felt like they were tutoring in real life with, uhm, their real personalities and style show. So some people were a lot more reticent, some people were a lot more proactive jumping in and trying to help people. And the way in which they engage with learners differed as well. Uhm, how does this compare to in-person tutoring?


[User] Please transcribe this text

[Assistant] Big advantages were they can control their own pace better, so they reported they love having a dashboard to have situational awareness so they could see what was going on and selectively help people instead of people raising their hand and them running around frantically. Uh, the history slider is super helpful and something you can't get in real life that they really liked. And, um,


[User] Please transcribe this text

[Assistant] They could help about, it turns out they could help about three people at once. Uh, so this is a log of how many chat windows they had opened, concurrent conversations, so they kind of, some people started off slow and chatted, some people chatted right away. It's around two or three at once, and uhm,


[User] Please transcribe this text

[Assistant] This, this number kind of makes sense because what happens when you're helping someone, they're not responding usually in real time. You give them a hand, you ask them a question, they go off for a while to do something, so it's nice to have this dashboard so you can help somebody else. Yes, question?


[User] Please transcribe this text

[Assistant] Presumably on different problems or at their own pace, but you can imagine if you deploy this in a class in a lab hour saying we are all working on merge sort now, start, and you all have the same context, we could do that and we could do even really cooler things with clustering and other stuff when we have the same context. That's a great, that's a great idea.


[User] Please transcribe this text

[Assistant] Active, right? So that's the, uh, that, uh, that prioritized activity has that drawback. One way, a simple way to fix this is, in a production version of this, is to have a get help button so the learners can summon the tutor privately for help, and that's a signal that they need help.


[User] Please transcribe this text

[Assistant] What do the learners, uh, do in these sessions? So about two to three learners confirm help in each 30 minute session. That's them saying, you know, blah, blah, blah, thanks, you fixed my problem. Or like, oh, I really wish I had this. Uh, this is despite the fact that these are completely unsolicited. So in a, you know, in a real class situation. That's right, that's right.


[User] Please transcribe this text

[Assistant] We injected those as well. Uhm, actually some of the ones, I would love to go behind the scenes. Some of the ones, you know, they were asking, are you a bot? Alright. And I said yes. They said no.


[User] Please transcribe this text

[Assistant] Uhm, yeaah, it turns out if you start too generic, people think you're a bot, right? You're like, oh, do you need help on this? Then they might actually think you're a bot, whereas if you are more targeted. So, it's interesting, some of the users, they started off not getting good responses, then they try to think of how do we pass the Turing test here, myself, and actually help them better.


[User] Please transcribe this text

[Assistant] Both types of knowledge again, this time at the lower two levels of move technotomy, and here's some examples of, uh, these are mostly kind of language or runtime specific things they're helping with. So this is typical of the sort of beginner errors you're, you're helping people debug.


[User] Please transcribe this text

[Assistant] Uhm, so a tutor, it ends up being a moderate of a few dozen learners to help out three at once. Uhm, again with this being their point is that, uh, pretty much everyone said that they wanted this tool even for their in-person courses. And, uhm, and a big benefit they said was that, you know, they just didn't want to run around.


[User] Please transcribe this text

[Assistant] They don't want to run around live and they want it to be able to have, feel more control over their tutoring.


[User] Please transcribe this text

[Assistant] Can start overcoming some of the limitations of real time monitoring multiple people. So just a simple idea of having a history slider really helps a lot. Because if you don't have histories, then you have to really see what goes on in real time. And that's not gonna, that's not gonna scale.


[User] Please transcribe this text

[Assistant] And then, despite this low bandwidth chat, the tutors found it to be an immersive and an authentic experience for doing this sort of light lab style kind of proactive tutoring.


[User] Please transcribe this text

[Assistant] So what Coachella, uh, CodeOpticon does is push out on the scale even more, right? So it goes from one to one to one to many.


[User] Please transcribe this text

[Assistant] I'm thinking beyond programming, so what if everybody could watch everybody else working all the time, right? What if we had a code-opticon for every student? So, you know, intellectual honesty issues and copying code issues notwithstanding, you know, what can we gain from actually seeing what other people in the class are doing?


[User] Please transcribe this text

[Assistant] Uh, a few minutes up in San Francisco and they have these labs where they just have a hundred stations with physical apparatuses and computers and everything. We have instructors and students just roaming around looking at what each other is doing and uhm, can we think about how to do this online? And we can even do better online because we have histories in other sorts of invisible state they don't have in person.


[User] Please transcribe this text

[Assistant] Uh, the, you know, the problem that came up when working on Code Opticon and Coachell, both the systems, is that we're presuming that there are tutors available, right? You're presuming there's some situation where there's someone there to help you, uh, but what if there are nobody available in the worst case scenario?


[User] Please transcribe this text

[Assistant] And for that, uh, we turn to, uh, the fellow learners. So if there's no tutors available, uh, what about if you summon other learners for help? So on the Rosetta site, we get about 60-ish people, up to about 100, depending on time. These are people online at once. At a MOOC, there may be around 100.


[User] Please transcribe this text

[Assistant] So, uh, here's some, you know, start of some ideas for, uh, how do you leverage the crowd? So, uhm, I have a student, Mitchell Gordon, who's working with me. We had one paper come out with some preliminary results on this, but can we have the crowd create tutorials? So, can we?


[User] Please transcribe this text

[Assistant] And have them be able to now run the code and surface the internal details and see what's going on and make annotations of what they think is happening and then do the usual, crowdy things of voting and filtering and aggregation. And can they come up with reasonable tutorials? It turns out for simple, basic kind of CS1 examples, they do a reasonable job.


[User] Please transcribe this text

[Assistant] Another thing is peer tutoring. So we have all these dozen people online. Can we match them up to each other or make suggestions for them to help each other? Uh, because there are no experts available. So we could use things like their activity level, their skill level based on what they're editing. So you can say like, oh, it looks like you're working on linked lists. Would you like to work together with Bob or Carol? We're also struggling with linked lists.


[User] Please transcribe this text

[Assistant] Uh, yes, well it looks like you're struggling with linked lists. Or maybe you can help, Bob or Carol. Uhm, uh, you, what, obviously there's ton of, you know, hand-waving here of, you know, how do you not make this into chat roulette or something, you know, totally mismanaged. Uh, but the crowd is there.


[User] Please transcribe this text

[Assistant] So to sum up really what's going on here is that we really want to provide someone like Mrs. Hamilton with more than just, uh, these static online materials, right? So if she can't get face to face, we really want the future in these online learning environments to provide a spectrum of options depending on kind of trade-off of fidelity and scale. I've presented these cases


[User] Please transcribe this text

[Assistant] For programming here, but we can imagine generalizing these ideas to learning other skills if you could think about how to visualize things and how to surface the histories.


[User] Please transcribe this text

[Assistant] Uhm, the ideal here obviously is high fidelity and high scale. So what does that mean? Uhm, maybe it means mobile wearables, because more and more people will just have them with them at all times. Maybe it means higher fidelity things like VR or even brain computer interfaces. So, you know, these are kind of wild blue sky ways to think about this. But, you know, so far we've had to make this trade-off.


[User] Please transcribe this text

[Assistant] So in sum, the question I started with is how do we scale up, uhm, the benefits of face-to-face learning online.


[User] Please transcribe this text

[Assistant] And the approach I've taken in this talk is to, uh, by surfacing procedural state, what's going on step by step inside of some black box, and then also surfacing the history. And then fostering an environment where you can do real time monitoring and chat so that people are actually engaged in real time.


[User] Please transcribe this text

[Assistant] And to put this much more simply, it's actually a really simple idea, which is to surface invisible and then talk about it. So just find out some way to surface what's going on underneath the covers and let people talk about those things so that they have a much more richer conversation than just talking about, say, the code.


[User] Please transcribe this text

[Assistant] And I've shown this for programming, but I think there's, these things can apply for kind of physical design activities, for things like medical education, and even for something as remote as law, where, you know, everything is very text-based and inherently, uh, not visual.


[User] Please transcribe this text

[Assistant] Uhm, and with that, I'd like to conclude my talk, and I'd be happy to take more questions. I'm around, so I'd be happy to take more questions as well. Thank you.


[User] Please transcribe this text

[Assistant] Yeah, I think that's an open challenge of, I mean, the question of, even if you have hundreds of people, they're in different time zones, they have different contexts. Uhm, one way to think about this is, uh, you know, I'm proposing something totally synchronous, real time. The existing state of the art in a MOOC is totally asynchronous as a forum. One way to think about this is bridging the two, right? So I think,


[User] Please transcribe this text

[Assistant] Something, oh, design things like Facebook Messenger, I like, or Slack, where you can chat in real time, but it's also kind of persistent, so you can come back in an hour and talk as well. So imagine some kind of design that bridges that, I think, maybe one way to scale across time.


[User] Please transcribe this text

[Assistant] So uhm, you probably don't want to debug something with a million nodes on, you know, because you can't draw on the board. So uhm, there's two issues with how long the code is and also how complex the runtime state is. So you could have a hundred lines of code, but it's generating only a few objects and the objects are changing. That's, that's fine. But in general, I would say 50 lines of code at most with


[User] Please transcribe this text

[Assistant] I haven't looked in it personally myself, but my suspicion is that, you know, given the right design and instrumentation you can't do it, because I don't know about how these are programmed, if it's some declarative language, if it's some kind of, you know, Verilog thing, or you know, with circuits or stuff.


[User] Please transcribe this text

[Assistant] Perhaps part of that. People can, and to surface that, you have, you know, time steps, right? It's like, you can step through time and see the ball accelerating. So each frame there, it's going down even more. The spring is accelerating or decelerating.


[User] Please transcribe this text

[Assistant] Yeah, I mean, people like Brett Victor have looked into these sorts of expanding time into space sorts of visualizations. So, this is all super interesting.


[User] Please transcribe this text

[Assistant] Uhm, to go into it for future work, though, there's a lot in programming, and I think, and I think even more broadly, uhm, Sean's point is, what about, like, 3D, like, like, physical world things like this, right? Like, this is even way harder, is there, do you do camera capture, do you, you know, do you have smart materials? So, those are really open. Uh, here, and then side.


[User] Please transcribe this text

[Assistant] With code and debug and also with chat, right? We have chat, uh, human annotated, you know, things about the code and using it to train machine learning or train, uh, program analysis and this tool. I think that's also an awesome other awesome direction. And the way, you know, as an HCI person, I don't want to put people out of work.


[User] Please transcribe this text

[Assistant] So the way I would diplomatically bridge that is to say that these tools can help triage and augment people even more. So imagine a Code Opticon thing where


[User] Please transcribe this text

[Assistant] Since the tutor can't help everybody at once, the simpler kinds of help


[User] Please transcribe this text

[Assistant] We can get you started with a triage with an Eliza-like thing that starts asking you how you're feeling about the weather, or, you know, how you're feeling about this part of the code. Did you look at the first line, second line? People who also were doing this were looking at this line of code. And as it gets conversation started, maybe the human can jump in.


[User] Please transcribe this text

[Assistant] To do the thoughts, you know, the more AI complete sort of thing. But yeaah, the data traces are definitely there and that's totally underexploited right now. And I would, it's another giant area I'd love to look into. And Sunday?


[User] Please transcribe this text

[Assistant] Yeaah, yeaah, we're going the other way, right? Like, given visualizing generative code, one immediate thing that comes up, the immediate next step adjacent there is to do active learning activities. So, the first step is seeing the visualization, you build the right model, and then you fade away that scaffold and you say,


[User] Please transcribe this text

[Assistant] Here's my code. Here's an incomplete diagram. Drag the parts that should be there in and we'll tell you if you're right. So you're turning the visualizations into an activity.


[User] Please transcribe this text

[Assistant] Could give you more active learning and engagement that way. So that's, that's one, you know, short-term thing. Longer-term thing is, you know, can you imagine learning a program with more visual paradigm rather than worrying about syntax? Which other, as you said, other people looked into as well.


[User] Please transcribe this text

[Assistant] Building this, you know, Stack Overflow-like Karma community leaderboard thing.


[User] Please transcribe this text

[Assistant] I think I want to draw on people's intrinsic motivation of, uh, you're on this site learning and wanting to help people, similar to why people are posting on the MOOC forum. I want to draw on that sort of motivation rather than make it into a more traditional paid microcast kind of crowdsourcing sort of situation.


[User] Please transcribe this text

[Assistant] And for the gamification thing, I think that's something to look into as well.


[User] Please transcribe this text

[Assistant] I'm, I'm a bit afraid of people, you know, I think communities that are geared around gamification, uh, you know, Alex Sacko will have issues with.


[User] Please transcribe this text

[Assistant] Exclusion and, and not being as friendly to beginners or people wanting to game it and, and game that way and such, uh, but.


[User] Please transcribe this text

[Assistant] Yeah, that's still very much in the future work section and I think we have to have some elements of that but I, uh, you know, I want to avoid money for now because I want to see how far can we push if we don't pay people because, you know, people voluntarily do this because it maybe helped them learn as well.


[User] Please transcribe this text

[Assistant] I don't have a great answer off the bat, but the way to, the way I approach it is, at first we want to at least get these common best practices, I mean, what's happening in lab, we want to at least get those online. And in the process, we can think about things like history or, or diagrams or, you know, multiple, you know, multiplexing to make it slightly better. Uhm, you know, the other part of that question is, you know,


[User] Please transcribe this text

[Assistant] What kind of undesirable things in the real world do we not want to replicate online and part of that


[User] Please transcribe this text

[Assistant] Is why I did, uh, kind of a lower bandwidth text sort of thing. So, uh, you know, if we had, let's say we had tried to really replicate the real world and have video for all the boxes instead of text, then I think that it would be much harder for people to use it because they feel more ashamed or more shy or so. So yeah, so there were decisions of, you know, both for scalability and also for


[User] Please transcribe this text

[Assistant] You know, wanting people to use that to actually step, step back from the real world. Awesome questions.


[User] Please transcribe this text

[Assistant] Perseus Club. I'm sorry, Mr. Martingale, I cannot be disturbed. No, I'm very sorry, but the membership committee is meeting.


[User] Please transcribe this text

[Assistant] It often reminds me of the interior of a whale's belly. It's only an intellectual association, of course.


[User] Please transcribe this text

[Assistant] Eternally hopeful for some small glittering flavor. Amazing place, this place here in the desert where the gems lie just a few inches below the surface. Free. Free for the taking. Were it not for certain unfortunate restrictions. Here, go ahead.


[User] Please transcribe this text

[Assistant] Heard a curious story the other day, Mr. Davis. It's about a young man, uh...


[User] Please transcribe this text

[Assistant] Hunter, who used to make his living around here as a guide until, uh, until he got into trouble. Kind of trouble.


[User] Please transcribe this text

[Assistant] Oh, a clerk that made a quick dash past a barrier and scooped up a bushel of a saw of diamonds. They're lying only a few inches under the sand.


[User] Please transcribe this text

[Assistant] How quickly our little trance tarnishes. How quickly a conch is turned heavy. How lovely in our arms.


[User] Please transcribe this text

[Assistant] As for the death of poor Dr. Hunter, he drank a little too much anyway, but I feel really sorry for Mademoiselle Renault. I just can't believe she killed him. There's no one who can prove differently. Oh, that a girl so beautiful should die.


[User] Please transcribe this text

[Assistant] Well I, I think as long as you're determined to, to sacrifice this great fortune and, and go back to diamond stud and help the girl I, I think I know a man who can arrange for us to make this trip, quietly.


[User] Please transcribe this text

[Assistant] They knew if they could make it to Chicago from Mississippi, from Alabama, from Arkansas, there was work here. Chicago was a gold mine if you lived in the South.


[User] Please transcribe this text

[Assistant] Black people come from the south, they definitely can't live with us.


[User] Please transcribe this text

[Assistant] Pushed to the worst neighborhoods. We're pushed to slums that others have moved out from.


[User] Please transcribe this text

[Assistant] Most people on the north side, the reason they don't know anything about the south side, because they've never been to the south side. Ever. Ever.


[User] Please transcribe this text

[Assistant] So it's easy to say, I don't worry about that. That's not in my backyard.


[User] Please transcribe this text

[Assistant] You see what's happening on the news. Boy shot, girl shot, hit shot on a playground. And you say to yourself, I have no idea what's going on.


[User] Please transcribe this text

[Assistant] Was the federal government and the Chicago Housing Authority pouring folks out that they didn't prepare. That they had for generations.


[User] Please transcribe this text

[Assistant] People that are poor are getting pushed into a smaller and smaller box as gentrification takes up more and more space.


[User] Please transcribe this text

[Assistant] Public housing had become so problematic in Chicago.


[User] Please transcribe this text

[Assistant] That many people just wanted it to go away. And in fact, that's what's happened.


[User] Please transcribe this text

[Assistant] Mademoiselle, Anne-Sanne, let the vigno. No, no, Sir Suzanne Raynaud.


[User] Please transcribe this text

[Assistant] Who is being held for the murder of Dr. Francis Kittredge Hunter, a diamond stud. Where did you ever learn how to...


[User] Please transcribe this text

[Assistant] You may be outside playing basketball hooping and somebody that you just passed the ball to might get shot in the head. I lost a good friend of mine. He was out just shooting jumpers and they came through and shot him 19 times, automatic.


[User] Please transcribe this text

[Assistant] It's very difficult to maintain peace and tranquility in the midst of confusion.


[User] Please transcribe this text

[Assistant] American women and children are living in a setting that is violent and demeaning and fearsome.


[User] Please transcribe this text

[Assistant] Drug commerce in Chicago is a very violent business, and it's settled not with writs and subpoenas, it's settled with guns. I think the murders in Chicago are mostly due to the drug trades in Chicago.


[User] Please transcribe this text

[Assistant] Hey, he has a good heart. All of our guys have good hearts. They just maybe haven't had the chances, the opportunities to support. And honestly, the love. And there's a lot of love in a room like this. And people who believe deeply in what he can accomplish, not just today, but for the next 10 years, for the west side, but also for the city. Alright. So proud. So proud. Hey, Josh. How you feel? You straight? Yep.


[User] Please transcribe this text

[Assistant] Versus James Corden, a late night television host who spends the majority of his day sitting, and the entire staff of The Late Late Show. Made up of all sorts of types of non-athletic people.


[User] Please transcribe this text

[Assistant] And the runners are taking their marks... In lane one, James Corden, late night host and self-proclaimed fastest man in the world. In lane two, Usain Bolt, nine-time gold medal winner and actual fastest man in the world.


[User] Please transcribe this text

[Assistant] In lane three, Owen Wilson, National Treasure.


[User] Please transcribe this text

[Assistant] To sniff out more clues, maybe dig up something we missed.


[User] Please transcribe this text

[Assistant] Alright, one more doggy pun out of you and I'm gonna have your nuts clipped. I hate to break it, dear horse. My sack's emptier than Santa's after Christmas.


[User] Please transcribe this text

[Assistant] Animals have a universal language, like Esperanto. But this one actually caught on.


[User] Please transcribe this text

[Assistant] Okay. I bet you see there's something about me that you don't know.


[User] Please transcribe this text

[Assistant] Dear participants, just in case if you wish to participate in Ma'am's question, you can unmute yourself and share the answer.


[User] Please transcribe this text

[Assistant] Dear participants, we've emailed the datasets to all of you. So we request all of you to kindly check the, uh, kindly check the email. In that, there's a dataset called as Day 7 Worksheet.


[User] Please transcribe this text

[Assistant] Day 7 worksheet, that is what man is using at present.


[User] Please transcribe this text

[Assistant] No ma'am, output sheet is not visible. The ma'am has to, uh, screen share once again.


[User] Please transcribe this text

[Assistant] Dear participants, if any of you have any further questions, you can unmute yourself and ask my question right now. Ma'am, there was one question that came from a participant. What we did previously was that one-tailed test.


[User] Please transcribe this text

[Assistant] Uh, if there are no further questions, uh, we can close, uh, the session for today, uh, before.


[User] Please transcribe this text

[Assistant] Can you give an example for a one tailed test?


[User] Please transcribe this text

[Assistant] Okay, I hope that question is clarified. Ma'am, we can continue, ma'am.


[User] Please transcribe this text

[Assistant] Okay, thank you ma'am. Uh, and on behalf of the entire organizing team, we wish to extend a bouquet of gratitude to Dr. Ajay Selsey, the resource person for this evening, for taking us through a wonderful session on the chi-square test. Thank you ma'am.


[User] Please transcribe this text

[Assistant] And dear participants, with that, we have come to the end of the national workshop on data analytics using this PSS. We have shared the feedback forms with all of you through email. We request all of you to respond to the feedback form.


[User] Please transcribe this text

[Assistant] Please provide us, uh, provide a feedback for the workshop that we have conducted so that we would be able to offer more such programs in the future and we would also be able to, uh, over, uh, improve ourselves if there are any shortcomings from, from our side.


[User] Please transcribe this text

[Assistant] We also wish to inform you that we would take 15 days to process your e-certificates. Once the e-certificates are ready, uh, we would be sharing that through, uh, we would be sharing the instructions to download the e-certificates through email.


[User] Please transcribe this text

[Assistant] With all of you, so we request you to kindly respond to the feedback form.


[User] Please transcribe this text

[Assistant] So with that note, I wish to propose the vote of thanks.


[User] Please transcribe this text

[Assistant] On the onset, I wish to thank the management of Kristo Jayanthi College for providing us with all the necessary resources and supporting all our initiatives in hosting this national level workshop.


[User] Please transcribe this text

[Assistant] Next, I would like to extend a bookie of gratitude to Professor Surendranath Reddy, Head, Department of Management, uh, Professor Ajay Abraham, Coordinator, Department of Management, for motivating us and for encouraging us to take forward, uh, this workshop. Thank you, Professors.


[User] Please transcribe this text

[Assistant] I extend a sincere bouquet of gratitude to all the resource person who have actively participated for the last seven days of this national level workshop. Thank you professors.


[User] Please transcribe this text

[Assistant] I also extend a bookie of gratitude to, uh, Professor Madhu Sudhanan R and Dr. Jisha Viji, uh, co-organizers of this national, uh, level workshop for rendering all the support and, uh, the guidance which was needed in executing this program. Uh, thank you, professors.


[User] Please transcribe this text

[Assistant] Finally, I extend a sincere gratitude to all the participants for actively taking part in this seven days national level workshop and be, uh, uh, active learners and asking your questions and clarifying your questions. We thank you professors.


[User] Please transcribe this text

[Assistant] So with that note, we've come to the end of the workshop for today. We would be sharing the recordings of the workshop with all of you once it is ready after editing.


[User] Please transcribe this text

[Assistant] Someone has also requested if we could share the presentations, uh, sharing of the presentations would be subjected to the, uh, uh, discretion of the resource people because it would be their intellectual property. However, we would be happy to share with you the recordings of all the sessions.


[User] Please transcribe this text

[Assistant] So with that note, thank you all. Uh, once again, a gentle reminder, we request all of you to kindly respond to the feedback forms which has been shared in an email.


[User] Please transcribe this text

[Assistant] We would be requiring 15 days to process the certificates and we would, once it is ready, we would be sharing that in an email. We will be sharing the instructions to download the certificates in the email.


[User] Please transcribe this text

[Assistant] So thank you all, wishing you all a wonderful evening and a wonderful season of learning ahead. Thank you.


[User] Please transcribe this text

[Assistant] Okay, sure, sure sir. We request all the participants to kindly turn on your cameras so that we could take a screenshot together for documentation.


[User] Please transcribe this text

[Assistant] So request all the participants to kindly please turn on your cameras and once you are ready please smile so that we could, uh, we could save the screenshot as a memory.


[User] Please transcribe this text

[Assistant] Looking forward for your active participation in the other activities and initiatives of the department. Thank you. Wishing you all a good evening.


[User] Please transcribe this text

[Assistant] And Elon said, darkness is merely the absence of light.


[User] Please transcribe this text

[Assistant] Look at the moon! It's a billion miles away! And he said, well no, it's actually under 250,000 miles away. And they said, oh Elon!


[User] Please transcribe this text

[Assistant] He was the youngest, he made about two days, and the shortest, and then he was this brilliant boy, and so people didn't really like him.


[User] Please transcribe this text

[Assistant] Not that he told me much about it, but he was picked on quite a bit.


[User] Please transcribe this text

[Assistant] And so, his social life was much less than my other two kids. And that's a typical nerd.


[User] Please transcribe this text

[Assistant] Anytime I had a question, my daughter Tosca would say, cool genius, boy.


[User] Please transcribe this text

[Assistant] My kids do funny things and I'm never too concerned about them because, you know, if Elon wanted to drop out of college, he could always go back.


[User] Please transcribe this text

[Assistant] You know, make a million or die tomorrow.


[User] Please transcribe this text

[Assistant] When Elon was going through his sad period, I was so sad I felt like I had a hole in my heart. And there's nothing you can do.


[User] Please transcribe this text

[Assistant] You just hurt so much. And I just didn't see him getting out of it. He was just so sad. And then the next thing I get this call saying,


[User] Please transcribe this text

[Assistant] For the first time, it was possible to drive over 200 miles and have performance that was directly comparable, competitive with what a gasoline car could do. And Tussel was the first company to


[User] Please transcribe this text

[Assistant] To take those principles and put that into practice and try it.


[User] Please transcribe this text

[Assistant] We were very passionate about trying to make sure that this car was going to just throw down the gauntlet on what the technology could do and really prove to the world that electric vehicles, you know, could be incredibly fast and could have incredibly long range.


[User] Please transcribe this text

[Assistant] There was kind of a sense of adventure, you know, doing these things for the first time and doing it in a really scrappy way. You know, we did some of the very first battery packs in my garage in Menlo Park before we actually were able to rent a real office.


[User] Please transcribe this text

[Assistant] You know, if they can't touch it, they can't drive it, it's not particularly real.


[User] Please transcribe this text

[Assistant] The challenge of converting a smart card to electric was doubly difficult because we couldn't find a smart card.


[User] Please transcribe this text

[Assistant] We didn't have much time at all. We knew that from the beginning and we kind of prepared for almost battle and you know, set up a war room in the shop.


[User] Please transcribe this text

[Assistant] And I can't convince them that they need to drill somewhere besides 200 feet from our house.


[User] Please transcribe this text

[Assistant] I'll show you where they wanted to put this location. One of the first places that they wanted to put it.


[User] Please transcribe this text

[Assistant] All of a sudden I just find new stakes out in the middle of my alfalfa field. I believe this, this stake represent their outer boundary of


[User] Please transcribe this text

[Assistant] Of their, uh, pad. Just guessing that it would be about 200 feet from our house. Which is awful close.


[User] Please transcribe this text

[Assistant] My dear Mrs. Winterbourne, allow me to tell you something.


[User] Please transcribe this text

[Assistant] Well, it's gone on long enough. I'm 25, and I have a right, a right, Mrs. Winterborne, to a very different life from this.


[User] Please transcribe this text

[Assistant] You are mistaken Lawrence in thinking that I intend to excuse myself. I merely wish to say this. We are at the beginning of our married life.


[User] Please transcribe this text

[Assistant] It all began here for us 25 years ago when my husband moved here. Then I moved here 18 years ago.


[User] Please transcribe this text

[Assistant] Orland and I, uh, were married in 1988. I was a pharmaceutical chemist for many years.


[User] Please transcribe this text

[Assistant] These are the two condensate and produced water tanks. The natural gas is immediately piped into a pipeline that goes across our fields and is sent off into the pipeline off to either Chicago or Los Angeles.


[User] Please transcribe this text

[Assistant] I have to say that living with this development has affected our lives in nearly every way imaginable.


[User] Please transcribe this text

[Assistant] You could never change, could you? A little while ago I thought you could. I thought you had. But you could never change.


[User] Please transcribe this text

[Assistant] This is, here are the reserve pits here.


[User] Please transcribe this text

[Assistant] Basically our first list of the chemicals that were being used was this very, very short


[User] Please transcribe this text

[Assistant] And, uh, an interesting list that EPA put together. It certainly wasn't comprehensive. We knew. We found out very rapidly that it was no small list. They don't tell you everything that's in a product. You may only get 5% of what's in that product.


[User] Please transcribe this text

[Assistant] And the rest of it is proprietary or they just don't give it. They don't have to.


[User] Please transcribe this text

[Assistant] In each fracking incident, they use approximately 1 million gallons of fluid. Each well can be fracked as much as 10 times. They're fracked at different depths as they come higher and higher toward the surface.


[User] Please transcribe this text

[Assistant] of something that you have taken from an evaporation fund and ask them to drink it.


[User] Please transcribe this text

[Assistant] She had been breastfeeding her daughter through the period when they were using the water that they were told was safe.


[User] Please transcribe this text

[Assistant] Laura was told her water was safe, but we found out later they never tested it for 2BE.


[User] Please transcribe this text

[Assistant] They waited till four years after the incident to go back to see if possibly they could find some. That was long gone.


[User] Please transcribe this text

[Assistant] She spoke to other people in her neighborhood. She began to see if anybody else was having the kind of health problem she had. And then others began telling me about people they heard about.


[User] Please transcribe this text

[Assistant] And as I sat there and looked out my window into my backyard, all I could think was, I'm getting involved in this. There's no way I can stay out of this.


[User] Please transcribe this text

[Assistant] And I'm sitting here with all of the right resources. These people need help.


[User] Please transcribe this text

[Assistant] What he's seeing, they have not been given the list of these chemicals that are being used. When someone comes to them with some of these look like clinical, ordinary disorders, they need to do a little questioning because it could very well be


[User] Please transcribe this text

[Assistant] That it is a chemical that they were exposed to and there may be a way to treat it, but they can't give proper treatment if they're not aware of what these people are exposed to.


[User] Please transcribe this text

[Assistant] Basically, we found that if you were to take all of the chemicals that are used in a particular state, always where you see the highest percentage of possible health effects, it's always skin, irritation, eye irritation, blistering, sinuses, asthma.


[User] Please transcribe this text

[Assistant] Coughing. And then this effect called sensitizing. Itchy skin. Burny skin.


[User] Please transcribe this text

[Assistant] Why would anyone think that something happening a couple miles down the road could possibly be causing this?


[User] Please transcribe this text

[Assistant] Health change in their body. You see blood disorders, what is called


[User] Please transcribe this text

[Assistant] Idiopathic hemorrhaging. So you get bloody eyes, bloody noses and also blood in the urine. And a number of people who called me said they had this condition.


[User] Please transcribe this text

[Assistant] Well, around 50% of the chemicals cause such things as kidney damage, cardiovascular problems, and then the next and very troublesome are the neurological effects.


[User] Please transcribe this text

[Assistant] Those chemicals again, neurotoxicants. People complain when they stepped out of their automobiles or out of their homes that they got a whiff of some air. They collapsed. They shook. Uh, they seem to have loss of memory.


[User] Please transcribe this text

[Assistant] We can make them do it better. The profits now in the industry are so high that there's no reason why they can't start using some new technology, develop the new technology.


[User] Please transcribe this text

[Assistant] Let's work on alternatives and let's, let's serve the country through alternative energy.


[User] Please transcribe this text

[Assistant] Morsi has seized all state powers and appointed himself Egypt's, quote, new Pharaoh.


[User] Please transcribe this text

[Assistant] Towards the end on creating new products for our riders and on the cancellations policy for the ride sharing product.


[User] Please transcribe this text

[Assistant] Uh, before that, uh, I was a PM at Postmates, uh, at a non-profit, uh, called Kiva, and then, uh, did some startup stuff for a while before that.


[User] Please transcribe this text

[Assistant] Uh, so I'm PM at Lyft, uh, I'm on the cancel side. Let's say cancels are up, uh, 5%. Obviously it's not a good thing. I want to see why that happened.


[User] Please transcribe this text

[Assistant] Uh, and then, uh, you have like a magical dashboard and I can ask you whatever question.


[User] Please transcribe this text

[Assistant] Uh, but you only give me like yes or no answers. I have to be pretty precise in like what things I ask you, right? Yes.


[User] Please transcribe this text

[Assistant] Cool, okay, I just make sure I understand how we're defining cancels. So first question, uhm.


[User] Please transcribe this text

[Assistant] Is this like, are we talking like rider versus driver cancels or like, is it like the admins, like the system behind the scenes cancelling?


[User] Please transcribe this text

[Assistant] I guess, yes or no questions. Is this driver, uh, cancels? Yes, it is driver cancels, yes.


[User] Please transcribe this text

[Assistant] Okay, so just to make sure our writer cancels, uh, different in any way, or is it just on the driver's side?


[User] Please transcribe this text

[Assistant] Okay, got it, got it. Uhm, has the way that we, uh, define the metric changed at all? Like as an example, like maybe the way we used to define it is, uh, you know, you requested the ride already.


[User] Please transcribe this text

[Assistant] Uh, and, uh, are we, like, logging it differently? Like, have we switched, like, you know, from, I don't know, Redshift, Hive, or like anything like that?


[User] Please transcribe this text

[Assistant] Uh, yeah, so I mean, maybe I'll try some other stuff for a while, and if we aren't really getting anywhere, then we can come back to, uh, instrumentation and see if there's like a technical problem with that. Uhm, sounds okay with you. Yeah, sounds good.


[User] Please transcribe this text

[Assistant] Uhm, maybe the metrics are being logged differently, but sounds like that's like...


[User] Please transcribe this text

[Assistant] I don't know, maybe it's like a routine thing that hasn't had much of an impact, but who knows, we can come back to it. Typically when I want to like, find the root cause behind this sort of thing,


[User] Please transcribe this text

[Assistant] I like to break it down into different types of like categories in which things might go wrong. A couple that come to mind right now and we can maybe think of more later.


[User] Please transcribe this text

[Assistant] Maybe there's some technical ways to slice and dice the data that will help us to find the segment that's experiencing this more, if such a thing exists.


[User] Please transcribe this text

[Assistant] Uh, maybe there's like user specific issues or like competitive issues. Those are maybe like the four, um, that I'll touch upon. Does that sound okay? Yep, that sounds great.


[User] Please transcribe this text

[Assistant] Yeah, sure. So let's start with Marketplace.


[User] Please transcribe this text

[Assistant] This is stuff like seasonality, like maybe time and location or something. So as a starting point, uhm, has this change, uh, did this, did this change happen, uh, like gradually, like decreased over time into what is now like.


[User] Please transcribe this text

[Assistant] So, cancels are up 5%, uh, from some point in the past, uh, was that point in the past, uh, like very recent, and it's like a stark change? Or has it been kind of gradually, uh, increasing up until where we are at now?


[User] Please transcribe this text

[Assistant] Yeah, so maybe like, there's like a specific city or like a state in America or something, but it's like kind of across the board. Right. Okay, that makes sense.


[User] Please transcribe this text

[Assistant] I guess you said it was a start change. Is there any like holiday or something? Like, uh, I dunno, is that like a big, like.


[User] Please transcribe this text

[Assistant] City specific, but it's like a national holiday, I guess, uhm, going on.


[User] Please transcribe this text

[Assistant] Uhm, yeaah, I mean, maybe like if you just had like a, you know, a magical calendar of uh, of holidays.


[User] Please transcribe this text

[Assistant] I'm going to skip forward to a different category. Let's talk about maybe, uh, technical, uh, you know, potential problems.


[User] Please transcribe this text

[Assistant] Uhm, is this, so I think lift only on iOS and Android, right? Maybe there's like a mobile product as well. Uhm, but like broadly is there like a specific platform in which we're seeing this happen more?


[User] Please transcribe this text

[Assistant] Okay, that makes sense. And, uh, we'll get into like user behavior type stuff later, but since you're talking about driver cancels, I think, uh, you know, probably not worth taking too much into like the passenger app, at least quite yet.


[User] Please transcribe this text

[Assistant] Uh, but on the driver app, uh, I think, I imagine there's like different versions of the app and stuff like that, right? Um, if it's the same across all the platforms, is there maybe like a, you know, like a version of the app that, uh, it's happening more on like some of the older versions, for example, or anything like that.


[User] Please transcribe this text

[Assistant] Uh, cool. And then I guess this is maybe like half technical, half user behavior stuff, but is there any like experiment that we've launched recently, like as lift, like, uh,


[User] Please transcribe this text

[Assistant] And or something that like maybe could have broken something and caused a bad experience that leads to more cancels or I guess like any new experience that we've launched.


[User] Please transcribe this text

[Assistant] Okay, interesting. Uhm, yeaah, I mean like intuitively doesn't feel to me like that would.


[User] Please transcribe this text

[Assistant] Uh, if nothing else, like fi- you know, yields like a bigger source of, uh, of problems, then, uh, maybe it'd be worth, like, doing some user research and asking drivers if the new color is, uh, somehow causing them to cancel more often, uh, if that sounds okay with you.


[User] Please transcribe this text

[Assistant] Uh, and just to like, kind of triple check here. So I think we've talked mostly about the example you gave at least was like a UI, uh, experiment, right?


[User] Please transcribe this text

[Assistant] Yeah, I guess broadly, uh, you know, back to the point that you brought up earlier, I guess we could just look like hypothetically, we could look at all the different experiments that are running and see if, uh, the varying versus the control or if it's like multivariate, like just compare all the different, uh,


[User] Please transcribe this text

[Assistant] Uh, one of the first things that I would want to think about is like the funnel of like the experience here, right? So we have, uhm,


[User] Please transcribe this text

[Assistant] Uh, set the destination, request a ride, they like, uh, basically like that's it, they can take to their destination.


[User] Please transcribe this text

[Assistant] Uhm, maybe I'll, I'll just like save it there and ask like.


[User] Please transcribe this text

[Assistant] Is there a specific step in that funnel from like app open, uhm, all the way to completing, uh, a ride, uh, that is, this is like disproportionately, uh, happening?


[User] Please transcribe this text

[Assistant] Uh, yeaah, I mean, yeaah, I guess like it could only happen from when the driver, uh, accepts the ride. Right. Right, right. Uhm, or I mean, I guess one clarifying thing here is, uh,


[User] Please transcribe this text

[Assistant] So, presumably the driver gets asked if they want a ride, uh, and they can, like, choose to accept it or not, right?


[User] Please transcribe this text

[Assistant] Okay, got it. Okay, let's get into this a bit more later.


[User] Please transcribe this text

[Assistant] Yeah, I guess I just want to close that loop on the connectivity side. Is the app crashing at all? That's basically what I'm getting at. Are we seeing app crashes higher than normal?


[User] Please transcribe this text

[Assistant] Okay, yeaah, that makes sense. Uh, cool. Okay, so now let's maybe go full on into the user side.


[User] Please transcribe this text

[Assistant] Uhm, so I just want to wrap up the funnel thing. So the driver accepted, uh, the ride. Uh, and this, from this point onwards, they, you know, might cancel.


[User] Please transcribe this text

[Assistant] Are we seeing a disproportionate amount of these, like, increased cancellations happening?


[User] Please transcribe this text

[Assistant] Uhm, from when you, uh, from you accept the ride until when you're like at the location, like picking up the passenger.


[User] Please transcribe this text

[Assistant] You might cancel like kind of early on because of something or later on because of something, right?


[User] Please transcribe this text

[Assistant] Are we seeing any like clustering of cancels, I guess is the question, uh, based on like the time between being, uh, dispatched to arriving at the pickup location.


[User] Please transcribe this text

[Assistant] Distance from the uh, place where the driver was dispatched.


[User] Please transcribe this text

[Assistant] And the destination that they're being, or the pickup location they're being asked to drive towards has that distance.


[User] Please transcribe this text

[Assistant] Uh, change meaningfully recently? Is it like, is it on average longer, for example?


[User] Please transcribe this text

[Assistant] Yeah, yeah, I guess I should take a step back for a second. I'm trying to think through like, uh, cause we're mostly focused on the driver. Like, is there anything from the driver's perspective that might?


[User] Please transcribe this text

[Assistant] Uhm, and to do that, they have to do a lot of jobs per hour. I think they're not getting paid hourly, right? So.


[User] Please transcribe this text

[Assistant] To do shorter pickups, right? So I guess the question is based on that, right? Uh, has the distance grown at all?


[User] Please transcribe this text

[Assistant] Ah, time, time as well, yeaah, yeaah. Okay, cool, yeaah, just wanna double check that. Uh, okay, cool, uhm, let's see, uhm,


[User] Please transcribe this text

[Assistant] I don't know if, we may, we may have already covered this, but something that comes to mind is like, are there more drivers lately? Like as a, you know, proportionately to the amount of riders, are we like oversupplied potentially? Like there's, you know, so many drivers that, uhm, I don't know, maybe they're frustrated because they're not getting as many rides or something like that.


[User] Please transcribe this text

[Assistant] Maybe one thing that comes to mind is like a more macro thing. Uhm, have you seen anything like in the news? Like any public sentiment? Like a strike?


[User] Please transcribe this text

[Assistant] Or like, uh, I don't know, like, delete Uber a while back seemed to be like a big thing for them. It's like something maybe similar, like bubbling up, uh, right now that we've seen.


[User] Please transcribe this text

[Assistant] Okay, that makes sense. Uh, and then maybe, uh, two, yeah, I think two more things that come to mind, uh, in users. And then I have some competitive stuff I want to ask about.