[User] Please transcribe this text

[Assistant] I mean, if you look at this, if I just try to take this over here, right, I couldn't get a light color very easily at all, could I?


[User] Please transcribe this text

[Assistant] That's why you want to come over, and that's true whether you're doing blue or anything or even a mix, like say you're tinting a secondary color, like you're tinting your purple, lightening it. You would want to take the purple to the white. That's a paint management strategy. Small amounts of your colorant to your tint, your white. It's easy to darken. I mean, just so easy to darken. Look at that. Hard to lighten.


[User] Please transcribe this text

[Assistant] Well, we are not going to do the more complicated using complementary colors to tone the paint too much. I'll show it to you a little bit when we get into secondaries. But toning in the course of what we mostly mean in color


[User] Please transcribe this text

[Assistant] Is it take a gray? So I'm gonna take a little bit of my black over to my white and make a mid gray.


[User] Please transcribe this text

[Assistant] There we go. A mid gray would be like one part white, one part black. It is easier, it's not easier. You could do this with a palette knife, but most people don't start out with a palette knife, so I'm demonstrating with a brush. So I've got a gray. I bring it over here. And I add that gray.


[User] Please transcribe this text

[Assistant] To the red and I have desaturated his value.


[User] Please transcribe this text

[Assistant] Come back and do like a little tint over here. We can show that.


[User] Please transcribe this text

[Assistant] Sometimes it'll happen. I'll get into my half inch angle.


[User] Please transcribe this text

[Assistant] Right, so we've got this nice little tint here, and then we can go down to some depth. As you can see, it's just easier to darken.


[User] Please transcribe this text

[Assistant] Knocked back the saturation of the color. Even if I take my red and I add a little gray directly into it, it does desaturate that color. Can you guys see that? That is less bright than the pure pigment.


[User] Please transcribe this text

[Assistant] I don't want to put the gray into it. Let's talk about shading.


[User] Please transcribe this text

[Assistant] And now the rule changes, cause the black, remember we talked about the dominant pigment, the higher tinting strength, right. So now I don't want to take the red over to the black, it would take so much red to impact the black. Right, so if I've got some red here, and I want to darken it, I want to shade it.


[User] Please transcribe this text

[Assistant] I bring the black over to the red. I barely brought any black, and did you see how much that shaded it? It darkened it.


[User] Please transcribe this text

[Assistant] To darken any color on my palette, so I've got to bring small amounts of the black to the less dominant color, to the less tinted color. The tinting strength is lower on red than black, so the black goes to the red. To recap, okay, if I'm trying to tint


[User] Please transcribe this text

[Assistant] Any color, I'm going to want to take that color to the white because it has a higher tinting strength. I would go from red to white or blue to white.


[User] Please transcribe this text

[Assistant] Or yellow to white. I wouldn't try to take the white to change this whole group of paint. That would go crazy on me.


[User] Please transcribe this text

[Assistant] But when I go to black, when I go to shade, I would want to take the black to any of the colors. Like if I'm going to take, uhm, like say I want to take a little black to shade my yellow.


[User] Please transcribe this text

[Assistant] Doesn't take very much black to shade that yellow. It would take a lot of yellow.


[User] Please transcribe this text

[Assistant] To shade the black. If I come back the opposite way over to the black with a little bit of yellow, that didn't do anything, did it?


[User] Please transcribe this text

[Assistant] I've got to bring the black to the yellow and work in small amounts so I can control the shading.


[User] Please transcribe this text

[Assistant] Tone to add gray. Tint to add white. These are just functions of lightening and darkening any of the colors that you're mixing on your palette. All that mean. Alright. Are we ready?


[User] Please transcribe this text

[Assistant] Making sure. Give John a second to check the chat. And then we'll move on to how to make secondary colors.


[User] Please transcribe this text

[Assistant] And you use what paint you have at home. Because you just need to know what your paint will do. Don't worry about what my paint will do. You gotta see what your paint will do.


[User] Please transcribe this text

[Assistant] Alright, secondary colors. We talked about it on the wheel a little bit. Let's put it into practice.


[User] Please transcribe this text

[Assistant] Now, one of the things, we don't want to have that hidden color in there, so we're gonna rinse our brush out thoroughly, aren't we?


[User] Please transcribe this text

[Assistant] Let's start with our first secondary color. I've got some red.


[User] Please transcribe this text

[Assistant] And I'm gonna take some yellow, about the same amounts, and I'm gonna mix them together.


[User] Please transcribe this text

[Assistant] Okay, no that's good. That's how we're gonna make orange. Is yellow and red. Now.


[User] Please transcribe this text

[Assistant] Not every red and every yellow makes the same hue of orange.


[User] Please transcribe this text

[Assistant] Sometimes they're desaturated because there's a hidden bias. Right?


[User] Please transcribe this text

[Assistant] So if you have two craft bottle paints, you've got a yellow and you've got a red and you mix them and you get an orange that's a little grayed out. Hidden bias in your craft paint. It's fine. You just need to know that's what's happening.


[User] Please transcribe this text

[Assistant] It's not that it's happening, it's that you don't know that it's happening or why.


[User] Please transcribe this text

[Assistant] There we go. Getting a little bit of that. So I'm just trying to get to the mid range.


[User] Please transcribe this text

[Assistant] See, I can get right back to it again and again and again and again. I've got to be able to make a mid range orange, which I can then. What? Can I tint it? I can. Can I tone it?


[User] Please transcribe this text

[Assistant] Green. Blue and yellow make green. Mostly.


[User] Please transcribe this text

[Assistant] Pretty much. If I were to do cad yellow and ultramarine blue, they have a hidden bias in there. Ultramarine is biased really red, and cad yellow has a warm bias to it too. So pretty much I'm getting an army green. But.


[User] Please transcribe this text

[Assistant] Now I'm gonna tell you something though, very important about green.


[User] Please transcribe this text

[Assistant] In general, I lighten my green with yellow first and then tint with white.


[User] Please transcribe this text

[Assistant] That's a thing that I do and we'll get into that on secondary colors and you'll see me mix and you'll see me talk about that all the time. Green is it's own weird world.


[User] Please transcribe this text

[Assistant] I yellow, I lighten red if I'm doing warm paintings with yellow too, before I lighten with white, cause it'll take it into the pink so fast. It's something you might not know.


[User] Please transcribe this text

[Assistant] Okay, I'm gonna tone, I mean, I'm toning with a gray.


[User] Please transcribe this text

[Assistant] There we go, pink tone shade, my secondary colors. All right, the hardest one.


[User] Please transcribe this text

[Assistant] So, if you have cad red medium in phthalo blue, you're gonna get a purple that is very gray. We have a black screen.


[User] Please transcribe this text

[Assistant] Red over. I'll take it up here. I'll take a little bit of my red over. And I'm gonna grab a little bit of my blue. And kind of mix them together one to one. And what I'm gonna get is my purple. It's not that...


[User] Please transcribe this text

[Assistant] It's not that I have, always have to have the best purple as purple. It's that I have to understand what my purple is. Right? I just need to know what my paints will do.


[User] Please transcribe this text

[Assistant] If you have a paint color that you just can't get, that's the one extra tube of paint you buy.


[User] Please transcribe this text

[Assistant] Alright, if you're on a budget and your primary colors don't give you a good green, then you buy a phthalo green.


[User] Please transcribe this text

[Assistant] If you're having primary colors and they don't give you good purple, then you buy your purple. If they don't give you good orange, you buy your orange.


[User] Please transcribe this text

[Assistant] You're gonna get two of the secondary colors good every time. Like in this set we get a good orange, we get a good green. We get a slightly desaturated, not as bright purple. If I wanted a very bright purple, that's the one tube I would buy. Hmm. But I'm gonna tint it. You can see it's pretty purpley. Purple enough.


[User] Please transcribe this text

[Assistant] For all the painting I'm gonna do through this series, it's more than purple enough. You know, and I can add the gray.


[User] Please transcribe this text

[Assistant] That's a lot of different colors that I can get. That's why I have a-


[User] Please transcribe this text

[Assistant] Color mixing recipe thing, 403 color recipes, cause this gets pretty wild fast. Alright, the last one we gotta do is gray.


[User] Please transcribe this text

[Assistant] Right, and it feels like, I think for beginners it feels like these should be


[User] Please transcribe this text

[Assistant] Okay, I'm gonna be weird here. When I was a new mom, I decided I would breastfeed. Not that you need to know this, but now you do. I decided that's how I wanted to feed my kid.


[User] Please transcribe this text

[Assistant] And uhm, I just assumed it would be easy and natural. Cause you know, it seemed like one of those things that should be easy and natural.


[User] Please transcribe this text

[Assistant] It was really hard. And I had to get a lactation consultant to help me.


[User] Please transcribe this text

[Assistant] You know, and nobody really had any like information they could give me. Sometimes in life there are things that seem like they should be the simplest thing in the world. Orange.


[User] Please transcribe this text

[Assistant] Green and purple seem like they should be the simplest thing in the world. Color mixing for primary seems like it should be the easiest thing in the world. Certainly felt easy when we were kids, didn't it?


[User] Please transcribe this text

[Assistant] Of course our pressure that we put on ourselves was a little bit lower than what she was an adult. So.


[User] Please transcribe this text

[Assistant] My weird analogy being sometimes things seem like they should be easy, they should be natural, but they require one, practice, and two, a little more information.


[User] Please transcribe this text

[Assistant] That's why I tell you these things, so you don't blame yourself when it's not your fault.


[User] Please transcribe this text

[Assistant] The only requirement is to feed your babies. Not how you feed your babies, I don't have to tell you. But that's what I would say.


[User] Please transcribe this text

[Assistant] Sometimes things are harder than they appear on the outset. So mixing gray is also one of those things that's harder than it seems on the outset.


[User] Please transcribe this text

[Assistant] One of the number one mistakes that beginners make is they try to lighten the black, they don't darken the white.


[User] Please transcribe this text

[Assistant] So what you do when you're trying to make grays is the first thing is take a small amount of black over to your white. And I'm gonna make a light gray. Oh, that's thin. I'm gonna take another little small amount of black over to my white and darken it again. Make a slightly darker gray.


[User] Please transcribe this text

[Assistant] And I can take some more white, uh, black over to my white and darken it again.


[User] Please transcribe this text

[Assistant] When you put out Tessa paint to see what your palette's actually mixing, sometimes it's hard to see a color until you get it out on the paper.


[User] Please transcribe this text

[Assistant] Most of your color classes will involve you doing this activity, which is swatching.


[User] Please transcribe this text

[Assistant] And you should swatch your colors. I have color chart videos and all of that because the number one thing you can do to save money is know what colors your paint already mixed. You don't buy paint, you don't eat.


[User] Please transcribe this text

[Assistant] That, uh, that dioxazine purple and phthalo green made purple if you've been painting with me for a while. You just didn't know you knew it.


[User] Please transcribe this text

[Assistant] They make a purple every bit as good as that. A blue every bit as good as, I mean, blue. Every bit as good as any of the blues that I could get out there.


[User] Please transcribe this text

[Assistant] Okay, are you guys ready? We're gonna make tertiary colors. We're almost done. We've learned a lot today. It was a big video. Alright, let's make some tertiary colors.


[User] Please transcribe this text

[Assistant] Now, I could, as a person, uh, go, okay, I'm gonna make yellow orange, so I need to make orange right, and then keep adding yellow to it.


[User] Please transcribe this text

[Assistant] So, what's my higher tinting strength? My red is a higher tinting strength than my yellow. Okay, I'll just put a little yellow over here to the side so I have it.


[User] Please transcribe this text

[Assistant] So that's my stronger tending string. That means that this


[User] Please transcribe this text

[Assistant] I bring over in small amounts to my yellow. I brought just a scotch over. I don't try to go from orange to yellow orange.


[User] Please transcribe this text

[Assistant] I take just a small amount of red over to my yellow, and I tint in increments.


[User] Please transcribe this text

[Assistant] If I'm trying to get to yellow orange, that's how I would get there. It's very easy. Did we put the yellow orange thing up? I think we did, yes. Okay. Now let's do red orange.


[User] Please transcribe this text

[Assistant] Now I can pull out a little of my red so you can see it.


[User] Please transcribe this text

[Assistant] I'm gonna take just a smidge of my yellow over to my red. And you're gonna see that that just kind of goes a little bit


[User] Please transcribe this text

[Assistant] Yellow, red. All it does is warm up my red.


[User] Please transcribe this text

[Assistant] Now if I'm trying to get to like more of a cad red color, or brighter red, I would do that.


[User] Please transcribe this text

[Assistant] Right. And I can add just a little bit more yellow.


[User] Please transcribe this text

[Assistant] I'm just slightly darker than the orange I had. So let's put them out on the palette so we can see them. Practice with me. Let's make a mid orange, a yellow orange, and a red orange now. Put it into practice.


[User] Please transcribe this text

[Assistant] So we can see where it is. It's red orange, mid orange, yellow orange.


[User] Please transcribe this text

[Assistant] And if you add tinting and toning and shading to that and the fact that you can mix complementary colors, wow.


[User] Please transcribe this text

[Assistant] You've got all the colors in the rainbow that you possibly need to do a painting.


[User] Please transcribe this text

[Assistant] Even with student paints, these are student, right? These are the, these are the artist grade kind of student price paints.


[User] Please transcribe this text

[Assistant] Dude, we're here. We're not struggling. We're not suffering, are we? No, we're not suffering. Shall we make? Ooh, let's get into the.


[User] Please transcribe this text

[Assistant] Alright, so what has the higher tinting strength over the red is the phthalo blue. Right, the blue has more tinting strength than the red. So if I put out some red.


[User] Please transcribe this text

[Assistant] I'm gonna want to bring out the blue in smaller, like incremental amounts. Right. And that is going to give me a purple.


[User] Please transcribe this text

[Assistant] I get a blue violet, or a blue violet, sorry.


[User] Please transcribe this text

[Assistant] Right, so that's blue violet when I add more blue to the mix. Where was that on the color wheel? Here we are.


[User] Please transcribe this text

[Assistant] Real practice, I added more paint, I got to my red violet.


[User] Please transcribe this text

[Assistant] Right, and at the end I'm gonna show you how you can do some cool things here. You're gonna love this.


[User] Please transcribe this text

[Assistant] It's where it all comes together. I love it where we can bring it all together for the students where they're like, oh wait, this is the thing. I'm gonna take it back to a mid purple.


[User] Please transcribe this text

[Assistant] And I think I'll make the mid green this time, just so I have it in the center. So we're gonna make a mid green. Just our regular green, right? That's where we mix the blue and yellow together to make a mid green.


[User] Please transcribe this text

[Assistant] With more yellow in it. Since the green, since the blue has higher tinting strength than yellow, I want to bring blue to the yellow, don't I? I don't bring yellow to the blue, I bring blue to the yellow. Is that process making sense to you guys? In your color mixing, um, palette management?


[User] Please transcribe this text

[Assistant] How you pick the tinting strength of the color to determine what color you move to the other to mix. I don't think I've ever taught this before, and I feel like that's a thing that you guys need to know. How do I mix the color? When you guys say how do I mix the color?


[User] Please transcribe this text

[Assistant] The, the, there's the question that you know, which is like I just want to get to this color, but there's the thing that you don't know, which is how you choose to take a color to another color. Right, cause look at my yellow already. It's got like so much tinting on it, doesn't it?


[User] Please transcribe this text

[Assistant] That got red in it, and that's gonna gray it, isn't it? Cause my paint at this point. This is why I do the landing strip.


[User] Please transcribe this text

[Assistant] I've got my mid green. Now I can very easily darken my green with blue.


[User] Please transcribe this text

[Assistant] And can get into a blue green. I think this is the strongest set of colors that it makes in this set. Just does really well in the green. Now.


[User] Please transcribe this text

[Assistant] Let's just real quick look at what we know from our color wheel and what we've mixed. Okay, so we know.


[User] Please transcribe this text

[Assistant] This is kind of like our recap. We know about compliments, don't we?


[User] Please transcribe this text

[Assistant] Taking a little yellow out. I want some pure yellow. There's some pure yellow. Okay. What's the compliment?


[User] Please transcribe this text

[Assistant] Get a good one going here. There's complementary colors.


[User] Please transcribe this text

[Assistant] So I didn't have the compliment to yellow in my primary set, but I can mix it.


[User] Please transcribe this text

[Assistant] So if I have blue out and I want to do a complimentary color painting.


[User] Please transcribe this text

[Assistant] If I wanted to have split compliments to something, well gosh, it's a red orange and a yellow orange, isn't it?


[User] Please transcribe this text

[Assistant] So what it is, is I just cover a topic really in depth for a year. The first time we did it was just try to explain everything you needed to know in the art store and just be more confident.


[User] Please transcribe this text

[Assistant] Uhm, and then, and then we started doing subjects, we did faces, we did, uh, fairy tales, uh, this year we're doing dogs.


[User] Please transcribe this text

[Assistant] We just do a lot of lessons based on that theme and there's a group for it. And it's just about that more deep dive into our learning process. So whenever you see BAIQ, it's a big art quest.


[User] Please transcribe this text

[Assistant] This actually counts as part of the big art quest for my big art quest students cause we're almost rebooting the original.


[User] Please transcribe this text

[Assistant] Not use gray to tone it. Like say if I had that, I could come over with this orange into my green. And tone it.


[User] Please transcribe this text

[Assistant] Good night. Just like my color wheel said. My color wheel didn't lie. See. Let me see if I can get a little more. You guys can really, really see it.


[User] Please transcribe this text

[Assistant] My color wheel did not lie as the color.


[User] Please transcribe this text

[Assistant] This is it. This is where it is. It's one of those things that seems simple when you just sit down, but if nobody tells you this stuff, how could you possibly know it?


[User] Please transcribe this text

[Assistant] Right. This isn't, it's like recipe. It's not something you just naturally could do. Right. Some things.


[User] Please transcribe this text

[Assistant] require a little assistance and mentorship and help.


[User] Please transcribe this text

[Assistant] And color theory is one of those things, especially your first toe, your first dip of your toe into the subject.


[User] Please transcribe this text

[Assistant] Uhm, you know, you need a little help, a little bridge to get you across. Now, if you get this skill set down,


[User] Please transcribe this text

[Assistant] All the other color theory classes will become easier and easier and easier sequentially as you go.


[User] Please transcribe this text

[Assistant] You get these first seven skills down from my videos. Everything else will flow in. Right?


[User] Please transcribe this text

[Assistant] And it's okay if it takes you a second to get it. It's called color theory because it's a, it's a, it's a force of study.


[User] Please transcribe this text

[Assistant] Right? Uhm, if you love this, I'm on black. Remember?


[User] Please transcribe this text

[Assistant] I'm gonna. All right. So guys, yeah, that's my color beginning class, right?


[User] Please transcribe this text

[Assistant] It's the seven videos plus ten plus the after party. And now you know why you deserve a certificate.


[User] Please transcribe this text

[Assistant] Because really honestly, if you do all this, you knew something. You literally just know.


[User] Please transcribe this text

[Assistant] You know more than some people who went to art school, to be real honest. Because I don't always think to, you know, do some of this in every single course, right? Not all people. Really art school covers a lot of the stuff, but I have


[User] Please transcribe this text

[Assistant] I have run into where this isn't necessarily core, so, but mostly it is. But you know those things that you have to know.


[User] Please transcribe this text

[Assistant] Do the courses. Or just take this one. If you just needed some color theory, this definitely will get you anywhere you need to go. And all the terms that people just so casually use in their beginner acrylic courses, or their beginning oil courses, or their beginner watercolor courses. All those terms that us teachers just so casually use. Knock it back. Tone it down. Suddenly have context.


[User] Please transcribe this text

[Assistant] We want to spray out those colors, right? What do the words mean? What do the colors actually do? For real. And why does it not work?


[User] Please transcribe this text

[Assistant] Alright guys, do you know what's next? Let's talk about what's next. So excited about what's next.


[User] Please transcribe this text

[Assistant] So you've done the seven, the super seven, the sweet seven, and you're ready to do your first ten paintings.


[User] Please transcribe this text

[Assistant] Ready to do them. Do you have the preview?


[User] Please transcribe this text

[Assistant] I wanna tell you something about the first ten paintings and then John's gonna hit play.


[User] Please transcribe this text

[Assistant] I've taught you concepts. I've taught you concepts about medium.


[User] Please transcribe this text

[Assistant] I've taught you concepts about studio and tools. I've taught you concepts about color and technique.


[User] Please transcribe this text

[Assistant] Right? In the beginner acrylic painting course. All those concepts are there.


[User] Please transcribe this text

[Assistant] Now, like I said, there's information, but it's not knowledge until you put it into practice.


[User] Please transcribe this text

[Assistant] That's when it becomes true knowledge for you. Something you can use more than conceptually. So.


[User] Please transcribe this text

[Assistant] No longer fantasy table artist, we're gonna go be artist artists and we're gonna do ten paintings and I've designed these ten paintings to have a series of skills that you've been introduced to and each skill, each painting


[User] Please transcribe this text

[Assistant] And I'm kind of famous for this in Acrylic April. Each painting teaches a sequential building block skill that leads into the next. So your first painting is going to be Happy Grumpy Cat. These are two cats, they're really funny.


[User] Please transcribe this text

[Assistant] We are using just the materials that we talked about. Right? These are just the materials.


[User] Please transcribe this text

[Assistant] For this course, it's just the primary colors, just the brushes, just the nine by twelve. Shall we show them? That's what you're gonna do tomorrow at one. This, right here.


[User] Please transcribe this text

[Assistant] I wonder if you can from this recognize some of the skills that the previous classes have demonstrated to you. Can you see them? I see all kinds of sketching in, drawing, mixing, I saw some tinting, right?


[User] Please transcribe this text

[Assistant] Uhm, form. Ah, layering. Got some layering. Sketching again, drawing with paint and chalk. Right, here we're going in, we're layering, using the light color to make up for transparency where we need to.


[User] Please transcribe this text

[Assistant] I'm gonna have a traceable, but I'm gonna show you how to draw these guys in. They're so super cute, the grumpy cats.


[User] Please transcribe this text

[Assistant] Right? You've gotta come tomorrow and see if you can put all this into practice with your little 9x12 canvas and your brushes. I think that you can.


[User] Please transcribe this text

[Assistant] I hope that you can, and we're gonna show this at the beginning with the materials too of the video so that if that fast view helps you


[User] Please transcribe this text

[Assistant] Which I've gotten feedback this year very, very much that it does. Uhm, you'll have that and then you'll have the step by step of the lessons. There's a mini book for this.


[User] Please transcribe this text

[Assistant] So if you've got, if you just need to print something out to kind of go over the concepts again, we have it.


[User] Please transcribe this text

[Assistant] Rewatch any time it's time stamped. If you're like, what was that golden thing? Go hit those, go do the chapter thing and click it. It'll take you right to that chapter. Like, comment and subscribe because we work really hard at teaching art.


[User] Please transcribe this text

[Assistant] We try to be, we try really really hard cause your experience matters. If you had some questions that I didn't get to in the show, we're gonna do a quick Q&A over on Facebook, on the Facebook page. Uhm, that's very informal if you are not here for the live show and we didn't get your questions.


[User] Please transcribe this text

[Assistant] Or the live didn't answer your questions. Or you just, you know, we're in Australia and this is 3am.


[User] Please transcribe this text

[Assistant] Leave a question in the comments below and I get to it. I check my channel all the time.


[User] Please transcribe this text

[Assistant] I couldn't do it without badrons. And they really put up with the light.


[User] Please transcribe this text

[Assistant] They do. The patrons are here, like, to be part of the mission. Cause, you know, that's how we do it.


[User] Please transcribe this text

[Assistant] Without super chats and emoji clubs and patrons. You can join the patronage on the, on the, on the website.


[User] Please transcribe this text

[Assistant] We have it set up where you just set your own patronage cause everybody's in, you know what? The future is so uncertain. You guys figure out what's in a budget that does not stress you out.


[User] Please transcribe this text

[Assistant] Whatever that is. Old Medici style patron.


[User] Please transcribe this text

[Assistant] And then I'll see you. So you guys, you, walk dogs, pet cat, take a rest and break. We're gonna make some coffee and then we'll go over at Facebook and kind of talk about what we learned today.


[User] Please transcribe this text

[Assistant] Hi everybody, welcome to Overboard, it's Polygon's board game show.


[User] Please transcribe this text

[Assistant] This week we are playing Fauna. It's a game about guessing things about animals, uh, and we are all experts, especially our two guests from the Dodo, Eli, and Matt.


[User] Please transcribe this text

[Assistant] Uh, so that's back around to me and I've already, I'm out so I'm not guessing anymore.


[User] Please transcribe this text

[Assistant] The Indonesian Archipelago.


[User] Please transcribe this text

[Assistant] Micronesia. Uh. Yes. The Northern Indian Ocean. Nice. Uh.


[User] Please transcribe this text

[Assistant] One foot to one foot three inches. The weight is two to two point six lb.


[User] Please transcribe this text

[Assistant] Alright, I'm going to guess that one of these fellers is looking at him. Just one.


[User] Please transcribe this text

[Assistant] The thing with birds is they are always way fucking bigger than you think. That's true to you. I'm gonna say.


[User] Please transcribe this text

[Assistant] If you grabbed each end of this guy and stretched him out, he'd be three to six feet.


[User] Please transcribe this text

[Assistant] The red-bellied piranha. Three areas, uh, weight, and total length.


[User] Please transcribe this text

[Assistant] Ribbit. So we, okay, on frog we would not be including the legs, we would just be like nose to frog butt. That's a, that is a good point, cause, yea.


[User] Please transcribe this text

[Assistant] You're talking frog ass to frog nose? I'm talking frog ass to frog nose.


[User] Please transcribe this text

[Assistant] No, don't do that right now until after. I'm going to...


[User] Please transcribe this text

[Assistant] I'm gonna say, I'm gonna go back to my friend the Amazon Basin.


[User] Please transcribe this text

[Assistant] And this has been Fauna. If you'd like to play it, you can't, cause it's not in print anymore. This has been Overboard. Thank you very much to Eli and Matt for joining us from the Dodo. Tell us a bit about the Dodo.


[User] Please transcribe this text

[Assistant] And nothing can be done about this escaped slave who murdered my father.


[User] Please transcribe this text

[Assistant] Why must we offer tribute to a man like that? What do we have an army for?


[User] Please transcribe this text

[Assistant] You were my father's counselor. So you will be to me. But it is I who will rule.


[User] Please transcribe this text

[Assistant] I will go to the horse fair at Naharina, but not to enjoy myself.


[User] Please transcribe this text

[Assistant] Thus, I make amongst those men of the desert, unknown to them.


[User] Please transcribe this text

[Assistant] I have often heard the songs of the desert, but they have never been so beautiful. I do not think I shall mind my bandage.


[User] Please transcribe this text

[Assistant] And what if they should happen on it by chance? They have so many men.


[User] Please transcribe this text

[Assistant] I can give no thought to such dreams as are made of moonlight, while slaves die in chains.


[User] Please transcribe this text

[Assistant] I could be useful, more useful than you realize. Once I lived in the palace at Chemis, I was a dancing girl. I could return there.


[User] Please transcribe this text

[Assistant] Would you send another to enjoy the moments we might spend together?


[User] Please transcribe this text

[Assistant] There is no proof that it was he. Only a banner flying over the Royal Chariot. A piece of colour cloth.


[User] Please transcribe this text

[Assistant] If I were a slave girl, or the daughter of a merchant, I could go to him. But I am a queen. And he's a rebel.


[User] Please transcribe this text

[Assistant] And because I am a queen, I must condemn the one man in my life.


[User] Please transcribe this text

[Assistant] There is a mountain pass. It is far away and difficult to find.


[User] Please transcribe this text

[Assistant] Yes, perhaps it is the will of the gods that I destroy the murderer of my father.


[User] Please transcribe this text

[Assistant] At the top is a black hole. On it, the stronghold of the slaves.


[User] Please transcribe this text

[Assistant] No, his only defense is the hidden location of the village. I will lead you.


[User] Please transcribe this text

[Assistant] No, I'm sorry. We're looking for Coalesce. C-O-A-L-E-S-C-E. Alright. Oh, Coalesce. Oh, wow. Alright. Alright, Jimmy's word here.


[User] Please transcribe this text

[Assistant] Jamal, Jamal, Jimmy is still up one to nothing. Poi, Jay, not.


[User] Please transcribe this text

[Assistant] They're gone. Every danged one of them. They've moved away somewhere. Grant, if you ride with that arm, you'll lose it.


[User] Please transcribe this text

[Assistant] I swear from the bottom of my heart, I never felt more like quitting my job than right now. Because I well know what this case may cost a family. A sorrowing mother and sister who have already lost one son, one brother. If my guess is right, may lose another to the law.


[User] Please transcribe this text

[Assistant] Yes, only she forgot she was married to a Callum. Bring a horse here, get me a rope.


[User] Please transcribe this text

[Assistant] Tomorrow I'm coming back. You're coming with me. If he tries to stop me, I'll kill him.


[User] Please transcribe this text

[Assistant] Take me out to the prairie and lay the sod over.


[User] Please transcribe this text

[Assistant] So now let's turn to finding the inverse for matrices that are larger than 2x2.


[User] Please transcribe this text

[Assistant] And I want to quickly point out something. Remember how we're trying to find an inverse, and an inverse was sort of two-sided? It was A A inverse is the same thing as A inverse A is equal to an identity matrix?


[User] Please transcribe this text

[Assistant] However, we know that matrix multiplication doesn't work for arbitrarily sized matrices. The matrices have to have the right sides, right? Like if I'm multiplying two matrices together, there has to be this matching of these sort of inner dimensions. But then,


[User] Please transcribe this text

[Assistant] If I'm also multiplying on the other side, there has to be a matching of the inner two dimensions here, which originally were the outer two dimensions.


[User] Please transcribe this text

[Assistant] So in other words, what I'm really saying is that my matrices have to be square matrices in order for this to even make sense. And if they're not square, then one of these two multiplications isn't gonna make any sense.


[User] Please transcribe this text

[Assistant] Alright, so that was okay, but we're gonna need a little bit more than that to figure out that it's invertible. N by N is gonna be a necessary condition, but not sufficient. So, I'm gonna do something that might at first seem a little bit strange. I want you to start with that matrix A.


[User] Please transcribe this text

[Assistant] And I'm going to imagine that we've got some, some particular matrix, say it's 3x3. And then what I'm going to do is kind of weird. I'm going to


[User] Please transcribe this text

[Assistant] And then I'm gonna append the identity matrix, which is another three by three, then, then what I'm gonna get is three rows and six columns. Okay. Now, let me suppose I take this.


[User] Please transcribe this text

[Assistant] And it's a big matrix, and I try to put it in its R-R-E-F form.


[User] Please transcribe this text

[Assistant] Is going to transform into being the identity matrix. As in, the RREF, the reduced row echelon form of an invertible matrix, is going to be the identity matrix. That's an equivalent property.


[User] Please transcribe this text

[Assistant] That's what happens to the A matrix. The A matrix turns into the identity. But what happens to that original identity matrix? So if I think about what my...


[User] Please transcribe this text

[Assistant] The process of putting it into RREF is I'm doing a bunch of elementary row operations.


[User] Please transcribe this text

[Assistant] So for example, I might be multiplying a row by two, and then interchanging two rows, a bunch of things like that.


[User] Please transcribe this text

[Assistant] However, we saw previously that elementary matrices were the process of


[User] Please transcribe this text

[Assistant] Taking some, uh, identity matrix as your starting point, doing a row operation to the identity matrix, and seeing what you get out of it.


[User] Please transcribe this text

[Assistant] So, as I put this into RREF, I'm starting with the identity matrix, and then I'm doing an elementary row operation to the identity matrix, and another row operation, and another row operation. I keep on transforming the identity matrix.


[User] Please transcribe this text

[Assistant] And we know eventually that it's gonna become the identity matrix, but by doing a whole bunch of row operations, and doing a whole bunch of row operations is like multiplying by elementary matrices. So like, for example, maybe I multiply by a first one, E1,


[User] Please transcribe this text

[Assistant] A whole bunch of matrices all multiplied together, this is a matrix, I'll call it now for A inverse, but it's some matrix where that matrix times A is the identity. That is going to be the inverse of the matrix A.


[User] Please transcribe this text

[Assistant] Well it's not the check, it works the other way around, but it will indeed do this. So, what have we deduced?


[User] Please transcribe this text

[Assistant] This identity matrix, which is just going to be starting the identity and doing EROs to it, is just this product of elementary matrices. In other words, what I get out of here, this is my triumphant conclusion,


[User] Please transcribe this text

[Assistant] And I haven't put in the closing bracket because I'm going to append to it the identity. So 1's down the main diagonal, 0's everywhere else.


[User] Please transcribe this text

[Assistant] And then, what I want to do here is, I want to do elementary row operations to the A matrix, and turn it into the identity, and then, the identity matrix that I have here, that's gonna transform as well under these row operations, and then, whatever it transforms into, will be my inverse.


[User] Please transcribe this text

[Assistant] Alright, so I've done a couple elementary row operations in this step, and what you might notice is that my A part, the portion on the left here, it's not the identity matrix yet, but it's getting a little bit cleaner.


[User] Please transcribe this text

[Assistant] And the stuff on the right hand side, which started with identity, is getting a little bit messier. So let's just keep on going and see what we get.


[User] Please transcribe this text

[Assistant] I'm going further than just doing it into REF form because I wanted to have the zeros above it. I'm going all the way to the reduced row echelon form. So I'm going a little bit further than sometimes we might do. One step left.


[User] Please transcribe this text

[Assistant] Alright, and then correcting this with a quick typo I found up here gives me this final form where


[User] Please transcribe this text

[Assistant] On the left I have the identity matrix, right? One's all the way down. And therefore, I can say that this matrix that I have here,


[User] Please transcribe this text

[Assistant] Turn. As the world turns? As the world, the life cycle, uhm.


[User] Please transcribe this text

[Assistant] Going green. Turning.


[User] Please transcribe this text

[Assistant] Billy, Charlie called as soon as he got out of the car. I turned toward the house, beckoning to Jacob as I ducked under the porch. I heard Charlie greeting them loudly behind me.


[User] Please transcribe this text

[Assistant] We get permits early on the res, Jacob said while I unlocked the door and flicked on the porch light. Sure you do, Charlie laughed. I have to get around somehow.


[User] Please transcribe this text

[Assistant] Watching anxiously as Charlie and Jacob help Billy out of the car and into his wheelchair.


[User] Please transcribe this text

[Assistant] I backed out of the way as the three of them hurried in, shaking off the rain. This is a surprise, Charlie was saying. It's been too long, Billy answered. I hope it's not a bad time.


[User] Please transcribe this text

[Assistant] His dark eyes flashed up to me again, their expression unreadable.


[User] Please transcribe this text

[Assistant] No, it's great. I hope you can stay for the game. Jacob grinned. I think that's the plan. Our TV broke last week. Billy made a face at his son.


[User] Please transcribe this text

[Assistant] And of course, Jacob was anxious to see Bella again, he added.


[User] Please transcribe this text

[Assistant] Nah, we ate just before we came, Jacob answered.


[User] Please transcribe this text

[Assistant] How about you, Charlie? I called over my shoulder as I fled around the corner. Sure, he replied, his voice moving in the direction of the front room in the TV. I could hear Billy's chair follow. The grilled cheese sandwiches were in the frying pan, and I was slicing up a tomato when I sensed someone behind me. So, how are things? Jacob asked. Pretty good, I smiled. His enthusiasm was hard to resist. How about you? Did you finish your car?


[User] Please transcribe this text

[Assistant] No, he frowned, I still need parts. We borrowed that one. He pointed with his thumb in the direction of the front yard. Sorry, I haven't seen any, what was it you were looking for? Master cylinder, he grinned. Is something wrong with the truck? He added suddenly. No? Oh, I just wondered because you weren't driving it. I stared down at the pan, pulling up the edge of a sandwich to check the bottom side.


[User] Please transcribe this text

[Assistant] I nodded non-committally, keeping my eyes down as I flipped sandwiches. My dad seemed to know him from somewhere.


[User] Please transcribe this text

[Assistant] Jacob, could you hand me some plates? They're in the cupboard over the sink. Sure. He got the plates in silence. I hoped he would let it drop now.


[User] Please transcribe this text

[Assistant] So who was it, he asked, setting two plates on the counter next to me. I sighed in defeat. Edward Cullen? To my surprise, he laughed. I glanced up at him. He looked a little embarrassed. Guess that explains it then, he said. I wondered why my dad was acting so strange. That's right. I faked an innocent expression. He doesn't like the Cullens.


[User] Please transcribe this text

[Assistant] I was really listening to the men's conversation, watching for any sign that Billy was about to rat me out, trying to think of ways to stop him if he began. It was a long night. I had a lot of homework that was going undone, but I was afraid to leave Billy alone with Charlie. Finally, the game ended.


[User] Please transcribe this text

[Assistant] I actually smoked in the evidence room I was totally controlled by my addiction.


[User] Please transcribe this text

[Assistant] All these convictions needed to be wiped away. When you're investigating somebody who's tampered with evidence, the scope of this could be very large.


[User] Please transcribe this text

[Assistant] The whole idea was to keep this as small as possible. Billions of dollars flood into this system to perpetuate this wrong drug.


[User] Please transcribe this text

[Assistant] The sun has fled. We've lost best half of our affair.


[User] Please transcribe this text

[Assistant] Well, that's a way. And say how much is done.


[User] Please transcribe this text

[Assistant] Chapter 10 Interrogations


[User] Please transcribe this text

[Assistant] It was very hard in the morning to argue with the part of me that was sure last night was a dream. Logic wasn't on my side or common sense. I clung to the parts I couldn't have imagined, like his smell. I was sure I could never have dreamed that up on my own. It was foggy and dark outside my window, absolutely perfect. He had no reason not to be in school today. I dressed in my heavy clothes, remembering I didn't have a jacket.


[User] Please transcribe this text

[Assistant] I swallowed a granola bar in three bites, chased it down with milk straight from the carton, and then hurried out the door. Hopefully the rain would hold off until I could find Jessica. It was unusually foggy. The air was almost smoky with it.


[User] Please transcribe this text

[Assistant] The mist was ice cold where it clung to the exposed skin on my face and neck. I couldn't wait to get the heat going in my truck.


[User] Please transcribe this text

[Assistant] It was such a thick fog that I was a few feet down the driveway before I realized there was a car in it, a silver car. My heart thudded, stuttered, and then picked up again in double time. I didn't see where he came from, but suddenly he was there pulling the door open for me. Do you want to ride with me today? He asked.


[User] Please transcribe this text

[Assistant] I noticed his tan jacket was slung over the headrest of the passenger seat. The door closed behind me.


[User] Please transcribe this text

[Assistant] Just a light gray knit v-neck shirt with long sleeves.


[User] Please transcribe this text

[Assistant] Again the fabric clung to his perfectly muscled chest.


[User] Please transcribe this text

[Assistant] It was a colossal tribute to his face that it kept my eyes away from his body. I'm not quite that delicate, I said, but I pulled the jacket onto my lap, pushing my arms through the two long sleeves.


[User] Please transcribe this text

[Assistant] Curious to see if the scent could possibly be as good as I remembered. It was better. Aren't you? He contradicted in a voice so low, I wasn't sure if he meant for me to hear. We drove through the fog shrouded streets, always too fast, feeling awkward. I was at least. Last night all the walls were down, almost all. I didn't know if we were still being escanded today. It left me tongue tied. I waited for him to speak.


[User] Please transcribe this text

[Assistant] He turned to smirk at me. What, no 20 questions today? Do my questions bother you, I asked, relieved. Not as much as your reactions do. He looked like he was joking, but I couldn't be sure. I frowned. Do I react badly? No, that's the problem. You take everything so coolly, it's unnatural. It makes me wonder what you're really thinking. I always tell you what I'm really thinking.


[User] Please transcribe this text

[Assistant] You edit, he accused. Not very much, enough to drive me insane. You don't wanna hear it, I mumbled, almost whispered. Soon as the words were out, I regretted them. The pain in my voice was very faint. I could only hope he hadn't noticed it. He didn't respond, and I wondered if I had ruined the mood. His face was unreadable as we drove into the school parking lot. Something occurred to me belatedly.


[User] Please transcribe this text

[Assistant] More than glad to be alone with him, but remembering that his car was usually full. They took Rosalie's car.


[User] Please transcribe this text

[Assistant] He shrugged as he parked next to a glossy red convertible with the top up. Ostentatious, isn't it? Wow, I breathe. If she has that, why does she ride with you? Like I said, it's ostentatious. We try to blend in. You don't succeed. I laughed and shook my head as we got out of the car.


[User] Please transcribe this text

[Assistant] I wasn't late anymore. His lunatic driving had got me to school in plenty of time.


[User] Please transcribe this text

[Assistant] So why did Rosalie drive today if it's more conspicuous? Hadn't you noticed? I'm breaking all the rules now. He met me at the front of the car, staying very close to my side as we walked onto campus. I wanted to close that little distance to reach out and touch him, but I was afraid he wouldn't like me to. Why do you have cars like that at all? I wondered aloud. If you're looking for privacy.


[User] Please transcribe this text

[Assistant] An indulgence, he admitted with an impish smile. We all like to drive fast. Figures, I muttered under my breath.


[User] Please transcribe this text

[Assistant] And finally, did you show copies of your memos to anyone outside of the Department of Justice?


[User] Please transcribe this text

[Assistant] I've been looking forward to this for months and months. Here it is, Christmas even.


[User] Please transcribe this text

[Assistant] It went off beautifully. My time is short. I'll come straight to the point. I've just heard some very disturbing rumors. Not to make things more painful than is necessary. Mrs. Fairfield, to be forthright, is your husband alive or dead?


[User] Please transcribe this text

[Assistant] Our church, broadly speaking, leaves these matters to the discretion of its clergy. It's a matter of conscience. You may be remarried, yes, but not by me.


[User] Please transcribe this text

[Assistant] Ah, Mrs. Fairfield, within a year, you and your husband would look back upon this episode as a dream, simply a bad dream.


[User] Please transcribe this text

[Assistant] What an amazing year you've had. Because this time last year, you know, this was not


[User] Please transcribe this text

[Assistant] But, er, you've had some practice now with the big celeb, because obviously there's Golden Globes. So who did you meet at the Golden Globes? Did you meet these guys at the Golden Globes?


[User] Please transcribe this text

[Assistant] Uh, based on just, uhm, everything that's been discovered up till now and just, uhm, your personal experience of the research that you've done, uhm,


[User] Please transcribe this text

[Assistant] I'm happy to be here. Uhm, so today I'm going to talk to you about considerations for collaborative robotics.


[User] Please transcribe this text

[Assistant] Uh, so I mentioned, uhm, I direct the ARM Lab, which stands for the Assistive Robotics and Manipulation Laboratory.


[User] Please transcribe this text

[Assistant] In our lab, we focus on three main research areas, robotic assistance, intelligent wearables, and connected devices. Where the idea of robotic assistance is, uh, you have these robots that are able to enable and amplify the efforts of a human collaborator or possibly perform a task for them.


[User] Please transcribe this text

[Assistant] Intelligent wearables, uhm, are robot systems that are usually augmenting the person, uhm, and provide situational awareness, uhm, and help the person achieve their objective. And connected devices, again, have the situational awareness, uhm, but usually don't have any sort of mobility, and their goal is to anticipate, uhm, service needs.


[User] Please transcribe this text

[Assistant] And so our overall goal is to develop intelligent assistive technology that improves human life.


[User] Please transcribe this text

[Assistant] So when we really look at this sort of problem, we focus on, uhm, a, the robot, which, you know, is, uh, essential to robotics. How does a robot model itself? How can a robot model a complex task? Uhm, but when you want robots to be collaborators, it becomes really important that they also model the human.


[User] Please transcribe this text

[Assistant] And so this graph is going to come back throughout this talk, but this is really the epicenter of the types of questions that we ask.


[User] Please transcribe this text

[Assistant] So here is a, uhm, actually a partial list of, uhm, the graduate and masters students, uh, in, uh, the ARM lab.


[User] Please transcribe this text

[Assistant] Currently, and they work on a host of different projects. So, uh, let's start off by thinking about the timeline of robotics itself.


[User] Please transcribe this text

[Assistant] So in the 80s up till about the 2000s, you know, when you thought of a robot, you thought of a system that was helping with automation. So doing some repetitive tasks, possibly on an assembly line. And it was extremely important that those robots be very precise when they executed their task.


[User] Please transcribe this text

[Assistant] We saw the movement toward autonomy where now the robots had to perceive unstructured environments and understand their place in it in order to function well.


[User] Please transcribe this text

[Assistant] We saw the onset of swarm technology, centralized systems, and then even decentralized systems where these big teams of robots need to work together in order to accomplish a larger goal.


[User] Please transcribe this text

[Assistant] And now in the past decade, uh, we've been approaching augmentation.


[User] Please transcribe this text

[Assistant] So how can these robots work effectively with human counterparts when there might be limited communication?


[User] Please transcribe this text

[Assistant] I'll give a few examples that are actually very pertinent because they're actually projects within my lab. And they include possibly wanting robots to work with humans to carry objects. You can imagine a wearable system where you want to detect where a person will walk, possibly to alert them that they might possibly fall.


[User] Please transcribe this text

[Assistant] You might want a robot to be able to learn how to do a new task, uh, and work alongside a person, such as carrying boxes from one location to another.


[User] Please transcribe this text

[Assistant] You might want to have an intelligent prosthetic, uhm, that can interpret a human's intent given the environment and limited inputs from the person to execute a task.


[User] Please transcribe this text

[Assistant] You might want to have shared autonomy between a robot here, a vehicle, uh, and a person so that the system knows when and how to transfer control between the human and the robot.


[User] Please transcribe this text

[Assistant] And maybe even more passively in that sort of connected devices phase if you want to have a system that can help observe the environment and do interpretation to help a person understand the people and the environment around them.


[User] Please transcribe this text

[Assistant] So, we're gonna focus on three main questions. The first being, what makes an effective collaborator?


[User] Please transcribe this text

[Assistant] What tools do we have in robotics and what do we need to make the robots effective collaborators?


[User] Please transcribe this text

[Assistant] And finally, what must be considered for human robot teams?


[User] Please transcribe this text

[Assistant] So what makes an effective collaborator? So here is, uh, uh, uh, uh,


[User] Please transcribe this text

[Assistant] Here, there are coordinated quadrotors that are flying together and they all know where to be because there is, uhm, an oracle, a single system that's telling every agent where to go. Here you can see that it's a sort of base station that's commanding them where to go.


[User] Please transcribe this text

[Assistant] It might actually be too much for a single oracle, uhm, to handle. And that limitation might be due to computational complexity or even just range. If you imagine all of these quadrotors that you see here, if they were to expand in their function or even be over an entire city,


[User] Please transcribe this text

[Assistant] Uhm, it might actually be very difficult for an oracle to govern the second to second or millisecond to millisecond action. And so they have to be decentralized, uhm, and communicate with each other to achieve that. But again, if you look at what the state of the art is, a lot of this is in fact, uhm,


[User] Please transcribe this text

[Assistant] But the problem definitely gets harder if you think about wanting to collaborate with biological systems or agents.


[User] Please transcribe this text

[Assistant] So here you have an example of, uh, basketball players that are working very well to do an operation tempo, op tempo, uh, task here of scoring on these teammates in a very fast, uh, play. And they clearly don't have digital communication. So what makes an effective team?


[User] Please transcribe this text

[Assistant] I would postulate that this includes having shared goals and objectives among teammates, the ability to model teammates behavior for prediction, and finally, the ability to plan given personal and teammate roles.


[User] Please transcribe this text

[Assistant] So clearly, communication is extremely important in this paradigm of teamwork. And if you think about biological systems here, like these colleagues that are herding these ducks for sport into the ring as you see here, usually the three main areas are audio, vision, and tactile.


[User] Please transcribe this text

[Assistant] Uhm, so if you have audio communication, this allows for extremely high specificity. Uhm, you know, such as giving this talk, passing information. But it can be very slow. And if you have a very fast operation, that might not be adequate. Uhm, and it's very sequential. I speak, then you speak.


[User] Please transcribe this text

[Assistant] If you consider vision, uhm, this can be done mutually. We can directly observe each other. And so the information exchange can be extremely fast as you see in this video, uhm, and very detailed.


[User] Please transcribe this text

[Assistant] So here this is, uh, sort of instantaneously they're watching each other, the dogs are watching each other and knowing what they should do in order to achieve the desired behavior.


[User] Please transcribe this text

[Assistant] But you can also observe, uhm, a very complex description through, uhm, vision, visual observation as well. So, all of us are very familiar with how honeybees are actually able to perform a dance and that is visually observed by, uhm, the other insects in order to know how to operate and find, uhm, a target, uhm, location.


[User] Please transcribe this text

[Assistant] Finally for tactile, uhm, touch and forces can be a communication method.


[User] Please transcribe this text

[Assistant] So a few key features for teamwork, uhm, I would say are role understanding and interagent trust. So role understanding, uh, will classify as given the actions of others, what is my role?


[User] Please transcribe this text

[Assistant] And interagent trust, uhm, is essentially the conditions you need before you take action and you know that, uhm, your actions will not lead to some suboptimal behavior. This is the main primary assumptions you're basing this on.


[User] Please transcribe this text

[Assistant] And they consist on of having an agreed objective and a known behavioral model. So for this case with these two cars, uhm, the basis for the red car trusting the blue car is it believes that the blue shares the objective that it doesn't want to collide and the behavioral model that to achieve that objective, the blue will maintain a particular velocity.


[User] Please transcribe this text

[Assistant] Based on that trust, uhm, the red may take the action of changing lanes.


[User] Please transcribe this text

[Assistant] You have limited dexterity and strength to weight ratio and possibly limited perception.


[User] Please transcribe this text

[Assistant] But how does biological systems actually overcome these? They do this through understanding, uhm, the role of the given team and for the task.


[User] Please transcribe this text

[Assistant] They think about that task modeling. How do you model the task? And they think about that teammate, uh, modeling as well.


[User] Please transcribe this text

[Assistant] So if we kind of want to match these two things together and make robots these sort of super effective teammates, the next question we need to ask is what tools do we already have in robotics that can make robots effective collaborators?


[User] Please transcribe this text

[Assistant] So I'll break these into three main categories that are worthy of note. You have proprioception and control or the sort of knowledge of self. You have perception where you're sensing the environment. You could think of computer vision or tactile sensing and intelligence or the ability to model and make decisions.


[User] Please transcribe this text

[Assistant] So going into proprioception and control, uhm, this is, does the robot know where it is in space? So these are pretty easy. This is more hardware. You can think of your encoders that are able to sense your forward kinematics.


[User] Please transcribe this text

[Assistant] And if you have a model of yourself, how much do each one of your links weigh, uhm, what is your torque or input capacity, uhm, you can quickly model the dynamics of your system here. The torque is your input, you have an inertia, uh, Coriolis and centrifugal terms, and possibly gravity.


[User] Please transcribe this text

[Assistant] On our way towards control, this also means you need to know how to move within yourself, possibly without avoiding self-collision.


[User] Please transcribe this text

[Assistant] So motion planning becomes a really important part. If you think of systems like manipulators, usually for high dimensional systems, you kind of find yourself in this sampling category in order to achieve your target pose.


[User] Please transcribe this text

[Assistant] And finally control, uhm, how does the state change given


[User] Please transcribe this text

[Assistant] So digging a little bit deeper into the space of controls, particularly if you're interacting with your environment, uhm, cause again, we're thinking about this sort of tactile sensing. Can you feel a force being applied or how do you react to your environment doing force on you? Uhm, two primary ones for this sort of contact, uhm, scenario is impedance and admittance control.


[User] Please transcribe this text

[Assistant] We're essentially impedance, the goal of impedance is to have a desired, uhm, dynamical system, particular mass that you want to have the system behave as if that's how much mass it had and resist motion, uhm, impeding motion.


[User] Please transcribe this text

[Assistant] And I have some input. I want to actually admit that behavior. I want to move in the way that's being asked of me. And depending on your different scenario, uhm, one of these or both of these might be useful. So you can think of an impedance case where if you want to actually push a door open, uhm, you want to actually have a particular amount of force you want to apply.


[User] Please transcribe this text

[Assistant] So impedance is great, but if you're interacting with a human and you were to think about safety and you were to accidentally bump them, you definitely want to have an admittance controller.


[User] Please transcribe this text

[Assistant] In the space of, uh, perception, uh, you can think of camera sensing as being extremely important here. You see autonomous vehicle, uh, camera looking at its surroundings, able to classify the different elements that are present.


[User] Please transcribe this text

[Assistant] You can think of laser range finders that are able to, uhm, literally send out light, uhm, and use that, uhm, reflection to give you information about your environment with precise distances if other conditions are allowed. Uhm, and finally tactile sensors, right? So sensors that can actually feel forces, possibly both normal and sheer, uhm, in their environment.


[User] Please transcribe this text

[Assistant] So in our lab, we realized that this tactile sensors is a really big barrier to having robots really present in your home. And so we asked the question, right, what does it take to really get us there? And so here on the left, you see this video of this place. Oh, excuse me.


[User] Please transcribe this text

[Assistant] That crashed. Uh, well you had a video of one arm that was actually manipulating an object.


[User] Please transcribe this text

[Assistant] And on the right, uh, we have a video of Gelsite that'll be back in one moment, uh, but it's really using vision to actually look at the boundary of a sensor to give you information about it.


[User] Please transcribe this text

[Assistant] And so that's really the question, right? So the first video that we were looking at, uhm, you had this robot that was capable of manipulating an object in hand, but the way that that actually worked was they had external cameras that were looking at the Rubik's Cube and they had spent hours in training learning how to


[User] Please transcribe this text

[Assistant] Uhm, actually manipulate that object in hand without it dropping out of the robot's hand.


[User] Please transcribe this text

[Assistant] So if you really look at this sort of space of manipulators and where we've come, uhm, the state of the art for quite a while was these sort of biotech sensors. Uhm, so they have smaller resolution than these vision-based sensors, uhm, but they're, uh, really effective.


[User] Please transcribe this text

[Assistant] Then we have the onset of like gel-site. Uh, and so here, uh, this is the digit, um, using gel-site, and it has a camera inside that's looking at that flat surface you see there, and it's able to detect impressions there that give it both normal and shear forces. Uh, but it's very flat, so we call that a 2D-shaped vision-based sensor.


[User] Please transcribe this text

[Assistant] And we finally have this, uhm, 3D-shaped vision-based sensor here, uhm, coming out of Berkeley OmniTech, where there are tiny, uh, micro cameras looking everywhere. So locally, they see a flat image because they're so close to the surface, uhm, but the whole sensor itself is curved, uhm, in 3D.


[User] Please transcribe this text

[Assistant] So uhm, in our lab for this work in tactile sensing, uh, we really realized that we can break this whole thing up into three main questions. The design, the modeling, and the motion planning.


[User] Please transcribe this text

[Assistant] So we're asking, can we leverage, uhm, anthropomorphic inspiration and curved fingertips, uhm, that are capable of deforming and sensing friction to drive our design? For modeling, can we effectively model a continuously curved fingertip, uhm, with high resolution?


[User] Please transcribe this text

[Assistant] And finally, if we have that design and that model, can we leverage this efficient continuous model of our fingertip to plan multi-finger graphs for in-hand manipulation?


[User] Please transcribe this text

[Assistant] And after a few iterations, we converged on actually using a fisheye lens looking at a curved hemispherical fingertip where the radius center for the fisheye lens and the sensor are shared.


[User] Please transcribe this text

[Assistant] And it gives us some really nice features in terms of our image model for the system.


[User] Please transcribe this text

[Assistant] And so this whole sensor and whole setup is pretty low cost and has some really cool features and so far has been performing.


[User] Please transcribe this text

[Assistant] And the idea of having such a system perform this really allows us to observe the shadowing that's produced when you actually imprint on the surface. So in the middle figures and on the left,


[User] Please transcribe this text

[Assistant] And so right now, uhm, we're going one step further to say, uhm, can we actually calibrate this, uhm, and actually know the depression as well as the forces given, uhm, our system.


[User] Please transcribe this text

[Assistant] So uhm, once this design has been completed and finalized, we'll move on to modeling and motion planning as well. And some ideas for this are, if you have this ability to measure the normal and shear forces which these type of vision sensors give you,


[User] Please transcribe this text

[Assistant] Uhm, and if the surface is, uhm, uh, actually soft and can conform to whatever it is that you're contacting,


[User] Please transcribe this text

[Assistant] Then through vision, you can actually determine how the surface was deformed and what the forces, both normal and sheer, were everywhere.


[User] Please transcribe this text

[Assistant] Which essentially tells you how much force you can apply and moments you can apply before slipping occurs. And particularly what's nice about these vision-based sensors is besides just seeing the boundary, uhm, uh, because you can actually see the depression, you can know where you are inside of the wrench


[User] Please transcribe this text

[Assistant] So if you want to plan in-hand manipulation, similar to the sliding work, uhm, where they actually were having a non-zero velocity for us, we're thinking about in-hand manipulation, so zero velocity.


[User] Please transcribe this text

[Assistant] Uhm, but you can use these sets of motion clones to, to plan your overall action. And so if you think about competing methods that just directly jump in with RL,


[User] Please transcribe this text

[Assistant] Uhm, and don't have the benefit of these sort of motion cones, uhm, or wrench cones, uhm, it's gonna take them a lot longer to learn. But if we can mathematically represent these, we think this is a tool that will cut down the rate of learning, uhm, for these other methods, uhm, exponentially.


[User] Please transcribe this text

[Assistant] So the last phase I'll talk about for them, uh, in the ones that I mentioned is intelligence, right? And so if you think about all the different types of


[User] Please transcribe this text

[Assistant] Uh, models you can use and we'll focus specifically on tools that you would use, um, for predicting the behavior or actions of either yourself or other agents, um, notably, uh, commented as trajectory forecasting. They break down into a few different categories and I love this breakdown, um, coming out of Marco Pavone's lab, uh, here at Stanford.


[User] Please transcribe this text

[Assistant] You have ontological methods which believe that there's some underlying structure to how the decisions are being made or how the state is changing.


[User] Please transcribe this text

[Assistant] And those can be states-based models or internal cost functions. And then you have on the other side, uhm, these, uh, phenomenological, uh, methods, these data-driven structures. So these can be deterministic regressors or even, uh, generative, uh, approaches. And so based on what, uh, what you actually have in these assumptions that you're making about your system,


[User] Please transcribe this text

[Assistant] Some may be better than others. And a lot of our work, we assume that there may not be, particularly because we're trying to predict behavior, these really nice ontological methods we may be able to pull from. And this pushes us more and more toward these sort of generative approaches.


[User] Please transcribe this text

[Assistant] So finally thinking about where robots are and where, you know, we'd like to be, you know, common algorithms for cooperation for robots, uhm, can be categorized as either having an oracle that's telling all agents what to do.


[User] Please transcribe this text

[Assistant] Uhm, all agents could in fact act independently toward a predetermined goal and not really interact, but just assume that everyone knows what they should do and where they should be.


[User] Please transcribe this text

[Assistant] And then finally, you have consensus through voting systems, uhm, in terms of the digital space where they share information and they converge on a solution or an action, uhm, based on that shared information.


[User] Please transcribe this text

[Assistant] Uh, but most of these implementations are conditioned on the availability of digital communication. And so this is something that we have to rethink when you want to actually have a human in the loop.


[User] Please transcribe this text

[Assistant] Right, so what do robots need to be effective teammates for humans? I postulate that they need the ability to model complex tasks, the ability to interpret and manipulate environments configured for humans, and finally the ability to model teammates' actions and roles.


[User] Please transcribe this text

[Assistant] So I believe you can also further kind of categorize robot collaborators as wearables, uhm, and an external collaborator where one is augmenting the human and the other is kind of performing like a partner.


[User] Please transcribe this text

[Assistant] And so in the wearable space, uhm, you actually can see, uhm, the ability to substitute, uhm, substituting device. So this, uh, wearable is actually performing the function, uh, part of you, uhm, or perhaps an ability enhancing device such as a phone.


[User] Please transcribe this text

[Assistant] So here you actually see a human on the left.


[User] Please transcribe this text

[Assistant] And I can kinda think of this control diagram, where the human has some cognition, they're planning what they wanna do, their brain knows what kind of controller they need to execute it, you send that to your body, those electrical signals to your body, which acts, and then you have some sensory feedback, uhm, that's telling you what happened after you took that action.


[User] Please transcribe this text

[Assistant] And so, particularly for wearables, the question we need to ask is what part of the human being is being augmented or substituted? And if we do, is the robot augmentation functioning adequately, uh, and does the presence of the robot feel natural or easy to use?


[User] Please transcribe this text

[Assistant] So to answer this, uh, my PhD student Shivani Guptasarma, um, and, uh, uh, Masters students Gabriella here, they are looking at, um, what we call the intelligent prosthetic arm.


[User] Please transcribe this text

[Assistant] So traditionally in the space of upper limb prosthetics, uhm, you have some powered prosthetics and others that are body powered as you see here.


[User] Please transcribe this text

[Assistant] Uhm, for shoulder disarticulation amputees. And as you see, this can be very challenging for them to manipulate, even though they're actually doing a fantastic job. Just the limited dexterity can just take them longer to perform some tasks. So we're thinking, how can robotics actually help?


[User] Please transcribe this text

[Assistant] And if you can kinda zoom in here on the center figure, uhm, we're proposing, uhm, a powered arm, uhm, that's informed by, uhm, the person wearing EMG signals, uhm, leveraging augmented reality, uhm, and, uhm, a robot arm that's outfitted with the ability to perceive the environment as well.


[User] Please transcribe this text

[Assistant] So kind of exploding, uhm, that methodology and going back to that original figure, uhm, I put on like the second slide, uh, here, you know, how does the robot sense the person, right? You have these EMG measurements. That's how the human is, uhm, sending information to the robot. But what's really difficult a lot of times is,


[User] Please transcribe this text

[Assistant] Ascertaining human intent and that line between the human and the environment.


[User] Please transcribe this text

[Assistant] It's usually very difficult to sort of intercept.


[User] Please transcribe this text

[Assistant] As well as extract information from what they're focusing on can be a key tool in intersecting that component between the human and the task and environment, which can allow us to better model the human because we understand what they're thinking about with the environment and how they're interacting directly with us, the robot.


[User] Please transcribe this text

[Assistant] So, let's go back to that sort of, uhm, control diagram. You have that cognition, which is a planner, controller, uhm, you're sending some, uh, electromyography signals, some impulses to your muscles, which are your actuators, your body has some dynamics that it acts through, and you have some sensory feedback, touch, and vision.


[User] Please transcribe this text

[Assistant] So clearly we have all of these capabilities, maybe to a lesser extent in some cases, uh, but present in robotics, cognition, right? The ability to plan and control your plant and actuator, body dynamics, sensory feedback, touch and vision.


[User] Please transcribe this text

[Assistant] So, if I look at this sort of expanded, uhm, uh, uh, uh, control diagram, the planner, controller, actuator, body dynamics, vision, touch, uhm, and this is all interacting with your environment and task on a bigger control loop, if you will.


[User] Please transcribe this text

[Assistant] Uh, then if you have a prosthesis, this means that the actuator and body dynamics are usually not available, right? And so someone with just a simple prosthesis would command a robotic prosthesis, uh, through, uh, say EMG.


[User] Please transcribe this text

[Assistant] If you think about a powered reflex, possibly with some basic reflexes, like maybe heat or contact, uhm, then maybe you want to outfit that prosthesis with the ability to have some sensory feedback, like vision or touch. So if the person is moving and they impact with a wall, it might respond immediately, uhm, but it doesn't really think, and the person is still controlling with EMG.


[User] Please transcribe this text

[Assistant] Finally, you can even have a system where the person has the ability to plan and control, but the robot also using the vision and touch of the environment can also understand its role and can help the human to be, uhm, uh,


[User] Please transcribe this text

[Assistant] Less cognitive load for the person who's using the system.


[User] Please transcribe this text

[Assistant] As I was mentioning before, augmented reality is a very powerful tool that can allow the robot to both extract and present information to the wearer. So the top two left corner images really show how with devices like the HoloLens 2, you can actually perform gaze tracking and see what the person is looking at or thinking about.


[User] Please transcribe this text

[Assistant] You can kind of model that as sort of a conic area of focus, which can be very helpful when ascertaining what the person actually wants to manipulate.


[User] Please transcribe this text

[Assistant] Uhm, and then if you have knowledge about your environment, uhm, such as affordances or the pose, if you're tracking them or seeing, understanding this can all be very useful and provide information back to the wearer.


[User] Please transcribe this text

[Assistant] So I want to set this kind of scene for you, if you will, uhm, on how you can imagine the system ultimately working.


[User] Please transcribe this text

[Assistant] So uhm, let's look at this, uhm, activity of daily living, ADL, of, uh, drinking some fluid with a pitcher of water to refill your cup and, uh, feeding yourself from this plate of food. And you have this simple state machine here on the right. Uhm, so, you know, when you're getting ready to start off, you have two choices. You could grasp the pitcher or you could grasp the fork.


[User] Please transcribe this text

[Assistant] Uhm, here this little circle with a plus sign designates where the person's area of focus is, let's say from just, uhm, the, uh, AR, uh, is focusing on. And then that allows you to actually put more weight on the state machine to transitioning to pouring, uh, say, the pitcher.


[User] Please transcribe this text

[Assistant] And so when you're pouring the pitcher, uh, you have two options. You could be, um, either filling the cup or replacing the pitcher. Uh, but maybe vision isn't as helpful in this scenario. So maybe here is where you might actually want to rely more on EMG to kind of signal which one of these operations at a high level you're trying to perform.


[User] Please transcribe this text

[Assistant] And then, uh, once it's replaced, you know, options you might logically go to from there are drinking from the food, uh, from the cup or possibly manipulating.


[User] Please transcribe this text

[Assistant] About a very practical way uhm, this system could ultimately be used. So let's define a few variables. We have the state of the system.


[User] Please transcribe this text

[Assistant] S, the gaze, G, the EMG signal, E, some desired action, AI, and the probability that the human wants to do a particular action.


[User] Please transcribe this text

[Assistant] And we assume that, um, that set of actions is, uh, finite and, uh, the probability over them sums to one.


[User] Please transcribe this text

[Assistant] So the question, uh, is we would like to predict the wearer's intent based on the state and behavior. So what action do they want to take given the state of the system, given the input from the EMG, given their gaze?


[User] Please transcribe this text

[Assistant] And we realized that we want to do that by modeling the human's behavior. What are they doing?


[User] Please transcribe this text

[Assistant] And so from, uh, Bayes rules, we actually have, um, the following relation.


[User] Please transcribe this text

[Assistant] Right? That the probability of your action given the EMG gaze and state


[User] Please transcribe this text

[Assistant] Is equal to this, uhm, relationship, uhm, between the reliance of the, uh, electro-myography, the gaze on the action in the state, both together and independently on the state, as well as the human intent model, what action you would take in a given state, uhm, scenario.


[User] Please transcribe this text

[Assistant] So if you have these probabilities, uhm, if you want to predict their intent, you simply take the argmax over that probability and this tells you what action you might expect the person wants to take.


[User] Please transcribe this text

[Assistant] So at this point, right, this has been a well phrased problem and you can actually branch off into many ways to solve this.


[User] Please transcribe this text

[Assistant] Uh, generative approaches, variational autoencoders, particularly conditional variational autoencoders have kind of risen to the top in a lot of literature as being a


[User] Please transcribe this text

[Assistant] Really nice way to solve these types of problems, but definitely not the only way. But here I'll present it here briefly as the method we're choosing to go forward with. So x here is your input, z is your latent space, and y is your output.


[User] Please transcribe this text

[Assistant] So here you have these arrows, the black and red, uh, on the left here that indicate, uhm, the, the testing process and the red indicate, uhm, the training process.


[User] Please transcribe this text

[Assistant] And here there are a lot of resources you can read that really looks at the mechanics of this implementation, but simply at the top you want to predict what is your output given your input and you can represent that in your latent space with the following parameter and distributions.


[User] Please transcribe this text

[Assistant] And so for our task, right, to actually solve our problem, we need, we realize we need to actually calculate renditions of the following, um, uh, models. What is the probability of an action given the state?


[User] Please transcribe this text

[Assistant] And probability of the, uhm, input given the state or given the state and the action. So you could actually have a separate one there that is just the state as well.


[User] Please transcribe this text

[Assistant] So what's really nice is because we can actually separate these and combine them at the end, uhm, that separation ability actually might mean that if we're very careful on how we choose our states, we can actually train these things independently.


[User] Please transcribe this text

[Assistant] So here you can imagine training an ADL or a task with augmented reality with able-bodied individuals that can put in a whole lot of work and benefit, uhm, the amputee community by actually putting in that work and training those models.


[User] Please transcribe this text

[Assistant] And then on the other side, similar to conventional prosthesis, we can actually calibrate by collecting behavioral data for these specified actions. And so you can imagine if you had this sort of setup with the, uh, an AR system, your EMG and your prosthetic,


[User] Please transcribe this text

[Assistant] If someone has trained how to do a whole set of tasks and maybe there's a library or an app store of tasks, right, then on the other side, if you can perform a lot of calibration routines, which


[User] Please transcribe this text

[Assistant] And now your system can quickly connect to the, to the,


[User] Please transcribe this text

[Assistant] So uhm, on the other side here, you have instead of ability replacing devices, you also can imagine ability enhancing devices. So in the ability enhancing devices, usually this kind of falls under the category of improving some sort of sensory feedback concerning the world.


[User] Please transcribe this text

[Assistant] So, uh, one project that we're working on here, uh, with my masters and undergrad students, Junwu, uh, Luciana, um,


[User] Please transcribe this text

[Assistant] So the question we ask is if these falls can be predicted, can they also be mitigated using a wearable sensor that observes the person and the environment?


[User] Please transcribe this text

[Assistant] And so this isn't really new. There's similar work, uhm, elderly fall risk assessment, uhm, and prediction based on gait analysis. But a lot of this isn't online work. They're not predicting if a fall is going to be happening on some significant


[User] Please transcribe this text

[Assistant] Time horizon and even those that do are looking at less than a second or even on the millisecond scale.


[User] Please transcribe this text

[Assistant] But we're saying if you can see far out in the environment, uhm, what's stopping you from trying to predict if down the road, uh, you might be able to predict the person might be at risk of falling?


[User] Please transcribe this text

[Assistant] And so, uh, if you observe the person's gait and surroundings, can you predict the path that they will take?


[User] Please transcribe this text

[Assistant] What the gate is gonna be over that path and the associated risk of falling.


[User] Please transcribe this text

[Assistant] And so the idea would be to have on the person a camera and IMU that is going to be in the final deployment that you just have to wear around your waist. And then we would train in order to allow such a system to correctly estimate the environment, the odometry, the gait and orientation of the wearer.


[User] Please transcribe this text

[Assistant] And once those were successfully implemented, uhm, then the objective would be to use these inputs, uhm, with, uhm, some sort of, uh, sequential method, like an LSTM, uhm, to actually predict, uhm, the future trajectory, gait, and orientation, which is associated with the fall risk of a person.


[User] Please transcribe this text

[Assistant] Uhm, again on this sort of ability enhancing device and more to the connected devices under a realm is our ASL to English translation mobile, uhm, application. So the idea we have here is, you know, if you really think about a lot of translation apps, a lot of them all clearly, uh, verbal and, uhm,


[User] Please transcribe this text

[Assistant] In the global community, it has been, uh, raising concern, how can we actually have a good, um, uh, automatic, uh, automatic interpreter for


[User] Please transcribe this text

[Assistant] Uhm, ASL back to English. There are a lot of things, uh, companies such as Handtalk that are able to go the other way. You say what you want to say and it shows you what the signs are, but going the other way, uhm, there's not a lot of resources in that space.


[User] Please transcribe this text

[Assistant] And those that do, as you can see here on this reference chart, really require a lot of GPU overhead. And so if you want to think about this being deployable, like carrying this ability in your pocket, we need to do something clever in order to achieve that.


[User] Please transcribe this text

[Assistant] So that's where our solution space is. How can we actually do this on the mobile computing scale? So to achieve this,


[User] Please transcribe this text

[Assistant] We're actually, uh, leveraging, um, uh, transformers to actually, um, characterize what the spoken words are. And we're doing preprocessing on our video data, which includes, um, the ability of extracting the person's facial expressions and hands.


[User] Please transcribe this text

[Assistant] Um, and, um, now we're actually much better at doing the torso as well, um, and correlating those to, um, the expected output from the words. And so if you can do that, you can imagine being far more efficient in terms of computation on your system because you're not dealing with raw videos.


[User] Please transcribe this text

[Assistant] How you're dealing with a subset of key points that can be very, uh, uh, essential for that.


[User] Please transcribe this text

[Assistant] So moving to the space of, uhm, external collaborators.


[User] Please transcribe this text

[Assistant] Here, uh, this is work, um, that's, uh, in collaboration with the Iliad group and CS Dorsa Siddique, uh, my, uh, PhD, uh, student, Eli, uh, Ing, Albert, uh, and JD. We're here. Our goal of this project is to model.


[User] Please transcribe this text

[Assistant] So we realize to be effective for this, we need to go back to that term of interagent trust, right, uhm, which we defined before. If you can, uh, model the other persons or the other teammates' behavior and actions, then you can understand better your role in order to carry, uhm, effectively.


[User] Please transcribe this text

[Assistant] And so in one version of this application, we're thinking, can you predict where your partner wants to go given where you've been as a team as and the actions you both have mutually taken as well as the environment of.


[User] Please transcribe this text

[Assistant] In this other space of an external collaborator here, we have this project, Efficient Learning from Demonstration and Collaboration with Aliyah Smith and Emanuel Rotana.


[User] Please transcribe this text

[Assistant] Um, and here the goal is to say, hey, uh, you know, if you imagine a scenario where, you know, you have first responders, you want to do a duty in an unstable environment, you want to move things, but maybe it's not safe to send a whole bunch of first responders into a space. Maybe it would be great to send a few first responders.


[User] Please transcribe this text

[Assistant] And more robots, uh, and there are many application spaces. You can kind of think of that where if the robot were able to work alongside a person, you could reduce risks to that person.


[User] Please transcribe this text

[Assistant] And so there are two main objectives within this set of problems here. One is can the human efficiently teach a robot a new task by identifying critical states using AR?


[User] Please transcribe this text

[Assistant] And secondly, given the described tasks, can the robot collaborator learn the human behavior model and leverage it to take action that optimizes performance for the entire team?


[User] Please transcribe this text

[Assistant] So here on the right, you see, imagine the case of opening a door. If you go back to one of those Google papers, they actually had a bunch of robots opening a door many, many times to be successful.


[User] Please transcribe this text

[Assistant] And a huge problem with RL and learning from demonstration is really the state space, right? But if I were able to define to the robot, what would be important states it should be paying attention to? You have a rigid object, which is a door. You have a constraint, which is the hinge. You have the handle is another rigid object and this constraint.


[User] Please transcribe this text

[Assistant] And if you operate these in this particular sequential action, uh, or state space action, then you can achieve your objective. And so then even, uh, transferring this to new abilities, you need to be looking for these corresponding elements, um, will allow robot to understand these tasks a lot more efficiently. And that brings a lot of power.


[User] Please transcribe this text

[Assistant] If you have these sort of behavioral models where you can predict the action the human would take given the set, maybe their actions are suboptimal.


[User] Please transcribe this text

[Assistant] Right, if the human carries a heavier box, or all the heavier boxes that the robot could have been carrying,


[User] Please transcribe this text

[Assistant] Then if you sum this up over the entire operation, it's going to just take longer because the human was doing suboptimal actions.


[User] Please transcribe this text

[Assistant] So maybe if you realize that human has a tendency to carry our heavy things, you should jump in and carry those heavier things for.


[User] Please transcribe this text

[Assistant] So you could think of any sort of POMTP, you know, belief tree solver using their model to propagate forward and explore different trees and then determine what best action you should take that will lead to optimality for the entire team where you're kind of playing their scenario.


[User] Please transcribe this text

[Assistant] In your mind, and this should ring some bells for you, I think, in terms of game theory approaches, but usually in those methods, uhm, it's adversarial, right? You're trying to beat the person here. You're asking, what can I do to be an effective, a more effective teammate and bring more optimality to the team?


[User] Please transcribe this text

[Assistant] Uh, this sort of thinking can be applied to the case of shared autonomy.


[User] Please transcribe this text

[Assistant] Alright, so if you actually look at the space of autonomous vehicles, uhm, a lot of it as we approach level five and six autonomy is these cars will become really smart and we'll just hand over control and we can just actually just rip the steering wheel out and we don't need this anymore.


[User] Please transcribe this text

[Assistant] Uh, and so I personally disagree. I think we will always want to drive cars, especially as they get newer, faster, sleeker.


[User] Please transcribe this text

[Assistant] You know, and Elon Musk puts out the best, next best Tesla, we're gonna wanna drive it. Period.


[User] Please transcribe this text

[Assistant] And so if you want to think about how you can actually get the best of both worlds, it becomes critical to think about how handover and control can be both safe and smooth.


[User] Please transcribe this text

[Assistant] Uhm, and, uh, on top of that, right, uh, you know, art has been warning us for a long time that there's gonna possibly be some reason you wanna take over. I don't subscribe to that, but, uhm, let's, let's cater to that and build for this eventuality.


[User] Please transcribe this text

[Assistant] So, uh, leveraging human intent for shared autonomy. If you kind of think of this controller and plant paradigm, uhm, here you have MH, which is the model of the, uh, human. You have MR, which is the action policy of the robot.


[User] Please transcribe this text

[Assistant] You're asking what action would the human take given the states of the system and the actions that the robot has taken to date? What's the state transition model and what is the human action policy?


[User] Please transcribe this text

[Assistant] So what's really fantastic with this sort of setup is, as you're driving, if you have a model of the human, before you actually hand over control and let it be just a boolean switch, as you might do in a lot of shared autonomy literature, if you actually just observe their input for a while, you can say, does this input match to a safe set?


[User] Please transcribe this text

[Assistant] So let's say you're on the highway and you're driving around a curve, there's a particular range or variance that's allowable in the gas pedal and a particular range and variance that's allowable in the steering angle.


[User] Please transcribe this text

[Assistant] Right? And if, uhm, you're going the other way, uhm, possibly handing over control at non-zero speeds from the human to the robot, vice versa, right? If something's going on,


[User] Please transcribe this text

[Assistant] And perhaps the robot doesn't sense uhm, something in its path, it can react uhm, in an efficient way. And particularly uhm, again, AR can play a really big role, because in addition to uhm, risk presentation,


[User] Please transcribe this text

[Assistant] Right, you can imagine, uhm, the robot's plan also being displayed in a sort of holographic way in the future in these sort of windshields. And I think that's really critical for self-driving cars.


[User] Please transcribe this text

[Assistant] Uhm, for us to actually build and maintain trust of these vehicles. So if you see something or someone getting ready to walk in your path, and the car is able to efficiently convey its plan to you, you can actually detect, uhm, hopefully, a non-frequent but possible fault.