[User] Please transcribe this text

[Assistant] The theoretical basis for that goes back really far, goes back to Osborne's work on applied imagination, where he argued that the best ideas come at the tail end of idea generation. So if you want to get to the really good ideas, then you have to generate many ideas.


[User] Please transcribe this text

[Assistant] The problem is that this behavior is not well supported in existing design tools or the practice of using those tools.


[User] Please transcribe this text

[Assistant] So, for instance, on the left, uhm, this is kind of the dominant model in digital drawing tools where they assume that you're going to work with a document model and you just map your alternatives to different file names that are aptly named, you know, concept one, concept two, concept three and so forth.


[User] Please transcribe this text

[Assistant] And in research, the way that we think about working with multiple ideas, for example, could be done through a tabbed interface. And so this is, uh, an example from the juxtapose system that, uh, Bjorn Hartman and Scott Clemmer did here.


[User] Please transcribe this text

[Assistant] The problem though is that other researchers argue is that these types of representations hinder


[User] Please transcribe this text

[Assistant] interpretation and the projection of meaning. What the heck does that mean, right? Let me give you an example. So take either of those two representations. Which is the leading alternative? Which one is favored? Which one was tried?


[User] Please transcribe this text

[Assistant] But no longer in contention and which ones are in between. We can't even see all the ideas anymore. I mean it's small but it's an important part of the creative process. And so what I wanted to do is try to bring that back as best as I could. And so my approach,


[User] Please transcribe this text

[Assistant] Was to explore a new model I called interactive spatial maps.


[User] Please transcribe this text

[Assistant] And the concept was implemented as part of a prototype sketching application. And the larger window is just a very crude approximation of what you would get in a digital drawing tool. And our spatial maps is in the lower right.


[User] Please transcribe this text

[Assistant] And it runs as a floating window. It can be on your primary display, or if you want to, you can move it to a secondary display if you have one.


[User] Please transcribe this text

[Assistant] And the key idea of the spatial maps is that we allow you to directly manipulate the representations on a zoomable workspace.


[User] Please transcribe this text

[Assistant] And so you can project meaning by using the dimensions of both space and scale. So for example, the user here has a number of alternatives. And the one that they think is favored at the moment, they can scale it.


[User] Please transcribe this text

[Assistant] Make it larger and position it near the center. And the one that maybe they tried but didn't necessarily like, they can scale it down and position it away and the others are somewhat in between. So when you look at the workspace relative to what I showed before, you should get an immediate intuitive sense as to how they're thinking about the design situation and you can see all the ideas at once.


[User] Please transcribe this text

[Assistant] So let me just show you uhm, a little video clip of this in action.


[User] Please transcribe this text

[Assistant] And we did an evaluation and I wanted to compare it to how it works with, uh, relative to a tab interface as represented say in Microsoft OneNote and a layered canvas as represented in Photoshop. So just briefly our procedure with it was a within subjects design. We had 18 users all trained in some form of visual design.


[User] Please transcribe this text

[Assistant] And we had a number of different measures. We measured process outcomes, so the number of ideas, the number of categories of ideas, the switches between the ideas.


[User] Please transcribe this text

[Assistant] We measured, uhm, strategies that they used for how they worked with the ideas during the task. We measured their preferences, and we got qualitative feedback. And we had a lot of interesting findings, and if you want to read about all of them, you can look at our GI 2010 paper. It's one of my all time favorite papers.


[User] Please transcribe this text

[Assistant] And let me just give you the highlights. So one of the highlights was that users actively organize and reflect.


[User] Please transcribe this text

[Assistant] During the idea generation task, it was a core part of generating ideas. And so the idea sets weren't passive, they weren't in archive. People actively engaged them throughout the task to help them come up with more ideas. I thought that was really interesting.


[User] Please transcribe this text

[Assistant] When they used our tool, for example, they had about 10 reflections where they'd look at the entire idea set at once and 12 organizational interactions where they'd map.


[User] Please transcribe this text

[Assistant] Either the idea, either onto space or scale. And that was more than what they would do with the other two tools. And my interpretation of that is that there's a need to engage those idea sets and when you give people the opportunity to do it, they'll take advantage of it. But they also wanted more.


[User] Please transcribe this text

[Assistant] Was that they wanted more ways to engage the idea set. And so we went back and extended the tool to offer a grid based view. We could look at all the ideas at the same scale.


[User] Please transcribe this text

[Assistant] Because we allowed them to directly manipulate and organize and reflect on ideas, there's a very strong preference for


[User] Please transcribe this text

[Assistant] Spatial maps. And maybe the biggest outcome of the study was that we showed that spatial maps is a viable approach and an effective approach for working with ideas. And so I would love to see this be extended and implemented in the growing class of digital drawing tools that are now available through app stores as well as uhm, just on the web.


[User] Please transcribe this text

[Assistant] But I also think there's a lot of opportunity to really dig deeper into how people do exploratory work in say areas of technical writing and programming. And I know this has been around for a really long time, but there's a lot more that we can do in this space and I hope to engage some of that work in the future.


[User] Please transcribe this text

[Assistant] But one of the things that really piqued my interest during this project was this notion of reflection. About how people look back to engage with things that they've done and try to extract meaning from it. And that led me to Don Schone's work.


[User] Please transcribe this text

[Assistant] He has uhm, a number of theories of reflection, and reflection is a theory of practice. And he developed that theory of practice through observations of how architectural designers work.


[User] Please transcribe this text

[Assistant] It was reflection on action. It's intentional critical thinking about prior work.


[User] Please transcribe this text

[Assistant] To learn from your experience and to project that into the future. So if you ever look back at, say, a talk or a CHI paper to think about what you might do next, you've engaged in reflection on action, okay? And I'm just gonna call that reflection for this talk. And I do it all the time. I actually did it for this talk, I did it for the last paper, and so forth.


[User] Please transcribe this text

[Assistant] Reflection typically benefits from having a representation of design process and


[User] Please transcribe this text

[Assistant] Designers often construct their own representations through different organizational and naming strategies. So this is just an example of one that we uncovered from a study that we did. So in this case, they use numbers to indicate project priority. They use, uhm, letters to indicate status. They have highlights to indicate unmet dependencies. And they organize iterations using holder structures. It's not that the naming schemes are bad.


[User] Please transcribe this text

[Assistant] It's just that the hierarchical representation doesn't reveal the relationships in a way that's meaningful to the user. So what did they learn from that project? What was a proud moment?


[User] Please transcribe this text

[Assistant] What was a critical incident? And what I'm arguing is that you can't really know because it doesn't come out of the visualization or the representation. Is it short?


[User] Please transcribe this text

[Assistant] That's where I'm gonna go. We're gonna try to make it explicit because what really kind of, I love Shown's theory, but what sort of kind of bothered me about it is it seemed very tacit and very implicit. And I wanted to think about what if we could actually make that explicit in a visualization so people can engage this in a more kind of direct way.


[User] Please transcribe this text

[Assistant] His whole book is dedicated to trying to communicate that people do this. And it's a very important part of how people work in the design profession.


[User] Please transcribe this text

[Assistant] It was a very nice visualization that you can construct by hand. Once you have the visualization, you can engage in reflection. The problem is the cost of construction works against the utility of the visualization.


[User] Please transcribe this text

[Assistant] And uhm, others, visualizations for state-based design histories and exploratory behavior, very nice contributions, but they didn't really target reflective behavior, and that's what I wanted to do.


[User] Please transcribe this text

[Assistant] And so we created a tool called Reflection Space. And the key innovation of Reflection Space is that it provides a condensed visualization of design process that allows you to engage with the process directly.


[User] Please transcribe this text

[Assistant] It was predicated on results from interviews that we had with graphic, product, and architectural designers. So let me just take you through what you see.


[User] Please transcribe this text

[Assistant] On the y-axis, we have a number of channels. It's called research, inspiration, concepts, prototype, and communication, as an attempt to capture a generalized design process. If you don't like that, that's fine. Uhm, you can change it and configure it in the tool.


[User] Please transcribe this text

[Assistant] Every filled circle is a representation of an artifact. It could be a concept sketch, it could be an AutoCAD prototype, it could be an example that you found on the web. And you can select them.


[User] Please transcribe this text

[Assistant] And look at the thumbnails and open them up for editing.


[User] Please transcribe this text

[Assistant] On the x-axis, that is time. And we cluster the artifacts based upon creation time. We use creation time because it gives us a stable representation. Once I place it, it stays there as a cross invocations of the tool.


[User] Please transcribe this text

[Assistant] So just to put this in context, uhm, what you're looking at is the visualization of process of an architectural designer who is working to remodel the dome of the observatory at our institution.


[User] Please transcribe this text

[Assistant] And what you notice is they start with some background research. In this case it was lighting conditions. They moved on to have a couple of bursts of inspirational example flying. In this case it was domes from around the world. The second burst was coincided with some concept sketches that they drew, uh, followed by some AutoCAD prototypes that they created. And there was continued communication with the supervisor and peers.


[User] Please transcribe this text

[Assistant] Throughout. And the point that I wanna make is that you can read and tell stories about process with this type of visualization that you cannot do with a file-centric approach.


[User] Please transcribe this text

[Assistant] One of our challenges though is we really wanted to make this as low cost as we possibly could.


[User] Please transcribe this text

[Assistant] And so the way that we approach this problem is that we try to map your organizational strategy to our visualization. And we use things such as your file names, your file extensions, your folder hierarchies, your folder names, as well as the creation times.


[User] Please transcribe this text

[Assistant] Okay, and the rules are a bit more complex than what we have here, but the basic idea was you could say something like, well, if your file is in, or is a PDF, or a document that ends with .doc, and it's in user, research, background, or source combination, then categorize it as research in our tool. Okay?


[User] Please transcribe this text

[Assistant] And so the more closely aligned the way that you organize is with what we expect.


[User] Please transcribe this text

[Assistant] If we map it and we get it wrong, you can move it around uhm, in the tool and that would, that would actually stay uhm, persistent.


[User] Please transcribe this text

[Assistant] Okay, so one of the things that you can do with the tool is you can capture stories. And the way that you capture stories is you select the icon in the upper right and you can put it into what we call the reflection channel. Once you do that,


[User] Please transcribe this text

[Assistant] The system prompts you to associate a number of artifacts with that story.


[User] Please transcribe this text

[Assistant] Okay, and a story can be anything that is memorable to you as a designer. It can be a proud moment.


[User] Please transcribe this text

[Assistant] It can be a lesson learned. It can be a critical incident. But what we're trying to do is make it explicit in the visualization so you can revisit it later on.


[User] Please transcribe this text

[Assistant] So you get prompted, you can, you can add and associate, uhm, the artifacts.


[User] Please transcribe this text

[Assistant] And you can put as many in there as you want. And actually one of my favorite features of the tool is you can come back later on and export that into a PowerPoint presentation to share some of those lessons learned with colleagues.


[User] Please transcribe this text

[Assistant] You can also mark your favorite materials. And so if you have an artifact that you're particularly proud of, you think it's inspirational, you want to revisit for any reason.


[User] Please transcribe this text

[Assistant] You can mark it and we do it with a star so you can pick as many as you want and it makes it pop out of the visualization so you can revisit it very quickly later on.


[User] Please transcribe this text

[Assistant] Goal that we had based upon the results of interviews that we did is that they wanted to look at the distribution of effort.


[User] Please transcribe this text

[Assistant] And this was a very challenging problem because we didn't want to have to instrument every application that one uses.


[User] Please transcribe this text

[Assistant] To record say editing time. And so the way that we approach the problem is that we looked at the changes to the modification date at the level of the file system. And so our assumption was that every time you do a save, it changes the modification date. And you're not going to do a save unless you made a meaningful change.


[User] Please transcribe this text

[Assistant] And so basically what we do is we're counting the number of times that you essentially save that artifact. We collect that data for everything that's in the system, we store it, and then we rank it per channel. And we display the ones that are at the top 10%, chosen arbitrarily, you can change it if you want, as an unfilled circle.


[User] Please transcribe this text

[Assistant] There's a lot of effort made to that particular concept and it was very early in the process. So that could indicate pre-mature commitment to a concept and that might warrant a change in process or at least a discussion as to why that was the case. So we do the evaluation of this.


[User] Please transcribe this text

[Assistant] And we had seven users all trained in either graphic or architectural design. And the way that worked is that


[User] Please transcribe this text

[Assistant] We built our representation in our system for them because we wanted to test the utility of the visualization first, not get too caught up in whether or not the mapping was perfect. And then we just recreated their own representation in the lab.


[User] Please transcribe this text

[Assistant] Uh, they would come in and we would engage them in a bit of storytelling. And so they would just talk about the project by itself, no representation. And then with one of the two representations, other project.


[User] Please transcribe this text

[Assistant] Other representation and we compared the differences in terms of the measured outcomes.


[User] Please transcribe this text

[Assistant] They would tell more and longer stories when they used our tool in terms of word count than with other representation.


[User] Please transcribe this text

[Assistant] And they rated it better for telling stories, noticing patterns, um, estimating effort, and just overall preference. So it's not definitive because the, you know, the sample size was small, it was a single study, but it is at least promising. Other interesting things that we found is that they wanted to mark iterations, uh, decisions and key,


[User] Please transcribe this text

[Assistant] Milestones in that reflection channel. So you can imagine having a more kind of elaborate visual vocabulary of things that you could store there.


[User] Please transcribe this text

[Assistant] And another thing that we found is that they would be more careful with the naming. If they knew that the visualization was based upon their strategy for how they organize things, then they might change that because they knew that that would be, uhm, impacting that visualization that would be created.


[User] Please transcribe this text

[Assistant] And the other things, there's a lot of enthusiasm. They were really interested in having a visualization that would allow them to engage in explicit acts of reflection. And that made me really excited. And so that got me to think a lot about, you know, how you might use this in the future. And so one of the things is everybody asked me to put my data in the cloud. So what if I did that? What if I put all design artifacts


[User] Please transcribe this text

[Assistant] And you can try to build very robust mappings and then construct that type of visualization as a value add and not just use it as a passive storage archive. So I think that could be really interesting, uh, moving forward.


[User] Please transcribe this text

[Assistant] And so let me just illustrate with an example why this is such an important component of design.


[User] Please transcribe this text

[Assistant] I can summarize this as JCPenney had a really great idea for a tea kettle. Unfortunately, it didn't work out so well for them.


[User] Please transcribe this text

[Assistant] And it's somewhat of an amusing example, but the reality was this was a national embarrassment for JCPenney. They had to take this tea kettle off all the shelves, they had to issue a public apology, uhm, they got a lot of negative press, and they wound up in my talk.


[User] Please transcribe this text

[Assistant] And so I think this is a really interesting example because it highlights one of the key reasons why you need feedback in design. And that's to compare intention to perception.


[User] Please transcribe this text

[Assistant] And so you as a designer often always have an intent. What is it you're trying to communicate? And other people may or may not get that perception. It's not like when you write a paper.


[User] Please transcribe this text

[Assistant] And you're trying to say something and you think it's unassailable, you give it to your advisor and it comes back with all red ink and you're like, how could you not possibly get what I was trying to say? And so it's that difference between intent and perception. And I would argue that good feedback is necessary but not always easy.


[User] Please transcribe this text

[Assistant] And so there are a number of known strategies for getting feedback. I mean, you can organize a critique, you can ask peers, you can submit to an online community, but.


[User] Please transcribe this text

[Assistant] They have both benefits and limitations. And so when you organize a critique, it's very heavyweight. It's unlikely you're going to do that for day to day decision making.


[User] Please transcribe this text

[Assistant] And if you're not familiar with the, with the kind of design domain, have a training in design, it's probably unlikely that you're even familiar with that technique at all. You can informally ask peers, but we know through research that


[User] Please transcribe this text

[Assistant] The research results on this uhm, don't really paint a very rosy picture and so the feedback that you get often is very limited in terms of the quantity and it's


[User] Please transcribe this text

[Assistant] How crowds might be engaged to offer deeper, more meaningful feedback. And I drew a lot of inspiration from work that Stephen Dow and Liz Gerber did. And what they had done is they used crowds to do A-B testing.


[User] Please transcribe this text

[Assistant] To collect preferences for which one you like, A versus B. And then as a designer you can get that signal and try to figure out what decision that you want to make.


[User] Please transcribe this text

[Assistant] And so I love this idea, but I want to try to push it a bit further and think about how to get crowds to provide more deeper, more meaningful feedback. The challenge is that crowds don't really have any expertise in design. So if I show you an idea,


[User] Please transcribe this text

[Assistant] And I say, what do you think of it? What do I expect somebody to say who doesn't have training in that discipline?


[User] Please transcribe this text

[Assistant] Okay, so that was, uh, a challenge. But if I asked the question in a different way, and I said, what's, you know, the first thing that comes to mind when you see that design? What's the first thing that you notice? How is my communication goal coming through to you? Then you can tell me your reaction. And that's something that you can control.


[User] Please transcribe this text

[Assistant] And that insight led to a system that we called, uh, voyant, which is a play on the word clairvoyant, helping you to see. Uhm.


[User] Please transcribe this text

[Assistant] And this is a system that leverages, uh, crowd workers, a non-expert crowd, to provide feedback


[User] Please transcribe this text

[Assistant] And the design of the system was predicated on the results of interviews that we did with people who had a wide spectrum of design experience from non-designers just doing design work all the way to, uhm, experts. And we asked them, if you had a large audience of non-experts, what would you want to ask them?


[User] Please transcribe this text

[Assistant] And from the responses we're able to extract, uh, five themes and we implemented those into the tool. And so you can get elements, this is literally what people can see, uhm, in the design.


[User] Please transcribe this text

[Assistant] What, first notice, so what people actually see first, and so establishing part of the visual hierarchy.


[User] Please transcribe this text

[Assistant] Guidelines, uhm, so we started with what's known as the CRAP, Humorously Principles, but Contrast, Repetition, Alignment, and Proximity. And what you would get with that is just a rating of 1 to 7 about how, how well people felt like your design was adhering to each of those principles, and very similar with the goals. So,


[User] Please transcribe this text

[Assistant] Goals are about your communication goals. Again, you get a rating from one.


[User] Please transcribe this text

[Assistant] So the way that it works is that you go to a publicly accessible website. It's called crowdfeedback.me if you have your web, uh,


[User] Please transcribe this text

[Assistant] Computer open, you can go there now. You upload your design image and here what you have is a poster design that was done as a student design project.


[User] Please transcribe this text

[Assistant] And the idea here was to try to engage, uh, customers in a restaurant to try a new smoothie station so that we put over the top and try to help them engage with that. Okay, so we're gonna try a new drink, grab attention. You can configure your audience.


[User] Please transcribe this text

[Assistant] So you can configure it based upon age, location, gender, and try to match that audience to the intended audience that you have for


[User] Please transcribe this text

[Assistant] The design. You can configure which of the types of feedback that you want, and for some of them you can add additional parameters. So if you configure the goals for instance, the system would prompt you to enter in, you know, with as few words as possible, make it accessible to a lay person. What is your communication goal for this design? You can enter as many as you want.


[User] Please transcribe this text

[Assistant] Once you click submit, it will decompose that task into a series of microtasks and put it onto Mechanical Turk.


[User] Please transcribe this text

[Assistant] And the, the workers are asked to do three things, very simple. First, give us your reaction. And so if it's the task about impression is tell us the first word or the phrase that comes to mind when you look at that design.


[User] Please transcribe this text

[Assistant] If it's a goal or a guideline, it's give us a rating on a scale of 1 to 7 of how well you think that design meets that rating or meets that guideline or goal. And the second thing is do an annotation. So,


[User] Please transcribe this text

[Assistant] Point out the elements in the visual design that relate to your reaction that you had. And the third thing, which is optional, is you can offer an explanation of why you gave the reaction and the annotation that you did.


[User] Please transcribe this text

[Assistant] And so we do this for a large number of people, and then we aggregate and present the results back to the user. And you get a visualization that looks something like this. This is the one for impressions, and if you're the user, this is what you see. Okay? And so in this case for impressions, we


[User] Please transcribe this text

[Assistant] Do the representation as a word cloud. And so all five types of feedback follow the same pattern. We give you an overview representation.


[User] Please transcribe this text

[Assistant] We give you uhm, the markers on the design that associate with each of the pieces of feedback and then the explanations. So here for example, you can select the different words in the word cloud and see which elements of your design went with that.


[User] Please transcribe this text

[Assistant] Alternatively, you can come over to the design and look at the markers and select a subset and see which words were associated with those markers.


[User] Please transcribe this text

[Assistant] And so the person who created this design found a number of interesting things from this.


[User] Please transcribe this text

[Assistant] Fresh, is that really what you're trying to communicate?


[User] Please transcribe this text

[Assistant] Uh, one of the interesting things that came out of the word cloud was the word sunny. And it turns out this was because of this element here. She had no idea that people were going to associate that with the sun. It looks like the sun to me, but it wasn't intentional on her part.


[User] Please transcribe this text

[Assistant] Another thing that was really interesting that she learned from the feedback


[User] Please transcribe this text

[Assistant] Was, people were associating the word strawberry, apple, and juice with the colors in the poster.


[User] Please transcribe this text

[Assistant] And she didn't really make that intentional. And what she discovered was that she could literally make people taste her design by choosing the appropriate colors. And so with a very simple set of feedback, she was learning really interesting things about what she was communicating, what she was not communicating, and how she might improve that.


[User] Please transcribe this text

[Assistant] The other set, another set of feedback that you get is the ratings of communication goals and here the goals that were entered were trying a new drink and grabbing attention. Okay, so again this is supposed to be above like a smoothie station in a restaurant and try to get people to come over and try it.


[User] Please transcribe this text

[Assistant] With this part, you get a data visualization that is an interactive bar chart.


[User] Please transcribe this text

[Assistant] Okay, and it assesses how well you're meeting your goals. So in this case, they did relatively well in both, but trying a new drink didn't rate as high as grabbing attention, and so you can maybe sharpen that particular goal on the next iteration.


[User] Please transcribe this text

[Assistant] The other thing that bar chart is interactive, and so you can select, um, different bars or subsets, and you can do this, um, to look at some interesting patterns that we've noticed. And so, for example, if you had a bimodal distribution where certain people really like something and others don't, you can compare and contrast that feedback.


[User] Please transcribe this text

[Assistant] Now there are additional types of feedback. You get elements and first notice and guidelines and it's very similar in terms of the data representation and the interactions that you have. And I would just refer you to the paper we have forthcoming on this and let me just talk a bit about the evaluation.


[User] Please transcribe this text

[Assistant] What I was really interested in the valuation is, can you actually learn anything about this feedback, about your design?


[User] Please transcribe this text

[Assistant] Okay, so we collected eight in-progress designs and it ran kind of the gamut. We had magazine covers, we had 3D renderings, we had graphic logos, we have webpage designs, and we worked with each of those, uhm, people who created the designs to understand their goals


[User] Please transcribe this text

[Assistant] We took, uh, their goals, we took their designs, we uploaded into our system, and once we had the feedback, we asked them to come back, and we talked about what they could learn as they walked through that feedback on their own.


[User] Please transcribe this text

[Assistant] So I just want to share with you one anecdote that kind of exemplifies what we found. And so in this case, this was a 3D rendering that the person who created it wanted to communicate two things. They wanted to communicate Chinese style and they wanted to communicate that it was a teapot.


[User] Please transcribe this text

[Assistant] Okay, and what they found from the feedback both from the impressions which are not shown and from the goals which are shown is that it didn't really communicate Chinese style very well and it didn't really communicate


[User] Please transcribe this text

[Assistant] A teapot. People actually perceive it as a coffee pot. And so based upon digging into the, uh,


[User] Please transcribe this text

[Assistant] Okay, so the point is that everybody learns something about their design. And in many cases, it wasn't just one set of feedback, it was the collection of things where they could synthesize, you know, interesting ideas.


[User] Please transcribe this text

[Assistant] Okay, well we also tracked how much it cost and how long it took. In terms of cost, it was about $10 and that was perceived as a fair trade-off because they didn't have to organize a critique or, uhm, repeatedly interrupt


[User] Please transcribe this text

[Assistant] I'm okay with that because finding problems is a significant outcome with the work. How do you iterate if you don't know what you're trying to address?


[User] Please transcribe this text

[Assistant] And so I can go through and optimize the cost by doing a better job of structuring my workflows. And I can use Michael's technique such as the retainer model to make it also ready fast. And by fast, I mean, what if I could give this to you like three to four times a day?


[User] Please transcribe this text

[Assistant] What would be the impact on design process if I could do that? I mean, I don't know, but I think it would be really, really interesting to think about how this is going to be integrated into the creative workflow. And another thing looking forward, what if I was to aggregate and collect?


[User] Please transcribe this text

[Assistant] All the data that you've got across all iterations and all projects that you've ever done. And I know a little bit about the data. I know they have word clouds. I know they have bar charts. It's connected to the design. I got explanations. I got uhm, the markers, right? What if I collected that over a long period of time?


[User] Please transcribe this text

[Assistant] And I created a visualization of that. What could you as a designer, what could you learn from that?


[User] Please transcribe this text

[Assistant] What else could you do with that? I think that's a really interesting thing. Go over to the short one.


[User] Please transcribe this text

[Assistant] And so right now all the audience comes back as being the same. And so there's many different ways that one wants to slice that audience. And so it could be that I'm gonna just separate based upon culture, the age, the gender. But one of the things that's really interesting is how do I separate on expertise?


[User] Please transcribe this text

[Assistant] So one of the challenges here, I think, is that crowd work often devalues expertise. Some people are much better at this type of task than others. So how would I differentiate away what people who are really good at that type of task, especially with the goals and the guidelines versus those


[User] Please transcribe this text

[Assistant] Who are not? Because right now we weigh them the same, I just don't think that's fair. Yeah, okay, but good question.


[User] Please transcribe this text

[Assistant] Okay, so the feedback from buoyant helps the individual make decisions on a design.


[User] Please transcribe this text

[Assistant] But, you know, decisions are often made in a group setting, not necessarily by the individual.


[User] Please transcribe this text

[Assistant] In online communities, for example, uhm,


[User] Please transcribe this text

[Assistant] The design discussions, or it's the community that often engages in design thinking in the discussions.


[User] Please transcribe this text

[Assistant] And uhm, I was really interested in how that transpires.


[User] Please transcribe this text

[Assistant] You have people who have never met and probably never will. They're using a fairly impoverished interface, which is the web forum, to engage in ideation about interesting design problems. So how does that work? How does it unfold? I really wanted to probe, uh, deeper about that.


[User] Please transcribe this text

[Assistant] One of the things that we had to do was pick a community that we wanted to study.


[User] Please transcribe this text

[Assistant] And we selected Drupal. It's a very well known, very successful open source content management system. It has about 500,000 websites that have been created with Drupal. It has roughly an equal number of people that have registered to contribute to that project.


[User] Please transcribe this text

[Assistant] The design discussions that relate to the user interface happen in this issue queue. So if you have a feature request or a usability problem that you want to report, you put it into the issue queue.


[User] Please transcribe this text

[Assistant] And other community members can come and join and talk about the issue, offer up concrete proposals for how to solve it, engage in deliberation, and then eventually if things go well, you can pick one and push it into, uh, the product.


[User] Please transcribe this text

[Assistant] And so we also needed to think about how we wanted to study those particular discussions and we chose the lens of what's called consensus building. Consensus building...


[User] Please transcribe this text

[Assistant] Is a distinct form of decision making where you value the social process of the discussion as much as the final outcome. And consensus is often determined to be reached when you have a decision and everybody can live with it.


[User] Please transcribe this text

[Assistant] We thought that was an interesting lens to look at because the design discussions often exhibit conflict due to people advocating strongly for their ideas and it's therefore really important for those discussions to unfold in a way that is respectful.


[User] Please transcribe this text

[Assistant] Uhm, and, well, respectful and courteous because the community depends upon people coming back and continuing to communicate and contribute to that, uhm, community.


[User] Please transcribe this text

[Assistant] So we analyzed a fairly large corpus of the data, a number of design discussions, uh, messages, patches, and uh, contributors. We had a very surprising finding.


[User] Please transcribe this text

[Assistant] About 42% of the UI design discussions do not reach consensus. It's a very significant finding because of all the potential consequences. So all the missed opportunities to improve the product


[User] Please transcribe this text

[Assistant] Uhm, all the effort that gets wasted into building the ideas, the deliberation, and it doesn't go anywhere.


[User] Please transcribe this text

[Assistant] And one of the problems is that that could cause people to leave the community and not come back. And it's actually not just Drupal.


[User] Please transcribe this text

[Assistant] Wikipedia has the same problem. So this is an article from a recent Wall Street Journal, uhm, uhm, article where they argued that there's a mass exodus of people from Wikipedia.


[User] Please transcribe this text

[Assistant] One of the really interesting things about this though, when I looked at that graph, is that maybe the right metric for consensus building,


[User] Please transcribe this text

[Assistant] It's not whether or not you reach an agreeable solution. It's whether you reduce the outflow of people from the community. That's a very different way of thinking about consensus building in communities than we have today.


[User] Please transcribe this text

[Assistant] But we think part of the reason is that the web forums that people use have no mechanism for consensus building at all. So we wanted to explore, you know, what would it mean from a technical side in terms of the interface to actually build in features that support consensus. This is a really challenging problem.


[User] Please transcribe this text

[Assistant] And so we were guided by two high-level principles. One is that we wanted to embed feasible and sound strategies into the interface. And our approach was to first enumerate many different strategies that we could locate in the literature, both from the theory and the practice of consensus.


[User] Please transcribe this text

[Assistant] A tool. We're not creating a human moderator. And so maybe as the first step, what we thought is let's try to give people a better discussion experience.


[User] Please transcribe this text

[Assistant] And maybe, you know, in the future we can try to lead and convert those discussions to a consensus, but maybe that wouldn't be a goal we could reach in the near, near term.


[User] Please transcribe this text

[Assistant] And we wanted to integrate into the existing community. And this also is a really difficult challenge. But all the work in Drupal, it happens in that issue queue. It's where the designers are. It's where the developers are. It's where all the work gets done. And so we wanted to have an impact on the community


[User] Please transcribe this text

[Assistant] And learn and discover principles of consensus through authentic use, as opposed to just creating a one-off interface that people would use, say, in the lab.


[User] Please transcribe this text

[Assistant] It's a script. It runs inside the client browser. And what it does, it basically rewrites the HTML, uhm, with the Drupal server. But we also use our own server to track certain state information that we need to render the interface the way that we do.


[User] Please transcribe this text

[Assistant] And so what you're looking at here, and the first thing I want to point out is on kind of the right side, this is the actual discussion in the Drupal community. We do not modify it at all.


[User] Please transcribe this text

[Assistant] The interface works with any discussion in that community that you want to go look at and participate in.


[User] Please transcribe this text

[Assistant] What we do is we basically wrap around the top and along the left. And so we have three different pages that we offer. The home page is what you see here. What we call an idea page with our show and an invitation page with our show.


[User] Please transcribe this text

[Assistant] One of the basic elements of consensus is you have to know what's going on in the discussion. So on the left panel, we give you a number of lenses to help you do that. We help you filter comments based upon the experience of the person who submitted the comment.


[User] Please transcribe this text

[Assistant] Patches and so the benefit is that you can use those lenses to help track and revisit key points of the discussion.


[User] Please transcribe this text

[Assistant] A second strategy is including users who can help contribute to moving that discussion forward.


[User] Please transcribe this text

[Assistant] And the way that we do this is through the invite page and we generate a list of all the community members and we show how they score on different criteria that we know to be important for consensus in the domain. So we talk about how much experience they've had in the community, the number of patches that they contributed, how many threads that they've participated in that have been closed or as a proxy for consensus.


[User] Please transcribe this text

[Assistant] And uhm, the number of interactions they've had with people in the current ongoing discussion.


[User] Please transcribe this text

[Assistant] And so you can use this to help fill perceived gaps. And so if you're in a discussion and you're like, and you wish that you had more experienced participants, you can come to this page and look at those who have more experience than what you guys have currently. And you can find those who have bridges to people who are already engaged in the discussion.


[User] Please transcribe this text

[Assistant] Because one is that we'll create an opportunity maybe to bring them in and second we know that it's much easier to reach consensus with people that we know than people that we don't.


[User] Please transcribe this text

[Assistant] I want to talk about the idea-centric page, but first I want to tell you about how do we know about ideas. And so we have a very clever strategy. We ask.


[User] Please transcribe this text

[Assistant] And so what we do is we rewrite the HTML on the page for authoring a comment. And we inject a very small amount of input that we request when you're authoring that comment. We simply ask, does this comment propose a concrete idea?


[User] Please transcribe this text

[Assistant] If not, does it refer to an idea? And if so, which comet number does it go with? And then tell us your disposition. Do you strongly object? Up to strongly support the idea that you're referring to? Now this has been a very long, uhm, thread of research in CSW, uh, work. Where basically what we say is that,


[User] Please transcribe this text

[Assistant] If you just give us a little bit of input, we can render the discussion better for you.


[User] Please transcribe this text

[Assistant] The problem, however, is that it's easy to fall into the trap to over-engineer the system.


[User] Please transcribe this text

[Assistant] Think design rationale systems, for instance, where without any kind of bound on what you're asking for the structured input, you just go too far, it becomes too cumbersome to use. And so we think this is interesting because.


[User] Please transcribe this text

[Assistant] So assuming that the users will give us that small amount of input, then we can render what we call an idea-centric page. And this is trying to organize the ideas.


[User] Please transcribe this text

[Assistant] You get a set of user defined criteria which I'll come back to and you get a set of comments that are organized in different channels. And so the top one is


[User] Please transcribe this text

[Assistant] Supportive comments. The middle one is neutral comments. And the bottom one are uhm, what we call constructive comments, which is a polite way of saying it's a critical comment. And so what you can see is that you can look for ideas that need further attention. You can look for conflict.


[User] Please transcribe this text

[Assistant] You can look for things that already have good support and things that have the lack of support. We do one other thing.


[User] Please transcribe this text

[Assistant] If you give me an extreme position, either I really don't like this or I object to it strongly or I strongly support it, we render it as a slightly larger circle to call attention to it by the other community members and give you a sense that you have a stronger voice in the discussion.


[User] Please transcribe this text

[Assistant] Okay, you can establish criteria, and the way you do the criteria is you select the icon here next to the criteria, uhm, uh, column, and then it prompts you with a dialog, and you can either modify what's there,


[User] Please transcribe this text

[Assistant] Or you can add an initial set. So if you add another piece of criteria, it gets replicated for every idea in that same column.


[User] Please transcribe this text

[Assistant] And what you end up with is criteria that's user defined, it's visually persistent, and it's interactive.


[User] Please transcribe this text

[Assistant] And so if you wanted to rate how an idea, uhm, adheres to one of the criteria, you can do that with a slider, and anybody in the community can do this. The system prompts you to enter a comment with it to create accountability, so it's unlikely that you're going to make a modification unless you feel strongly about that change should happen.


[User] Please transcribe this text

[Assistant] The last one that I'm going to talk about that we actually do more than this is promoting a constructive


[User] Please transcribe this text

[Assistant] And we simply remind you why it's a good thing to have a constructive tone. And you can just choose to go forward and post it as is, or you might go back and revise it. Okay, it's up to you. And the second way is...


[User] Please transcribe this text

[Assistant] To promote a constructive tone is that we have a constraint in the system. So it's that when you have a new idea, the first comment has to be supportive. And so we try to extract the good of an idea.


[User] Please transcribe this text

[Assistant] It's an early prototype. We know that we have a lot more work to do. But we had it to the point where we could actually get some feedback from the community members themselves. And so the way that we did this is we did it as a walkthrough. We found some active discussions that had some tension in them where people were advocating strongly for their ideas.


[User] Please transcribe this text

[Assistant] And we downloaded the discussions and we mirrored them on our server and we constructed and rendered that.


[User] Please transcribe this text

[Assistant] We gave them a link to their own discussion so they could see it and we gave them a set of tasks to experience the different key features of the system.


[User] Please transcribe this text

[Assistant] And so, you know, overall, we got very favorable reactions from the committee members, which I was really excited about.


[User] Please transcribe this text

[Assistant] And their favorite aspects were the idea-centric and views and the, um, the criteria lists were the things they liked the most. And in terms of their subjective responses, they rated our tool as being more effective for consensus building than the current web forum that they use. And I know that's not definitive, but it's at least promising.


[User] Please transcribe this text

[Assistant] Another couple interesting things that we found is that they desire minimal change. They have such a...


[User] Please transcribe this text

[Assistant] And that's exactly what we've done and we're just now trying to look at where that discussion is heading and starting to collect some data.


[User] Please transcribe this text

[Assistant] And looking forward, I mean one of the things I think is really intellectually challenging is


[User] Please transcribe this text

[Assistant] How do you measure the impact of a tool like this in a community? And when I first approached the problem, I was thinking, wow, we should really just be looking at the number of discussions that now reach consensus that didn't before.


[User] Please transcribe this text

[Assistant] But when I started looking at the data and looking at, you know, the graphs of say Wikipedia, it started to think about well maybe that's not the right metric after all. Maybe what we should really be looking at.


[User] Please transcribe this text

[Assistant] It's whether or not we reduce the outflows of volunteers from this type of community and maybe there's other metrics that we can use too.


[User] Please transcribe this text

[Assistant] Okay, so let me, uhm, offer a couple of pathways forward for future work and then I will conclude.


[User] Please transcribe this text

[Assistant] And that really spawned 40 years of research into how we do better idea generation.


[User] Please transcribe this text

[Assistant] But now I wanna argue that the world has changed. And ideas are now cheap. And one of the reasons that we're cheap is because we can engage crowds. So for example, this is from Starbucks, mystarbucksideas.com.


[User] Please transcribe this text

[Assistant] And what they do is they engage the crowd to give them ideas for how they might improve products or consumer experiences. And people do engage with that. It generates about a thousand ideas a month, if not more.


[User] Please transcribe this text

[Assistant] And so what I'm arguing is that the problem is no longer that we don't have enough ideas, we have too many.


[User] Please transcribe this text

[Assistant] And so future design tools, especially those that try to engage with the crowd, need to think more about idea selection and not just idea generation. And maybe that's just a stepping stone to something even bigger where we really think about how crowds change the creative workflow throughout. And I think that's something very interesting to think about in the future.


[User] Please transcribe this text

[Assistant] About how do we help people select the best or the most innovative idea and overcome the psychological bias of always choosing the safest or the most familiar idea, which we know happens a lot. And the second pathway I want to talk about just briefly is


[User] Please transcribe this text

[Assistant] The motivation of creative effort. So I'm a really big fan of Teresa Ammobile's work.


[User] Please transcribe this text

[Assistant] Who argues that intrinsic motivation is one of the three, one of three key ingredients for creative work.


[User] Please transcribe this text

[Assistant] Look, I can give you examples, I can give you feedback, uhm, I can give you spatial maps, but unless you're motivated to translate that into creative outcomes, it's probably not going to happen.


[User] Please transcribe this text

[Assistant] How do you build technology that motivates creative effort? That's really interesting. And so as an analogy,


[User] Please transcribe this text

[Assistant] Uhm, I was thinking about Nike's social media application, Nike Plus Run, and it's one of my favorite ones. And what you can do with this before you go on a run is you can invoke the application and put a link onto your Facebook page and people can cheer you on.


[User] Please transcribe this text

[Assistant] And I suppose in the headphones it's like go, go, go, go, right? Having been a writer in my earlier years, I would find that really motivating. What is the equivalent for design thinking?


[User] Please transcribe this text

[Assistant] On that rhetorical note, sorry for the cliffhanger, uhm, I'm going to conclude.


[User] Please transcribe this text

[Assistant] And uhm, I've made a number of contributions to design thinking tools for individuals, for groups, and for communities. Uhm, I've talked about the work we've done on spatial maps. We've developed visualizations to help extract uhm, meaning for prior activity.


[User] Please transcribe this text

[Assistant] We showed how to use non-expert crowds to generate feedback on, uh, preliminary designs. And we talked about some results and a tool that supports consensus building in online communities. And so thanks for your attention. I can take questions now.


[User] Please transcribe this text

[Assistant] I think you can do the same thing here. And so on multiple iterations, or even if you want to do an exploration at certain points, you can compare and contrast if you're improving. So one of the things that I really want to do is I'm really sensitive to engineering trust into the system.


[User] Please transcribe this text

[Assistant] And if you make what you think to be really good improvements on your design, and it comes back as having worse feedback, you're gonna call into question whether or not that's meaningful or not.


[User] Please transcribe this text

[Assistant] So, what I really want to do is try to help people elaborate what the changes are, look and see if the crowd gives you an improved response, and then compare that to the changes that you said that you made. And hopefully give people confidence that the system's actually, uhm, recognizing when you make improvements.


[User] Please transcribe this text

[Assistant] They're not really going to have a lot of confidence in the system, so I want to give them confidence that we give them consistent feedback. And if they actually make improvements, the crowd recognizes that those improvements were made.


[User] Please transcribe this text

[Assistant] And what that tells me is that we don't really know how to do this very well yet. And so hopefully we can kind of discover some principles and communicate that and they get engendered in some of these other, other tools. Yeah.


[User] Please transcribe this text

[Assistant] But hopefully I can come back and give you a data point. Uhm, will people make changes? I, I actually think that they will.


[User] Please transcribe this text

[Assistant] Uhm, but I don't know how suggested that we need to be. Right now I'm very kind of, I don't want to make interpretations for them. I want them to come to their own conclusion that they need to make an adjustment. But I, I...


[User] Please transcribe this text

[Assistant] I think, I think they will. I really do. I'm, I'm very optimistic about that. Uhm, cause when you see it, and it's in front of you, and you look at all the different words that I've highlighted, it has just kind of a visceral kind of impact on you. It's like, wow, I'm really not being very positive. And so you might go back and change it. And there's, you know, we know that,


[User] Please transcribe this text

[Assistant] You know, from reviews and doing reviews, we know that there's ways to say things exactly the same way, but it's more constructive. And that's what I really want people to do. I mean, you have to change what you say. Just make it more constructive in the way that you communicate it to the other person.


[User] Please transcribe this text

[Assistant] And it's really important because that social process is such a key element. It's so easy to leave that community that if you get frustrated and you're upset about what somebody says, you can just leave and not come back.


[User] Please transcribe this text

[Assistant] Well, of course there's many different solution possibilities that one has here. Right? And so, you know, we had one that we thought was effective, but we'd be open to changing it. But we map your disposition, in other words, if you like it or not, we map it to space. Right? So the top channel,


[User] Please transcribe this text

[Assistant] Gives you what people, you know, the people who like it and the bottom channel gives you those who do not or want to be constructive. So we just map it onto space to be direct about your question.


[User] Please transcribe this text

[Assistant] So you can look for things that if you don't have anything in the bottom row, that means that people are very supportive. There's a lot on the top. If there's an equal number top and bottom, and there's a lot of them, that means there's tension and conflict.


[User] Please transcribe this text

[Assistant] If there's only things on the bottom and nothing on the top row, that means that, well, maybe that's not really gonna go anywhere. And then the ones that are rendered, like, a little bit larger, like, wow, I really, like, either really do like this or I really don't. And it calls attention to it. That's what we're trying to do. It's so easy to miss comments.


[User] Please transcribe this text

[Assistant] In the, in the discussion. And you feel as if I said something and nobody's listening to me. Here we call it out. So if you really don't support it, or you really do, it's apparent.


[User] Please transcribe this text

[Assistant] And a way to allow us to probe what people really want in that community and then be responsive to that and try to iterate on the design. So if certainly other things come up like that, we'd be happy to incorporate it.


[User] Please transcribe this text

[Assistant] And so now what you can do is you can engage crowds through your local workspace. So literally with, you know, a pull down, say in Photoshop, you can engage that crowd.


[User] Please transcribe this text

[Assistant] And now it's probably cost too much, it takes too long, but you know, what if we can do this in a couple of hours and get you feedback three to four times a day? That's where we really want to go. And so we're making the feedback really inexpensive to get. And so now when you try out different things, you can immediately test it and then use that to have very tight iterations. I think that's really exciting.


[User] Please transcribe this text

[Assistant] Yeah, that's exactly what you can do. And so what we wanted to do is not just think about a visualization as output, but think about the visualization as input. And so we can offer some constraints through this way. And so this is where we have this constraint that if you have a newly submitted idea that has no comments yet, then the first one needs to be in the top row.


[User] Please transcribe this text

[Assistant] That way people see the good of the idea and can read the good about the idea before they start making constructive comments. And the part is, again, about the social process, you know, making people feel better about the discussion. And even if my idea doesn't get picked, well at least people said something that was positive about it, right?


[User] Please transcribe this text

[Assistant] And what this sort of will revolve around is my work at the interplay between learning and control, particularly for robotics. Uhm, so what this sort of motivated by is that uhm, if you look at each of these robotic systems on their own, at least from a dynamics perspective,


[User] Please transcribe this text

[Assistant] We're pretty good at coming up with some sort of first principles model of each of these systems.


[User] Please transcribe this text

[Assistant] Uhm, and sort of, these are the reasons why we sort of turn to data-driven learning. Where we can't write down the model by hand, so we look to data and try and figure out, uhm, a model for the system overall, or at least some addendum to the model, some model we already have.


[User] Please transcribe this text

[Assistant] Uhm, but a lot of the fundamental, uhm, and sort of naive ways you might think about doing learning from machine learning don't always work very well in practice when you're talking about closed loop control. So, for example, naive regression, where you just minimize, you know, two norm less, uhm, may work well in minimizing one step open loop error.


[User] Please transcribe this text

[Assistant] Uhm, but control tasks happen over long time horizons in closed loop.


[User] Please transcribe this text

[Assistant] So regression, what you might find is that regression for the system and then deploying a learned model, uh, for the system with, that was learned with naive regression would diverge over time with compounding errors over multiple time steps during close loop control. So what we need to do,


[User] Please transcribe this text

[Assistant] At least what I argue in this talk is that we need to orient the learning problem around what we're ultimately interested in, which is closed loop control or the downstream control objective is the terminology that we use. So rather than just the model fitting objective.


[User] Please transcribe this text

[Assistant] So the idea, which we sort of call control-oriented learning, is that we should condition the model to perform better and be more adaptable in closed loop, in comparison to models that are learned naively to minimize open loop error.


[User] Please transcribe this text

[Assistant] Uhm, so today I'll discuss three different lines of work that we've developed, time permitting, to orient learning around closed loop control. The first one involved translating the concept of stabilizability or controllability into algebraic conditions that we're going to use as a regularizer for closed loop control performance during offline learning.


[User] Please transcribe this text

[Assistant] The second take is a little bit different, uh, it involves structuring our model appropriately offline in a manner that's generally applicable, but it naturally yields a useful control, uh, close-up controller based on LQR.


[User] Please transcribe this text

[Assistant] Uhm, and then the final work I'll talk about is shifting more towards the online setting and specifically using meta-learning to learn a good extension to a nominal model that you specify maybe through first principles and learning an extension to it that we can adapt well online for adaptive control. Okay?


[User] Please transcribe this text

[Assistant] So I'll get started with the first project, which was joint work with some folks at Google and, uh, MIT. Uhm, and it's going back to this regression problem.


[User] Please transcribe this text

[Assistant] Uhm, and what we want to do is integrate some notion of stabilizability or the ability to stabilize the system as a constraint with the key idea being that what this should do is prune the hypothesis base of functions that we're fitting over, that we're searching over.


[User] Please transcribe this text

[Assistant] Uhm, and finding one that somehow attuned to closed loop control.


[User] Please transcribe this text

[Assistant] So the key question, so in stabilizability what I mean by that is there exists a controller.


[User] Please transcribe this text

[Assistant] That, as a function of my current state, where I am, where I want to be, x star, and ensures that x of t, my current state, goes towards my x star of t over time. That's what I mean by stabilizability, in the general sense. Okay?


[User] Please transcribe this text

[Assistant] The key thing we need to sort of overcome here is if we want to try this out is we have to be able to describe stabilizability algebraically to construct an optimization problem that we could solve to fit a model. So to do this, we turn to something called contraction theory.


[User] Please transcribe this text

[Assistant] Uhm, so if we look at, let's consider control offline dynamics, x dot equals f of x plus beta x times u. Uhm,


[User] Please transcribe this text

[Assistant] And what contraction theory starts off by doing is linearizing the dynamics, but maintaining the dependence on your linearization point. So you look everywhere.


[User] Please transcribe this text

[Assistant] State and input wise, and you look at the local linear dynamics or the variational dynamics to be more specific.


[User] Please transcribe this text

[Assistant] This is a family of linear systems, and it's an exact description of small variations of your dynamics near your linearization point.


[User] Please transcribe this text

[Assistant] And this is a linear system, at least in your variations, delta x and delta u. So what you can do is, we can describe, we can construct this notion of local energy or local distance. Uhm, and this is a square distance, but it's weighted by some, uhm, a metric called m of x, it's a positive definite function, a matrix function.


[User] Please transcribe this text

[Assistant] It, uh, basically generalizes the notion of distance and allows it to depend on your linearization point.


[User] Please transcribe this text

[Assistant] Like I said, we're sort of generalizing the notion of squared distance with some, some quantity that allows it to depend on where you're linearizing it.


[User] Please transcribe this text

[Assistant] Uhm, and this thing that we're weighting the distance with is what's known as a control contraction metric, or CCM. And the idea is that we have a quadratic notion of energy, we have a linear system.


[User] Please transcribe this text

[Assistant] So anybody who's done any sort of control theory that should scream out to you, do some sort of linear control design. That, that's sort of our bread and butter as control theorists is linear control design with quadratic costs or quadratic energy functions or Lyapunov functions. So uhm, the idea is that if we can design a,


[User] Please transcribe this text

[Assistant] Linear controller, a local linear controller, a variational controller that's linear in the variation in x. And make, design in such a way that this differential energy is decreasing locally everywhere. Okay?


[User] Please transcribe this text

[Assistant] Well if it's decreasing locally everywhere and I integrate that differential control dealt to you between


[User] Please transcribe this text

[Assistant] Uh, two points, x and x star, so long shortest paths between x and x star. I back out another control, and what, through some,


[User] Please transcribe this text

[Assistant] Uh, a lot more math than I intend to get into in this presentation. We can show that by integrating out control for the original system, we can get something that's stabilizing for the original system.


[User] Please transcribe this text

[Assistant] And it shows, it makes, basically it makes x of t go towards x star of t. So, the metric m of x together with this


[User] Please transcribe this text

[Assistant] Energy decreased condition, which I haven't really put any specifics on yet.


[User] Please transcribe this text

[Assistant] So we have dynamics and we've introduced this additional function along with some conditions that need to be satisfied to make the system stabilize. So it's a, so uhm,


[User] Please transcribe this text

[Assistant] Uhm, the idea that, there's a few things I want to point out, is that it depends on the dynamics, which go into, now our model fitting objective, the top,


[User] Please transcribe this text

[Assistant] And we also have this decree condition that ends up looking like a matrix inequality that


[User] Please transcribe this text

[Assistant] And it acts as a certificate for the stabilizability of the thing, the dynamics functions that you're fitting to data. Okay?


[User] Please transcribe this text

[Assistant] And there's no, there's no data associated with these constraints. There's no output or labels for this data.


[User] Please transcribe this text

[Assistant] It's, I can sample as many axes in the state space as I want. So I can cover the state space as densely or as sparsely as I want.


[User] Please transcribe this text

[Assistant] The only labels that sort of show up are in the actual model fitting objective. So,


[User] Please transcribe this text

[Assistant] The other, the other problem here is that it's non-convex. You have matrix inequalities at multiples of f and w.


[User] Please transcribe this text

[Assistant] Uhm, but it's bi-convex. So what we do to solve this, we linearly parameterize our metric and dynamics, and alternating, and do alternating convex, uh, solves. Until we converge to some local, uh, optimum.


[User] Please transcribe this text

[Assistant] And so, we actually did this on hardware, with real data, with a quadrotor. And on the left, we've done sort of naive model fitting. And on the right, we've done our approach.


[User] Please transcribe this text

[Assistant] And what we find is that uhm, the one on the left is not gonna do so well. It's gonna start oscillating a bit. Right here.


[User] Please transcribe this text

[Assistant] Then it starts to scroll out of control and ends up crashing. While this one, on the right, manages to


[User] Please transcribe this text

[Assistant] And keep in mind, this is sort of a quadruple model that's sort of learned from scratch. So it's sort of, it's, we're not providing any base model for it. Sort of wanted you to see how well does this method work for learning a model from scratch. Okay?


[User] Please transcribe this text

[Assistant] So with that, uhm, I'll move into sort of the second project which sort of takes a different spin on things. Uh, and it'll still be an offline model work, uh, learning. And this was with some folks at MIT. But it sort of takes the approach instead of trying to


[User] Please transcribe this text

[Assistant] Come up with an addendum, like an extra function, an extra certificate function. It looks at how, how do we do control? How do we?


[User] Please transcribe this text

[Assistant] So, for a mechanical system, for example, it has this nice ligand gene form that, for example, you could do feedback linearizing control on it.


[User] Please transcribe this text

[Assistant] And it depends on the, you know, the second order form is specific to yielding that controller.


[User] Please transcribe this text

[Assistant] Something that's a little more fundamental are linear systems. When we have linear systems, we have a lot of good, like I said, bread and butter for control theorists. Linear systems, there's a lot of theory in designing, stabilizing controllers.


[User] Please transcribe this text

[Assistant] Specifically, we can, you know, specify quadratic cost and do LQR control, which is optimal for linear systems.


[User] Please transcribe this text

[Assistant] So it's this idea of structure informing control design that we're sort of latching onto here.


[User] Please transcribe this text

[Assistant] It turns out that this linear-like structure can also be identified for nonlinear systems. So, uh,


[User] Please transcribe this text

[Assistant] If we have f of x plus b of x times u again, let's say that we can always factor this, it's fundamental theorem of calculus, we can always factor this as f at zero plus a of x times x.


[User] Please transcribe this text

[Assistant] Some constant offset plus all the terms in your Taylor expansion. That you can always factor out an x from.


[User] Please transcribe this text

[Assistant] Uhm, and for such systems, okay, well this looks like a linear system, so we can do, you know, we can solve a Riccati equation at each feedback timestamp for each x we encounter, and do LQR-like control for it. It turns out in practice this has worked pretty well, uhm, and,


[User] Please transcribe this text

[Assistant] Uhm, so this, a similar factorization exists for aerodynamics, for any control affine system. So the idea of being if you


[User] Please transcribe this text

[Assistant] Write the dynamics of x tilde dot, where x tilde is x minus x star, so where you are minus where you want to go.


[User] Please transcribe this text

[Assistant] It turns out you can factor in a similar way where you have some matrix function times x tilde.


[User] Please transcribe this text

[Assistant] And you can solve a state, what's called a state-dependent Riccati equation, and do feedback in a nonlinear manner, using linear-like tools. And it works, uh, pretty well in practice.


[User] Please transcribe this text

[Assistant] There's some, I think there's some analogy with what we talked about before with the certificate based feedback with contraction theory. So uhm,


[User] Please transcribe this text

[Assistant] CCM based, or contraction theory based feedback, you, as long as you have a decreased condition satisfied, you're guaranteed to be exponentially stable. It's a guarantee.


[User] Please transcribe this text

[Assistant] Where this bottom line here is just a decreased condition I've written out.


[User] Please transcribe this text

[Assistant] Uhm, for, uh, the closed loop system with some controller pi. And without getting


[User] Please transcribe this text

[Assistant] You don't have to know every term in here, but the critical thing is to notice that it depends on


[User] Please transcribe this text

[Assistant] Three different things you're learning. Dynamics, controller, and some certificate function. It's a lot of things to try and fit to data. Uhm, in contrast, this


[User] Please transcribe this text

[Assistant] Riccati equation based feedback, it's a heuristic, it's locally optimal, uhm, it's no guarantees on stabilizability, but it does work well in practice.


[User] Please transcribe this text

[Assistant] Uhm, and what you're doing is you're only depending on some factors forming the dynamics and then you're doing Riccati equation based control. You're not learning a separate certificate function, you're not learning a separate controller.


[User] Please transcribe this text

[Assistant] The structure of the dynamics yields your controller design.


[User] Please transcribe this text

[Assistant] So what prior work has done is they've, what they've tried to do is


[User] Please transcribe this text

[Assistant] Let's try and, let's take this regression loss, let's take this certificate decreased condition, turn it into an auxiliary loss term, and try and fit all these functions to data at the same time.


[User] Please transcribe this text

[Assistant] Try to extract the structure out of the dynamics. Just learn the dynamics and the structure that enables Riccati equation based control. With an auxiliary loss that sort of, uh, ensures that these factorizations are, uhm, are valid or consistent.


[User] Please transcribe this text

[Assistant] So instead, focus on learning the dynamics and some details about them, and do Riccati equation based control. So, in a sense that you're learning less, but the hope is that it still yields an effective controller.


[User] Please transcribe this text

[Assistant] So this is precisely what we did. So we did, in simulation we did this planar quad example where we wanted to track this double loop trajectory.


[User] Please transcribe this text

[Assistant] How much data do we need for each different type of method to do well?


[User] Please transcribe this text

[Assistant] And on the bottom I'm going to plot the closed loop trajectory error. So uhm, this is a baseline linearization based LQR. So linearize around your current point, solve our quad equation, and apply that control. Okay, so this is the oracle, so to speak, with the known dynamics. So it does pretty well. Uhm, then we did, so this method is,


[User] Please transcribe this text

[Assistant] Basically fit your dynamics to data, just do naive regression and do linearization based LQR on that model. And we find it takes a while before it starts to do well.


[User] Please transcribe this text

[Assistant] We then tried out the CCM base, basically the idea of learning dynamics, controller, and certificate. And again, also struggles a bit to get up to the same level of performance.


[User] Please transcribe this text

[Assistant] So this is purely learning the dynamics and at the same time learning some structural dynamics to enable nonlinear LQR-like control.


[User] Please transcribe this text

[Assistant] Uhm, and in this plot, we did it through two different systems, and along the x-axis is again the data set size, and on the right, on the upper, uhm,


[User] Please transcribe this text

[Assistant] Vertical axis is, uhm, sort of RMS, trajectory error.


[User] Please transcribe this text

[Assistant] And, uh, we've taken sort of multiple random seeds and figured out the distribution in this, these results. And we get similar case where, uhm, as you increase the amount of data, you could do better. But our method in green already does really well for low amounts of data. And in fact, for enough data, it beats out the oracle.


[User] Please transcribe this text

[Assistant] And the key thing here is to remember that we're learning a controller that's inherently non-linear while the oracle is using a linearization-based controller. So remember I said that


[User] Please transcribe this text

[Assistant] So we can find, even now when we're learning the dynamics, we still get that happening.


[User] Please transcribe this text

[Assistant] So, overall what we've seen is that enforcing this factorized structure, uhm, enables a type of control.


[User] Please transcribe this text

[Assistant] I might have to go a little fast to this next section. Uhm, forgive me, but uhm, we'll move into sort of the final section, which will be more about the adaptive and online setting. So, last two were about learning a model offline and deploying it online.


[User] Please transcribe this text

[Assistant] Uhm, but like I said, often, we're pretty good at driving at least some basic model by hand. And then maybe what we want to do is uhm, so fine tune it online. So uhm,


[User] Please transcribe this text

[Assistant] Here we have a typical feedback loop from state to controller to system and then you have your feedback.


[User] Please transcribe this text

[Assistant] And usually in a controller, like I said, we're relying on some knowledge of the system, and the system model that we're using consists of maybe two things, both structure and parameters.


[User] Please transcribe this text

[Assistant] Structure could be the fact that force equals mass times acceleration.


[User] Please transcribe this text

[Assistant] Parameters are maybe the mass and shape of your payload. Maybe those are uncertain. Especially if you're doing pick and place operations, your mass.


[User] Please transcribe this text

[Assistant] So what adaptive control does is it says, okay, well, maybe we don't know the value of these parameters, but we maintain estimates and try and change them on online, and learn them online.


[User] Please transcribe this text

[Assistant] And what's interesting is that a defining characteristic of adaptive control is that it can guarantee tracking convergence.


[User] Please transcribe this text

[Assistant] But parameter convergence is only guaranteed as a secondary result. So you know, parameters don't have to converge, but you'll get tracking convergence.


[User] Please transcribe this text

[Assistant] Naturally, a, uhm, a form of control-oriented learning or adapting parameters on a need-to-know basis for control.


[User] Please transcribe this text

[Assistant] But for a drone in wind conditions, it's hard maybe to derive.


[User] Please transcribe this text

[Assistant] So the idea is that what we propose to do is learn this model, a model for these forces, and the adaptive part is maybe just the last layer of this deep model that we work with.


[User] Please transcribe this text

[Assistant] And I think I'm running out of time, but the idea is that if we have data from a drone from multiple different days when we're flying it or multiple different conditions that we fly it in,


[User] Please transcribe this text

[Assistant] So the idea of meta-learning is learn some shared structure, some shared features for each of these different conditions.


[User] Please transcribe this text

[Assistant] And adapt some parameters online that are specific to either task.


[User] Please transcribe this text

[Assistant] So this is the concept of meta-learning, and uhm,


[User] Please transcribe this text

[Assistant] What prior work has done is done this with a regression objective. Basically differentiating through least squares.


[User] Please transcribe this text

[Assistant] And then backpropagate through a simulat- closed loop simulation, instead of just backpropagating through least squares objective.


[User] Please transcribe this text

[Assistant] Baselines are in orange and green, and our method is in blue, so you can see a drastic difference in performance there. Apologize for rushing at the end, but uhm, I'll turn it over to questions I guess now. So, if you have any questions, including in the last part, go for it.


[User] Please transcribe this text

[Assistant] I think, I mean, the method is pretty, uhm, agnostic to the system. If you look previously, we did it in two different systems where


[User] Please transcribe this text

[Assistant] So the difference between some of the baselines was not so drastic for this system because it was closer to being linear.


[User] Please transcribe this text

[Assistant] With contraction theory, there are some assumptions about the system that you would, that make it simpler to apply these, uhm, inequality conditions to construct them.


[User] Please transcribe this text

[Assistant] One of which is some sparsity in that b of x.


[User] Please transcribe this text

[Assistant] Which, for example, from mechanical systems, uhm, we hope that we can divide.


[User] Please transcribe this text

[Assistant] Uhm, are actuated degrees of freedom and are unactuated degrees of freedom. And that comes up with some sparsity in B of X.


[User] Please transcribe this text

[Assistant] Uhm, I think much like in model order reduction, you'd have to maybe specify how many states are uncontrollable.


[User] Please transcribe this text

[Assistant] A little before two o'clock, Trumpkin and the badger sat with the rest of the creatures at the wood's edge, looking across at the gleaming line of Mirage's army, which was about two arrow shots away.


[User] Please transcribe this text

[Assistant] In between, a square space of level grass had been staked for the combat.


[User] Please transcribe this text

[Assistant] At the two far corners stood Glozell and Sopespian with drawn swords.


[User] Please transcribe this text

[Assistant] At the near corners were Giant Wimbleweather and the Bulgy Bear, who in spite of all their warnings was sucking his paws and looking, to tell the truth, uncommonly silly.


[User] Please transcribe this text

[Assistant] To make up for this, Glenstorm on the right of the lists, stock still except when he stamped a hind foot occasionally on the turf, looked much more imposing than the Telmarine Baron who faced him on the left. Peter had just shaken hands with Edmund and the Doctor and was now walking down to the combat.


[User] Please transcribe this text

[Assistant] It was like the moment before the pistol goes at an important race, but very much worse.


[User] Please transcribe this text

[Assistant] I wish Aslan had turned up before it came to this, said Trumpkin.


[User] Please transcribe this text

[Assistant] So do I, said Truffle Hunter, but look behind you.


[User] Please transcribe this text

[Assistant] Huge people, beautiful people like gods and goddesses and giants.


[User] Please transcribe this text

[Assistant] It's the Dryads and the Hammer Dryads and Sylvanes, said Truffle Hunter. Aslan has wakened them. Hmph.


[User] Please transcribe this text

[Assistant] Said the dwarf, that'll be very useful if the enemy try any treachery, but it won't help the High King very much if Miraz proves handier with his sword.


[User] Please transcribe this text

[Assistant] The badger said nothing, for now Peter and Miraz were entering the lists from opposite ends, both on foot, both in chain shirts with helmets and shields. They advanced till they were close together.


[User] Please transcribe this text

[Assistant] Both bowed and seemed to speak, but it was impossible to hear what they said.


[User] Please transcribe this text

[Assistant] Next moment, the two swords flashed in the sunlight.


[User] Please transcribe this text

[Assistant] For a second the clash could be heard, but it was immediately drowned because both armies began shouting like crowds at a football match.


[User] Please transcribe this text

[Assistant] Well done, Peter! Oh, well done! Shouted Edmund as he saw Miraz reel back a whole pace and a half. Follow it up quick!


[User] Please transcribe this text

[Assistant] But then Miraz pulled himself together, began to make real use of his height and weight.


[User] Please transcribe this text

[Assistant] Miras, miras, the king, the king, came the roar of the Telmarines.


[User] Please transcribe this text

[Assistant] Caspian and Edmund grew white with sickening anxiety.


[User] Please transcribe this text

[Assistant] Peter is taking some dreadful knocks, said Edmund. Hello, said Caspian, what's happening now?


[User] Please transcribe this text

[Assistant] When there was such a clapping and baying and throwing up of hoods among the old Narnians that it was nearly deafening.


[User] Please transcribe this text

[Assistant] What was it? What was it? Ask the doctor. My old eyes missed it.


[User] Please transcribe this text

[Assistant] The high king has pricked him in the armpit, said Caspian, still clapping, just where the armhole of the Horberg let the point through. First blood.


[User] Please transcribe this text

[Assistant] It's looking ugly again now though, said Edmund. Peter's not using his shield properly. He must be hurt in the left arm.


[User] Please transcribe this text

[Assistant] It was only too true. Everyone could see that Peter's shield hung limp. The shouting of the Telmarines redoubled.


[User] Please transcribe this text

[Assistant] You've seen more battles than I, said Caspian. Is there any chance now? Precious little, said Edmund. I suppose he might just do it. With luck.


[User] Please transcribe this text

[Assistant] Oh, why did we let it happen at all? said Caspian.


[User] Please transcribe this text

[Assistant] Suddenly all the shouting on both sides died down.


[User] Please transcribe this text

[Assistant] Edmund was puzzled for a moment, then he said, Oh I see, they both agreed to arrest. Come on doctor, you and I may be able to do something for the High King.


[User] Please transcribe this text

[Assistant] They ran down to the lists and Peter came outside the ropes to meet them, his face red and sweaty, his chest heaving.


[User] Please transcribe this text

[Assistant] It's not exactly a wound, Peter said. I got the full weight of his shoulder on my shield, like a load of bricks, and the rim of the shield drove into my wrist. I don't think it's broken, but it might be a sprain. If you could tie it up very tight, I think I could manage. While they were doing this, Edmund asked anxiously, What do you think of him, Peter? Tough, said Peter, very tough.


[User] Please transcribe this text

[Assistant] To everyone at home. Ed, if he gets me. Here he comes into the list again. So long, old chap. Goodbye, doctor. And I say, Ed.


[User] Please transcribe this text

[Assistant] He walked back with the doctor to his own lines with a sick feeling in his stomach. But the new bout went well.


[User] Please transcribe this text

[Assistant] Peter now seemed to be able to make some use of his shield, and he certainly made good use of his feet.


[User] Please transcribe this text

[Assistant] He was almost playing TIG with mirrors now, keeping out of range, shifting his ground, making the enemy work.


[User] Please transcribe this text

[Assistant] Coward, booed the Telmarines. Why don't you stand up to him? Don't you like it, eh? Thought you'd come to fight, not dance. Yeaah. Oh, I do hope you won't listen to them, said Caspian.


[User] Please transcribe this text

[Assistant] But indeed there was no need to egg the usurper on. He was on top of Peter already. Edmund bit his lips till the blood came as the sword flashed down on Peter. It looked as if it would slash off his head.


[User] Please transcribe this text

[Assistant] I couldn't see what happened, said the doctor. How did he do it?


[User] Please transcribe this text

[Assistant] They were certainly at it hammer and tongs now, such a flurry of blows that it seemed impossible for either not to be killed. As the excitement grew, the shouting almost died away. The spectators were holding their breath.


[User] Please transcribe this text

[Assistant] It was most horrible and most magnificent.


[User] Please transcribe this text

[Assistant] Great shout arose from the old Narnians. Miraz was down, not struck by Peter, but face downwards having tripped on a tussock.


[User] Please transcribe this text

[Assistant] Peter stepped back, waiting for him to rise. Oh bother, bother, bother, said Edmund to himself. Need he be as gentlemanly as all that? I suppose he must. Comes of being a knight and a high king. I suppose it is what Aslan would like. But that brute will be up again in a minute and then... But that brute never rose.


[User] Please transcribe this text

[Assistant] The lords Glozell and Sopespion had their own plans ready. As soon as they saw their king down, they leapt into the lists, crying, Treachery! Treachery! The Narnian traitor has stabbed him in the back while he lay helpless! To arms! To arms! Telmar! Peter hardly understood what was happening.


[User] Please transcribe this text

[Assistant] If all three had set upon him at once, he would never have spoken again, but Glozell stopped to stab his own king dead where he lay.


[User] Please transcribe this text

[Assistant] That's for your insult this morning, he whispered as the blade went home.


[User] Please transcribe this text

[Assistant] Peter swung to face Sopespion, slashed his legs from under him, and with the back cut of the same stroke, walloped off his head.


[User] Please transcribe this text

[Assistant] Edmund was now at his side crying, Narnia, Narnia, the lion!


[User] Please transcribe this text

[Assistant] The whole Telmarine army was rushing towards them, but now the giant was stamping forward, stooping low and swinging his club.


[User] Please transcribe this text

[Assistant] The centaurs charged. Twang, twang behind and hiss, hiss overhead came the archery of dwarfs. Trumpkin was fighting at his left. Full battle was joined.


[User] Please transcribe this text

[Assistant] Come back, Reapy Cheap, you little arse! shouted Peter. You'll only be killed! This is no place for mice!


[User] Please transcribe this text

[Assistant] But the ridiculous little creatures were dancing in and out among the feet of both armies, jabbing with their swords. Many a Telmarine warrior that day felt his foot suddenly pierced as if by a dozen skewers, hopped on one leg cursing the pain, and fell as often as not. If he fell, the mice finished him off. If he did not, someone else did.


[User] Please transcribe this text

[Assistant] But almost before the old Narnians were really warmed to their work, they found the enemy giving way. Tough looking warriors turned white, gazed in terror not at the old Narnians, but on something behind them, and then flung down their weapons shrieking,


[User] Please transcribe this text

[Assistant] But soon neither their cries nor the sound of weapons could be heard anymore.


[User] Please transcribe this text

[Assistant] For both were drowned in the ocean-like roar of the awakened trees as they plunged through the ranks of Peter's army and then on in pursuit of the Telmarines.


[User] Please transcribe this text

[Assistant] Have you ever stood at the edge of a great wood on a high ridge when a wild southwester broke over it in full fury on an autumn evening? Imagine that sound.


[User] Please transcribe this text

[Assistant] And then imagine that the wood, instead of being fixed to one place, was rushing at you, and was no longer trees, but huge people.


[User] Please transcribe this text

[Assistant] It still liked trees because their long arms waved like branches and their heads tossed and leaves fell round them in showers. It was like that for the Telmarines. It was a little alarming even for the Narnians.


[User] Please transcribe this text

[Assistant] In a few minutes all Miraz's followers were running down to the great river in the hope of crossing the bridge to the town of Baruna


[User] Please transcribe this text

[Assistant] Mmhm. Why, I hope you don't mind my bringing her. Oh, gosh.


[User] Please transcribe this text

[Assistant] Chapter 12 Sorcery and Sudden Vengeance


[User] Please transcribe this text

[Assistant] Meanwhile, Trumpkin and the two boys arrived at the dark little stone archway, which led into the inside of the mound.


[User] Please transcribe this text

[Assistant] And two sentinel badgers, the white patches on their cheeks were all Edmund could see of them, leapt up with bared teeth and asked in snarling voices, Who goes there?


[User] Please transcribe this text

[Assistant] Trumpkin, said the dwarf, bringing the High King of Narnia out of the far past. The badgers nosed at the boy's hands. At last, they said, at last. Give us a light, friends, said Trumpkin. The badgers found a torch just inside the arch, and Peter lit it and handed it to Trumpkin.


[User] Please transcribe this text

[Assistant] The DLF had better lead, he said. We don't know our way about this place. Trumpkin took the torch and went ahead into the dark tunnel.


[User] Please transcribe this text

[Assistant] It was a cold, black, musty place with an occasional bat fluttering in the torchlight and plenty of cobwebs.


[User] Please transcribe this text

[Assistant] The boys, who had been mostly in the open air since that morning at the railway station, felt as if they were going into a trap or a prison. I say, Peter, whispered Edmund, look at those carvings on the walls. Don't they look old? And yet we're older than that. When we were last here, they hadn't been made. Yes, said Peter, that makes one think.


[User] Please transcribe this text

[Assistant] The dwarf went on ahead, and then turned to the right, and then to the left, and then down some steps, and then to the left again. Then at last...


[User] Please transcribe this text

[Assistant] They saw a light ahead, light from under a door.


[User] Please transcribe this text

[Assistant] And now for the first time they heard voices for they had come to the door of the central chamber. The voices inside were angry ones.


[User] Please transcribe this text

[Assistant] Someone was talking so loudly that the approach of the boys and the dwarf had not been heard. Don't like the sound of that, whispered Trumpkin to Peter. Let's listen for a moment. All three stood perfectly still on the outside of the door.


[User] Please transcribe this text

[Assistant] You know well enough, said a voice. That's the king, whispered Trumpkin. Why the horn was not blown at sunrise this morning. Have you forgotten that mirrors fell upon us almost before Trumpkin had gone and we were fighting for our lives for the space of three hours and more? I blew it when first I had a breathing space.


[User] Please transcribe this text

[Assistant] I'm not likely to forget it, came the angry voice, when my dwarfs bore the brunt of the attack and one in five of them fell. That sneak a break, whispered Trumpkin. For shame dwarf, came a thick voice. Truffle hunters, said Trumpkin.


[User] Please transcribe this text

[Assistant] We all did as much as the dwarfs and none more than the king.


[User] Please transcribe this text

[Assistant] Tell that tale your own way for all I care, answered Nickerbrick. But whether it was that the horn was blown too late, or whether there was no magic in it, no help has come. You, you great clerk, you master magician, you know-all, are you still asking us to hang our hopes on Aslan and King Petyr and all the rest of it?


[User] Please transcribe this text

[Assistant] I must confess, I can't deny it, that I am deeply disappointed in the result of the operation, came the answer. That'll be Dr Cornelius, said Trumpkin.


[User] Please transcribe this text

[Assistant] The help will come, said Truffle Hunter. I stand by Aslan. Have patience, like us beasts. The help will come. It may be even now at the door.


[User] Please transcribe this text

[Assistant] Pah! Snarled Nickerbrick. You badgers have us wait till the sky falls and we can all catch larks. I tell you we can't wait. Food is running short. We lose more than we can afford at every encounter. Our followers are slipping away. And why? Asked Truffle Hunter. I'll tell you why.


[User] Please transcribe this text

[Assistant] Because it is noised among them that we have called on the kings of old and the kings of old have not answered.


[User] Please transcribe this text

[Assistant] If you must blow the horn, do not let the army know why you blow it or what you hope from it. But that same evening, everyone seemed to know.


[User] Please transcribe this text

[Assistant] You'd better have shoved your grey snout in a hornet's nest badger than suggest that I am the blab, said Nickerbrick. Take it back or...