﻿Leslie: So, welcome to our webinar sponsored by the National Institute of Corrections. I am Leslie LeMaster, with NIC, your host and one of the facilitators for today's webinar. You will notice as you join today, that you are muted and the voices you're going to be hearing are mine, as your host, those of your facilitators, and your producer. Your way of communicating with us is via the chat function, which is located in the lower right hand corner of the Webex webinar screen. To ensure that your chat function is turned on because we are going to be using it a lot during the webinar to engage you in the content and activities, I want you to make sure it is turned on. So, hover your your mouse over the bottom center of your screen and you'll see some tools pop up. I want you to click on a tool that looks like a person talking. That will turn on the participant panel for you in the upper right hand corner of your screen. And then turn on a tool by clicking it that looks like a chat bubble and that will turn on the chat function for you. And then to make sure it's working, and a whole bunch of people have done this, but some probably have not yet who are just joining us and continue to join us, if you would chat to us where you are joining us from today or what the weather is like and why you are interested in the topics of this webinar, which is "Training from Analysis (A) To evaluation (E): Are you applying the Science of Learning and Performance?" And if you would send that to all of participants in the program, and you do that by choosing that on the drop-down menu. Oh wow! It's snowing in South Dakota, and then just click enter and your chat will go out to all. So yes, oh boy. We've got all kinds of things happening here and, well, definitely it's winter in the country. It was snowing here, also, too. So, training commanders we've got lots of people from training. Yes, oh boy! Lots of things going on here. So, it looks like the chat function is working. In the first couple of times we use it, we're gonna ensure that everybody has got that working for us because we are going to be using it a lot. So, we're just all happy that weather is rain--raining or snowing or sun shining or sleeting or whatever it could be doing that you are all joining us here today. That's the power of virtual training, right? We can do it no matter what the weather conditions might be. So, we wanted to let you know that the webinar is being recorded and you will have access to that recording at a later date. We will be posting it on the NIC website when it is available. We'll also have the opportunity for questions and answers so please do chat your questions and comments all along the way using the chat function as the webinar progresses. So, without further ado, we do want to get started, and I am going to start that out by, actually, having my colleagues introduce themselves. I'll turn it over to my chief, first, for him to introduce himself to all of you. Jeff: Well, good morning everyone my name is Jeff Hadnot on the chief of NIC's academy division. I'm happy you guys were able to join us this morning, especially on the short notice, for the rescheduled event. We're excited to bring this webinar to you today on training. A personal connection I kind of have with this is being a former training director, I can share some of the frustrations that that we all have in the training field when it comes to training and the role of training and organization. So, we're excited to bring some ideas to you today that you might be able to implement into your bag of tricks and we're going to get started. So, Bernie, I'll turn it over to you. Bernie: Good morning, everyone. Am I coming through, Leslie? Leslie: Absolutely loud and clear, Bernie. Bernie: Okay, from cold, but really beautiful Colorado, I've been with the academy division now for ten years. I can't believe it's been ten years, it's gone by so quickly. My particular interest is in learning and performance. And, the ADDIE model has been around for a while and, of course, in the learning performance field, things change, there's new technologies, there's all kinds of new things going on, but the ADDIE model has still stood up as a foundational piece in designing and delivering curriculum and can be effective in the pieces we're talking about today and connecting what we do in the training to what the agency needs. So, good morning, everyone, especially good morning to North Dakota because that's where I'm from. And, yeah, I can--I don't even have to guess that it's cold there, but glad you're all here from all parts of the country. Leslie. Leslie: All right. Thanks, Bern. Thanks, Jeff. Again, I'm Lesley Lemaster and that's me. I have been with the academy division of NIC for 20 years and I feel like I just got off the plane yesterday. I can't believe it's been 20 years. And this is one of my passion areas because I believe that excellent analysis, development, design, delivery, and evaluation is really the basis and foundation for everything we do in corrections. That leads to the great delivery of services. Because if we don't train people well and then evaluate the training that we do to see if it really does what we needed to do then kind of that loss. So, we're really happy that we're able to bring you this webinar, and that all of our technology is working fabulously today. We're hoping it's working fabulously for you on your end, as well. So, there's someone else who is joining us, also today, who's gonna bet going to be behind the scenes, but we would love for her to introduce herself to you, as well. Linda. Linda: Thank you, Leslie. Hello, everyone and good morning. Happy to have you here. I'm Linda, as Leslie said. I've been a management assistant for 10 years with NIC. And I work with Jeff, Leslie, and Bernie in various learning and performance projects, but today I will be behind the scenes as your webinar producer. And now I will hand it back to Leslie. Leslie: All right. Thanks, Linda. So, Linda's going to be responding to you. If you happen to have any technical difficulties, but she, also, will be keeping track of any questions that you happen to have along the way or any intriguing and interesting comments. So, that when we get to the question and answer period, we're all ready to go. So, very important role here in the webinar. So, to get us started, Bernie, I think you've got an engagement activity to get us going and up and running today. Bernie: Okay, so technology-wise we've got 80 people online. Hopefully, you all have audio and you got your computer working and we can all hear each other or you can hear us and see what's going on. That's part of the engagement for distance learning. The other part of the engagement is how engaged will you be in this webinar. So, if you can rate yourself on a scale of one to seven, seven being incredibly engaged and one being not engaged at all, and kind of rate where you are in that. And and just put that into the chat. Jeff: All right, looks like we've got a lot of fives all the way up to seven. I saw some seven pluses in there. We got some motivated individuals today for you. Bernie: It looks like that. There's all kinds of different ways that we get disengaged with technology and we're in the classroom. So, just getting ourselves oriented. So, thank you for putting it in there. We're got five, sixes, lots of five and six, and fours. So, considering where you are, if you wanted to raise your number, I want you to put into chat what would you do to increase your number? What would you do to increase your number to be more engaged? And just put that into the chat. Just put your ideas into chat. In- person training. Sometimes, in-person can be more engaging, but we know even with classroom training people check out on us. "Lunch and a nap" would help. "Starbucks" Okay, "Stop eating lunch." "More coffee." "Stay focused." "Clear my desk." "Away from the work desk". Cool. Jeff... what else have we got? Jeff: Looks like I saw "Shut down the email." I think that there's some value to be in on that. Maybe an out-of-office on the phone. It looks like we've got some information there... "Hanging a sign on your door" is also a good way to raise that number up. Looks like we got some pretty good answers here, Bernie. Bernie: Okay. Thanks, Jeff. So, we know whether it's a webinar, whether it's classroom, we struggle with the concentration. We struggle with being engaged because we have so many more things pulling us away from concentration. Why should we care about that? Well, we want you to get the most that you possibly can from this webinar, and so, being engaged and distancing ourselves from distractions can help. The whole idea that people can multitask is really a lie. The research tells us there is no such thing as multitasking, which you cannot do three things at once and give them all attention at the same time. So, we encourage you to be as engaged as you can, just because we want you to get the most out of this, but certainly when it comes to webinars, there's things you can do. I make sure one thing is, I do is make sure my cell phone is shut off and I can't see it, so, I'm not tempted to grab it when I'm in a webinar. And so, these things whether in the classroom or webinar, and in the classroom, what I see is people check out and you'll see them move, you'll see them look away. You just know, you have this sense when you work with classrooms after a while is "Okay... they've checked out right now." So, and we know--we know it's just a human thing to do and, so we encourage you to do everything you can to be engaged in this. So, we encourage you to be here now with us in this webinar in the present, fully engaged. And, so now, that you are here, now, with us fully engaged, what Leslie will be focusing on during today's session? Leslie: Well, Bernie. I'm so glad you asked that.Before I actually focus on that, I just wanted to point out one particular thing about this objective. You'll notice there's two words that are highlighted here in different colors, which is analysis in and evaluation. We just have one, you know, two focus areas, instead of the entire ADDIE framework, we're going to be working on applying the ADDIE principles of analysis and evaluation to the training in your organization. So, that's our real focus area. So, the first part the webinar is about an analysis and some activities around that, and then, the second part is evaluation. So, we hope will also be modeling for you how you can do very interactive training in a webinar format and then you can use these ideas in doing virtual training in your own agency. So, we're hope where we're gonna kind of an unwritten objective here for you, as well. So, if you're like me, sometimes you just have the best idea ever and "I just have to do something with it and I know just what I'll do. I'm going to design training." But, then I think to myself, "Is that actually going to meet a need?" So, what I'd like for you to do, is chat to me what do you do when you have the best idea ever. What do you do when you have the best idea ever? Ah! "Share it with peers to get feedback," says James. "Write it down," says Fanmaron says, "share it with somebody." "Tell somebody else." "Share the well." Ooh "Sticky note," "Write an outline." Haha. "Forget it," "Research it." "Write an outline." Ooh! All kinds of great ideas coming down here! Be sure and look at the chat. Oh I love it! "The iPhone memo pad," I love it! Don't you love that? "Journal it." "I brain storm it." I use a little... I like to mind map things. But, almost inevitably, sometimes I always seem to want to design training for some reason. That always be my first place I'd go, but I have to stop myself. So, what we're going to do now, is we're going to visit... we're going to kind of peek in on a conversation that's happening right now. Somewhere here in the United States between a CEO and a training director, and find out what's going on with them. Let's begin. [knock, knock, knock, knock, knock, knock] CEO: Bernie, if I see one more CO mess of a pat-down search, I'm going to blow a gasket. Training Director: What's the problem? CEO: We spend thousands of dollars on eight weeks training these new correctional officers and they can't even do a simple pass search correctly. I want you to double the hours in basic that we trained on pat searches and I want you to add four hours of it to annual refresher training. Training Director: That could take some analysis and design time. CEO: No, no, Bernie. We don't have any time for that. We need this done ASAP. Training Director: So, what is it that you want to accomplish with this training? CEO: I want the pat search done correctly. Leselie: Oh, boy. That's a really interesting kind of a thing going on. So what's going on in that conversation, you guys that we were just eavesdropping on? Chat, what is happening in that situation? And, have you ever experienced that? Haha, Yeah right, Spencer, "Blaming training." Leslie says, "Pretty typical." "Reactive approach." Yeah, Courtney, you got it. "Blaming training." "Policies are not clear." Oh. "Trying to solve a bigger problem and assuming it's training." Oh, yeah. Maria says, "All the time." "Something bad's happened. Fix it now." Ah! "They're not really getting to the bottom of what's happening." "Staff not being held accountable." Oh, "Not looking for a root cause," says Courtney. "A communication shortfall," says Linda. "Expectations are not clear" from Jeff. "A knee-jerk reaction" said Kathleen. "Without looking into it to see the real problem." I think everybody is spot on with that. So, Bernie, um how does this begin to play out in the workplace and and then, what do we need to do to find out what's really going on in this situation? Bernie: Well, we see that a lot of times what happened is this what actually, what everybody said in chat is that it's a knee-jerk reaction. We experience somewhere there's an issue and they were asked to design something. Usually, really quickly, to fix it in response to this on-the-job problem. And so, that happens often we hear that often. I remember it happening when I was in the field. And so,... so, let's begin to look and see how this plays out in the workplace. And we already saw how this plays out with CEOs. So, what can we do to...and yes, some of you already address this. What can we do to get to what's really going on here? Why? And why conduct a needs analysis before designing training? So, why conduct a needs analysis? In the chat, we already had some people tell us. already. You already picked up on that, but if you have a knee-jerk reaction, and it may not be... that may not be the issue. To see if it's actually a training issue...so, what other ideas... we used... "We trained using training as punishment." Yes, that happens, doesn't it? "To look at the full picture." Okay. "Why do it needs analysis? See... if it's a training issue or if the supervisor/leadership." What if it's... if it's a widespread issue or just just local?" Okay, so, anything else, Leslie? Leslie: Oh, yeah."Identifying the problem. If it's time, money and resources that are being spent in the wrong area." It's all kinds of clarifying. It seems to be a theme here of what's really going on versus the perspective that the CEO is bringing in that conversation. Bernie: Thanks, Leslie. So, we see... you understand that if we don't diagnose the problem correctly, how are we sure that our fix is going to fix it? How are we sure that what we do is going to fix it? I saw one of them said too, "What if this is a small issue in one place or is it widespread?" That could make a difference, as well. And, "Finding out how is it going to be addressed by training?" The other problem is that sometimes when we are...just do the knee jerk and click deliver something if we haven't addressed the real issue, then what I see is training turns around and gets blamed. And. you'll hear, "Well, you know. See that didn't make any difference anyway. So, training really doesn't matter." Well, it really doesn't matter when it isn't fixing the right problem. So a lot of great ideas here. That it really is understanding what's really going on is important. And so, doing a needs analysis, and when you look at that, does... Can we again and chat talk about this? Does your agency conduct a needs or gap analysis prior to designing a solution? In chat tell us, do you do that with your training? Do you do a needs analysis? "No." Got some yeses. "We've started to in some cases." "Our agency is moving toward calibration techniques." Oh, that's interesting! I want to know more about that. "Basically, train according to what is sent down to us." Yeah, from the top down. "Sometimes, yeah." So it appears... Somewhat spotty... ties back to that. So, some agencies do, some don't. Some do certain kinds of needs assessment. And others... and it says, "In some forms, but not necessarily, but, officialy your best practices." And "Some some of us don't know how to do a proper needs analysis." So, let's look at analysis a little bit and keep putting things in there. Really curious about that one, that I'd heard not of before. We need to find a lot more about that. So, let's look at needs analysis, and let's take a closer look at how we determine what is going on by answering two questions to close that gap. So, we're going to look at two questions to ask that we when we see in the graphic between performance and the solution. So the first question is the "why." Why don't people perform as desired? Why don't they do what we trained them to do? Put your answers in chat. Why...what are the things that you see, or you've experienced? Why don't they do what we trained them to do with training? "They did it in training." "Lazy sometimes." "It's not reinforced on the floor." "Folks want to take shortcuts." Oh. "The senior staff show them how it's really done." That's pretty standard when people get back to their workplace that they're told and certain times, you know this. I know what you learned at training, Forget all that. Here's how we do it here." "Complacency." Oh. "Not feeling the training policies are worthwhile." So they're really not connecting the reason why it needs to be done a certain way, or why their tasks need to be done a certain, way with why...why bother with it doing it that way." Okay, so... so again, why don't they do what... So, you see there's multiple reasons, right? There's multiple reasons why we have performance gaps, and it's different from person to person. It may be. But, as trainers, we want to look at that. The... And so, finding out, that's our first question, is why don't people perform as desired. What's... what's going on? And you've already given us a list of some of those things. What's going on that they aren't performing? So, finding out what that is can be really helpful. The second one is... The second step is... to begin to create a possible solution to those identified performance gaps. And so, what are some possible solutions to performance gaps? And, we... please put those in chat. What are some possible solutions that you've tried or experienced to performance gaps? And put those in... PIPs. "I don't know what pips are." Performance improvement plan. That's what... that is probably. "Evaluation." "Post-training." "Quality assurance practices. Leslie: All kinds of interesting... oh, "Focus performance meetings." "Regular proficiency checks. There's all kinds of things here, Bernie. Lots of great ideas. Don't you love it when we all learn from each other as we're here? Bernie: "Incentivize performerformance," okay. Leslie: Touchdown. Bernie: "Hold supervisors accountable." If you noticed, I've seen a lot of that in here. Supervisors, supervisors/leadership. Leslie: Yeah, so, who should be coaching back on the job after training is done to make sure that performance is truly transferred into the workplace? Who should be doing that everybody? Bernie: Oh, that's interesting. "Creating an environment of safety, so staff feel they could admit when they don't understand." Yes, we all want to be seen as competent. So, it can be difficult to admit that we don't know how to do something or can't do it right. Great. Lots of great ideas. "Get buy-in from the staff." Buying in is important, isn't it? Because you can teach them how to do it. But if they don't really believe it's important or valuable how do you get them to actually do it? Oh, good. Good. Bernie: So, we see that some of the solutions that you've tried or worked with have to do with supervisors... have to do with leadership. So, training isn't just about creating training, putting it out in the classroom, or however you want to deliver it. And, then they go and do it. This is... Everybody needs to be involved with this, especially supervisors. What do supervisors expect back on the job? They may not have the resources we've talked about that. Or authority needed to do the job, or sometimes a job training changes. Oftentimes, if I've looked back at any job description and, I have, it isn't very long until a year or two -- until my job shifts from that. So, oftentimes, that happens. "The job is changed." "They don't get feedback." "Feedback from supervisors or leadership." One way or the other as to whether they're specifically doing the job right or not. Or "They're punished for doing the right thing." They want it. They, oftentimes, will new pre-service training and all excited and ready to go do their job and it's done and then I get there and they're told, "Well, you know forget all that stuff you learned at training, here's how we really do it." And so, sometimes they can even be rewarded for not doing in the way they were trained. There's a lot of different reasons and a lot of different ways that we can come up with solutions for that. So, there's only one analysis that should lead to training. One answer to these analysis questions that should lead to training... ...and that is they don't know how to do it. Because you're right... all the things you've listed in the chat. All those things, a lot of them were not training. A lot of them were other things that around training, or have to do with training, but aren't specifically training. And that's all training can really do, is teach them knowledge skills and attitude. "KSAs" Some of these other things are not going to get fixed with training. We know that. So, the only answer that should lead to the creation of training is they don't know how to do it. And this could be so important. And again getting to some solution, because if you aren't focusing on the right thing, then, you're not going to get the outcomes that you want. And we know there's times where we're asked to create training, and it has more to do with legal issues and someone freaking out over something that happened. And quick we got to do something, so that we can show we did something! We know that we're called on to do things like that, and we know there's times when that's the main point of what's going on. As learning professionals in this field, and you've pointed it out, there's times where leadership needs to understand what training can and can't do and what can be expected of us and what kind of solutions we can offer them. So, identifying the solution for the problem is training. Let's say that your answers are, it does go to training. How do we figure out what should be in the training? How do we know what the content should be? Well, one of the ways, this is just an example there's other ways of doing this analysis, but this is one way and one part of a way to develop training. So, we're going to say, we have an issue a performance gap. Here's how we're going to address this and we're gonna address this in pre-service before people get the work. There's certain things they need to know. One of the things that I see often when I go in the field, is training and particularly, pre-service. Pre-service is based on policy procedure. Policy procedure. Policy procedure. And policy procedure can tell you the limitations of what you can and can't do, but it doesn't get to...what it often doesn't get to is what people actually do on the job. And so, one of the things we use that NIC is what's called a DACUM, which is really just a job task analysis. And it is just "what does a person do on the job." It's the process of identifying and documenting what is done on a specific job. What do they actually do on the job? And it's a process starting with the job itself. The title...and that's a position or the title, usually a sentence or two. And this is just a particular job, like I'm a correctional program specialist. That's my title or position. A corrections officer would be a job title. And so, that's a basic... that's where you start, is with that job title. And having it clearly... clearly defined as possible. Do that because that can change over time too. I'm sure there's people in positions where you may still be called a corrections officer, but you may do other things. So jobs, starting with the job title and then you work your way down through this process called a DACUM. Developing a curriculum. And as you work your way down it, within that job, what are the duties? And you know what duty is something that's a responsibility, but it has no discrete beginning or ending. It's just ongoing. So, part of corrections officer's duties would be security and safety of inmates in the facility. That would be a duty. That is ongoing. That never stops. That's always part of their job. Within a duty, there are tasks, and the tasks have a discrete beginning and ending and they're what you do to fulfill that. And one of those may be count. Count could be a task. Running count. That would be a task. So, that would be a task that you do to fulfill your duty, which is security and safety within the institution. And so, you get down to task and within a task there steps. There's a certain way to do a count. There's certain... there may be, there may be paperwork involved. There may be... But, there's certain steps to doing that. And when you do a DACUM, you get all the way to task. When it comes to steps you do it after that. The DACUM process for us at NIC, is...take several days, you get a group of high performers that enter in that position. You work your way through these steps to find out what they do on a regular basis. And then, once you get to tasks and you take your DACUM charts, you take your policies and procedures, you take all the other things you need to create training, and that's what you use to create your curriculum. Because, you want to get to steps. You want to get to behaviors. You want to get to the place where you know what you can teach them... what they actually do. And I've seen curriculum that doesn't get there at all. I mean, I've seen pre-service curriculum where it's the policy and procedure dumped on a powerpoint and read off to the class. Which is not going to be terribly helpful for that new corrections officer when they have to go in that first day and do what they do. So, that's a Dacum... that's an example of one way to do that. So, I don't know if anybody, if you put in a chat, whether you use a DACUM or not. You may do a different job task analysis. And so on. Somebody lost audio. Okay, so, that's one way to do this analysis. There are other ways. If there's other ways that you do this let us know. Then, we move to the middle part of ADDIE because we're focusing, again, on A and E. So, we're kind of jumping over the middle part. Not because it's unimportant. It is important. But it's usually the part that we focus on and we forget the A and the E. Or we can't do it. So we're just gonna work-- Again, hopping over the top of this because we're very familiar, most of us, with the middle part of this. So, the design part is actually creating this. You take everything you have around you, like I said, the DACUM if you have that, and so on, You create your outline. You come up with your performance objectives. And these are really important here, because you can't measure something that you don't have as a performance objective. What are you gonna measure? So, need to have those up front. And then, how are you going to deliver this? The development part is where you're actually starting to create the pieces of it that you use in your lesson plans, your facilitator guide, your participant guide, your slides, all your materials. And also, creating your evaluation tools. Because evaluation, again, needs to be all the way through here. And then implementing. This is where you're piloting. Piloting your training on getting some feedback, evaluating the process, and also evaluation phase. And see are they understanding, are they getting it? So, this is the middle part of ADDIE, and again we're kind of skipping over this quickly because it is the part we tend to focus on. And so, in talking about these parts, talking about the analysis, we're now going to move to evaluation. Jeff: Thanks, Bernie. As we begin to explore the last piece of the ADDIE process, "evaluation", I want you guys to think about how you guys are working on evaluation in your agency. How do you define training in your agency? In your current position? Thinking about, however, you guys do evaluation. How would you define training evaluation? You guys can go ahead and chat me some responses in there. Anything you guys are doing for evaluating training. Leslie: "Job performance," says Louie. "Connect to the mission and vision." "Does it do what we designed it to do and does it give the skills we wanted?" Ah! "A good video system." Jeff: Alright, so it looks like we've got a pretty varied way of doing some evaluations. Everything from a pre and post assessment, survey responses, all kinds of good... good answers there. Okay, before we move on too far past this, what I'd like to do is... let's talk about how OPM actually defines training evaluation and let's see how it either compares or contrasts with what we've already answered, here. So, the OPM says, "Training evaluation is a continual and systematic process of assessing the value or potential value of a training program, course, activity or event. So, yeah. I think it looks like a lot of the connections there... a lot of these surveys... A lot of these pre and post-tests, evaluations and things are going to kinda tie together with this. Now, that we have defined training evaluation and the many facets of it. Let's take a look at what we can do with some of the results of the training evaluation. So, I'd like you guys to take a look at this poll question. It answers A through H and you may have to scroll down. Leslie's going to open up poll on the side for you to answer on. And, read through this, and what do you think we can do with the results of a training evaluation? Leslie: And, you may have to open up the polling function on your screen. They'll be... a polling will pop up and hopefully, the poll has popped up for you in the right hand side of your screen. Just click on the little polling word and it should pop up for you. Let me open up the poll and it is open. Jeff: Thank you. Leslie: We do have people responding, Jeff. Jeff: All right. And if you're not seeing the poll for some reason, if you click on the little three dots at the bottom in your little tools, you'll see an option where you can turn the polling on. Just in case it's not showing up for you. Leave it open just a little bit longer, Leslie and see if we get some more responses in. Leslie: Now, 32 out of 84 respond. Jeff: Ok. Leslie: Numbers are going up. In about another 20 seconds. Leslie: Got about half of people respond. About another 10 seconds. Okay, I'm closing the poll. Calculating. Jeff: And while we are waiting for this to calculate, I'd like you guys to think about what else... are there any other things that we can do with the results of a training evaluation? So if we're running training evaluations, what else can we do? We see some of the answers here, but are there other answers that we're thinking about. Think about some of those. Leslie: Poll results should be up. Bernie: Leslie, I see we have a few people... ...about "determine the effectiveness." A majority of our people are "all of the above." They identified "all of the above." It is overwhelming majority Leslie. And Jeff. Thank you, Bernie. Alright. So, we can go ahead and close out our poll. If you guys can minimize your polling on the left or on the right hand side there, you can either do it by that down arrow next to polling, or you can click on the X next to the word "Polling" and that will get rid of that. You can open your chat window back up. If you guys have any answers on what else you can do. Is there something that you're doing with training evaluations that we didn't already talk about, we'd be curious to share those with the rest of the group. "Identifying training gaps and needs." Good. Yeah, there's a lot of different things we can do. We can use them to guide decision-making around various components of training, such as instructional design, delivery results, and its overall continuation, modification, or even elimination. Think about that. We can determine if the program's objectives were met. Assess whether or not training was properly implemented. So, that's something else to think about. Its effectiveness and ability. Good. All right. Now, that we've discussed what we can do with the results of training evaluation, why then is evaluation such a critical step in the ADDIE process? So, why do you think it's so critical that we that we don't overlook this phase? And a lot of times we do that, right? So, what do you think? What? Why is it so critical that we don't overlook the evaluation piece of it? Okay. "Identifying material to see if meets the current issues." Excellent response. "Efficient use of resources." Great, Maron. Appreciate that response. "Program effectiveness." Good. Definitely, Ah! "Many times, what's on paper doesn't work in practice." We've seen that,right? "Policy comes down from people sitting in headquarters and it doesn't work behind the wire." So, there's a lot of times that that happens. Some good responses coming in here. "Training's expensive. You have to get it right." That's exactly a good point. If we won't evaluate the results, we won't know if it works." We see that all the time, right? Unfortunately, many organizations do misunderstand the evaluation stage. Sometimes we believe that if you evaluate learning by collecting the feedback forms, then the process is done. But we often know that that's not enough. One-and-done evaluations are not the only part of the evaluation process, as it doesn't consider the long-term impact of the training. So, as someone said, the training is expensive. We want to make sure that we get this done right and what's the long-term impact going to look like? Organizations that do the evaluation process will continually evaluate the impact the training is having on employees, the department, and the entire organization. They do this to ensure that the training is achieving the desired results. If the training is not doing this, then it is changed or replaced with something more appropriate. Now, sometimes that means starting over with the ADDIE process again because a different direction is needed. Leslie: Alright. Thanks Jeff. So, now, that we've defined evaluation, and we've explored the criticality of it in its uses, now we need to move on to how to do it effectively. This is where the rubber meets the road. As we know, we just cannot stop at a participant feedback form, if we truly want to measure the impact and effectiveness of our training. So how do we do it in the deliberate and structured way? And what types of tools and evaluation methods should we really consider using? Well, we here, at NIC, use Kirkpatrick's levels of evaluation. And what we're going to do now, is take a look at each one of those four levels. And some examples of how you can measure each one of the four levels. So, the first level, level one, is reaction. And this is where you're measuring the degree to which participants find the training favorable, engaging, and relevant to their jobs. So, when you look at that first little picture there on the left, that's a person. That's supposed to be a participant with that wonderful ideal light bulb. And that's an example of a trainer or a curriculum designer, who is designed in pulse checks as an evaluation strategy. And pulse checks are a facilitator determining the level of engagement or knowledge of participants during the training by observing, questioning, or conducting any number of large or small group activities. And so, this is actually something any of these different evaluation strategies that i'm going to be actually sharing with you at each one of these four levels, are things that would actually be designed as the training is the bus designed all along the way. So, the first one there is a pulse check. And by the way, the definitions of all of these are included in the white paper, in which, this webinar is based on, as well as, the OPM document that is in the resources of the white paper. That second little picture there is supposed to be kind of getting us to think about using surveys and questionnaires as a way to measure level one. And that's kind of a time-honored way that we are used to in a traditional way to measure level one. But, there's all kinds of creative ways to do level one surveys and questionnaires. Some people like to do end-of-day reactions. I know I'm a fan of doing those. A lot of times, I like to get people to self evaluate where they are in terms of their own learning and what they need to do to get the most out of the training, in addition to, the typical closed-ended, open-ended, end of course surveys and questionnaires. In general, surveys are more generally closed-ended, and questionnaires have a tendency to be more-- be more open-ended. So, there's another way you can measure level one. And then, that third one, those people, they're talking together in a group. That's an example of using individual or group interviews. So, you can open two-way communication channels about the training, the content, the delivery strategies. So, you get lots of really valuable information as training designers and facilitators about what people are learning. What's really working for them. Because, sometimes if you just rely on a survey or questionnaire, you're only getting one-way communication and it's kind of open to interpretation about exactly what some of the information means. So, chat to us, how do you measure level one in your agency? What are you doing to measure reaction, the degree to which participants find the training favorable, engaging, and relevant to their jobs? Bernie: "Survey monkey software." Oh, survey monkey... Survey monkey is really easy isn't it? It's easy for the person putting it out and really easy to do the analysis. Leslie: Ah! "Open forum with new hires." Alright, Sarah. Bernie: Just so you know, there was a discussion about level one and the value of it or not. And at the end of this, when we have a few minutes for questions, we can address that. So, hang on to that idea. And "Verbal feedback." Absolutely. "Dialogue" absolutely. We would encourage you to do just more than surveys or questionnaires to get more substantive feedback and measurement for level one. We would totally encourage you to do more. "Really find out things." There you go. Alright, level two, which is measuring learning. The increase of intended knowledge, skills, and attitudes, as a result of participating in the training. Because remember, that's what Bernie said, is the only reason why we would design training and use training as a solution is because we want to increase and change knowledge, skills, and attitudes or KSAs. So, at level two, that's what we're actually trying to measure. So, this first little person here, they are actually working on the job. So, this is an example of using skills and behavior observations. And, that is where an expert, peer, or supervisor observes and documents if an employee is performing the appropriate skills on the job completely and correctly. And, that would be using a checklist. So, that's a creative and different way of measuring learning as a result of the training that you're delivering. So again, that evaluation strategy would be designed into the training as you are designing it. On the middle one is an example of using teach backs as a way to evaluate the learning and behavior change as a result of the training. And teach backs are when a presentation is prepared and delivered to others to demonstrate the employees new knowledge, skills, and attitudes. This can be done during the training itself, such as, you know, any kind of new knowledge and skills. I always think of our trainer development, training that we do. But I can think this would work with any skills based training particularly. And it also could be done to peers back on the job while a... yes there's a learning theory. Absolutely, Stan. Also, with a peer supervisor, watching, using a checklist for evaluation. And then of course, time-honoured and traditional pre-test and post-test here. But there's lots of other ideas in the white paper, in the OPM document. For level three, again, this is where learning and behavior change kind of work together. And there's one here that is shared that some skills observations on the job. We have our little officer there at the control station. That's a really fantastic way to measure if behavior change, if the new knowledge skills and attitudes are actually being applied on the job. I, personally, can't think of a better way to find out if that is happening. The one in the middle is... it might be a new one. It might be new to you, and this is a work review. An expert, peer, or supervisor reviews the employee's work on the job and documents if the work is being performed correctly. And that would be, probably, according to policy, practice, and standards, generally, using a checklist and a template. The on-the-job observation, that first one to the left, that's a shorter term. The work review is of longer duration, looking at work, dockets, documents, work product over a longer period of time. The third one to the right, you notice the people working together in a group. A level one...three behavior change is actually action learning, so again, on-the-job behavior change, what new knowledge, skills, and attitudes are being applied back on the job as a result of the training. Participants create a plan that they commit to carry out when the training is over. Ideally, a supervisor would follow up on the plan and provide coaching and reinforcement. This is actually something that we do in our leadership programs. We have action learning projects, we work with the agencies that we're working with to help develop those. Those are real agency projects to impact something new that needs to happen in the agency, and they practice their new leadership skills on the job with their peers, with supervisors, with leaders in the agency. So, that's something to think about. So, how are you currently actually measuring leveltwo2 and level three in your agencies? Just chat those ideas to us. And we don't want to leave out level four. This is the one we always hear is the most difficult to do because level four gets to organizational results. It measures the impact of the training on the organization as a whole. And when we look at this, that one to the left. You design the training and try to tie in measures of key agency and human resource metrics. And, you look at how the results of the training tie in to begin meeting the mission, the vision, and the values of the agency and key hr metrics, like looking at retention rates, looking at recidivism rates for the agency, looking at other offender outcomes, and other agency outcomes as a result of your training. And then on the right, a request for validation. And when you look at a request for a validation, a request for validation might be a new one to you. And this is a letter sent to supervisors or managers asking for quantitative evidence that training is being applied on the job. In other words, it seeks evidence of results of the training being applied on the job and impacting the organization as a whole. And how do you measure level four for your agency? How do you measure level four results or impact of your training of the organization? How is it happening for you? Oh, we've got all kinds of interesting things popping up here. Oh, yeah. Very common is follow up separate surveys months after the training. "Statistical data and meeting required measurements." thanks, Denise. "Oral presentations," "Observation of staff," "Supervisor audits". Excellent. "After-action reviews," "Unannounced evaluations and audits." Oh, there we go Mike! You're talking about some agency-level outcome measures. "Number of diversions from court detentions." "Length of supervision." "Client feedback." Tracking stats from audits." Thanks, Tara. Sarah, excuse me. Stanley, thanks. "Feedback from supervisors and management." Great, that is fantastic. So, we do have... we're going to see as a result of this webinar, we're going to go back and peek in. We're going to check in on our supervisor and our CEO because they have been with us during this webinar. And see if their conversation has changed as a result. [knock, knock, knock, knock, knock, knock] CEO: Bernie, if I see one more CO mess up a pat-down search, I'm going to blow a gasket. Training Director: Oh, this is a serious issue for you. Obviously, correct pat searches are a security issue. CEO: Bernie, we spend thousands of dollars and eight weeks training these new corrections officers, and they can't even do a simple pat-down search correctly. Now, I'm not certain this is a training issue, and I don't want to overreact in this situation, but I... and I know that we can just throw more training at everyone, but sometimes that's not really the problem. I'd like you to take a look at this issue and see if you can figure out what's going on. So, if it is serious enough to invest some staff, time, and money, we need to get this right. CEO: Bernie, we have the responsibility to ensure pat-down searches are done correctly. As the subject matter expert, what do you recommend we start with, and how can I help support you? Training Director: Well, before we decide on a solution, I recommend some analysis to identify the root cause. We need to look for specific performance issues, Once we've identified the root cause, we can design an effective solution that we can evaluate. One way you can support this process at the very beginning is just give me some specifics on what you're seeing with pat searches. We need to start with sitting down and going through what you're actually seeing. CEO: I understand that Bernie, I'm seeing some specific issues as I walk around on third shift with the new officers and they're taking shortcuts and not performing these searches the way that we've instructed them to do that. But of course, we need this done ASAP. How long do you think this will take the training team to do this analysis and bring me some potential solutions? Training Director: Well, give the training team three weeks to analyze a problem and will recommend some solutions that will get to the outcomes that you're looking for. CEO: Bernie, as always I thank you for your leadership, your thoughtful approaches to training, and ensuring that we're meeting our mission. I look forward to working with you as we correct this deficiency. Thank you. Leslie: All right, everybody. What are you hearing in our roleplay revisited between the CEO and training director? Has this webinar had any impact? Chat us your ideas? What are you hearing? Ah, "Better communication," says Denise. "Sounds as if he's learned something," says James. "Management support!" with an exclamation point," from Sarah. "Buy-in," says Yolanda. Oh yeah. "Not one size fits all." "Systemic approach considered." Oh yeah, Bernie with her managing up aproach. Ah, he's listening to his...me!" "Problem-solving, more open input." Absolutely Kathleen. Oh an "understanding of training needs," right Marie. You got it. Absolutely. So a little bit of a different approach there, and you know, sometimes you just have to do some groundwork layin, in terms of that. So, we do have some time for Q & A. A couple of minutes. I did want to open the floor from that. Linda I'm not sure... the chat has been going through fast and furious, but now, is the time to chat any questions that you have for your facilitators Jeff, myself, and Bernie. So, what questions do you have? Linda, do you have a question? Bernie: Leslie, I did see somebody ask for an explanation on what's a teach back. Leslie: Okay, great. And just to let you know, any of those examples of evaluation methods that I mentioned in Kirkpatrick, are located on page 17 of the white paper, of which, this webinar is based upon. But basically, a teach back, or a presentation, or teach back is a presentation prepared by the participant that is delivered to others to demonstrate the employee's knowledge or skill. And this can be done during the training, itself, or to peers back on the job using some kind of checklist or evaluation form. And of course, you would like to give feedback from that. And so. Mike is asking more info and how other agencies use teach backs. I know that we use teach backs, not only in our learning and performance, which are our trainer development programs, but we also use them like in our leadership programs. We use the peer feedback delivery method as a way to actually deliver some of the content in our leadership programs. It's an instructional strategy that we have designed into our programs. That's how we at NIC use it. How about others right out there on the webinar? And Robin there was a link to the white paper in the advertisement for this webinar. Other questions? The powerpoints will be a part of the copy of this webinar. Bernie: And if I can address one, there was a discussion going on about level one, with the students... how they rate the class by how fun the class is. Just so you know, when we had our training administrators network meeting last year, this was a point of discussion when it came to evaluation. When we looked at... there were some that found it useful, some that didn't find it useful. Just so you know, that even Kirkpatrick agency is looking at the one, two, three, four, and have actually said that if they had to do it over again, they'd flip it on its head. That the numbering doesn't necessarily mean value in that order, and that it doesn't... and also that it's more based on chronological, like, you know, students reaction is right after the training, right? Where the other stuff is at a later date, and so they're looking at even changing that around and also trying to help people understand what is meant by levels rather than that it's what what we tend to think of. But, the discussions, just so you know, that is a discussion that goes on. And there is some of the stuff on the Kirkpatrick website that you may want to look at about that. Because there were some people who said, "Yes, it's not useful at all." And it depends on what you ask the class. It asks... it depends, and then, also what you're trying to get. What information are you trying to get? Leslie: I just posted the link to the white paper that this is based upon in chat. And I want to let you know as I mentioned earlier on in the webinar, all NIC webinars are recorded. As soon as we get this made 508 compliant, we will be actually posting this to the NIC website, and we will post to the website when it is available. The actual recording of the webinar will include the slides. So, that is all the time we have today. We do thank you for being here with us today and joining us in spite of the difficulties that we had back on November the 15th. We thank you all for joining us here today for this rescheduled webinar. These are selected resources that we highly recommend, but there's so much out there on training evaluation. You can look at not only our information center resources, but very specifically, the OPM training evaluation field guide is just a wonderful resource to have. It's available online and that is in the resources link in the white paper. Linda has posted for you our names and the email addresses of all of us who have been on here. And, if you would like to contact any of us, if you have very specific questions, I see there, Joshua that you have a very specific question. We're not going to be able to answer it here and right here during the webinar, but please do email us after it's over with or we'll try to grab that question and get back to you. On that, thank you all for being here with us today. We really do appreciate you being here, and you'll be getting an email from us from the webex system on how to retrieve your certificates when those are available from the NIC Learning Center. For those of you who completed the webinar today and we really appreciate it. Have a wonderful holiday season, and we hope to see you in another nNIC webinar in the future. Thank you, everybody.