Verint Best Practices – 
Calibration

Transcript

Matthew: Hello and welcome everyone, my name is Matthew Storm. I’m with Group Elite, I’m joined by my colleagues Andrew and Eric today, and we have a really great session for you today. Calibration is one of those things that that I love talking about because I’ve like many of you I’ve sat in rooms and argued with coaches and evaluators and supervisors about all kinds of things, and it's one of those topics that everyone likes to talk about. And we're going to talk about how to do that uniquely in your Verint system today. So to just get us started. First of all, just as a quick introduction. Group Elite is a Verint strategic partner and we do installations and implementations and upgrades and consulting engagements all around the Verint portfolio. We've been around for almost 10 years, we're having our anniversary next year. We've done hundreds of integrations and migrations from all kinds of different vendors and platforms, we have a wonderful NPS score. We have a lot of happy clients and I see a few of them on the call today welcome and thank you for being a customer. We also do a variety of services from again new and upgrade but also very KPI focused, maybe around average speed of answer or NPS or compliance or sales, and those are usually very custom engagements that we do in consulting services. So that's my little pitch about Group Elite let's just jp right into calibration. What I’m going to do is Andrew I’m going to stop sharing and turn this over to you. So that you can share your screen and we can just jump straight into the topic at hand,and welcome. So Andrew tell us a little bit about yourself and let's get started.

Andrew: Absolutely, Hi I’m Andrew Tolbert, I’m looking at the attendee list and I know a good number of you. You guys have sat in training sessions with me and probably know my experience a little bit. I got to sit through a lot of calibration sessions just like Matt was telling us about, where you're in those rooms and you have those good fun debates with folks over who got the right answer and who got the wrong answer right. So let's let's dive in and let's talk about the calibration process inside of Verint. Let's talk about a little bit of of of, Sort of what I do what I like to what I like to implement for folks, and we'll start you know kind of at the beginning and just just work our way through you know, and as you can see on my screen right now, you've probably recognized this form designer depending on you know where where you're at you know how many calibrations you've you've sat through yourselves. You're you're probably very familiar with the evaluation form at the very least and and you know what hey this looks a lot like that, if you want to look at my screen here for a second, you can actually see I’ve I’ve named my form assessment right, and the form type down here is assessment. One thing I want to point out inside the the Verint platform is that an assessment should be synonymous with a calibration. Right those two things should walk hand in hand together, because inside the Verint platform we have an assessment form that we like to create to do our calibrations with, and the reason for that is simply because assessments are not tied to an agent's score directly. So we're not we're not assessing agents we're not calibrating agents right we're calibrating contacts and we're calibrating phone calls. So that's one big big change there from what your normal evaluation form would look like is that that assessment blank in there. Now as far as as far as creating an assessment versus an a calibration or a an evaluation rather, it's real simple right. All you have to do is create your evaluation form or maybe you already have your evaluation form created today, you can come over here and click the save as button and just change that form type over to an assessment and give it a give it a good name right. You want to make sure you name it. Something to do with assessment. So you don't accidentally do a lot of evaluation forms on your on your calibration form right. So you put the word calibration or assessment in there to delineate that, those facts in there. Also one thing I would like to point out on our assessment form here we got a lot of questions on this for we'll be focusing on these top few other lists we won't we won't have to go through an entire calibration process with you today, but we'll be focusing on these top few here specifically the opening and greeting I’ll look at these these questions, as in going through the list here looking at that that question probably looks like a lot of your evaluation forms look. You know the the question type yes and no questions, you know question text uses appropriate greeting as you're probably aware this is just built off of a template that already exists inside of Varint today. These are these are probably questions similar to what you're already currently grading maybe the scoring's a little bit different but we'll talk about that in a bit. So you notice this is a mandatory question and I have some instructional text. Instructional text is very important and keeping people calibrated not only on your evaluation forms, but your calibration forms right. All of your calibration forms or assessments that you're going to be using you want to put some instructional text in there right. We all have that that calibration bible that that word docent or pdf file that has how to answer every question, what to do in this scenario, what to do in that scenario and sometimes when we're doing those calibrations it's difficult to reference back and forth. So we know exactly how to answer that. So you can go ahead and cut and paste right out of that word docent and put it down here in your instructional text. So that you know how to answer every question, it'll help keep your folks calibrated. In fact I like to live a little messy inside my assessment forms, I probably put a lot of information down here just to kind of help out with the process . So that we can kind of go over. Some of those weird one-off scenarios right, since the agents aren't tied to this they're not going to have to see this and ask you questions like what why would why would this be in here you know you don't have to worry about that inside of an assessment since this is just your quality team or your calibration team it's going to be looking at this one. I'll flow through to my next one, my next question is a little bit more subjective it's not as easy as a yes or no. So my success for my question type is actually listed down here at the bottom, I’ve got my five criteria for success and as you can see this is a five rank list, and I have five criteria for success makes it real easy for calibration right. I can say you know somebody maybe may have spoken confidently and they were direct but they weren't really professional or easy to understand or inquisitive. So I might answer that one at a two instead of a four or. Something like that. So you know I kind of have my guidelines in here of you know oh they hit all five objects. So let's go ahead and give them an excellent score on that particular question. So it makes it easy for us to understand and follow through with and also you know we we we're all in calibration before we even get started with our calibration session. Again this is all before we actually get started. All right just a reminder as I’m switching screens here you know we got a  Q&A going on. So if you have any questions drop that in the chat and Matt’s gonna keep me honest, he's gonna keep his eyes on that one and interrupt me if we have if we have some questions come in there. So let's talk about the Verint screen right you guys are familiar with this one surely. So you've got this screen in front of you here you know we got to have a moderator for our calibration right, somebody that controls the conversation and keeps us on topic, as we know those sessions even remote can get out of hand. They got to go in and pick a call, and I’m going to select from my interactions and I’m just going to bring something up here to select. And this is this is the process that I start out with right. So I’ve got a a couple of calls listed here that I’ve already flagged but if this was a call that I listened to you know and I checked it out and I said hey this is this is going to be an interesting call to calibrate on, I’m going to come in here to my contact flags and I'll go ahead and mark this as a calibration call. And what this does is it's actually going to transfer it over to a folder and make sure that you know we're all in sync right where we can we can you know all find the same call, sure I could send out an email and I could say hey you know go to this contact id that happened on this date and and you know kind of put some extra steps in there but if I know that I have a calibration session every month the first day of the month I’m the moderator, I can come in here and check that that contact flag, and it's gonna go ahead and go straight over that folder, which those folders can be found over here underneath the the flags right. So I’m going to go over and look, I’ve got a couple of calls already in the folder for us to calibrate on. Sometimes we know this can take up to about 15 minutes to get in there. So if you if you were setting that up and gonna send out an email to let everybody know hey check your calibration folder we got this month's calls to calibrate on in there. Just make sure you give it a few minutes to get over here into this folder, you know in my in my laboratory environment sometimes it'll it'll work a lot faster than it does in in production takes up to about 15 minutes sometimes to get that in there. Alright I’m gonna go back in and we'll we'll go ahead and do our pre-work and get started on this on this lovely calibration of my friend Victor's calls hopefully . Some of you probably worked with him before recognize the name. Alright I’ve got my default form coming up, I I went ahead and prepped for this session and I knew that up in preferences I could set my default form to come to my marvel assessment, you know if if you're not not got your assessment form set up or your evaluation form set up you wouldn't want to find it in that drop down menu right there. But I’m going to go ahead and and go along and answer this, we're I’m not gonna make you listen to the call either we'll just go ahead and give him a a good grade, but if I did have questions on how to grade these subjective questions here, I would be able to find answers to him right here right. So I’ve got that that instructional text listed I can hit that drop down and we can see my friend criteria for success in here already already laid out for us. So I can go ahead and rank him in excellent because I know he hit all five of those topics, and then it's it's easy to just hit submit after we've listened to the call and finished up, and now once once we hit submit on that one it's going to go ahead and submit that and now you know if I’m going in and moderating the session I probably want some speaking points to talk about, because if you look at my form if I’ve got a 30 minute calibration session and a 15 minute call to listen to and I have 15 questions on my form that's going to be about a minute of conversation per question, it's going to get real tight real fast right. So we want to look at aome. Some docentation maybe you know some reporting on the back end and see where I need to calibrate, what are my evaluators saying you know, how how can I help these assessors out. So I will go in here and I’ve already pulled up a report for us, this particular report is the calibration report by question and on this calibration report by question I’m looking for the current months calibrations on that specific form, I’ve got five folks that have all calibrated on this one call from Victor in here. Obviously it doesn't have my results in it because I pulled the report beforehand but we'll go through and look and see where our five calibrators all landed on this one. So we have the used appropriate greeting question, I probably am not gonna mention much about that one in my calibration session we were all in alignment all five of us said yes on that, we were good to go. If you notice over here we do have our standard deviation along with our average score of what the what the question was with. The average score it is the score per question right. So you do need to know what those points are just to be mindful of you know how many points each question's worth and again we have our standard deviation if you're into the the mathematics there for the the standard deviation and really you know that's how you drive your calibration sessions. 

Matthew: Actually Andrew I just looked up standard deviation on that super computer that I carry around in my pocket, and it says that it's a measure of how dispersed the data is in relation to the mean. So if it's low that means that the data is clustered around the mean and high means that the data is more spread out. So I’m guessing in this case you would want that number to be low you would want the numbers to be really close together if they were all plotted as opposed to all spread out. So if it looks like obtains and verifies customer information is a little higher, he means a little you know a little more spread out a little bit more, you'd probably see a lot more if you had you know a lot of people doing the assessment, and that the standard deviation don't let the the calculus term there scare you. I think it's just a you know are are we pretty close are we not pretty close what is it. So but anyway. 

Andrew: Absolutely you know me personally I’m more of a more of a look at the question and see where everybody lies than looking up calculus terminology on my on my phone, but but yeah and you're right on that one one one thing we have to look at you know like you said this obtains ver and verifies customer information this is a pretty high point question and because that one person answered it no we've got a pretty high standard deviation as compared to this question that's you know in one point increments. It's a little bit lower than than the than the other standard deviation even though we're we're a little a little further off right on on the calibration. So that's that's a good callout, Matt good call out.

Matthew: I see here I think Karina put it best she she just posted in the comments, low is better it means that the team is evaluating in the same way rather than each person doing their own thing.

Andrew: Absolutely.

Matthew: Thank you Karina, I appreciate that. By the way if you have other questions about some of this process or other calculus related questions that you know probably will give to Eric we'll fire fire away ask away in in the chat or the  Q&A. So back back to you Andrew.

Andrew: Perfect perfect I’m glad we'll give those questions to Eric all right. So if we look at our our follows script for introduction, this is where one of one of these these questions we're probably gonna get a little heated right in the in those those rooms that we were talking about earlier Matt. So we've got everybody's a little bit all over the place out of our five evaluators, we've got a couple of them that think excellent a couple that think good and one that thinks average. So I don't know if that average person's just a little meaner than our our standard and you know maybe our average score being at four right probably probably should land somewhere in the middle of that. But but we really don't know until we have that good hearty conversation to to know exactly because, see these folks that are that are answering excellent they may know something that the person that answered average doesn't know they they could be using a different rule book right remember we had those have those rule books that are sitting around, maybe they ignored the instructional text and was using the rule book from 1997 instead of the new updated 2021 version and so they were a little off on their they're answering the question. So you never know what you might know that someone else doesn't know or vice versa. So it's good to get everybody calibrated everybody on the same rule book and and grading these questions all at the same way. 

Matthew: So Andrew we had at the kind of the question that came in the same way, where can we go to see how each individual evaluator answered. So they can compare and discuss their answers. We also had the same question you know, can you see who answered it I mean how do we get to know who is the person there who did average and good.

Andrew: Absolutely absolutely. There's two options really. One thing that I like to do in the past was I would take the interaction, press the download button or the the print button right and actually all of us bring our own evaluation that we filled out to the to that conference room. But you know the recording is still there. So I’ve got this report where Danny Rand, he actually evaluated the scores and since he was all by himself we just selected him as the assessor, and we can get all of Danny Rand's answers. So we can actually pull these reports at the individual level, or again we can we can have those those come out. Now the awesome part about the reports you might think oh do I have to pull a report every single month for every single evaluator. Go ahead and automate that process and set that report to you know come to your email right before your calibration session. So you know you can you can go ahead and set up those reports and then every month they get emailed out to you, whenever you're about ready to have your your calibration session. So you can have the report of every assessor or you know again overall encompassing question by question for everyone. 

Matthew: So Andrew you mentioned the word every month, but that kind of leads to the next question is what's the best practice as to the frequency of calibration assessments. So do do you like sending them out every month or every week or a couple of quarter or what do you think.

Andrew:  I think I think for most people most people can fit every month into their into their lineup, right especially with this session where we're answering it you know in our own time and then we have that that hard due date in in the month where we're gonna go talk about our results right . So you know it's good to set that time aside every month and I think that's that's a good cadence. Now if you notice that you start to get boring right because everybody's calibrated you haven't changed anything about your form in a year you know maybe that then does back off to every quarter. I had at one point in time eight call centers and eight eight teams I was calibrating with every single month, it was it was great I had them scheduled on Tuesdays and Thursdays and I just ran through that cadence and hey we we we did it everybody knew when when to come and when to show up. So it was good stuff.

Matthew: Next question, is there a report that would show who hasn't done their assessments yet. So if I had you know in your particular case eight sites and everything say you know I need to go send an email or something to these 12 people that haven't done their assessment yet.

Andrew: Yeah absolutely, that that was one that I sent every day to myself the number of assessments done per assessor is the name of the report and it's under productivity reports inside your system, and again I sent it to myself every single day, and you know when I noticed we were having issues guess what I forwarded it out to people and you know. So there was that visibility there. So I didn't forward on a on a regular cadence just once we started getting closer and closer to the due dates, because you know the first day of the month of course everybody knows they haven't done an assessment right, because it's the first day of the month. So yeah that's the that's the name of the report, and again I just I just put the heat on closer to due date time right.

Matthew: Gotcha all right, well let's if you have some other questions feel free to put them over in the chat or the  Q&A. Just as a quick you know we do help organizations with with topics like calibration and like many of these engagements that we talk about here on the screen. I will say that if you have a request for a demo or you want to see some examples of how people have done calibration sessions or how we've helped clients do that we can help you with that. As always on these best practices sessions, we're also giving away some free consulting if you wil.l We will do a calibration review with you, maybe help you set up your first one or maybe the next one with the forms that you have today, or maybe help you look at maybe your first collective calibration session. So the first three people who email back to Eric after today's call, I just email and say yes I'd like you to know some help with that or I'd like to discuss instructional text or you know putting all that together, or or maybe you'd just like to see a demo of some additional things. Maybe you'd like to see a demo of how this works and then prepare yourself for AQM or you'd like to see the AQM demo perhaps you'd like to see some things about ad hoc reporting. Just reply to Eric or or book a time there and there's a link in the email that you'll get here in a few minutes that will help and and and get to that. So next session this is our last session for 2021, but our next session is going to be on automating reporting and so you you won't want to miss that one. We do have a little bit more time for some questions. So I’m going to just go back up here and I’m going to pull open that the chat session here and see if we have a couple other questions. Let's see you know a lot of people saying thank you, yes a copy of the presentation can be made available. If you missed it or you'd like to see a recorded session, yes that that one is is also available. Let me just check my  Q&A window and just make sure that I have everything answered. So is there a suggestive strategy on the call selection, for example to see specifically how everyone differs on a specific interaction with versus a range of interactions. Is that the flag that you mentioned earlier?

Andrew: Yeah. So I had two schools of thoughts of what I flagged right because it's a manual process to flag the call, you know normally there's a little bit of of looking and listening and one one of the things I I selected was you know calls that would be tough to to answer you know. Some things that I knew would get subjective and a new conversation would happen and then other times I selected calls that were very straightforward and basic you know, I tried to alternate month to month to kind of you know keep everybody expecting something a little bit different. So yeah that's that's sort of what I did. But as far as they knew, they were just it was an ordinary call to them.

Matthew: Okay, very good very good. We had a couple other questions come in about seeing some reporting even some ad hoc reporting for WFM or other products within Verint. Absolutely just reply to Eric and and we could set up some time and discuss some of that with you. And well let's just we're about four minutes over. So it's perfect and I I just want to wish everyone a happy holidays.I hope you have a wonderful time with you and yours and and I hope that you you know no more webinars for the next two weeks how about that right. And thank you Andrew for your time today, thank you for for joining and and spending some time with us today we'll be doing these 20 minute sessions again starting again in january. So with that I'll say goodbye to everyone, thank you again Andrew.

Andrew: Hey no problem I appreciate it.

Matthew: Very good and have a good holiday Eric. We'll be looking for your email here in a few minutes.

Eric: Thank you always a pleasure.

Matthew: Alright well goodbye everyone, and we'll see you next time.