Everything We Learned from Starting An AI CX Company - OttoQa

00:00Tom Laird here Today we're going to break down contact center technology on the podcast time and time This is I from a call center geek, a the weekly podcast with a, a focus on all things call center But he's our host. Welcome back everybody to another episode of advice from a call center geeky call center calling center podcast. We try to give you some action items to take back in your your context and improve the overall quality improve the H and experience hopefully can improve the customer experience as well. Oh, how's, how's everybody doing?. My name is I have been slammed with Exp. of Your Things and with Auto-QA really launching to take off with lots of companies. So, I get a lot of questions on Auto. I wanted to answer as many of them as I possibly can on my list here and talk about everything that we basically have learned in the last 13, 14,.15 of starting from scratch with an AI tool, doing it for our internal customers first and then, and then moving it to to a product that customers are actually paying for using. It's been it's been really awesome. So I'm live on LinkedIn live on TikTok. If you guys have any questions, please, please, please, ask away. I'm here to answer anything we'll do this as we normally do as a as a full AMA. But let's start with some things that I think I didn't think about. And I think some interesting things, starting So when ChatGPT for came out And that's who we kind of tested it and And that's who- we kind of tested it And a lot of people just said, well, all you guys are doing is building a GUI on top of a large language model.

02:52And I guess, yeah, that's kind of what we're doing, but there's so- So many things for what, we are using this for. I don't know if people disagree with this, I've not really talked to, nobody's really talked to me about what models they're using in this auto CX space. Everybody's hum and hush on that, a pretty open with it. Although it's cheaper, when 4.0 came out, it cut our costs in half, but our act also cut in half. Very quickly, the excitement of for turned into, man, this thing just, it's almost worse than four. So that's when we started to say, all right, well, what other models are out there, and we, we tested pretty much all of them, the ones, the only ones that we have found to be absolutely the best use case for us the best use case for us have been entropics products.

03:57Right.

[Gap 03:57–04:07: no reliable transcription]

04:07poorly on those 150 calls that we've evaluated, we just sent it out to, that does a really, that does a really good job of just kind of summarizing very basic stuff. We initially started with Sonnet.

[Gap 04:19–04:30: no reliable transcription]

04:30And it's not consistent. Is it really accurate? I mean, the answer is no No, we would see out of 10 questions that we asked There's something there. But when it came to just the consistency of scoring. That's where some really cool things we have. Seven kind of checks within our system prompt to make sure that we are being as accurate as possible with our answer. So I think you guys have heard me talk about this a little bit, but we reiterate every single question. So even if it costs us more in tokens, you know, what is the answer? is to be the most accurate and the most consistent auto C auto CX on the market. So the one thing that we found is that the

[Gap 05:41–06:01: no reliable transcription]

06:01they'll match, then boom, we give the answer yes or no or whatever we're scoring.

[Gap 06:05–06:31: no reliable transcription]

06:31with some different. I- I'm. I- I- I- I- I- I- I- I- I- I- I- I-, I- I- I- Being, being a contact center, a, a tool like this, like we know what we- we're not just kind of looking for specific keywords. the phrase that I which is intent and outcomes. We're, which is intent and outcomes. We're looking for, did the proper outcome take place. And that's where we can score this thing at. And, again, looking for specific keywords. and impact, and you can be expensive for them, it's not that bad. the a ton of overhead. We are doing this as a contact center would do it, not as a software company to do it, which I don't know if that is good or bad. We have found that obviously the system prompt is absolutely vital to tell the exact outputs that we want. We We have found that we can't trust any of the large early on that we that we we tried to work through the new model that just came out yesterday as I recorded some recording this on on the 21st it's been beaten in a T-40 for a really long time, but no one's using it in a special sense.

09:33That is our And we. storylines, but what we're finding is that it is better than Opus as well, and maybe I'll get into the calibration. How we calibrate these calls is a customer will send us ten, So weB will have the call and we'll have the maybe it And So we, we can then match the scoring culture of every organization. But what we did started yesterday is we'd have a row for the human, a row for, we said auto 3.0, meaning Opus 3.0, And then auto 3. Cloud 3 Op. We match the human a little bit more but 3.5 has been actually more correct. So, I'm finding more quote-un, unquote human error in the calls when we're using 3.5. So, we're moving to 3.5 it looks like we're gonna test this for another day or to, as kind of our core model. It's been easier to kind of tweak?

12:11And also. We'll say this. we just ran a have a very large customer that we ran the first

[Gap 12:18–12:27: no reliable transcription]

12:27accurate, right? [QA coverage remark omitted] So, really excited about, it's been a little bit of a difference too, that initial run is so much closer now, that we, we have less tweaking, which means less cost for us, because we're not running as many, as many kind of calibration runs of the AI. So I think that's kind of been really, really

[Gap 13:07–13:37: no reliable transcription]

13:37F4, it's not even close. Right. We, we, we can't, we almost can't even use for how poorly it has been able to ask questions of data. So I think that, you know, some of the getting, getting into some of the, the tricks and some I think that some of the getting the tricks and some of the things that What we have found on the operation has been something that we glared and spent a lot a lot of money on to get it to the point where we've been fully calibrated. I think it was really cool to learn some of the tricks like reiteration. Anthropic likes to here or prompt, I if we're, if we're having a a big big and to be good on both of the models is to use the forms, feed those forms, and let them see the form that a customer has at the beginning to start our initial build-out.

15:20The longest period of time that we have or the longest length is the actual initial build-out. A customer will send us their rubric, they'll send us all the questions. How do you score these questions? Here's some background on this. To build those out takes a little bit of time. We do have an AI tool that we're utilizing to speed that up, but that's something that we're looking to do to make this even quicker. We're currently about 10 days to two weeks. We've been longer for some customers, shorter for others on the build-out. We won't go to proof of concept until we're at least 96 percent calibrated. I think that there's a lot to be said for the tools that are coming out and for how smart the models are getting, to be able to start to take other pieces of data input that will help us kind of as we move forward.

16:11I'm very excited. The next big step, I think from the model standpoint that we are excited and we are preparing for is the one thing that we stink at is that we can't see the screen, right? We thought that 4.0 was gonna be the panacea that we are going to be able to take the screen recordings and QA what an agent is doing actually on the screen as well. And while we can get data from that, we haven't to get it to the point where it can QA it. I think that's coming very shortly. That was a little frustrating. That kind of was the, oh my god, Foros here. Oh my god, I can do this kind of video and this imaging. This is unbelievable. But it just, the processing power, I think that it needed, it just never could get to that point where we could put it into a, to use it in production.

17:03So we're still testing that here and there, but it's been a little bit of a struggle there. So we're still sticking with audio. So voice, email, chat, SMS, helped us tickets. Those are all the things that we're really good at QA-ing. I think the other thing is customers been extremely helpful with how we should roadmap this thing out. I'm glad that we went out early. Certain things that customers want, a lot of different changes to the dashboard that we will be able to see, starting even today with being able to show differentiation how much better or worse an agent is getting. I think those basic type of reporting is important. Auto fails were not easy. To do auto fails was something that we're now doing programmatically. So for a section of a QA form, we will say, we'll just click it into this is an auto fail section. So if any of them and we'll say, we'll click the auto fail and we'll say yes. So if any of those questions are answered yes, then we will still score the call, but it'll be a zero for the overall call, although you'll be able to still see the sections.

18:17So we've been able to tweak some things there. rated calls we're getting much better better at. I feel very comfortable now, now going out with customers to say, hey, I don't want a yes or no, but I want to scale up on one through 10. We have one customer that is scoring this zero, one, two, or three, right? and they get points, you know, based on those different levels. And I think we're very close and very accurate with those guys. So I think that customer is pretty happy with where things are going, because that's a a little bit more subjective, the, the difference between a two and a a three, like, they did this but they did a little better as a three compared to a yes or no, they either did it or they didn't, there's a big difference in kind of how the aI need to think that, that through, it's a little bit more subjective, but, there's some prompting techniques that, that we've been able to utilize for that as well, so, yeah, it's been a, it's been a lot of a fun, we're getting better, and better, and better, and faster, and faster, I'd love to build this, for you guys again, remember no contracts, right now there's still the setup, I don't,t know how much longer we're going to be able to do that for, but right now there's still no setup.

19:27Total usage model, no seat licenses or any of that. It doesn't cost you anything to run this from us building your format to a full proof is pennies, so I think that you can utilize your teams in a lot of different ways that are more productive as well you know, hopefully we can really do some cool, cool cost stamp, you know, we can work and the cost of this compared to you know, you can utilize your teams in a lot of different ways that are more productive as well. So thank you guys- again, check us out autoc-QA.com, love that to talk with you all so on on our, our auto QA platform as well.