Do You Need an Experimentation Therapist? with Chris Gibbins

No hacks invited to the party. No hacks invited to the show. We've got a good thing going. No hacks invited today.

[00:00:18] Sani: Welcome to No Hacks Show a weekly podcast where smart people talk to you about better online experiences. The episode you're about to listen was done in partnership with Experimentation Elite, the UK's premier CRO and experimentation event taking place on June 11th and 12th in Birmingham uk.

So you know what to do. Go to exploitation elite.com and get your ticket today. My guest is an experimentation elite legend and this year's event MC, Chris Gibbons, chief experience officer at Creative CX, a man with 20 years of experience in the fields of UX and experimentation. Chris, welcome to No Hacks Show.

[00:00:51] Chris: Hey, Sonny. It's fantastic to be here. Really excited. Um, yeah. Oh my God. I'm sounding a bit old, aren't I? Well,

[00:00:57] Sani: all are. We all are, but that's not a problem. Wiser is what I would say. Not, not older. So, uh, the app today we're talking, we're going to talk about experimentation lead a lot, of course. Uh, but let's talk about what the, the main topic of this episode is the concept of having an experimentation therapist, because as soon as you mentioned that in your email, I knew this is what I wanted to talk to you about.

So tell me about the origins of this nickname. You have experimentation therapist,

[00:01:24] Chris: it came, it came up completely spontaneously and it wasn't my idea. It came from a client we're working with and we're helping them to build a centre of excellence. We're helping them to move from being a traditional optimisation to a really kind of scaling experimentation around the whole retail business.

Because of the nature of all the many calls I've been having with her around, and all the challenges she's been having, and we've been discussing a lot of these challenges around how to get around them, how to deal with different people, she just spontaneously came up with saying, you're my experimentation therapist, she said.

Which made me laugh, and I'd never heard that before, but actually, It got me thinking it's quite fitting really for the type of work that we need to do in experimentation. It's, it's those people challenges. So

[00:02:13] Sani: right? And. Why do you think there's still that problem that you even need something like an experiment or anyone, not you need something like an experimentation therapist. What is the core issue there?

[00:02:25] Chris: Well, I don't think it's, uh, I don't think it's someone you will actually go look to hire, but I think it's a trait. It's something that if you're, if you're really into what we're about, which is, um, And I think this, it almost goes down to what is the mindset change of being an experimenter versus being an optimizer who's, who just wants to get those winning A B tests, who just wants to increase the win rates and do an amazing job and are really into the actual idea themselves.

An experimenter is more about how can we scale this, all this great stuff? How can we actually share all these practices and get everybody in the company doing it? And use experimentation to make everything they're already doing much better. And of course experimentation therapist is useful because a lot of the challenges are people and processes it's like the um I always remember some real examples where uh, when you start working with designers who are very used to working just Usually just doing what people tell them to do which isn't the most exciting thing to do And then you get an experimentation team coming along and saying actually we want you to test more variations, ideally we need to aim for say two or three variations in each A B test, A B N test.

At the start they can see it as a blocker, like more work to do. So the challenge from the experimentation therapist or the experimenter is actually to try and win them over, to try to sell experimentation into them, to change the, the whole, the whole, their mindset. but to present it in a way that it's going to benefit what they want to achieve.

And of course, that's almost, it's almost a psychological challenge, isn't it? And I think that's why it's quite fitting.

[00:04:12] Sani: Changing the way someone works. There's always psychology involved, right? The first question I have after, after what you just told me about expert, the concept of experimentation therapist is the domain available experimentation therapist. com. Do you know if it's available or not?

[00:04:28] Chris: Oh god, I haven't looked actually,

[00:04:29] Sani: Maybe you should, maybe you should.

And second, this reminds me in some ways to centers of excellence. Like you said, center of enablers as, as Robin Deboer likes to call them. So it's really about others how to do experimentation properly versus just focusing on, like you said, optimizers. But let's talk about that difference.

Optimization versus experimentation or optimizers versus experimenters. What are some other core differences there?

[00:04:55] Chris: Well, I think you touched upon it. It's for teaching people to fish. Um, it really isn't. It's the, uh, when you see the benefit, when you, I mean, a lot of us, me included, came from UX background, came from getting excited about optimization and the magic of optimization. Yeah. But then there comes a point, maybe this is just about getting a bit wiser too, there comes a point where actually, if we want to make the biggest impact we can at an organisation, it's not going to be from us personally going out there and working 10 times as hard to do 10 times as many A B tests.

It's actually, um, and I think the best way to describe that is, Uh, I was on a panel, um, last year and, and I remember a question came up and it was, So how do you scale experimentation then? What's the best way to scale it in, in our organization? And a few people answered it who were in a more of a CRO optimization team.

They answered the question by saying, Well, actually we want to bring in more developers into our CRO team. We, we, we want to bring copywriters in as well. Oh, actually we need to bring a whole lot of designers into our CRO team. We want to bring UX research into it. I was thinking, hang on a second. So you want to bring everybody in the company into your CRO team.

And of course that's not the way to do it. Those people are professionals in their areas already. So you need to go out instead of bringing everybody in to your own little world. You need to position experimentation, and that's what we do, in that how it can benefit copywriters in that team, how it can benefit the trading team, so that they can do a better job.

And it's that change shift in mindset. And then of course, you're just massively amplifying the impact that experimentation can bring to the organization. But it is a kind of completely like turning things on their head, isn't it? The mindset and the people who are not used to it. It's quite tricky and to make it work amazingly well.

That's why the, the, I think we, we, um, helped a client five years ago build a center of excellence. And then more recently, the last few years, it's been a real thing that everyone's going towards. Everyone's talking about it, isn't it? But it seems to be the best. It seems to work the best, um, in, kind of, when you go to scaling experimentation to have that kind of hybrid model and that operating model.

So yeah, that's, uh, that's why, and I love what Ruben said, calling it a center of enablers, actually, I thought that was

[00:07:22] Sani: That's a much, much nicer name. Yes. I it's still the same acronym, which I love. So he kind of politically use the same thing without getting into debate, but it's a much more appropriate name is what I think what you said about teaching men how to fish, that that's just a perfect example, because if, if your experimentation team can just.

What do you want me to do for you? And just do that every time. Sure. You might be able to squeeze more tasks and more experiments out of them. But if you teach every single department, and I think Johnny Long then had some really great LinkedIn posts. And we talked about that on this podcast as well, but when your web development division, when they start experimenting before building the feature, that's when you know, you, you're doing it properly.

So rather than build something and then when it's done, let's, you know, Get the experimenters to fix it or to make it better. No, let's just experiment as we're building it to figure out what we are actually trying to build and what the users actually need, which is even more important. So,

[00:08:19] Chris: but it does come down to, and that's why, again, the other people approach to get people engaged and bought into this idea. You don't kind of dictate a way of working on to them. You don't say, here's your new process, now like, like it or lump it. This is what you need to work. You need to make sure that you put your feature flags in, that everything is it.

You don't start like that, because that just gets people's backs up, because you're coming in. I mean, you're not a designer. I'm a creative director or something, and you're coming in to tell me that I need to change on my processes. What works much better, and again this probably goes back to the therapist type of approach, this more psychological approach, is, is to first figure out what they want to achieve.

What type of behaviors do they want to, um, to, what problems do they want to solve at the moment? What, what problems of, of, are their UX team currently trying to tackle? And then you can start to talk about, um, for a start with, with designers, we can include more of their ideas in every experiment. Because of, of, often what happens is that if you're only talking about A B testing as if it's validation, It's just a way of measuring what they're going to do anyway, right?

Experimentation is a bit different because experimentation is a lot more creative in some ways because You're not saying i'm just going to measure what you're doing You're actually saying in that design process. Did you have any other ideas that you wrote off early on? Did you actually um, maybe there was some more risky ideas That there's no way that you could have actually tested previously Now we can help you test those riskier ones as well.

We can do two variations You can do your safe one and you can take a bigger risk so you can do a more creative one Because experimentation can now let you be more creative in your art And as soon as you approach things like that that designers start thinking Oh, really? Oh, I see. So that's, I've never thought of it like that, and they can get excited about it.

And then the best feeling in the world is about three months later, and you've got designers coming to the experimentation team with all kinds of ideas. And it's almost, it's no longer somebody else dictating, telling designers what to do. It's actually designers suddenly having a lot more autonomy. And in some ways, that's how experimentation can massively benefit what they do.

But unfortunately, people don't really talk about it that way, and then they just see A B testing as still like a very technical, almost a barrier to what they're doing, and someone's coming in and telling me what to do, and

[00:10:53] Sani: exactly. That is really well said. And I'll go back to designers taking bigger swings and trying some more risky solutions or risky designs and options. It's about. Letting them try what they want to try without the risk of completely failing. So if you, if you let them experiment and test those riskier designs without, uh, the risk of completely tanking the company's revenue or, or whatever else it is that they're working on. Who wouldn't like that? Like, is there a designer that is not going to say, I sign me up. I want to do that as soon as possible. So going back to what you said, it's really about, let me help you do your job in the way, in a way that's going to be better. For the bottom line, yes, but also in a way that you want to do it in the first place.

So yeah, everyone needs a therapist because we're coming to that conclusion. If this is a therapist, everyone needs a therapist. So, but no one would ever have that need to, to basically have a experimentation therapist if there were not challenges within experimentation Units or in the entire organization.

So what are some of the key challenges and key blockers that you see in your 20 years of experience, obviously, that teams face in digital optimization and experimentation?

[00:12:07] Chris: I think the biggest problem at the moment is, is just the operating model, it's their current structure, it's, it's the responsibilities that the teams have It's like, what were they doing? And it's a way, like, for example, so many product teams at the moment are just feature delivery teams. Um, they're not, they don't have autonomy.

They're just told what to do, what features to deliver. And there's, they're not, they don't have the, um, the ownership over solving the problem. It's okay to be given the role for senior teams actually is to decide on what problems and opportunities to go after in any particular quarter. That's it, that's okay for, for that top down.

It's not really a hippo approach. That's okay. But the reality is that so often the actual Senior people are just given the solutions and the team setup isn't there. So that's,

[00:12:58] Sani: But why is it like that? Is it just how it was decades ago? And we're still dealing with the same organizational structures or did something happen and made it worse?

[00:13:07] Chris: I think it's, it's, it's always been like that in some companies. I think it's, it's, um, the other factor is if you're in a senior position, traditionally you've been, you've got there by making decisions, haven't you? So therefore more experimentation approaches kind of goes against that a little bit. So it's a little bit of a, that, that's a bit of a problem.

Um, I mean, the other part of. people and processes and team structures which gets in the way, blocker, is the fact that, um, no one's really in charge of uncovering new opportunities and uncovering those problems. It's a bit, it's not clear whose responsibility it is. You'll find that, you would think it would be the UX team and the, um, who have that, uh, who have that responsibility.

But actually in so many of the organizations I've been talking to, UX teams are busy just designing and validating ideas. That's their job. They're validating ideas. They're, they're, they're at the end of the process validating, and, and then you find they're not really involved in the A B testing side. So you end up with this, um, which is where a lot of companies start off, and ones who we start working with.

An optimization team doing some great A B testing, but very small scales, completely separate to a product team. So, so the key challenge is to, is to embed it, embed experimentation into the product. Way that product work. Um, but yeah, I think the it's just and that's like a huge one So I've just started with the huge one there

[00:14:38] Sani: And this is more about larger medium to large organizations. How is it different in, in very agile, agile, small startups, or is it different at all?

[00:14:47] Chris: I think in small startups is where they have a challenge of traffic actually. Um, so you do need to be smart with the type of A, B, A, B tests that you carry out. I mean, if you have too little traffic, it's impossible really to get any significant result as, as you know. Um, But you can still use the same kind of method.

You can still go out there and dedicate time to uncovering the biggest opportunities and customer problems and then address them. And you just need to choose your validation method. Maybe then you do need to do prototypes and usability testing or Only to validate hopefully you can start to um, find some leading metrics and still carry out some A B tests But you'll have to do it on leading metrics which are nearer to to the area Um, so that's that's what you would have to do but I can't stress enough the thing that's missing so in so many teams is Somebody even if it's one person who's dedicated to continually exploring the end to end user journey with real users real qualitative research and then quantifying that with With all the analytics and things that's the thing that's missing and then people should be Ideating off the problems that come out of that and then that gives you your roadmap But unfortunately that doesn't happen people jump straight to ideas And then they and then they retrospectively look at the prioritization framework and think like oh, um, we need to prioritize those ideas now um, therefore Oh, let's go and find some research or cherry pick some Some data to back up my idea because I really want to do it or not, which is Which is actually what led to our filth bias, which we, which we, which is forcing insights and learnings to hypotheses, which is a real confirmation bias, which is one of the next big challenge, I think, in, in what we do at the moment.

[00:16:34] Sani: Filth bias is something I have not heard that term before, but it doesn't sound good. It's not a good, clearly not good

[00:16:41] Chris: Filthy, it's bad.

[00:16:42] Sani: about the end to end optimization. And this is something that I've been thinking about. A lot. I I've been in digital, uh, in websites, e commerce for 17 years now. I've never worked in an organization where there was a person who was responsible for end to end optimization or end to end, even just following the journey of a customer.

And I'm talking the. Things that are as simple as, do you know what your confirmation email looks like when someone signs up? Has, has your optimization team looked at that? Do you know what happens a week after? Do you know what the package looks? Is there a note in the packaging? If we're talking about e commerce, why doesn't this happen in every, you can tell I'm frustrated by this.

Why doesn't this happen in every single organization? Is it that difficult to just put yourself in someone else's shoes and just order the product, go through the checkout and just even as simple as testing your website on mobile, most teams don't even do that. Why?

[00:17:39] Chris: I know, I've been talking, it's incredibly frustrating, isn't it? I think, in a way, this is why I found, I think it's so important to focus on the root cause of these problems, rather than just addressing every single form element. Rather than, because of course we talk about all the time, you know, there's date of, date of birth fields, where it's impossible to describe.

But I think the, what's missing is end to end, just even the basic usability testing end to end. And what we try to get our clients doing is just every quarter at least. Just make sure you explore the main journey with a different target audience and just explore end to end as much as possible. We've even carried out usability testing on, in two parts, where we've done the first part on the phone, which is the researching, and the second part actually in the store.

So it's a full omni channel experience to try to get to the end, from end to end. But I think the end to end user research is actually your key and the most practical way to get people to optimize end to end. Because it is there are some practical differences if you're in if you're in a large organization You have different product teams who are most likely responsible for different parts of that journey.

So there Therefore you do get these problems where people are very like only focus on optimizing one page at a time and they're blinkered and that's all they can see and and and then they end up with Stupid test ideas basically because they're just a problem. They're solving is how do I improve conversion rate on pdps?

You I mean, that's not a problem. It's like, um, or how can I get more people to buy our products from PDPs? It's like so narrow in focus. The problem may not even be the PDPs, even though people are dropping out there. The problem is up up there somewhere. It's And if only they would, um, if they conduct end to end user research, like, all the way from the start, and at least do that regularly, then the great thing that does, it kind of, it puts everything into perspective.

It means that you identify all those little form issues on mobile, um, really, really quickly, at least every time you do this. It means that you can just chuck all those JFDIs into the roadmap and get them fixed, all those silly things. But also it means that your optimization efforts are based on solving the biggest problems at the time.

And that's why I think a big thing at the moment I'm talking about, and we all are at CriticsCX, is trying to get people to prioritize the problems first, rather than the solutions. So the whole, the whole optimization field has always been focused on prioritizing ideas, which is. Again, very idea centric like come up with an idea And then retrospectively find some research to back up your idea telling you how great it is Whilst over there, there's a massive problem that nobody ever addresses because it's on like a not, not a sexy part of the site where everyone's really interested in.

And it just gets, it just remains that big usability issue you were talking about. It stays there for like years sometimes and it's never addressed. And it's because We're all too kind of page focused to idea first. And, and if, and that's why I'm really keen to get people to prioritise problems first, not the solutions to it.

And, and then that can, you can structure everything around what are the most important, five most important problems that you have at the moment. And you'll be surprised how many organisations don't actually know. It's really interesting question to ask in that one.

[00:21:22] Sani: The question is why? The question is always why? I mean, If an organization is too small, they don't have the resources to understand the entire journey and to do full proper research end to end. If they're too big, they're only looking at details. They're only looking at one page or one template or one email and just focusing on it, like you said, let's get the PDP from, I don't know, eight to 9 percent click through rate when you're leaking somewhere else and you're bleeding somewhere else.

And maybe you should be fixing that. There must be a sweet spot in the organization size where, where like the magic happens, I haven't seen it yet. I doesn't sound like you have either based on everything you told me. But my final experimentation therapy question is who's your experimentation therapy in the community

[00:22:06] Chris: Ah, oh my goodness.

[00:22:09] Sani: or what? It doesn't have to be a person

[00:22:12] Chris: I think Honestly, the therapy for me is actually just still putting in the hours watching real users go through end to end but actually it's the researchers which is my therapy when I when I um what I love doing actually when the team have been crazy busy on usability testing recently all kinds of brands that we're working with and and and I love to um, Just uh log in and watch a few sessions and that's my therapy because then everything becomes You kind of Rebalanced in some ways I start to see what really matters and I start to see what doesn't matter and all and off so often All these areas that everybody's spending tons of money on in a company just doesn't really matter for users at all And it's it's some and that's my therapy It's almost like it realigns everything and I feel kind of right This is this is the way we need to go then and it makes me feel better.

[00:23:08] Sani: That's a very, uh, big aha moment when you spend hours and a lot of money fixing something, and then you look at the users and what they're doing and, Oh my God, they're not even using this page or they're, they're going to another page. Very, very, uh, plausible scenario for any websites that is trying to do optimization, by the way.

Now let's talk about the importance of in person events for. You know, getting your therapy, let's just call it that. I don't know if that's appropriate, but what does experimentation community, especially meeting in person, uh, not just to validate the ideas you have and what you're working on, but how does that help your career, uh, as an experimenter,

[00:23:47] Chris: Oh, it's been amazing for me I've absolutely loved Going to all these different conferences of all different styles and different setups and experimentation elite has always been One of the best out there, of course in the UK anyway What I love is of course, I love learning from people who are on stage, but it's the in between bits It's all those conversations and it's the It's the surprising scenarios.

Like, there's been a few times where I've been certain of a certain approach. And then I just go to talk to different teams and different setups. And then it, they get me thinking. And then I start, well actually that's not going to work, is it? It's almost like Testing your ideas and your, what you're thinking about at the time with other people in the room in, in the, like the networking and in the evening events, that's been the most valuable thing to me.

And I think when, when Experimentation Elite started, um, straight after COVID, it was, it was really quite, um, a bit daunting to, to be honest, because it was, it was soon after COVID, we hadn't really been let out much. And then suddenly there's a few hundred people all in a room. It was quite kind of quite But it was absolutely amazing and I think a part of that goes down to what the experimentation community is like It's like they're they're very open so you can talk to anybody you can just and and again I think there's something that makes a good somebody who's into experimentation is naturally More curious.

They're naturally more interested in other people again back to it being more of a like a um A people function, like, uh, uh, like therapy, like, like psychology, it's more of it. And that naturally makes people more inquisitive and more open, I think, to other people. So, um, yeah, I mean, I'm just dead excited to see how it goes in Birmingham, the next one.

So,

[00:25:38] Sani: Cannot wait just like you. So, so what is the best way for people to connect with you LinkedIn or any other way?

[00:25:45] Chris: LinkedIn is great, actually. And, um, and then, yeah, if you want to catch up, I'm, um, I'm in London a lot more now since, um, tend to go in three or four days a week. So I'm up. What I love doing is just having, like, um, a coffee catch up with people and just discussing all this stuff, so,

[00:26:02] Sani: So your LinkedIn profile will be in the description and let's get that domain, uh, experimentation therapies. com before someone else does. Let's just make sure that's, or let's create a custom GPT agent called experimentation therapist that people can use and bounce ideas off of. Just getting there. Uh, final call to action to everyone listening to this, go to experimentation elite.

com. Get your ticket today for the first time ever. It's a two day event. So June 11th and June 12th. A lot of things happening. Chris will be emceeing the event for the first time in his life from what I hear. And, uh, to everyone listening to this episode, thank you very much for doing that and please consider rating, reviewing and sharing it, and I will talk to you next week.