Experimentation Elite Awards with Judges Dinah Alam and Joe Doveton

​

[00:00:15] Sani: Welcome to NoHacksShow, a weekly podcast in which smart people talk to you about better online experiences. It is that time of the year again, the winter's almost over, the days are getting longer, the smell of experimentation awards is in the air. And this is another episode we're doing with Experimentation Elite, the UK's premier experimentation and CRO experience taking place on June 11th and 12th in Birmingham.

So head over to experimentationelite. com and get your ticket now. But we're not talking only about the conference. We're talking about the experimentation elite awards and the award show is recognizing the best of the best in our industry. And to discuss that today, I have two guests, two of the judges on the panel, Dina Alam, head of UX and CRO at RocketMill and Joe Dovedon, founder and CEO of BinaryBear.

Dina, welcome to the podcast and Joe, welcome back to the podcast.

[00:01:05] Dinah: But

[00:01:06] Joe: Thank you.

[00:01:07] Sani: Really nice to have you. And thank you for just doing this, for selecting the best of the best. I understand it's not an easy process. It's a lot of work, from what I heard. On top of the work that you need to do already, right?

[00:01:18] Joe: Uh, yeah, you, uh, yeah, you could say that. It's hundreds of man hours go into this, this thing.

[00:01:26] Sani: Oh wow. So Joe, tell me about, you've been in the CRO industry, experimentation for a while. Uh, tell me about awards show in general, and specifically, experimentation league, and why it's so special.

[00:01:37] Joe: Yeah, sure. So, um, my background is years ago I used to work for technology publishers. And so, um, I, we used to have, uh, magazine based, uh, awards events. And so I worked on awards events for things like, uh, T3, the gadget magazine, and so on. And then when I, um, moved more into the digital marketing side of things, I was, I attended and entered things like the European Search Awards.

And I kind of thought, you know, this is a space that's absolutely right for, um, for doing something similar. Because. CRO experimentation is a, is a very broad area, but a big chunk of it is at least, you know, performance led or measurable. So it seems like a good candidate for, for doing, uh, for doing something in that space.

So about 2017, I looked into, I was started working on, um, what I was called the CRO Awards at the time. And I got wind of the fact that Jackie. was also looking at doing something. So Jackie's the, uh, the lady who runs Experimentation Elite, the conference. So we just kind of joined forces and, uh, we ended up from, from there.

So I think it's, it's my gut feel was that this is something that, um, That the industry would like and that's it. The response to it's been absolutely incredible. And, you know, we we've even had people in tears, uh, who've been when they've been given awards and, you know, hopefully it's going to become a permanent fixture in the sort of CRO experimentation calendar now.

[00:03:10] Sani: When was the first one?

[00:03:12] Joe: So we did the first one, um, was in 2020. Now, obviously that was when COVID hit. So, uh, you know, that one, like all conferences that year actually was, was, it was an in person, uh, awards event. Uh, then we had the first, sorry, it was a, it was a virtual one. We had the first in person in 20, um, 2022. Uh, and, uh, we continued it again last year.

So this is the fourth, uh, time that we'll be running in 2024.

[00:03:41] Sani: And the interest has been growing, I would assume, from from the audience and just people getting excited about this event, right?

[00:03:48] Joe: Absolutely. I mean, we're now getting badgered by, you know, people saying that when you're going to, when you're going to open the category, uh, option on the website, we're, you know, we're ready to go. So, um, yeah, people seem to be very receptive to it.

[00:04:02] Sani: That is absolutely incredible. I mean, this is year three, basically, of the

[00:04:07] Joe: Mm hmm.

[00:04:08] Sani: event, and it's grown significantly. So, you two are on the panel. Who else is there?

[00:04:13] Dinah: So, um, we've got, um, some legends on the panel. I mean, me and Joe are obviously really experienced judges and specialists in our industry as well. But we've got, um, Yorgi Yorgiev, um, author of Statistical Methods in Online AB Testing. If you've not read it, you should. Um, Tim Stewart of TRS Digital, Craig Sullivan, Optimize or Die.

They both have won awards themselves at Experimentation Elite for being legends in the industry. Our new Telemans, um, online market think, and we've got two new judges this year, Joe. Um, we've got Iqbal Ali, um, he's a consultant and also an illustrator, but he worked at the train line for four years, heading up their optimization team.

And Abby Howe, um, uh, user experience and optimization specialist. So, um, grow the team out of judges so you don't feel like you've got the kind of same people looking at the same awards all the time. But one thing I would say is like, you know, to add to what Joe said about how the awards have kind of become a kind of important part of the CRO calendar for the year.

The standard has also gone up every year and that's really great to see. Not only are people really engaging with it. But this whole industry is, you know, changing and the awards entrants that we get put in every year really show the way the industry is moving on every year as well. So it's a great thing to be involved in.

[00:05:40] Sani: absolutely. And the panel, uh, I'll just say, uh, uh, Craig, Abby and Iqbal, former guests of this podcast and dear guests of this podcast, and wonderful people as well. So, uh, hey, you may have a lot of work to do, but it's with nice people. So let's, let's

[00:05:58] Dinah: And what a chance to get your work in front of some of those people as well. And, you know, sometimes, you know, you get the, uh, you get Craig reaching out to people to give feedback, so you start, um, you know, it kind of, So it helps the community atmosphere, and that's what Experimentation Elite is all about.

It's about, yeah, it's about networking and building your network and, um, really being part of a community.

[00:06:19] Sani: sounds absolutely beautiful. So let's talk about, you know, tell me about the categories for the awards, because obviously that's what people need to know before they apply and, and how they can apply and all that.

[00:06:27] Dinah: Yeah, and these are still aligned with what we had, um, last year. We've, um, kind of aligned campaign awards with, um, industry sectors. And hopefully there's an industry sector here that, um, you feel that your work can fit within. And then we have practitioner awards as well, um, and then team awards. So, um, I'll start off with the kind of campaign awards.

The reason why we call it a campaign award is that you could submit, um, you know, it could be a single experiment, or it could be a whole series of work and a program of work that you've done in your organization or for one of your clients. So, um, we've got a charity, um, campaign award, e commerce, finance and insurance, um, SAS and B2B.

Travel and hospitality, and then media, marketplace, and real estate. And, so there's six sectors there, and if you needed reminding about those, they're obviously really clearly displayed on the, um, awards website, which you'll get to through experimentationelite. com. And, um, I don't know, Joe, do you want to go through the practitioner and team awards, or?

[00:07:33] Joe: Yeah. So, um, you know, it's really about kind of. You know, reward rather than individual programs. It's kind of rewarding individual achievement in the industry. So, you know, we've, um, we kind of varied this over the years, but we kind of have like a overall experimentation team with award for in house teams.

So if you're working for a brand. You can submit your team for, for that. Um, and also agencies. So there's an, there's an experimentation team for agencies. You don't have to be a CRO specialist agency. You can be, um, an agency, a full service agency. It's just got a really good, a really good team that does that.

So, um, and yeah, in terms of individual awards, there's kind of, um, an award for, um, You know, overall individual achievement. There's also a community award for somebody who's contributed, um, to, you know, communicating great ideas, um, through whatever method, whether it's a podcast like yours, uh, or whether it's a blog, um, you know, that you can, you can submit yourself or you can get a friend or a colleague to, to put yourself through for that one.

[00:08:49] Dinah: Yeah, and, um, we had, to be honest, we had one individual, um, practitioner award, um, last year for agency or in house. And the, um, entrants we had were so diverse, um, from entrants that are doing a load in the community, like, um, you know, reaching out, trying to train people, trying to share their knowledge, um, Entrants that are really rising stars, um, Entrants that are really doing innovative work with technology, AI, etc.

And they became quite uncomparable.

[00:09:24] Joe: Yeah,

[00:09:26] Dinah: that's why we've ended up this year with rising star, community and industry contribution. And we look forward to seeing the entrants that we get this year. We'll probably end up with another category because obviously That happens, but it's, it's, it's really quite exciting.

And then something to say about the awards, if you're not sure, um, where you sit. So like, it's always really hard to chop up the whole world of, um, you know, kind of optimization into business sectors. Um, you might feel that you've got a website that you work on that crosses a couple of sectors. Um, so just reach out and ask us, um, like we'll always try and get back to people to support them getting their entry in.

And yeah, as Joe says. If you want to, um, submit an award on someone's behalf, um, if you work in an agency or in house and you're a manager and you've got a rising star on your team, then, um, you know, it's great for practitioners to be entered by multiple people. And in fact, when we see an, uh, An entry come in and then we see a, another entry come in for the same person.

Um, it gives us a really strong view of that person's work from multiple angles. So it adds a lot of power and authority to their, to their entrance.

[00:10:35] Sani: That's a really good point. Is the Rising Star a new award? Is it new this year or has it been around before?

[00:10:42] Dinah: It was new last year because we, um, pretty much had people that were kind of new in their role, um, within the last couple of years that had got, um, really, really, um, good at best practice, I guess. And, and then people that have been around in the world a while and were able to really share their knowledge and do training and stuff online.

So we just felt like it was worthwhile separating out. So it's, it was new last year.

[00:11:07] Sani: I think it's great. And once you have a former Rising Star winner win one of the senior awards, you know you've grown as a conference and this has become a huge thing. So I hope that happens in the future for the last year winner or this year's winner at least. So, um, the address is experimentationally.

com slash award. All of the links will be in the description of this episode, obviously. What's the process like? Like you go there, and what do you need to do?

[00:11:31] Joe: Yeah, sure. So it's a really good question. And we get a lot of, we do get a lot of people reaching out saying, you know, am I eligible? How do we, you enter, the whole mechanism is online. I think you mentioned that a minute ago, Sally. So it's a web form. It's on that page that you mentioned. I'll repeat again, experimentationelite.

com slash awards. Um, there's a click on the entry submit button and, um, after you, then you add your contact details in, you need to do that first. And then there are basically six sections that you need to complete. Um, the first one's the problem, the problem of research. Um, second one, hypothesis and build.

The third section, solution and variation designs. Uh, fourth one is planning data. Um, fifth one's results data. And then the sixth one is outcomes, you know, what happened from that. The reason we do this is we, we kind of don't want people really to just sort of submit their case studies, if you like, that they might do on a sales pitch.

We, the range of people that we've got on the, uh, the awards panel is such that, you know, people have got specialists in different areas. We've literally got the guy who wrote the book on A B testing statistics, you know, on

[00:12:44] Sani: Don't, don't try to fool that guy,

[00:12:46] Joe: Absolutely. So, you know, we

kind of want to see, it's a bit like in a maths paper where you kind of want to see, you don't want the answer, you want to see the workings.

So we kind of, we, we look at the whole thing. We're not, you know, it's not about saying, Oh, this experiment had a bigger uplift than another one. Therefore, that one should win. Actually, we're interested in people who had a unique challenge and then approached it in a really innovative way. And actually, if it's a program, you know, it's, it's going to be a more interesting entry if they can say, well, these experiments didn't work.

And these are the reasons that they didn't work. And this is what we learned from it. And this is what we did next on the back of it. Because that's the whole point of the process, right? So, um, yeah, so we, hence the reason it's broken down into chunks. We want to see the thinking that went into every single step.

Um, and they will all be equally considered, not just. We did this and we got this result. Um, so it's, it's not, it's not super quick to fill in, but, but we're not talking, you know, hours or days. Uh, it just needs a little bit of thinking. What we've done this year is we've also produced on that page, there's a little breakdown of, these are the questions that we need to consider so that you can do a bit of prep before you start submitting your entry.

And, um, Craig Sullivan's also, uh, got a little video, uh, which is basically kind of introducing, this is the awards, this is what we're looking for, and this is how to do an entry. So, you know, hopefully there should be enough information online. Um, but if you, as Dina said, if you're in any doubt, just, just, uh, get in contact and we'll get back to you.

[00:14:24] Dinah: Yeah, definitely read that information first, um, the more information we've been putting out about what we expect from an entry, the, the higher the standards gone up to be honest as well. And, um, yeah, there's a, a video and definitely watch that first and download those PDFs as to what to expect. Um, cause if there's something missing, um, Whilst you maybe are copying and pasting from a case study sometimes.

Um, and then you end up feeling like there's a form field that you can't fill in. There's form fields that we've probably, um, got some scoring mechanism against it. So you can get scored down and, um, talking about, um, Jorgi, he's quite a harsh scorer actually. Um, sometimes when something's left blank. You know, do you score it, um, you know, 50 50, because you think I'll give the, give the entrant the benefit of the doubt.

Some judges are, that's a zero, because you missed out on the question, so make sure you fill in everything that's asked of you, and have a good think about it beforehand, and absolutely, as Jo said, we, we're interested in, you know, how interesting and innovative it is, um, so, um, have a good think around the whole, whole piece, and how you make sure that your, your entry stands out, um, but write it well.

Keep it on point. We have sometimes hundreds to go through. Um, don't find it, don't make it too hard for us to find the critical points that you want to make as well. Cause, um, then, you know, you don't want your story to get lost. And, um, yeah, make sure the evidence is really strong for why you're doing what you're doing.

Um, and it's, um, becoming harder to do it. And so if you get shortlisted for a winner, it's a huge accolade. So, but I would say it can be quite tough to get through. Yes.

[00:16:06] Sani: a very, very good point. So, as you said, write it well, but keep it on point, because if it takes a while to write it, and there's hundreds, we have the judges to think about. We don't want them to spend days reading these applications. But another thing about that video, How to Win Awards and Impress People by Greg Sullivan. If you didn't get something CRO related explained to you by Craig Sullivan, like do you even do CRO? So go and watch that video because it's good. It's pretty good. It's Craig Sullivan explaining things and that's kind of the point of CRO That's how a lot of us got started. That's how I got started with CRO at least.

So, uh, yes How many submissions do you expect this year? Can you share that?

[00:16:45] Joe: Um, well, like Dina said, sometimes there are, you know, uh, in the low hundreds, I would say. Um, but yeah, I mean, obviously the, you know, we, we want as many people to put themselves forward as, as possible. Um, the agencies, to be fair, are very hot on this and we get a lot of agency submissions. Um, we'd love it if audiences, if your audience for this podcast who are at their, at the brand also get inspired to submit an entry.

We wanna see, uh, as many, um, you know, submissions from brands because sometimes there is a certain amount of homogenity amongst the way that agencies approach things that you don't necessarily get at brands. Maybe brands do things in a slightly interesting or slightly different way, and if you are doing something that you think is interesting, yeah, sure.

We would have a. We want to have a listen to what you're doing.

[00:17:41] Dinah: Um, I think that's probably indicative of the industry. Like, we get a lot more in e commerce than we maybe do in charity. But I know that charities are testing and they're doing experimentation. Um, same goes for, you know, finance and travel. You know, a lot of clients in my own business that are obviously experimenting.

So, you know, um, get these, um, Category's filled and we want to hear from more and more people every year.

[00:18:08] Sani: and the deadline is

[00:18:10] Dinah: Oh, yeah,

[00:18:11] Joe: 15th of

[00:18:12] Sani: 15th. Yes.

[00:18:13] Joe: So, you know, the clock is ticking.

[00:18:16] Sani: And then, uh, obviously the shortlist will be announced at some point between March 15th and the event itself. And then,

[00:18:24] Dinah: Well, we actually have a deadline for that. We oh, there's a deadline for that as

that we stick to. Yeah, I can't remember,

[00:18:29] Joe: it's April the 15th, I think. So if you just think the 15th, yeah.

[00:18:32] Sani: So something like that. Uh, and Dina, did you go to the previous events, the Experimentation Lead Award Shows?

[00:18:39] Dinah: Yes, and lots of fun and, and, Joe did as well.

[00:18:42] Sani: So walk me through that day, the award ceremony. You, you mentioned tears, you mentioned tears from some of the award winners. So what, what did that look like?

[00:18:50] Dinah: Well, it's actually an event that happens after the conference event. Um, and obviously this year, Experimentation Elite is over two days. So, there'll have already been a day of learning and networking events, and then we'll have the main conference, and the awards happens as the party on the second evening.

So, um, it's like, you know, it involves food, it involves wine, and it involves music. Um, so it's, you know, a real celebration of not just It's the, um, shortlisted teams, but also a kind of wrap up of the whole conference event, so it's lots of fun. Um, and it takes the kind of format of, you know, everyone sits down and has a bit of food and then the awards are then announced in a kind of quite glitzy kind of way.

[00:19:35] Joe: I think the first, the first year we did it in person, when you've got a, a dance, i, a, so a dance floor full of optimizers, uh, super geek optimizers. I thought, my god, my work here is done. You know, when you've got, I dunno, John Alderson and uh, Charles Meen bopping on the dance floor.

[00:19:57] Dinah: There were some good moves.

[00:19:58] Sani: What's a good

[00:20:00] Joe: Yeah.

[00:20:01] Dinah: I, from what I can

[00:20:02] Joe: And then an after party, after party.

[00:20:04] Dinah: Oh yeah, I think at quarter past three Jay, you were still asking me if I wanted to go out to a club.

[00:20:10] Sani: Oh, wow, that, that is

[00:20:12] Dinah: Couldn't keep up with Joe, but, um, yeah, um, it'd be, it'd be lots of fun and I really look forward to it. So it's a great way to, um, you know, get your team involved and if you get shortlisted to celebrate it.

And I, you know, what better way as well to, um, report back to your stakeholders, especially if you're trying to get recognition for your test and learn culture. Um, brilliant way to kind of go, we got, you know, we got shortlisted, we went to this event, we were able to, um, you know, Celebrate, um, our hard work and what a way to kind of give your, your, your work some acknowledgement as well within your organisation.

[00:20:47] Sani: And one more thing about the awards. There's an actual trophy that people get. It's not just, you know, you won this award and here's your PDF file. There's an actual take home trophy that people get. I think that is important because

you can put that on the shelf with your name on it. Okay.

[00:21:03] Dinah: absolutely.

[00:21:04] Joe: Like I say, I've seen photos of people, you know, afterwards, holding their awards at home. And I'm sure I saw a tear in the eye. Looking slightly, slightly glassy eyed, where they're proudly holding their You know, experimentation, uh, uh, elite campaign award up,

but Yeah.

there's a, there's a real trophy.

[00:21:23] Sani: So one, one last thing about the event itself, not the award show. If you need to pitch this to someone, Elevator pitch, basically. How would you convince someone to go to the award show? To the event, not the award show.

[00:21:36] Joe: Who wants to, should we never go with that, Dina? Well, I mean the name, you know, the, the, the, the name is in the title, experimentation elite. So this is, it's, it's the creme de la creme of the experimentation, CRO, optimization, whatever you want to call it, industry. Talking about the, the live topics that are, uh, that are, are, are, uh, in the minds of the most prominent practitioners.

So, you know, we've, we have some of the best names from the biggest companies, the biggest agencies, the biggest brands, um, who are speakers. Um, you know, it's, it's elite topics for people who are, uh, who are already at the elite stage or aspire to be, become an elite practitioner. It's, it's not. Really entry level as such, but if you were new to the industry, you would learn so much by going through to this conference and listen to those speakers.

[00:22:37] Dinah: Yeah, absolutely. If you're an experimentation specialist, CRO practitioner, data analyst, growth marketer Working on a product team that are experimenting, a UX designer, um, working around experimentation. Um, there'll be content for you at the conference and especially now the conference is extended and the event is extended over two days.

So, um, there's loads of valuable stuff to learn. And these are also the people we want to, um, hear from in terms of entrance to the awards. Um, if you want to make your experimentation program award winning. Get the recognition you deserve, then get your entries in by March

[00:23:18] Joe: And it's not like some conferences where you get superstar speakers and they clear off after, you know, this is, you mentioned the community earlier on, Sani, you know, everybody sticks around, you can grab somebody over a coffee or a beer. You know, pick their brains. Everybody's really approachable. It's small enough to be, you can get around the whole room, you know?

Um, so yeah, definitely come along. A

[00:23:44] Sani: to be elite, you gotta hang out with the elites. Like, that's your step one. And this is where the elites are on June 11th and 12th in Birmingham. So, get the Peaky Blinders references ready. I had to throw that in. I just had to. It's a great show. And, uh, to two of you, get your coffee ready, because there's gonna be a lot of sleepless nights.

For you to, I guess, going through all the submissions.

[00:24:06] Joe: of long nights, I should think.

[00:24:08] Dinah: I did a lot of reading of awards on, um, on trail and bush journeys last year, so I expect to be doing the same again this year. Every, every sort of spare minute in between work was spent reading somebody's awards, so yeah.

[00:24:20] Sani: on behalf of everyone doing CRO, uh, I want to thank you for doing this for, for, for just. Donating your time to, to picking the best of the best. And on that note, uh, I want to thank you for being on this episode. It was really a pleasure to talk to you and I cannot wait to meet you in person in Birmingham.

And to everyone listening to this episode, please consider rating, reviewing, sharing, and spread the word. Experimentationelite. com slash awards, or without the slash awards, depending on what you want, what you are interested in. And I will talk to you next week.