Documenting Experimentation with Nils Koppelmann

[00:00:00] Sani: Welcome to no hacks show, a weekly podcast in which smart people talk to you about better online experiences. My guest today is the founder of 3tech agency, publisher of your weekly dose of experimentation, co founder of conversion stash, and constantly driving growth through experimentation. Nils Koppelmann, welcome to no hacks show. 

[00:00:19] Nils: Hey, Sani, how are you doing?

[00:00:21] Sani: Doing great. Such a, such a pleasure to talk to you on this podcast, man. So, uh, before we start talking about the main topic, I have a few questions. The main topic is of course, documenting experimentation, you know, the boring thing, nobody needs, nobody wants to do, but everybody needs to do basically. I mean, I was a web developer code, you know, code comments and documentation.

Same thing. Nobody wants to do that. And then you regret it. Eventually. The first question for you. And this is something that I started asking recently in this podcast. So let's say you need to hire someone for your experimentation team, your CRO team, whatever, but because the economy is as great as it is in 2024, everybody has a job and there's no humans available.

So you have to hire one animal. Who are you hiring? Which animal, not who, which animal are you hiring for your team? You have five seconds.

[00:01:06] Nils: Alright, I think I would hire somebody who can wear multiple hats, because especially when you don't have too many people to hire, you need someone who can wear multiple hats. But eventually, as you get more mature, then be able to specialize in one direction and

[00:01:25] Sani: Which animal, which animal is that? Which, which animal is that?

[00:01:28] Nils: In the CRO context, I would probably be a conversion manager, uh, who's very good with talking, uh, talking with people.

[00:01:34] Sani: No, I mean literally which animal would do that job in

[00:01:37] Nils: Oh, rich animal? Like, literally?

[00:01:39] Sani: animal literally animal. Yeah

[00:01:41] Nils: Um, a chameleon.

[00:01:45] Sani: I love that. That is so, that is so good. Yes. That is amazing. Thank you very much for this response. Next question is an audience question and it is from Bjorn Brunenberg, who I just met for coffee two hours ago. Great life living here in Portugal. Why should people even care about documentation?

[00:02:03] Nils: You just mentioned, uh, that you come from a web development background and, uh, that a lot of times when you do not document, and presumably that is It's just even the notion of writing comments in your code, um, make, like creates problem downstream. And the same thing is true for, for running experiments in a CRO program.

Think about all the experiments you've ran in the last six months. I bet you do not remember the majority of them. And even if you do, um, try to draw conclusions, um, from these experiments, uh, after you've concluded them. And. While it seems like documentation is this tiresome, boring topic, if you do it correctly, and if you, if you actually do it with the intent to learn and to, to foster a culture of experimentation, because that is part of that entire thing, you can get so many benefits from doing the simple task of writing down what you do.

Um, and even if that means sometimes having to write down that an experiment. did not go the way you expected. So there you

[00:03:12] Sani: Especially that I would say, so it's basically, it's like insurance, but also more. It's not just covering your ass in the future. It's also about learning and making it easier to learn. Thank you. From everything

[00:03:22] Nils: I think, I think the experimentation part of that is already the insurance. Um, but, um, knowing to document what happened and how helps you actually get the money back from the insurance in a way.

[00:03:35] Sani: well said. Very well said. So now you, you run an agency called 3tech and you do experimentation and growth. How did you get into experimentation? And when did you start your agency and why did you decide to do your own thing?

[00:03:48] Nils: Yeah, just just for clarification, um, like about half a year ago, we transitioned from offering the typical agency services to, um, now being, uh, being more on the consultative side, still offering agency services, but primarily, um, consulting with clients. But how did we get there? Um, I think in 2017, I created the, Agency at the time with a heart, like strong focus on web development, web design, um, at some point realized, uh, fancy, fancy web designs, fancy relaunches is not just the way to go.

Um, because clients were actually putting, uh, into the, the, um, into the briefings, uh, goals. Right. And I was like, wait, we just want to fancy new website. And they were like, no, we want to increase conversion rate. We want to have more leads. Uh, and stuff like that. And so I was like, okay, that's interesting.

So, and this like started with, okay, let's first analyze websites before we, uh, throw the idea completely away and start from scratch. Let's start understanding how users interact on the site. Uh, start, yeah, this entire process that we now potentially do as part of an audit, um, and that, uh, step by step slippery slope kind of thing, uh, led to, uh, delving into CRO and Uh, getting to understand and, and, and fidget around with A B tests.

And well, that's just where things started and it never ended.

[00:05:16] Sani: Never answer that. That's exactly what growth marketing and experimentation is. Uh, so what's the onboarding process for you with the client? Like they, they decide to work with you. How do you start?

[00:05:26] Nils: Uh, it's, I would say it totally depends on the kind of problem we're trying to solve for them. Um, so if you have a typical Ecom client who wants to, who comes with, um, the wish to say increase conversion rate or whatever it is, um, usually we would start with an audit to understand a like. What does the site do?

Where do current problems lie? And that then includes, um, like quantitative, qualitative research, but also understanding the company and talking with people there. Um, and then starting, usually starting to build up, um, or hook into, um, uh, testing or CRO program that they either already have, or, um, we're just building it up.

Um, that's one scenario with other companies that have. Um, more of the wish to who are already more mature, um, and, and want to build up their experimentation program or, or extended, uh, rather, um, make it spread throughout the entire organization. Usually we'd start by again, doing an audit, but with a focus on finding out what their level of maturity is.

And, uh, yeah, well, with the goal to, to understand how can we help them to, uh, level up their, their experimentation game and. ultimately win by testing, right? Or win by failing in a way.

[00:06:40] Sani: just went by doing something and learning exactly, so let's move on to the main topic then the, the, the thing that most people don't want to do the documentation, like, why, why are we even talking about this? Why isn't this something that is like, you know, you need to have to brush your teeth every day and there's no need to tell people they need to brush their teeth every day.

Why, why aren't we at that level yet? Or will we ever be, I guess.

[00:07:05] Nils: So the, the point is when, when you stop brushing your teeth, people will at some point stop talking to you. Uh, in, in the case of experimentation and documentation, it is not that easy because, um, sure, if you don't document, um, at some point you will not, it won't be the people will stop talking to you, but you will have to you will stop having answers for questions that will come up eventually.

Um, and that creates a problem, uh, in an organizational sense that creates problems in, um, people feeling less confident in what you're actually doing. So documenting has, uh, like a few pillars. Um, one of them, uh, being actually having a log of what has been done. Um, to, um, creating visibility for what has been done and for what's been going on for a lot of learnings that have been drawn, um, not just for your team, but maybe for the entire company, because eventually what do you want to have is you don't necessarily just want to have five, 10 people working on experiments, which is already, um, more than probably the majority of companies have, but you want this information to spread out across the company.

You want to be more people to be engaged or at least interested in what you're doing. Um, and. . The only way, not, not the only way, but, but a good way to do this is to, to document, um, the two of us. We've been talking about content creation a lot and one of the notions is, uh, document and not create. And I would say to a large degree, this is also one of the success factors in experimentation because if you document, you're already doing a good part of the work.

Um, and you're helping to, to move the motion forward, spin the wheel faster. Um, yeah,

[00:08:52] Sani: So, uh, wonderful and eloquent answer there. Uh, the idea is that this is not something you do once every three months. Let's document everything we've done. You document as you go continuously, that this is the only way that really makes sense. That is sustainable. Now, what are the tools that you use for documentation?

What is your favorite, your go to tool for documenting experimentation?

[00:09:15] Nils: so we'll, we'll, we'll get to the point of what my favorite tool is. But, um, if we talk reality for what most companies are working with today, uh, documentation usually lives in a Jira, right? If depending on how, um, how the company or the organization is built up, a lot of things happen in Jira and Confluence.

Um, Some might have an Airtable, uh, some work with Google Sheets. The point for me is not necessarily like what's the tool, but can you make the tools that you have work for you in the way, um, that support documentation, um, that build institutional memory in a way. Um, and that is a challenge, um, because usually tools are not built, um, With the idea in mind to store experimentation data to store, um, and then also in a way run experiment analysis or run, um, meta analysis on the data you have.

Um, and this is like a little pitch, but, um, it's not out yet, but like why I started creating, uh, like a software to, to, to help with this entire process to go from first experiment ideas all the way to running meta analysis of experiments you have and make this data available and accessible to the entire organization.

But that said, people should use what they have and not over complicate about tools, um, because we're already doing way too much of that in regard to A B testing tools, in regard to everything else. So if you can document today in an Excel or a Google Sheets. an Airtable or your Jira or Confluence. I would highly recommend just doing that, but following a rigid structure.

That means not just documenting today, uh, and, and the next day documenting a different way, but being consistent about what you document, um, and creating a way to make things searchable. I think this is, like, one of the things, um, that will help you later also make this data accessible. Um, yeah, I think that's one way to start in the future.

[00:11:24] Sani: So have a system is, is the number one thing and stick to the system and make sure it works. Let me touch on that software real quick. And we'll talk about that software on this podcast when it's out. If that's okay with you, we can have another episode and we can talk all about that software because I cannot wait to hear about it, but basically if, if there's an organization that does some experimentation, but.

Barely documents anything or just documents nothing at all. You know, uh, What is the minimum that they can do to say this is acceptable level of documentation compared to what you've been doing before? What is, what is the absolute minimum to document that experiment?

[00:12:03] Nils: In, in my opinion, and I know other people don't agree, but documenting and writing down a hypothesis pre starting or even running or building the experiment. And then after the experiment is finished, however you are doing that on fixed horizon or other ways, documenting the results. This is, I think, the minimum that you need to do, um, knowing what you're testing, what the reasoning is behind it, um, noting down, um, the, the KPIs or metrics that you're tracking, um, and being very, very clear about what that is, which is the primary metrics, secondary metrics, uh, guardrails, et cetera.

But this is maybe even very advanced for some, um, But if you'd ask me what's the minimum, that's what I would answer probably, yeah.

[00:13:00] Sani: basically it's going back to having a system for how you're documenting. I think if you, if you do that, You'll at least have some, that's one thing to push you into documenting and you know how to document. So for those who are not doing it, like even having a spreadsheet, even as basic as, as a sample spreadsheet, and that might be good enough for years for an organization, but I want to hear what your favorite tool is.

[00:13:25] Nils: My favorite tool today? Well, I'm, I'm, I'm building it, but, um, I hate that question because the favorite tool, um, for me is obviously the one, uh, that, that I'm building because I'm, I'm building it to what I believe, uh, to be the, um, the end all, be all solution. But, Um, I would generalize a bit. The best tool that you can do use is the one that you will actually lose, uh, use and not, um, not try to find, uh, the next best shiny object.

Documentation is not a shiny object and it shouldn't be treated as such, but it should be something reliable that, you know, people will actually use, not just yourself, but actually that it fits your organization. Um, And with that in mind, the Jira is perfect if, if that's what you're using daily. Um, and yeah, just making sure the, the, the principles I mentioned before apply.

[00:14:25] Sani: That makes perfect sense. And, uh, and I think that you sent me before this recording is who should be responsible. Like this, this is like a big question that needs to be answered. You have, maybe you have a team, maybe you have a person for experimentation in small organizations. It could be just one person running the whole thing or a consultant who should be responsible, who should own the documentation process.

[00:14:48] Nils: Um, depending on the kind of organization, um, the documentation process eventually should, um, should be steered potentially or overseen Um, by people who understand the greater picture, um, in the, if you have a larger organization, but, um, ultimately everybody should know how to document, um, and not need any further guidance.

But if that's necessary, then maybe something like a center of excellence would step in and help, um, owning good question. And I would say owning in that sense, if you have the resources. using a center of excellence who builds up the ideas and these kinds of things. Um, in smaller teams, um, I would make one or two people responsible for, for documentation.

Um, but not in the sense that they document everything. Um, but in the sense that they make sure people actually document. Um, and yeah, that's, it's very, it depends on very much on, on who also runs experiments, who plans experiments. Um, if you have developers running and planning everything, maybe it makes sense that they get help from project management, um, or they document themselves and own that process.

[00:16:07] Sani: So that's a good point. I think, I mean, The more people get involved in contributing to documentation and experimentation in general, the healthier. That organization will be long term. And I just recently spoke to Ruben DeBoer about centers of excellence or yeah, you just mentioned that. So that, that's kind of his thing as well that he loves to talk about, uh, and a fascinating talk about it.

Uh, so what are the biggest, like, what are the top two or three benefits that an organization will have from, from documenting their experiments? And I mean, beyond just being able to remember what they did and justify the money that they spent.

[00:16:45] Nils: Yeah, so I think justifying the money they spend is, is, uh, still on another page, um, because for a lot of companies that is not as easy and simple, uh, to answer, um, if we're talking ROI. But from. If we just look at documentation, um, visibility is for me, if you actually spread that information out into the organization, visibility and accessibility are probably the one or two most, um, important, uh, benefits, um, and being able to.

yeah, to refer back to experiments. I mean, you mentioned this like might not be on the list, but I think this is still one of the top, the top reasons why you should experiment, document your experiments. Um, and yeah, at the end, I would, I would argue if you make it as easy for people to, to, to document it's actually, and, and, and automate a lot of the otherwise busy work, it even is a bit of fun.

So, um, because you actually see the value you're creating, you see the kind of work you're doing, and this gives people working in a company also a tool to show upper management or their higher ups or however the company is structured, uh, that they're, that they're contributing to, to, to the company goal.

Um, which is otherwise something they'd need to do somewhere themselves showing, Oh, look, this is how many experiments I've done. So I think it can contribute to the culture of wanting to run experiments.

[00:18:21] Sani: that's a great point. I think, uh, if you get the full buying from the entire organization and get everyone to contribute, let's start with ideas and then documentation as well. I mean, someone who doesn't have a lot of say in what the website will have on it, like what, what, what it will say, who gets an AB test from their idea.

That's a huge morale boost for that employee. It, it, it really, yeah,

[00:18:46] Nils: It democratizes in a way also, uh, who can contribute to, um, company overall company success. Um, but it also shows, um, what kind of ideas work and which don't, um, without necessarily pointing too much fingers at, oh, look what person A, person B did, because it creates a plain level field, like, like a level playing field, um, uh, for people to understand this is about the thing.

This is about our customers and not about who has the best idea. But on the other hand, provides them with the means to show their contribution to, to the overall, uh,

[00:19:26] Sani: It also leaves ego in front of the door outside. If you have a democratic organization where everybody can contribute and win with their tasks,

[00:19:35] Nils: yeah,

[00:19:36] Sani: there's no hippos, there's no ego, there's nothing like that. It's kind of a beautiful thing if you reach

[00:19:41] Nils: it, if you reach that level, that is,

[00:19:44] Sani: If you reach that, of course, of course. But there are many hurdles to reaching that level.

Some of them being hippos or most of them being hippos, but there are many, many hurdles to getting there. One last thing I want to talk to you about is your weekly dose of experimentation. Tell me about that. Pitch me that. And the link to subscribe page will be in the description. So tell me about that newsletter.

[00:20:06] Nils: sure. So, uh, in all honesty, it hasn't been too weekly, uh, recently, but, um, on the track to, to getting back there

[00:20:13] Sani: still in the name. So it counts. It's in the

[00:20:15] Nils: um, so, so. Experimentation Tuesday and like your weekly dose of experimentation came, um, from the motivation to, um, to make accessible the resources information that I on a weekly basis come up with.

Sometimes it's obviously from, from a different week, but, um, I regularly share with people in the space or ask me, uh, stuff. I regularly share resources. Um, and I just wanted to be able to, uh, do that on a larger scale, help more people. And so far the feedback I've been getting, um, it is, it is amazing. Um, and I think it is, this sounds so cliche, but it is probably something I would have wanted, um, starting out, um, when there maybe was not as good of a resource directory for, or like a single source where I could get things from.

That being said. This is not the only amazing newsletter out there. There is amazing others, um, that I cherish and, and, and like reading. But yeah, that's my take on, on how it

[00:21:22] Sani: But that's the one we're talking about today.

[00:21:24] Nils: Exactly. There

[00:21:25] Sani: about it. Yes, of course, there are many others, including Kevin Anderson's newsletter, which

[00:21:30] Nils: I love his newsletter. Definitely.

[00:21:32] Sani: an incredible, incredible member of the community and gives so much back to the community through that newsletter and everything else.

But what was your, when you started? Let's go back to the first edition of the newsletter.

[00:21:44] Nils: Yeah.

[00:21:45] Sani: What were you hoping to achieve with that? Because I have sort of something like that with the podcast here as well. What were you hoping to achieve?

[00:21:54] Nils: So I think there's two things. One, um, people told me, um, If you don't have a newsletter or something like that, you should definitely start, um, because it adds a lot of value to the business. Um, that was one of the first thoughts so far to the business. I'm not sure if it has added that much value, but it has added a lot of value to, um, uh, presence in the community and being able to share my ideas.

Um, and through that, I think as part of building a brand, uh, and as part sharing information that I deem, um, valuable to others. I think it's a way for me to, to contribute to this community.

[00:22:37] Sani: That is spot, spot on and a hundred percent true. Let me tell you that. I mean, this is how we connected in the first place. You and I, through the newsletter and the content that we're both putting out there. And a bunch of other people that are doing the same thing. So, yeah, uh, to everyone who's listening, Go ahead and subscribe to your weekly dose of experimentation and experimentation Tuesday, right?

That's the, what's the difference?

[00:22:59] Nils: it's, it's, it's experimentation Tuesday. And initially your weekly dose of experimentation was just the, the subtitle, but I, um, with the, with the, well, little data at the time that I had available, I tested headlines. Um, or like subject lines and weekly experimentation, um, even though probably not statistically significant, but was, uh, ended up being the winner, so I just stick with it.

[00:23:24] Sani: It's a cool

[00:23:25] Nils: but yeah, yeah,

[00:23:26] Sani: It's a very cool name. So everyone go ahead and subscribe and Nils, I'm, I'm beyond thankful for you being on this podcast. This was such a pleasure to talk to you, to interview you about this and I'll see you in a few months, I guess. Right. There's, there's a few conferences happening throughout Europe.

So I'll

[00:23:42] Nils: so. Are you going to Birmingham?

[00:23:45] Sani: Uh, these are still pending. We'll see. Yeah.

[00:23:48] Nils: Ah, right. I remember. All right. We'll see each other at, at one

[00:23:52] Sani: growth mark growth marketing summit.

[00:23:54] Nils: Definitely will be there.

[00:23:55] Sani: Definitely. We'll be there. Okay. I'll see you soon. That is, that is

[00:23:58] Nils: forward to it, man.

[00:23:59] Sani: And yeah, me too. Me too. And to everyone listening, uh, I know you enjoyed this one and please consider waiting, sharing whatever you do with podcast and I will talk to you next week.