E021: Shane Rostad on CRO: The DOs and DON'Ts of Conversion Rate Optimization

[00:00:00] [SPEAKER_00]: Welcome to episode 21 of the Marketing Operators podcast. Today we are doing our second episode

[00:00:05] [SPEAKER_00]: on conversion rate optimization with a special guest, Shane Rostad. Shane is one of my favorite

[00:00:10] [SPEAKER_00]: most thoughtful folks in CRO. Been following him for a handful of years, got the chance

[00:00:15] [SPEAKER_00]: to meet about six months ago and yeah, we go we go deep on this one. So we're basically

[00:00:19] [SPEAKER_00]: talking generally about how to run tests properly to get confident and actionable

[00:00:23] [SPEAKER_00]: results. We're talking about philosophy behind CRO testing, how to correctly set

[00:00:27] [SPEAKER_00]: up and interpret results. We're talking about some process and operation stuff around testing

[00:00:32] [SPEAKER_00]: and yeah, this is a fun one. There's a lot of nuance to conversion rate optimization.

[00:00:37] [SPEAKER_00]: So we do a pretty good job of addressing that here. As always, thank you to our sponsors

[00:00:41] [SPEAKER_00]: Motion Rich Panel, Pression AI and House. And also as always, if you're enjoying

[00:00:45] [SPEAKER_00]: the Marketing Operators podcast, make sure you like, subscribe, share with your buddies.

[00:00:50] [SPEAKER_00]: We want to get this in the hands as many tacticians as possible and we appreciate

[00:00:54] [SPEAKER_00]: any help in doing that.

[00:01:04] [SPEAKER_01]: At Ridge, we are loyal users of motion app.com and why is that? It's because creative is the

[00:01:09] [SPEAKER_01]: new ad targeting four or five, six years ago you'd show up to meta, you'd tell them

[00:01:12] [SPEAKER_01]: exactly who you wanted to target and that is no longer the case. If you want to be

[00:01:16] [SPEAKER_01]: reaching new audiences, you need to be doing that through creative and a strategic

[00:01:19] [SPEAKER_01]: approach to it. So what we talk about on our team all the time is creative

[00:01:23] [SPEAKER_01]: diversity. What are the motivators, the reasons to believe the new content

[00:01:27] [SPEAKER_01]: pillars that we're using to reach new audiences and all of that gets catalogued

[00:01:30] [SPEAKER_01]: within motion so that we can both measure what's working within each of those

[00:01:34] [SPEAKER_01]: categories as well as how do we measure against one another. With all the talk of

[00:01:38] [SPEAKER_01]: creative strategy, Motion has become an industry leader in content that my team

[00:01:42] [SPEAKER_01]: learns from and your team can learn from too. So you can go to motionapp.com

[00:01:45] [SPEAKER_01]: and check out their webinars, subscribe to their newsletter. We've done

[00:01:48] [SPEAKER_01]: exclusive marketing operators podcast with their audience to be up to date

[00:01:53] [SPEAKER_01]: on cutting edge creative strategy techniques. So if you're ready to join

[00:01:57] [SPEAKER_01]: the best DTC and e-commerce brands using motion to win at meta, TikTok,

[00:02:01] [SPEAKER_01]: Snapchat, go to motionapp.com, tell them the marketing operator sent you

[00:02:05] [SPEAKER_01]: and get 50% off your first month.

[00:02:10] [SPEAKER_00]: How'd you guys, how'd you guys map before?

[00:02:12] [SPEAKER_03]: Just on I feel like just via Twitter DMs.

[00:02:15] [SPEAKER_03]: Yeah, let's go.

[00:02:18] [SPEAKER_01]: But I mentioned Shane, you were a huge help with we did the sections API

[00:02:22] [SPEAKER_01]: project. So you talked with James.

[00:02:23] [SPEAKER_01]: So we're big, big fans of you.

[00:02:26] [SPEAKER_03]: Awesome.

[00:02:27] [SPEAKER_01]: You did.

[00:02:28] [SPEAKER_03]: Did who's API project?

[00:02:30] [SPEAKER_03]: It was the sections API.

[00:02:32] [SPEAKER_03]: So it was like when I didn't intentionally do it.

[00:02:35] [SPEAKER_03]: But it's always like something I like kind of

[00:02:37] [SPEAKER_03]: point in a roast.

[00:02:39] [SPEAKER_03]: Well, it's like, you know, you just have an example of like, hey, when you

[00:02:42] [SPEAKER_03]: click on the like, if you're on a black wallet, you want to go to green,

[00:02:46] [SPEAKER_03]: it has to reload the entire page and that because they use separate

[00:02:49] [SPEAKER_03]: products and that's just like not how you have to do it in Shopify.

[00:02:52] [SPEAKER_03]: It's definitely way easier.

[00:02:54] [SPEAKER_03]: But they came out this new thing where you could just like instantly

[00:02:57] [SPEAKER_03]: re-render the entire page without like refreshing it.

[00:03:01] [SPEAKER_03]: And so I made a video about how like it looked there and then another

[00:03:05] [SPEAKER_03]: store where like how they had it set up.

[00:03:07] [SPEAKER_03]: And then, you know, I felt kind of like, so it's always like

[00:03:12] [SPEAKER_03]: you feel a little bad roasting.

[00:03:14] [SPEAKER_03]: But you know, it's like, it's not that's not the intention of it.

[00:03:17] [SPEAKER_03]: But then we talked about it and I talked to the dev team about

[00:03:20] [SPEAKER_03]: how to actually go about implementing those changes to make it work.

[00:03:25] [SPEAKER_03]: And I think you're doing it now, right?

[00:03:27] [SPEAKER_03]: You have it set up.

[00:03:27] [SPEAKER_03]: I know that last time we talked to James, they were like, there,

[00:03:29] [SPEAKER_03]: we ran into another issue because you run into all these issues

[00:03:32] [SPEAKER_03]: with like firing pixels and stuff like that, like becomes a massive headache.

[00:03:35] [SPEAKER_03]: But it seems like they figured it out.

[00:03:37] [SPEAKER_00]: Well, we got it all figured out.

[00:03:38] [SPEAKER_00]: Yeah.

[00:03:38] [SPEAKER_00]: We're going to have to add that conversation because if you go

[00:03:41] [SPEAKER_00]: to our pan page and you start clicking through all the different variants,

[00:03:45] [SPEAKER_00]: it's it loads a new page every time.

[00:03:47] [SPEAKER_00]: So which is actually selfish.

[00:03:49] [SPEAKER_00]: Yeah.

[00:03:49] [SPEAKER_03]: I how about I just I'll just forward you the entire thread of my

[00:03:53] [SPEAKER_03]: conversation with James and all the all the stuff and we can talk about it

[00:03:57] [SPEAKER_03]: because there was the back and forth on how to get certain stuff implemented.

[00:04:00] [SPEAKER_03]: But yeah, dude, it makes a huge, a huge, it's one of the we could

[00:04:04] [SPEAKER_03]: we could even talk about this during the pod because it's one of those things

[00:04:06] [SPEAKER_03]: that's like not really something you would test.

[00:04:09] [SPEAKER_03]: It's just kind of obvious that it's better.

[00:04:12] [SPEAKER_03]: Yeah.

[00:04:12] [SPEAKER_03]: Yeah.

[00:04:12] [SPEAKER_00]: Well, that was self selfishly.

[00:04:14] [SPEAKER_00]: I that's why I mean Cody, like Cody, I don't know when this was

[00:04:18] [SPEAKER_00]: like 12 episodes ago is 12 episodes when we started talking to an

[00:04:21] [SPEAKER_00]: after like we are about to roll out international.

[00:04:23] [SPEAKER_00]: This is purely me just wanting to pick your brain about how to do it.

[00:04:28] [SPEAKER_00]: Cause both Ridge and Hexclad now both have pretty robust international businesses.

[00:04:33] [SPEAKER_00]: And that's kind of how I felt about this one.

[00:04:34] [SPEAKER_00]: I'm like, I think we've gotten better at CRO in the last year and a half.

[00:04:38] [SPEAKER_00]: I still think there was a lot to be desired and just like best practices.

[00:04:42] [SPEAKER_00]: Web stuff.

[00:04:42] [SPEAKER_00]: So I'm like, all right, this is great.

[00:04:44] [SPEAKER_00]: I'm just going to, I'm just going to pick Shane's brain on the

[00:04:46] [SPEAKER_00]: podcast and try to figure it out and probably going to want to have like

[00:04:50] [SPEAKER_00]: hire him for an audit after the show.

[00:04:54] [SPEAKER_03]: Yeah, man.

[00:04:55] [SPEAKER_03]: Well, I mean, always happy to ask, you know, the fun thing is doing, trying

[00:04:59] [SPEAKER_03]: to do less and less consulting stuff now that's some working on.

[00:05:02] [SPEAKER_03]: We actually just placed a P.O. for product.

[00:05:05] [SPEAKER_03]: This probably would derail us for a while if we started talking about this.

[00:05:09] [SPEAKER_03]: But we just played a place to P.O.

[00:05:11] [SPEAKER_03]: For product that should arrive by October 15th, like a full initial inventory.

[00:05:18] [SPEAKER_00]: Well, that's a good segue a little bit into this kind of first topic

[00:05:22] [SPEAKER_00]: because you have been in the services game for a while, specifically

[00:05:27] [SPEAKER_00]: in conversion rate optimization.

[00:05:29] [SPEAKER_00]: You're now taking a swing at a brand that you're founding or co-founding, I think.

[00:05:34] [SPEAKER_00]: And this is, I believe the first, the first brand you're founding, right?

[00:05:39] [SPEAKER_00]: Is this your first swing at a brand?

[00:05:43] [SPEAKER_03]: Yeah.

[00:05:43] [SPEAKER_03]: Besides like when I was 18, like trying to sell, you know, all sorts, like sourcing

[00:05:49] [SPEAKER_03]: random and Alibaba and all of that, you know, like the first in-roads

[00:05:53] [SPEAKER_03]: to e-commerce besides like fidget spinners and other random product.

[00:05:57] [SPEAKER_03]: That's not the wrong way to put it.

[00:06:00] [SPEAKER_03]: There's a mess.

[00:06:02] [SPEAKER_03]: That's such a funny.

[00:06:03] [SPEAKER_03]: I got we got burned pretty hard on fidget spinners.

[00:06:06] [SPEAKER_03]: No, it was double for a crush on that one.

[00:06:10] [SPEAKER_03]: Really? Yeah.

[00:06:11] [SPEAKER_03]: Well, we ordered them and we were early.

[00:06:14] [SPEAKER_03]: Yeah, we were early and a big bundle of them, a big huge box of them arrive

[00:06:18] [SPEAKER_03]: and we dropped one of them and it instantly just shattered into pieces

[00:06:22] [SPEAKER_03]: because they didn't want to tell us before shipping them that they were like,

[00:06:24] [SPEAKER_03]: we changed the dimensions a little bit so they were so tight

[00:06:27] [SPEAKER_03]: that like if you tapped it, it would just shatter basically.

[00:06:30] [SPEAKER_03]: And so we got all of them.

[00:06:31] [SPEAKER_03]: We're like, OK, by the time we could send them back

[00:06:33] [SPEAKER_03]: and we're more the whole craze was already like in full swing.

[00:06:36] [SPEAKER_03]: We're like, all right, I guess on to the next one.

[00:06:39] [SPEAKER_03]: Yeah, that was my first foray into dealing with stuff.

[00:06:42] [SPEAKER_03]: But yeah, that's awesome.

[00:06:43] [SPEAKER_00]: I think Nick, I think Nick was involved in fidget spinners in some way.

[00:06:47] [SPEAKER_00]: I don't know if he like was just buying ads for them

[00:06:50] [SPEAKER_00]: or like had some ownership of like the because like there is the fidget spinner brand.

[00:06:55] [SPEAKER_00]: Right? I know they're all called fidget spinners, but there's a brand

[00:06:57] [SPEAKER_00]: that was like it's like Kleenex fidget spinner.

[00:07:00] [SPEAKER_03]: Right. The original. Yeah.

[00:07:01] [SPEAKER_03]: Yeah, there was an original and then people like me just were like,

[00:07:05] [SPEAKER_03]: hey, these are super easy to make and they cost like 50 cents or something.

[00:07:09] [SPEAKER_03]: I love it.

[00:07:10] [SPEAKER_03]: It's the best. I'm spending this thing around all day long.

[00:07:13] [SPEAKER_03]: Yeah, I'm a big fidget.

[00:07:14] [SPEAKER_03]: Now I have my I have my wedding.

[00:07:16] [SPEAKER_03]: I'm actually not wearing a wedding ring now.

[00:07:18] [SPEAKER_03]: So got married recently, which is awesome.

[00:07:20] [SPEAKER_03]: But so I just fidget with that all day.

[00:07:22] [SPEAKER_03]: Like I'm always flipping it around and stuff.

[00:07:24] [SPEAKER_03]: It's basically like expensive.

[00:07:25] [SPEAKER_03]: Sounds like you're in market for a for a ridge ring at some point.

[00:07:28] [SPEAKER_03]: Yeah, I think I.

[00:07:30] [SPEAKER_03]: Yeah, I think we purchased the ring before a ridge even launched range.

[00:07:33] [SPEAKER_03]: So maybe you'll have to double back and get that lifetime guarantee.

[00:07:37] [SPEAKER_01]: Or need it resized.

[00:07:39] [SPEAKER_01]: You go to Ridge.com. Perfect.

[00:07:41] [SPEAKER_00]: What's so so I'm yeah, we don't we don't usually spend too much time

[00:07:45] [SPEAKER_00]: like going into like the bio stuff, but at least when I yeah,

[00:07:50] [SPEAKER_00]: hear a little bit like like who are you?

[00:07:52] [SPEAKER_00]: How'd you get into growth marketing?

[00:07:54] [SPEAKER_00]: What's your kind of trajectory been?

[00:07:56] [SPEAKER_00]: What's like the three to five minute timeline?

[00:07:59] [SPEAKER_03]: Yeah, yeah, I'll try to keep it like compact

[00:08:02] [SPEAKER_03]: because there's plenty of detail to go into.

[00:08:04] [SPEAKER_03]: But basically I started working in marketing

[00:08:06] [SPEAKER_03]: just like out of college as an intern.

[00:08:08] [SPEAKER_03]: I went to school for like entrepreneurship and finance

[00:08:11] [SPEAKER_03]: and the whole entrepreneurship side of things.

[00:08:12] [SPEAKER_03]: Just an absolute joke.

[00:08:13] [SPEAKER_03]: It was basically just like a general business degree.

[00:08:17] [SPEAKER_03]: And but yeah, what started I was working for a company doing,

[00:08:21] [SPEAKER_03]: you know, just basic marketing.

[00:08:23] [SPEAKER_03]: They had no marketing department.

[00:08:24] [SPEAKER_03]: I was the only person like, hey, figure out my brand.

[00:08:26] [SPEAKER_03]: They were building.

[00:08:28] [SPEAKER_03]: No, it was actually they were building apps for big brands

[00:08:32] [SPEAKER_03]: using like using Salesforce.

[00:08:35] [SPEAKER_03]: So it was super B2B focused and I had no clue what I was doing.

[00:08:38] [SPEAKER_03]: So I just basically tried to learn absolutely everything I could.

[00:08:41] [SPEAKER_03]: That led me into like content and SEO

[00:08:43] [SPEAKER_03]: because that was kind of like the craze at that time.

[00:08:46] [SPEAKER_03]: And also for like selling B2B services

[00:08:49] [SPEAKER_03]: was definitely like the thing to be doing.

[00:08:51] [SPEAKER_03]: But tried learning everything I could

[00:08:53] [SPEAKER_03]: about all aspects of marketing,

[00:08:55] [SPEAKER_03]: copyrighting, content, SEO, etc.

[00:08:57] [SPEAKER_03]: And but we were building apps and I realized,

[00:09:00] [SPEAKER_03]: oh, design is kind of cool.

[00:09:02] [SPEAKER_03]: And I met some people who are working there

[00:09:03] [SPEAKER_03]: that were doing design and development

[00:09:04] [SPEAKER_03]: actually building the apps.

[00:09:06] [SPEAKER_03]: And I always had this like weird, like this

[00:09:07] [SPEAKER_03]: not weird nerdy like techie side where I was interested in that.

[00:09:11] [SPEAKER_03]: So I was making these landing pages

[00:09:12] [SPEAKER_03]: for partnership things that we were doing.

[00:09:14] [SPEAKER_03]: And I was like having a great time doing it.

[00:09:16] [SPEAKER_03]: So I just started doing a little bit more design stuff

[00:09:18] [SPEAKER_03]: and decided, hey, I'm just going to take a course

[00:09:21] [SPEAKER_03]: on like how to do UI design, essentially.

[00:09:24] [SPEAKER_03]: So I'm having fun building these landing pages

[00:09:25] [SPEAKER_03]: and like actually getting a little bit more into it.

[00:09:27] [SPEAKER_00]: You were actually building funnels for the apps

[00:09:31] [SPEAKER_00]: targeting these like businesses that were in your your ideal profile.

[00:09:36] [SPEAKER_03]: I was I was building

[00:09:38] [SPEAKER_03]: I was basically building funnels for

[00:09:42] [SPEAKER_03]: the like enterprise companies, essentially.

[00:09:44] [SPEAKER_03]: So I started it like trying to get an enterprise

[00:09:46] [SPEAKER_03]: to sign up to build an application for their thing on using it.

[00:09:52] [SPEAKER_03]: It was very convoluted, essentially.

[00:09:53] [SPEAKER_03]: And so essentially what we ended up doing

[00:09:56] [SPEAKER_03]: though is like after I was making all these landing pages

[00:09:58] [SPEAKER_03]: because we were doing partnership events, etc.

[00:10:00] [SPEAKER_03]: So like all the time we had different stuff going on

[00:10:02] [SPEAKER_03]: to try to generate new business, essentially.

[00:10:07] [SPEAKER_03]: And I just got super into design, realized

[00:10:10] [SPEAKER_03]: I like to I've been working like for myself for the past five years.

[00:10:14] [SPEAKER_03]: So I'm very like I like to do everything,

[00:10:16] [SPEAKER_03]: which is almost to a fault.

[00:10:18] [SPEAKER_03]: And so I was like, well, if I can design it,

[00:10:19] [SPEAKER_03]: but now I have this page in Figma at the time

[00:10:22] [SPEAKER_03]: it was like in Sketch and like what the hell am I going to do with this?

[00:10:25] [SPEAKER_03]: Like I can't actually like I can use a landing page builder or whatever.

[00:10:29] [SPEAKER_03]: But like I want to actually make the thing.

[00:10:30] [SPEAKER_03]: So I ended up leaving that job to learning how to to learn how to code.

[00:10:35] [SPEAKER_03]: Went through that process.

[00:10:37] [SPEAKER_03]: So I was like, I want to be able to design and build stuff.

[00:10:40] [SPEAKER_03]: Basically ran out of money

[00:10:41] [SPEAKER_03]: and was like, I need to start freelancing.

[00:10:43] [SPEAKER_03]: And the one thing when trying to get marketing clients

[00:10:45] [SPEAKER_03]: at the time, that's what my like most of my experience was in.

[00:10:49] [SPEAKER_03]: I could get people that were like, yeah, I need to do website.

[00:10:51] [SPEAKER_03]: And so I just started like basically fell into doing a ton of web design

[00:10:55] [SPEAKER_03]: and development. But I always like I think the thing that sets me up

[00:10:58] [SPEAKER_03]: that set me up really well to do CRO work is that

[00:11:02] [SPEAKER_03]: I always looked at design and development from like a business perspective.

[00:11:05] [SPEAKER_03]: Like I don't come from like the art school background of design of like,

[00:11:09] [SPEAKER_03]: oh, it's designed to make it look pretty.

[00:11:10] [SPEAKER_03]: Like you get one of some of these like not to like call out names

[00:11:13] [SPEAKER_03]: like the red antler style of designs that are like super over the top

[00:11:17] [SPEAKER_03]: and look amazing, but like aren't easy to use.

[00:11:21] [SPEAKER_03]: And it's like, OK, how do I even buy a product on this website?

[00:11:23] [SPEAKER_03]: Like there's so many animations.

[00:11:25] [SPEAKER_03]: I come from the like, hey, let's ignore all of that.

[00:11:27] [SPEAKER_03]: It might not be as like aesthetically pleasing,

[00:11:31] [SPEAKER_03]: but I like to get results essentially.

[00:11:33] [SPEAKER_03]: And so when doing web design, I kind of like kept that in the back.

[00:11:37] [SPEAKER_03]: We all have those friends by the way, right?

[00:11:38] [SPEAKER_00]: All those friends that are where their business is more of an art project

[00:11:42] [SPEAKER_00]: than it is like a business.

[00:11:44] [SPEAKER_00]: And I have some of these buddies that I'll be texting about their businesses

[00:11:47] [SPEAKER_00]: and like, oh, like this is great that this happened.

[00:11:50] [SPEAKER_00]: We should do ABC and D and there's like, nah.

[00:11:52] [SPEAKER_00]: I'm like, all right.

[00:11:53] [SPEAKER_00]: All right. And like I'm done with you.

[00:11:56] [SPEAKER_00]: Yeah. Go make your brand.

[00:11:58] [SPEAKER_00]: Yeah. And I look for your brand that hasn't done a dollar in revenue.

[00:12:01] [SPEAKER_00]: Like that's fine. Yeah, exactly.

[00:12:03] [SPEAKER_03]: Yeah. And I was always like, I care about making money at the end of the day

[00:12:08] [SPEAKER_03]: and like design is fun and interesting.

[00:12:10] [SPEAKER_03]: And I think it has like such a powerful impact on the business.

[00:12:13] [SPEAKER_03]: Like at the end of the day, like websites are just like they are our storefronts, you know?

[00:12:18] [SPEAKER_03]: And so I kind of was like, hey, this is like your end at the beginning.

[00:12:21] [SPEAKER_03]: It was like local businesses then getting into like more national businesses.

[00:12:24] [SPEAKER_03]: It's like this is where people are finding you

[00:12:26] [SPEAKER_03]: and trying to actually solicit your services or buy your product.

[00:12:29] [SPEAKER_03]: It's like clearly this should be done well.

[00:12:33] [SPEAKER_03]: Like, or if it's being done poorly and you're succeeding,

[00:12:35] [SPEAKER_03]: then like probably would be doing better if you just had,

[00:12:38] [SPEAKER_03]: if you were put some focus and attention to it.

[00:12:41] [SPEAKER_03]: So yeah, basically from there, I was like, where are my services most valuable?

[00:12:45] [SPEAKER_03]: And it was either SaaS or how many years in?

[00:12:48] [SPEAKER_03]: Are you at this point?

[00:12:49] [SPEAKER_00]: Like from the moment you started at that agency to the moment,

[00:12:51] [SPEAKER_00]: now you're like in this kind of I'm learning how to develop in code stage.

[00:12:57] [SPEAKER_03]: Probably like, well, I was only two years into working at a year and a half,

[00:13:01] [SPEAKER_03]: two years into working at the agency when I left to just do my own thing

[00:13:04] [SPEAKER_03]: and kind of dive deep into this.

[00:13:06] [SPEAKER_03]: And then probably doing that for a few years before being like, OK,

[00:13:11] [SPEAKER_03]: like I'd worked with a couple of you confines, but I was like, where?

[00:13:14] [SPEAKER_03]: Who actually gets value out of the service that I'm providing?

[00:13:16] [SPEAKER_03]: And if you're selling to be to be enterprises,

[00:13:19] [SPEAKER_03]: like the guy making the sales calls is the one that's actually generating

[00:13:22] [SPEAKER_03]: business, not your website at the end of the day.

[00:13:24] [SPEAKER_03]: So like that's where I was like, OK, I have previous experience in e-com.

[00:13:28] [SPEAKER_03]: It's what I'm interested in.

[00:13:29] [SPEAKER_03]: And then I just kind of went all in on that and started specifically being

[00:13:33] [SPEAKER_03]: like, how do I instead of just designing and building websites,

[00:13:36] [SPEAKER_03]: how do I?

[00:13:38] [SPEAKER_03]: You know, run experiments.

[00:13:41] [SPEAKER_03]: Like how do I run an experimentation program because I feel like any

[00:13:44] [SPEAKER_03]: like I could redesign a website, but like you don't know,

[00:13:48] [SPEAKER_03]: like you can only do that once essentially.

[00:13:50] [SPEAKER_03]: I mean, you could do it multiple times, but like, OK, you redesign the website.

[00:13:52] [SPEAKER_03]: Now what? Like, is it better or worse?

[00:13:55] [SPEAKER_03]: Are you going to redesign the whole thing again?

[00:13:56] [SPEAKER_03]: I really wanted to like, I was really interested in the experimentation,

[00:14:00] [SPEAKER_03]: the test obviously I'm super results oriented.

[00:14:03] [SPEAKER_03]: So it drew me in.

[00:14:04] [SPEAKER_03]: And yeah, so that's how I got.

[00:14:06] [SPEAKER_03]: That's like my background that I think sets me up uniquely to do CRO work

[00:14:10] [SPEAKER_03]: is like I have this marketing background.

[00:14:11] [SPEAKER_03]: I understand UX design and development basically like that's kind of where

[00:14:16] [SPEAKER_03]: I think most of my expertise is on that UX side, but I also like

[00:14:19] [SPEAKER_03]: have the background in copywriting, etc., etc.

[00:14:22] [SPEAKER_03]: So it's like a weird mix of skills that come together

[00:14:24] [SPEAKER_03]: to be someone who can provide a full service like CRO offering, essentially.

[00:14:29] [SPEAKER_00]: Yeah, that makes sense.

[00:14:30] [SPEAKER_00]: And did you did you always have it in the back of your mind

[00:14:32] [SPEAKER_00]: that you were going to try to launch like a big swing at a brand

[00:14:36] [SPEAKER_00]: that you owned and were operating?

[00:14:38] [SPEAKER_03]: Yeah, I mean, I think my for a while, my I guess my.

[00:14:42] [SPEAKER_03]: Deces on how I should go about things was like, hey,

[00:14:45] [SPEAKER_03]: I can get really, really good at something and command like

[00:14:49] [SPEAKER_03]: a decent hourly rate and kind of set myself up where I could always

[00:14:53] [SPEAKER_03]: be employable and then I can.

[00:14:58] [SPEAKER_03]: Basically just take shots on goal and know that I always have like a solid

[00:15:02] [SPEAKER_03]: income, like essentially I was because when I stopped,

[00:15:04] [SPEAKER_03]: when I left that first shop, I was like I ended up I was

[00:15:07] [SPEAKER_03]: and I started freelancing like I was broke.

[00:15:09] [SPEAKER_03]: Like I was like there's nobody I'm just call.

[00:15:12] [SPEAKER_03]: I'm I'm doing all the outreach I can try to get people to like

[00:15:15] [SPEAKER_03]: I had never done my own thing before I was still pretty young.

[00:15:18] [SPEAKER_03]: And I was like, yeah, let's like solidify a base.

[00:15:21] [SPEAKER_03]: So let's get really good at something so that like I'm always

[00:15:24] [SPEAKER_03]: employable at like a decently high level.

[00:15:27] [SPEAKER_03]: And then once I have that flexibility, I can that kind of gives me the

[00:15:31] [SPEAKER_03]: you know, ability to just take shots on goal with bigger projects.

[00:15:35] [SPEAKER_03]: And this one just kind of came up recently.

[00:15:39] [SPEAKER_03]: I was like, yeah, why not?

[00:15:40] [SPEAKER_03]: Like, let's go.

[00:15:40] [SPEAKER_00]: Yeah, I don't think that's that model that like services,

[00:15:44] [SPEAKER_00]: which is very cash flow positive, very high margin, especially when

[00:15:48] [SPEAKER_00]: you're doing it at like a, like an individual freelance scale that

[00:15:51] [SPEAKER_00]: model of doing services to generate cash.

[00:15:54] [SPEAKER_00]: And then yeah, taking swings at brands that aren't going to probably

[00:15:58] [SPEAKER_00]: generate you or that should be a drain on cash upfront and resources,

[00:16:02] [SPEAKER_00]: but eventually could turn into something really big, especially if

[00:16:06] [SPEAKER_00]: you don't want to scale that agency model, which is I don't personally

[00:16:11] [SPEAKER_00]: like I would never be interested in launching and scaling an agency the

[00:16:14] [SPEAKER_00]: way that like Zach Stuck's done with Homestead or Taylor holidays done

[00:16:18] [SPEAKER_00]: with Common Thread like hats off to those guys.

[00:16:20] [SPEAKER_00]: That's just not a business model.

[00:16:21] [SPEAKER_00]: I would be very interested in scaling out.

[00:16:23] [SPEAKER_00]: But yeah, I think it seems like that was Zach Stuck's model all along.

[00:16:27] [SPEAKER_00]: I mean, from the moment I met him, he's like, yeah, we're going to

[00:16:29] [SPEAKER_00]: we're going to do the agency thing.

[00:16:30] [SPEAKER_00]: It's going to be a good cash flow business.

[00:16:32] [SPEAKER_00]: And then we're I want to I want to take some swings at some brand.

[00:16:34] [SPEAKER_00]: So cool, man.

[00:16:36] [SPEAKER_00]: That's that's a cool story.

[00:16:37] [SPEAKER_00]: I'm excited to see and stay in touch with you on the on

[00:16:40] [SPEAKER_00]: the how the brand stuff goes.

[00:16:41] [SPEAKER_00]: Congrats on that.

[00:16:42] [SPEAKER_00]: Yeah, thanks, dude.

[00:16:45] [SPEAKER_01]: So what are the most successful brands have in common?

[00:16:47] [SPEAKER_01]: They have great profit margins.

[00:16:49] [SPEAKER_01]: Their lean businesses are actually making money.

[00:16:51] [SPEAKER_01]: This is something that the DTC industry didn't do for a really long time.

[00:16:55] [SPEAKER_01]: And number two, they have a brand that consumers love.

[00:16:58] [SPEAKER_01]: Now, what if I told you that you could achieve both using AI?

[00:17:01] [SPEAKER_01]: The team at Rich panel has developed an AI that can significantly

[00:17:03] [SPEAKER_01]: reduce your support staff while providing memorable customer service

[00:17:06] [SPEAKER_01]: so you can reduce costs while ensuring that consumers continue to love your brand.

[00:17:11] [SPEAKER_01]: So here's how it works.

[00:17:12] [SPEAKER_01]: The AI imports data from your existing help desk and uses this information

[00:17:15] [SPEAKER_01]: to train itself.

[00:17:16] [SPEAKER_01]: This entire process takes less than a day.

[00:17:18] [SPEAKER_01]: Afterwards, it begins to automate numerous tasks, thereby reducing customer response times.

[00:17:23] [SPEAKER_01]: We just switched to Rich panel at Ridge.

[00:17:26] [SPEAKER_01]: We had been on customer.

[00:17:27] [SPEAKER_01]: I didn't feel we had a modern stack for our customer experience team.

[00:17:32] [SPEAKER_01]: What we're looking forward to doing is a number of things I just mentioned.

[00:17:34] [SPEAKER_01]: We're going to reduce headcount.

[00:17:35] [SPEAKER_01]: We're going to offer a better customer experience.

[00:17:37] [SPEAKER_01]: We're going to make information more readily available to the consumers

[00:17:40] [SPEAKER_01]: that are looking for it, and we think Rich panel is a great tool to do that.

[00:17:43] [SPEAKER_01]: So the best aspect of the transition was its simplicity.

[00:17:45] [SPEAKER_01]: We're ready to go live in two weeks with no data loss.

[00:17:48] [SPEAKER_01]: Our team of 30 agents were quickly on boarded and thanks to Rich panel's

[00:17:51] [SPEAKER_01]: tutorial videos and user friendly interface.

[00:17:53] [SPEAKER_01]: It was all very seamless.

[00:17:55] [SPEAKER_01]: If you're looking to leverage AI to save on support, payroll

[00:17:57] [SPEAKER_01]: and deliver a customer service experience that your customers will truly appreciate.

[00:18:01] [SPEAKER_01]: Check out rich panel dot com and schedule a demo with their team.

[00:18:06] [SPEAKER_00]: Let's get into CRO.

[00:18:07] [SPEAKER_00]: So I figured I think we're going to probably get into a lot of technical

[00:18:10] [SPEAKER_00]: stuff here, which is great.

[00:18:11] [SPEAKER_00]: I think our audience really likes that before we do that.

[00:18:15] [SPEAKER_00]: And this is going to be like come off as an obvious question, but philosophically,

[00:18:20] [SPEAKER_00]: like, well, like why?

[00:18:22] [SPEAKER_00]: Like why, why CRO, like why even test stuff on your website?

[00:18:27] [SPEAKER_00]: Like outside of just like, well, to improve conversion rate and to improve

[00:18:31] [SPEAKER_00]: AOV and to improve revenue per site visitor.

[00:18:35] [SPEAKER_00]: Like what is the philosophical why behind CRO testing and like,

[00:18:40] [SPEAKER_00]: is there different lanes?

[00:18:44] [SPEAKER_00]: Is it all one big lane?

[00:18:46] [SPEAKER_00]: Like what are your thoughts on that?

[00:18:48] [SPEAKER_03]: So I think I actually have like the a different opinion on this than most people

[00:18:54] [SPEAKER_03]: is that and we'll get into some of the more like fun hot takes before.

[00:18:57] [SPEAKER_03]: But one of the things that I mentioned was like, I don't think that

[00:19:00] [SPEAKER_03]: profit should be the ultimate goal of your CRO program.

[00:19:03] [SPEAKER_03]: Like when it comes to running tests, I think fundamentally, like at the

[00:19:07] [SPEAKER_03]: baseline, what you're hoping to do is just get better every day.

[00:19:12] [SPEAKER_03]: And it's not every day you're not running a test every day, but like

[00:19:14] [SPEAKER_03]: you just want to be improving week over week over week.

[00:19:17] [SPEAKER_03]: Like that's why you continue iterating, continue trying to improve.

[00:19:20] [SPEAKER_03]: You want your store like, like I mentioned, your store is your business.

[00:19:24] [SPEAKER_03]: Like of course people are going on the channel.

[00:19:26] [SPEAKER_03]: I think that's great.

[00:19:27] [SPEAKER_03]: But for a lot of brands, 90% of sales are coming through their

[00:19:31] [SPEAKER_03]: their Shopify store or whatever provider they're using,

[00:19:35] [SPEAKER_03]: whatever platform they're using.

[00:19:37] [SPEAKER_03]: But I just think it makes sense to continually improve your store

[00:19:41] [SPEAKER_03]: every like on a monthly basis, like know that you're just getting better

[00:19:45] [SPEAKER_03]: and leveling up every month.

[00:19:47] [SPEAKER_03]: And I think at a fundamental level, like that should be one of your

[00:19:51] [SPEAKER_03]: priorities because, you know, you don't know what you're kind of like.

[00:19:56] [SPEAKER_03]: Of course the whole goal at the end of the day, like we're in business

[00:19:58] [SPEAKER_03]: to make a profit.

[00:19:59] [SPEAKER_03]: So it's kind of like silly to say profit shouldn't be your goal.

[00:20:02] [SPEAKER_03]: Like at really like the only reason you're in business is to make a

[00:20:04] [SPEAKER_03]: profit. But when it comes to A.B.

[00:20:06] [SPEAKER_03]: testing in particular, I think that a profit motive kind of skews people's

[00:20:11] [SPEAKER_03]: behavior in a certain way.

[00:20:14] [SPEAKER_03]: And we can get to that.

[00:20:15] [SPEAKER_03]: But like I guess generally speaking, I think you should run tests

[00:20:19] [SPEAKER_03]: because it's a way to consistently and reliably improve your store month over

[00:20:23] [SPEAKER_03]: month.

[00:20:24] [SPEAKER_03]: And I think that's a good thing to do for like the healthier business.

[00:20:27] [SPEAKER_00]: And when you say that like profit motive can skew people's motivations,

[00:20:32] [SPEAKER_00]: are you saying that because not in your opinion, not all like on-site

[00:20:37] [SPEAKER_00]: zero test should have that final KPI of like conversion rate or revenue per

[00:20:41] [SPEAKER_00]: user? Are you saying that there's a lot more to be improved on and like

[00:20:47] [SPEAKER_00]: there's different metrics depending on the test that you're trying.

[00:20:49] [SPEAKER_00]: Like if you're trying to increase the amount of people that go to a certain

[00:20:52] [SPEAKER_00]: page and consume some content that you think is incredibly important for

[00:20:55] [SPEAKER_00]: your brand story, like you might not necessarily expect that to turn into

[00:20:59] [SPEAKER_00]: revenue, but you think that's lifting up.

[00:21:01] [SPEAKER_00]: Like is that what you're referring to?

[00:21:02] [SPEAKER_00]: There's different.

[00:21:03] [SPEAKER_03]: I don't take it that far.

[00:21:06] [SPEAKER_03]: Like from my perspective, I think that conversion rate or revenue

[00:21:10] [SPEAKER_03]: procession, whichever one you choose to use, is a good indicator of

[00:21:15] [SPEAKER_03]: something being better.

[00:21:17] [SPEAKER_03]: Like fundamentally we want people to come to the store and feel like

[00:21:20] [SPEAKER_03]: they've been properly educated enough to make a purchase.

[00:21:24] [SPEAKER_03]: Like we've removed enough barriers in order for them to make a purchase.

[00:21:28] [SPEAKER_03]: That would be a good experience for them.

[00:21:30] [SPEAKER_03]: Like they're coming to your store for a reason and then finding something

[00:21:32] [SPEAKER_03]: that they actually like is ultimately and choosing to go ahead and purchase

[00:21:36] [SPEAKER_03]: that is ultimately a positive experience.

[00:21:40] [SPEAKER_03]: And so when I'm saying like we want to improve the store, it's

[00:21:43] [SPEAKER_03]: like conversion rate is just a marker because we'll get into this later

[00:21:48] [SPEAKER_03]: too on how to actually interpret test results because I genuinely think

[00:21:51] [SPEAKER_03]: like everybody is doing it wrong and I have occasionally too.

[00:21:54] [SPEAKER_03]: I think most people are just totally misinterpreting what their

[00:21:58] [SPEAKER_03]: testing tool is actually telling them.

[00:22:00] [SPEAKER_03]: But if you have a positive effect on conversion rate, that's better.

[00:22:07] [SPEAKER_03]: So okay, that's better.

[00:22:08] [SPEAKER_03]: Now let's continue like let's continue down that path and just be like

[00:22:12] [SPEAKER_03]: that's a win.

[00:22:13] [SPEAKER_03]: Let's book that as a win, but not just like let's look for us.

[00:22:16] [SPEAKER_03]: Let's try to push for 60% improvements in conversion rates.

[00:22:19] [SPEAKER_03]: 50%.

[00:22:20] [SPEAKER_03]: Like I think those numbers don't really even exist in a brand

[00:22:23] [SPEAKER_03]: that's at scale, like that already has product market fit.

[00:22:26] [SPEAKER_03]: You can if you lower your prices.

[00:22:28] [SPEAKER_03]: And I think Connor has talked about that.

[00:22:30] [SPEAKER_03]: Like there are ways that you can achieve that for different offers

[00:22:33] [SPEAKER_03]: and things like that, but I just think generally if you're always

[00:22:37] [SPEAKER_03]: thinking how do we maximize our dollars in using our website?

[00:22:43] [SPEAKER_03]: I think it leads you to trying to make short term decisions

[00:22:47] [SPEAKER_03]: because in a two week test, you might see that adding that

[00:22:49] [SPEAKER_03]: little four people are looking at this to get it now before it

[00:22:53] [SPEAKER_03]: runs out.

[00:22:53] [SPEAKER_03]: You might actually see, wow, that increases our conversion rate

[00:22:56] [SPEAKER_03]: by 20%, but like the long term damage that it does to your

[00:23:00] [SPEAKER_03]: brand's reputation is not being factored into that two week

[00:23:03] [SPEAKER_03]: thing.

[00:23:03] [SPEAKER_03]: So it's like by always shooting for the biggest results and

[00:23:06] [SPEAKER_03]: like there I guess the one the easiest way to visualize is

[00:23:09] [SPEAKER_03]: that there's this.

[00:23:10] [SPEAKER_03]: And I think Connor has also mentioned something like this

[00:23:12] [SPEAKER_03]: in the past too where there's essentially this.

[00:23:16] [SPEAKER_03]: Look, there's this global maxima essentially of like the best

[00:23:19] [SPEAKER_03]: your site will ever convert.

[00:23:21] [SPEAKER_03]: And as you get closer to that, like a qualified visitor that is

[00:23:26] [SPEAKER_03]: like in market, the best that your site will ever convert the

[00:23:29] [SPEAKER_03]: average qualified in market visitor is like there is a peak

[00:23:32] [SPEAKER_03]: to that.

[00:23:33] [SPEAKER_03]: You're like, you will essentially hit a peak and the closer

[00:23:36] [SPEAKER_03]: that you get to that peak, the harder it is to see bigger

[00:23:39] [SPEAKER_03]: gains because obviously you're getting closer to the maximum.

[00:23:42] [SPEAKER_03]: So there's not massive like Delta between you and there

[00:23:46] [SPEAKER_03]: when you're a new brand, you don't have product market

[00:23:47] [SPEAKER_03]: to fit and you hit on an ad or an angle or a concept or

[00:23:50] [SPEAKER_03]: something with a specific landing page.

[00:23:53] [SPEAKER_03]: It can drastically spike your conversion rate because you're

[00:23:55] [SPEAKER_03]: like way down here as far as the the maximum of the peak

[00:23:58] [SPEAKER_03]: goes, but as you start to scale that you just get into this

[00:24:01] [SPEAKER_03]: like, okay, we have to go more short term.

[00:24:03] [SPEAKER_03]: We have to go more aggressive.

[00:24:04] [SPEAKER_03]: We have to go more kind of like.

[00:24:07] [SPEAKER_03]: Influence psychology of persuasion stuff.

[00:24:09] [SPEAKER_03]: And I think it creates a weird like incentive in that way.

[00:24:14] [SPEAKER_00]: And that's like, that's I always say this about it's the same

[00:24:17] [SPEAKER_00]: with an ad account.

[00:24:18] [SPEAKER_00]: Uh, like when I walked into hex clad in 2021, they hadn't

[00:24:22] [SPEAKER_00]: launched new creative in three or three to four months and

[00:24:26] [SPEAKER_00]: I'm sitting there drooling because I'm like, well, we're

[00:24:28] [SPEAKER_00]: going to have a one in three hit rate probably in terms of

[00:24:31] [SPEAKER_00]: launching creative that has an incremental or marginally

[00:24:34] [SPEAKER_00]: better performance.

[00:24:36] [SPEAKER_00]: Uh, in terms of our like actual performance metrics now,

[00:24:40] [SPEAKER_00]: you know, two and a half years later, it's like, shit,

[00:24:42] [SPEAKER_00]: we might, we might find a winner every one in 10, two and

[00:24:45] [SPEAKER_00]: 10, three, you know that hit rate is that diminishing return.

[00:24:49] [SPEAKER_00]: Connor talks about this a lot with like local maximums on

[00:24:52] [SPEAKER_00]: certain ad creatives and like people often pigeonhole

[00:24:56] [SPEAKER_00]: themselves because they're like, oh, we found a winner and

[00:24:58] [SPEAKER_00]: now we're going to spend the next two months iterating on

[00:25:00] [SPEAKER_00]: it.

[00:25:00] [SPEAKER_00]: And in reality, you're like just inching towards this,

[00:25:03] [SPEAKER_00]: like you already got 90% of the way there with your,

[00:25:05] [SPEAKER_00]: with that initial launch.

[00:25:07] [SPEAKER_00]: Now you're just kind of inching towards 95%.

[00:25:09] [SPEAKER_00]: And instead you could have unlocked like a totally new 90%

[00:25:12] [SPEAKER_00]: that you're not even tapping into.

[00:25:14] [SPEAKER_00]: So it's, um, it's, yeah, it's interesting to see, you know,

[00:25:18] [SPEAKER_00]: we've talked a lot about that in the context of ad creative.

[00:25:20] [SPEAKER_00]: It's interesting to hear you talk about and that's not a

[00:25:22] [SPEAKER_00]: unique concept to ad creative.

[00:25:24] [SPEAKER_00]: I think that's, that's generally the case as you're

[00:25:26] [SPEAKER_00]: finding wins and incrementally, marginally improving

[00:25:29] [SPEAKER_00]: across all of your channels.

[00:25:32] [SPEAKER_01]: We last week, we talked about cheaper ads, do cheap

[00:25:36] [SPEAKER_01]: ads work or something like that?

[00:25:37] [SPEAKER_01]: And it was like, they do.

[00:25:39] [SPEAKER_01]: Like they can get cheaper.

[00:25:40] [SPEAKER_01]: It's exact same thing where like if you're just optimizing

[00:25:42] [SPEAKER_01]: for like short term best row as like you're going to,

[00:25:45] [SPEAKER_01]: you're going to opt for like big markdowns, aggressive

[00:25:48] [SPEAKER_01]: messaging, you're going to pull out all the like spammy

[00:25:51] [SPEAKER_01]: little stuff spin to win, which we all know it too.

[00:25:54] [SPEAKER_01]: We all know that.

[00:25:56] [SPEAKER_03]: Yeah.

[00:25:57] [SPEAKER_03]: We all know the little tricks like you're with

[00:25:59] [SPEAKER_03]: the spin to win.

[00:26:00] [SPEAKER_03]: Hey, you're running it right now, but you know, like

[00:26:02] [SPEAKER_03]: this is when you're trying to maximize on those signups

[00:26:04] [SPEAKER_03]: and like I think that you don't run spin to win

[00:26:07] [SPEAKER_03]: like year round, right?

[00:26:09] [SPEAKER_03]: We don't know.

[00:26:09] [SPEAKER_03]: Yeah.

[00:26:09] [SPEAKER_03]: So like that's the thing where you get into this, I think

[00:26:12] [SPEAKER_03]: with testing on your site in particular, there's a

[00:26:15] [SPEAKER_03]: there's so much damage that can be done by seeing

[00:26:20] [SPEAKER_03]: a positive test result on something like that, because

[00:26:22] [SPEAKER_03]: you think that removing it is going to have the now

[00:26:26] [SPEAKER_03]: bring your efficiency down, but like having it

[00:26:29] [SPEAKER_03]: there long term impacts your act like how people

[00:26:32] [SPEAKER_03]: perceive your brand and you get into this belief

[00:26:35] [SPEAKER_03]: system, like you have this belief about how your

[00:26:37] [SPEAKER_03]: website operates.

[00:26:38] [SPEAKER_03]: And that's every test kind of tells you like builds

[00:26:41] [SPEAKER_03]: upon your beliefs in a certain way, which is why

[00:26:43] [SPEAKER_03]: I think we'll talk about like how to run a test

[00:26:45] [SPEAKER_03]: properly, because if you're incorrectly interpreting

[00:26:47] [SPEAKER_03]: results and getting into incorrect beliefs, then

[00:26:50] [SPEAKER_03]: you just have all this like, Hey, I want to remove

[00:26:52] [SPEAKER_03]: this as someone's like no way.

[00:26:54] [SPEAKER_03]: That was a 20% improvement.

[00:26:56] [SPEAKER_03]: Like we can't remove that.

[00:26:57] [SPEAKER_03]: And you're like, I don't think it's good for

[00:26:59] [SPEAKER_03]: our brand.

[00:26:59] [SPEAKER_03]: And someone's like, well, we already ran the test

[00:27:02] [SPEAKER_03]: and are we going to use another like iteration

[00:27:03] [SPEAKER_03]: of an experiment or another set of resources to

[00:27:06] [SPEAKER_03]: retest that same thing that we already tested.

[00:27:08] [SPEAKER_03]: And it's like, there's a lot of like potholes that

[00:27:10] [SPEAKER_03]: you can find yourself falling into like that.

[00:27:13] [SPEAKER_03]: And then we all know the tricks to get conversion

[00:27:15] [SPEAKER_03]: rates and stuff up, but at what cost essentially?

[00:27:19] [SPEAKER_03]: Yeah.

[00:27:19] [SPEAKER_01]: So yeah, I'd like this point and then we can

[00:27:22] [SPEAKER_01]: get to some of the hot takes because I think

[00:27:23] [SPEAKER_01]: we can talk about this for a long time.

[00:27:25] [SPEAKER_01]: Yeah, because because I

[00:27:28] [SPEAKER_01]: I think well, I think I think two things.

[00:27:30] [SPEAKER_01]: One, I think subjectively applying constraints to

[00:27:33] [SPEAKER_01]: your business makes sense.

[00:27:35] [SPEAKER_01]: Like you can't like unbound optimized for profit

[00:27:37] [SPEAKER_01]: because you'll like you'll fall into dark patterns

[00:27:40] [SPEAKER_01]: and like all the like yeah, you'll pull out

[00:27:42] [SPEAKER_01]: all the tricks or whatever.

[00:27:43] [SPEAKER_01]: I also feel many brand owners apply those

[00:27:47] [SPEAKER_01]: constraints too aggressively where they're like,

[00:27:49] [SPEAKER_01]: Oh, spin like spin to win.

[00:27:51] [SPEAKER_01]: For instance, like people hate it.

[00:27:52] [SPEAKER_01]: And like there's a place for spin to win.

[00:27:54] [SPEAKER_01]: Yeah.

[00:27:54] [SPEAKER_01]: And if you don't believe that there's a place

[00:27:57] [SPEAKER_01]: of first bit like whatever, if you're Ramoa,

[00:27:59] [SPEAKER_01]: there's no place for spin to win.

[00:28:01] [SPEAKER_01]: OK, I'll believe that.

[00:28:02] [SPEAKER_01]: But like for most people, they're like applying

[00:28:05] [SPEAKER_01]: those constraints too early for some other reason.

[00:28:08] [SPEAKER_01]: It's like a form of like it's like ego or something.

[00:28:12] [SPEAKER_01]: Right?

[00:28:12] [SPEAKER_01]: People like to poo poo the spin to win, but it's like.

[00:28:16] [SPEAKER_03]: Yeah, you can you can make it.

[00:28:18] [SPEAKER_03]: I've experienced this with post purchase a lot.

[00:28:21] [SPEAKER_03]: Like I've talked to brands who feel like post purchase

[00:28:24] [SPEAKER_03]: like those hey, like one the one click aspect of it

[00:28:28] [SPEAKER_03]: where like they just add it to their order.

[00:28:30] [SPEAKER_03]: And then some people naturally accidentally click

[00:28:32] [SPEAKER_03]: that button somehow, but like people do that.

[00:28:35] [SPEAKER_03]: And then they get these customer support requests

[00:28:37] [SPEAKER_03]: and they're like, we're getting people who are upset.

[00:28:39] [SPEAKER_03]: They feel like we wronged them.

[00:28:41] [SPEAKER_03]: Like we should just remove it.

[00:28:42] [SPEAKER_03]: And it's like, OK, well, like it's to your point,

[00:28:45] [SPEAKER_03]: a lot of people will be like, let's not do that

[00:28:47] [SPEAKER_03]: because there is like some small downside,

[00:28:49] [SPEAKER_03]: but you have to weigh the massive upside

[00:28:52] [SPEAKER_03]: in that increased revenue that you're gaining.

[00:28:55] [SPEAKER_03]: And it's very hard to make.

[00:28:57] [SPEAKER_03]: I think you just have to make those decisions wisely.

[00:29:00] [SPEAKER_03]: And that's where pushing for.

[00:29:04] [SPEAKER_03]: It's it pushing too hard for like profit optimization

[00:29:06] [SPEAKER_03]: is I think where you get this.

[00:29:08] [SPEAKER_03]: That's why I'm not like, oh, yeah, let's optimize

[00:29:09] [SPEAKER_03]: for like trying to get people to read a piece of content

[00:29:11] [SPEAKER_03]: like narrow we want sales like at the end of the day.

[00:29:14] [SPEAKER_03]: Like I want I want to see conversion rate going up.

[00:29:16] [SPEAKER_03]: But at the same time, if all you're looking for is like

[00:29:20] [SPEAKER_03]: as a service provider, if you're being measured

[00:29:22] [SPEAKER_03]: on how many winning tests or how many tests

[00:29:25] [SPEAKER_03]: are over a 5 percent gaining conversion rate,

[00:29:26] [SPEAKER_03]: like you're going to start scraping the bottom of the barrel

[00:29:29] [SPEAKER_03]: in order to find those tests that hit that because hey,

[00:29:33] [SPEAKER_03]: we need one of those a month and you're like, OK, well,

[00:29:35] [SPEAKER_03]: I know 12 different tactics that I could implement

[00:29:37] [SPEAKER_03]: that like will scrape the bottom of the barrel

[00:29:40] [SPEAKER_03]: to get that increase.

[00:29:41] [SPEAKER_03]: But like that's not good for the.

[00:29:43] [SPEAKER_03]: So it's just that incentive alignment is kind of difficult.

[00:29:46] [SPEAKER_03]: Yeah. So that's where I think it gets interesting.

[00:29:49] [SPEAKER_00]: That graph, it's not as popular anymore.

[00:29:52] [SPEAKER_00]: It was it was really popular like three years ago

[00:29:54] [SPEAKER_00]: where it would usually have like three images

[00:29:57] [SPEAKER_00]: over three different timelines.

[00:29:58] [SPEAKER_00]: All right, L 14, like it's like sales over time.

[00:30:02] [SPEAKER_00]: It's like just like massive spikes.

[00:30:03] [SPEAKER_00]: Like what's going on?

[00:30:04] [SPEAKER_00]: Then you zoom out to three months.

[00:30:05] [SPEAKER_00]: It's like, all right, some some more spikes, but it's moved out.

[00:30:08] [SPEAKER_00]: And then it was like the people that were like just like

[00:30:10] [SPEAKER_00]: zoom out on the big picture.

[00:30:11] [SPEAKER_00]: Don't get you know what I mean?

[00:30:12] [SPEAKER_00]: It's like, all right, over time we've grown

[00:30:14] [SPEAKER_00]: and it's been positive.

[00:30:15] [SPEAKER_00]: The same thing applies here.

[00:30:17] [SPEAKER_00]: Like those cheap like do cheap tactics work?

[00:30:19] [SPEAKER_00]: Yes. If in the short term.

[00:30:21] [SPEAKER_00]: But if you zoom out and you say, OK, like let's go

[00:30:23] [SPEAKER_00]: from a month outlook to three years, it probably is going

[00:30:27] [SPEAKER_00]: to show actually a lower conversion rate at some point

[00:30:29] [SPEAKER_00]: in a lower revenue per user.

[00:30:30] [SPEAKER_00]: It's hard to like actually map that because you're never going

[00:30:33] [SPEAKER_00]: to run that test for that long or whatever.

[00:30:34] [SPEAKER_00]: But like that's ultimately what you're saying here, right?

[00:30:36] [SPEAKER_00]: Is like you need to be optimizing for for conversion

[00:30:39] [SPEAKER_00]: rate wins, revenue per site visitor wins that are going

[00:30:41] [SPEAKER_00]: to give you short term wins that also translate into,

[00:30:44] [SPEAKER_00]: you know, wins over the course of two years, three years,

[00:30:47] [SPEAKER_00]: four years, like OK, yeah, totally.

[00:30:49] [SPEAKER_00]: I want to dig into this note because this is a juicy

[00:30:51] [SPEAKER_00]: this is a juicy quote from the notes.

[00:30:53] [SPEAKER_00]: The note was 99% of people listening to this podcast.

[00:30:57] [SPEAKER_00]: Oh, damn, calling out the listener.

[00:30:58] [SPEAKER_00]: Shit are running the test wrong or at least interpreting

[00:31:01] [SPEAKER_00]: the test wrong.

[00:31:02] [SPEAKER_00]: I've seen firsthand people are considered top class growth

[00:31:04] [SPEAKER_00]: marketers in this space run a B tests that make absolutely

[00:31:07] [SPEAKER_00]: no sense and interpret results from them that have no basis

[00:31:10] [SPEAKER_00]: reality.

[00:31:11] [SPEAKER_00]: Let's just start calling people out like who's who's a fraud.

[00:31:14] [SPEAKER_00]: I'm not I'm just kidding.

[00:31:17] [SPEAKER_00]: Yeah, yeah, yeah.

[00:31:18] [SPEAKER_03]: All right. Well, here's the list actually.

[00:31:20] [SPEAKER_03]: Taylor holiday.

[00:31:22] [SPEAKER_03]: No, I'm just kidding.

[00:31:26] [SPEAKER_03]: If anybody understands statistics, I'm sure it's that guy.

[00:31:29] [SPEAKER_03]: So I think that, you know, really what it is that like

[00:31:34] [SPEAKER_03]: people statistics is really complicated.

[00:31:36] [SPEAKER_03]: I'm not a statistician.

[00:31:38] [SPEAKER_03]: You know what I mean?

[00:31:38] [SPEAKER_03]: Like this is that's not my profession.

[00:31:40] [SPEAKER_03]: That's not my expertise.

[00:31:41] [SPEAKER_03]: But like there are some baseline things that you

[00:31:43] [SPEAKER_03]: should probably understand when running tests that you

[00:31:46] [SPEAKER_03]: interpret things correctly.

[00:31:47] [SPEAKER_03]: But like there's this quote that I really like by Richard

[00:31:51] [SPEAKER_03]: Feynman. It's like the first the first principle is not to

[00:31:54] [SPEAKER_03]: fool yourself and you're the easiest person to fool.

[00:31:56] [SPEAKER_03]: And it kind of goes hand in hand with like this book.

[00:31:58] [SPEAKER_03]: I think on the latest operators podcast episode, Jason

[00:32:02] [SPEAKER_03]: mentioned like the book of how to lie with statistics.

[00:32:05] [SPEAKER_03]: I like that one.

[00:32:06] [SPEAKER_03]: Like if you have statistical metrics on a page,

[00:32:09] [SPEAKER_03]: it's so easy to lie using them.

[00:32:11] [SPEAKER_03]: Like it's so easy to tell a story about like, oh, yeah,

[00:32:14] [SPEAKER_03]: the confidence is only 73 percent.

[00:32:16] [SPEAKER_03]: But like, you know, we know this about that.

[00:32:19] [SPEAKER_03]: And all of a sudden it's like, well, now you're actually

[00:32:21] [SPEAKER_03]: talking yourself up to the confidence being like 95 percent or something.

[00:32:24] [SPEAKER_03]: Like I think people use these metrics to like in it can be used

[00:32:28] [SPEAKER_03]: in various ways, but I just want to maybe we should just get into like some

[00:32:32] [SPEAKER_03]: I'll try to keep these in short, but like we can get into some rules,

[00:32:37] [SPEAKER_03]: I think that people could follow where the reason why I bring this up

[00:32:41] [SPEAKER_03]: is like we mentioned this briefly earlier, but.

[00:32:44] [SPEAKER_03]: If you run a test incorrectly and you get a result

[00:32:48] [SPEAKER_03]: that tells you either big win or big loss, you're skewing your perspective

[00:32:52] [SPEAKER_03]: for the future and usually you're kind of you're you're codifying that as

[00:32:57] [SPEAKER_03]: like a rule of like, hey, we tried that already.

[00:32:59] [SPEAKER_03]: And because you've tried it already, you're less likely,

[00:33:02] [SPEAKER_03]: even if you're not totally against doing something again,

[00:33:04] [SPEAKER_03]: you're way less likely to try it again because you're like, hey,

[00:33:07] [SPEAKER_03]: we've already done that.

[00:33:07] [SPEAKER_03]: Like we have all these other any, but all of us probably have

[00:33:10] [SPEAKER_03]: 50 things that like you look at a website and you're like, hey,

[00:33:12] [SPEAKER_03]: let's try changing this.

[00:33:13] [SPEAKER_03]: Let's try changing that.

[00:33:14] [SPEAKER_03]: Like you can just look at it and be like, what if we tried that?

[00:33:16] [SPEAKER_03]: You can come up with a super long list.

[00:33:18] [SPEAKER_03]: And when you have that list and you're like, we've already tried this one.

[00:33:21] [SPEAKER_03]: You you're so much less likely to double back.

[00:33:23] [SPEAKER_03]: And anytime in the future, six months in the future,

[00:33:25] [SPEAKER_03]: someone looks at it and is like, oh, we've already tested that.

[00:33:28] [SPEAKER_03]: And this is the result, but they don't know that you ran

[00:33:30] [SPEAKER_03]: the test wrong and that the results are like running a test

[00:33:33] [SPEAKER_03]: incorrectly and getting results and interpreting them.

[00:33:35] [SPEAKER_03]: And you're not even it's like worse than just being wrong.

[00:33:41] [SPEAKER_03]: It's just like you're not actually getting anything out of it.

[00:33:44] [SPEAKER_03]: Like the results are totally random.

[00:33:46] [SPEAKER_03]: So you're 50, 50 chance that you're in one direction or the other.

[00:33:49] [SPEAKER_03]: You might be overly confident under a confident.

[00:33:52] [SPEAKER_03]: And so there's a lot of downside to running testing correctly

[00:33:55] [SPEAKER_03]: interpreting results, which is why I feel like if they're like people

[00:33:58] [SPEAKER_03]: listening to this, if they're running experiments or they're talking

[00:34:00] [SPEAKER_03]: thinking about working with an agency to run experiments,

[00:34:02] [SPEAKER_03]: having this baseline understanding of like, hey, let's set these

[00:34:05] [SPEAKER_03]: ground rules or at least asking them like, hey, why are we not doing this?

[00:34:08] [SPEAKER_03]: Because they should be able to speak through like very confidently

[00:34:11] [SPEAKER_03]: why they're not following these kind of procedures.

[00:34:14] [SPEAKER_03]: But yeah, I can just kind of go through them and we can.

[00:34:18] [SPEAKER_00]: Yeah, let's.

[00:34:19] [SPEAKER_00]: The rules and we can rest on them.

[00:34:21] [SPEAKER_00]: I like I think it's helpful to give people just like frameworks

[00:34:24] [SPEAKER_00]: to to operate off of.

[00:34:25] [SPEAKER_00]: So I think this would be really interesting.

[00:34:27] [SPEAKER_03]: Cool.

[00:34:27] [SPEAKER_03]: So the first rule, which is the toughest thing is that you need a

[00:34:32] [SPEAKER_03]: I would say a minimum test duration.

[00:34:34] [SPEAKER_03]: So a minimum number of days or weeks that a test is going to run

[00:34:38] [SPEAKER_03]: for and included in that is this notion of sample size.

[00:34:42] [SPEAKER_03]: Like I've had people send me test results that are like,

[00:34:45] [SPEAKER_03]: this person ran a test and there's four conversions on this one

[00:34:47] [SPEAKER_03]: and two on the other and eight on the other.

[00:34:50] [SPEAKER_03]: And they're showing a hundred and eighty percent increase.

[00:34:52] [SPEAKER_03]: And they're trying to say that this is statistically significant.

[00:34:54] [SPEAKER_03]: And it's like, yeah, well, like it's not.

[00:34:57] [SPEAKER_03]: That's obviously not like it's obviously not.

[00:34:59] [SPEAKER_03]: But like all they know is statistical significance

[00:35:02] [SPEAKER_03]: and they see the conversion rate increase.

[00:35:03] [SPEAKER_03]: But they don't know like the the likelihood is it's actually

[00:35:06] [SPEAKER_03]: maybe negative, like there's a high likelihood it's negative one hundred and eighty percent.

[00:35:10] [SPEAKER_03]: Is that right?

[00:35:10] [SPEAKER_00]: So like or it probably depends on like what the

[00:35:14] [SPEAKER_00]: what like the performance spread is between like the variants that you're testing.

[00:35:18] [SPEAKER_00]: But is there a number of like sample sizes is this at least

[00:35:23] [SPEAKER_00]: in the number of conversions is this at least?

[00:35:26] [SPEAKER_00]: Or is that not really how it works?

[00:35:28] [SPEAKER_00]: I'm not a statistician.

[00:35:28] [SPEAKER_03]: I think that it's it's not necessarily how it works.

[00:35:32] [SPEAKER_03]: But I think like a good rule of thumb that most people put out there

[00:35:36] [SPEAKER_03]: is that at minimum, you should be getting like

[00:35:42] [SPEAKER_03]: at absolute minimum, you shouldn't be running a test

[00:35:44] [SPEAKER_03]: if you can't get 300 conversions on both sides.

[00:35:48] [SPEAKER_03]: Using like this is using like Bayesian statistics, analysis, etc.

[00:35:51] [SPEAKER_03]: But like there's there's this concept from

[00:35:53] [SPEAKER_03]: there's two ways of kind of interpreting test results.

[00:35:56] [SPEAKER_03]: And we I won't get into statistics mainly because it's

[00:35:59] [SPEAKER_03]: it also gets over my head at a certain point.

[00:36:01] [SPEAKER_03]: But like there's this method called frequentist statistics.

[00:36:04] [SPEAKER_03]: It requires a lot more data.

[00:36:06] [SPEAKER_03]: Everybody's moved to this Bayesian approach that kind of says,

[00:36:10] [SPEAKER_03]: well, we have this data from beforehand so we can kind of like speed up the process.

[00:36:14] [SPEAKER_03]: Most people use that, which gives you like a probability of success.

[00:36:17] [SPEAKER_03]: But in this like frequentist model, which some people still use for testing,

[00:36:21] [SPEAKER_03]: there's actually a like.

[00:36:23] [SPEAKER_03]: A there's there's a metric called a confidence interval,

[00:36:26] [SPEAKER_03]: which is like you have your statistical significance, which is like 95 percent.

[00:36:29] [SPEAKER_03]: But you have this thing called a no, you have statistical power.

[00:36:33] [SPEAKER_03]: And so statistical power is this metric of like,

[00:36:36] [SPEAKER_03]: have we powered the test enough?

[00:36:38] [SPEAKER_03]: Like have we pushed enough people into the test to feel confident

[00:36:41] [SPEAKER_03]: that we're not getting false positives?

[00:36:43] [SPEAKER_03]: And so the the kind of going

[00:36:46] [SPEAKER_03]: like standard for that is 80 percent.

[00:36:49] [SPEAKER_03]: And there's like, you know, this is kind of complicated,

[00:36:51] [SPEAKER_03]: but there are tools online like conversion XL, like CXL.com.

[00:36:54] [SPEAKER_03]: They have a test calculator where you can kind of input

[00:36:57] [SPEAKER_03]: the visitors and the conversion rate, and it will tell you like an estimated

[00:37:01] [SPEAKER_03]: minimum test duration because the smaller the impact is like one to four percent.

[00:37:06] [SPEAKER_03]: You need a lot more data to feel confident in that anything more than that,

[00:37:10] [SPEAKER_03]: you can have less data to feel confident, like the bigger the impact,

[00:37:13] [SPEAKER_03]: the more confident you feel.

[00:37:15] [SPEAKER_03]: That's all to say that like that's not how the Bayesian statistics analysis

[00:37:19] [SPEAKER_03]: works necessarily, but it gives you a good rule of thumb where like

[00:37:24] [SPEAKER_03]: when you do that and you're looking at how what your traffic numbers are

[00:37:27] [SPEAKER_03]: and like what your conversion numbers are, like, OK, let's set a rule of thumb

[00:37:31] [SPEAKER_03]: of like we should run a test for like I have I have kind of some rules here

[00:37:34] [SPEAKER_03]: where I think for most seven figure brands, you can get really confident.

[00:37:39] [SPEAKER_03]: Like you'd be really confident in your results if you run a test for four weeks.

[00:37:42] [SPEAKER_03]: And I think with most eight figure brands, it can be three weeks.

[00:37:47] [SPEAKER_03]: And this obviously eight figures is a pretty big spread,

[00:37:49] [SPEAKER_03]: but nine figure brands, I think it's two weeks.

[00:37:52] [SPEAKER_03]: And the reason why it only goes to two weeks is because I think that more

[00:37:56] [SPEAKER_03]: important than I think most brands that are above like five million dollars a year

[00:38:00] [SPEAKER_03]: or ten million dollars a year are going to get significant results.

[00:38:04] [SPEAKER_03]: But the main thing with the two week period is that like things happen in the world,

[00:38:08] [SPEAKER_03]: like the stock market was down five percent in a single day the other day.

[00:38:12] [SPEAKER_03]: Like to say that that has no impact on people's buying behavior is kind of insane.

[00:38:16] [SPEAKER_03]: So your your impact like people's psychology coming to your website

[00:38:21] [SPEAKER_03]: has an impact on your ultimate conversion rate and what things impact like

[00:38:25] [SPEAKER_03]: if it's Black Friday, for example, the oh, there's only two left

[00:38:30] [SPEAKER_03]: like order now sort of thing on your on your PDP might have a much bigger impact

[00:38:35] [SPEAKER_03]: than if it's like a Wednesday in June, you know.

[00:38:39] [SPEAKER_03]: So by running the test for, you know, a minimum duration of like, let's say two weeks,

[00:38:44] [SPEAKER_03]: you kind of smooth out these like random events that might skew the results

[00:38:49] [SPEAKER_03]: and you get to like full business cycles where like, hey, you're running a sale

[00:38:52] [SPEAKER_03]: for a few days in there and then you're not running a sale.

[00:38:54] [SPEAKER_03]: You kind of like smooth out the results.

[00:38:56] [SPEAKER_03]: And I think you can get away with running a test every week.

[00:39:00] [SPEAKER_03]: But like at the same time, if you stop the test, are you do you have another

[00:39:04] [SPEAKER_03]: test already built and developed and ready to launch that day that you're going to launch?

[00:39:09] [SPEAKER_03]: Like because most people are like, hey, let's stop it after a week

[00:39:10] [SPEAKER_03]: and then let's build out another test.

[00:39:12] [SPEAKER_03]: And then we run that for, you know, we start that a week later.

[00:39:15] [SPEAKER_03]: So it's like you could have just ran the test for two weeks while building

[00:39:18] [SPEAKER_03]: out the other, the a new test, you know, but that's one rule of thumb there.

[00:39:27] [SPEAKER_00]: So we use Prussian at a hex clad.

[00:39:29] [SPEAKER_00]: It's used by other top brands like Good American, Symbiotica, Timbuktu,

[00:39:33] [SPEAKER_00]: Nude and dozens more Prussian AI helping brands scale spend

[00:39:36] [SPEAKER_00]: confidently with AI powered marketing mix models.

[00:39:39] [SPEAKER_00]: So if you want to check out Prussian AI's marketing mix modeling tool,

[00:39:43] [SPEAKER_00]: go to PrussianAI.com forward slash operators.

[00:39:46] [SPEAKER_00]: That's P R E S C I E N T A I dot com forward slash operators.

[00:39:56] [SPEAKER_00]: Could we talk about the confidence interval stuff?

[00:40:00] [SPEAKER_00]: Because I'm very, I'm very curious about that.

[00:40:03] [SPEAKER_00]: And I held me out with this example.

[00:40:06] [SPEAKER_00]: We're running this test right now.

[00:40:07] [SPEAKER_00]: We've been talking about it on a few episodes, this desktop navigation redesign.

[00:40:11] [SPEAKER_00]: I'm back at it.

[00:40:13] [SPEAKER_00]: We have, we have over 1200 conversions on each.

[00:40:16] [SPEAKER_00]: We have over 60,000 visits that people got followed into the test on each.

[00:40:21] [SPEAKER_00]: The new version that we launched has a lower conversion rate slightly,

[00:40:25] [SPEAKER_00]: but a way better AOV.

[00:40:27] [SPEAKER_00]: So the revenue per visitor is way better.

[00:40:30] [SPEAKER_00]: Confidence interval on it though is and let me actually.

[00:40:34] [SPEAKER_00]: So this is we have a seven dollar and 91 cent revenue per visitor

[00:40:38] [SPEAKER_00]: on the one or the variation one and then seven dollar and 60 cent

[00:40:43] [SPEAKER_00]: revenue per visitor.

[00:40:45] [SPEAKER_00]: So not like a massive, massive bump, but like a bump nonetheless.

[00:40:50] [SPEAKER_00]: I'll calculate the percentage, but my confidence interval on this is only 75.5%.

[00:40:55] [SPEAKER_00]: Right?

[00:40:55] [SPEAKER_00]: Even though I've had over 1200 conversions on each.

[00:41:00] [SPEAKER_03]: Well, and yeah.

[00:41:01] [SPEAKER_03]: And this is where I think that the, this so depends on what tool

[00:41:06] [SPEAKER_03]: you're actually using because every tool has a different

[00:41:11] [SPEAKER_03]: like way of doing the statistical analysis and present.

[00:41:14] [SPEAKER_03]: Like they all use similar tools, but what they present to you is different.

[00:41:17] [SPEAKER_03]: And what you're probably, what you might be looking at is that 70%

[00:41:21] [SPEAKER_03]: is like the probability that it is better than the other ones.

[00:41:25] [SPEAKER_03]: So this is where I think people get confused and we can talk about

[00:41:28] [SPEAKER_03]: and analyze and test results, but it's not actually a probability

[00:41:31] [SPEAKER_03]: that you're getting like, is that a 3% like gain or something?

[00:41:35] [SPEAKER_00]: Again, revenue per visitor.

[00:41:37] [SPEAKER_03]: So 4%.

[00:41:38] [SPEAKER_03]: So it's like basically saying that there is a 76% chance, 76% probability.

[00:41:47] [SPEAKER_03]: So likelihood that that result not, not, it says nothing about that 4% increase.

[00:41:53] [SPEAKER_03]: All it says is that it's probably better than the control.

[00:41:58] [SPEAKER_03]: So it's, there's a 76% chance that it is essentially 0.01%

[00:42:03] [SPEAKER_03]: better to 4% to 8% better than your control.

[00:42:07] [SPEAKER_03]: Actually, a lot of tools like I use in telegrams with clients that I used

[00:42:11] [SPEAKER_03]: what they present is that probability to be best, which is essentially

[00:42:15] [SPEAKER_03]: all you're saying is that it's not worse than the control.

[00:42:18] [SPEAKER_03]: You're not saying anything about like that 4% and the reality

[00:42:21] [SPEAKER_03]: of that 4%, if that makes sense.

[00:42:23] [SPEAKER_00]: Right. But there's still a 25% chance that the original page

[00:42:26] [SPEAKER_00]: is actually better than the variation that the results that we're seeing

[00:42:30] [SPEAKER_00]: are just from a variety of noise like consumer behavior and a million

[00:42:36] [SPEAKER_00]: other things that could be going on.

[00:42:38] [SPEAKER_03]: We just launched a new product and this is where it's risky to,

[00:42:41] [SPEAKER_03]: this is why it's risky to not wait to get actual results.

[00:42:45] [SPEAKER_03]: Because like if you, like there's different ways that people kind of go about

[00:42:49] [SPEAKER_03]: this, but if you're at 76% waiting to 95%, the problem is at that 4%,

[00:42:54] [SPEAKER_03]: if it goes down to 2%, it's just going to take even longer to get 95.

[00:42:59] [SPEAKER_03]: Like 95% because like the smaller the change, the harder it is to be

[00:43:04] [SPEAKER_03]: confident essentially like, hey, we're measuring really small differences.

[00:43:07] [SPEAKER_03]: So like it's pretty hard to be sure that it's better.

[00:43:10] [SPEAKER_03]: It's just like the logical way of thinking through it.

[00:43:12] [SPEAKER_03]: So if over time you see that go from four to three, well, you might

[00:43:17] [SPEAKER_03]: have just like added another couple of days before you get to that 95%,

[00:43:20] [SPEAKER_03]: which is like one of the challenges of interpreting.

[00:43:25] [SPEAKER_03]: So what do I do?

[00:43:25] [SPEAKER_03]: Essentially it's like you kind of running like, hey, we're just.

[00:43:28] [SPEAKER_03]: Well, how long have you been running it for?

[00:43:30] [SPEAKER_03]: Like three weeks at this point.

[00:43:31] [SPEAKER_00]: Okay.

[00:43:32] [SPEAKER_03]: Three weeks.

[00:43:32] [SPEAKER_03]: And are you segmenting out mobile traffic?

[00:43:36] [SPEAKER_00]: Uh, yeah.

[00:43:36] [SPEAKER_00]: Then this is, this is a desktop only test specifically.

[00:43:39] [SPEAKER_00]: We have two separate.

[00:43:40] [SPEAKER_00]: Yeah.

[00:43:40] [SPEAKER_00]: This is just for our desktop redesign.

[00:43:43] [SPEAKER_00]: So yeah.

[00:43:43] [SPEAKER_00]: The test by, by design is nothing changes if you go and like filter

[00:43:47] [SPEAKER_00]: out desktop only because it's only showing to those people.

[00:43:51] [SPEAKER_03]: Okay.

[00:43:51] [SPEAKER_03]: Well, this is one of those.

[00:43:53] [SPEAKER_03]: Like a lot of people will give you different answers.

[00:43:56] [SPEAKER_03]: I would say that if you've been running it for a few weeks ready, this

[00:43:59] [SPEAKER_03]: gets to the point of like your mobile traffic is way higher than

[00:44:02] [SPEAKER_03]: your desktop.

[00:44:02] [SPEAKER_03]: So like it makes it really hard to get confident results on desktop.

[00:44:05] [SPEAKER_03]: I would say that on desktop, like on mobile, you probably can run

[00:44:09] [SPEAKER_03]: a test and have results within a week realistically and like feel

[00:44:12] [SPEAKER_03]: really confident in them.

[00:44:13] [SPEAKER_03]: But on desktop, you're probably more like at that four week threshold for

[00:44:16] [SPEAKER_03]: like, Hey, we're running a, I think setting your expectations

[00:44:18] [SPEAKER_03]: differently for the amount of traffic that you're actually receiving

[00:44:22] [SPEAKER_03]: into that experiment is a pretty important thing to do up front.

[00:44:25] [SPEAKER_03]: So I would probably lean towards like, Hey, we're going to have

[00:44:28] [SPEAKER_03]: to run this test for four weeks to get a significant result.

[00:44:32] [SPEAKER_03]: And then if it was on mobile, it would only be like one to two

[00:44:35] [SPEAKER_03]: weeks.

[00:44:35] [SPEAKER_03]: Obviously, like scale changes things.

[00:44:37] [SPEAKER_03]: I know your position you're in, it's probably like you could run

[00:44:40] [SPEAKER_03]: a test in one week, even though I say two weeks should be the minimum.

[00:44:44] [SPEAKER_03]: But I would say just let it run four weeks and then you get

[00:44:47] [SPEAKER_03]: into this position where you can't actually call a result.

[00:44:50] [SPEAKER_03]: I think that you should try to actually like it's so hard because

[00:44:57] [SPEAKER_03]: it's so context dependent on like your where you guys are in

[00:45:00] [SPEAKER_03]: like with all the background that led up to this.

[00:45:03] [SPEAKER_03]: But likely what you want to do is not just say, Hey, we're 70%

[00:45:08] [SPEAKER_03]: confident in this result.

[00:45:10] [SPEAKER_03]: You might just say, cool, let's go on and move it.

[00:45:13] [SPEAKER_03]: But there's that there's the downside risk is that it's

[00:45:16] [SPEAKER_03]: actually worse.

[00:45:17] [SPEAKER_03]: And now you're just iterating on a worse concept.

[00:45:20] [SPEAKER_03]: And so what I would tell you to do is try to assess, okay, this

[00:45:24] [SPEAKER_03]: only was a 4% improvement, but we made big, did we change

[00:45:27] [SPEAKER_03]: enough?

[00:45:27] [SPEAKER_03]: Like, did we actually make big enough changes?

[00:45:29] [SPEAKER_03]: What did we change?

[00:45:30] [SPEAKER_03]: How can we modify it even further to try to see a bigger

[00:45:33] [SPEAKER_03]: impact to see if this like, what is the thesis behind

[00:45:36] [SPEAKER_03]: the change in navigation and like, how can we lean into

[00:45:39] [SPEAKER_03]: that even stronger because you're the baseline.

[00:45:43] [SPEAKER_03]: If you end a test without a comp, like a confident result,

[00:45:46] [SPEAKER_03]: it's like, it's really no different is essentially what

[00:45:49] [SPEAKER_03]: you're getting out.

[00:45:49] [SPEAKER_03]: Like maybe you didn't change enough like your thesis of

[00:45:51] [SPEAKER_03]: like, Hey, we're driving people to collection pages a lot

[00:45:54] [SPEAKER_03]: harder instead of trying to get them directly to a product

[00:45:57] [SPEAKER_03]: isn't actually impacting customer behavior.

[00:46:00] [SPEAKER_03]: So like that's one takeaway, you know, but if, if that's

[00:46:04] [SPEAKER_03]: what you're testing that I would say just try to make

[00:46:05] [SPEAKER_03]: it go even further with the change essentially to try to

[00:46:09] [SPEAKER_03]: get a bigger result.

[00:46:10] [SPEAKER_00]: Cause do you have this?

[00:46:11] [SPEAKER_00]: Do you have any thought that like you should never

[00:46:12] [SPEAKER_00]: conclude a test before it's at a minimum confidence

[00:46:15] [SPEAKER_00]: interval?

[00:46:15] [SPEAKER_00]: Like we use our, and we should go into this at some

[00:46:18] [SPEAKER_00]: point, but like basically we use convert.com and then

[00:46:20] [SPEAKER_00]: we have a web hook that's connecting Shopify and like

[00:46:23] [SPEAKER_00]: GA to convert.

[00:46:24] [SPEAKER_00]: And then so we're, we're that's, that's the

[00:46:26] [SPEAKER_00]: confidence interval that I'm looking at here is

[00:46:28] [SPEAKER_00]: convert.com's reported, uh, confidence interval.

[00:46:31] [SPEAKER_00]: Like, but do you have a minimum that you're

[00:46:32] [SPEAKER_00]: like, you should never include as how or is it

[00:46:34] [SPEAKER_00]: context?

[00:46:35] [SPEAKER_03]: No, I, well, this goes into like how to, this

[00:46:37] [SPEAKER_03]: goes into like how to run with the principles

[00:46:39] [SPEAKER_03]: for how to run an experiment.

[00:46:40] [SPEAKER_03]: Like I, I think you should honestly run the

[00:46:43] [SPEAKER_03]: test for the duration that you pick beforehand and

[00:46:48] [SPEAKER_03]: then end the test when it hits that duration and

[00:46:51] [SPEAKER_03]: interpret the results from there.

[00:46:52] [SPEAKER_03]: Because with most of these, like with at least, and

[00:46:56] [SPEAKER_03]: I'm not super familiar with, with convert, but

[00:46:59] [SPEAKER_03]: with, for example, like in, in intelligence, they

[00:47:02] [SPEAKER_03]: have this ability to look at your, um, your, you

[00:47:06] [SPEAKER_03]: kind of have a tolerance level and a probability

[00:47:08] [SPEAKER_03]: to be best.

[00:47:08] [SPEAKER_03]: So it'll tell you like, Hey, if I'm willing

[00:47:10] [SPEAKER_03]: for this to be 1% worse, like what is the odds

[00:47:14] [SPEAKER_03]: that this is no worse than 1% worse than the

[00:47:16] [SPEAKER_03]: control, but actually this much better, you know,

[00:47:20] [SPEAKER_03]: that whatever we're getting, like actually

[00:47:21] [SPEAKER_03]: better, like what's the probability it's going

[00:47:23] [SPEAKER_03]: to beat it.

[00:47:23] [SPEAKER_03]: And if it doesn't beat it, what's probably

[00:47:25] [SPEAKER_03]: it's going to be 1% worse.

[00:47:26] [SPEAKER_03]: So that might be for you, that might be like

[00:47:29] [SPEAKER_03]: only 54%.

[00:47:30] [SPEAKER_03]: And so when you're looking like you can dive

[00:47:32] [SPEAKER_03]: a little bit deeper and then just be like, I

[00:47:34] [SPEAKER_03]: would just end the test and say let's interpret

[00:47:37] [SPEAKER_03]: the results because otherwise you just, you

[00:47:39] [SPEAKER_03]: have to have a rule for stopping it.

[00:47:41] [SPEAKER_03]: And this is why I think duration works because

[00:47:44] [SPEAKER_03]: you could just have another thing queued up and

[00:47:47] [SPEAKER_03]: ready to go while you're interpreting this

[00:47:49] [SPEAKER_03]: result.

[00:47:49] [SPEAKER_03]: So it kind of lends itself to a process of

[00:47:51] [SPEAKER_03]: improvement where you have, Hey, you're

[00:47:53] [SPEAKER_03]: running for this duration.

[00:47:55] [SPEAKER_03]: And then at the end of that, you know, Hey, we

[00:47:56] [SPEAKER_03]: have this next test, maybe in a different

[00:47:58] [SPEAKER_03]: area, whatever, we have another test queued

[00:48:00] [SPEAKER_03]: up to run for the next two weeks.

[00:48:02] [SPEAKER_03]: And while you're kind of running that, you

[00:48:04] [SPEAKER_03]: can interpret the results and try to be like,

[00:48:06] [SPEAKER_03]: Hey, let's like have an honest conversation

[00:48:08] [SPEAKER_03]: about, is this navigation actually having an

[00:48:10] [SPEAKER_03]: impact at all?

[00:48:11] [SPEAKER_03]: And like, is it worth putting more effort

[00:48:13] [SPEAKER_03]: into because if you're not seeing like 3%

[00:48:15] [SPEAKER_03]: is obviously a change, but it's on your

[00:48:16] [SPEAKER_03]: desktop traffic, like we're segmenting down

[00:48:18] [SPEAKER_03]: into segments, you know.

[00:48:20] [SPEAKER_03]: So that's where how I would kind of go

[00:48:23] [SPEAKER_03]: out the process.

[00:48:25] [SPEAKER_01]: So I have a quick question on this one.

[00:48:27] [SPEAKER_01]: The process of interpreting results.

[00:48:30] [SPEAKER_01]: I'm curious, oh, we talk about conversion,

[00:48:32] [SPEAKER_01]: right?

[00:48:32] [SPEAKER_01]: Or you look at like contribution margin

[00:48:33] [SPEAKER_01]: per visitor, things like that.

[00:48:35] [SPEAKER_01]: And I'm sure this changes like significantly

[00:48:37] [SPEAKER_01]: depending on what you're actually testing.

[00:48:39] [SPEAKER_01]: But what are some of the other data sources

[00:48:42] [SPEAKER_01]: and like behaviors that you actually look

[00:48:44] [SPEAKER_01]: at when interpreting test results?

[00:48:46] [SPEAKER_03]: Yeah, I mean, the main obviously the

[00:48:47] [SPEAKER_03]: main thing is just looking at the

[00:48:48] [SPEAKER_03]: core metric of conversion rate

[00:48:50] [SPEAKER_03]: and revenue per session.

[00:48:51] [SPEAKER_03]: But then ideally you want to look at

[00:48:53] [SPEAKER_03]: like how it's affecting new versus

[00:48:55] [SPEAKER_03]: returning visitors.

[00:48:56] [SPEAKER_03]: Like you can kind of segment down.

[00:48:58] [SPEAKER_03]: This is another reason to run tests

[00:49:00] [SPEAKER_03]: for longer because you get more data.

[00:49:01] [SPEAKER_03]: So when you go into segments, you

[00:49:03] [SPEAKER_03]: can be more confident in those segments

[00:49:05] [SPEAKER_03]: essentially.

[00:49:06] [SPEAKER_03]: So I honestly used to use Google

[00:49:09] [SPEAKER_03]: Analytics for this a lot.

[00:49:11] [SPEAKER_03]: Just actually like looking at the two

[00:49:12] [SPEAKER_03]: different groups, I don't use it as much

[00:49:14] [SPEAKER_03]: anymore because GA4 is like painful

[00:49:17] [SPEAKER_03]: and I haven't spent the time to really

[00:49:19] [SPEAKER_03]: double down on it.

[00:49:21] [SPEAKER_03]: But looking at like you can

[00:49:23] [SPEAKER_03]: start to look at when you're trying

[00:49:24] [SPEAKER_03]: to make a justification of like, hey,

[00:49:26] [SPEAKER_03]: do we think this is actually having an

[00:49:27] [SPEAKER_03]: impact?

[00:49:28] [SPEAKER_03]: You could start to go down the list

[00:49:29] [SPEAKER_03]: of like time on site.

[00:49:32] [SPEAKER_03]: Like how you can look at how

[00:49:34] [SPEAKER_03]: different cohorts of users are

[00:49:35] [SPEAKER_03]: behaving essentially on the site.

[00:49:37] [SPEAKER_03]: How many pages they're looking at,

[00:49:39] [SPEAKER_03]: etc. To think like, OK,

[00:49:41] [SPEAKER_03]: is there something here?

[00:49:42] [SPEAKER_03]: Because when you start looking into

[00:49:43] [SPEAKER_03]: those behavioral metrics, for example,

[00:49:45] [SPEAKER_03]: with what Connor's doing, you might

[00:49:46] [SPEAKER_03]: see, OK, these metrics also look

[00:49:49] [SPEAKER_03]: exactly the same.

[00:49:50] [SPEAKER_03]: Like people are seeing just as many

[00:49:51] [SPEAKER_03]: pages, people are spending just

[00:49:52] [SPEAKER_03]: much time on site.

[00:49:54] [SPEAKER_03]: Like clearly this isn't actually

[00:49:56] [SPEAKER_03]: impacting their behavior very

[00:49:59] [SPEAKER_03]: much.

[00:49:59] [SPEAKER_03]: So we need to make a much bigger

[00:50:01] [SPEAKER_03]: change or like you might see,

[00:50:02] [SPEAKER_03]: wow, people are spending a lot more

[00:50:03] [SPEAKER_03]: time on site, seeing a lot more

[00:50:05] [SPEAKER_03]: pages and then that might change

[00:50:07] [SPEAKER_03]: the way. Hey, for this next

[00:50:08] [SPEAKER_03]: variation, let's change these

[00:50:10] [SPEAKER_03]: things, etc.

[00:50:11] [SPEAKER_03]: So I like to look at the two

[00:50:13] [SPEAKER_03]: different segments in those

[00:50:15] [SPEAKER_03]: behavior metrics.

[00:50:15] [SPEAKER_03]: But really that's only if it's

[00:50:18] [SPEAKER_03]: up for debate, essentially.

[00:50:20] [SPEAKER_03]: Like if you have a confident

[00:50:21] [SPEAKER_03]: result that this is better and

[00:50:22] [SPEAKER_03]: you feel like from a conversion

[00:50:23] [SPEAKER_03]: rate revenue procession, like

[00:50:24] [SPEAKER_03]: your core metric perspective

[00:50:27] [SPEAKER_03]: then like publish it,

[00:50:30] [SPEAKER_03]: like just launch it and then

[00:50:31] [SPEAKER_03]: move on to the next test

[00:50:32] [SPEAKER_03]: essentially.

[00:50:34] [SPEAKER_03]: I really go into those detailed

[00:50:35] [SPEAKER_03]: metrics only when it's like a

[00:50:38] [SPEAKER_03]: hard debate of like, are we

[00:50:39] [SPEAKER_03]: going to spend more time trying

[00:50:40] [SPEAKER_03]: to like improve this when we're

[00:50:41] [SPEAKER_03]: not seeing any actual

[00:50:43] [SPEAKER_03]: difference?

[00:50:43] [SPEAKER_03]: You know, yeah.

[00:50:45] [SPEAKER_00]: Totally.

[00:50:45] [SPEAKER_00]: So as I'm doing this live,

[00:50:47] [SPEAKER_00]: this is great.

[00:50:48] [SPEAKER_00]: This is why I wanted to have you

[00:50:48] [SPEAKER_00]: on, because I'm now doing this

[00:50:49] [SPEAKER_00]: live as we were talking about it.

[00:50:52] [SPEAKER_00]: So our new user, you're

[00:50:53] [SPEAKER_00]: working.

[00:50:53] [SPEAKER_00]: I'm working.

[00:50:54] [SPEAKER_00]: This is like you're on the

[00:50:55] [SPEAKER_00]: clock for so long.

[00:50:56] [SPEAKER_00]: Yeah.

[00:50:57] [SPEAKER_00]: Our when I actually broke it

[00:50:59] [SPEAKER_00]: out and I just selected new

[00:51:00] [SPEAKER_00]: users revenue per user up 10

[00:51:02] [SPEAKER_00]: percent, 91 percent confidence,

[00:51:04] [SPEAKER_00]: which is ultimately what I was

[00:51:05] [SPEAKER_00]: what I was hoping to get out of

[00:51:07] [SPEAKER_00]: this was a bump in new when I

[00:51:08] [SPEAKER_00]: look at returning our revenue

[00:51:10] [SPEAKER_00]: per user is actually down.

[00:51:11] [SPEAKER_00]: Not significantly, but it is

[00:51:13] [SPEAKER_00]: down.

[00:51:13] [SPEAKER_00]: But I'm less concerned about that

[00:51:15] [SPEAKER_00]: because at the end of the day,

[00:51:15] [SPEAKER_00]: like most of the repeat orders

[00:51:17] [SPEAKER_00]: that we're going to get are

[00:51:18] [SPEAKER_00]: probably going to come through

[00:51:20] [SPEAKER_00]: like we're creating awareness

[00:51:22] [SPEAKER_00]: right about all the products we

[00:51:23] [SPEAKER_00]: have and like people are going

[00:51:24] [SPEAKER_00]: to come to site either through

[00:51:25] [SPEAKER_00]: a like marketing funnel that's

[00:51:27] [SPEAKER_00]: going to be super streamlined

[00:51:29] [SPEAKER_00]: or they're going to come to

[00:51:30] [SPEAKER_00]: the site and like know exactly

[00:51:31] [SPEAKER_00]: where they want to go.

[00:51:32] [SPEAKER_00]: I am really I am more like one.

[00:51:34] [SPEAKER_03]: Yeah, one thing I'd say is that

[00:51:37] [SPEAKER_03]: with this is where it's so

[00:51:39] [SPEAKER_03]: hard to like actually get

[00:51:41] [SPEAKER_03]: good data on some of this stuff

[00:51:42] [SPEAKER_03]: and you have to go layers deeper.

[00:51:44] [SPEAKER_03]: Like for example, I'll just say

[00:51:45] [SPEAKER_03]: we're getting that one thing

[00:51:46] [SPEAKER_03]: you can do is tag orders by

[00:51:48] [SPEAKER_03]: test and then do like order

[00:51:49] [SPEAKER_03]: analysis afterwards, which can be

[00:51:51] [SPEAKER_03]: really helpful to see like the

[00:51:52] [SPEAKER_03]: different types of orders

[00:51:53] [SPEAKER_03]: people are placing.

[00:51:54] [SPEAKER_03]: Are they buying sets more?

[00:51:55] [SPEAKER_03]: They buying individual pants

[00:51:56] [SPEAKER_03]: more etc.

[00:51:57] [SPEAKER_03]: But like in your position

[00:51:59] [SPEAKER_03]: new versus returning users,

[00:52:00] [SPEAKER_03]: that can be someone like came to

[00:52:02] [SPEAKER_03]: your site and left and then came back.

[00:52:04] [SPEAKER_03]: Not somebody like came back

[00:52:07] [SPEAKER_03]: because they already purchased

[00:52:08] [SPEAKER_03]: before and they're buying again

[00:52:09] [SPEAKER_03]: because GA just gives you that

[00:52:11] [SPEAKER_03]: like basically session based.

[00:52:14] [SPEAKER_03]: So someone could even like, you

[00:52:15] [SPEAKER_03]: know, Google does their best to

[00:52:16] [SPEAKER_03]: kind of like pull these people

[00:52:17] [SPEAKER_03]: together.

[00:52:17] [SPEAKER_03]: But you get into this like

[00:52:18] [SPEAKER_03]: that's somebody who's come to

[00:52:20] [SPEAKER_03]: your site left and then come back.

[00:52:22] [SPEAKER_03]: So what you can do and this is

[00:52:24] [SPEAKER_03]: where at someone at your scale

[00:52:26] [SPEAKER_03]: can do some stuff that's more

[00:52:28] [SPEAKER_03]: sophisticated like this.

[00:52:28] [SPEAKER_03]: You can trigger tests based on

[00:52:31] [SPEAKER_03]: certain query parameters,

[00:52:35] [SPEAKER_03]: essentially.

[00:52:35] [SPEAKER_03]: And so if you're running,

[00:52:37] [SPEAKER_03]: essentially, if you're like, hey,

[00:52:37] [SPEAKER_03]: we know that we're getting on this

[00:52:39] [SPEAKER_03]: campaign, we're getting 80% new

[00:52:42] [SPEAKER_03]: customers, like we're like 80% of

[00:52:44] [SPEAKER_03]: people are net new.

[00:52:46] [SPEAKER_03]: Let's trigger a test based off

[00:52:48] [SPEAKER_03]: of those query parameters or the

[00:52:49] [SPEAKER_03]: landing page we're sending them

[00:52:51] [SPEAKER_03]: to and only enter people.

[00:52:52] [SPEAKER_03]: Like if you want to see how

[00:52:53] [SPEAKER_03]: the impact it has on new

[00:52:54] [SPEAKER_03]: customers, you can kind of do

[00:52:56] [SPEAKER_03]: that because you know that

[00:52:58] [SPEAKER_03]: those are people if you want

[00:52:59] [SPEAKER_03]: another variation is like this is

[00:53:00] [SPEAKER_03]: again at scale, you can do this

[00:53:02] [SPEAKER_03]: because you can do it like you

[00:53:03] [SPEAKER_03]: can get a reasonable sample

[00:53:04] [SPEAKER_03]: size, maybe not on desktop.

[00:53:05] [SPEAKER_03]: But like, hey, if you send an

[00:53:07] [SPEAKER_03]: email only to returning customers

[00:53:10] [SPEAKER_03]: for a specific promotion, you

[00:53:12] [SPEAKER_03]: can trigger a test based off

[00:53:13] [SPEAKER_03]: of that and be like, hey, these

[00:53:14] [SPEAKER_03]: are we only started this test

[00:53:16] [SPEAKER_03]: with returning customers.

[00:53:18] [SPEAKER_03]: And you might be able to get

[00:53:19] [SPEAKER_03]: some signal out of that at the

[00:53:21] [SPEAKER_03]: volumes that you're doing.

[00:53:22] [SPEAKER_03]: But again, getting so segmented

[00:53:24] [SPEAKER_03]: in detail this like I think

[00:53:25] [SPEAKER_03]: if someone's a low eight

[00:53:27] [SPEAKER_03]: figure brand, it's going to be

[00:53:28] [SPEAKER_03]: really hard to get that detailed.

[00:53:30] [SPEAKER_03]: So that's why I lean on like,

[00:53:32] [SPEAKER_03]: hey, you can dig in to see if

[00:53:34] [SPEAKER_03]: it's worth putting more effort

[00:53:35] [SPEAKER_03]: into this and dig into these

[00:53:36] [SPEAKER_03]: metrics. But at the end of the

[00:53:38] [SPEAKER_03]: day, I would say just like end

[00:53:41] [SPEAKER_03]: the test at a set duration.

[00:53:42] [SPEAKER_03]: Don't just leave them running

[00:53:43] [SPEAKER_03]: forever and move on to the

[00:53:45] [SPEAKER_03]: next one or make a decision.

[00:53:47] [SPEAKER_03]: Hey, let's try again.

[00:53:48] [SPEAKER_03]: Essentially, let's let's try

[00:53:49] [SPEAKER_03]: again to get a confident

[00:53:50] [SPEAKER_03]: result. If that makes sense.

[00:53:52] [SPEAKER_00]: Totally makes sense.

[00:53:53] [SPEAKER_00]: All right, we got we went

[00:53:54] [SPEAKER_00]: in the weeds there.

[00:53:55] [SPEAKER_00]: We got we got stopped at rules

[00:53:56] [SPEAKER_00]: to follow. All right, number one,

[00:53:57] [SPEAKER_00]: you have to have a minimum

[00:53:58] [SPEAKER_00]: test duration in days or weeks.

[00:54:01] [SPEAKER_00]: It scales up or down depending

[00:54:02] [SPEAKER_00]: on your revenue levels.

[00:54:03] [SPEAKER_00]: All right, rule number two, what

[00:54:05] [SPEAKER_00]: what's let's keep going through

[00:54:06] [SPEAKER_04]: these. Yeah, this is

[00:54:08] [SPEAKER_03]: that you guys were talking

[00:54:09] [SPEAKER_03]: about this literally like last

[00:54:10] [SPEAKER_03]: week. I don't know when you

[00:54:10] [SPEAKER_03]: recorded it, but you were like

[00:54:11] [SPEAKER_03]: talking about taking a test

[00:54:12] [SPEAKER_03]: results and one of your one

[00:54:14] [SPEAKER_03]: of your tweets of like how

[00:54:15] [SPEAKER_03]: fun it is and how everybody

[00:54:16] [SPEAKER_03]: does it. But like there is

[00:54:19] [SPEAKER_03]: a downside, I think to

[00:54:20] [SPEAKER_03]: actually doing that because

[00:54:21] [SPEAKER_03]: you were even saying like you

[00:54:22] [SPEAKER_03]: get this euphoria where like

[00:54:24] [SPEAKER_03]: you look at it, like it's

[00:54:25] [SPEAKER_03]: a winner and then like the

[00:54:26] [SPEAKER_03]: result come down so

[00:54:28] [SPEAKER_03]: significantly and you're like,

[00:54:28] [SPEAKER_03]: oh, that 30% improvement is

[00:54:31] [SPEAKER_03]: now only 7% and cool 7% still

[00:54:34] [SPEAKER_03]: win. But it feels way less

[00:54:36] [SPEAKER_03]: compelling than 30% like your

[00:54:38] [SPEAKER_03]: way.

[00:54:38] [SPEAKER_03]: You can literally hear it in my

[00:54:39] [SPEAKER_03]: voice when I was just looking

[00:54:41] [SPEAKER_03]: at it. I was like, oh, users,

[00:54:42] [SPEAKER_00]: it's 10%. I'm like, exactly.

[00:54:45] [SPEAKER_03]: Yeah, what you start to look

[00:54:47] [SPEAKER_03]: at these things and you

[00:54:48] [SPEAKER_03]: you you kind of bias yourself

[00:54:49] [SPEAKER_03]: and look, this is like on

[00:54:51] [SPEAKER_03]: the margins level of impact.

[00:54:53] [SPEAKER_03]: But when you bias yourself

[00:54:54] [SPEAKER_03]: like in a continuous testing

[00:54:56] [SPEAKER_03]: program where you're trying to

[00:54:57] [SPEAKER_03]: get better month over month,

[00:54:58] [SPEAKER_03]: you have to make decisions

[00:55:00] [SPEAKER_03]: about what you're going to

[00:55:01] [SPEAKER_03]: test. And if you see

[00:55:02] [SPEAKER_03]: something on day one, have a

[00:55:04] [SPEAKER_03]: 30% improvement like this

[00:55:05] [SPEAKER_03]: like this navigation, if you

[00:55:06] [SPEAKER_03]: were like, hey, day one, we

[00:55:07] [SPEAKER_03]: saw 30% improvement in

[00:55:09] [SPEAKER_03]: revenue procession, this is

[00:55:10] [SPEAKER_03]: going to crush it. And then

[00:55:11] [SPEAKER_03]: two weeks in, you're at

[00:55:12] [SPEAKER_03]: 4% and you're like, I

[00:55:14] [SPEAKER_03]: don't know man, I think

[00:55:15] [SPEAKER_03]: we could get to 30%. If we

[00:55:17] [SPEAKER_03]: just tweak this and we really

[00:55:18] [SPEAKER_03]: work it out, let's put

[00:55:19] [SPEAKER_03]: some more effort into it,

[00:55:20] [SPEAKER_03]: like let's double down.

[00:55:21] [SPEAKER_03]: But in reality, if you let

[00:55:22] [SPEAKER_03]: this test run for another

[00:55:23] [SPEAKER_03]: four weeks, it might go to

[00:55:24] [SPEAKER_03]: zero. And it's like, hey,

[00:55:25] [SPEAKER_03]: maybe your whole thesis around

[00:55:27] [SPEAKER_03]: the desktop navigation was

[00:55:30] [SPEAKER_03]: wrong or just isn't actually

[00:55:32] [SPEAKER_03]: important. And you should

[00:55:33] [SPEAKER_03]: like, there's a reality where

[00:55:34] [SPEAKER_03]: you shouldn't be spending any

[00:55:35] [SPEAKER_03]: more energy on it because

[00:55:37] [SPEAKER_03]: it's not going to have some

[00:55:38] [SPEAKER_03]: massive impact. Hey, implement

[00:55:40] [SPEAKER_03]: the new one, there's a high

[00:55:41] [SPEAKER_03]: probability that it's going

[00:55:42] [SPEAKER_03]: to be better and just like

[00:55:43] [SPEAKER_03]: be okay with it, try maybe

[00:55:45] [SPEAKER_03]: doing another iteration on

[00:55:46] [SPEAKER_03]: that, like can we improve

[00:55:47] [SPEAKER_03]: this? But if you're not

[00:55:47] [SPEAKER_03]: seeing big changes, you

[00:55:49] [SPEAKER_03]: should be making the

[00:55:50] [SPEAKER_03]: decision to focus your

[00:55:51] [SPEAKER_03]: resources and your energy

[00:55:52] [SPEAKER_03]: elsewhere, especially with a

[00:55:54] [SPEAKER_03]: desktop navigation test

[00:55:56] [SPEAKER_03]: that's like 10% of your

[00:55:58] [SPEAKER_03]: user base or something

[00:55:59] [SPEAKER_03]: like that. It's likely that

[00:56:01] [SPEAKER_03]: like that energy could be

[00:56:02] [SPEAKER_03]: better spent on a mobile

[00:56:03] [SPEAKER_03]: navigation test where it's

[00:56:04] [SPEAKER_03]: 90% of your user base.

[00:56:06] [SPEAKER_03]: You know what I mean? So

[00:56:07] [SPEAKER_03]: that's where I think peaking

[00:56:09] [SPEAKER_03]: at results and seeing things

[00:56:10] [SPEAKER_03]: that just really mean

[00:56:11] [SPEAKER_03]: nothing at the end of the

[00:56:12] [SPEAKER_03]: day can impact how you

[00:56:14] [SPEAKER_03]: kind of make decisions going

[00:56:15] [SPEAKER_03]: forward.

[00:56:16] [SPEAKER_03]: And there's also that's

[00:56:17] [SPEAKER_03]: even I know nobody's

[00:56:18] [SPEAKER_03]: gonna actually do this. So

[00:56:19] [SPEAKER_03]: it's fine. Like I still

[00:56:21] [SPEAKER_03]: do it like I try to

[00:56:24] [SPEAKER_03]: look away for a few days.

[00:56:25] [SPEAKER_03]: And then like I end up

[00:56:26] [SPEAKER_03]: checking but like dude,

[00:56:27] [SPEAKER_03]: definitely don't look at it

[00:56:28] [SPEAKER_03]: the day after because it's

[00:56:29] [SPEAKER_03]: just so meaningless and it

[00:56:31] [SPEAKER_03]: does really mess with your

[00:56:33] [SPEAKER_03]: emotions. Like give it at

[00:56:34] [SPEAKER_03]: least a couple of days

[00:56:35] [SPEAKER_03]: before you take a look

[00:56:35] [SPEAKER_01]: day after all 24 hours.

[00:56:38] [SPEAKER_01]: We probably have a

[00:56:39] [SPEAKER_01]: probably four or five

[00:56:40] [SPEAKER_01]: versions in that.

[00:56:41] [SPEAKER_01]: Yeah. Yeah.

[00:56:42] [SPEAKER_00]: And I think that confirmation

[00:56:43] [SPEAKER_00]: bias is really irrelevant in

[00:56:45] [SPEAKER_00]: like, like there's ego.

[00:56:47] [SPEAKER_00]: There's a lot of ego and

[00:56:48] [SPEAKER_00]: ego hurts and like that

[00:56:50] [SPEAKER_00]: confirmation bias becomes

[00:56:51] [SPEAKER_00]: even more enhanced when

[00:56:53] [SPEAKER_00]: when you add in like very

[00:56:54] [SPEAKER_00]: strong egos which like I

[00:56:56] [SPEAKER_00]: understand why that's the

[00:56:57] [SPEAKER_00]: case and in a role in a job

[00:56:59] [SPEAKER_00]: where I mean 99% of people

[00:57:02] [SPEAKER_00]: in the in the workforce in

[00:57:04] [SPEAKER_00]: the United States have or

[00:57:05] [SPEAKER_00]: probably higher have no

[00:57:06] [SPEAKER_00]: direct attachment between

[00:57:08] [SPEAKER_00]: their work and like how

[00:57:09] [SPEAKER_00]: that drives top line or

[00:57:11] [SPEAKER_00]: bottom line revenue top line

[00:57:14] [SPEAKER_00]: revenue growth or bottom

[00:57:15] [SPEAKER_00]: line profit margin growth.

[00:57:16] [SPEAKER_00]: And that's one of the

[00:57:16] [SPEAKER_00]: coolest things about this

[00:57:17] [SPEAKER_00]: job is like you have in

[00:57:19] [SPEAKER_00]: immediate instantaneous

[00:57:21] [SPEAKER_00]: connection to what you're

[00:57:23] [SPEAKER_00]: doing and how that affects

[00:57:24] [SPEAKER_00]: those things.

[00:57:25] [SPEAKER_00]: But yeah, I think that

[00:57:26] [SPEAKER_00]: that confirmation bias that

[00:57:28] [SPEAKER_00]: can come into play is like

[00:57:29] [SPEAKER_00]: even more enhanced because of

[00:57:30] [SPEAKER_00]: that.

[00:57:31] [SPEAKER_03]: Yeah.

[00:57:32] [SPEAKER_03]: Yeah.

[00:57:33] [SPEAKER_03]: And it's you know, I

[00:57:34] [SPEAKER_03]: I don't think anybody's

[00:57:35] [SPEAKER_03]: actually going to change

[00:57:36] [SPEAKER_03]: that behavior.

[00:57:37] [SPEAKER_03]: But like there is there is

[00:57:38] [SPEAKER_03]: a downside if you end

[00:57:39] [SPEAKER_03]: tests if you're under the

[00:57:41] [SPEAKER_03]: mindset or your agency

[00:57:42] [SPEAKER_03]: ends tests when you get

[00:57:44] [SPEAKER_03]: a confident result like a

[00:57:45] [SPEAKER_03]: statistically significant

[00:57:46] [SPEAKER_03]: result.

[00:57:46] [SPEAKER_03]: For example, if you're

[00:57:48] [SPEAKER_03]: looking in intelligence and

[00:57:49] [SPEAKER_03]: says 95% probability to be

[00:57:51] [SPEAKER_03]: best on day two, but

[00:57:53] [SPEAKER_03]: you know you should be

[00:57:54] [SPEAKER_03]: running tests for two weeks

[00:57:55] [SPEAKER_03]: after listening to this

[00:57:56] [SPEAKER_03]: conversation and they end

[00:57:57] [SPEAKER_03]: the test.

[00:57:58] [SPEAKER_03]: There is actually like a

[00:57:59] [SPEAKER_03]: statistical problem with

[00:58:01] [SPEAKER_03]: the analysis where like you

[00:58:02] [SPEAKER_03]: can get false positives

[00:58:04] [SPEAKER_03]: and you can get

[00:58:05] [SPEAKER_03]: Hey, in reality, like I

[00:58:06] [SPEAKER_03]: think even in like a a

[00:58:08] [SPEAKER_03]: test, you get like 11%

[00:58:10] [SPEAKER_03]: false positive rates where

[00:58:12] [SPEAKER_03]: like there's absolutely no

[00:58:13] [SPEAKER_03]: change.

[00:58:13] [SPEAKER_03]: You will get false

[00:58:15] [SPEAKER_03]: positives throughout the

[00:58:17] [SPEAKER_03]: course of when you

[00:58:18] [SPEAKER_03]: should be running the test.

[00:58:20] [SPEAKER_03]: So there is actually a

[00:58:21] [SPEAKER_03]: downside to it like just you

[00:58:23] [SPEAKER_03]: can see it whatever but like

[00:58:24] [SPEAKER_03]: don't end your tests because

[00:58:25] [SPEAKER_03]: you're like, oh my god,

[00:58:26] [SPEAKER_03]: look at this result.

[00:58:27] [SPEAKER_03]: Let's end the test.

[00:58:28] [SPEAKER_03]: We have 95% probability of

[00:58:30] [SPEAKER_03]: it being winner and then you

[00:58:31] [SPEAKER_03]: end it and then you think

[00:58:32] [SPEAKER_03]: you just like did the

[00:58:34] [SPEAKER_03]: e-commerce equivalent of

[00:58:35] [SPEAKER_03]: curing cancer with your

[00:58:36] [SPEAKER_03]: test because it's 80%

[00:58:38] [SPEAKER_03]: improvement.

[00:58:38] [SPEAKER_03]: But like that's not

[00:58:39] [SPEAKER_03]: actually the case.

[00:58:40] [SPEAKER_03]: So yeah, I would say

[00:58:43] [SPEAKER_03]: that's the last piece on

[00:58:44] [SPEAKER_03]: that.

[00:58:44] [SPEAKER_03]: But I really like I don't

[00:58:45] [SPEAKER_03]: know.

[00:58:46] [SPEAKER_03]: I think that the the next

[00:58:49] [SPEAKER_03]: thing on the list we can skip

[00:58:51] [SPEAKER_03]: one of them, which is just

[00:58:52] [SPEAKER_03]: run your desktop and mobile

[00:58:53] [SPEAKER_03]: tests separately.

[00:58:54] [SPEAKER_03]: Like they're different

[00:58:54] [SPEAKER_03]: audiences.

[00:58:55] [SPEAKER_03]: You should treat them

[00:58:56] [SPEAKER_03]: differently.

[00:58:57] [SPEAKER_03]: Most people just run tests

[00:58:58] [SPEAKER_03]: for mobile because you're

[00:59:00] [SPEAKER_03]: 90% of your traffic if you're

[00:59:01] [SPEAKER_03]: a seven figure store, you're

[00:59:02] [SPEAKER_03]: not going to get results on

[00:59:03] [SPEAKER_03]: desktop that actually mean

[00:59:04] [SPEAKER_03]: anything.

[00:59:04] [SPEAKER_03]: But on the next one,

[00:59:06] [SPEAKER_03]: what real quick is like

[00:59:07] [SPEAKER_00]: also now when are you saying

[00:59:08] [SPEAKER_00]: because like obviously you

[00:59:09] [SPEAKER_00]: can go into any test that

[00:59:10] [SPEAKER_00]: you run and say, all right,

[00:59:12] [SPEAKER_00]: we ran this to both mobile

[00:59:13] [SPEAKER_00]: and desktop and I can

[00:59:14] [SPEAKER_00]: filter out by mobile and

[00:59:15] [SPEAKER_00]: desktop.

[00:59:15] [SPEAKER_00]: Are you basically saying that?

[00:59:16] [SPEAKER_00]: Yeah, like the actual

[00:59:18] [SPEAKER_00]: like user experience swings

[00:59:19] [SPEAKER_00]: that you're taking should be

[00:59:20] [SPEAKER_00]: different.

[00:59:21] [SPEAKER_00]: You shouldn't just like have

[00:59:21] [SPEAKER_00]: this end all user.

[00:59:23] [SPEAKER_03]: They should be different.

[00:59:24] [SPEAKER_03]: Yeah, they should be different.

[00:59:25] [SPEAKER_03]: But also you're just not

[00:59:26] [SPEAKER_03]: for most brands up until

[00:59:28] [SPEAKER_03]: like a certain size, you're

[00:59:29] [SPEAKER_03]: just not going to get

[00:59:30] [SPEAKER_03]: like for example, your desktop

[00:59:31] [SPEAKER_03]: test, you have to run that

[00:59:32] [SPEAKER_03]: test for four weeks to get

[00:59:34] [SPEAKER_03]: a significant result and

[00:59:35] [SPEAKER_03]: your hex clan.

[00:59:37] [SPEAKER_03]: Most people listening this

[00:59:37] [SPEAKER_03]: aren't hex glad they don't

[00:59:38] [SPEAKER_03]: have the volumes that

[00:59:39] [SPEAKER_03]: you're doing.

[00:59:40] [SPEAKER_03]: And so it's like they

[00:59:41] [SPEAKER_03]: probably can't actually

[00:59:42] [SPEAKER_03]: like it's not reasonable

[00:59:44] [SPEAKER_03]: for them to run a test on

[00:59:45] [SPEAKER_03]: desktop unless they're going to

[00:59:46] [SPEAKER_03]: run it for six, seven.

[00:59:47] [SPEAKER_03]: There's a mobile test

[00:59:48] [SPEAKER_00]: or just that like with

[00:59:49] [SPEAKER_00]: certain context where we're

[00:59:50] [SPEAKER_00]: doing things like you

[00:59:51] [SPEAKER_00]: mentioned where we're only

[00:59:52] [SPEAKER_00]: like we're using a trigger

[00:59:53] [SPEAKER_00]: URL that's only accessible

[00:59:55] [SPEAKER_00]: through our Facebook ads

[00:59:56] [SPEAKER_00]: and we have a super long

[00:59:58] [SPEAKER_00]: consideration period.

[00:59:59] [SPEAKER_00]: So a lot of people are

[01:00:00] [SPEAKER_00]: getting this test seeing it

[01:00:01] [SPEAKER_00]: not converting whatever

[01:00:03] [SPEAKER_00]: maybe converting later.

[01:00:04] [SPEAKER_00]: We've been running this

[01:00:05] [SPEAKER_00]: knives landing page test

[01:00:06] [SPEAKER_00]: for four weeks now

[01:00:07] [SPEAKER_00]: and we still only have 30

[01:00:08] [SPEAKER_00]: conversions per variant.

[01:00:09] [SPEAKER_00]: I'm like what do I do to

[01:00:10] [SPEAKER_00]: sit here and let this

[01:00:11] [SPEAKER_00]: thing run for for four months

[01:00:13] [SPEAKER_00]: like yeah, exactly.

[01:00:14] [SPEAKER_00]: I'm like, Jesus forever.

[01:00:16] [SPEAKER_00]: Yeah, it's like it's

[01:00:18] [SPEAKER_03]: it's hard to work on.

[01:00:19] [SPEAKER_03]: That's where I think you have

[01:00:21] [SPEAKER_03]: to make the decision on like

[01:00:22] [SPEAKER_03]: what is actually worth your

[01:00:23] [SPEAKER_03]: time because if you're only

[01:00:24] [SPEAKER_03]: getting that many conversions,

[01:00:25] [SPEAKER_03]: the impact on your revenue

[01:00:27] [SPEAKER_03]: is also going to be that little

[01:00:28] [SPEAKER_03]: like you know what I mean.

[01:00:29] [SPEAKER_03]: So like when you run experiments

[01:00:30] [SPEAKER_03]: that's where one of the

[01:00:31] [SPEAKER_03]: choosing what what tests to run

[01:00:33] [SPEAKER_03]: is like look at what

[01:00:34] [SPEAKER_03]: percentage of your traffic

[01:00:35] [SPEAKER_03]: is impacted by this experiment.

[01:00:37] [SPEAKER_03]: Like if it's on a singular

[01:00:39] [SPEAKER_03]: product page, it might

[01:00:40] [SPEAKER_03]: only be nine percent

[01:00:42] [SPEAKER_03]: of your traffic, even though

[01:00:43] [SPEAKER_03]: like a navigation thing is

[01:00:45] [SPEAKER_03]: like 100 percent of your traffic

[01:00:46] [SPEAKER_03]: because everybody sees it.

[01:00:47] [SPEAKER_03]: But on a specific PDP

[01:00:49] [SPEAKER_03]: that might only be nine

[01:00:50] [SPEAKER_03]: percent of your traffic.

[01:00:50] [SPEAKER_03]: So you're like, oh my God,

[01:00:51] [SPEAKER_03]: we increased conversion rate

[01:00:52] [SPEAKER_03]: for this bundle by

[01:00:54] [SPEAKER_03]: and it might be your core

[01:00:55] [SPEAKER_03]: bundle, whatever caveats set

[01:00:57] [SPEAKER_03]: aside, but like a 10 percent

[01:00:58] [SPEAKER_03]: improvement there is only a

[01:00:59] [SPEAKER_03]: 10 percent improvement

[01:01:00] [SPEAKER_03]: on 10 percent of your traffic.

[01:01:03] [SPEAKER_03]: So that's only a 1 percent.

[01:01:05] [SPEAKER_03]: Like you know what I mean?

[01:01:05] [SPEAKER_03]: Like when you start really

[01:01:06] [SPEAKER_03]: thinking through like how

[01:01:08] [SPEAKER_03]: crazy do you want to get

[01:01:09] [SPEAKER_03]: about trying to find,

[01:01:11] [SPEAKER_03]: you know, spend time on these things.

[01:01:13] [SPEAKER_00]: Those are the examples where

[01:01:15] [SPEAKER_00]: I think you do need to go

[01:01:16] [SPEAKER_00]: and dig into the data outside

[01:01:18] [SPEAKER_00]: of this revenue per visitor

[01:01:19] [SPEAKER_00]: because my my where my head

[01:01:21] [SPEAKER_00]: immediately goes then is OK,

[01:01:22] [SPEAKER_00]: we clearly don't have enough

[01:01:23] [SPEAKER_00]: conversions per variant

[01:01:25] [SPEAKER_00]: to conclude this test.

[01:01:26] [SPEAKER_00]: But is there any is there

[01:01:28] [SPEAKER_00]: any connection I could set?

[01:01:29] [SPEAKER_00]: I can make in saying, OK,

[01:01:33] [SPEAKER_00]: do folks that hit this

[01:01:34] [SPEAKER_00]: landing page ultimately

[01:01:36] [SPEAKER_00]: have a higher chance

[01:01:37] [SPEAKER_00]: of converting than over all

[01:01:39] [SPEAKER_00]: time because that first

[01:01:40] [SPEAKER_00]: touchpoint is just better.

[01:01:42] [SPEAKER_00]: And like that's when it's like,

[01:01:43] [SPEAKER_00]: how can I create that?

[01:01:44] [SPEAKER_00]: Like is there some sort of?

[01:01:46] [SPEAKER_00]: I don't know if you could, right?

[01:01:48] [SPEAKER_00]: Because if someone well,

[01:01:49] [SPEAKER_00]: that's where I think I think.

[01:01:50] [SPEAKER_03]: Yeah, I think Facebook is a better

[01:01:52] [SPEAKER_03]: avenue for testing sometimes

[01:01:53] [SPEAKER_03]: for things like that, because

[01:01:54] [SPEAKER_03]: you can see spend data is a little bit

[01:01:57] [SPEAKER_03]: like I feel like you see your

[01:01:59] [SPEAKER_03]: you can be a little bit more confident

[01:02:00] [SPEAKER_03]: in like Facebook being like, hey,

[01:02:02] [SPEAKER_03]: you can profitably acquire

[01:02:03] [SPEAKER_03]: a customer to this page,

[01:02:04] [SPEAKER_03]: but to this page you can

[01:02:06] [SPEAKER_03]: essentially and like that's a different

[01:02:08] [SPEAKER_03]: you might be able to get less

[01:02:09] [SPEAKER_03]: conversions to have Facebook

[01:02:10] [SPEAKER_03]: tell you from like an acquisition

[01:02:12] [SPEAKER_03]: perspective, hey, this isn't

[01:02:13] [SPEAKER_03]: like this page is just not working

[01:02:16] [SPEAKER_03]: to convert customers.

[01:02:17] [SPEAKER_03]: And that's a pretty good indicator.

[01:02:19] [SPEAKER_03]: Whereas if you tried to split

[01:02:20] [SPEAKER_03]: that traffic evenly,

[01:02:21] [SPEAKER_03]: like send traffic to one page

[01:02:22] [SPEAKER_03]: from Facebook and then split it,

[01:02:24] [SPEAKER_03]: you kind of have only

[01:02:26] [SPEAKER_03]: a certain amount of conversions.

[01:02:27] [SPEAKER_03]: And so it's kind of hard to like

[01:02:28] [SPEAKER_03]: there are different avenues

[01:02:30] [SPEAKER_03]: that you can use for testing

[01:02:31] [SPEAKER_03]: the same things from your

[01:02:32] [SPEAKER_00]: like either you set it up

[01:02:34] [SPEAKER_00]: post click like this

[01:02:35] [SPEAKER_00]: or I have to go in and say,

[01:02:36] [SPEAKER_00]: all right, I have three

[01:02:38] [SPEAKER_00]: separate assets all at the same

[01:02:40] [SPEAKER_00]: ad, all at the same ad spend

[01:02:41] [SPEAKER_00]: product page collection page

[01:02:43] [SPEAKER_00]: landing page.

[01:02:43] [SPEAKER_00]: So it's like either way, it's

[01:02:45] [SPEAKER_00]: right. Isn't it?

[01:02:46] [SPEAKER_00]: I mean, it's like either a front

[01:02:48] [SPEAKER_00]: loaded or or post click

[01:02:49] [SPEAKER_00]: back loaded in terms of where

[01:02:51] [SPEAKER_00]: you're splitting the traffic.

[01:02:52] [SPEAKER_00]: But yeah, I spend a

[01:02:53] [SPEAKER_03]: well, it's just that I think

[01:02:55] [SPEAKER_03]: that what you get when you do

[01:02:56] [SPEAKER_03]: it in Facebook is you get

[01:02:57] [SPEAKER_03]: the actual like the different

[01:02:59] [SPEAKER_03]: set like ad sets and like

[01:03:01] [SPEAKER_03]: their performance essentially,

[01:03:02] [SPEAKER_03]: which you don't get when you

[01:03:03] [SPEAKER_03]: split it when they get to

[01:03:04] [SPEAKER_03]: your site. Yeah, essentially.

[01:03:05] [SPEAKER_03]: Like when they get to your site

[01:03:07] [SPEAKER_03]: and then you split the traffic,

[01:03:08] [SPEAKER_03]: you don't get the difference

[01:03:09] [SPEAKER_03]: in like those actual like

[01:03:12] [SPEAKER_03]: in platform Facebook metrics

[01:03:14] [SPEAKER_03]: of like your return on ad spend

[01:03:15] [SPEAKER_03]: was because you're sending all

[01:03:16] [SPEAKER_03]: the spend to one

[01:03:18] [SPEAKER_03]: and then that's true.

[01:03:19] [SPEAKER_03]: People off to the other.

[01:03:20] [SPEAKER_03]: Yeah.

[01:03:21] [SPEAKER_03]: So that's where you can see

[01:03:22] [SPEAKER_03]: like in Facebook, you could see

[01:03:24] [SPEAKER_03]: like, hey, we tried sending

[01:03:25] [SPEAKER_03]: people this landing page

[01:03:26] [SPEAKER_03]: with these these ads

[01:03:27] [SPEAKER_03]: and like they're not converting

[01:03:28] [SPEAKER_03]: at all like, OK, you could

[01:03:29] [SPEAKER_03]: probably even if you just

[01:03:31] [SPEAKER_03]: sent other traffic via split

[01:03:33] [SPEAKER_03]: test, it's probably going to

[01:03:34] [SPEAKER_03]: show you a similar.

[01:03:35] [SPEAKER_00]: And in your MTA, you get a longer

[01:03:37] [SPEAKER_00]: like if someone doesn't convert

[01:03:39] [SPEAKER_00]: in this post click test,

[01:03:40] [SPEAKER_00]: like that data is never going to

[01:03:41] [SPEAKER_00]: show up in that test.

[01:03:42] [SPEAKER_00]: But if you set it up

[01:03:44] [SPEAKER_00]: at the Facebook level,

[01:03:45] [SPEAKER_00]: like I could go into North

[01:03:46] [SPEAKER_00]: Beam and be like, well,

[01:03:47] [SPEAKER_00]: show me 60 day clicks

[01:03:48] [SPEAKER_00]: and view show me lifetime

[01:03:50] [SPEAKER_00]: clicks and views.

[01:03:51] [SPEAKER_00]: And then you actually can see

[01:03:53] [SPEAKER_00]: that attribution over time,

[01:03:54] [SPEAKER_00]: even if they don't convert

[01:03:55] [SPEAKER_00]: like in that actual session

[01:03:57] [SPEAKER_00]: that they clicked on and saw

[01:03:59] [SPEAKER_00]: right after they saw the ad.

[01:04:01] [SPEAKER_00]: So yeah, I guess you do get

[01:04:02] [SPEAKER_00]: you get that longer tail,

[01:04:03] [SPEAKER_00]: which is it's a good point.

[01:04:07] [SPEAKER_02]: So one of the buzzwords this year

[01:04:09] [SPEAKER_02]: is income mentality last year

[01:04:10] [SPEAKER_02]: is all about attributions.

[01:04:11] [SPEAKER_02]: Here's all about income

[01:04:12] [SPEAKER_02]: mentality, and I think that's a

[01:04:13] [SPEAKER_02]: great thing because we are all

[01:04:14] [SPEAKER_02]: trying to grow our businesses

[01:04:16] [SPEAKER_02]: profitably and we want to make

[01:04:17] [SPEAKER_02]: sure that the spend

[01:04:18] [SPEAKER_02]: that we are putting out into

[01:04:19] [SPEAKER_02]: market is actually

[01:04:20] [SPEAKER_02]: is working for us.

[01:04:22] [SPEAKER_02]: So I know that we are all

[01:04:23] [SPEAKER_02]: house customers and you guys

[01:04:25] [SPEAKER_02]: have sought out houses here.

[01:04:26] [SPEAKER_02]: What what led you guys to

[01:04:27] [SPEAKER_02]: to do that?

[01:04:29] [SPEAKER_01]: We'll start with Ridge.

[01:04:30] [SPEAKER_01]: For us, it's all about new

[01:04:31] [SPEAKER_01]: channels where we're in the

[01:04:32] [SPEAKER_01]: business of reaching net

[01:04:33] [SPEAKER_01]: new male audiences.

[01:04:34] [SPEAKER_01]: So when we're launching

[01:04:35] [SPEAKER_01]: across Pinterest, Twitter,

[01:04:36] [SPEAKER_01]: Snapchat, Live Intent,

[01:04:38] [SPEAKER_01]: et cetera, we use house to measure

[01:04:40] [SPEAKER_01]: the total impact of those new

[01:04:42] [SPEAKER_01]: channel experiments.

[01:04:43] [SPEAKER_01]: That's awesome. How about

[01:04:44] [SPEAKER_01]: Hexclad?

[01:04:44] [SPEAKER_02]: What are you guys looking at?

[01:04:45] [SPEAKER_00]: Definitely the new channels as

[01:04:47] [SPEAKER_00]: we roll those out.

[01:04:48] [SPEAKER_00]: But because we already had such

[01:04:50] [SPEAKER_00]: a diversified marketing mix

[01:04:52] [SPEAKER_00]: in terms of channels and we

[01:04:53] [SPEAKER_00]: are already spending at pretty

[01:04:54] [SPEAKER_00]: high levels prior to

[01:04:55] [SPEAKER_00]: onboarding house, the first

[01:04:57] [SPEAKER_00]: batch of testing is just for us

[01:04:58] [SPEAKER_00]: to find those optimal spend

[01:05:00] [SPEAKER_00]: levels with our existing

[01:05:02] [SPEAKER_00]: marketing stack and then

[01:05:03] [SPEAKER_00]: comparing those to one another

[01:05:04] [SPEAKER_00]: to kind of see within our

[01:05:05] [SPEAKER_00]: existing marketing stack.

[01:05:06] [SPEAKER_00]: Where should we bring spend

[01:05:07] [SPEAKER_00]: down? Where should we bring

[01:05:09] [SPEAKER_00]: spend up and really using that

[01:05:11] [SPEAKER_00]: incremental ROAS, that

[01:05:12] [SPEAKER_00]: incremental CAC number to

[01:05:15] [SPEAKER_00]: inform how we move those

[01:05:16] [SPEAKER_00]: budgets up and down?

[01:05:17] [SPEAKER_02]: I love that. Yeah.

[01:05:18] [SPEAKER_02]: And for us the same, we've

[01:05:19] [SPEAKER_02]: definitely tested new channels,

[01:05:20] [SPEAKER_02]: but we've also calibrated spend

[01:05:22] [SPEAKER_02]: on on existing channels.

[01:05:24] [SPEAKER_02]: We've we've saved six figures

[01:05:26] [SPEAKER_02]: a month just by not running

[01:05:27] [SPEAKER_02]: brand search because we've ran

[01:05:28] [SPEAKER_02]: multiple tests and it's not

[01:05:29] [SPEAKER_02]: incremental.

[01:05:30] [SPEAKER_02]: So I know for all of us,

[01:05:31] [SPEAKER_02]: it's had a huge impact and

[01:05:32] [SPEAKER_02]: we love it and we couldn't

[01:05:34] [SPEAKER_02]: live with that at this point.

[01:05:34] [SPEAKER_02]: And so if you want to be like us

[01:05:36] [SPEAKER_02]: and want to add scientific rigor

[01:05:37] [SPEAKER_02]: to your marketing stack, I

[01:05:39] [SPEAKER_02]: recommend you go to house.io

[01:05:41] [SPEAKER_02]: slash operators.

[01:05:42] [SPEAKER_02]: That's h a u s dot i o slash

[01:05:45] [SPEAKER_02]: operators to get a demo today.

[01:05:49] [SPEAKER_01]: I just worry you introduce like

[01:05:51] [SPEAKER_01]: so many more variables.

[01:05:53] [SPEAKER_01]: Like all of this and you have

[01:05:54] [SPEAKER_01]: that you'd add sets like one

[01:05:56] [SPEAKER_01]: ad sets gets a couple

[01:05:56] [SPEAKER_01]: conversions earlier.

[01:05:57] [SPEAKER_01]: Like is it optimizing better?

[01:05:59] [SPEAKER_01]: It's like a something at that

[01:06:00] [SPEAKER_01]: point. So we love like we

[01:06:02] [SPEAKER_01]: prefer doing it all post click.

[01:06:04] [SPEAKER_01]: So we'd like set it up based

[01:06:05] [SPEAKER_01]: on UTMs just run it in

[01:06:07] [SPEAKER_01]: intelligence.

[01:06:08] [SPEAKER_01]: And then you're like you're

[01:06:09] [SPEAKER_01]: forcing all the traffic

[01:06:11] [SPEAKER_01]: through like whatever ad sets

[01:06:12] [SPEAKER_01]: you've set up in the account.

[01:06:14] [SPEAKER_03]: Yeah, no, I think that's the

[01:06:15] [SPEAKER_03]: that's the way to do it.

[01:06:16] [SPEAKER_03]: But when looking at these small

[01:06:18] [SPEAKER_03]: like data sets, essentially

[01:06:21] [SPEAKER_03]: like the hey this for example

[01:06:22] [SPEAKER_03]: like this 30 conversions on

[01:06:24] [SPEAKER_03]: each thing over the course of

[01:06:25] [SPEAKER_03]: weeks, like you might want

[01:06:27] [SPEAKER_03]: to look at other if you

[01:06:27] [SPEAKER_03]: really want to learn that

[01:06:29] [SPEAKER_03]: you might need to look at

[01:06:30] [SPEAKER_03]: other avenues to learn that

[01:06:31] [SPEAKER_03]: because otherwise like you're

[01:06:33] [SPEAKER_03]: not going to get it through

[01:06:33] [SPEAKER_03]: that post click like

[01:06:36] [SPEAKER_03]: testing and Conor do you have a

[01:06:37] [SPEAKER_03]: sense of but is it even

[01:06:39] [SPEAKER_03]: worth it?

[01:06:39] [SPEAKER_00]: Do you have a sense of like

[01:06:40] [SPEAKER_00]: your like roughly your average

[01:06:42] [SPEAKER_00]: like from like awareness to

[01:06:45] [SPEAKER_00]: conversion path

[01:06:46] [SPEAKER_00]: because like the brands that

[01:06:47] [SPEAKER_00]: have quicker shorter

[01:06:49] [SPEAKER_00]: windows, they're certainly at

[01:06:51] [SPEAKER_00]: an advantage when it comes to

[01:06:52] [SPEAKER_00]: having these tests get more

[01:06:53] [SPEAKER_00]: conversions and hit stats

[01:06:55] [SPEAKER_00]: quick quicker.

[01:06:56] [SPEAKER_00]: Whereas like we are like

[01:06:58] [SPEAKER_00]: the worst like for that

[01:07:01] [SPEAKER_00]: it's like we're at the

[01:07:02] [SPEAKER_00]: total opposite end of the

[01:07:03] [SPEAKER_00]: spectrum like we are just

[01:07:04] [SPEAKER_00]: not an impulse by one day,

[01:07:06] [SPEAKER_00]: seven days are least common

[01:07:07] [SPEAKER_00]: consideration periods.

[01:07:09] [SPEAKER_00]: It just makes it much

[01:07:10] [SPEAKER_00]: harder I think.

[01:07:12] [SPEAKER_01]: Yeah.

[01:07:13] [SPEAKER_01]: No we have I mean we have a

[01:07:14] [SPEAKER_01]: pretty significant

[01:07:14] [SPEAKER_01]: consideration period.

[01:07:15] [SPEAKER_01]: What our customers tell us is

[01:07:16] [SPEAKER_01]: 40% of people have known about

[01:07:18] [SPEAKER_01]: us for 30 days once they

[01:07:19] [SPEAKER_01]: purchased.

[01:07:20] [SPEAKER_01]: Lot and we see that like

[01:07:22] [SPEAKER_01]: I mean when we go on promo

[01:07:23] [SPEAKER_01]: we see like the prospect

[01:07:25] [SPEAKER_01]: list convert things like that

[01:07:27] [SPEAKER_01]: like there's a there is

[01:07:28] [SPEAKER_01]: a certain type of customer

[01:07:29] [SPEAKER_01]: who takes a while.

[01:07:30] [SPEAKER_01]: That being said it's like

[01:07:31] [SPEAKER_01]: a hundred dollar AOV like

[01:07:32] [SPEAKER_01]: when we're doing CRO tests

[01:07:35] [SPEAKER_01]: or we're looking at something

[01:07:36] [SPEAKER_01]: like conversion rate

[01:07:36] [SPEAKER_01]: we're looking at it on a very

[01:07:38] [SPEAKER_00]: short time.

[01:07:38] [SPEAKER_00]: Yeah.

[01:07:39] [SPEAKER_00]: Which is also a problem

[01:07:40] [SPEAKER_00]: because now you're only getting

[01:07:41] [SPEAKER_00]: data from like a very certain

[01:07:43] [SPEAKER_00]: type of customer

[01:07:44] [SPEAKER_00]: like your your like

[01:07:45] [SPEAKER_00]: we're getting this data from

[01:07:47] [SPEAKER_00]: our least common customer type

[01:07:48] [SPEAKER_00]: which is concerning

[01:07:50] [SPEAKER_00]: like the people that yeah

[01:07:52] [SPEAKER_00]: it's so I don't know

[01:07:54] [SPEAKER_00]: it's there.

[01:07:55] [SPEAKER_00]: You have to take everything

[01:07:56] [SPEAKER_01]: it's all the way down.

[01:07:57] [SPEAKER_01]: Yeah.

[01:07:59] [SPEAKER_01]: Yeah.

[01:08:00] [SPEAKER_03]: Which is why again

[01:08:01] [SPEAKER_03]: I think this is why

[01:08:02] [SPEAKER_03]: like you you start having

[01:08:03] [SPEAKER_03]: these conversations

[01:08:03] [SPEAKER_03]: and you start getting into

[01:08:05] [SPEAKER_03]: reality and you're like

[01:08:05] [SPEAKER_03]: oh yeah maybe like

[01:08:06] [SPEAKER_03]: these testing programs

[01:08:08] [SPEAKER_03]: are not where we should be

[01:08:09] [SPEAKER_03]: like trying to drive

[01:08:10] [SPEAKER_03]: massive growth.

[01:08:11] [SPEAKER_03]: Like I think we all

[01:08:12] [SPEAKER_03]: understand that this isn't

[01:08:13] [SPEAKER_03]: like a massive lever for growth

[01:08:14] [SPEAKER_03]: but it's like

[01:08:15] [SPEAKER_03]: there's so many caveats

[01:08:17] [SPEAKER_03]: and stuff like that

[01:08:17] [SPEAKER_03]: that I just go back to like

[01:08:19] [SPEAKER_03]: hey are we actually getting

[01:08:20] [SPEAKER_03]: like are we at least

[01:08:21] [SPEAKER_03]: confident that things are

[01:08:22] [SPEAKER_03]: getting better every month.

[01:08:23] [SPEAKER_03]: Yeah.

[01:08:23] [SPEAKER_03]: You know like I was going

[01:08:24] [SPEAKER_03]: to make a point too.

[01:08:25] [SPEAKER_03]: Yeah.

[01:08:26] [SPEAKER_01]: Yeah.

[01:08:26] [SPEAKER_01]: Yeah.

[01:08:26] [SPEAKER_01]: Yeah.

[01:08:27] [SPEAKER_01]: As a you need tools

[01:08:29] [SPEAKER_01]: to improve

[01:08:31] [SPEAKER_01]: like you need tools to get

[01:08:32] [SPEAKER_01]: better and that's what this is

[01:08:33] [SPEAKER_01]: and it's not necessarily

[01:08:34] [SPEAKER_01]: as you're not going to get

[01:08:35] [SPEAKER_01]: the clarity or the certainty

[01:08:36] [SPEAKER_01]: to say we've gotten 40%

[01:08:38] [SPEAKER_01]: better with this test

[01:08:40] [SPEAKER_01]: that we just ran over

[01:08:41] [SPEAKER_01]: the last three and a half days.

[01:08:42] [SPEAKER_03]: Yeah.

[01:08:43] [SPEAKER_03]: And I also I think

[01:08:44] [SPEAKER_03]: that leads into like

[01:08:44] [SPEAKER_03]: I just want to say

[01:08:45] [SPEAKER_03]: we can talk about things

[01:08:46] [SPEAKER_03]: that like you shouldn't do.

[01:08:47] [SPEAKER_03]: There's one other piece

[01:08:48] [SPEAKER_03]: I wanted to mention

[01:08:49] [SPEAKER_03]: but I think one thing

[01:08:49] [SPEAKER_03]: with interpreting results

[01:08:51] [SPEAKER_03]: to what you just said

[01:08:52] [SPEAKER_03]: of like hey we got

[01:08:52] [SPEAKER_03]: this 40% improvement.

[01:08:54] [SPEAKER_03]: I think the main thing

[01:08:55] [SPEAKER_03]: with interpreting results

[01:08:56] [SPEAKER_03]: that I think most people

[01:08:57] [SPEAKER_03]: should understand

[01:08:58] [SPEAKER_03]: is that when people say

[01:08:59] [SPEAKER_03]: that a test will generate

[01:09:02] [SPEAKER_03]: a certain amount of incremental

[01:09:03] [SPEAKER_03]: revenue into the future

[01:09:04] [SPEAKER_03]: it just makes absolutely

[01:09:06] [SPEAKER_03]: no sense.

[01:09:07] [SPEAKER_03]: Like it will it just

[01:09:09] [SPEAKER_03]: when they say oh we

[01:09:10] [SPEAKER_03]: like this test generated

[01:09:11] [SPEAKER_03]: an incremental

[01:09:12] [SPEAKER_03]: $40,000 a month

[01:09:14] [SPEAKER_03]: in sales for this brand.

[01:09:15] [SPEAKER_03]: It's like you never see

[01:09:17] [SPEAKER_03]: the $40,000

[01:09:18] [SPEAKER_03]: in incremental sales

[01:09:19] [SPEAKER_03]: in the month.

[01:09:20] [SPEAKER_03]: Like I don't know anybody

[01:09:21] [SPEAKER_03]: who's been like oh yeah

[01:09:22] [SPEAKER_03]: you know what we saw

[01:09:23] [SPEAKER_03]: a massive revenue spike

[01:09:24] [SPEAKER_03]: on the back of this

[01:09:25] [SPEAKER_03]: one experiment

[01:09:26] [SPEAKER_03]: that's not what happens.

[01:09:28] [SPEAKER_03]: And so the reason

[01:09:29] [SPEAKER_03]: the reason why is because

[01:09:30] [SPEAKER_03]: what you're actually seeing

[01:09:32] [SPEAKER_03]: with that like conversion rate

[01:09:33] [SPEAKER_03]: increase that's 4%

[01:09:35] [SPEAKER_03]: you're seeing is

[01:09:37] [SPEAKER_03]: that is the midpoint

[01:09:38] [SPEAKER_03]: in the distribution

[01:09:39] [SPEAKER_03]: of conversion rates essentially.

[01:09:41] [SPEAKER_03]: So 50% of people are below that

[01:09:44] [SPEAKER_03]: and 50% of people are above that.

[01:09:46] [SPEAKER_03]: And so the actual

[01:09:47] [SPEAKER_03]: thing that you're being

[01:09:49] [SPEAKER_03]: confident in is probably

[01:09:51] [SPEAKER_03]: like that distribution

[01:09:52] [SPEAKER_03]: does go over a negative

[01:09:54] [SPEAKER_03]: conversion rate.

[01:09:55] [SPEAKER_03]: Like there is still

[01:09:56] [SPEAKER_03]: chances that that's negative.

[01:09:57] [SPEAKER_03]: So there's a

[01:09:58] [SPEAKER_03]: like let's say at that 4%

[01:10:00] [SPEAKER_03]: there's probably a 75% chance

[01:10:02] [SPEAKER_03]: the change is within 2%

[01:10:04] [SPEAKER_03]: and 6%.

[01:10:05] [SPEAKER_03]: And so hey

[01:10:07] [SPEAKER_03]: like you keep running the test

[01:10:08] [SPEAKER_03]: you might get more confident

[01:10:09] [SPEAKER_03]: as all those bands start to come in

[01:10:11] [SPEAKER_03]: like you know converge

[01:10:12] [SPEAKER_03]: but the reality is

[01:10:14] [SPEAKER_03]: that 8% you're seeing

[01:10:16] [SPEAKER_03]: is not 8%

[01:10:17] [SPEAKER_03]: like you're not 100% confident

[01:10:19] [SPEAKER_03]: you got an 8% result.

[01:10:20] [SPEAKER_03]: There's a good chance

[01:10:21] [SPEAKER_03]: it's 0 to 8%

[01:10:23] [SPEAKER_03]: and it might even be 2%

[01:10:24] [SPEAKER_03]: which is why the whole like

[01:10:26] [SPEAKER_03]: yeah we're getting this major

[01:10:27] [SPEAKER_03]: like incremental revenue boost

[01:10:28] [SPEAKER_03]: or we're getting like

[01:10:29] [SPEAKER_03]: we're using this as like a lever

[01:10:30] [SPEAKER_03]: for growth.

[01:10:32] [SPEAKER_03]: It's just

[01:10:33] [SPEAKER_03]: there's so many caveats

[01:10:35] [SPEAKER_03]: within that where I feel

[01:10:36] [SPEAKER_03]: like people should just be aware

[01:10:37] [SPEAKER_03]: of like the ROI

[01:10:38] [SPEAKER_03]: you're getting on your testing program

[01:10:40] [SPEAKER_03]: is not always

[01:10:43] [SPEAKER_03]: as great as it seems

[01:10:45] [SPEAKER_03]: when using that sort of a way

[01:10:47] [SPEAKER_03]: of attributing

[01:10:48] [SPEAKER_03]: you know success

[01:10:50] [SPEAKER_03]: or measuring success.

[01:10:51] [SPEAKER_01]: I have a question.

[01:10:54] [SPEAKER_01]: So we talk about

[01:10:56] [SPEAKER_01]: I'm trying to think like

[01:10:56] [SPEAKER_01]: what do I want to take away

[01:10:57] [SPEAKER_01]: from this generally?

[01:10:58] [SPEAKER_01]: And I and I overall like the approach

[01:11:00] [SPEAKER_01]: of like hey what we're trying to do

[01:11:01] [SPEAKER_01]: is get better

[01:11:01] [SPEAKER_01]: and these are tools

[01:11:02] [SPEAKER_01]: in order to do that

[01:11:03] [SPEAKER_01]: and like there's just a lot of caveats

[01:11:05] [SPEAKER_01]: and a lack of clarity

[01:11:06] [SPEAKER_01]: and yeah

[01:11:08] [SPEAKER_01]: largely statistical

[01:11:09] [SPEAKER_01]: insignificance.

[01:11:11] [SPEAKER_01]: But when it comes to

[01:11:13] [SPEAKER_01]: getting like improving your site

[01:11:15] [SPEAKER_01]: making it better

[01:11:16] [SPEAKER_01]: providing a better experience

[01:11:17] [SPEAKER_01]: for customers.

[01:11:18] [SPEAKER_01]: Do you see any parts

[01:11:19] [SPEAKER_01]: that brands undervalue

[01:11:22] [SPEAKER_01]: like often overlook?

[01:11:24] [SPEAKER_03]: Yeah, I think

[01:11:25] [SPEAKER_03]: the biggest one is probably merchandising.

[01:11:28] [SPEAKER_03]: Like I think the one of the

[01:11:29] [SPEAKER_03]: it's just like

[01:11:30] [SPEAKER_03]: I mean obviously physical

[01:11:31] [SPEAKER_03]: and digital is different.

[01:11:32] [SPEAKER_03]: But like when you go into a store

[01:11:34] [SPEAKER_03]: you kind of like what products

[01:11:35] [SPEAKER_03]: they actually have

[01:11:36] [SPEAKER_03]: and how they present them

[01:11:37] [SPEAKER_03]: is like a big thing

[01:11:39] [SPEAKER_03]: that will impact

[01:11:39] [SPEAKER_03]: what you ultimately purchase

[01:11:41] [SPEAKER_03]: and like having a strategy

[01:11:42] [SPEAKER_03]: around how you show

[01:11:44] [SPEAKER_03]: people specific products, etc.

[01:11:45] [SPEAKER_03]: Like showing people for example

[01:11:47] [SPEAKER_03]: like when they land on your site

[01:11:48] [SPEAKER_03]: if the main thing

[01:11:49] [SPEAKER_03]: you're pushing them to

[01:11:50] [SPEAKER_03]: is a singular product

[01:11:51] [SPEAKER_03]: versus if the main thing

[01:11:52] [SPEAKER_03]: you're pushing them to

[01:11:53] [SPEAKER_03]: is a bundle of products

[01:11:54] [SPEAKER_03]: is going to have a pretty big difference on

[01:11:56] [SPEAKER_03]: because most people are going to see

[01:11:57] [SPEAKER_03]: the one that you push them to

[01:11:59] [SPEAKER_03]: so you can kind of guide

[01:12:00] [SPEAKER_03]: people's journey

[01:12:01] [SPEAKER_03]: and you could try to get them

[01:12:03] [SPEAKER_03]: to click on the things

[01:12:04] [SPEAKER_03]: that are maybe higher AOV

[01:12:05] [SPEAKER_03]: or bring more

[01:12:06] [SPEAKER_03]: higher contribution dollars

[01:12:08] [SPEAKER_03]: like per session to your store.

[01:12:11] [SPEAKER_03]: And so much of what we're trying

[01:12:12] [SPEAKER_03]: to do with CRO

[01:12:13] [SPEAKER_03]: is just do better merchandising

[01:12:15] [SPEAKER_03]: if we're trying to get

[01:12:15] [SPEAKER_03]: people to the right products

[01:12:18] [SPEAKER_03]: or the products that they want.

[01:12:19] [SPEAKER_03]: One example is

[01:12:21] [SPEAKER_03]: is kind of like a

[01:12:21] [SPEAKER_03]: example is

[01:12:23] [SPEAKER_03]: there

[01:12:23] [SPEAKER_03]: I just spoke with someone recently

[01:12:25] [SPEAKER_03]: there's a brand called

[01:12:25] [SPEAKER_03]: Paragon Fit where they sell

[01:12:27] [SPEAKER_03]: like shapewear leggings

[01:12:29] [SPEAKER_03]: and we were having a chat

[01:12:30] [SPEAKER_03]: and they told me that

[01:12:31] [SPEAKER_03]: a couple of years ago

[01:12:33] [SPEAKER_03]: they tried to release

[01:12:34] [SPEAKER_03]: like loungewear

[01:12:35] [SPEAKER_03]: and they had

[01:12:36] [SPEAKER_03]: like their whole thing was like fit

[01:12:38] [SPEAKER_03]: like very fit women

[01:12:39] [SPEAKER_03]: you know selling like

[01:12:40] [SPEAKER_03]: shapewear leggings

[01:12:41] [SPEAKER_03]: and they tried to release this loungewear

[01:12:43] [SPEAKER_03]: and they had some models

[01:12:44] [SPEAKER_03]: that were like not in their core demo

[01:12:46] [SPEAKER_03]: and they tried to expand

[01:12:47] [SPEAKER_03]: their product categories essentially

[01:12:49] [SPEAKER_03]: and their conversion rate tanked

[01:12:52] [SPEAKER_03]: all they did was

[01:12:53] [SPEAKER_03]: add the loungewear to their site

[01:12:55] [SPEAKER_03]: essentially like and their

[01:12:56] [SPEAKER_03]: their return on their ad

[01:12:58] [SPEAKER_03]: their evergreen ads

[01:12:59] [SPEAKER_03]: that they were running to their shapewear

[01:13:00] [SPEAKER_03]: dropped because as people were exploring

[01:13:02] [SPEAKER_03]: like oh maybe this brand

[01:13:04] [SPEAKER_03]: isn't for me like they're selling these products

[01:13:05] [SPEAKER_03]: it's kind of like

[01:13:06] [SPEAKER_03]: if HexGuy just started selling

[01:13:08] [SPEAKER_03]: or Ridge just started selling

[01:13:09] [SPEAKER_03]: random like obscure

[01:13:11] [SPEAKER_03]: like I don't know

[01:13:12] [SPEAKER_03]: maybe you tell

[01:13:13] [SPEAKER_03]: yeah like a deodorant exactly

[01:13:15] [SPEAKER_03]: and then they're like

[01:13:16] [SPEAKER_03]: why the hell are you selling

[01:13:17] [SPEAKER_03]: deodorant? It's just weird

[01:13:18] [SPEAKER_03]: like it throws you off like is this

[01:13:19] [SPEAKER_03]: is this a marketplace

[01:13:20] [SPEAKER_03]: or is this a brand

[01:13:21] [SPEAKER_03]: like you start getting all these

[01:13:22] [SPEAKER_03]: weird things popping into your head

[01:13:24] [SPEAKER_03]: and so I think

[01:13:25] [SPEAKER_03]: making those decisions

[01:13:26] [SPEAKER_03]: and the one other example

[01:13:28] [SPEAKER_03]: I'll give is of a brand

[01:13:29] [SPEAKER_03]: who is selling a product where

[01:13:31] [SPEAKER_03]: it is

[01:13:32] [SPEAKER_03]: it performs so much worse

[01:13:34] [SPEAKER_03]: that like when you remove

[01:13:36] [SPEAKER_03]: that products from the site entirely

[01:13:38] [SPEAKER_03]: and you only have the one hero product

[01:13:40] [SPEAKER_03]: like all the metrics

[01:13:41] [SPEAKER_03]: just get better because having

[01:13:43] [SPEAKER_03]: even just adding one additional product

[01:13:45] [SPEAKER_03]: it increases confusion

[01:13:46] [SPEAKER_03]: and if that product doesn't work

[01:13:48] [SPEAKER_03]: it's like

[01:13:49] [SPEAKER_03]: it's really bad for

[01:13:51] [SPEAKER_03]: confusion rates overall

[01:13:52] [SPEAKER_03]: and so the actual decision of what

[01:13:54] [SPEAKER_03]: like this goes all the way down

[01:13:55] [SPEAKER_03]: to like the product development

[01:13:56] [SPEAKER_03]: sort of side of things

[01:13:57] [SPEAKER_03]: like you can't help it sometimes

[01:13:58] [SPEAKER_03]: but making some products

[01:14:00] [SPEAKER_03]: less apparent or more apparent

[01:14:02] [SPEAKER_03]: or you know putting things

[01:14:03] [SPEAKER_03]: in certain places

[01:14:05] [SPEAKER_03]: you know that's the sort of thing

[01:14:06] [SPEAKER_03]: that I think tends to have

[01:14:07] [SPEAKER_03]: a pretty big impact

[01:14:09] [SPEAKER_03]: on sales

[01:14:10] [SPEAKER_03]: that a lot of people just like

[01:14:11] [SPEAKER_03]: they have a collection

[01:14:12] [SPEAKER_03]: and they haven't even changed

[01:14:13] [SPEAKER_03]: how it sorts

[01:14:14] [SPEAKER_03]: like they just sort by like

[01:14:15] [SPEAKER_03]: you know whatever

[01:14:17] [SPEAKER_03]: like featured

[01:14:17] [SPEAKER_03]: or however they made the collection

[01:14:19] [SPEAKER_03]: it's just random

[01:14:20] [SPEAKER_03]: and they haven't even thought

[01:14:21] [SPEAKER_03]: about maybe we should have our best

[01:14:23] [SPEAKER_03]: sellers or maybe we should have our

[01:14:24] [SPEAKER_03]: most expensive items

[01:14:25] [SPEAKER_03]: or maybe like those sorts of things

[01:14:26] [SPEAKER_03]: do have an impact ultimately

[01:14:29] [SPEAKER_03]: on conversion rates

[01:14:30] [SPEAKER_03]: and on the amount of money

[01:14:31] [SPEAKER_03]: that you'll make

[01:14:31] [SPEAKER_01]: totally

[01:14:32] [SPEAKER_01]: but even that like

[01:14:33] [SPEAKER_01]: OK so we have like a

[01:14:35] [SPEAKER_01]: collection page full of wallets

[01:14:37] [SPEAKER_01]: like A.B. testing

[01:14:39] [SPEAKER_01]: well what do we want

[01:14:40] [SPEAKER_01]: towards the top or the bottom

[01:14:41] [SPEAKER_01]: is surprisingly difficult

[01:14:42] [SPEAKER_04]: yeah it is

[01:14:43] [SPEAKER_04]: yeah

[01:14:44] [SPEAKER_01]: it just comes down to like

[01:14:46] [SPEAKER_01]: I like

[01:14:47] [SPEAKER_01]: we make decisions

[01:14:48] [SPEAKER_01]: like that all the time

[01:14:48] [SPEAKER_01]: and it comes down to like

[01:14:50] [SPEAKER_01]: well what do we feel better about

[01:14:51] [SPEAKER_01]: like or we're duplicating

[01:14:53] [SPEAKER_01]: the collections page

[01:14:54] [SPEAKER_01]: I guess you could do that

[01:14:55] [SPEAKER_01]: you could duplicate the collections page

[01:14:56] [SPEAKER_01]: you could reorganize

[01:14:57] [SPEAKER_01]: you could A.B.

[01:14:57] [SPEAKER_01]: test it that way

[01:14:58] [SPEAKER_01]: or something

[01:14:58] [SPEAKER_01]: but

[01:15:00] [SPEAKER_01]: yeah a lot of these things

[01:15:01] [SPEAKER_01]: just end up feeling

[01:15:03] [SPEAKER_01]: like surprisingly tricky

[01:15:05] [SPEAKER_03]: yeah they're tricky

[01:15:06] [SPEAKER_03]: and then at the end

[01:15:07] [SPEAKER_03]: you might get a result

[01:15:07] [SPEAKER_03]: like what Connor is dealing with

[01:15:09] [SPEAKER_03]: which is like

[01:15:09] [SPEAKER_03]: hey we don't really know

[01:15:10] [SPEAKER_03]: like which way this should go

[01:15:11] [SPEAKER_03]: and then you're like

[01:15:12] [SPEAKER_03]: well where do we go from here

[01:15:13] [SPEAKER_03]: and the result is like

[01:15:14] [SPEAKER_03]: I think this is why

[01:15:16] [SPEAKER_03]: I think most tests

[01:15:17] [SPEAKER_03]: when you're building them

[01:15:18] [SPEAKER_03]: like you should have a

[01:15:19] [SPEAKER_03]: thesis behind like

[01:15:21] [SPEAKER_03]: why you're making

[01:15:22] [SPEAKER_03]: like

[01:15:22] [SPEAKER_03]: why are you changing

[01:15:24] [SPEAKER_03]: the collection page

[01:15:25] [SPEAKER_03]: to look this way

[01:15:25] [SPEAKER_03]: like I think a lot of people

[01:15:26] [SPEAKER_03]: go into tests without

[01:15:27] [SPEAKER_03]: not saying either of you do

[01:15:29] [SPEAKER_03]: but a lot of people go

[01:15:29] [SPEAKER_03]: and test with just a general

[01:15:31] [SPEAKER_03]: like I think this is better

[01:15:32] [SPEAKER_03]: but it's like

[01:15:33] [SPEAKER_03]: what actually

[01:15:35] [SPEAKER_03]: makes this better

[01:15:36] [SPEAKER_03]: for example

[01:15:37] [SPEAKER_03]: like hey we change

[01:15:39] [SPEAKER_03]: our collection layout

[01:15:40] [SPEAKER_03]: and so we have these

[01:15:41] [SPEAKER_03]: like offers at the top

[01:15:42] [SPEAKER_03]: that are like our best

[01:15:44] [SPEAKER_03]: performing while it's like

[01:15:45] [SPEAKER_03]: these are the ones

[01:15:45] [SPEAKER_03]: that are actually our best selling

[01:15:47] [SPEAKER_03]: or really just like

[01:15:48] [SPEAKER_03]: I don't know

[01:15:49] [SPEAKER_03]: we have these collaborations

[01:15:50] [SPEAKER_03]: or we have these offers

[01:15:50] [SPEAKER_03]: etc.

[01:15:51] [SPEAKER_03]: And we put those at the top

[01:15:52] [SPEAKER_03]: because of X Y and Z

[01:15:53] [SPEAKER_03]: rather than like

[01:15:55] [SPEAKER_03]: I don't know

[01:15:56] [SPEAKER_03]: let's just we think that

[01:15:57] [SPEAKER_03]: these wallets are better

[01:15:58] [SPEAKER_03]: or something like that

[01:15:59] [SPEAKER_03]: like that's not the best example

[01:16:01] [SPEAKER_03]: but just like

[01:16:02] [SPEAKER_03]: the better example

[01:16:04] [SPEAKER_03]: like a give is like

[01:16:04] [SPEAKER_03]: when making changes on

[01:16:05] [SPEAKER_03]: for example

[01:16:06] [SPEAKER_03]: home page

[01:16:06] [SPEAKER_03]: if you're going to change

[01:16:07] [SPEAKER_03]: hey we should update

[01:16:09] [SPEAKER_03]: our copy on our home page

[01:16:10] [SPEAKER_03]: and it's like

[01:16:11] [SPEAKER_03]: well

[01:16:12] [SPEAKER_03]: why are like

[01:16:13] [SPEAKER_03]: what is the actual

[01:16:14] [SPEAKER_03]: underlying thesis

[01:16:14] [SPEAKER_03]: for updating

[01:16:15] [SPEAKER_03]: right

[01:16:15] [SPEAKER_03]: if you're just like

[01:16:16] [SPEAKER_03]: we want it to be better

[01:16:17] [SPEAKER_03]: it's like okay

[01:16:17] [SPEAKER_03]: but like who do you

[01:16:18] [SPEAKER_03]: want to speak to

[01:16:19] [SPEAKER_03]: like the idea could be

[01:16:20] [SPEAKER_03]: hey we have

[01:16:21] [SPEAKER_03]: you know

[01:16:22] [SPEAKER_03]: female runners

[01:16:23] [SPEAKER_03]: that are largest demo

[01:16:24] [SPEAKER_03]: so our

[01:16:25] [SPEAKER_03]: our copy

[01:16:26] [SPEAKER_03]: and maybe even the images

[01:16:27] [SPEAKER_03]: on our home page

[01:16:28] [SPEAKER_03]: should be

[01:16:28] [SPEAKER_03]: really geared towards

[01:16:29] [SPEAKER_03]: like the problems that

[01:16:30] [SPEAKER_03]: that group of people has

[01:16:31] [SPEAKER_03]: and like by changing

[01:16:33] [SPEAKER_03]: that

[01:16:33] [SPEAKER_03]: that will improve

[01:16:34] [SPEAKER_03]: our conversion rates

[01:16:35] [SPEAKER_03]: because we'll be speaking

[01:16:36] [SPEAKER_03]: to our core demo

[01:16:36] [SPEAKER_03]: and if you don't see

[01:16:38] [SPEAKER_03]: that then you're like

[01:16:38] [SPEAKER_03]: oh okay like

[01:16:39] [SPEAKER_03]: it doesn't really matter

[01:16:41] [SPEAKER_03]: you know that

[01:16:41] [SPEAKER_03]: we speak to those people

[01:16:42] [SPEAKER_03]: or maybe there's

[01:16:43] [SPEAKER_03]: you could try a different

[01:16:44] [SPEAKER_03]: thing

[01:16:44] [SPEAKER_03]: but if you get a significant

[01:16:45] [SPEAKER_03]: result then you're like

[01:16:46] [SPEAKER_03]: oh cool

[01:16:46] [SPEAKER_03]: it does work

[01:16:47] [SPEAKER_03]: we should change

[01:16:48] [SPEAKER_03]: maybe try this elsewhere

[01:16:49] [SPEAKER_03]: we should

[01:16:49] [SPEAKER_03]: you know

[01:16:50] [SPEAKER_03]: maybe target our ad creative

[01:16:52] [SPEAKER_03]: a little bit more

[01:16:52] [SPEAKER_03]: double down on that audience

[01:16:54] [SPEAKER_03]: and

[01:16:55] [SPEAKER_03]: yeah I think a lot of people

[01:16:56] [SPEAKER_03]: just like

[01:16:57] [SPEAKER_03]: it is hard because

[01:16:59] [SPEAKER_03]: people just like

[01:17:01] [SPEAKER_03]: you fall into the default

[01:17:02] [SPEAKER_03]: of not having

[01:17:03] [SPEAKER_03]: like a strong thesis

[01:17:04] [SPEAKER_03]: going into it

[01:17:05] [SPEAKER_03]: or afterwards

[01:17:06] [SPEAKER_03]: like a strong opinion

[01:17:08] [SPEAKER_03]: because the data's

[01:17:08] [SPEAKER_03]: just kind of mild

[01:17:09] [SPEAKER_00]: there's also

[01:17:10] [SPEAKER_00]: business

[01:17:11] [SPEAKER_00]: like Taylor Holiday

[01:17:12] [SPEAKER_00]: talks about this

[01:17:13] [SPEAKER_00]: or has in the past

[01:17:14] [SPEAKER_00]: there's business context

[01:17:16] [SPEAKER_00]: that is

[01:17:17] [SPEAKER_00]: so critical

[01:17:18] [SPEAKER_00]: like

[01:17:18] [SPEAKER_00]: people will go into

[01:17:20] [SPEAKER_00]: an ad account

[01:17:20] [SPEAKER_00]: and be like

[01:17:21] [SPEAKER_00]: why are you running

[01:17:21] [SPEAKER_00]: this campaign

[01:17:22] [SPEAKER_00]: or this ad

[01:17:23] [SPEAKER_00]: so like it's way lower

[01:17:23] [SPEAKER_00]: performing than these

[01:17:24] [SPEAKER_00]: other

[01:17:25] [SPEAKER_00]: campaigns

[01:17:25] [SPEAKER_00]: but in reality

[01:17:26] [SPEAKER_00]: businesses have agendas

[01:17:27] [SPEAKER_00]: like they need to liquidate

[01:17:28] [SPEAKER_00]: certain products

[01:17:29] [SPEAKER_00]: they can't

[01:17:30] [SPEAKER_00]: they can't just have

[01:17:31] [SPEAKER_00]: a product sitting on

[01:17:32] [SPEAKER_00]: shelf

[01:17:33] [SPEAKER_00]: forever

[01:17:34] [SPEAKER_00]: just because the one

[01:17:35] [SPEAKER_00]: day click row as is

[01:17:35] [SPEAKER_00]: 20% lower than

[01:17:36] [SPEAKER_00]: like the best performing

[01:17:38] [SPEAKER_00]: product

[01:17:38] [SPEAKER_00]: and it's the same thing

[01:17:39] [SPEAKER_00]: with

[01:17:39] [SPEAKER_00]: Sierra like Ridge might

[01:17:41] [SPEAKER_00]: make the business

[01:17:42] [SPEAKER_00]: decision to merchandise

[01:17:43] [SPEAKER_00]: some wallets

[01:17:44] [SPEAKER_00]: towards the top of the

[01:17:45] [SPEAKER_00]: collection

[01:17:45] [SPEAKER_00]: for a few weeks

[01:17:46] [SPEAKER_00]: even if short term

[01:17:47] [SPEAKER_00]: that actually drives

[01:17:48] [SPEAKER_00]: a worse revenue per user

[01:17:50] [SPEAKER_00]: because like

[01:17:50] [SPEAKER_00]: that's the reverse

[01:17:51] [SPEAKER_00]: example of what

[01:17:52] [SPEAKER_00]: we were talking about

[01:17:53] [SPEAKER_00]: earlier or cheap tactics

[01:17:54] [SPEAKER_00]: where like short term

[01:17:55] [SPEAKER_00]: that is going to drive

[01:17:56] [SPEAKER_00]: of like net

[01:17:58] [SPEAKER_00]: you know

[01:17:58] [SPEAKER_00]: negative for the business

[01:17:59] [SPEAKER_00]: but long term

[01:18:00] [SPEAKER_00]: that's a good thing

[01:18:00] [SPEAKER_00]: because it's way better

[01:18:01] [SPEAKER_00]: for Ridge to have liquidated

[01:18:02] [SPEAKER_00]: that product

[01:18:03] [SPEAKER_00]: or hex cloud

[01:18:03] [SPEAKER_00]: have liquidated

[01:18:04] [SPEAKER_00]: that product

[01:18:05] [SPEAKER_00]: then it is

[01:18:05] [SPEAKER_00]: debit sitting there

[01:18:06] [SPEAKER_00]: for the next four years

[01:18:08] [SPEAKER_00]: you know catching cobwebs

[01:18:11] [SPEAKER_00]: so yeah

[01:18:12] [SPEAKER_03]: this is why I think

[01:18:13] [SPEAKER_03]: most best

[01:18:14] [SPEAKER_03]: like best practices

[01:18:15] [SPEAKER_03]: they don't apply in a lot

[01:18:17] [SPEAKER_03]: of places

[01:18:17] [SPEAKER_03]: but especially when it comes

[01:18:18] [SPEAKER_03]: to Sierra O

[01:18:19] [SPEAKER_03]: like

[01:18:19] [SPEAKER_03]: oh you should like

[01:18:20] [SPEAKER_03]: I've

[01:18:21] [SPEAKER_03]: I'm guilty of this

[01:18:22] [SPEAKER_03]: like I've done this before

[01:18:23] [SPEAKER_03]: like I did it with Ridge

[01:18:24] [SPEAKER_03]: for example

[01:18:24] [SPEAKER_03]: we were talking about that

[01:18:25] [SPEAKER_03]: before we started about the

[01:18:27] [SPEAKER_03]: like

[01:18:27] [SPEAKER_03]: you know

[01:18:27] [SPEAKER_03]: switching the products

[01:18:28] [SPEAKER_03]: and using the sections API

[01:18:30] [SPEAKER_03]: or whatever

[01:18:30] [SPEAKER_03]: it's like

[01:18:30] [SPEAKER_03]: you should do things

[01:18:31] [SPEAKER_03]: this way

[01:18:32] [SPEAKER_03]: because of like

[01:18:33] [SPEAKER_03]: a best practice

[01:18:35] [SPEAKER_03]: it just usually

[01:18:36] [SPEAKER_03]: doesn't take into account

[01:18:37] [SPEAKER_03]: any of the actual

[01:18:38] [SPEAKER_03]: like priorities

[01:18:39] [SPEAKER_03]: of the business

[01:18:41] [SPEAKER_03]: and so

[01:18:42] [SPEAKER_03]: that's why the

[01:18:42] [SPEAKER_03]: Sierra work that needs to be

[01:18:43] [SPEAKER_03]: done is like

[01:18:44] [SPEAKER_03]: very

[01:18:45] [SPEAKER_03]: intensive

[01:18:45] [SPEAKER_03]: and can't really be done

[01:18:46] [SPEAKER_03]: from like a

[01:18:48] [SPEAKER_03]: it's hard to do it

[01:18:49] [SPEAKER_03]: just that

[01:18:49] [SPEAKER_03]: like

[01:18:49] [SPEAKER_03]: a

[01:18:50] [SPEAKER_03]: glance essentially

[01:18:52] [SPEAKER_00]: honestly we should do

[01:18:53] [SPEAKER_00]: like

[01:18:53] [SPEAKER_00]: I would love to do

[01:18:54] [SPEAKER_00]: like a part two

[01:18:55] [SPEAKER_00]: part three here

[01:18:56] [SPEAKER_00]: because there's a lot

[01:18:56] [SPEAKER_00]: we haven't gone into yet

[01:18:57] [SPEAKER_00]: and this has been awesome

[01:18:59] [SPEAKER_00]: so

[01:19:01] [SPEAKER_00]: I don't know

[01:19:01] [SPEAKER_00]: yeah

[01:19:01] [SPEAKER_00]: I think we could

[01:19:02] [SPEAKER_00]: probably turn this into

[01:19:03] [SPEAKER_00]: like a

[01:19:03] [SPEAKER_00]: nice little series

[01:19:04] [SPEAKER_00]: if we want

[01:19:05] [SPEAKER_00]: which would be cool

[01:19:05] [SPEAKER_00]: but should we

[01:19:06] [SPEAKER_00]: I think we talked

[01:19:07] [SPEAKER_00]: a lot about what

[01:19:08] [SPEAKER_03]: not to do

[01:19:08] [SPEAKER_03]: which I think is important

[01:19:09] [SPEAKER_03]: for a lot of brands

[01:19:10] [SPEAKER_03]: in the space

[01:19:10] [SPEAKER_03]: because most brands

[01:19:11] [SPEAKER_03]: are working with a

[01:19:12] [SPEAKER_03]: I think

[01:19:12] [SPEAKER_03]: for context

[01:19:13] [SPEAKER_03]: I think a lot of brands

[01:19:14] [SPEAKER_03]: in this

[01:19:14] [SPEAKER_03]: in your audience

[01:19:15] [SPEAKER_03]: are probably working

[01:19:17] [SPEAKER_03]: with a vendor

[01:19:18] [SPEAKER_03]: to do their

[01:19:18] [SPEAKER_03]: CRO

[01:19:19] [SPEAKER_03]: because it's not something

[01:19:20] [SPEAKER_03]: that

[01:19:21] [SPEAKER_03]: you usually have

[01:19:22] [SPEAKER_03]: the skill set to do

[01:19:23] [SPEAKER_03]: or

[01:19:23] [SPEAKER_03]: maybe they're doing

[01:19:24] [SPEAKER_03]: internally

[01:19:24] [SPEAKER_03]: but

[01:19:25] [SPEAKER_03]: you want to know

[01:19:26] [SPEAKER_03]: that

[01:19:26] [SPEAKER_03]: the decisions

[01:19:27] [SPEAKER_03]: that are being made

[01:19:28] [SPEAKER_03]: like you can have

[01:19:29] [SPEAKER_03]: confidence in them

[01:19:29] [SPEAKER_03]: and so having

[01:19:30] [SPEAKER_03]: like a little bit

[01:19:31] [SPEAKER_03]: of a gut check

[01:19:31] [SPEAKER_03]: on like

[01:19:31] [SPEAKER_03]: are you following

[01:19:33] [SPEAKER_03]: these practices

[01:19:34] [SPEAKER_03]: that at least

[01:19:34] [SPEAKER_03]: you can be sure

[01:19:35] [SPEAKER_03]: that the changes

[01:19:36] [SPEAKER_03]: that someone's making

[01:19:37] [SPEAKER_03]: to your store

[01:19:39] [SPEAKER_03]: are

[01:19:40] [SPEAKER_03]: like they're confident

[01:19:41] [SPEAKER_03]: that they're actually

[01:19:41] [SPEAKER_03]: going to move

[01:19:42] [SPEAKER_03]: the business forward

[01:19:44] [SPEAKER_03]: and

[01:19:45] [SPEAKER_03]: and yeah

[01:19:45] [SPEAKER_03]: that's

[01:19:45] [SPEAKER_03]: that's the biggest

[01:19:46] [SPEAKER_03]: I think there's a lot

[01:19:47] [SPEAKER_03]: of downside risk

[01:19:48] [SPEAKER_03]: that we want to protect

[01:19:49] [SPEAKER_03]: with brands

[01:19:50] [SPEAKER_03]: at these scales

[01:19:50] [SPEAKER_03]: by actually doing

[01:19:51] [SPEAKER_03]: things correctly

[01:19:53] [SPEAKER_03]: and making sure

[01:19:53] [SPEAKER_03]: that they're actually

[01:19:54] [SPEAKER_03]: getting real results

[01:19:55] [SPEAKER_03]: but there's a lot

[01:19:56] [SPEAKER_03]: of stuff that like

[01:19:57] [SPEAKER_03]: there's the other side

[01:19:58] [SPEAKER_03]: of like what should

[01:19:59] [SPEAKER_03]: you do

[01:19:59] [SPEAKER_03]: it's very brand

[01:20:00] [SPEAKER_03]: specific

[01:20:00] [SPEAKER_03]: but there's some

[01:20:01] [SPEAKER_03]: general things

[01:20:01] [SPEAKER_03]: that like

[01:20:02] [SPEAKER_03]: I don't think

[01:20:03] [SPEAKER_03]: we got into

[01:20:04] [SPEAKER_03]: a ton here

[01:20:04] [SPEAKER_03]: but

[01:20:05] [SPEAKER_03]: well let's do

[01:20:05] [SPEAKER_03]: it

[01:20:05] [SPEAKER_00]: let's do

[01:20:06] [SPEAKER_00]: I mean

[01:20:06] [SPEAKER_00]: we only have

[01:20:07] [SPEAKER_00]: we have next week's

[01:20:08] [SPEAKER_00]: episode

[01:20:09] [SPEAKER_00]: slotted

[01:20:10] [SPEAKER_00]: but after that

[01:20:10] [SPEAKER_00]: like we're

[01:20:12] [SPEAKER_00]: we're pretty open

[01:20:12] [SPEAKER_00]: so we could definitely

[01:20:13] [SPEAKER_00]: slot in some

[01:20:14] [SPEAKER_00]: CRO part two

[01:20:15] [SPEAKER_00]: part three

[01:20:15] [SPEAKER_00]: if we want

[01:20:16] [SPEAKER_00]: yeah

[01:20:16] [SPEAKER_00]: we can go deep on

[01:20:17] [SPEAKER_00]: like more

[01:20:17] [SPEAKER_03]: less

[01:20:19] [SPEAKER_03]: on

[01:20:19] [SPEAKER_03]: more actionable

[01:20:20] [SPEAKER_03]: things for people

[01:20:21] [SPEAKER_03]: to do

[01:20:22] [SPEAKER_03]: instead of just

[01:20:22] [SPEAKER_03]: allowing people

[01:20:23] [SPEAKER_03]: not to look at their

[01:20:23] [SPEAKER_03]: not to pick at

[01:20:24] [SPEAKER_03]: their test results

[01:20:25] [SPEAKER_03]: you know

[01:20:26] [SPEAKER_03]: and that's

[01:20:27] [SPEAKER_03]: like being like

[01:20:27] [SPEAKER_03]: that's gonna be the

[01:20:28] [SPEAKER_03]: advice

[01:20:28] [SPEAKER_05]: taking a really good time

[01:20:29] [SPEAKER_00]: out

[01:20:30] [SPEAKER_00]: yeah exactly

[01:20:31] [SPEAKER_00]: yeah

[01:20:32] [SPEAKER_00]: sweet well

[01:20:33] [SPEAKER_00]: should we

[01:20:33] [SPEAKER_00]: get into

[01:20:34] [SPEAKER_00]: the test of the week

[01:20:36] [SPEAKER_00]: yeah

[01:20:37] [SPEAKER_03]: yeah

[01:20:39] [SPEAKER_03]: so

[01:20:39] [SPEAKER_03]: with the

[01:20:40] [SPEAKER_03]: the test of the week

[01:20:41] [SPEAKER_03]: the thing I want to talk about

[01:20:42] [SPEAKER_03]: and

[01:20:44] [SPEAKER_03]: again this is kind of

[01:20:45] [SPEAKER_03]: in the spirit

[01:20:45] [SPEAKER_03]: maybe of this

[01:20:46] [SPEAKER_03]: conversation

[01:20:47] [SPEAKER_03]: is that

[01:20:48] [SPEAKER_03]: the test that

[01:20:49] [SPEAKER_03]: running with someone

[01:20:50] [SPEAKER_03]: currently

[01:20:50] [SPEAKER_03]: for context

[01:20:51] [SPEAKER_03]: everybody

[01:20:51] [SPEAKER_03]: I'm

[01:20:52] [SPEAKER_03]: as build

[01:20:53] [SPEAKER_03]: working on this brand

[01:20:54] [SPEAKER_03]: building and I'm

[01:20:55] [SPEAKER_03]: doing less

[01:20:56] [SPEAKER_03]: and less

[01:20:56] [SPEAKER_03]: CRO work now

[01:20:58] [SPEAKER_03]: like trying to

[01:20:58] [SPEAKER_03]: actually

[01:20:59] [SPEAKER_03]: feel like

[01:20:59] [SPEAKER_03]: hope someone thing

[01:21:00] [SPEAKER_03]: but still work

[01:21:01] [SPEAKER_03]: with

[01:21:01] [SPEAKER_03]: work with brands

[01:21:01] [SPEAKER_03]: so one test

[01:21:02] [SPEAKER_03]: that we're

[01:21:03] [SPEAKER_03]: running right now

[01:21:04] [SPEAKER_03]: as a

[01:21:04] [SPEAKER_03]: free shipping

[01:21:05] [SPEAKER_03]: threshold test

[01:21:06] [SPEAKER_03]: which I think

[01:21:06] [SPEAKER_03]: is always

[01:21:07] [SPEAKER_03]: kind of like

[01:21:07] [SPEAKER_03]: a

[01:21:07] [SPEAKER_03]: hot

[01:21:08] [SPEAKER_03]: topic of

[01:21:09] [SPEAKER_03]: conversation

[01:21:09] [SPEAKER_03]: of like

[01:21:10] [SPEAKER_03]: are you throwing away

[01:21:11] [SPEAKER_03]: margin

[01:21:12] [SPEAKER_03]: are you actually

[01:21:13] [SPEAKER_03]: getting

[01:21:13] [SPEAKER_03]: like

[01:21:14] [SPEAKER_03]: better performance

[01:21:16] [SPEAKER_03]: on your

[01:21:16] [SPEAKER_03]: you know

[01:21:17] [SPEAKER_03]: in your ad account

[01:21:17] [SPEAKER_03]: or not

[01:21:18] [SPEAKER_03]: like

[01:21:18] [SPEAKER_03]: where does this land

[01:21:19] [SPEAKER_03]: and so the test

[01:21:20] [SPEAKER_03]: that we're running is

[01:21:20] [SPEAKER_03]: a

[01:21:21] [SPEAKER_03]: it's a

[01:21:21] [SPEAKER_03]: haircare brand

[01:21:22] [SPEAKER_03]: and the test is

[01:21:24] [SPEAKER_03]: 30

[01:21:24] [SPEAKER_03]: pa

[01:21:25] [SPEAKER_03]: it's a UK brand

[01:21:26] [SPEAKER_03]: so it's

[01:21:26] [SPEAKER_03]: 30 pounds

[01:21:27] [SPEAKER_03]: 50 pounds

[01:21:28] [SPEAKER_03]: or 60 pounds

[01:21:29] [SPEAKER_03]: as a free shipping

[01:21:29] [SPEAKER_03]: threshold

[01:21:30] [SPEAKER_03]: historically

[01:21:31] [SPEAKER_03]: for a long time

[01:21:32] [SPEAKER_03]: it's been

[01:21:32] [SPEAKER_03]: 50 pounds

[01:21:34] [SPEAKER_03]: we thought

[01:21:35] [SPEAKER_03]: hey

[01:21:36] [SPEAKER_03]: this is like

[01:21:36] [SPEAKER_03]: protecting margin

[01:21:37] [SPEAKER_03]: maybe we should

[01:21:38] [SPEAKER_03]: move it up

[01:21:38] [SPEAKER_03]: because we could

[01:21:39] [SPEAKER_03]: protect margin

[01:21:40] [SPEAKER_03]: even more

[01:21:40] [SPEAKER_03]: and if it's not having

[01:21:41] [SPEAKER_03]: that big of an

[01:21:41] [SPEAKER_03]: impact on sales

[01:21:42] [SPEAKER_03]: then like

[01:21:43] [SPEAKER_03]: why not set it to 60

[01:21:44] [SPEAKER_03]: and just

[01:21:45] [SPEAKER_03]: get back some of that

[01:21:46] [SPEAKER_03]: shipping revenue

[01:21:46] [SPEAKER_03]: essentially

[01:21:48] [SPEAKER_03]: but there's also the

[01:21:49] [SPEAKER_03]: other side of the coin

[01:21:49] [SPEAKER_03]: where

[01:21:50] [SPEAKER_03]: like most people

[01:21:51] [SPEAKER_03]: they're trying to

[01:21:51] [SPEAKER_03]: reduce their

[01:21:52] [SPEAKER_03]: customer acquisition

[01:21:53] [SPEAKER_03]: costs

[01:21:53] [SPEAKER_03]: and we're trying to

[01:21:54] [SPEAKER_03]: see if

[01:21:54] [SPEAKER_03]: hey maybe if we lower

[01:21:56] [SPEAKER_03]: it

[01:21:56] [SPEAKER_03]: and most people

[01:21:57] [SPEAKER_03]: with their entry

[01:21:57] [SPEAKER_03]: offers

[01:21:58] [SPEAKER_03]: it ends up being

[01:21:59] [SPEAKER_03]: like 30

[01:21:59] [SPEAKER_03]: pounds or

[01:22:00] [SPEAKER_03]: or around

[01:22:01] [SPEAKER_03]: 40

[01:22:01] [SPEAKER_03]: pounds

[01:22:02] [SPEAKER_03]: like if we

[01:22:02] [SPEAKER_03]: include all of

[01:22:03] [SPEAKER_03]: those people

[01:22:03] [SPEAKER_03]: in this free

[01:22:04] [SPEAKER_03]: shipping

[01:22:05] [SPEAKER_03]: is the

[01:22:06] [SPEAKER_03]: increase in performance

[01:22:07] [SPEAKER_03]: going to be

[01:22:07] [SPEAKER_03]: enough to

[01:22:08] [SPEAKER_03]: offset

[01:22:09] [SPEAKER_03]: the shipping costs

[01:22:10] [SPEAKER_03]: and what I found

[01:22:11] [SPEAKER_03]: what I thought

[01:22:12] [SPEAKER_03]: I would find

[01:22:12] [SPEAKER_03]: and this

[01:22:13] [SPEAKER_03]: kind of spawned

[01:22:14] [SPEAKER_03]: because

[01:22:14] [SPEAKER_03]: we were talking about it

[01:22:15] [SPEAKER_03]: and then I heard

[01:22:16] [SPEAKER_03]: on the operators

[01:22:16] [SPEAKER_03]: podcast

[01:22:17] [SPEAKER_03]: Mike Beckham

[01:22:17] [SPEAKER_03]: say that they

[01:22:18] [SPEAKER_03]: change their

[01:22:19] [SPEAKER_03]: free shipping

[01:22:19] [SPEAKER_03]: threshold

[01:22:20] [SPEAKER_03]: down

[01:22:20] [SPEAKER_03]: for similar reasons

[01:22:21] [SPEAKER_03]: they moved it up

[01:22:22] [SPEAKER_03]: because

[01:22:22] [SPEAKER_03]: that people

[01:22:23] [SPEAKER_03]: were telling them

[01:22:23] [SPEAKER_03]: increase your AOV

[01:22:24] [SPEAKER_03]: increase your AOV

[01:22:25] [SPEAKER_03]: and then they moved it

[01:22:26] [SPEAKER_03]: down

[01:22:26] [SPEAKER_03]: and they saw

[01:22:26] [SPEAKER_03]: a massive improvement

[01:22:27] [SPEAKER_03]: why I thought

[01:22:28] [SPEAKER_03]: this was interesting

[01:22:29] [SPEAKER_03]: is because

[01:22:30] [SPEAKER_03]: what we're

[01:22:30] [SPEAKER_03]: actually seeing

[01:22:31] [SPEAKER_03]: from these

[01:22:31] [SPEAKER_03]: test results

[01:22:32] [SPEAKER_03]: is

[01:22:33] [SPEAKER_03]: basically no difference

[01:22:35] [SPEAKER_03]: people are

[01:22:36] [SPEAKER_03]: the conversion

[01:22:37] [SPEAKER_03]: rate is actually

[01:22:37] [SPEAKER_03]: down

[01:22:41] [SPEAKER_03]: at 30

[01:22:41] [SPEAKER_03]: which I feel like is

[01:22:42] [SPEAKER_03]: kind of

[01:22:43] [SPEAKER_03]: counterintuitive

[01:22:44] [SPEAKER_03]: and I

[01:22:44] [SPEAKER_03]: want to get into

[01:22:45] [SPEAKER_03]: why I think

[01:22:46] [SPEAKER_03]: that's the case

[01:22:46] [SPEAKER_03]: and it's kind of

[01:22:47] [SPEAKER_03]: gets into

[01:22:47] [SPEAKER_03]: how to

[01:22:48] [SPEAKER_03]: analyze a test

[01:22:49] [SPEAKER_03]: and make

[01:22:49] [SPEAKER_03]: plans for

[01:22:50] [SPEAKER_03]: what to do

[01:22:50] [SPEAKER_03]: next

[01:22:51] [SPEAKER_03]: we're seeing

[01:22:52] [SPEAKER_03]: no

[01:22:52] [SPEAKER_03]: an actual

[01:22:53] [SPEAKER_03]: reduction of

[01:22:54] [SPEAKER_03]: like 1.8%

[01:22:55] [SPEAKER_03]: in conversion rate

[01:22:56] [SPEAKER_03]: can't be

[01:22:57] [SPEAKER_03]: super confident

[01:22:57] [SPEAKER_03]: in that

[01:22:58] [SPEAKER_03]: but

[01:22:58] [SPEAKER_03]: that tells us

[01:22:59] [SPEAKER_03]: like

[01:22:59] [SPEAKER_03]: that doesn't

[01:22:59] [SPEAKER_03]: really make sense

[01:23:00] [SPEAKER_03]: we're basically

[01:23:01] [SPEAKER_03]: giving

[01:23:01] [SPEAKER_03]: a ton

[01:23:02] [SPEAKER_03]: of people

[01:23:03] [SPEAKER_03]: free shipping

[01:23:04] [SPEAKER_03]: and it's having

[01:23:04] [SPEAKER_03]: no impact

[01:23:05] [SPEAKER_03]: on diversion

[01:23:05] [SPEAKER_03]: like

[01:23:05] [SPEAKER_03]: what's going

[01:23:06] [SPEAKER_03]: on here

[01:23:07] [SPEAKER_03]: and

[01:23:07] [SPEAKER_03]: the takeaway

[01:23:08] [SPEAKER_03]: that we have

[01:23:09] [SPEAKER_03]: is like

[01:23:10] [SPEAKER_03]: first

[01:23:11] [SPEAKER_03]: the order analysis

[01:23:12] [SPEAKER_03]: so we're

[01:23:12] [SPEAKER_03]: tagging every

[01:23:13] [SPEAKER_03]: order

[01:23:13] [SPEAKER_03]: because we're using

[01:23:14] [SPEAKER_03]: we're using

[01:23:15] [SPEAKER_03]: intelligence

[01:23:15] [SPEAKER_03]: to run the test

[01:23:16] [SPEAKER_03]: you can

[01:23:17] [SPEAKER_03]: set up using

[01:23:17] [SPEAKER_03]: like Shopify flow

[01:23:18] [SPEAKER_03]: you can

[01:23:18] [SPEAKER_03]: tag all of your

[01:23:19] [SPEAKER_03]: orders based on

[01:23:20] [SPEAKER_03]: their

[01:23:21] [SPEAKER_03]: intelligence

[01:23:21] [SPEAKER_03]: test ID

[01:23:22] [SPEAKER_03]: and so we're

[01:23:23] [SPEAKER_03]: tagging all the

[01:23:23] [SPEAKER_03]: orders

[01:23:24] [SPEAKER_03]: then using an

[01:23:24] [SPEAKER_03]: analytics tool

[01:23:25] [SPEAKER_03]: in order to

[01:23:26] [SPEAKER_03]: actually look at

[01:23:27] [SPEAKER_03]: how the

[01:23:28] [SPEAKER_03]: split of

[01:23:29] [SPEAKER_03]: order changes

[01:23:29] [SPEAKER_03]: so one thing

[01:23:30] [SPEAKER_03]: you can do

[01:23:30] [SPEAKER_03]: when doing this

[01:23:31] [SPEAKER_03]: order analysis

[01:23:31] [SPEAKER_03]: is you could

[01:23:32] [SPEAKER_03]: say

[01:23:32] [SPEAKER_03]: what is the

[01:23:34] [SPEAKER_03]: percentage

[01:23:34] [SPEAKER_03]: of new customers

[01:23:36] [SPEAKER_03]: that actually

[01:23:37] [SPEAKER_03]: we acquire

[01:23:37] [SPEAKER_03]: during this

[01:23:38] [SPEAKER_03]: system was

[01:23:38] [SPEAKER_03]: a percentage

[01:23:39] [SPEAKER_03]: of repeat customers

[01:23:40] [SPEAKER_03]: because we talked

[01:23:40] [SPEAKER_03]: about beforehand

[01:23:41] [SPEAKER_03]: you can't really

[01:23:42] [SPEAKER_03]: get that

[01:23:43] [SPEAKER_03]: new versus returning

[01:23:44] [SPEAKER_03]: visitor

[01:23:44] [SPEAKER_03]: kind of split

[01:23:45] [SPEAKER_03]: you have to get it

[01:23:46] [SPEAKER_03]: somehow afterwards

[01:23:47] [SPEAKER_03]: and so by doing

[01:23:48] [SPEAKER_03]: the order analysis

[01:23:49] [SPEAKER_03]: you can see

[01:23:50] [SPEAKER_03]: hey

[01:23:50] [SPEAKER_03]: of these orders

[01:23:51] [SPEAKER_03]: was this the

[01:23:52] [SPEAKER_03]: first order

[01:23:52] [SPEAKER_03]: or was this

[01:23:53] [SPEAKER_03]: like a

[01:23:53] [SPEAKER_03]: second

[01:23:54] [SPEAKER_03]: plus order

[01:23:54] [SPEAKER_03]: and then

[01:23:55] [SPEAKER_03]: you can

[01:23:55] [SPEAKER_03]: compare those

[01:23:56] [SPEAKER_03]: rates

[01:23:56] [SPEAKER_03]: and see

[01:23:57] [SPEAKER_03]: what the difference

[01:23:58] [SPEAKER_03]: is between

[01:23:58] [SPEAKER_03]: the two test

[01:23:59] [SPEAKER_03]: groups and say

[01:24:00] [SPEAKER_03]: hey in

[01:24:00] [SPEAKER_03]: this test

[01:24:01] [SPEAKER_03]: group or the

[01:24:01] [SPEAKER_03]: three test

[01:24:01] [SPEAKER_03]: groups in this case

[01:24:02] [SPEAKER_03]: what is the

[01:24:03] [SPEAKER_03]: you know

[01:24:04] [SPEAKER_03]: percentage change

[01:24:05] [SPEAKER_03]: in

[01:24:05] [SPEAKER_03]: you know

[01:24:06] [SPEAKER_03]: group A

[01:24:07] [SPEAKER_03]: in new customer

[01:24:07] [SPEAKER_03]: revenue

[01:24:08] [SPEAKER_03]: because what we're

[01:24:08] [SPEAKER_03]: trying to achieve

[01:24:09] [SPEAKER_03]: ultimately

[01:24:09] [SPEAKER_03]: something that

[01:24:10] [SPEAKER_03]: you mentioned

[01:24:11] [SPEAKER_03]: Connor is like

[01:24:12] [SPEAKER_03]: we're trying to

[01:24:12] [SPEAKER_03]: ultimately achieve

[01:24:13] [SPEAKER_03]: this new customer

[01:24:15] [SPEAKER_03]: acquisition

[01:24:16] [SPEAKER_03]: cost reduction

[01:24:17] [SPEAKER_03]: essentially

[01:24:18] [SPEAKER_03]: we're trying to acquire

[01:24:18] [SPEAKER_03]: customers more efficiently

[01:24:20] [SPEAKER_03]: so that's one thing

[01:24:21] [SPEAKER_03]: we're doing

[01:24:21] [SPEAKER_03]: here come

[01:24:22] [SPEAKER_03]: like Monday

[01:24:23] [SPEAKER_03]: where we're

[01:24:24] [SPEAKER_03]: going to actually

[01:24:24] [SPEAKER_03]: dive into that

[01:24:25] [SPEAKER_03]: and try to get

[01:24:26] [SPEAKER_03]: some answers on that

[01:24:27] [SPEAKER_03]: but

[01:24:28] [SPEAKER_03]: the more

[01:24:29] [SPEAKER_03]: more important

[01:24:30] [SPEAKER_03]: takeaway from this

[01:24:30] [SPEAKER_03]: is that the

[01:24:31] [SPEAKER_03]: design of the test

[01:24:32] [SPEAKER_03]: was probably

[01:24:33] [SPEAKER_03]: just

[01:24:33] [SPEAKER_03]: not done super well

[01:24:35] [SPEAKER_03]: and so even

[01:24:35] [SPEAKER_03]: like I

[01:24:36] [SPEAKER_03]: I'm doing this

[01:24:36] [SPEAKER_03]: while still

[01:24:37] [SPEAKER_03]: you know

[01:24:38] [SPEAKER_03]: set things up

[01:24:39] [SPEAKER_03]: where I'm like

[01:24:39] [SPEAKER_03]: why is this

[01:24:40] [SPEAKER_03]: not showing an increase

[01:24:40] [SPEAKER_03]: and the problem is that

[01:24:42] [SPEAKER_03]: on the site

[01:24:44] [SPEAKER_03]: 60% of

[01:24:45] [SPEAKER_03]: customers are returning

[01:24:46] [SPEAKER_03]: customers

[01:24:47] [SPEAKER_03]: and they've

[01:24:47] [SPEAKER_03]: bought before

[01:24:48] [SPEAKER_03]: and the free shipping

[01:24:49] [SPEAKER_03]: threshold has been

[01:24:50] [SPEAKER_03]: the same for two years

[01:24:51] [SPEAKER_03]: and so you have this

[01:24:52] [SPEAKER_03]: like

[01:24:53] [SPEAKER_03]: you can't just ignore

[01:24:54] [SPEAKER_03]: the fact that

[01:24:54] [SPEAKER_03]: like

[01:24:54] [SPEAKER_03]: things have been

[01:24:55] [SPEAKER_03]: a certain way

[01:24:56] [SPEAKER_03]: for years

[01:24:57] [SPEAKER_03]: and a lot of

[01:24:57] [SPEAKER_03]: customers are coming back

[01:24:58] [SPEAKER_03]: and they just

[01:24:59] [SPEAKER_03]: don't expect

[01:24:59] [SPEAKER_03]: things to change

[01:25:00] [SPEAKER_03]: like they don't expect

[01:25:01] [SPEAKER_03]: prices to change

[01:25:02] [SPEAKER_03]: that

[01:25:02] [SPEAKER_03]: they don't even notice

[01:25:03] [SPEAKER_03]: like these small

[01:25:04] [SPEAKER_03]: things because

[01:25:04] [SPEAKER_03]: there's like

[01:25:04] [SPEAKER_03]: I'm just buying

[01:25:05] [SPEAKER_03]: my shampoo

[01:25:05] [SPEAKER_03]: and conditioner again

[01:25:06] [SPEAKER_03]: and so

[01:25:08] [SPEAKER_03]: my thesis is that

[01:25:10] [SPEAKER_03]: hey

[01:25:10] [SPEAKER_03]: for these

[01:25:10] [SPEAKER_03]: returning customers

[01:25:11] [SPEAKER_03]: we're not making it

[01:25:13] [SPEAKER_03]: obvious enough

[01:25:13] [SPEAKER_03]: because the only place

[01:25:14] [SPEAKER_03]: that it's seen is in

[01:25:15] [SPEAKER_03]: the announcement bar

[01:25:15] [SPEAKER_03]: and then

[01:25:16] [SPEAKER_03]: in the cart

[01:25:16] [SPEAKER_03]: but it's not like

[01:25:17] [SPEAKER_03]: super obvious

[01:25:18] [SPEAKER_03]: in the cart

[01:25:19] [SPEAKER_03]: it's not like

[01:25:19] [SPEAKER_03]: a whole progress bar

[01:25:20] [SPEAKER_03]: sort of thing

[01:25:20] [SPEAKER_03]: it's very like

[01:25:21] [SPEAKER_03]: sort of minimal

[01:25:22] [SPEAKER_03]: and so

[01:25:23] [SPEAKER_03]: my thesis is

[01:25:24] [SPEAKER_03]: hey

[01:25:25] [SPEAKER_03]: let's

[01:25:25] [SPEAKER_03]: rerun this test

[01:25:26] [SPEAKER_03]: so talking about

[01:25:27] [SPEAKER_03]: hey we don't really

[01:25:28] [SPEAKER_03]: have a result here

[01:25:28] [SPEAKER_03]: so how do we actually

[01:25:29] [SPEAKER_03]: get a result

[01:25:30] [SPEAKER_03]: let's rerun this test

[01:25:31] [SPEAKER_03]: but let's put it

[01:25:32] [SPEAKER_03]: in kind of like

[01:25:33] [SPEAKER_03]: bold text

[01:25:34] [SPEAKER_03]: on the product page

[01:25:35] [SPEAKER_03]: like

[01:25:35] [SPEAKER_03]: free shipping

[01:25:36] [SPEAKER_03]: on orders over

[01:25:37] [SPEAKER_03]: this amount

[01:25:39] [SPEAKER_03]: so we can make it

[01:25:39] [SPEAKER_03]: very clear like

[01:25:40] [SPEAKER_03]: hey

[01:25:40] [SPEAKER_03]: if you are in the

[01:25:41] [SPEAKER_03]: 60 threshold

[01:25:42] [SPEAKER_03]: 60 camp

[01:25:42] [SPEAKER_03]: you're no longer

[01:25:43] [SPEAKER_03]: getting free shipping

[01:25:44] [SPEAKER_03]: on your 50 pound order

[01:25:45] [SPEAKER_03]: which will

[01:25:47] [SPEAKER_03]: reduce conversion rate

[01:25:48] [SPEAKER_03]: likely

[01:25:49] [SPEAKER_03]: and the other group

[01:25:50] [SPEAKER_03]: where they're like

[01:25:50] [SPEAKER_03]: oh my god

[01:25:50] [SPEAKER_03]: I'm getting free shipping

[01:25:51] [SPEAKER_03]: on a 30 pound order

[01:25:52] [SPEAKER_03]: like

[01:25:53] [SPEAKER_03]: I didn't realize that

[01:25:54] [SPEAKER_03]: that will actually

[01:25:54] [SPEAKER_03]: impact them more

[01:25:56] [SPEAKER_03]: and so

[01:25:56] [SPEAKER_03]: that's how

[01:25:57] [SPEAKER_03]: we're kind of

[01:25:57] [SPEAKER_03]: redesigning the test to

[01:25:59] [SPEAKER_03]: try to restart it

[01:26:01] [SPEAKER_03]: one day to be like

[01:26:01] [SPEAKER_03]: okay let's just

[01:26:02] [SPEAKER_03]: make it really obvious

[01:26:04] [SPEAKER_03]: and try to really

[01:26:05] [SPEAKER_03]: because

[01:26:06] [SPEAKER_03]: granted

[01:26:06] [SPEAKER_03]: this is a big change

[01:26:07] [SPEAKER_03]: like free shipping

[01:26:08] [SPEAKER_03]: or not free shipping

[01:26:08] [SPEAKER_03]: is usually

[01:26:09] [SPEAKER_03]: a pretty big change

[01:26:11] [SPEAKER_03]: and so

[01:26:12] [SPEAKER_03]: that's how

[01:26:12] [SPEAKER_03]: we're trying to

[01:26:13] [SPEAKER_03]: change the experiment

[01:26:14] [SPEAKER_03]: to make it more obvious

[01:26:14] [SPEAKER_03]: to get a real result

[01:26:16] [SPEAKER_03]: instead of just being

[01:26:17] [SPEAKER_03]: like

[01:26:17] [SPEAKER_03]: well it looks like

[01:26:19] [SPEAKER_03]: you know

[01:26:19] [SPEAKER_03]: there's no different

[01:26:20] [SPEAKER_03]: selection

[01:26:21] [SPEAKER_03]: and the test

[01:26:22] [SPEAKER_03]: and keep 50

[01:26:22] [SPEAKER_03]: we're gonna kind of

[01:26:23] [SPEAKER_03]: do that next set

[01:26:24] [SPEAKER_03]: of iterations

[01:26:25] [SPEAKER_03]: and so I just thought

[01:26:27] [SPEAKER_03]: that was a good example

[01:26:27] [SPEAKER_03]: of like

[01:26:28] [SPEAKER_03]: how to work through

[01:26:30] [SPEAKER_03]: ambiguity

[01:26:30] [SPEAKER_03]: in test results

[01:26:31] [SPEAKER_03]: and also kind of

[01:26:32] [SPEAKER_03]: interesting that

[01:26:32] [SPEAKER_03]: maybe the result is

[01:26:33] [SPEAKER_03]: to send you guys

[01:26:34] [SPEAKER_03]: a message afterwards

[01:26:36] [SPEAKER_03]: once we actually

[01:26:37] [SPEAKER_03]: rerun the test

[01:26:38] [SPEAKER_03]: like

[01:26:38] [SPEAKER_03]: the result might be that

[01:26:39] [SPEAKER_03]: nobody cares about the

[01:26:40] [SPEAKER_03]: threshold

[01:26:41] [SPEAKER_03]: which would be just as

[01:26:42] [SPEAKER_03]: interesting to us

[01:26:43] [SPEAKER_03]: so yeah

[01:26:44] [SPEAKER_03]: So

[01:26:45] [SPEAKER_01]: is your guys

[01:26:46] [SPEAKER_01]: is KPI

[01:26:47] [SPEAKER_01]: really around new customers

[01:26:49] [SPEAKER_01]: like that

[01:26:49] [SPEAKER_01]: that's ultimately

[01:26:50] [SPEAKER_01]: what

[01:26:51] [SPEAKER_01]: you guys care about

[01:26:52] [SPEAKER_01]: Ultimately

[01:26:52] [SPEAKER_03]: that is what

[01:26:54] [SPEAKER_03]: the business

[01:26:55] [SPEAKER_03]: cares about

[01:26:55] [SPEAKER_03]: right now

[01:26:56] [SPEAKER_03]: given its priorities

[01:26:57] [SPEAKER_03]: for

[01:26:58] [SPEAKER_03]: growth

[01:26:59] [SPEAKER_03]: essentially

[01:27:00] [SPEAKER_03]: because it's trying to

[01:27:03] [SPEAKER_03]: in doing really well

[01:27:04] [SPEAKER_03]: at

[01:27:04] [SPEAKER_03]: um

[01:27:06] [SPEAKER_03]: extracting value

[01:27:07] [SPEAKER_03]: from existing customers

[01:27:08] [SPEAKER_03]: and that's going really well

[01:27:09] [SPEAKER_03]: but new customer growth

[01:27:10] [SPEAKER_03]: has slowed

[01:27:11] [SPEAKER_03]: and customer acquisition

[01:27:12] [SPEAKER_03]: costs have risen

[01:27:13] [SPEAKER_03]: and so while they're

[01:27:14] [SPEAKER_03]: growing

[01:27:15] [SPEAKER_03]: they're growing mostly

[01:27:16] [SPEAKER_03]: based off of their

[01:27:16] [SPEAKER_03]: returning customer revenue

[01:27:18] [SPEAKER_03]: and so they're like

[01:27:19] [SPEAKER_03]: hey

[01:27:20] [SPEAKER_03]: that

[01:27:20] [SPEAKER_03]: over a long enough

[01:27:21] [SPEAKER_03]: period of time

[01:27:22] [SPEAKER_03]: we need to keep the

[01:27:22] [SPEAKER_03]: we need to keep

[01:27:23] [SPEAKER_03]: acquiring new customers

[01:27:24] [SPEAKER_03]: and so

[01:27:25] [SPEAKER_03]: that's part of their

[01:27:26] [SPEAKER_03]: that's the big

[01:27:27] [SPEAKER_03]: objective with this

[01:27:28] [SPEAKER_03]: but obviously

[01:27:29] [SPEAKER_03]: there's business case

[01:27:30] [SPEAKER_03]: of like

[01:27:30] [SPEAKER_03]: 60% of customers

[01:27:32] [SPEAKER_03]: are returning

[01:27:33] [SPEAKER_03]: we're losing a ton of money

[01:27:34] [SPEAKER_03]: on shipping revenue

[01:27:35] [SPEAKER_03]: that's not gonna work either

[01:27:36] [SPEAKER_03]: so

[01:27:36] [SPEAKER_03]: we're keeping that stuff

[01:27:37] [SPEAKER_03]: in mind

[01:27:38] [SPEAKER_03]: but the ultimate goal

[01:27:38] [SPEAKER_03]: going into the test is

[01:27:39] [SPEAKER_03]: like

[01:27:39] [SPEAKER_03]: can we get

[01:27:40] [SPEAKER_03]: much more profitable

[01:27:42] [SPEAKER_03]: acquisition spend

[01:27:43] [SPEAKER_03]: on this

[01:27:45] [SPEAKER_03]: yeah

[01:27:45] [SPEAKER_01]: yeah totally

[01:27:46] [SPEAKER_01]: I was curious just because

[01:27:49] [SPEAKER_01]: in theory

[01:27:50] [SPEAKER_01]: there's a negative impact

[01:27:51] [SPEAKER_01]: on returning customer

[01:27:52] [SPEAKER_01]: conversion rate

[01:27:53] [SPEAKER_01]: because they are familiar

[01:27:54] [SPEAKER_01]: with the certain price

[01:27:55] [SPEAKER_01]: threshold

[01:27:55] [SPEAKER_01]: and then if you find

[01:27:56] [SPEAKER_01]: like

[01:27:56] [SPEAKER_01]: hey new customer acquisition

[01:27:57] [SPEAKER_01]: is better at

[01:27:58] [SPEAKER_01]: a different threshold

[01:27:58] [SPEAKER_01]: then those customers

[01:28:00] [SPEAKER_01]: will be familiar

[01:28:00] [SPEAKER_01]: with that one

[01:28:01] [SPEAKER_01]: like it slowly

[01:28:02] [SPEAKER_01]: kind of

[01:28:02] [SPEAKER_01]: normalize over time

[01:28:03] [SPEAKER_03]: yeah

[01:28:04] [SPEAKER_03]: 100%

[01:28:05] [SPEAKER_03]: and it's

[01:28:06] [SPEAKER_03]: it's tough because

[01:28:07] [SPEAKER_03]: the other alternative

[01:28:07] [SPEAKER_03]: that we're

[01:28:08] [SPEAKER_03]: probably going to do

[01:28:09] [SPEAKER_03]: is just run a

[01:28:11] [SPEAKER_03]: offer

[01:28:12] [SPEAKER_03]: of like a free shipping

[01:28:13] [SPEAKER_03]: offer in like

[01:28:14] [SPEAKER_03]: for new customers

[01:28:15] [SPEAKER_03]: essentially

[01:28:16] [SPEAKER_03]: of like

[01:28:16] [SPEAKER_03]: hey you not only

[01:28:17] [SPEAKER_03]: get this like

[01:28:17] [SPEAKER_03]: small introduction

[01:28:18] [SPEAKER_03]: of three offer

[01:28:19] [SPEAKER_03]: but now you also

[01:28:19] [SPEAKER_03]: get

[01:28:20] [SPEAKER_03]: free shipping

[01:28:20] [SPEAKER_03]: via landing pages

[01:28:22] [SPEAKER_03]: or something like that

[01:28:22] [SPEAKER_03]: and seeing how it affects

[01:28:23] [SPEAKER_03]: performance

[01:28:25] [SPEAKER_03]: but

[01:28:26] [SPEAKER_03]: then

[01:28:26] [SPEAKER_03]: does that

[01:28:28] [SPEAKER_03]: actually

[01:28:29] [SPEAKER_03]: or like

[01:28:29] [SPEAKER_03]: not free shipping

[01:28:30] [SPEAKER_03]: but like

[01:28:30] [SPEAKER_03]: free shipping over

[01:28:31] [SPEAKER_03]: this amount

[01:28:31] [SPEAKER_03]: which they would get

[01:28:32] [SPEAKER_03]: free shipping by default

[01:28:33] [SPEAKER_03]: because the bundle

[01:28:34] [SPEAKER_03]: that they're being offered

[01:28:34] [SPEAKER_03]: is over that amount

[01:28:36] [SPEAKER_03]: but like

[01:28:38] [SPEAKER_03]: how does that translate

[01:28:39] [SPEAKER_03]: to the rest of the site

[01:28:41] [SPEAKER_03]: and

[01:28:41] [SPEAKER_03]: you know it's kind of

[01:28:42] [SPEAKER_03]: like

[01:28:42] [SPEAKER_03]: again

[01:28:43] [SPEAKER_03]: working within this ambiguity

[01:28:44] [SPEAKER_03]: and trying to attack it

[01:28:45] [SPEAKER_03]: from different angles

[01:28:46] [SPEAKER_03]: to see

[01:28:47] [SPEAKER_03]: if we can get

[01:28:48] [SPEAKER_03]: really confident

[01:28:49] [SPEAKER_03]: in one direction

[01:28:50] [SPEAKER_03]: or another

[01:28:50] [SPEAKER_00]: there's so much

[01:28:51] [SPEAKER_00]: thoughtfulness

[01:28:52] [SPEAKER_00]: that goes into this

[01:28:53] [SPEAKER_00]: and

[01:28:54] [SPEAKER_00]: in the same way

[01:28:54] [SPEAKER_00]: that

[01:28:56] [SPEAKER_00]: like

[01:28:56] [SPEAKER_00]: I feel like

[01:28:57] [SPEAKER_00]: in like

[01:28:58] [SPEAKER_00]: business

[01:28:58] [SPEAKER_00]: like consumer businesses

[01:28:59] [SPEAKER_00]: in general

[01:29:00] [SPEAKER_00]: over the last five years

[01:29:00] [SPEAKER_00]: there's this been

[01:29:01] [SPEAKER_00]: this shift from

[01:29:02] [SPEAKER_00]: like revenue

[01:29:02] [SPEAKER_00]: as king

[01:29:03] [SPEAKER_00]: to

[01:29:03] [SPEAKER_00]: profitability

[01:29:04] [SPEAKER_00]: and contribution

[01:29:05] [SPEAKER_00]: margin is king

[01:29:06] [SPEAKER_00]: and I feel like

[01:29:07] [SPEAKER_00]: within growth marketing

[01:29:08] [SPEAKER_00]: there's also been a similar

[01:29:09] [SPEAKER_00]: not

[01:29:10] [SPEAKER_00]: an analogous shift

[01:29:11] [SPEAKER_00]: in the last like

[01:29:12] [SPEAKER_00]: three to

[01:29:13] [SPEAKER_00]: from the lat

[01:29:13] [SPEAKER_00]: like

[01:29:14] [SPEAKER_00]: four

[01:29:14] [SPEAKER_00]: three to five years ago

[01:29:15] [SPEAKER_00]: to today

[01:29:16] [SPEAKER_00]: where it's

[01:29:17] [SPEAKER_00]: like

[01:29:18] [SPEAKER_00]: everyone was talking about

[01:29:18] [SPEAKER_00]: volume

[01:29:19] [SPEAKER_00]: volume

[01:29:19] [SPEAKER_00]: volume

[01:29:20] [SPEAKER_00]: two years ago

[01:29:21] [SPEAKER_00]: three years ago

[01:29:21] [SPEAKER_00]: still some people today

[01:29:22] [SPEAKER_00]: and like

[01:29:23] [SPEAKER_00]: I used to

[01:29:24] [SPEAKER_00]: think about

[01:29:25] [SPEAKER_00]: think about that

[01:29:26] [SPEAKER_00]: a lot too

[01:29:26] [SPEAKER_00]: and I just

[01:29:27] [SPEAKER_00]: believe in that

[01:29:27] [SPEAKER_00]: less and less now

[01:29:28] [SPEAKER_00]: and really believe that

[01:29:29] [SPEAKER_00]: like

[01:29:30] [SPEAKER_00]: more intentional

[01:29:31] [SPEAKER_00]: more thoughtful

[01:29:31] [SPEAKER_00]: bigger

[01:29:32] [SPEAKER_00]: swings

[01:29:33] [SPEAKER_00]: are

[01:29:34] [SPEAKER_00]: are what's king here

[01:29:35] [SPEAKER_00]: and like

[01:29:35] [SPEAKER_00]: that's

[01:29:36] [SPEAKER_00]: that's coming out

[01:29:36] [SPEAKER_00]: and

[01:29:37] [SPEAKER_00]: and you

[01:29:37] [SPEAKER_00]: talking through

[01:29:38] [SPEAKER_00]: like

[01:29:38] [SPEAKER_00]: what a great example

[01:29:39] [SPEAKER_00]: of

[01:29:39] [SPEAKER_00]: of how like

[01:29:40] [SPEAKER_00]: this surface level

[01:29:41] [SPEAKER_00]: metrics

[01:29:42] [SPEAKER_00]: is not where we ended

[01:29:43] [SPEAKER_00]: like

[01:29:43] [SPEAKER_00]: there was so many

[01:29:44] [SPEAKER_00]: other metric to dig into

[01:29:45] [SPEAKER_00]: and

[01:29:46] [SPEAKER_00]: next steps

[01:29:46] [SPEAKER_00]: based on that

[01:29:47] [SPEAKER_00]: and it's

[01:29:49] [SPEAKER_00]: like that's the level

[01:29:49] [SPEAKER_00]: of thoughtfulness

[01:29:50] [SPEAKER_00]: that's going to

[01:29:50] [SPEAKER_00]: drive

[01:29:51] [SPEAKER_00]: big

[01:29:52] [SPEAKER_00]: it that also

[01:29:52] [SPEAKER_00]: like facilitates

[01:29:53] [SPEAKER_00]: it ensures

[01:29:54] [SPEAKER_00]: that you're only

[01:29:55] [SPEAKER_00]: taking big swings

[01:29:56] [SPEAKER_00]: if you're putting

[01:29:56] [SPEAKER_00]: that level of

[01:29:57] [SPEAKER_00]: thoughtfulness

[01:29:58] [SPEAKER_00]: into each test

[01:29:58] [SPEAKER_00]: that you run

[01:29:59] [SPEAKER_00]: it's just so intentional

[01:30:00] [SPEAKER_03]: yeah

[01:30:01] [SPEAKER_03]: and that's why

[01:30:01] [SPEAKER_03]: I think

[01:30:02] [SPEAKER_03]: CRO

[01:30:02] [SPEAKER_03]: is a very difficult

[01:30:03] [SPEAKER_03]: service to provide

[01:30:05] [SPEAKER_03]: and so

[01:30:05] [SPEAKER_03]: no hate

[01:30:05] [SPEAKER_03]: anybody who is

[01:30:06] [SPEAKER_03]: running

[01:30:07] [SPEAKER_03]: like a CRO agency

[01:30:08] [SPEAKER_03]: like it is

[01:30:08] [SPEAKER_03]: it's hard

[01:30:09] [SPEAKER_03]: it's a really hard thing

[01:30:10] [SPEAKER_03]: to scale

[01:30:10] [SPEAKER_03]: because

[01:30:10] [SPEAKER_03]: as you can see

[01:30:11] [SPEAKER_03]: you're dealing

[01:30:12] [SPEAKER_03]: with a ton of ambiguity

[01:30:13] [SPEAKER_03]: and you need

[01:30:13] [SPEAKER_03]: a lot of

[01:30:14] [SPEAKER_03]: you need a deep

[01:30:15] [SPEAKER_03]: understanding

[01:30:15] [SPEAKER_03]: of

[01:30:17] [SPEAKER_03]: many different aspects

[01:30:18] [SPEAKER_03]: of the business

[01:30:19] [SPEAKER_03]: to be able to

[01:30:19] [SPEAKER_03]: actually put forward

[01:30:20] [SPEAKER_03]: ideas

[01:30:20] [SPEAKER_03]: that are going

[01:30:21] [SPEAKER_03]: to move the

[01:30:21] [SPEAKER_03]: business forward

[01:30:22] [SPEAKER_03]: and so

[01:30:22] [SPEAKER_03]: it's a really difficult

[01:30:24] [SPEAKER_03]: service to provide

[01:30:26] [SPEAKER_03]: at scale

[01:30:26] [SPEAKER_03]: and really even

[01:30:27] [SPEAKER_03]: just like

[01:30:27] [SPEAKER_03]: one off

[01:30:28] [SPEAKER_03]: it requires a lot

[01:30:29] [SPEAKER_03]: of focus

[01:30:30] [SPEAKER_03]: and so

[01:30:30] [SPEAKER_03]: that's why I'm

[01:30:32] [SPEAKER_03]: scaling back

[01:30:33] [SPEAKER_03]: on doing it

[01:30:34] [SPEAKER_03]: as much

[01:30:34] [SPEAKER_03]: and doing it

[01:30:35] [SPEAKER_03]: with like

[01:30:35] [SPEAKER_03]: select clients

[01:30:36] [SPEAKER_03]: and then

[01:30:37] [SPEAKER_03]: you know

[01:30:38] [SPEAKER_03]: building something

[01:30:38] [SPEAKER_03]: on the side

[01:30:40] [SPEAKER_03]: so yeah

[01:30:40] [SPEAKER_03]: shout out to anybody

[01:30:43] [SPEAKER_03]: building a CRO agency

[01:30:44] [SPEAKER_03]: because it's

[01:30:44] [SPEAKER_03]: probably the hardest

[01:30:45] [SPEAKER_03]: service business

[01:30:46] [SPEAKER_03]: to build

[01:30:47] [SPEAKER_03]: and the after

[01:30:48] [SPEAKER_03]: round is tough

[01:30:49] [SPEAKER_00]: like

[01:30:50] [SPEAKER_00]: that's what

[01:30:51] [SPEAKER_00]: I've noticed

[01:30:51] [SPEAKER_00]: the same thing

[01:30:52] [SPEAKER_00]: with like

[01:30:52] [SPEAKER_00]: the agencies

[01:30:52] [SPEAKER_00]: we work with

[01:30:53] [SPEAKER_00]: like the

[01:30:54] [SPEAKER_00]: the design

[01:30:54] [SPEAKER_00]: and dev

[01:30:55] [SPEAKER_00]: like

[01:30:55] [SPEAKER_00]: it's a

[01:30:56] [SPEAKER_00]: it's a big

[01:30:57] [SPEAKER_00]: process

[01:31:00] [SPEAKER_00]: strategy

[01:31:01] [SPEAKER_00]: and like

[01:31:02] [SPEAKER_00]: analysis

[01:31:04] [SPEAKER_00]: is incredibly hard

[01:31:05] [SPEAKER_00]: but then

[01:31:06] [SPEAKER_00]: so it's just

[01:31:06] [SPEAKER_00]: like getting the designs

[01:31:07] [SPEAKER_00]: and developments

[01:31:08] [SPEAKER_00]: out the door

[01:31:09] [SPEAKER_00]: in a non

[01:31:09] [SPEAKER_00]: buggy way

[01:31:10] [SPEAKER_00]: like

[01:31:10] [SPEAKER_00]: it's just

[01:31:10] [SPEAKER_00]: a bit

[01:31:11] [SPEAKER_00]: it's hard

[01:31:11] [SPEAKER_03]: yeah

[01:31:12] [SPEAKER_03]: there's a lot of

[01:31:13] [SPEAKER_03]: work that goes into it

[01:31:13] [SPEAKER_03]: you can get

[01:31:14] [SPEAKER_03]: into a nice flow

[01:31:14] [SPEAKER_03]: like I

[01:31:15] [SPEAKER_03]: as far as a process

[01:31:16] [SPEAKER_03]: what I

[01:31:17] [SPEAKER_03]: used to do

[01:31:18] [SPEAKER_03]: was just

[01:31:19] [SPEAKER_03]: run a set

[01:31:20] [SPEAKER_03]: number of tests

[01:31:21] [SPEAKER_03]: on a set

[01:31:21] [SPEAKER_03]: cadence

[01:31:22] [SPEAKER_03]: so like

[01:31:22] [SPEAKER_03]: we talked about

[01:31:23] [SPEAKER_03]: two weeks

[01:31:24] [SPEAKER_03]: four weeks

[01:31:25] [SPEAKER_03]: but it's like

[01:31:25] [SPEAKER_03]: hey when those

[01:31:26] [SPEAKER_03]: tests are done

[01:31:27] [SPEAKER_03]: like the next set

[01:31:28] [SPEAKER_03]: of tests are ready

[01:31:29] [SPEAKER_03]: to go live

[01:31:30] [SPEAKER_03]: because we've been working out

[01:31:31] [SPEAKER_03]: like if it was a

[01:31:32] [SPEAKER_03]: four week cadence

[01:31:33] [SPEAKER_03]: that's why I think people

[01:31:34] [SPEAKER_03]: are worried about this

[01:31:35] [SPEAKER_03]: duration thing

[01:31:36] [SPEAKER_03]: of like

[01:31:36] [SPEAKER_03]: letting tests run too long

[01:31:37] [SPEAKER_03]: but it's like

[01:31:38] [SPEAKER_03]: it takes time

[01:31:39] [SPEAKER_03]: to make new tests anyway

[01:31:40] [SPEAKER_03]: so like

[01:31:41] [SPEAKER_03]: you're not exactly

[01:31:42] [SPEAKER_03]: by just running tests

[01:31:43] [SPEAKER_03]: for a week

[01:31:43] [SPEAKER_03]: like

[01:31:44] [SPEAKER_03]: you're probably not

[01:31:45] [SPEAKER_03]: running more tests

[01:31:46] [SPEAKER_03]: or just like

[01:31:46] [SPEAKER_03]: being very frantic

[01:31:47] [SPEAKER_03]: about it

[01:31:48] [SPEAKER_03]: and so what I like

[01:31:49] [SPEAKER_03]: to do is just

[01:31:49] [SPEAKER_03]: be on a cadence

[01:31:50] [SPEAKER_03]: of like

[01:31:50] [SPEAKER_03]: hey

[01:31:50] [SPEAKER_03]: let's run this test

[01:31:51] [SPEAKER_03]: experiment for four weeks

[01:31:52] [SPEAKER_03]: that gives us

[01:31:53] [SPEAKER_03]: plenty of time

[01:31:54] [SPEAKER_03]: to

[01:31:54] [SPEAKER_03]: build out

[01:31:55] [SPEAKER_03]: the new set

[01:31:56] [SPEAKER_03]: of experiments

[01:31:56] [SPEAKER_03]: regardless of how

[01:31:57] [SPEAKER_03]: you know

[01:31:57] [SPEAKER_03]: big of a lift

[01:31:58] [SPEAKER_03]: it is

[01:31:58] [SPEAKER_03]: in order to get that

[01:31:59] [SPEAKER_03]: done

[01:32:00] [SPEAKER_03]: and then we'll have those

[01:32:01] [SPEAKER_03]: ready to go

[01:32:01] [SPEAKER_03]: that next day

[01:32:03] [SPEAKER_03]: so that when we end the test

[01:32:04] [SPEAKER_03]: we launch a new set of tests

[01:32:05] [SPEAKER_03]: and then

[01:32:06] [SPEAKER_03]: that four weeks again

[01:32:07] [SPEAKER_03]: like obviously

[01:32:08] [SPEAKER_03]: on your second round of that

[01:32:09] [SPEAKER_03]: you're now doing

[01:32:10] [SPEAKER_03]: the analysis

[01:32:10] [SPEAKER_03]: and really

[01:32:11] [SPEAKER_03]: digging into those tests

[01:32:12] [SPEAKER_03]: to be like

[01:32:12] [SPEAKER_03]: are these

[01:32:14] [SPEAKER_03]: how do we feel about these