Learn, Make, Learn
Learn, Make, Learn is two product geeks sharing qualitative & quantitative perspectives to help you make, better. Hosted by Ernest Kim and Joachim Groeger.
Learn, Make, Learn
Experimentation in Product Innovation
In this episode, we challenge tech industry dogma around A/B testing & Minimum Viable Products (MVPs), and propose alternatives better suited to today’s post-low interest rate economy.
FOLLOW-UPS – 01:41
Apple Vision Pros & Cons
How to book a Vision Pro demo at an Apple Store
Apple Vision Pro has finally made virtual reality worth it
First insights about Apple Vision Pro optics
Steve Jobs on the Post-PC Era
Apple Vision Pro review: magic, until it’s not
Anil Dash on Apple Vision Pro
EXPERIMENTATION IN PRODUCT INNOVATION – 19:43
Building a Culture of Experimentation
The Surprising Power of Online Experiments
Computers suck now. They run on superstition, not science.
CRITIQUE OF THE MVP APPROACH – 25:03
Minimum Viable Product
Cory Doctorow on the enshittification of platforms
PIXAR, BRAINTRUSTS & SAFETY – 33:45
The Marginalian reviews Ed Catmull’s Creativity, Inc.
Toolbox: How “Jobs to Be Done” Can Help You Make, Better
LIES, DAMN LIES & STATISTICS – 48:24
Science has been in a “replication crisis” for a decade
REAL ARTISTS SHIP – 54:21
Andy Hertzfeld on the origin story behind this Jobs-ism
WEEKLY RECS – 56:52
“Onigiri” Rice Balls: Kings of the Convenience Store
History of Seven & I Holdings
The Greatest Night in Pop
The story behind “One Love” from Nas’ legendary Illmatic album
CLOSING & PREVIEW – 01:07:20
(Image credit: Young Frankenstein / Twentieth Century Fox Film Corp)
****
Rant, rave or otherwise via email at LearnMakeLearn@gmail.com or on Threads @LearnMakeLearnShow.
CREDITS
Theme: Vendla / Today Is a Good Day / courtesy of www.epidemicsound.com
Drum hit: PREL / Musical Element 85 / courtesy of www.epidemicsound.com
Hello and welcome to Learn Make Learn where we share qualitative and quantitative perspectives on products to help you make better. My name is Ernest Kim and I'm joined by my friend and co-host Joachim Groeger. Hey, Joachim, how's it going?
Joachim:I am doing well again, another Sunday. I'm gonna be honest with everyone. This is our second take, and now I know it's Super Bowl Sunday. I could now pretend that I am fully immersed in American culture and I have a full knowledge of the teams that are playing. But now I've already stumbled and I don't know who's playing. So this, maybe it's, this is not going so well. I'm doing fine. Ernest, how are you doing?
Ernest:Thanks. Yeah, I'm, very excited about this big game Sunday as, I think we're meant to call it instead of using the actual trademark term, but
Joachim:Okay.
Ernest:I. But yeah, definitely looking forward to, uh,, it's a bit of a tradition on my end to, uh,, watch the game with family.
Joachim:Oh, nice. Okay. Very good.
Ernest:But, um,, we're excited about this episode as well. This is episode six and our topic today is Experimentation in Product Innovation. And by experimentation we mean a specific kind of outlook on experimentation that's been embraced by the tech sector and has started to spread beyond it. Reid Hoffman of LinkedIn fame captured this outlook when he said, if you're not embarrassed by the first version of your product, you've launched too late. Is this move fast and break things approach a good thing? Or is this embrace of continuous experimentation and A/B testing eroding the discipline necessary to create meaningful lasting products? But before diving into these questions, let's start with a follow up and,, I'll get us started with a follow up to our second episode from a few weeks back. Apple Vision, pros and Cons. And this is based on, um,, the fact that my wife and I both went through the Apple Vision Pro demo experience this past week, and I thought I'd just share a few takeaways from those sessions. The first thing they're 30 minute sessions. We actually did them separately. We weren't able to get sessions back to back, but we were able to get'em on the same day. And the first thing that I found really exciting was the diverse range of people that I saw going through the demo experience. And there were plenty of, middle-aged geeky guys like me. But um, just over the span of those two different sessions, I saw um, a young girl in, with her parents doing a demo session, a disabled older gentleman doing a session. So it was really clear that. While Vision Pro isn't, certainly, isn't accessible from a price perspective, it is absolutely growing the tent of people interested in VR and adjacent technologies in a way that no one else has managed to so far. It brings to mind a quote from Palmer Lucky who founded Oculus VR and eventually sold the company to Facebook. He said that in his view. Virtual reality has to become something that everybody wants before it can become something that everybody can afford. You can't reverse those two steps and ex expect mainstream acceptance. And I think he's actually right about that. And Apple is delivering on this desirability in a way that no one else in the VR space has managed before now. So that was the first thing. The second thing is that in terms of the tech itself, I thought it was pretty astonishing. I was very impressed. I've had a chance to try a few different VR products in the past, and it was quite a bit beyond any of those other things that I've tried, And I would say that. The vision system, the fact that it's showing you the real world, it's, it's not perfect. Similarly, the gesture based interface, wasn't perfect, but it was really well developed. You can tell that it was something that had gone through quite a few iterations and kind of hinting at our main topic for today. It, they didn't, neither the vision system nor the interface felt like a version 1.0 experience uh, felt much richer than that of the. Kind of things that they took us through during the demo. The thing that I found the most compelling were the immersive experiences. So even down to that simple I think they call it immersive environments, where you can turn the digital crown and it'll make it look like you're, on the moon or near Mount Hood. In the, marketing imagery, they look fine, but when you have the headset on, those immersive experiences are pretty cool. And I was surprised, I didn't realize this, but they're animated. I thought that they were just basically 3D pano images, but they're actually animated. For example, I was looking at the Mount Hood environment and it started raining. And not only do you get that visual, but you also get the slight sound of the rain falling in the lake that's in front of you. So that was a lot more compelling than I thought it was gonna be. And then overall, the immersive video experiences I found to be very. Compelling and much more exciting when you experience it firsthand than in, any of the promotional materials. A lot of people have mentioned there's this Alicia Keys rehearsal studio experience where you really feel like you're in the studio space with her and her band. But the thing that I found the most compelling was they showed an example of a spatial video captured via the um, vision Pro. And it's the one they've shown in their promotional materials where it's a birthday party and the daughter blows out the candles towards you. and you know, you see the promo promo stuff and it's like, okay, whatever. That's fine. But experiencing it firsthand, it is really compelling. They've done a remarkably good job of capturing and then conveying that in a way that. Really is highly emotional. I was really surprised by that. They also showed a demo of a spatial video captured via an iPhone 15 Pro. And it was also very good, not as rich as the version captured via the Vision Pro headset because I think the Vision Pro has the benefit of cameras that are more widely spaced apart. So you get a much richer spatial experience from those videos. But even the version captured with the iPhone 15 PRO was pretty compelling. So that to me was kind of the highlight. Third thing I wanted to point out was around comfort and motion sickness. Um, for me, you know, I know that that's been a, a concern voice by a lot of people was comfort given the weight of the headset. For me, the comfort was actually okay. Uh, I used it with the standard solo knit band, the one that's in all the promotional photos. you know, It was a little bit heavy on my face, but, I felt like it was fine and I felt like I could have worn it for much longer than the, the span of my 30 minute demo for my wife. It was much less comfortable. She really couldn't get a good fit with that standard solo net band. And I was surprised because I didn't realize they would do this. But the person running the demo for my wife brought out the other band, the dual loop band, which goes over the head as well as behind it. And. While she wasn't able to get a perfect fit, she felt like that was much more comfortable and she was able to go through the demo with that dual loop version of the band. In terms of the motion sickness piece of this, I was really worried about that because I had seen a few people online mention that they actually weren't able to make it through the whole demo because they got motion sickness. And I was worried about that because I, am sensitive to motion sickness when it comes to even first person shooter type games. They, give me nausea. And then pretty much all of the VR experiences I've tried in the past have made me feel nauseous pretty quickly. So I was worried about that, but I'm happy to be able to report, I guess, that I didn't experience motion sickness or nausea during the demo and neither did my wife, but we did feel a bit of it afterwards. And I think that might've been in part because of the nature of the demo experience. You know, They cut through a bunch of different spatial video type demos. And I don't think that's the sort of thing you would do in typical use of the device. So, um, I'm chalking it up to the demo itself being a little bit more quick cut than you would typically experience in terms of the motion sickness piece of it. I was. Excited to read, as I was researching vision Pro, a quote from um, a professor of AI and spatial computing from Liverpool Hope University. This was in an interview with the BBC last year. He mentioned that, the main problem with VR emotion sickness is something called Virgins Accommodation Conflict, or VAC, and he noted that Apple has reduced motion sickness as much as possible by reducing lag and delay and utilizing high quality displays. And on that point of displays as important as the Vision Pro's, ultra high resolution Micro, OLED displays are the lenses that direct the light. Projected from those displays into your eyes are at least as important. I. For addressing this issue of VAC and motion sickness and the optical technology is really fascinating and it's, you know, it's a pretty deep rabbit hole, so I won't go into it in detail, but uh, we'll include links in the show notes for anyone who's interested in diving into this. But the reason I've highlighted the. The displays and these remarkable optical technologies at work in the Vision Pro is that they're related to my final observation, which is that I came away from the demo feeling really, really strongly that the product concept embodied by Apple Vision Pro, which is fundamentally a VR headset that shuts you off from the real world and then applies just massive amounts of technology towards projecting reality back into your field of view. I think that that approach is a dead end, and I have to believe Apple recognizes this as well. The reason I believe it's a dead end is that the technology required for this approach will never scale down to a device that can reach mainstream audiences. The specs for Vision Pro, that sounds so impressive. You know, over a dozen cameras and related sensors to capture the real world, two high powered Uh. processors to take all that data and stitch it into something humans can comprehend with just 12 milliseconds of lag, two displays with 23 million pixels of resolution. As impressive as they sound, those are actually just a baseline necessary to make a pass through mixed reality experience palatable for a broad audience. You know, I'm sure Apple is gonna be able to reduce cost in some other areas, but those are just gonna be nibbling away at the edges compared to the cost of these fundamental ingredients. Now, I know a lot of people on social media have argued that meta has already done all of this in their Quest Pro and the Quest three at a fraction of the price. But the reality is that they haven't, you know, if you go through the, this Vision Pro experience firsthand, you'll see that, compared to Vision Pro, there's a reason those other devices haven't reached mainstream adoption and it's because they're just not good enough for people who aren't geeks willing to put up with their, really significant limitations. And I think the people at Apple are certainly not dumb, you know, and so they certainly recognize all of this. There is a lot of speculation that there will be a lower cost version of Apple Vision Pro people are saying around the price of an iPhone pro, that's just not gonna be possible within this paradigm of a VR device that delivers ultra realistic mixed reality. Where it could be possible though, is in an optical augmented reality device. One that doesn't shut you off from the real world and does not try to recreate reality in a headset, but instead overlays information atop your true view of reality. This is by no means an easy thing to do and to be delivered in a form. acceptable to mainstream audiences is probably gonna take breakthroughs in direct retinal projection. But an optical ar approach eliminates the need for a bulky headset that shuts out the real world, eliminates the need for ultra high resolution displays sharp enough to recreate the real world, and also eliminates the need for uh, the massive compute horsepower necessary to project the real world onto those displays with nearly zero lag. And all that means you also end up with a product with much lower power requirements. So a much smaller battery and a device that's much lighter and more wearable overall. And vitally. It also means you end up with a product that can be much less expensive. And this is what I see as. Sort of an Apple Vision Air, not a cheaper version of the Vision Pro, but a fundamentally different device. And I'm guessing the folks at Apple see this as well, which is why I find it remarkable that they've nonetheless been willing to invest the massive resources required to bring Vision Pro to market. You know, I do expect that there will continue to be a place for a Pro-like product built around the fundamentally VR paradigm. So, you know, There's gonna continue to be a return on the huge investment apple's made on the VR centric aspects of this device. But I think that return will be very small compared to the opportunity around optical ar. You know, I I was trying to think of an analogy to explain this, and the one that came to mind was in the automotive space, imagine an entirely new car brand started up, and even though they knew that fully battery electric vehicles would be, it would represent the majority of new car sales within the next, say, five to seven years, they nonetheless decided to start by building a hybrid vehicle and invested a huge amount of their innovation capacity into the internal combustion aspects of the car. This is pretty much what I think Apple's done here in the way that they've entered the spatial computing market with what is fundamentally a VR product. This um, automotive, automotive analogy brings to mind something that Steve Jobs actually said to describe the post PC transition. He said when uh, we were an agrarian nation, all cars were trucks because that's what you needed on the farm. But as vehicles started to be used in the urban centers. Cars got more popular and things that you didn't care about in a truck as much started to become paramount in cars. PCs are going to be like trucks. He said they're still gonna be around. They're still gonna have a lot of value, but they're gonna be used by one out of X people as compared to cars that are gonna be used by far more people. Now, after using Vision Pro, it's clear to me that VR devices are like the trucks in Jobs' analogy here. They've defined spatial computing up till now, and they'll continue to offer value to certain audiences, but they're going to be supplanted by optical AR products even though those optical R products won't be able to deliver on the immersion that VR headsets are uniquely great at. You know, As I mentioned just a minute ago, the spatial videos were the thing that most impressed me during my Vision Pro demo, and that sort of experience is massively amplified when you can shut out the real world, which isn't going to be possible in optical AR devices, but optical AR is going to enable so many other use cases while keeping us connected to the real world and. It's gonna do that in smaller, more affordable form factors, which is why I'm convinced it's going to be the dominant spatial computing paradigm. And all that makes me really curious to, to know why Apple nonetheless made this decision to start with Vision Pro. You know, Maybe knowing that retinal projection is still several years out, they decided that it was better to launch the best possible product they could launch today, rather than waiting for optical AR tech to mature. And maybe there were business imperatives at play here. You know, With iPhone sales already plateauing Vision Pro creates a new revenue stream. Small as it is, that offers growth now and maybe they were concerned about meta establishing a leadership position in this space. I really hope that someday down the line after the team that created Vision Pro is all retired, we'll hear some of the backstory to this. Um, Alright, now finally to the question of whether I bought it I do, I will admit that my purchase finger did tingle after my demo. And it is entirely possible that if they had my right configuration, you know, with the right um, light seal for my face uh, in stock at the store, I might have actually gone ahead and bought one, but they didn't, which is probably fortunate for me. And I'm fairly certain I'm gonna hold off on this first incarnation, but. I would really strongly recommend that people try the demo, it's free. At least in my area, in the Portland area, it's really easy to get an appointment. It's only 30 minutes and they don't try to push you to buy. They're not at all pushy. I think it's absolutely worth it if you have any interest in technology, but also if you have interest in retail. I think it's the pinnacle of retail theater as it exists today to be able to do what they're doing at scale. This sort of an incredibly high touch experience, I think is really fascinating to see firsthand. So I think for a lot of reasons it's absolutely worth giving the demo experience a try. Joachim, do you have any interest in giving it a try?
Joachim:I think, well now you put me on the spot, so I'll have to say yes. I think I will give it a try. It's been a while since I've tried a VR experience. I think the last time was an HTC Vive set. I. Seven years ago, which was very enjoyable and it was a game experience. So yeah I was reading Nli Patel's review on The Verge watching the associated video. And a lot of what he was saying came through also in what you were saying as well, regarding shutting out the outside world and the technology still being a little bit hobbled to bring in reality onto those screens and into your eyes. I think N Patel said, um. reality is just so much better than a screen showing me reality. and it is a little bit lonely and isolating to be in that space. I was very interested in the spatial video piece because that birthday candle sequence strongly reminded me of late runner 2049 again, where there's one of the characters is designing a memory that is a birthday scene, I think. So I was getting a Blade runner vibe that I would like to have experienced as well firsthand. And it actually reminded me of another perspective on on all of this and an dash who I follow on Mastodon and will share a post that he he posted the other day. This is what he said after a closer look at the I Apple Vision Pro reviews and talking to people who've used it, my prediction of the most likely path is that its evolution resembles the Apple Watch, which went from big vague promises to a simple health and notifications device. So a future vision air, using your term as well, will ditch external I displays and complexity and mostly be a very large, smart display. You can tell any Mac, iPhone, iPad or Apple TV may be HomePod to extend to this display. That's the big basic deal. So that was his take. And he also, shared an old tweet of his from, I think 14 years ago when the first iPad airs were announced or iPads, I should say, with the, with apple's own chip set inside of And so he said, that's the real big deal here is not really the iPad the real show is this chip and that's gonna change the course of Apple. And quote me on that five years from now is what he asked is his readers to do. I quite like this succinct, slightly downer discussion of what the Vision Pro will become. But some of it echoes what you're saying as well, which is just ditch the complexity and focus on more on the essential pieces that make it an experience that would be useful and that. Makes it useful as extension of all of the devices that you already have. But I'm interested in that demo now.
Ernest:Yeah, I really would love to hear what you think of it as well. I think you'd um, bring a, maybe a'cause I'm, I was bullish going into it, so it'd be great to hear your perspective. I think as someone who's maybe was, has a little bit more of a bearish view going in uh, see what you
Joachim:right, yeah, for sure.
Ernest:Alright, now let's move on to our main topic, which is continuous experimentation in product innovation. Joachim, can you set the stage for us?
Joachim:Yeah. With pleasure. As Ernest mentioned at the top of the episode, we're gonna talk about this culture of experimentation within the context of product innovation. So not medical experimentation or anything like that really focused on the innovation of products and most of my experiences involve experimentation in the digital domain. So the classic tech platform experimentation that we are discussing at the beginning of the episode. The way I think about experimentation is encapsulated by this introduction to an Harvard Business Review article that I stumbled on by Stefan Tomkin. And so he's writing about booking dot com's in late 2017. Quote, in December, 2017, just before the busy holiday travel season, booking dot com's director of design proposed a radical experiment testing an entirely new layout for the company's homepage. All the content and design elements, pictures, text buttons, and messages that booking.com had spent years optimizing would be eliminated. Jillian Tan's booking dot com's, CEO at the time was skeptical. She worried that the change would cause confusion among the company's loyal customers. Given that pessimism, why didn't senior management just veto the trial? Because doing so would've violated one of booking dot com's core tenets. Anyone at the company can test anything without management's permission. So the, traditional approach people take to experimentation is you have some kind of idea, in this case, booking dot com's, design lead felt simplification was going to be the solution. You have an idea, you run it through a test, and you hope that something comes out at the other end of that. And so if that's the way we do things, why aren't we just experimenting on every single dimension that we can imagine? Why aren't we just testing every possible feature in this way? As we heard, booking.com has no limits on who can run a test. The reality is that's exactly what they do. Quote again from the article, booking.com runs more than 1000 rigorous tests simultaneously, and by my estimates, more than 25,000 tests a year at any given time. Quadrillions. Millions of billions of landing page permutations are live, meaning two customers in the same location are unlikely to see the same version. So yes, they're going nuts. They're testing every single thing they can imagine under the sun. And because the experimentation platform is easy to use and low cost, people run wild. So where's the discipline? How can there be a thousand different ways to innovate on a homepage? And are these tiny tweaks really doing anything to the person? I wanna bring in our own experiences when we're dealing with products. There's a lot of anecdotes that suggest something is deeply wrong with our interface design right now. For example, I stumbled on this Mastodon toot that's tweet for Mastodon that reminded me of this exact conversation. To quote, when I was a smart ass computer nerd in the eighties and nineties, an eternal theme was friends and family sheepishly asking me for tech support help and me slowly, patiently explaining to them that computers aren't scary. They're actually very predictable, but that's not true anymore. User interfaces, lag. Text lies, buttons, don't click. Buttons don't even look like buttons. Panels pop up and obscure your workspace and you can't move or remove them. A tiny floating X in a few horizontal lines is all you get My tech support calls now are about me sadly explaining there's nothing I can do. Computers suck now. They run on superstition, not science. It's a real tragedy for humanity and I have no idea how to fix it. That was very dramatic. But I think it captures a lot of our own frustrations when we're dealing with technology these days. I think it's safe to say that we have all experienced these frustrations with technology coming from these platforms. And if these platforms are constantly tweaking things and using experiments to measure our responses, surely things should be unequivocally getting better. Why aren't our frustrations registering in the data? So that's what we want to talk about today. What's your overall take on this earnest experimentation, prototyping. How do you see that Helping the innovation process and sometimes maybe even causing it to be undisciplined and unfocused.
Ernest:This actually brings me back to our very first episode where we talked about ourselves and our backgrounds. And I think you had asked, I had mentioned that I got my start um, at university learning how to design multimedia interfaces and specifically interfaces for CD ROM based experiences. And you had asked if that um, helped inform my eventual work in product innovation around physical products. And I think it did. And, to the point you're making here, I do think it, it informs my perspective on this question as well, in that if you're producing a physical good it imposes some very real constraints and some very real discipline. You can't take back that physical product and revise it uh, after it's been shipped. But still, I think that my perspectives on this topic do apply beyond physical goods as well. And actually a lot of my ire I guess when it comes to this concept of experimentation is directed towards the idea of the minimum viable product or MVP that's uh, become so popular in the digital domain and is really starting to creep over into the physical product creation world as well. For anyone who's not familiar with it, the concept of minimum viable product was introduced by a guy named Eric Reese as part of his broader lean startup methodology, and he defines. The MVP as the version of a new product, which allows a team to collect the maximum amount of validated learning about customers with the least effort. And I think on the surface that sounds laudable, right? Wow, that's, that sounds like a great thing to try to do. But the problem is that, at least in my experience in the real world, it's often implemented as the minimum achievable product as in, you know, what can we do within the constraints of an arbitrary deadline where that deadline is usually defined by someone focused on business goals. And the achievability is defined by someone with engineering goals. And typically what I've found is that the customer is completely left out of the equation. And it also brings to mind an example that a friend shared. He had worked at one of the big automakers and he said they had this system in place where of similar to booking.com. Anyone can run an experiment in their case, anyone could offer up a suggestion to quote unquote improve a product. That would be, you know, anyone in the design part of the organization, engineering, even down onto the um, assembly line. But really what they meant by improve was make it cheaper. And so you would end up with these, death by a thousand cuts along the way, someone would make a suggestion and, oh, actually, yeah, that does reduce the cost and it doesn't lead to any issues, so let's run with it, and 1, 2, 3 of those, maybe that's okay. But as you go, across the years that it takes to get a product from concept to actual production, you'd end up with something at the back end that bear. There's no resemblance to the original concept because you had these series of decis decisions, experiments, being conducted and decisions being made that would degrade the product because they weren't rooted in a customer's perspective. So I think that's my concern with this approach and something that I've seen firsthand is that you end up with these experiments that really have little to do with the needs of the customer, which is exactly what you were talking about. And really are rooted in imperatives of other organizational, requirements, right? Whether it's engineering or finance or whatever the case may be. And I think that. In a boom time, or during periods of low interest rates essentially zero interest rates that could work, but I think that certainly in our current reality where money is much more constrained, it's much less viable as an approach because the whole minimum viable product uh, ideology assumes you're gonna get a second chance. You know, It's all based on the idea that you're gonna have more opportunities to iterate on that product. But you know what, in today's environment, you might not, you might just get that first shot. And if you put out that a product that, building on Reid Hoffman's quote, that's embarrassing um, in the first go round, you might not get a second shot at it. So I really, even in periods of the boom times, I wasn't a fan of the concept, but. Today I think it's an irresponsible approach that maybe it'll work. I think there's gonna be chances, opportunities where it does work, but I think there's a high likelihood that it's not gonna work.
Joachim:Yeah I like the death by a thousand cuts. I'm surprised to hear that would happen in the domain that is so physical and so difficult to get right, which is the automotive industry. I've experienced so many versions of that in tech platforms, and in fact, one example that comes to mind is one where we had, backend systems that had been tweaked over many years. Each tweak had been tested in an AB experiment and. Obviously each one of those had passed some sort of statistical test. And so we had many, many, many, many, many components inside of the system that were just being added to this code base And the system became illegible at the end it was so incomprehensible We ran an experiment where we shut everything off and maybe unsurprisingly everything, looked better without all of those things. So it was a death by a thousand cuts. The complexity of all of those extra little tweaks that seemed by themselves really harmless and really cool and easy. The classic low hanging fruit when they start interacting with each other and when they are part of this single experience you'll be surprised it could actually be a really detrimental final product. And like you said, no one is keeping an eye on the big picture, which is at the end of the day, there is a person at the end who is experiencing this product. So yeah, death by a thousand cuts, I think is, I. very very easy to justify. And you wouldn't call it that, you would just call them incremental tweaks. Oh, this is an incremental tweak and we tested it, and so on and so forth. And so you're on this path where you've lost sight of the longer horizon Actually Cory Doctorow, who is a great technologist and writer and novelist as well, coined a the word enshitification, meaning something that starts off really engaging and really great, but then slowly gets degraded bit by bit until it reaches, and I'm adding this color to it. It reaches a critical threshold when it becomes just absolute garbage. But, it's like the frog in the water. You're just raising the temperature very slowly and everything's fine. And everything's fine until it's no longer fine if your experimental measurement framework isn't disciplined, which in most cases it is not, and it's not capable of looking into the future, what the future consequences are, you will find yourself in this position where each success of tweaks feels great, but really you've been destroying the thing and you're gonna boil those frogs very soon.
Ernest:Yeah, I, that's such a great point. I think too, something I've seen, which you've touched on, is this increasing atomization of corporations. Where, as they grow, they, the. Continue to specialize and specialize to the point where you'll have a product manager focused on just this one little, 300 by 300 pixel part of the overall page. And, they want to be able to run their experiments on that little portion of it, but. In that process you've got these hundreds of teams all doing their own thing, and just to your point, you end up with this thing that becomes completely incomprehensible for the end user because there's no one looking over the whole of that experience and ensuring that it's still delivering, that job to be done for the customer. And so I think it is so vital to maintain some function, whether it's a person or a small group of people that can continue to have this overarching perspective and overarching focus on the customer experience in my, professional life that's been called the product manager. It might be called different things in different companies, but I think that's the, you need to have that role because absent that. You're just, there's so many rational reasons to run all these different experiments, right? There's really good arguments to make for all of these things that booking.com has done, all these other companies have done, but without someone there to act as a safeguard and, looking out for the interest of the customer, I think you just naturally, inherently are going to see things go off the rails. Like, you know, We're seeing now in that to the point that people actually have language for it now they call it in enshitification because it's just so common. One approach that I've seen that I think could. Help to combat this trend is actually something from the world of uh, the creative world. It's from um, Pixar, the World of Cinema. And it's interesting because in some respects it might seem really similar to Reid Hoffman's quote about being embarrassed by the first version of, of your product. But. Fundamentally it's very different. So this comes from Ed Catmull, who co-founded Pixar and was president of Pixar through its most successful years. He's since retired. Uh, He wrote a book some years back called Creativity Inc. And we'll provide a link to that and in it. So this is where it might sound very similar to Reid Hoffman's quote, but in it, in the book, Catmull explained that early on, all of our movies suck. And he said, that's a blunt assessment. I know, but I choose this phrasing because saying it in a softer way fails to convey how bad the first versions really are. I'm not trying to be modest or facing Pixar films are not good at first. And our job is to make them so to go, as he says, from suck, to not suck. And you know, and obviously it's a different situation that, when you're making a movie, you don't have a second chance, right? Unless you get a sequel, but you don't have a second shot at that first film, at that film. You, the idea that you can come back in and re revise it is not applicable. But I think, you know, as I said earlier, that's kind of the whole problem with the MVP approach is this assumption that you're going to have a second shot. I think this discipline that's um, displayed by Pixar and I think other great product companies is something worth trying to mimic. I mentioned Catmull is really agreeing with Hoffman's. Take that the first version of the product is gonna suck, right? Is gonna be embarrassing. But unlike Hoffman, they don't release that first draft. They don't think that's the version that you should put out into the world. But the, I think the thing that maybe is a good learning here is that the way that Pixar. Was able to get from that first version that sucked to the version that ultimately went out to the world that was great was through this process and culture rooted in candor. And Catmull talks about that a lot in his book as well, that um, candor could not be more crucial to their creative process of Pixar. And it's because that's the only way that they found that they could actually make that. sucky product in the beginning, better by being honest with each other. And one really important me mechanism they created to facilitate this ca candor was something that they called the Brain Trust, which was just a group of trusted colleagues that got together periodically to review the progress of any given Pixar film during its development. And as Catmull explained, the job of the Brain Trust was to push towards excellence and root out mediocrity, forgetting about any technical issues or budget issues or anything else? It was all about the story. And how do we get this to be an excellent story? And a really important thing about the way the Braintrust worked is that it actually had no authority. The director didn't report to the Brain Trust. They weren't his boss. And so they weren't required to act on any of the suggestions given by the Brain Trust. Instead, it was the director's job to take that feedback and decide for themselves what they were gonna do about it, and. Based on that dynamic, the suggestions that the brain trust provided weren't prescriptive. They weren't mandates, they were just advice provided by peers, but in a way that was very forthright and honest. But always rooted in that shared goal of making the best product possible. Uh, And then another important thing was that, that Brain Trust were able to view the project with the luxury of perspective, which is something that is really is very much a luxury when you're, neck deep in a project, right? It's very easy to lose perspective. So that was another really important function of the brain trust is there were people who weren't working on the project so that they could look at it with objective eyes, but then bring the, their, pool of experience to you. So I think that's that's something that I've seen firsthand as well, that in products whether it's physical products or digital products, it's so vital to have. This feedback mechanism in-House that's rooted in judgment and experience, not just, Hey, we ran this AB test and, incrementally more people per preferred this version than the other one. It's about something we've talked about repeatedly over our episodes, which is this importance of a point of view, um, Rooted in experience, right? And so Pixar found a way to create a system, a process around that that could help them to get those, their products, their films from that first version that sucked to the ultimate version that in more cases than not was excellent. So I think there are ways to do, this is basically my point here. It's not just it's not just experimentation or nothing. There's other processes you can place put in place to get you to better outcomes.
Joachim:I think what you are hitting on right there, Ernest, is exactly the delineation that we would want to have in our minds, which is the innovation process. And the development process has to be inherently experimental. It you are flying blind, you're trying something new, and to a certain extent, if you're able to reduce it to a simple AB test. Maybe you're not pushing the boundaries that much. If it is so straightforward to just spin up an experiment It suggests that maybe these aren't such groundbreaking ideas that they don't require you to rethink the entire framework that you're working with. When I want to do something innovative, I want it to be different and groundbreaking. So that process is experimental. And I remember as also when I was in research, I worked with a far more senior colleague when I was starting out as an assistant professor, and he was, I. He had tenure, so he had a lot of security and insurance. That's, and I will come back to that point'cause I think that's important as well. But what was great about him was he had no ego in the process of innovation. He just threw things on the board and he did that. And I remember once catching a basic arithmetic error. And I said, oh, I don't think you can do that. And he is ah, nevermind. We're gonna try things, throw things on the board, and we're gonna react to it. We're gonna respond to it. And sometimes you stumble on something very powerful. And I think the pixel example is a beautiful example of that. They're exactly right. What's interesting as well, related to my point where I said about tenure. I think you can only have a candid conversation with a colleague or a teammate or someone who's produced something. If you trust each other and you are all very safe, meaning literally your job is safe, that we all agree that this is actually difficult, what we're doing here, it takes time to figure stuff out and you create an environment that. It doesn't have this type of MVP approach, where you say, I want to try this very ambitious thing, and you have the safety to try and do that. And as you said, that in boom times is much easier to do. In hard times people are less likely to wanna wait to see what happens, but it's exactly the other way around. It's in those tough times that you really want to have the thoughtful, longer run project and the process to be deep and well thought out before you go out there, because you probably only have one shot. If everyone's money is constrained, then your customer's money is constrained and you're only gonna get a one shot at bringing them in and convincing them that this is worthwhile. So the long and short of it, I think of it is innovation involves experimentation. The prototyping and the internal stress testing of the idea is critical, but it only works well when you trust the people around you and you feel safe sharing those ideas with them. I think safety, it sounds very touchy feely, but it's a really. Important aspect of your company. And so when I hear something like Brain Trust already creates in my mind a certain hierarchy to it. In all of this, the overarching piece that's coming out is, of course, the person that is going to be consuming or experiencing this thing is the person that matters, right? So the point of the brain trust at Pixar is precisely to be that outside voice, and it's supposed to be the voice that says to you, look, I know you're very enamored with how, I don't know how this light ray is bouncing off this surface, but ultimately your viewer won't care. And The brain trust role is, as you alluded to, to step outside of the weeds, look above it and remind everyone that what they're making here is entertainment and the job to be done is to entertain. And so what matters is not that all of those technical details, which might be awesome and are really cool, but they're not for the end user. And that reminds me a lot of the tech platform thinking, a lot of the excitement that people, especially in engineering groups, they get very excited about technical innovations. Your platform systems and the infrastructure that operates at machine time, right? So you want to measure everything in milliseconds and microseconds and all of those things. But your actual end user, the human being that's responding to you, is responding in human time and the implication of that is you should be able to explain to me how the human, the person experiencing this tweak is going to respond to it. And that goes back again, right back to jobs to be done. Jobs to be done was a causal chain of events needed to be illuminated so that you could then innovate. And similarly here, if you have a tweak for the system, it doesn't matter that the machine is doing all these things, it's doing it faster and lower latency, dah. dah. It needs to come back to the human response time. The measurement framework should involve all of the dimensions that you think you're influencing. If you are only looking at one dimension and it makes you look good but you look at all the other dimensions of how a user is interacting with what you've done, and those signals don't look right, you have not, you've not really done the thing that you claim you've done. I think it comes back to the, that causal chain of events that leads a person to be engaged and enjoy. And that feels like a very unifying theme with what Pixar's trying to do. Trying to a person be the customer there. And I think there's a way to do those things. I've, I try and do that in my quantitative work as well, is to represent that human being. I don't care about what the machine is doing. What I care about is, how a human being is responding to that flow as they're being pushed through that system. And surprisingly, you'll find measurements and KPIs that you never thought about coming out of that, that become really impactful for identifying whether something actually has done something good. Yeah. So I just wanted to pick up on those things that you had been saying, Ernest as well.
Ernest:Oh no, those are all amazing points. You made a bunch of amazing ones. It brings to mind your point about this tendency to get really attached, as a domain expert, you, it's very easy to get attached to something very specific. Like you mentioned, an engineer might get very attached to some algorithm that might speed up, some process by certain milliseconds. And the way Catmull talked about it was that their job as a brain trust was to weed out the ugly babies. You know, No one. Wants to get rid of their own baby,
Joachim:Yeah.
Ernest:but everyone has their babies over the course of a project. But they, this group, because they were outside of the process, were able to bring that perspective, but also that they had the expertise to understand what was really important, what the customer really did want that they could find out where those ugly babies were and highlight them so that the director could then weed them out, weed out the ugly babies. Another thing, what you mentioned brought to mind was just getting back to this point about um, having a point of view and prioritization, figuring out what the. The key problems to solve are I think that is coming back to this minimum viable product concept. One of the things that I've found really troubling about that process is that it often leads to people solving the problems that are solvable instead of the problems that are important. And I think to your point too about that doing things this way become even more difficult in times of, in economic challenging economic times it really does become difficult to think more deeply about, what the real problems to solve are when. Times are tough, but exactly to your point, that's exactly when we need to focus on those real problems instead of just the checklist. Things that, we know we can get done and look good on your, annual review, but really don't help, deliver on the needs of the customer or even on the overarching long-term needs of the company. So, um, I think that's such a great point to highlight that in times like this are when it's most vital to. To get away from this sort of checkbox approach and dig into the real fundamental problems to solve and innovate against. And I think that's why you see oftentimes that companies that have the sort of perspective like Apple do often tend to innovate most effectively during downturns. You know, I think uh, looking back, I think the iPod was released during an economic downturn. I think the first iPhone was released during an economic downturn. It's because they have this discipline that even when times are tough, they're able to deliver really groundbreaking products because they've always, they're always focused on the right questions, the right problems to solve.
Joachim:Yeah, I agree with that for sure. The iPhone did come around 2007, 2008, which is of course the credit crisis that was hitting everyone, but yeah the device still, Had impact. It really it shifted the conversation on what a phone could be and what it should be. They were coming in with a very strong alternative, but they thought about it for so long, which is that's what we're getting at, right? We're saying innovation experimentation, but do it quietly first for yourself and use your careful judgment and expertise to, to build out what you think that product should look like. I had a few unrelated, but related, and I think this is, purely from a very technocratic perspective, I think it's very important to remember what actually happens when you run an AB experiment. This is gonna get a little philosophical, but basically when you run something, you're gonna do a statistical test, and that's gonna then end up with someone saying, there is a statistically significant effect in this experiment. Now, statistical significance is a pretty meaningless concept. So you assume at first there should be no effect. I put someone in a treatment, I put someone in a control, and my original null hypothesis, as they say, says that these two groups are identical. And if you find a difference between those two groups, you then ask yourself, is the magnitude of that difference surprising? And there's a little heuristic that people use to define what they think is surprising. So surprise means a little bit out there in the tails now. Tail event means something that's a low probability, but not zero probability. So that doesn't mean anything. In statistics and probability theory, it is totally possible that I could flip a coin and get all heads in the first 10 flips. So when it comes to measuring effects, even if there is no meaningful causal relationship to suggest this intervention does something, I can still detect statistically significant effects because all they're saying is, oh, here's a surprising magnitude that I wasn't expecting. If things should be zero, but. I could also have situations where the hypothesis is true and there is no measurable effect, or the hypothesis is false. And there is a surprising effect because probability is all about surprise. Like all kinds of surprising things can happen with probability. So when people, parade their statistically significant results around you have to take it with a pinch of salt. And so coming back to the earlier point of you need to be able to tell me convincingly, what is happening to the human being in the loop here? What is the customer actually experiencing? And. If you have a convincing enough story, then we can say, okay, let's go out and let's measure those other dimensions that you failed to mention now, and let's see if that's actually taking place. And then if we gather enough of those statistically significant, surprising magnitudes, maybe then we could say, okay, there's a good chance this could be true. And if you look closely at the scientific literature there's a replication crisis happening right now precisely because of that abuse of statistical significance. We're seeing people not able to get the same effects in the same experiment with the same settings, and that's important to remember for. You as a practitioner as well. Maybe you did get lift this time, but that could have been a fluke. And so there's also a good chance, like I described before, when you switch everything off again, you might actually be in a better off situation in that experiment because these numbers can move around. And in the absence of a compelling causal chain that describes what's happening to the customer, at the end of the day it's all just metrics trading, and so I find that in my own practice as a data scientist and economist, I really try and center the behavioral aspects and the measurement of those behavioral aspects. So there is a way to do It's a little bit harder, but just looking at a KPI and hoping for the best and seeing that it went green. It's going to happen. You're gonna be able to find an effect. And I have seen all kinds of stuff where people change the window of the experiments or the number comes up, or partitioning the groups in certain ways. It gets funky really fast. And I think that's important to call out as well, because if you're new to this world and you hop into this thing and you'll think there's some sort of scientific rigor going on here that you need to be exposed to, otherwise you don't understand what's going on. Sometimes you just have to call bullshit and if you can't understand why something works the way it does, you should feel free to call bullshit if you feel safe enough to do that. But statistics are very tricky and once you've spent enough time with statistics, you will be surprised how easy it is to get things to look one way or ano another way. We take for granted that there's some sort of scientific approach that's handling the inference for you and you can outsource it to that thing. So again, empower everyone who is non-technical, to just ask the basic questions about what is the human feeling in this thing.
Ernest:That actually brings to mind two quotes. First it's something from a famous person in advertising named David Ogilvy who noted, and this was probably in the seventies when he said this, but he said, I've noted an increasing tendency on the part of marketing professionals to use research as a drunkard, uses a lamppost for support rather than illumination. And to your point, I do, I have seen that a lot over my career. People just very, cherrypicking data. To support an argument. And then the second quote it brought to mind was from Steve Jobs, and he said this in that lost interview documentary we talked about a little while back. He said, I don't care about being right, I care about success. And there are many anecdotes from people that said that was actually true. He was really surprisingly willing to change his mind. Um, He had very strong views, but loosely held. And if you could make a compelling argument, he'd be willing to change his mind because his ultimate goal was success. Delivering a product that would succeed with his customer. So I. I think that there, I've seen that firsthand over the course of my career. I think a big part of the success, such as it is that I've had in product creation, has been based on my willingness to be honest with the feedback we were getting back. Not just cherry picking, not just cooking the books to, to get to positive feedback because that's ultimately going to play out once the product reaches the marketplace. And that leads me to something that might sound like it's counter to a lot of what I've said so far today. And it's another Steve Jobs-ism, which is, he'd love to say that great artist ship. That, as talented as you might be, you're judged by what you eventually put out there into the marketplace. And that's a reality, right? To, as a company, if involved in the commercial enterprise, you have to put products out there that can generate revenue. But what I'd say to that is, for me, the approach that's driven success most consistently is not that minimum viable approach. Um, I don't have a acronym to uh, explain my perspective on this, but basically it's put the best product you can out there given your constraints of time and budget, because you're always gonna have constraints and, deadlines can be arbitrary, but they can be important too. You know, Maybe there is a key date you have to hit because there's a, an event that's surrounding that date or, you do have financial imperatives that are associated with that date. So those are realities that you're gonna have to deliver against. But to a lot of the points you've been making, Joachim, I think the key is to root your decisions on the questions that matter, not just on what you can achieve, which is really what I think MVP is often, how it's often implemented, but on the things that matter. Within those constraints of time, budget, and the things that matter, what's the best product that I can create, solve those problems instead of just what can I achieve and what's gonna maybe show an incremental effect in some ab test or experiment that I can cook up.
Joachim:Steve Jobs man, always with the pithy one liners,
Ernest:He full of them.
Joachim:I think need to spend a good chunk of time, you and me, and us just coming up, going through the podcast scripts and coming up with one liners for everything so that we can just have a book as a, these are our one summarize
Ernest:Yeah. Yeah. That should be uh, once we get to a hundred episodes, maybe we could pick through them and find our one liner.
Joachim:There's goal a goal.
Ernest:All right, now that you've heard our perspectives, we wanna hear from you. Do you think Reid Hoffman was right in his assessment that if you're not embarrassed by the first version of your product, you've launched too late? Or do you agree with our takes that this sort of fixed in, in post attitude is a road to mediocre products? We wanna hear from you, so let us know at LearnMakeLearn@gmail.com. Now let's move on to our recommendations of the week. Joachim, what has you excited this week?
Joachim:Again, a funny one for me. I think I had some friends come back from Japan recently and they were, they're super excited and it got me thinking about what it was like. I'm half Japanese One of the things that excited me when I was visiting Japan, when you go to like a seven 11 or a family mart or Lawson, the triangular rice ball, the onigiri. The packaging for that is absolutely incredible. So if you haven't experienced it, you basically have a triangle with numbers on it, and you can see the rice ball is there, the rice is there. You can't see the rice because it's wrapped in the dry seaweed. But the packaging is such that the dry seaweed is not touching the rice, so it remains crispy. And then you peel the central plastic line and then pull it apart, and you have crispy seaweed with, refrigerated rice in the filling. But man, that nty stays crispy throughout that. And it's a product that Affects me in many ways. One is I just imagine the years of innovation that went into designing something like that and getting the machines to make and be able to deliver that. It, there's something really obsessive about the people that were working on that to get it. So right. So tremendous amount of respect for that. But then another part of me asks, is it that tragic if the nori is not that crunchy anymore? Was it really necessary to go through this whole rigamarole and probably build a new machine that can make that wrapping? So I experienced that every time I unwrap one of those onigiri, and I just ask myself, man, this is so great, but was it necessary? So that was my submission for the recommendations. If you have a chance, if you somehow can get to an Asian supermarket, you should grab one and just marvel at the ingenuity of human beings that worked on that. But then also ask yourself, was it all strictly necessary as well? So yeah, that, that's my submission. I'm sure you've encountered these packages, but you've also visited Japan recently Ernest, right?
Ernest:Yeah. Yeah. Oh I'm a big fan of conbini in Japan. I I, there's so genuinely convenient and the food is often really good. And I share that love and fascination with the packaging. And something that I think a lot of lay people aren't familiar with, but in product, physical product creation, a really important consideration is order of operations, sequencing of steps in a manufacturing process because um, you know, that's really important. There's things that you have to do in sequence, oftentimes in order for the end product to be possible. And the packaging it, as you were talking about it, it just brought to mind that, wow, that is a serious order of operations problem to solve to, get the rice in first. And then the packaging between the rice and the naughty as you were explaining. It must be a really interesting machine on the backend that actually does that,
Joachim:Yeah. But was it all worth it?
Ernest:you know, um, Seven 11 is also such a, it's a fascinating story. Maybe we could dig into it in the future, but in that. It was licensed, seven 11 started in the us It was licensed by a company in Japan. Became very popular in Japan, I think largely based on their focus on the needs of their customers. And they became so successful that they acquired the parent company. What is, yeah. So what is seven 11 now globally is owned by the company that was previously the licensee of the seven 11 brand in Japan. And what's excited me is that, I think it was just last week, the head of seven 11 Corporation, I forget what the corporation name is, but he noted that he thinks the experience that they've created in Japan is something that could be applied more broadly globally. So it would be a dream if they would bring that focus on the needs of the customer. And I would love to see the uni brought to the us but um, even if it's just that focus at least uh, and making. Convenience stores like seven 11, truly convenient. That would be fantastic. I would love to see that happen in the us.
Joachim:for sure. Seven 11 right now is, it's not comparable to the Japanese ones be kind. That's all I can Yeah.
Ernest:That's a fantastic uh, recommendation of the week. Um, So yeah, on my end it's something pretty different. It's actually uh, a show on Netflix, a documentary called The Greatest Night in Pop, directed by Bao Nguyen. And it's about the creation of we Are the World, the song that was released in support of famine Relief for Africa. And it was inspired by a ADE that came out of the uk. But um, this one focuses on uh, the, we Are the World version, which was more focused on the American artists. And I thought, if you're interested in music, I think it's a really interesting thing to watch. But I think for people interested in product and particularly people interested in the product management and product marketing role I. A real, it's a masterclass in what it takes to work with, effectively work with domain experts and particularly people in creative fields. And'cause what you see is Lionel Richie is the, I guess the star of the episode and I didn't realize, but he really was instrumental in pulling all this together. And he co-wrote the song with Michael Jackson, but. You can see all the steps in essentially creating a product. Some people might think it's crass to call music product, but it is ultimately, and he thought of it in that, about it in that way too. You could see in the very beginning knowing that their goal was to reach as many people as possible so that they could raise as much money as possible for famine relief. It. It informed his choice of meter and beat for the song. It informed his choice of the artist that he wanted to pull in to perform the song. And then even to the point of sequencing, even the sequence in which they engaged artists to, increase their odds of getting as many of these artists as possible. So it's such a, if you look at it through that lens, I think you can take away a lot of lessons. And then during the recording itself, there's some things that Lionel Richie and Quincy Jones, who was the producer for the song, some things that they do that I think are great lessons as well for people who work with domain experts to help to, guide them along in the process of creating a product. I'll just cite one example. I guess two examples. One is the importance of momentum. So they talk about that a lot and in their case, they had one night to record the song. So it was really important that they keep things moving. But I think that applies to any kind of a product creation experience where it's so important to keep that momentum going, to keep people engaged, the those domain experts that you're working with. But then there was also a key moment that ties back to some of the things you were talking about as well, Joachim in that. Before they started recording, they had the person who had spearheaded the farm aid project, I forget his name now is English guy. Come in and just speak to why they were doing this, what were the stakes here on the ground? The people, suffering, dying from starvation, he gave a beautiful like, I don't even know if it was like a minute long, really brief speech just to set the stakes and let everyone in the room and to ground them, and just like you were saying, to focus everyone's attention and efforts against that ultimate outcome that they were working towards. And I think that was so important to the success, the ultimate success of that project as well. But, so I think, you know, even just for entertainment it's a great watch. If you're interested in music, it's a great watch. But I think if you're interested in product creation, it's also a really interesting watch that you could take a lot away from.
Joachim:I found that it was really interesting your point about the sequencing of artists and yes, we don't like calling music product, but it is. But the sequencing of getting these very big names to perform altogether on a piece of music is pretty tricky. But that reminded me about. Nas' Illmatic album. Okay, so in that one, I think the story was that they had already had producers working on some songs, and then they showed them to Q-Tip Q-tip got incredibly competitive. And so that led to a production battle happening. For this album, the Illmatic an incredible hip hop album. They refer to as the metal sharpening where they're trying to outdo each other with their beats productions and something incredible comes out of that. So again, you have to be pretty smart about the sequencing.'cause sometimes it is good to sequence as opposed to have everyone in the room at the same time. Let someone make a statement and then have someone respond to that statement and then do that back and forth. You get something pretty credible. So when you said of artists, that made me think about that as an example. I think I watched a YouTube video that described that in great detail. We'll put that in the link for the show notes. It was quite interesting.
Ernest:Oh, that'd be awesome. One last thing I'll mention, something you said reminded me of as well was in terms of the sequencing. It comes back to this point of judgment and perspective and bringing expertise to the table. The judgments that Lano Richie and Quincy Jones um, I think uh, the person who was the producer also had a voice in this, but the judgments they made in terms of the sequencing of the artists who sang was also so interesting to see and hear their thought processes. And, they. Ended up with these juxtapositions that I think for a non-expert might seem so weird. Paul Simon before Willie Nelson uh, you know, or I'm not sure if that was exactly right, but these really interesting combinations. But once you hear it come to life, you are like, oh my goodness, that yes, that is amazing. That's such great sequencing that they came up with. So it was a product and they didn't test any of that, right? It was a product of their expertise, their hard won expertise over many years of um, of making great art. I thought that was a really interesting takeaway as well. All right, I think that does it for us. Thank you so much for joining us here at Learn Make Learn. As I mentioned, we want to hear from you, so please send any questions or feedback to LearnMakeLearn@gmail.com and tell your friends about us. Now across our two next episodes, Joachim and I are actually gonna interview each other with a focus on our careers for the benefit of anyone interested in pursuing our career paths. We're gonna start with an episode we're calling What the Bleep is an Applied Micro Economist where I'll interview Joachim to find out what an Applied Micro Economist does, how to become one and more. And if you have questions about this career path, please send them our way and I'll pose them to Joachim during the episode. Alright, well thanks for joining us today and we hope you'll join us again next week for Learn Make Learn.
