Evidence that quality is impossible at scale
I believe that quality becomes impossible as software grows. Here’s a collection of quotes that support my belief.
Patrick McKenzie tweeted about the culture of quality at Stripe. Patrick ended with “none of it is sufficient. We are actively dissatisfied with where quality is at the moment”.
Benji Taylor of Family and Honk said that “big corporate software seems to be getting buggier each year, while indie software [seems] to be getting smoother and more reliable”.
Some notable people replied to Benji’s tweet:
“the attention to detail is inversely proportional to the size of the team” (Jordan Singer)
“details are best cared for when they can be thought about holistically from a macro point of view. large organizations, by design, don’t allow for that.” (Brandon Jacoby)
“craft just becomes more and more expensive the bigger your product/team is” (George Kedenburg III)
“I wholeheartedly agree high craft is excruciatingly hard to scale” (Kurt Varner)
“craft was impossible to maintain at Twitter as we rapidly grew” (Sean Thompson)
In their article Optimizing For Feelings, The Browser company says that “as our everyday software tools and media became global for the first time, the hand of the artist gave way to the whims of the algorithm. And our software became one-size-fits-all in a world full of so many different people. All our opinions, beliefs, and ideas got averaged out”.
David Khourshid, founder of Stately, says that “adding even a single ‘feature’ (new states + transitions) to an app can introduce an exponentially greater number of ways the app can be used”.
Benji Taylor tweeted that “something all these products have in common is that they are … built by small teams… Too often products lose their soul once they reach a certain scale”.
Paul Stamatiou said that “responses here so far confirm my theory that… high-quality software products are only made by small teams”.
Nathan A. Curtis said “I love Figma and value the experimentation they’re doing to add more and more. But I can sense it… their UI is ‘Adobe Creative Suite’ing. And that carries a usability cost.” Note that he said this in April of 2022, months before Adobe acquired Figma.
Repeating an argument Steve Jobs made, Marty Cagan said, “as a company gets bigger, product historically became less important… Good product people don’t want to work there anymore… They go to a company that values product. I think that’s a better explanation than any other I’ve heard.”
Steve Ruiz said, “unfortunately taste doesn’t scale”.
Packy McCormick wrote about this. Here are some good bits:
“Magic can, even must, be a strategy for startups. It’s something they can uniquely create, that incumbents often can’t.”
“Figma was able to choose … sufficiently advanced technologies at that moment in time without worrying about two decades’ worth of technical debt. As a young startup, it could focus on creating magic and ride that magic until, one day, it, too, had too much technical debt… until the demands of the public markets forced it to make the kinds of practical business-driven decisions that kill the magic.”
“whether Adobe accelerates the downfall or not … the magic would have, eventually, degraded anyway.”
“A startup, if it’s lucky, creates magic, turns that magic into dollars, and transitions to life as a successful Big Muggle Company, capable of enormous profits and power but no longer able to conjure magic.”
“I was planning to bemoan the loss of magic in the products I once loved and try to figure out why the technology I use isn’t as magical as it once was. But that’s not what’s happening. What’s happening is more fascinating. The last generation of magical products has matured, turned into Muggles, and in so doing, has cleared space for a new cohort of Magicians.”
“Blame familiarity. Blame business models. Blame technical debt. Blame multivariate testing and optimization. Blame the cruel coldness of the market. Blame the hedonic treadmill. Blame our unending pursuit of shiny new things. Blame human nature.”
Paul Graham said, “One theme I’ve noticed talking to people in Silicon Valley this trip is that Google has become terminally bureaucratic”.
On the Tim Feriss show, Stripe co-founder Patrick Collison said “Just think of any big, major company … for whatever reason, they can’t turn capital into good software. And it would be immensely valuable for them if they could. But they can’t. Or at least, they don’t. And I don’t think it’s for lack of trying or lack of realizing this. And so I think actually, small companies don’t realize how much of an upper hand they have here where if they can create a product that is so much better … it’s going to be really freaking hard for a big company to effectively compete because, again, this organizational transformation into being good at software is just so profoundly hard.”
Dan Luu has said that “while it’s not exactly common, there are plenty of small firms out there with a culture of excellence that generally persists without heavyweight processes or big incentives”, but that “this doesn’t work at big tech companies since they’ve all gone through a hypergrowth period where it’s impossible to maintain such extreme (by mainstream standards) cultural values.”
Kevin Yien says that “you know you are no longer a startup when your [organisationl] structure prevents well-intentioned people from fixing problems”.
Cemre Güngör, a product manager at Instagram, says that “Big tech companies hire expensive senior designers but still end up with poor design because it’s much harder to make structural changes to incentives and slavish metric chasing. There’s great talent in most companies already. They’re just stifled by the organization around them.”
Craig Mod said “… the three primary pieces of software on macOS are probably Finder, Safari, and Mail. To have two of these show signs of instability is … just weird. It shouldn’t happen, especially when these are new, critical bugs in decades-old programs. It makes you wonder … what’s broken with the development cycle to allow for these bugs to ship… we’re seeing once-reliable applications suffer from feature creep and bloat. Perhaps this is endemic to the very nature of public companies and their conflation of features with user growth? For example: Dropbox has gone from a svelte, hyper-reliable file syncing service to a bloated curiosity that pegs the cpu at 200% for unclear reasons.”
The Financial Times’ Tim Bradshaw said “… developers that respond by bundling ever more features into a super app risk frustrating users. Single-purpose apps are faster and easier to navigate. By contrast, the super-app approach involves cramming as many features as possible into one bloated piece of software… Spotify’s interface, for instance, has become cluttered and confusing since it added podcasts… This is my real issue with super apps: most solve a problem for the company, not the customer.”
Adam Michela, on the topic of Airbnb’s misleading pricing design, says “the problem is that Airbnb’s chief metric is Nights Booked (units sold) and manipulating users with basic pricing psychology moves units. It takes a Jobs-like chief to prioritize user experience over business performance.”
Adam Michela also said that “We’ve helped hundreds of 2-200 person startups build products … I can confidently agree that the ~30 person moment (5-10 engineers and 2 designers) is the quality sweet spot.”
Dennis Brotzky, coufounder of Fey, says that “the smallest and most focused teams deliver the best results. Keep your team small and give them time and freedom.”
Michael Karnjanaprakorn, who founded Skillshare, says that “quality and speed goes down as team size gets bigger”.
Rasmus Andersson says that “I’m convinced that—generally speaking—there’s an inverse correlation between number of people working on a software product and its quality (from end-user perspective)”.
Kyle Rubenok, a PM at Microsoft, says that “the best iOS design is exclusively in indie/small team apps today.”
Cameron Moll says “quality and craft don’t seem to scale as well as headcount.”
Box Baxley says “Nothing about creativity scales—and it never has. Production scales. Creativity doesn’t.”
Hardik Pandya says, “Don’t be afraid of large teams. Be afraid of small & determined teams.”
Packy McCormick says “taste is a fickle thing on which to hang the fate of a company. At some point, growth pressures can get in the way of making sure that every detail is perfect, that every hire is great. Taste, like trust, takes a lifetime to build and minutes to destroy.”
Jasmine Friedl said “while these [measures of quality] worked for us, we acknowledged that it wouldn’t scale much past our small team; for example, a team with hundreds of designers shipping daily wouldn’t likely have the bandwidth for manual reviews.”
Paul Stamatiou wrote an article about craft which has some gems in it:
“That’s where the challenge of building quality products starts to creep in. The constant tension of shipping faster versus shipping better. Falling into a cycle of “Ship, then iterate” is a trap. … Things happen and that “fast-follow” V1.1 release or V2.0 you had imagined probably won’t. There’s always a new shiny initiative, re-org or new leadership hire that throws a wrench into things and changes all plans. Don’t rely on a future release to clean up today’s mess.”
“If support for quality does not come from the top, it’s far too easy for ICs to skirt over issues and pay a little less attention to details they think their reporting chain won’t notice.”
In a 1995 interview for TV show “Triumph of the Nerds”, Steve jobs said “It turns out the same thing can happen in technology companies that get monopolies, like IBM or Xerox. If you were a product person at IBM or Xerox, so you make a better copier or computer. So what? When you have monopoly market share, the company’s not any more successful. So the people that can make the company more successful are sales and marketing people, and they end up running the companies. And the product people get driven out of the decision making forums, and the companies forget what it means to make great products. The product sensibility and the product genius that brought them to that monopolistic position gets rotted out by people running these companies that have no conception of a good product versus a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts, usually, about wanting to really help the customers.”
Gergely Orosz says “Very few if any people and teams [at a large tech company] get rewarded for fixing low impact issues impacting 0.002% of users. Also few get punished by not doing so. At any Big Tech, incentives that drive promotions and performance reviews end up driving a surprisingly large amount of decisions. Most of these incentives are impact, revenue generation at scale & savings at scale. Fixing small issues fits none of these. In fact, fixing one-off issues is one of the most expensive things to do. It would - and does! - pull profits down. This is bad for business.”
Dan Luu says “The pervasiveness of … technical decisions made without serious technical consideration, is a major reason that the selection pressure on companies to make good products is so weak. There is some pressure, but it’s noisy enough that successful companies often route around making a product that works, like in the Mongo example from above, where Mongo’s decision to loudly repeat demonstrably bogus performance claims and making demonstrably false correctness claims was, from a business standpoint, superior to focusing on actual correctness and performance; by focusing their resources where it mattered for the business, they managed to outcompete companies that made the mistake of devoting serious resources to performance and correctness.”
Arvind Narayanan says “some products are sold directly to the end user, and are forced to prioritize usability. Other products are sold to an intermediary whose concerns are typically different from the user’s needs. Such products don’t HAVE to end up as unusable garbage, but usually do.”
Nick Lockwood says “development is like Zeno's paradox, where effort rises exponentially. You never actually reach completion, you just get ever nearer to it, with each unit of effort getting you fractionally closer… this fact applies *fractally*, to every feature. At any point in a product's lifecycle, it will be the same (or more) effort to fix a small bug or papercut than to implement a major new feature (or rather, to implement 80% of it, minus the last few bugs and papercuts).”
Alexandr Wang says “Unintuitive fact of engineering: Big teams (>8 people) ship less than small teams. Not only less per capita—less overall!”
On Twitter, Dan Luu said “… a sentiment I’ve heard from literally all of the highest impact/most effective people I’ve talked to at large companies: You can make a big difference, but you’re constantly fighting a self-sabotaging organization”.
In his article Tiktok’s enshittification, Cory Doctorow said “here is how platforms die: first, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die”.
George Kedenburg III wrote an article, “The Cost of Craft”, on this topic:
“Over time Instagram continued to grow and evolve. The product got bigger and more complicated as it accumulated more competitors and employees. Three years went by and I noticed that it was starting to feel more and more like the environment I left Facebook to escape — a place where metrics and growth mattered more than creating experiences that people love.”
“It appears inevitable that all digital products must eventually trade craft for scale.”
“The cost of craft rises with each additional person.”
“The secret sauce I was searching for was really just a perfect blend of size and talent… It’s simply so much easier to keep 10 people in sync than 10,000. Each additional person makes it harder for everyone to stay focused and in sync.”
Patrick McKenzie wrote that “… as software companies get larger … they increasingly make it difficult to get relatively simple software done, which leads to surprising results … You just need a landing page, but the process of getting a landing page done requires three quarters of lead time and sixteen reviews, and the Landing Page as a Service fits on a purchasing card and takes two hours…”
Adam Stoddard wrote about craft at scale, and said that “Funded startups and public tech companies need to keep the high-growth engine running, which inevitably leads progressively larger and more complex product offerings that attempt to attract more and more users. No single team can deliver on that, which leads directly to disjointed and/or competing teams, and a steady erosion of vision that’s backfilled by process.”
Joel Beukelman said, “After ~8 years at Google I have a deep respect for product at scale, but my $.02 is there’s some sparkle lost when a team gets too big.”
Chief Product and Engineering Officer Claire Vo wrote a post on this topic. Some excerpts:
“Almost everything about growing the size and scale of your team has immediate and detrimental impacts to the thing that matters the most: building awesome products.”
“… meetings, process, policy and operating models are necessary for high scale and trusted delivery AND YET teams that forget to actively and aggressively prune the worst effects of these things out of their company will soon regret it.”
Karri Saarinen, co-founder of Linear, wrote a post about quality and said, “As companies grow, they often have a hard time with quality, and usually just give up on it. [The] main reason is that quality is something which cannot be easily measured or defined. As the companies scale, the way they operate or make decisions [is] based more on measurements.”
In a conversation with Linear about quality, Dick Costolo said, “The best product teams will stay as small as possible, as long as possible. It just becomes too difficult for the product to look and feel like the outcome of a singular simple vision for what the future could be as you layer more and more and more people into the thing.”