Team Structure for AI — chapter audio overview

Podcast Script: Team Structure for AI Host: Welcome back to the GenAI Engineering Leadership course. This is Chapter 2 of 20, and today we're tackling something most engineering leaders underestimate until it hurts — team structure for AI organizations. In the last chapter, we covered how to hire GenAI engineers. Now the question shifts: once you've hired them, how do you organize them? Because the way you group these engineers into teams determines what they can build, how fast they can ship, and whether they can sustain velocity as your organization scales. Here's the scene to picture while you drive. You've got twelve talented engineers. Some train models, some build the application layer, some wire up infrastructure. Right now they sit on one team, and it kind of works. But tickets are piling up, cognitive load is spiking, and everyone feels stretched thin. What do you do? This chapter gives you the framework to answer that. For any team building production AI systems, this is a core competency — the kind of depth that turns an engineer who uses AI tools into one who helps design the organization that builds them. You'll practice this across six hands-on exercises, but first, let's build the mental model. We'll walk through three team archetypes, learn how to combine them into hybrids, and then cover the interaction protocols that make the whole thing hold together. Let's start with the platform team. Expert: Great framing. Before we dive into the platform team specifically, let me define the umbrella concept that everything else sits under. A team topology is just an organizational design pattern — it names the kinds of teams you have, what each one owns, and how they interact with each other. The idea comes from a framework by Matthew Skelton and Manuel Pais, and we adapt it for the unique coordination challenges that GenAI work introduces. Now, the AI platform team. The simplest way to think about a platform team is that it's a team whose customers are other engineers, not end users. Its mission is to reduce cognitive load on every other team that builds GenAI applications. Cognitive load, in this context, just means the total amount of stuff a team has to hold in its head to do its work — the domain, the tooling, the on-call duties, all of it. When you shrink that load for application teams, they ship faster and more reliably. So what does a platform team actually provide? Think of it as the self-service aisle of your AI organization. An agent developer shouldn't have to think about scheduling GPUs. A data scientist shouldn't be standing up their own vector database — which, by the way, is a specialized database that stores the numerical representations that AI models use for search and retrieval. An ML engineer shouldn't be writing model serving configurations from scratch. The platform team builds all of that infrastructure once, wraps it in clean interfaces, and lets everyone else consume it through simple APIs. The staffing model matters a lot here. A solid AI platform team has a mix of infrastructure engineers who handle things like Kubernetes and GPU scheduling, ML operations engineers who handle model serving and pipeline orchestration, and — this is critical — at least one developer experience engineer from day one. A developer experience engineer is someone whose entire job is making the platform easy to adopt. They write the documentation, they build the command-line tools, they make sure a new consumer team can get productive in under fifteen minutes. Why is this role so important? Because the most common failure mode for platform teams isn't technical. It's that they build powerful infrastructure that nobody adopts because it's too hard to use. The second most common failure is scope creep — the platform team starts building features for end users instead of capabilities for engineering teams, loses focus, and becomes a bottleneck. There are two practices every mature platform team uses. First, the internal customer interview. Every quarter, the platform lead sits down with engineers from each consumer team and asks three questions: What platform capabilities saved you the most time? What gaps forced you to build workarounds? What new capabilities would most accelerate your roadmap? This prevents the team from building what they think is needed rather than what consumers actually need. Second, a platform product board. One representative from each consumer team meets monthly with the platform lead to review the roadmap, prioritize new capabilities, and resolve conflicting requests. Without this governance mechanism, the platform team either serves whoever shouts loudest, or spreads too thin trying to satisfy everyone at once. One leading indicator of trouble: when a platform team owns more services than it has engineers, on-call burden explodes and quality degrades. If you have four engineers and six services, you cannot provide reliable coverage. That ratio — services to people — is a simple metric you can track. Host: Okay, so the platform team is the self-service backbone — it exists to reduce cognitive load for everyone else, and its success is measured by how much its consumer teams adopt what it builds. That brings us to the teams doing that consuming. If the platform team serves internal engineers, who serves the end users? That's the agent development team. What makes this team fundamentally different? Expert: The defining difference is pace and customer. The agent development team is the most product-facing team in a GenAI organization. Its output is the AI-powered applications that real users actually touch — chatbots, autonomous workflows, copilots, multi-step reasoning systems. Revenue and user satisfaction depend directly on what this team ships. And because of that, the team structure has to be optimized for something very specific: rapid iteration under uncertainty. Let me unpack that. When you build an agent system, you're experimenting with prompt designs, tool configurations, orchestration strategies, and safety guardrails. Every one of those variables affects quality, and you can't reason about them from first principles — you have to test them against real evaluation data. The cycle time from hypothesis to production test should be measured in days, not sprints. A typical agent team ships a new version every three days. That cadence drives every other structural choice. So what's the composition? You need prompt engineers who specialize in conversation design and safety guardrails. You need full-stack developers who build the application layer, the orchestration logic, and the state management. You need an engineering manager, and you need a product manager for user research and success metrics. But the most critical hire — the one that separates a real agent team from a demo factory — is the evaluation engineer. Let me explain why. An evaluation engineer is someone whose full-time job is measuring the quality of the AI system. They maintain test datasets, they run regression tests whenever a prompt changes, they design A/B test analysis, and they provide the quantitative evidence that separates "this feels better" from "this measurably improves accuracy on our evaluation set." Without this role, your team will optimize for demo quality — looks great in a scripted walkthrough — rather than production quality, which means handling the long tail of real user inputs reliably. Every agent team that ships to real users should have at least one dedicated evaluation engineer. Agent teams also need specialized review processes that traditional code review does not cover. There are three that matter. A prompt review is triggered any time a prompt template changes — because small tweaks can produce large behavioral shifts. A safety review fires when you add a new tool integration or change a guardrail. And an evaluation gate is the final check before any change reaches production — the evaluation engineer signs off that the new version passes the regression suite. All three should be blocking, meaning you cannot ship without them. There's a powerful practice called prompt review as code review. Just as code changes go through structured review before merging, prompt template changes go through a dedicated review process with explicit criteria. Does the prompt handle edge cases from the evaluation dataset? Does it maintain the established safety guardrails? Has it been tested against the regression suite? Treating prompts as first-class artifacts with the same rigor as code prevents the "just tweak the prompt" anti-pattern, where small untracked changes accumulate into large behavioral drifts that nobody can explain later. One last practice: the failure catalog. This is a documented collection of real production failures, what caused them, and how they were resolved. New team members study the catalog during onboarding to build intuition about the failure modes specific to your systems. It prevents the organization from relearning the same lessons repeatedly, and it gives evaluation engineers real-world test cases that synthetic datasets can't generate. Host: So the agent team ships user-facing product fast, with evaluation engineers and specialized review gates keeping quality in check. Now, neither the platform team nor the agent team decides which underlying models to use or how to fine-tune them. That's a separate discipline. Let's talk about the ML and model team — what's its job, and how does it differ from the other two? Expert: The ML team sits at the intersection of research and production engineering. Its primary responsibility is making sure the organization uses the right models, configured correctly, evaluated rigorously, and served reliably. Note what it does *not* do — it doesn't build end-user products directly. Instead, it provides model artifacts, evaluation frameworks, and fine-tuning capabilities that downstream teams consume. When the ML team works well, application teams can treat models like reliable building blocks. When it doesn't, every team makes its own model selection decisions with incomplete information, and you end up with duplicated effort, inconsistent quality, and runaway cost. A quick definition — fine-tuning is the process of taking a pre-trained model and further training it on your own data so it performs better on your specific tasks. It's one of the core techniques the ML team owns. The key tension inside this team is between exploration and productionization. Research-oriented members want to experiment with new architectures and adaptation techniques. Production-oriented members want stable model artifacts with predictable latency and cost. Both are necessary. A healthy ML team typically has something like a one-to-one or two-to-three research-to-production ratio, skewing toward production as the organization matures. To manage this tension, the ML team structures its work around a clear model lifecycle with four stages: experimental, staging, production, and deprecated. And critically, they define explicit handoff protocols between each stage. A handoff protocol is a set of gate criteria — objective quality checks — that a model must pass before it can advance. So to move from experimental to staging, the model has to beat baseline evaluation scores on core benchmarks, have a reproducible training pipeline documented, and have a cost estimate for serving at target scale. To move from staging to production, it has to pass a load test at twice expected traffic, meet latency targets at the ninety-ninth percentile, pass safety evaluation, and have a documented and tested rollback procedure. Without these protocols, models sneak into production through informal agreement — "it seems good enough" — rather than verified quality gates. The ML team also owns three shared services for the whole organization. First, the evaluation framework. When every application team builds its own evaluation methodology, you lose the ability to compare model quality across use cases. Centralizing evaluation expertise in the ML team creates a consistent quality bar while still allowing application teams to define their own domain-specific test cases within that shared framework. Second, model cards. A model card is a standardized document for every model in the registry, covering its architecture, training data, known limitations, evaluation scores, recommended use cases, and explicitly documented failure modes. Model cards prevent the painful situation where an application team adopts a model without understanding its limitations, then discovers in production that it performs poorly on their specific use case. Third, the experiment tracking infrastructure. When experiments are tracked centrally rather than in individual engineers' notebooks, the team can answer questions like "which embedding models have we already evaluated for retrieval tasks?" without interrupting researchers to go digging. Experiment tracking as a shared service transforms individual knowledge into organizational knowledge — and that's a huge multiplier as the organization scales. Host: Good. So we have three archetypes — platform, agent, and ML — each with its own mission, staffing model, and review processes. But pure archetypes live in textbooks, not in the real world. Most organizations end up blending these. How do you design a hybrid structure intentionally, instead of letting one emerge by accident? And once you have multiple teams, how do they actually talk to each other without drowning in meetings? Expert: Exactly the right question, because those two problems — designing the hybrid and defining the interaction protocols — are really one problem. Let me take them together. The hybrid design process starts with identifying your organization's primary constraint. A constraint is whatever bottleneck is currently limiting your team's ability to deliver. If your constraint is infrastructure complexity, weight the org toward a strong platform team. If your constraint is product iteration speed, weight toward autonomous agent teams. If your constraint is model quality and cost, weight toward a dedicated ML team. Most organizations have all three constraints at once, which is exactly why hybrid structures are the norm. Then map the constraint against organizational maturity, which is essentially a function of headcount. Under twelve engineers, the answer is always a single full-stack AI team that does everything. Don't split prematurely. Between twelve and thirty engineers, split into two teams organized around your primary constraint — agent-first with an embedded platform function, or ML-first with thin application pods, or platform-first with product pods. Above thirty engineers, you need the full three-team topology with platform, agent, and ML as distinct units with defined interaction modes. Now, interaction modes. This is the second core concept. When two teams work together, they do it in exactly one of three modes. Collaboration means two teams working closely side by side, which is high-bandwidth but high-cost — reserved for genuinely novel problems. X-as-a-service means one team consumes the other's output through APIs, contracts, or self-service tools — zero coordination cost, which is what you want by default. And facilitating means one team temporarily coaches another to adopt a new capability, then steps back. Each pair of teams should have exactly one documented interaction mode, so nobody is confused about how to engage. The overarching principle for cross-team work is: asynchronous by default, synchronous by exception. Most inter-team requests should flow through well-defined channels — tickets, API calls, self-service provisioning — that don't require both teams to be available at the same time. Synchronous meetings should be reserved for the rare cases where the teams genuinely need to reason through a novel problem together. A team that spends thirty percent of its time in cross-team meetings is structurally broken. Every pair of teams that needs to collaborate should have an explicit interaction protocol. The protocol specifies the communication channel, the response time SLA, the escalation path, and whether it's synchronous or asynchronous. For example: the agent team requests new infrastructure from the platform team via a ticket workflow with a two-day SLA. The agent team consumes model serving from the ML team via an API contract with zero latency. Incident escalation between them happens through on-call paging with a one-hour response. These protocols should be documented and reviewed quarterly. Here's a concrete metric to track: the coordination tax. That's the percentage of total engineering hours spent on cross-team coordination — meetings, ticket management, waiting for responses — versus building. A healthy GenAI organization keeps it below twenty percent. If it exceeds thirty percent, your protocols need redesign, typically by converting ticket workflows into self-service capabilities, or replacing sync meetings with API contracts. And one more concept worth naming: Conway's Law. This is the observation that organizations end up designing systems that mirror their communication structures. If your teams communicate in silos, your architecture will have silos. The Inverse Conway Maneuver is the practice of using this deliberately — you design your team structure first, knowing that the system architecture it produces will match. It's a strategic lever, not an accident. Host: That's a lot of moving pieces. So team structure is never set in stone — it's a hypothesis about the best way to organize given current conditions. How do engineering leaders know when to evolve the topology? And what's the production wisdom — the stuff that separates organizations that scale gracefully from ones that implode every six months? Expert: The key shift is treating organizational design as an engineering process driven by data, rather than a political process driven by opinions. You do this with a team health assessment framework that you run quarterly. The idea is simple. For each team, you measure a handful of quantitative metrics — deployment frequency, cognitive load score, consumer satisfaction, on-call burden, cross-team wait time. Each metric has a healthy threshold and a critical threshold, so every reading lights up green, yellow, or red. A team with any red metric needs intervention. A team with two or more yellow metrics is at risk. The rule for when to reorganize is conservative: if half or more of your teams are unhealthy at the same time, that usually signals a structural problem rather than a team-specific one, and it's time to redesign the topology. Otherwise, optimize within the existing structure. Always pair this with a qualitative component. Numbers tell you what's happening, not why. After collecting metrics, run a brief anonymous survey asking each team member one thing their team does well and one thing that needs improvement. A declining deployment frequency could be an infrastructure bottleneck, a staffing gap, or a morale problem — only the team members know which. Now, the production wisdom. If you remember nothing else from this chapter, remember these lessons. First, start with a single team and split only when cognitive load signals demand it. Premature splitting creates coordination overhead without any benefit. Wait until the team consistently reports they cannot hold the full system in their heads — which typically happens between eight and twelve engineers. Second, define explicit interaction protocols between every pair of teams that need to collaborate. Specify the channel, the SLA, the escalation path, synchronous or asynchronous. Document them. Review them quarterly. Third, invest in a developer experience role on the platform team from day one, even if you can only hire one platform engineer total. This is the single highest-leverage hire for platform success. Fourth, measure team health quantitatively and share the raw data transparently with the teams themselves, so they can self-correct before top-down intervention is needed. And now the "never do this" list. Never let team boundaries emerge accidentally from project assignments — that creates teams aligned to historical projects that quickly diverge from current strategy. Never skip the transition plan when reorganizing. Run a two-week overlap where both the old and new structures operate at the same time, so knowledge transfers and in-flight work doesn't drop. Never default to synchronous meetings as the primary interaction mode between teams. And never reorganize more frequently than every six months unless a team is in critical health — organizational changes need three months just to stabilize before you can meaningfully measure them. Frequent reorganizations prevent any topology from proving its value. Two more production considerations. Keep every team between five and nine engineers — below five lacks resilience, above nine drowns in internal communication overhead. And avoid matrix reporting. Every engineer should have exactly one team and one manager. Matrix structures with dual reporting create competing priorities and diffuse accountability. Host: Excellent. Now that the concepts are in place, let's preview what you'll build in the labs. You'll work through six hands-on exercises that map directly to what we've covered. In the first lab, you'll design the data model and schema for capturing team structure. In the second, you'll implement the core service logic — the staffing rules, the cognitive load calculations, the interaction modes. The third lab builds the API and interface layer. The fourth integrates with external systems so your topology data can flow into other tools. The fifth covers testing and validation, making sure your team health metrics behave correctly. And the final lab walks you through deploying and operating the whole system in production. Each lab has its own audio overview that goes deeper on the specifics. Host: Let's close by anchoring what you now understand. First, you understand the three core team archetypes in a GenAI organization — the platform team that reduces cognitive load through self-service infrastructure, the agent team that iterates rapidly on user-facing AI applications with dedicated evaluation engineers, and the ML team that manages model selection, fine-tuning, and the full model lifecycle with explicit handoff protocols. Second, you understand how to combine these into hybrid structures based on organizational maturity and primary constraint, and how to design interaction protocols that default to asynchronous communication. Third, you understand how to evaluate team health with quantitative metrics and evolve the topology using data rather than opinion. You now have the depth to contribute to architecture and organizational design conversations in your team — to evaluate whether your current structure fits your constraints and to explain the trade-offs of different topologies to your leadership. The chapter quiz will focus on five concepts: which topology fits which maturity stage, the Inverse Conway Maneuver, Conway's Law itself, the three interaction modes, and agent team staffing. Pay special attention to the decision framework that recommends topologies based on size and primary constraint — that's where most of the application questions live. In Chapter 3, we'll build on this by looking at career ladders for AI engineers — how to define levels, progression criteria, and specialization tracks inside the archetypes we covered today. Team structure defines the boxes; career ladders define the paths people take through them. Until next time.