Git Workflows for AI Teams — chapter audio overview

Podcast Script: Git Workflows for AI Teams Host: Welcome back to the DevOps Foundations for GenAI Engineers course. This is Chapter 1 of 8, and we're opening the course with something that sounds deceptively basic — Git Workflows for AI Teams. Now, if you're listening and thinking, "I already know Git, I commit and push every day," I want you to stick with me, because this chapter is not about Git operations. It's about something your team is almost certainly getting wrong if nobody has explicitly designed for it. Here's the scene. You're on a team shipping a production AI system. Three engineers are working in parallel. One is iterating on a system prompt. Another is appending training examples to a JSONL file — that's a text file where every line is a standalone JSON record, heavily used for model training data. A third is tweaking a model configuration file where one parameter change can shift evaluation metrics by double digits. They all branch off main on Monday. By Thursday, the merge conflicts are a nightmare no three-way diff tool can resolve. Your organization invested in this training because this is a core competency for any team building production AI — the kind of depth that turns an engineer who uses AI tools into one who builds the infrastructure behind them. You'll practice this across six hands-on exercises. But first, let's build the mental model. We'll walk through four pillars: a branching strategy, a protection layer, a review workflow, and a conflict resolution approach. Host: Let's start with the branching strategy. The chapter centers on something called trunk-based development. Before we get into why AI projects need it specifically — can you explain what trunk-based development actually is, in plain language, for someone who has never heard the term? Expert: Absolutely. So in Git, the word "trunk" just means your primary branch — usually called main. Think of it like the trunk of a tree. Every other branch grows out of it and eventually folds back into it. Trunk-based development is a discipline that says: every engineer on the team works on tiny, short-lived side branches — we call these feature branches — and merges them back into main within hours, or at most a day or two. Never weeks. Now contrast that with the older style, sometimes called gitflow, where a feature branch might live for three weeks while somebody polishes a user interface. That works okay for traditional software, where the code is mostly independent functions and classes. But it falls apart for AI projects, and here's why. AI teams ship three kinds of artifacts that don't behave like normal code. The first is prompt templates — the instruction text you send to a model. Two engineers tweaking the same system prompt on different long-lived branches will produce changes that Git's text-based merge cannot reconcile, because prompts are natural language where every word matters. The second is JSONL training data. When two engineers each append new training examples to the same file, Git sees overlapping changes at the end of the file and flags a conflict — even though logically, you just want both sets of examples. The third is model configuration files, where a temperature value or a stop sequence change looks trivial in a diff but can shift model behavior dramatically. So trunk-based development solves this by making the integration window tiny. If your branch only lives for 24 hours, the opportunity for conflicting changes shrinks dramatically. The rule of thumb in the chapter is 48 hours maximum. Beyond that, you're accumulating what the reading calls "merge debt" — drift between your branch and main that becomes nearly impossible to reconcile cleanly. The practical discipline has a few parts. First, every branch gets a structured name that encodes intent. A branch that changes a prompt might be named something like "feat slash prompt dash summarization." A branch that adds training data might be named "data slash training dash batch" and then a date. A branch that fixes a bug is "fix slash" and then the problem. This matters because your continuous integration pipeline — the automated checks that run on every change — can look at the branch name pattern and apply the right validation. Prompt branches trigger the evaluation suite. Data branches trigger schema validation. Fix branches trigger the full regression test suite. Second, for work that isn't finished in two days, you don't extend the branch lifetime — you use what are called feature flags. A feature flag is just a runtime switch that lets half-finished code exist on main but stay turned off. That way the main branch always reflects the latest state, and nobody is stuck rebasing a week-old branch against it. Third, when you do merge, the recommended strategy is called squash merge. A squash merge takes all the little commits on your feature branch — the typo fixes, the work-in-progress saves — and collapses them into one clean commit on main. This matters enormously for AI teams because when model behavior regresses two weeks later, you want to be able to run a tool called git bisect, which does a binary search through your commit history to find the exact change that caused the regression. A clean squash-merged history makes bisect reliable. A messy history of intermediate commits makes it useless. Host: Okay, that's a powerful reframe — the branching strategy is the first line of defense. Short branches, clear names, clean history. So once a developer opens a pull request to merge their branch, something has to enforce all those rules at the repo level. Let's talk about that layer next — branch protection rules. What are they, and why do AI teams need them configured more strictly than typical software teams? Expert: Great question. A branch protection rule is a setting you configure on your repository — whether you're using GitHub, GitLab, or something else — that blocks changes to a protected branch unless specific conditions are met. Think of it like a bouncer at the door of your main branch. No pull request gets through unless it passes the checks. Traditional software teams use branch protection to prevent broken builds. AI teams need it for something more dangerous: silently bad changes. Here's the distinction. A broken import statement fails your test suite immediately, loudly. But a prompt template change that subtly degrades response quality? That passes every syntax check. It looks fine in a diff. It might even pass unit tests. And yet it costs your organization real money through degraded user experience and increased token usage. So branch protection rules for AI projects need two parallel tracks of enforcement. Track one is required status checks. A status check is just a named validation — a CI job that reports pass or fail on your pull request. For AI repos, you want the standard checks — lint and unit tests — plus three AI-specific ones. You want a prompt validation check that makes sure every prompt template has the required placeholders and structure. You want a JSONL schema check that reads every line of every training data file and confirms it parses as valid JSON with consistent keys. And you want a manifest validation check that runs your Kubernetes deployment files through a validator before they can reach the cluster. Track two is required reviews. This is where human judgment enters. You configure a minimum number of approving reviewers — typically two for anything touching prompts or model configs. And you enable something called stale review dismissal. Here's why that matters: imagine a reviewer approves your pull request, and then you push three more commits afterward. Without stale review dismissal, your approval still counts, even though the code has changed. With it enabled, the new commits invalidate the prior approval, forcing a fresh review of the current state. For prompt changes, where a single word swap can change model behavior, this is non-negotiable. Now, one setting in this chapter that deserves special attention is called "strict" mode on status checks. When strict is turned on, your feature branch must be up to date with main before the checks are considered valid. Why does this matter? Picture two engineers with independent pull requests. Both pass all checks in isolation. But when merged sequentially, their combined changes break. Strict mode prevents that by forcing the second pull request to rebase onto the updated main and re-run every check. Slightly slower, dramatically safer. There's one more piece here: a file called CODEOWNERS. That name will come up on the chapter quiz, so listen carefully. CODEOWNERS is a text file you put in your repository that maps file path patterns to specific teams or people. So you might say: any change to the prompts directory requires review from the prompt engineering team. Any change to JSONL files under the data directory requires review from both the data engineering team and the ML engineering team. Any change to Kubernetes manifests requires platform team review. When a pull request touches those paths, GitHub automatically requests review from the listed owners. You're not relying on someone remembering to tag the right reviewer — the repo enforces it. One critical do-not: never disable the "enforce admins" flag on your branch protection rules. The moment administrators can bypass the protections, you've created an escape hatch that will absolutely be used under deadline pressure, and that's exactly when the risk of an unreviewed prompt change is highest. Host: So protection rules are the server-side gate, CODEOWNERS routes reviews to domain experts automatically, and strict mode prevents silent integration conflicts. That sets up the next question beautifully. Once a reviewer is looking at a pull request, how do you give them the right context to actually review an AI change — especially when a one-line prompt tweak can have massive downstream impact? Expert: This is where most teams fall down, and the fix is structural. The problem is that a standard pull request shows you a diff — the lines added and removed. That's fine for reviewing a new function, because a reviewer can read the code and reason about correctness. But AI artifacts don't work that way. A prompt is natural language, and its correctness is measured by downstream evaluation metrics that the reviewer cannot compute by reading the text. A model configuration change swapping temperature from point seven to point two looks like a trivial diff, but it dramatically changes output diversity. The solution is structured pull request templates. A pull request template is a pre-written markdown form that appears in the description box every time an engineer opens a pull request. It shifts the burden of context from the reviewer to the author. Instead of the author pasting a one-line description and calling it done, the template forces them to fill in specific sections before the pull request can pass validation. And here's the key insight — different change types need different templates. A prompt change template should require fields for the previous prompt text, the new prompt text, which evaluation suite the author ran, the baseline metrics before the change, and the new metrics after the change. Without those numbers, a reviewer cannot possibly make an informed approval decision. A model configuration template should require a rollback plan, because config regressions need an immediate reversion path. A training data template should require the data source, any filtering applied, and a sample of records. Now, how do you enforce this at merge time? You build a CI script that parses the pull request body and checks for the required section headers based on which files the pull request modifies. If someone changes a file in the prompts directory and the pull request body is missing the evaluation results section, the status check fails and the merge is blocked. This pairs perfectly with the branch protection rules we just discussed. The second piece is automated reviewer routing. Beyond the CODEOWNERS file doing path-based routing, you can layer on a labeler that reads the file paths in the pull request and applies classification labels. A change touching the prompts directory gets a "prompt change" label and routes to the prompt engineering team. A change to model configs gets a "model config" label and routes to ML platform. A change to Kubernetes manifests routes to the infrastructure team. These labels then power dashboards, review metrics, and filtering. The third piece is local feedback through pre-commit hooks. A Git hook is just a script that runs automatically at a specific moment in the Git workflow. A pre-commit hook runs before a commit is even created. So if your prompt template file is missing a required field like the model identifier or the evaluation dataset reference, the hook blocks the commit right there on the developer's machine — seconds of feedback instead of minutes waiting for CI to fail. Here's the relationship to understand clearly: pre-commit hooks are a convenience layer. They're fast, but engineers can bypass them with a flag called no-verify. Branch protection status checks are the real gate — server-side, mandatory, no bypass. You need both: fast local feedback for normal development, and the unbreakable server-side gate at the merge point. One important don't — never use a single monolithic status check that bundles every validation into one job. If that job fails, you have no idea what broke. Keep the checks separate and named clearly — lint, test, validate prompts, validate JSONL, validate manifests — so that when something fails, the error points to the specific validation that rejected the change. Host: Structured templates for the author, automated routing for reviewers, and layered hooks for fast feedback. That brings us to the final pillar, and this one is genuinely unique to AI work. Merge conflicts in JSONL training data and prompt template files. Why do these break standard Git tools, and what's the strategy? Expert: This is the most AI-specific problem in the chapter, and it exposes an assumption baked into Git that simply doesn't hold for our artifacts. Git's three-way merge algorithm works by comparing text line by line. For a Python file, a line corresponds to a meaningful unit — a statement, a function signature. For JSONL files, every line is a completely independent JSON record. And for prompt templates, a line is just natural language text with no structural meaning at the line level. Consider the most common conflict. Engineer A branches from main and appends 20 new training examples to a JSONL file. Engineer B branches from the same point and appends 15 different training examples to the same file. Both pull requests pass all checks. The first one merges cleanly. The second one hits a conflict at the end of the file — because Git sees both branches modifying overlapping lines near the bottom. But logically, there's no conflict at all. The correct resolution is simply the union of both additions. Git cannot infer that. The fix is a custom merge driver. A merge driver is a script you register for specific file patterns in a file called dot-gitattributes. When Git encounters a merge conflict in a JSONL file, instead of running its default text merge, it hands the three versions — the common ancestor, your version, the other version — to your custom script. The script parses each file as JSONL, treats each line as an independent record, deduplicates using the canonical JSON form so that records with the same data in different key order don't appear twice, and produces the correct union of both branches. Git just writes the result back as if it resolved the merge itself. You still want a safety net, because a custom driver can't handle every case. That's where pre-commit hooks earn their keep again. You build a hook that runs on every commit containing a JSONL file, validates that every line parses as valid JSON, and verifies that all records follow the same schema — the same set of keys. This catches schema drift, like accidentally interleaving chat-format records with completion-format records after a manual conflict resolution. Prompt template conflicts need a different strategy entirely, because a custom merge driver cannot reliably resolve natural language. Two engineers restructuring the same system prompt for different behavioral outcomes have produced genuinely conflicting intents that no algorithm can merge. The recommended pattern is prompt versioning. Instead of modifying a prompt file in place, every change creates a new versioned file. So instead of overwriting "system underscore prompt dot txt," you create "system underscore prompt version three dot txt." A registry file tracks which version is active. This eliminates merge conflicts in the prompt content entirely, because no two branches ever modify the same file. The only possible conflict is on the single-line "active version" pointer, which is trivial to resolve. This versioning approach also gives you a free audit trail and rollback capability. If a new prompt version regresses in production, rolling back is a one-line change to the registry. You don't have to hunt through commit history and reconstruct what the old prompt looked like. One final consideration — automated dependency update tools. Tools like Renovate and Dependabot open pull requests automatically when a library like PyTorch or Transformers releases a new version. These bots can trigger the same conflicts as human engineers, so you configure them to batch updates on a schedule rather than firing PRs all day. And critically — never enable auto-merge for ML framework dependencies. A passing test suite does not guarantee identical model outputs across library versions. Every ML dependency update needs manual approval after the full evaluation suite runs. Host: Four pillars covered. If the listener takes nothing else away from this chapter, what are the two or three production lessons you want burned into their mental model? Expert: Three lessons. First — branch lifetime is the single most predictive indicator of merge pain. If your team's average feature branch lives more than 48 hours, you will accumulate conflicts in JSONL files and prompt templates that no tool can resolve cleanly. Set up automated alerts when branches age past that threshold. Require daily rebase against main. Treat long-lived branches as a process failure, not an inconvenience. Second — never allow direct pushes to main, not even for trivial prompt typo fixes. What looks like a harmless wording correction can fundamentally alter model behavior. Every change, regardless of size, passes through a pull request with at least one reviewer who has access to the evaluation dashboard. The five minutes saved by pushing directly is never worth the hours debugging a production regression. Third — layer your enforcement. Pre-commit hooks catch format errors in seconds on the developer's machine. Status checks catch schema and evaluation failures in CI. Branch protection rules and CODEOWNERS enforce human review by the right domain expert. Each layer addresses a different failure mode, and skipping any one of them leaves a gap that will eventually cause a production incident. And the top do-not from the reading: never disable branch protection temporarily for an urgent fix, even with the best intentions. The standard failure pattern is that protections get turned off during an incident and never turned back on. Use an audited bypass mechanism with automatic reinstatement after a timeout instead — and log every override. Host: Now that the concepts are in place, here's what you'll practice hands-on. The chapter has six exercises, each with its own audio overview that goes deeper. You'll implement trunk-based development for an AI project. You'll configure branch protection rules and required status checks on a real repository. You'll build pull request templates specifically for prompt changes and model configuration updates. You'll manage merge conflicts in JSONL and prompt files using the custom drivers we described. You'll implement pre-commit hooks and automated dependency updates. And finally, you'll version AI artifacts using Git tags and releases. Host: Let's close it out. After this chapter, you now understand three things at a deeper level. First, you understand why trunk-based development with short-lived feature branches is the only practical strategy for AI codebases where JSONL files, prompt templates, and model configurations are shared resources. Second, you understand how branch protection rules, required status checks, and the CODEOWNERS file combine into a layered enforcement system that prevents unreviewed changes from reaching production. Third, you understand why JSONL and prompt template merge conflicts need custom tooling — structural merge drivers for data files, versioning for prompts. You now have the depth to evaluate your own team's Git workflow, identify the gaps, and bring concrete proposals to your architecture discussions. The chapter quiz will test your understanding of which files belong in CODEOWNERS, which merge strategy preserves bisect-ability, and what happens after a new commit is pushed to a pull request that already has an approval. Pay close attention to the stale review dismissal behavior and to the CODEOWNERS routing patterns. In the next chapter — Chapter 2, CI Pipelines with GitHub Actions — we build directly on every protection rule we configured here. Those required status checks we talked about? Chapter 2 is where you actually write them, wire them to your pull request events, and run them on every commit. The branch protection layer we designed today becomes the foundation for everything that follows. See you there.