How We Built It, Part 9: Maintain It

In Part 8 we pushed the site to production on a Docker Swarm behind Traefik. Now comes the part nobody puts in the brochure: the next ten years. A site is not "done" when it launches — it is launched. The slow, unglamorous work that follows is what separates "we still run our site" from "we abandoned it three months in." This final post is the maintenance plan: updates, backups, security, and the one mental model that ties it all together.

Applying security updates

Drupal and its modules ship security releases. You want them, and you want them without surprises. On a containerized, recipe-driven site there are two valid ways to apply an update, and they serve different jobs.

• The admin UI path — Drupal's built-in updater (/admin/reports/updates) can stage and swap module versions from the browser. It is fast for a one-off patch, but it needs the codebase to be writeable by PHP-FPM and leaves no diff to review. We don't lean on it.
• The git + composer path — edit dependencies locally, run composer update, commit composer.lock, push, redeploy. Every change is reviewable in git, every change is reversible with git revert, and it works the same on every environment. This is the path we use.

The update procedure is deliberate. Snapshot first, then update the packages explicitly (listing them by name so composer only touches what you intend), then run the database updates:

composer update --no-interaction --with-all-dependencies drupal/core-recommended drupal/canvas drupal/smart_date drupal/fullcalendar_view drupal/feeds

Then apply any pending schema changes and clear caches:

docker compose exec php drush updatedb -y
docker compose exec php drush cr

Finally, prove it: composer audit should report no vulnerabilities, and a quick visual check of the homepage, navbar, and a blog post confirms nothing broke. One honest gotcha worth naming up front: right after a fresh install or a composer update, Drupal's update report can show modules as "out of date" for up to 24 hours — its cached release feed from drupal.org is simply stale. Don't panic. If you need a fresh answer immediately, force a re-check:

docker compose exec php drush state:set update.last_check 0 && docker compose exec php drush cron

How do you find out an update exists in the first place? Drupal cron emails you. The deploy pipeline sets the notification recipients (and throttles the check to weekly so you get a digest, not a daily nag) — which only works because the email pipeline itself is configured correctly, the subject of an earlier post.

When a minor upgrade bites: a real one

Everything above makes updates sound tidy. Here is one that wasn't, because the honest version is more useful than the tidy one.

We bumped the page-builder module from 1.7 to 1.8 — a minor release, the kind you apply without ceremony. Every page rendered perfectly afterward. The homepage, the navbar, the blog: all fine. The visual check we just recommended passed completely.

Then the site owner tried to edit the header and got this:

2 Errors — Header region:
  'view_mode' is a required key.
  'view_mode' is a required key.

The site was readable but not editable. That distinction is the whole lesson.

What happened: 1.8 tightened a validation contract. Our announcements banner is a content block placed as a page-builder component, and the new version started validating those components against Drupal core's schema for content blocks — a schema that marks view_mode as a required key. Our banner was created before that rule existed, so it didn't have one. Nothing about the banner changed; the rules changed underneath it. Content that was valid on Monday was invalid on Tuesday.

Three things had to be true to fix it, and missing any one of them left it broken:

• The view mode had to actually exist. Adding view_mode: default wasn't enough on its own — the schema also requires that the named view mode be a real config entity, and a fresh build of this site ships none for content blocks. We had to create it.
• The obvious value was the wrong one. The block plugin's own default is full, so that looks like the safe answer. But no full view mode exists here either, so it fails the same check. default was correct because it matches the display our recipe already provisions.
• Repairing existing content is not the same as creating it. Our recipe creates the banner only when it's absent — which meant a site that already had a broken banner would skip the fix forever. The repair has to detect the broken state, not just check for existence. And it had to delete and recreate the component rather than edit it in place, because an in-place edit quietly reverted to that wrong full default.

All three now live in the recipe's post-install script, so the fix is not a thing someone did once on a server — it re-applies on every deploy, and it heals a broken site as readily as it configures a fresh one. That is the same "content versus structure" discipline the next section is about, applied to a bug.

The takeaway that generalizes past this one module: after a minor upgrade, test the things you do, not just the things you see. Publish a change to each region. Save a node. Submit the form. A page can render beautifully while the editor behind it is quietly unusable — and the person who finds that out shouldn't be your client.

Backups that restore themselves

A backup you have never restored is not a backup; it is hope. We run three layers so that hope never has to do any work.

1. Committed snapshots live in git as the seed for a fresh build. They carry site structure, not editorial content — more on that split below.
2. Weekly off-site backups run every Sunday at 03:00 (the cron container's local time). The production crond service runs scripts/backup-offsite.sh, which dumps the live database, tars the uploaded files, and uploads both as dated assets to a GitHub Release tagged backups-chattanooga-prod. That copy lives entirely outside the Swarm cluster — it survives losing the whole server.
3. Auto-reseed on redeploy is the layer that makes the other two effortless. When the production stack is rebuilt from scratch, the init container checks that GitHub Release first and restores the newest db-chattanooga-prod-*.sql.gz and files-chattanooga-prod-*.tar.gz it finds, falling back to the committed snapshot only if the release is missing or unreachable.

The practical payoff: in most disaster scenarios you do not run a manual restore at all. You delete the stack and its volumes, redeploy, and the init container pulls the most recent weekly backup on its own. The decision it makes is logged, so you can see in Portainer whether it used a release asset or the committed snapshot. If you ever do need a hand-restore, the assets are plain files you can pull with the GitHub CLI:

gh release download backups-chattanooga-prod -R TortoiseWolfe/Chattanooga-Digital -p 'db-chattanooga-prod-*.sql.gz'

Security hardening

A public site with a login form is a target. We ported a small, boring, effective account-security suite — boring being the highest compliment you can pay a security control. Each piece is a Drupal module wired in through the recipe and the deploy pipeline:

• login_security — tracks failed login attempts per user and per IP, and emails a configured security address when something looks like a brute-force attempt. (The whole suite was added after a real suspicious-login alert, not as a checkbox exercise.)
• captcha — a self-contained math CAPTCHA on the login form. No third-party service, no tracking, just enough friction to stop a script.
• advban — a maintained IP-ban backend for shutting out an address that has clearly turned hostile.
• Two-factor authentication via tfa and real_aes — time-based one-time codes (the authenticator-app kind), with the secret encrypted at rest.

One design choice here is worth dwelling on, because it is a lesson learned the hard way. We loosened the lockout: 2FA ships as optional, self-enroll rather than forced on every account. A forced-but-misconfigured second factor is a fantastic way to lock the site owner out of his own site permanently, because the encryption key that protects every enrolled secret is created once and must never change — regenerate it and every stored TOTP secret becomes undecryptable. So we make it available, encourage it, and let the owner turn it on for himself at /user/{uid}/security/tfa. Security that locks out the people it is meant to protect is not security; it is an outage with good intentions.

Content versus structure: the discipline that makes it forkable

Every site has two kinds of state, and confusing them is the root of most operational pain. Get this split right and almost everything else follows.

• Structure lives in code, in the recipe YAML, in git. Content types, fields, Views, menus, Canvas page templates, roles and permissions, module and theme versions. This is developer-authored, reviewable, and rebuildable from scratch. A structure change is a recipe edit plus a commit plus a redeploy.
• Content lives in the database. Blog posts, uploaded media, user accounts, form submissions. This is authored through the admin UI by people who should never have to touch git. A content change needs no deploy at all.

This is why the off-site backup and the recipe do different jobs. Recipes recreate the structure; the backup snapshot restores the content the recipes cannot possibly know about (the recipe has no idea what the editor blogged last Tuesday). On a full rebuild you restore the snapshot for content, then apply the recipe for structure — in that order, because the recipe is idempotent and won't stomp the content it finds.

There is one subtlety that bites everyone eventually. On staging and production the database is restored before the recipe runs, so any config the snapshot already carries is "already there." Drupal's recipe installer compares against active config and silently skips installing anything that already exists — which means editing a plain config file for an existing setting does nothing on a deployed site. To change a value that already shipped, you use a recipe config-action (which merges into active config on every apply) or a per-deploy drush cset in the deploy pipeline. New config that the deployed databases have never seen can ship as a plain file; everything already live must go through an action. Internalizing that one rule is the difference between "your change deployed" and "you patched it in the admin UI and it vanished on the next redeploy."

What's next

Nothing — and that is the whole point. Across this series we installed the site in Docker, learned what a Drupal Recipe is, built components, modeled real data, wired up a calendar and forms, automated the boring parts with cron, deployed to a production Swarm, and now we maintain it for the long haul. What you are left with is not a screenshot of a finished website. It is a recipe — a set of version-controlled files that rebuild a real, running, maintainable site from scratch, on your laptop or on a server, identically, every time.

So take it. The whole thing is open: clone the repository, run make up && make install && make build, and make it your own. That was the promise in Part 0 — we are, quite literally, giving away the recipes. Thanks for building it with us.

Beta-test this post with your AI

This series is open-source documentation, and documentation drifts. Help us keep it honest: point your AI coding assistant (Cursor, Claude Code, or similar) at this post and the repo, and if it finds something that no longer matches the code, open a ticket. First time? Read A Token Is an Identity (in this series) so your AI opens issues as you, with credit. Edit the three variables at the top of the block below, then paste it into your assistant:

# ─────── EDIT THESE FIRST ───────
GITHUB_USERNAME = "your-github-username"   # your handle, so the issue is credited to YOU (see the Token Is an Identity post)
DEMO_REPO       = "TortoiseWolfe/gig-city-drupal"   # the repo to file against — or your own fork: GITHUB_USERNAME/gig-city-drupal
ARTICLE_URL     = "https://www.chattanooga.digital/blog/2026-06/how-we-built-it-part-9-maintain-it"
# ────────────────────────────────

Read ARTICLE_URL, then clone DEMO_REPO into my local environment and explore it.

Cross-check Part 9 (Maintain It) against the actual code — the update procedure, the self-restoring backups, the security hardening, and the content-vs-structure rule.

If anything is inaccurate, out of date, or contradicts the code, open an issue on
DEMO_REPO (authenticated as GITHUB_USERNAME) so the maintainers can fix the tutorial.

A good issue:
  - one problem per issue (not a catch-all list)
  - a clear title naming the section, e.g. "Part N:  no longer matches the code"
  - cite the exact spot: the post section + the repo file/path (and line if you can)
  - say what the post claims vs. what the code actually does
  - suggest the fix if it's obvious

← Part 8: Deploy It