iOS Build Notifications

Real-time Slack notifications for iOS build status - from upload to TestFlight ready.

Overview

When an iOS build is uploaded to App Store Connect, Apple processes it before making it available on TestFlight. This processing can take anywhere from a few minutes to over an hour, with no predictable timeline.

This system provides real-time notifications to #alerts-ios-builds when:

✅ A build finishes processing and is ready for testing
❌ A build fails during the GitHub Actions CI/CD pipeline

Architecture

Components

1. App Store Connect Webhooks

Apple introduced native webhook support in App Store Connect (WWDC 2025). When build processing state changes, Apple sends a signed HTTP POST to our endpoint.

Registered Webhooks:

App	Bundle ID	Webhook ID
Eli Health (Production)	`health.eli.Eli`	`96ca30ed-75f1-482d-9b0d-9333e834bda7`
Eli Stage (Staging)	`health.eli.Eli.staging`	`9e733b2f-7f4a-48da-b25b-5f9d3ad2a883`
Eli Dev (Development)	`health.eli.Eli.dev`	`61f62ac6-79d6-4cb9-9dd3-3e48fa0854f8`
Eli Dev 2 (Development2)	`health.eli.Eli.dev2`	`d8b86d07-8be3-441b-a984-c7e1f1d1e40f`

Event Types Monitored (all apps):

Event Type	Description
`BUILD_UPLOAD_STATE_UPDATED`	Build processing state changes (uploading → processing → complete)
`BUILD_BETA_DETAIL_EXTERNAL_BUILD_STATE_UPDATED`	TestFlight external testing status changes
`APP_STORE_VERSION_APP_VERSION_STATE_UPDATED`	App Store version state changes (submitted, in review, approved, released, rejected)
`BETA_FEEDBACK_CRASH_SUBMISSION_CREATED`	TestFlight user submitted a crash report
`BETA_FEEDBACK_SCREENSHOT_SUBMISSION_CREATED`	TestFlight user submitted screenshot feedback

All webhooks are configured with the same event types for consistency.

2. Cloud Function (Webhook Handler)

A Google Cloud Function receives webhooks from Apple and posts notifications to Slack.

Location: eli-devops/tf/modules/global/appstore-webhook/

Endpoint: https://appstore-webhook-production-pqpontvmva-ue.a.run.app

How It Works:

Receive Webhook - Apple sends POST request with JSON payload
Verify Signature - HMAC-SHA256 signature validation using shared secret
Parse Event - Extract event type and build upload ID from payload
Fetch Build Details - Call App Store Connect API to get version, build number, app name
Post to Slack - Send formatted notification with TestFlight link

Apple's Webhook Payload Format:

{
  "data": {
    "type": "buildUploadStateUpdated",
    "attributes": {
      "oldState": "PROCESSING",
      "newState": "COMPLETE"
    },
    "relationships": {
      "instance": {
        "data": {
          "type": "buildUploads",
          "id": "uuid-here"
        }
      }
    }
  }
}

App Store Connect API Response Structure:

The function fetches build details from ASC API to enrich notifications:

// GET /v1/builds/{id}?include=app,preReleaseVersion
{
  "data": {
    "type": "builds",
    "attributes": {
      "version": "259"  // CFBundleVersion (build number)
    }
  },
  "included": [
    {
      "type": "preReleaseVersions",
      "attributes": {
        "version": "1.4.16"  // CFBundleShortVersionString (marketing version)
      }
    },
    {
      "type": "apps",
      "attributes": {
        "name": "Eli Dev"  // App name from App Store Connect
      }
    }
  ]
}

Signature Verification:

# Apple sends signature in X-Apple-Signature header with format: hmacsha256=<hex>
signature_header = request.headers.get("X-Apple-Signature")
signature = signature_header.replace("hmacsha256=", "")

expected = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
return hmac.compare_digest(signature.lower(), expected.lower())

3. GitHub Actions Failure Notifications

If a build fails before reaching Apple (compile error, code signing issue, upload failure), the webhook won't fire because nothing was uploaded.

The iOS build workflows include a failure notification step:

- name: Notify Slack on failure
  if: failure()
  env:
    SLACK_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
  run: |
    curl -X POST https://slack.com/api/chat.postMessage \
      -H "Authorization: Bearer $SLACK_TOKEN" \
      -d '{
        "channel": "alerts-ios-builds",
        "text": "iOS build failed",
        "blocks": [...]
      }'

Files:

eli-app/.github/workflows/ios-build.yml - Reusable workflow with Slack notification
eli-app/.github/workflows/ios-dev2-build.yml - Development2 build trigger
eli-app/.github/workflows/ios-dev-build.yml - Development build trigger
eli-app/.github/workflows/ios-staging-build.yml - Staging build trigger
eli-app/.github/workflows/ios-prod-build.yml - Production build trigger

4. Build Tagging and Commit Tracking

Each successful build is tagged in Git and commit information is stored in GCS, enabling Slack notifications to show which commits are included in each build.

Git Tagging Strategy

After a successful build upload, GitHub Actions creates a Git tag with the pattern:

build/{environment}/{build_number}

Examples:

build/dev/265 - Development build 265
build/staging/265 - Staging build 265
build/prod/265 - Production build 265

This allows tracking commits per environment independently. A dev build and staging build can have the same build number but different commit ranges.

How Commits Are Tracked

Environment Isolation

Tags are filtered by environment prefix to ensure each environment tracks its own commit history:

# Find previous dev build tag (ignores staging/prod tags)
git tag -l "build/dev/*" --sort=-v:refname | head -1

# Find previous staging build tag
git tag -l "build/staging/*" --sort=-v:refname | head -1

# Find previous prod build tag
git tag -l "build/prod/*" --sort=-v:refname | head -1

This means:

Dev builds show commits since the last dev build
Staging builds show commits since the last staging build
Production builds show commits since the last production build

GCS Storage Structure

Commits are stored in GCS for retrieval by the webhook handler:

gs://eli-health-prod-appstore-webhook-source/
└── commits/
    ├── 6747853441/          # Eli Dev app ID
    │   ├── 264.json
    │   └── 265.json
    ├── 6747853426/          # Eli Stage app ID
    │   ├── 264.json
    │   └── 265.json
    └── 6471992170/          # Eli Health (Prod) app ID
        ├── 264.json
        └── 265.json

JSON Format:

{
  "build_number": "265",
  "environment": "dev",
  "app_id": "6747853441",
  "commit_sha": "abc123def456...",
  "branch": "main",
  "commits": [
    "abc123 Add new feature",
    "def456 Fix bug in login",
    "ghi789 Update dependencies"
  ],
  "timestamp": "2026-01-15T10:30:00Z"
}

App ID Mapping

Environment	App ID	App Name
dev2	`6758223635`	Eli Dev 2
dev	`6747853441`	Eli Dev
staging	`6747853426`	Eli Stage
prod	`6471992170`	Eli Health

Notification Examples

Build Processing Complete (from Apple Webhook)

✅ Eli Dev Build Ready for Testing
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Version: 1.4.16          Build: 265
Status: Ready for Testing
Previous: PROCESSING

Commits in this build:
• abc123 Add new feature
• def456 Fix bug in login
• ghi789 Update dependencies

2026-01-15 01:28:55 UTC

[Open in TestFlight]

Build Failed (from GitHub Actions)

❌ iOS Build Failed
━━━━━━━━━━━━━━━━━━━━━
Environment: staging
Branch: staging

[View Workflow Run]

Build Processing States

State	Emoji	Description
PROCESSING	⏳	Build is being processed by Apple
COMPLETE/VALID	✅	Ready for testing on TestFlight
FAILED	❌	Processing failed
INVALID	⚠️	Build has issues

Infrastructure

Terraform Resources

The webhook infrastructure is managed in Terraform:

Module: eli-devops/tf/modules/global/appstore-webhook/

Resources Created:

google_cloudfunctions2_function - The webhook handler
google_storage_bucket - Source code storage
google_secret_manager_secret - Webhook secret
google_service_account - Function identity

Configuration in production.tfvars:

appstore_webhook_enabled       = true
appstore_webhook_secret        = "<HMAC secret>"
appstore_webhook_slack_channel = "alerts-ios-builds"

Secrets

Secret	Location	Purpose
`appstore-webhook-secret`	GCP Secret Manager	HMAC signature verification
`appstore-connect-api-key`	GCP Secret Manager	ASC API private key for fetching build details
`slack-summarizer-token`	GCP Secret Manager	Slack API access
`SLACK_BOT_TOKEN`	GitHub Secrets (eli-app)	GitHub Actions Slack notifications

App Store Connect API Credentials

The function uses ASC API to fetch build details (version, build number, app name):

Variable	Value	Description
`ASC_API_KEY_ID`	`RRB6366394`	API Key ID
`ASC_ISSUER_ID`	`63d33c15-d2de-41f0-a5be-09411fb12e74`	Issuer ID
Private Key	In Secret Manager	ES256 key for JWT signing

Registering a New Webhook

Important: Webhooks are registered per-app, not globally. Each app in App Store Connect (Eli Health, Eli Stage, Eli Dev, Eli Dev 2) needs its own webhook registration.

Step-by-Step Guide

1. Add the App to the Management Script

Edit eli-devops/tf/modules/global/appstore-webhook/scripts/manage_webhooks.py:

APPS = {
    # ... existing apps ...
    "health.eli.Eli.newapp": {
        "name": "Eli New App",
        "webhook_id": None,  # Will be set after creation
    },
}

2. Register the Webhook

cd eli-devops/tf/modules/global/appstore-webhook/scripts

# Get the webhook secret
SECRET=$(gcloud secrets versions access latest --secret=appstore-webhook-secret --project=eli-health-prod)

# Create the webhook
python manage_webhooks.py --secret "$SECRET"

The script will create the webhook with all standard event types and print the new webhook ID. Update the APPS dict with the new ID.

2. Update the Cloud Function (if new app)

If this is a new app not already in the system, add it to the APP_NAMES mapping:

# In eli-devops/tf/modules/global/appstore-webhook/function/main.py

APP_NAMES = {
    "6471992170": "Eli Health",      # Production
    "6747853426": "Eli Stage",       # Staging
    "6747853441": "Eli Dev",         # Development
    "6758223635": "Eli Dev 2",       # Development2
    # Add new app here: "APP_ID": "App Name",
}

3. Add Unit Test for the New App

# In eli-devops/tf/modules/global/appstore-webhook/function/test_main.py

def test_new_app_mapping(self):
    assert APP_NAMES.get("NEW_APP_ID") == "New App Name"

4. Run Tests and Deploy

# Run unit tests
cd eli-devops/tf/modules/global/appstore-webhook/function
python3 -m pytest test_main.py -v

# Deploy the updated Cloud Function
cd eli-devops/tf
gcloud config set project eli-health-prod
TMPDIR=~/terraform_tmp terraform apply -var-file=production.tfvars -target=module.appstore_webhook -auto-approve

5. Update GitHub Actions Workflow

Add the new app ID to the commit tracking in .github/workflows/ios-build.yml:

# Map lane to app_id
case "${{ inputs.lane }}" in
  dev2) APP_ID="6758223635" ;;
  dev) APP_ID="6747853441" ;;
  staging) APP_ID="6747853426" ;;
  prod) APP_ID="6471992170" ;;
  # Add new lane here
esac

6. Update Documentation

Update this documentation file with the new webhook information in the "Registered Webhooks" table.

Using the Webhook Management Script

All webhooks can be managed programmatically using manage_webhooks.py:

cd eli-devops/tf/modules/global/appstore-webhook/scripts

# Install dependencies
pip install pyjwt cryptography requests

# Sync all webhooks (creates missing, updates existing)
python manage_webhooks.py

# If creating new webhooks, provide the secret:
SECRET=$(gcloud secrets versions access latest --secret=appstore-webhook-secret --project=eli-health-prod)
python manage_webhooks.py --secret "$SECRET"

# List current webhook configurations
python manage_webhooks.py --list

# Delete a webhook
python manage_webhooks.py --delete <WEBHOOK_ID>

The script maintains all apps with the same event types for consistency. When adding a new app:

Add it to the APPS dict in manage_webhooks.py
Run python manage_webhooks.py --secret "$SECRET" to create the webhook
Update the APPS dict with the new webhook ID

Note: The script uses App Store Connect API credentials from eli-devops/github/tfvars/github.tfvars:

API Key ID: RRB6366394
Issuer ID: 63d33c15-d2de-41f0-a5be-09411fb12e74

Testing

Simulating Webhooks

To test all event types without waiting for real events from Apple:

cd eli-devops/tf/modules/global/appstore-webhook/scripts

# Simulate all event types (sends to #alerts-ios-builds)
python simulate_webhooks.py

# Simulate a specific event
python simulate_webhooks.py --event crash
python simulate_webhooks.py --event released

# List available event types
python simulate_webhooks.py --list

Available simulations:

build_complete - Build ready for TestFlight
build_failed - Build processing failed
crash - TestFlight user crash report
screenshot - TestFlight user screenshot feedback
in_review - App entered App Store review
approved - App approved (pending release)
released - App released to App Store
rejected - App rejected by Apple

Integration Tests

Integration tests verify the deployed Cloud Function works correctly:

cd eli-devops/tf/modules/global/appstore-webhook/function

# Run all integration tests (16 tests)
python3 -m pytest test_integration.py -v

# Run specific test class
python3 -m pytest test_integration.py -v -k TestSignatureVerification

The integration tests cover:

Signature Verification - Valid/invalid/missing signatures
Build Events - Complete, failed, TestFlight external
Feedback Events - Crash reports, screenshot feedback
App Store Events - In review, approved, released, rejected
Edge Cases - Unhandled events, malformed JSON, wrong HTTP method

Unit Tests

The webhook handler also has unit tests that serve as documentation for the API structures.

Run Tests:

cd eli-devops/tf/modules/global/appstore-webhook/function
python3 -m pytest test_main.py -v

Test Coverage (25 tests):

Signature Verification - HMAC-SHA256 validation with hmacsha256= prefix
Message Formatting - All build states (PROCESSING, COMPLETE, FAILED, VALID, INVALID)
Webhook Processing - End-to-end with mocked dependencies
ASC API Parsing - Realistic mocks based on actual API responses
App Mappings - Eli Health, Stage, Dev, Dev 2 app IDs
Commit Display - Commits in notifications, truncation for >10 commits

Key Test Files:

main.py - The webhook handler with process_webhook() for testability
test_main.py - Unit tests with realistic mock payloads

The test fixtures document the exact JSON structures from both Apple webhooks and ASC API responses.

Troubleshooting

Webhook Not Firing

Check Cloud Function logs:

gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="appstore-webhook-production"' \
  --limit=20 --project=eli-health-prod

Verify webhook is registered for the correct app:
- Production, Staging, and Dev are separate apps in App Store Connect
- Each needs its own webhook registration
Check signature verification:
- Apple prefixes signatures with hmacsha256=
- Ensure the secret matches what's in Secret Manager

Build Number Conflicts

If you see "The bundle version must be higher than the previously uploaded version":

Cause: Two builds ran simultaneously and calculated the same build number
Prevention: The concurrency setting in workflows cancels in-progress builds
Fix: Wait for any in-progress builds to complete, then re-trigger

GitHub Actions Notification Not Sending

Check the SLACK_BOT_TOKEN secret exists in eli-app repository
Verify the Slack bot has access to #alerts-ios-builds channel
Check workflow logs for curl errors

Alert Summarizer - Smart alerting system
Terraform - Infrastructure as code

Changelog

Date	Change
2026-01-24	Added crash feedback, screenshot feedback, and App Store version state notifications
2026-01-24	All webhooks now have consistent event types across all environments
2026-01-24	Unified webhook management into single `manage_webhooks.py` script
2026-01-24	Added Eli Dev 2 webhook support
2026-01-15	Initial implementation - webhooks for all three apps (prod, staging, dev)
2026-01-15	Added GitHub Actions failure notifications
2026-01-15	Fixed Apple webhook payload parsing (event type in `data.type`)
2026-01-15	Added ASC API integration to fetch build details (version, build number, app name)
2026-01-15	Added unit tests (22 tests) with realistic mock payloads
2026-01-15	Fixed preReleaseVersions parsing (`version` not `versionString`)
2026-01-15	Added TestFlight button link in Slack notifications
2026-01-15	Added build tagging strategy (`build/{env}/{build_number}`) for commit tracking
2026-01-15	Added commit storage in GCS (`commits/{app_id}/{build_number}.json`)
2026-01-15	Added commit messages to Slack notifications (up to 10, with truncation)
2026-01-15	Environment-isolated tag searching ensures correct commit ranges per environment

Overview​

Architecture​

Components​

1. App Store Connect Webhooks​

2. Cloud Function (Webhook Handler)​

3. GitHub Actions Failure Notifications​

4. Build Tagging and Commit Tracking​

Git Tagging Strategy​

How Commits Are Tracked​

Environment Isolation​

GCS Storage Structure​

App ID Mapping​

Notification Examples​

Build Processing Complete (from Apple Webhook)​

Build Failed (from GitHub Actions)​

Build Processing States​

Infrastructure​

Terraform Resources​

Secrets​

App Store Connect API Credentials​

Registering a New Webhook​

Step-by-Step Guide​

1. Add the App to the Management Script​

2. Register the Webhook​

2. Update the Cloud Function (if new app)​

3. Add Unit Test for the New App​

4. Run Tests and Deploy​

5. Update GitHub Actions Workflow​

6. Update Documentation​

Using the Webhook Management Script​

Testing​

Simulating Webhooks​

Integration Tests​

Unit Tests​

Troubleshooting​

Webhook Not Firing​

Build Number Conflicts​

GitHub Actions Notification Not Sending​

Related Documentation​

Changelog​