← Back to Projects

Case Study

Delta Order Orchestration Backend

Delta is a multi-tenant backend for customers and operators managing standard and priority order workflows. It combines authenticated APIs, PostgreSQL state machines, transactional outbox events, and asynchronous validation, fulfillment grouping, asset-generation, webhook, and notification workers.

Outcome

Coordinates validation, fulfillment grouping, asset generation, storage, and notifications while keeping order submission decoupled from long-running work.

  • Python
  • FastAPI
  • PostgreSQL
  • SQLAlchemy
  • Pydantic
  • Supabase
  • Pillow
  • systemd
Delta Order Orchestration Backend project preview

Problem & Context

What Needed to Change.

The system coordinates orders through capacity checks, optional external eligibility validation, fulfillment grouping, asset generation, controlled lifecycle transitions, and delivery notifications. These stages must remain consistent when workers fail or retry, without processing the same job twice or exposing records across server tenants.

Documented roles include customers, operational runners, and administrators. Customers submit and review their own orders, while administrators manage same-server orders, capacity, renewals, invoices, generated outputs, and configuration.

Constraints & Tradeoffs

The Shape of the Decision Space.

Constraints

  • Every protected resource must be scoped by the server identifier loaded from the authenticated user's database record, with inaccessible cross-server records returned as not found.
  • Order state changes, audit records, and durable outbox events must be committed together so workers never observe an event without its corresponding state.
  • Capacity consumption and worker claims must remain safe under concurrency through row-level locking and SKIP LOCKED queries.
  • External validation, fulfillment grouping, asset generation, uploads, webhooks, and notifications must tolerate retries without creating duplicate work items or state transitions.
  • Headless asset generation must preserve expected dimensions, file contracts, and acceptable visual similarity to committed reference images.

Tradeoffs

  • PostgreSQL outbox and job polling avoids a separate message broker and keeps coordination transactional, at the cost of polling latency and additional database activity.
  • Workers process one item or fulfillment group at a time and use SKIP LOCKED for deterministic claiming, simplifying failure handling while limiting throughput per process.
  • Filesystem inventory and atomic folder renaming preserve the existing standard asset workflow, but couple generation workers to managed disk paths and strict folder contracts.
  • Replacing Photoshop with a Pillow asset generator enables headless Ubuntu deployment while requiring committed masks, transforms, fonts, visual references, and calibration checks.

Architecture

A Deliberately Legible System.

  1. 01

    Authentication and Tenancy

    FastAPI authenticates Supabase JWTs, database-backed API keys, or a legacy master key, then derives the trusted role and server scope from the user record.

  2. 02

    Validation and Capacity

    Pydantic schemas validate requests before writes, while ordering gates, personal limits, daily caps, inventory capacity, and priority-workflow counters use database locks where concurrent consumption is possible.

  3. 03

    Commit and Emit

    The API persists orders, state-change audit rows, and outbox events in the same PostgreSQL transaction before asynchronous processing begins.

  4. 04

    Eligibility Processing

    A worker claims pending checks, calls the external eligibility service with bounded retries, and moves orders to the queue or cancellation based on the result.

  5. 05

    Fulfillment Grouping

    A worker locks compatible queued orders from the same server, prioritizes reserved duration groups, emits grouping events, and records order transitions.

  6. 06

    Asset Generation

    Standard and priority workers allocate resources, create processing records, generate assets with Pillow, upload previews to storage, persist generated data, and transition work items to ready.

  7. 07

    Controlled Lifecycle

    Administrative APIs move work items through ready, processing, completed, failed, and rejected states using centralized transition rules.

  8. 08

    Delivery and Integrations

    Outbox-driven workers deliver notifications and signed webhooks with retry scheduling, audit history, dead-letter metadata, and operator replay support.

Screens & States

The Interface in Context.

Temporary placeholder artwork for Delta generated outputs
Temporary placeholder — replace with anonymized standard and priority workflow outputs using synthetic data.
Temporary placeholder artwork for the Delta order orchestration workflow
Temporary placeholder — replace with an anonymized API, generated-output review, or event-processing capture.

Results

What the System Delivers.

  • Separate inventory-backed and priority workflows from authenticated order creation through fulfillment grouping, generated outputs, and controlled lifecycle management.
  • A shared headless Pillow asset generator suitable for Ubuntu deployment while preserving established file contracts, storage paths, state transitions, and retry behavior.
  • Independent notification events with capped retries, dead-letter metadata, and operator replay so one delivery failure does not block unrelated recipients.
  • Framework-free contract checks covering asset generation, notification delivery, session recovery, and inventory behavior.

Lessons

What I Would Carry Forward.

  • A transactional outbox is most useful when state transitions, audit records, and worker payloads share one consistency boundary.
  • Separating retryable technical failures from terminal business rejection prevents unsafe retries and makes operational recovery explicit.
  • Tenant isolation must come from authenticated server-side identity and be repeated in API queries, worker claims, resource allocation, and event payloads.
  • Migrating a visual pipeline away from desktop software requires committed assets and measurable image comparisons, not only functionally equivalent rendering code.