CASE_STUDY.mdView source

Case Study: OpenResponses iOS AI Playground

Last updated: 2026-06-27

OpenResponses is a native iOS and macOS (Catalyst) developer playground for the OpenAI Responses API. This case study details the core engineering decisions, architecture patterns, and technical challenges solved during its implementation.

It is also the active successor to Gunnar Hostetler’s older OpenAssistant Assistants API client, shifting the product line away from thread-and-run polling and toward direct Responses API execution.

1. The Problem

Developers and prompt engineers working with advanced generative models face a major gap between mobile usability and technical control. Most standard AI chat applications obscure the details of request execution. They hide:

  • Token usage counts and real-time network statuses.
  • Raw response reasoning traces (crucial for optimizing models like o1 and o3-mini).
  • Outbound API request payload details and raw JSON inputs/outputs.
  • The step-by-step lifecycle of background tool calls (such as code interpreters and browser automation).

OpenResponses was built to bridge this gap: providing an observable developer workspace directly on iOS and iPadOS.

2. Technical Constraints

Building a fully featured AI playground within the iOS sandboxed environment introduced several major constraints:

  1. Zero Intermediate Servers: To guarantee privacy and key security, all network transactions must connect directly to destination endpoints (OpenAI, Notion API) from the device. This excludes the use of intermediary backend proxy servers to handle API formatting, token parsing, or automation routing.
  2. High-Velocity Concurrency: OpenAI’s Responses API streams Server-Sent Events (SSE) at rates exceeding 100 completion or reasoning tokens per second. The application must process and render these deltas without freezing the SwiftUI main thread.
  3. Local Automation Boundaries: Enabling “Computer Use” browser automation on iOS requires managing active WKWebView viewports inside a sandboxed environment, preventing layout reload loops while enforcing strict step-by-step user security approvals.
  4. Platform Permission Gating: Accessing calendars, contacts, reminders, and local documents requires compliant handling of security-scoped bookmarks and native iOS permission dialogs without causing crash conditions.

3. Architecture Design: The MVVM-S Pattern

To resolve these constraints, OpenResponses implements the MVVM-S (Model-View-ViewModel-Service) pattern, structured with clear separation of concerns and dependency injection via the AppContainer service locator:

                      ┌─────────────────────────┐
                      │    SwiftUI View Layer   │
                      └────────────┬────────────┘
                                   │ observes State
                                   ▼
                      ┌─────────────────────────┐
                      │   View Model Layer      │
                      └────────────┬────────────┘
                                   │ coordinates actions
                                   ▼
 ┌─────────────────────────────────┴─────────────────────────────────┐
 │                           Service Layer                           │
 ├──────────────────┬──────────────────┬─────────────────────────────┤
 │  OpenAIService   │  ComputerService │  ConversationStorageService │
 └──────────────────┴──────────────────┴─────────────────────────────┘
  • View Layer: Pure, declarative SwiftUI interfaces that observe state published by ViewModels. They forward user gestures or input strings and contain zero database, network, or business logic.
  • ViewModel Layer: The ChatViewModel orchestrates the current chat state, validates prompt configurations, manages safety approval sheets, and controls streaming animations. To avoid monolithic file growth, streaming and tool hooks are isolated into extensions (e.g., ChatViewModel+Streaming.swift). All state updates are scheduled back to the @MainActor.
  • Service Layer: Stateless classes that wrap specific API payloads or system frameworks. This includes OpenAIService (SSE decoder, request payload builder), ComputerService (browser automation state tracker), KeychainService (secure credential wrapper), and FileConverterService (multi-format extraction pipeline).

4. Key Technical Challenges & Solutions

A. WKWebView Reload Loops and UI Freeze (The Scrolling Thrasher)

  • Challenge: Early iterations of the Computer Use feature rendered the active browser viewport inside a SwiftUI UIViewRepresentable wrapping a WKWebView. Whenever the user scrolled the chat timeline or typed a prompt, SwiftUI ran layout sweeps, repeatedly triggering updateUIView(_:context:). Because the target URL was bound dynamically, this caused WKWebView to continuously invoke .load(URLRequest), creating infinite reload loops, freezing the UI, and flooding system logs with NSURLErrorCancelled states.
  • Solution: I resolved this by introducing a state coordinator that tracks the last requested URL. The app checks the target URL against coordinator.lastRequestedURL and only load the request if they differ. 
    func updateUIView(_ webView: WKWebView, context: Context) {
    guard let url = context.environment.targetURL else { return }
    if url != context.coordinator.lastRequestedURL {
        context.coordinator.lastRequestedURL = url
        webView.load(URLRequest(url: url))
    }
}
    This implementation eliminated reload thrashing entirely, reducing main-thread layout blocking to 0% and enabling smooth concurrent scrolling.

B. High-Velocity Concurrency in SSE Streaming

  • Challenge: High-frequency Server-Sent Events (SSE) payloads caused race conditions and data corruption when background thread decoders attempted to update the conversation timeline concurrently with user scroll gesture bindings.
  • Solution: I structured the OpenAIService.streamChatRequest to decode SSE stream tokens line-by-line and yield them in a Swift Concurrency AsyncThrowingStream<StreamingEvent, Error>. The ChatViewModel consumes this stream and schedules mutations of the active message list exclusively back to the Main Actor. 
    for try await event in stream {
    await MainActor.run {
        self.processStreamingEvent(event)
    }
}
    This architecture isolates background parsing from main actor UI updates, eliminating concurrency crashes during high-velocity token delivery.

C. Safe Startup Keychain Migration

  • Challenge: Early prototype builds stored testing API keys as plaintext in standard UserDefaults (openAIAPIKey, pineconeAPIKey), which exposed developer credentials on jailbroken or backed-up devices.
  • Solution: I implemented an initialization migration hook during app startup: 
    KeychainService.shared.migrateApiKeyFromUserDefaults()
    On launch, if a key is detected in UserDefaults, KeychainService writes it securely to the Keychain generic password descriptor and deletes the legacy UserDefaults keys immediately. This migrated existing beta installations to secure Enclave storage without losing active session keys.

D. File Conversion Pipeline for 43 Document & Image Types

  • Challenge: To feed documents and media to the OpenAI API payload, the app must parse diverse file formats (PDFs, plain texts, RTF, Microsoft Office docs, images) directly on-device without using remote parsing APIs.
  • Solution: FileConverterService integrates native iOS framework decoders:
    • PDFKit to extract text layouts from multi-page PDFs.
    • Apple’s Vision framework (OCR text recognition) to extract text content from images.
    • Data mapping to convert 43 specific file extensions into normalized plaintext segments or compressed PNG payloads, packing them into the API request structure.

E. Scalable Settings via ResponseSettingsRegistry

  • Challenge: As OpenAI frequently adds parameters to the Responses API, manually hand-coding new settings rows and validation checks led to massive UI churn, incomplete payload coverage, and configuration drift.
  • Solution: I centralized all parameter definitions into a declarative ResponseSettingsRegistry. A ResponseSettingDescriptor dictates the API key mapping, UI grouping, default values, and valid bounds. The UI dynamically iterates over this registry to build the settings form, ensuring 100% parameter coverage without touching view code when new parameters arrive.

F. Chat-Native Computer Use and Tool Execution Timelines

  • Challenge: Initial Computer Use execution forced users into a heavy, modal-driven UI that broke the conversational flow of a chat app.
  • Solution: I introduced the ToolExecutionTimeline model mapped to each assistant message. As the Responses API streams tool call events, ChatViewModel dynamically upserts granular state (queued, running, awaiting approval) into the timeline. MessageBubbleView then renders expandable inline ToolExecutionCard components directly inside the chat transcript. This embeds browser automation loops natively into the conversation without requiring dedicated modal workflows.

5. Architectural Tradeoffs

  • Direct Connections vs. Server-Side Middleware: Bypassing proxy middleware ensures maximum privacy and absolute credential ownership. However, it means the client must handle all response formatting and tool execution locally, which increases on-device battery consumption and request payload sizes.
  • Keychain Enclave vs. Cloud Synchronization: Storing keys in the Secure Enclave ensures that credentials never leave the device. The tradeoff is that users must enter their API keys manually on every new device they set up, as keys are not synced via standard iCloud key-value stores.
  • Local WKWebView Automation: Running the browser automation loop inside a local WKWebView allows users to see and approve automation actions step-by-step. However, this restricts browser automation to websites that render correctly inside the iOS WebKit container, lacking support for heavy desktop-only plugins.

6. Engineering Metrics

The application incorporates the following integrations and components:

  • APIs Integrated: 5 (OpenAI Responses, OpenAI Embeddings, Notion, Apple Calendar, Apple Contacts/Reminders).
  • Core Architecture Layers: 5 (Views, ViewModels, Services, Keychain Security Enclave, Sandboxed Local Storage).
  • File Modalities Supported: 43 distinct file extensions converted locally.
  • Preflight & QA Scripts: 2 (secret_scan.py and preflight_check.sh verifying credential safety and Info.plist compliance).
  • Zero-Data Leak Guard: 100% of credentials stored in Secure Keychain; zero external analytics tracking libraries.

7. What I Would Improve Next

  1. Local Pyodide Sandboxing: Transition the Code Interpreter from OpenAI’s remote container to a local WebAssembly-based Pyodide workspace, executing calculations entirely offline.
  2. Dynamic MCP Autodiscovery: Allow local network multicast DNS (mDNS) scanning to automatically detect and pair with running Model Context Protocol (MCP) servers on the local network.