LIMITATIONS.mdView source

Limitations

OpenIntelligence ships on the App Store for iPhone, iPad, and Mac, with paid subscription tiers. It is a real product, and the limits below are the honest boundaries of what it does well — not a disclaimer that it is a prototype.

Corrected 2026-08-05. This document previously opened by calling the app “experimental software… a proof-of-work repository and prototype” and listed “It is not production-ready” and “It is not a commercial product offering” as limits. Both were false: the app has been selling pro_monthly, pro_annual, and lifetime_cohort through StoreKit since v4.4, and README.md states it is shipping. A limitations document that understates what the product is undermines the limits that actually matter, which are the safety and technical ones below.

Corrected 2026-10-20 against the working tree (b868fc0 plus the 5.7 work in progress): the Technical Limits bullet on how much of the window Standard’s evidence fills said Deep Think and Maximum use fixed character budgets. From 5.6 Deep Think’s chain sizes each session in tokens from the device’s window, and the characters are its fallback. The same bullet now also says that Standard’s first pass is sized for the Private Cloud Compute window the app assumes when Hybrid can reach it. The answer modes, the agentic loop and its checks are set out in Docs/CANONICAL_OPENINTELLIGENCE_SOURCE_OF_TRUTH.md, section 19.

Product Limits

  • It is not a finished enterprise SDK, and OIEngine still routes through app-owned services rather than standing alone.
  • It is not a sealed binary SDK or a buyer-ready handoff.
  • Building from source requires Xcode and Apple-platform development familiarity. Using the shipped app does not.
  • Quality modes are not equally proven. Standard has a measured accuracy baseline. Deep Think has been benchmarked, but not in a way that separates it from Standard, and Maximum has no valid figure. See the Technical Limits below. Corrected 2026-09-29: from 2026-08-05 this line read “Standard has a measured accuracy baseline; Deep Think and Maximum do not.”

Safety Limits

  • It is not validated for regulated workflows.
  • It is not intended for clinical, legal, financial, or safety-critical decision-making.
  • It is not a clinical decision-support system.
  • It is not a diagnostic tool.
  • It does not guarantee complete, correct, or exhaustive answers.

Technical Limits

  • Retrieval quality depends on document quality, extraction quality, and query scope.
  • OCR, layout parsing, chunking, and entity extraction can fail or lose context.
  • Citations indicate supporting sources, not formal proof.
  • Confidence signals are engineering aids, not certified reliability metrics.
  • Only Standard mode has a measured accuracy baseline: 0.410 exact-match across 83 cases from 40 real research papers (QASPER), local-only. Corrected 2026-08-21: this read “80% across 20 ground-truthed cases with zero hallucinations” until that figure was withdrawn, because the run behind it averaged 7 seconds per case, under the 60-second threshold at which BenchmarkRuns/PROGRESSION.md flags generation as almost certainly not having run, and the corpus was synthetic and authored alongside its own questions. Exact-match is a floor, not a quality score. Deep Think has since been benchmarked after both fixes named in Docs/EVALS.md (665da0a, acfbfbd). BenchmarkRuns/LEDGER.md records its first quality number, 2 of 8 against Standard’s 4 of 7 on 2026-08-19 (paired-retry), and its largest paired run, 9 of 25 against Standard’s 10 of 25, run overnight from 2026-08-20 (overnight-25case-nodeadlock). Each is a single run in the Mac benchmark harness, with the harness forcing a sampling temperature of 0.7, which the ledger found does not reproduce the shipping app, and the ledger does not read either gap as a difference between the modes: at 25 cases it cannot resolve less than about a four-case swing. Maximum still has no valid figure. Corrected 2026-09-29: from 2026-08-21 this sentence read “Deep Think and Maximum still have no valid figure; the one Deep Think run attempted scored 1 of 5 with a timeout at 1800s.” Both runs above were already in the ledger when it was written, and the raw data behind the 1-of-5 run had been deleted on 2026-08-19, so that figure cannot be checked. [evidence_level: artifact_derived, confidence: high, evidence_source: BenchmarkRuns/LEDGER.md rows paired-retry (1b700ed) and overnight-25case-nodeadlock (73fff4f), sections "The noise floor, measured" and "No benchmark has ever measured the shipping configuration", and the overnight-smoke row with its deletion note; git merge-base --is-ancestor places 665da0a and acfbfbd before 1b700ed and 73fff4f; git log dates the two ledger entries 2026-08-19 and 2026-08-21 02:45 and this sentence 2026-08-21 20:00 (e3dcab1)]
  • Corrected 2026-09-29: the Standard-mode figure above can be re-run from committed inputs. Benchmarks/ResearchFixtures/qasper_external_v1/ holds its 83 cases over 40 papers with external ground truth, committed 2026-08-12 (1fa32c2), and python3 scripts/build_external_fixtures.py --check verifies the pack offline against its lock. The 0.410 was measured on that pack: the committed summary Docs/AuditArtifacts/Benchmarks/20260812-215108-matrix.md names its manifest and records the same corpus SHA-256 prefix as the lock, 4a470066470988a0 (34 correct of 83 attempted). A re-run reproduces the inputs, not necessarily the figure: BenchmarkRuns/LEDGER.md measured a one-case swing at n=24 under topk sampling and none under --sampling greedy. From 2026-08-05, before the pack existed, this line read “The repository cannot currently reproduce its own quality numbers” and listed committed fixtures with external ground truth as roadmap work. Still true: Docs/TestDocuments/ holds six small ingestion edge cases that test no answer quality, and Benchmarks/rag_validation_sample.json points at a gitignored PDF. [evidence_level: test_verified+artifact_derived, confidence: high, evidence_source: build_external_fixtures.py --check run 2026-09-29 printed "OK: qasper_external_v1 matches its lock (83 cases, 40 papers, allenai/qasper @ fdc9d8214fba)"; fixtures.lock.json fixtures_sha256 4a470066470988a0...; 20260812-215108-matrix.md; BenchmarkRuns/PROGRESSION.md row qasper-overnight (34 correct, 0.410), the run Docs/EVALS.md cites as summarised in that file; LEDGER.md "The noise floor, measured" and greedy-25-a/greedy-25-b; .gitignore:250 ignores Benchmarks/Fixtures/private_*]
  • Some Apple Intelligence or on-device model paths may depend on device, OS, and regional availability.
  • Native PCC execution is owner-confirmed on a physical iOS 27 device running v4.6. Quota exhaustion, mid-stream network-transition fallback, background/App Intent consent behavior, and Archive/TestFlight distribution signatures were not part of that confirmation and remain unverified; source and simulator evidence cannot prove Apple’s production route for them. [evidence_level: user_confirmed+code_verified, confidence: high_for_execution_path_unverified_for_edge_scenarios, evidence_source: owner device testing 2026-07-28, ModelExecutionPlanner.swift, FoundationModelSessionFactory.swift, MessageBubbleV2.swift]
  • AFM 3 Core Advanced — Apple’s 20B sparse on-device model announced at WWDC26 — is real, but it is OS-managed. An app cannot select it: in the release SDK (Xcode 27.0, 27A266a) SystemLanguageModel construction still accepts only .default, a UseCase (.general/.contentTagging, a task selector, not a size selector), or a LoRA adapter, and the adapter initializer is obsoleted in iOS 27. An app can observe which on-device model it has, in two ways. The same SDK has SystemLanguageModel.variant (iOS 27.0, macOS 27.0), which reads .core3 or .coreAdvanced3; Apple’s reference pages name them “AFM 3 Core” and “AFM 3 Core Advanced”. And every response names its assets: each Transcript.Response in a response’s transcriptEntries carries assetIDs, public since iOS 26.0 and documented as “Version-aware identifiers for all assets used to generate this response.” OpenIntelligence reads neither, so it labels the local route as On-Device rather than asserting a parameter count. On the owner’s Mac (M3 Pro, 18 GB, macOS 27.0 26A428) variant reads .coreAdvanced3, one response’s assetIDs read com.apple.fm.language.instruct_3b.fm_api_generic_3b?variant=generic_sparse_16.1.0.13.103120,0 beside a draft model and a tokenizer, and contextSize reads 8192, where Apple’s written documentation says 4096 tokens: a 7,949-token prompt was answered and a 9,691-token prompt was refused as exceeding “the maximum allowed context size of 8192”. The variant reference pages do not say which devices get which variant. Not measured in this pass: an iPhone or iPad. Measured 2026-10-06: a Private Cloud Compute response’s assetIDs read two com.apple.fm.language.instruct_server_v2 assets in each of five runs of a development-signed probe that carried the PCC entitlement, so on that Mac the field told the two routes apart; Docs/PRIVACY_AND_ROUTING.md (“What Apple reports”) has the identifiers. scripts/probe_afm_advanced_canary.sh is not a working alarm. The Xcode Cloud post-clone script calls it as ../scripts/probe_afm_advanced_canary.sh after its own cd .., which points outside the repository, and || true hides the failure, so it has most likely never run there; no build log was read to confirm that. Run by hand it tests one spelling, SystemLanguageModel.advanced, against the iPhone simulator SDK and exits 0 either way, so it would miss selection shipped under another name and could not have noticed variant. Corrected 2026-10-05: from 2026-07-28 (cfbbd0d) this bullet said the SDK “exposes no way for an app to select it or to observe whether it (rather than the 3B Core model) served a request”, citing the interface of Xcode 27.0 beta (27A5194q), and said the canary “runs in CI and alerts the moment Apple exposes developer selection”. Selection is still absent. Observation is possible through variant in the release SDK and through assetIDs, and the bullet had not been re-checked against either. [evidence_level: sdk_verified+documented+measured+code_verified+externally_corroborated, confidence: exact_for_installed_sdk_and_one_mac, evidence_source: FoundationModels.swiftinterface for iOS and macOS in Xcode 27.0 27A266a read 2026-10-05 (the variant property, the Variant struct with core3 and coreAdvanced3, the SystemLanguageModel initializers, Response, Transcript.Response.assetIDs); developer.apple.com/documentation/foundationmodels/systemlanguagemodel/variant-swift.property, variant-swift.struct with its core3 and coreAdvanced3 pages, transcript/response/assetids, and managing-the-context-window ("4096 tokens per session"), fetched 2026-10-05; command-line probes on macOS 27.0 26A428 on 2026-10-05 printing SystemLanguageModel.default.variant, contextSize and one response's assetIDs and metadata, and sending prompts sized with tokenCount(for:); a development-signed probe carrying the PCC entitlement on the same Mac on 2026-10-06 printing a Private Cloud Compute response's assetIDs (five runs, 51 requests); grep -E for \.variant\b, \.core3\b, coreAdvanced3 and assetIDs across the app, engine and test Swift (no match); LLMModel.swift:245-255; ci_scripts/ci_post_clone.sh:80, 104-106 and 125-130 (build 426); sh scripts/probe_afm_advanced_canary.sh run from the repository root on 2026-10-05 printed "still absent from this SDK (expected)" and exited 0, and sh ../scripts/probe_afm_advanced_canary.sh from the same directory printed "No such file or directory" and exited 127; git log -S on this file for the quoted phrases (cfbbd0d only); Apple ML Research AFM-3 announcement for the 20B figure] On an iPhone (2026-10-19): the app’s own trace on the owner’s iPhone 16 Pro Max (A18 Pro, iOS 27.0.1) shows the Standard path’s base window as 4096 in three answers on 2026-10-16 and 10-17 (Context budget: base=4096, which RAGService takes from SystemLanguageModel.default.contextSize), so the window differs by device: 8,192 on that Mac, 4,096 on that iPhone. [evidence_level: measured (the app's pipeline_trace.log, copied from the phone with devicectl), confidence: high for that iPhone] Changed 2026-11-17: the same iPhone, on iOS 27.2.1 (24B156), reports 8,192: its trace reads base=4096 in nine answers through 2026-11-16 10:10 and base=8192 from 2026-11-17 08:10, with no context-size error since. What changed it overnight is not known. A device’s window can change with the system or the model, so the app reads it every time and the in-app text names no device’s figure. [evidence_level: measured (pipeline_trace.log copied with devicectl on 2026-11-17), confidence: high for that iPhone]
  • In the one corpus measured, Standard’s on-device evidence filled roughly a quarter of the model’s window. Standard reads the window from SystemLanguageModel.default.contextSize rather than assuming 4096, but it sizes its evidence in characters, converting its token budget at 1.4 characters per token. A 10,000-character ceiling sits above that conversion and cannot bind at a window of 8192, where the conversion yields at most 9,756 characters; questions the app classes as trivial are held to 2,500 characters. 1.4 is close to what a dense table of part numbers and measurements costs (1.38 measured), and English prose costs far less per character (4.2 to 5.5 characters per token measured). In the Mac benchmark run sourceonly-after-greedy25 of 2026-09-23 (25 research-paper cases, on-device, window read as 8192 in every case) the evidence sent was 3,177 to 9,494 characters, median 9,324. The assembled prompts were not saved. The same length of each case’s recorded chunk text counts as 723 to 2,158 tokens, median 1,993, by the model’s own tokenizer, so the token figures are an estimate from like text and the character figures are exact. Structured answer generation was skipped in all 25: that path is gated at 3,600 estimated tokens whatever the window is, and every skip logged an estimate of 4,247 or more. Deep Think and Maximum give their reasoning sessions fixed character budgets that are not derived from the window: 3,500 or 3,000 characters in the reasoning chain, cut to 2,200 once a session carries earlier findings, and 2,500 in Maximum’s own loop. Two of their budget helpers do read the window. Not measured: an iPhone or iPad, a corpus of tables, and how much of the window Deep Think and Maximum fill in practice. Corrected 2026-10-20 against the working tree (b868fc0 plus the 5.7 work in progress): the chain’s fixed character budgets are now its fallback, not its rule. From 5.6, wherever the OS can count tokens (iOS and macOS 26.4 and later), Deep Think’s chain, and the chain Maximum runs on a “good” pool, read each session’s passages whole within a token budget planned from the window the device reports: at most 3,200 evidence tokens, the window less 700 for the reply, 400 for the prompt, 256 for safety and the carried text, with 4 passages a window on a 4,096-token window and up to 7 on a window of 8,000 or more, as long as at least 4 passages fit whole. The 3,500 and 3,000 characters of extracted sentences, cut in the prompt to 4,000 for the first session and 2,200 for later ones, still apply where tokens cannot be counted, to a window where fewer than 4 passages fit whole, and to chain sessions bound for Private Cloud Compute when it is picked in the model picker. Maximum’s own loop still reads 2,500 characters of extracted sentences a session. One more limit on the first sentences of this bullet: Standard reads the window from the device only when the cloud route is not eligible. When Hybrid can reach Private Cloud Compute, its first pass is sized for the 32,768-token PCC window the app assumes (a hard-coded value, while the route planner reads the window PCC reports) at 2.5 characters a token, capped at 45,000 characters, and an answer then planned on the device is re-packed to the device’s budget. The line numbers in the evidence below are those of the tree measured on 2026-10-05. In the working tree as read at 22:17 on 2026-10-20 the same code is at RAGService+QueryPipeline.swift:503-527 and RAGService+QueryContext.swift:1142-1205 (window, ratios and ceilings) and RAGService+Generation.swift:1504 (structured gate), and AgenticOrchestrator+ReasoningChain.swift:232-235 (chain budgets), AgenticOrchestrator+MultiChain.swift:248-251 (buildChainPrompt) and AgenticOrchestrator+Maximum.swift:851-855 (Maximum’s 2,500). [evidence_level: code_verified, confidence: high, evidence_source: OpenIntelligence/Services/Agentic/SessionEvidencePlan.swift:30-76, :92-106 (fewer than 4 whole passages fall back) and :134-158, OpenIntelligence/Services/Agentic/AgenticOrchestrator+ReasoningChain.swift:231-287, :303-435 and AgenticOrchestrator+MultiChain.swift:248-253, OpenIntelligence/Services/RAG/Orchestration/RAGService+QueryPipeline.swift:483-495 and :503-527, RAGService+QueryContext.swift:1142-1205 and RAGService+QueryRetrieval.swift:122-231, OpenIntelligence/Services/AIPlatform/AppleFoundationModels/FoundationModelTokenBudget.swift:25-40, the v5.6.0 tag (2bee971) for the 5.6 SessionEvidencePlan] [evidence_level: code_verified+artifact_derived+measured, confidence: high_for_characters_medium_for_tokens_one_mac_one_corpus, evidence_source: RAGService+QueryPipeline.swift:503-527 and RAGService+QueryContext.swift:1142-1205 (window, 1.4 ratio, the 10,000 and 2,500 ceilings; 9,756 is (8192 - 121 - 1 - 150) x 0.88 x 1.4 from those lines) and RAGService+Generation.swift:1497-1505 (structured gate); FoundationModelTokenBudget.swift:28-50; AgenticOrchestrator+ReasoningChain.swift:232-234 (chain budget), AgenticOrchestrator+MultiChain.swift:250-253 (buildChainPrompt), AgenticOrchestrator+Maximum.swift:851 (executeTrueUnlimitedReasoning), AgenticOrchestrator+Retrieval.swift:214 and AgenticOrchestrator+MultiChain.swift:558 (helpers that read the window); the "Context budget: base=8192" and "Structured answer generation skipped" log lines and context_chars in BenchmarkRuns/2026-09-23-sourceonly-after-greedy25/results.jsonl; SystemLanguageModel.default.tokenCount(for:) on macOS 27.0 26A428 on 2026-10-05 over the first context_chars characters of each case's RETRIEVED CHUNK TEXT listing (the harness's chunk dump, not the assembled prompt), and over a 2,783-character synthetic table (2,024 tokens)]
  • From 5.7, some features read a very large library through a sample, with fixed limits. The Knowledge Atlas maps at most 6,000 passages and takes its keyword statistics from at most 2,000,000 characters of text spread over the documents. The vocabulary used to expand a question, and the pool that suggested questions are drawn from, are sampled above 4,000 passages. The scan that looks for a stated value in passages retrieval did not return covers every such passage up to 20,000 and an even sample above that, so in a larger library it can miss a value that sits only in a passage outside the sample. Vector and keyword search still read every passage. [evidence_level: code_verified for the limits; test_verified for the sampling functions (LargeLibraryCostTests, SystemToolsAndScanTests), confidence: high, evidence_source: RAGService.swift visualizationPassageLimit, suggestedQuestionPoolLimit and specScanLimit; QueryEnhancementService.swift vocabularySampleLimit; SQLiteFullTextService.swift statisticsCharacterBudget; Docs/ai/DECISIONS.md 2026-10-10]
  • From 5.7, one document holds at most 50,000 passages unless it is imported in pieces. A text, Markdown or JSON Lines file over 8 MB is imported in pieces and is not held to that figure (one finished at 144,247 passages on an iPhone 16 Pro Max on 2026-10-10). Any other file whose text would need more is refused with a message; through 5.6 the chunker stopped at the limit and said nothing. CSV, HTML and Office files are still read whole whatever their size, and a .json file is read whole up to 64 MB, as records up to 16 MB and as plain text above that. [evidence_level: code_verified, confidence: high, evidence_source: SemanticChunker.swift maxChunks; DocumentProcessor.swift importsInPieces and tooManyPassages; Docs/INGESTION_PIPELINE.md]
  • The two copies of the embedding model have been timed alone, not compared in use (5.7). The rates behind the 5.7 default (Core ML for a new library on iPhone and iPad, Core AI on macOS 27) are for the model by itself. How closely the two copies’ vectors agree, and how the two compare across a whole import, have not been measured, which is why no existing library is moved from one to the other. [evidence_level: measured for the model-only rates, confidence: medium, evidence_source: Docs/ai/DECISIONS.md 2026-10-10]

Demo Limits

Demo flows should use generic sample documents. Do not use private credentials, employer material, hospital-specific material, patient data, or sensitive personal data in public examples.