StarGateVR: AI System Architecture
Updated March 2026
How the voice-driven AI assistant works under the hood.
Early Access: The AI Assistant is available as an opt-in panel in the app (tap “AI Assistant— Early Adopter” to reveal). It is functional and in active development. Reliability is improving with each build as we refine intent classification and archive integration. Researcher feedback drives that development directly.
Speech Pipeline
StarGateVR’s AI processes voice interactions through four stages:
• Transcription — OpenAI Whisper converts spoken queries to text
• Intent Classification — A fast model (GPT-4o-mini) determines query type: general question, dataset analysis, enrichment request, or new dataset retrieval
• Reasoning & Execution — A full-capability model (GPT-4o) handles complex analysis and orchestrates database queries
• Response Synthesis — OpenAI TTS converts the text response back to speech
Tiered Model Approach
The system uses two models to balance speed and cost. GPT-4o-mini handles routine intent classification quickly and cheaply (~$0.0001/query). Only when a task requires deeper reasoning — constructing complex ADQL queries, interpreting multi-step enrichment requests, or synthesizing analysis — does the system escalate to GPT-4o. Typical voice interaction latency: 3–6 seconds. As models improve, the system improves incrementally without code changes.
AI-First Intent Classification
Rather than keyword matching (“if user says ‘add’…”), the system uses the LLM itself to classify intent. This handles the natural variation in how researchers phrase requests. When a request is ambiguous — for example, “get X-ray data” could mean add X-ray columns to the current dataset or fetch a catalog of X-ray-detected sources — the system asks for clarification with numbered options the user can select by voice.
Real Data First
The LLM never generates scientific data. Every measurement a researcher sees comes directly from established astronomical archives through deterministic queries. The LLM translates natural language into archive API calls — it is not a source of astronomical facts. For a detailed description of the data processing assumptions, quality cuts, and default parameters applied during queries, see the Hidden Defaults & Implicit Assumptions page.
Token Management and Consent
Database queries are bounded to control token usage and response time:
• ≤10 stars: automatic execution, no consent required
• 11–100 stars: system asks for user consent
• >100 stars: always samples down to 100
• >1000 stars: hard limit
• If estimated enrichment time exceeds 120 seconds, the system recommends sampling
For large enrichment requests, the system provides real-time progress updates with elapsed time visible in VR. Results for large datasets are chunked for transmission and recombined automatically.
Collaboration and Audio
In multi-user VR sessions, AI-generated datasets are shared with all collaborators automatically. A practical challenge in collaborative VR: researchers need to talk to each other and talk to the AI, but mobile VR hardware only allows one audio consumer at a time. The system solves this by tapping into the existing voice chat audio stream for AI recording — so a researcher can switch between group discussion and AI queries without interruption.
Development Stack
• VR Platform: Unity (Meta Quest)
• AI Backend: Python server with OpenAI API integration
• LLM Architecture: Modular — intent classification and reasoning layers can be swapped to alternative models
Frequently Asked Questions
How do you handle cross-matching accuracy?
Positional cross-match with configurable radius (default 5 arcsec), preferring source_id matching when available via Gaia cross-match tables. See the Hidden Defaults & Implicit Assumptions page for the full list of radii and quality filters.
What about LLM hallucination?
The LLM never generates scientific data. When the system reports an X-ray detection, that detection comes from eROSITA, not from the model’s training data. If there is no match in a catalog, you get an explicit null — not a plausible guess. For interpretive responses, the LLM-generated nature is transparent, and the underlying data is always visible in VR for direct inspection.
What does it cost per query?
API costs roughly $0.01–0.10 per substantive query, depending on complexity and number of archives queried.
What is the typical latency?
Simple voice queries: 3–6 seconds. Multi-database enrichment: 10–60 seconds depending on dataset size. Live Gaia ADQL queries: 30 seconds to several minutes depending on query complexity and archive load, with elapsed-time progress visible in VR.
Could this work with other LLM providers?
The architecture is modular. The intent classification and reasoning layers could be swapped to use other models with interface changes.