Hybrid Vision-Language Models and On-Device Object Detection for Mobile Augmented Reality Learning using the NARV Algorithm

Abstract

Mobile Augmented Reality (MAR) is increasingly employed to deliver immersive instructional content in underserved classrooms, with object recognition typically offloaded to cloud-based vision-language models. The challenge encountered in such deployments diverges fundamentally from conventional assumptions. Empirical field measurements reported in this article demonstrate that the binding constraint is not elevated latency, but rather intermittent cloud service unreachability, rendering strategies built around extending timeouts or deferring transmissions until connection recovery of no benefit to students standing directly before physical objects awaiting an immediate answer. Existing reliability mechanisms recover passively through local caching and manual user intervention, steer computation based on input content complexity under simulated bandwidth throttling, or achieve network independence only by necessitating interconnected edge device clusters. This article proposes Network-State Adaptive Resilient Vision (NARV), a pipeline that combines bounded retry orchestration governed by linear backoff with a reactive substitution pathway to an on-device detector running continuously throughout the scanning phase, ensuring that substitute label sets are pre-accumulated and readily available the moment all cloud attempts are exhausted without incurring additional latency. Quantitative in-the-wild evaluations were conducted across 90 sessions from 90 unique participants on three heterogeneous Android hardware tiers in Sumbawa Besar, Indonesia. The pipeline achieved a continuity rate of 100.0%, with 84.4% of sessions resolved via the cloud pathway and 15.6% via on-device substitution. Out of 49 failed cloud attempts, 47 failed instantaneously with a median of 29 ms rather than reaching the timeout boundary, 30 out of 35 backoff schedulings executed within a 0.038-second deviation from the model, and memory growth remained bounded at 11.86 MB per session.