--- title: "Seeing Through Uncertainty: A Free-Energy Approach for Real-Time Perceptual Adaptation in Robust Visual Navigation" authors: ["Maytus Piriyajitakonkij", ", Rishabh Dev Yadav", ", Mingfei Sun", ", Mengmi Zhang", ", and Wei Pan", "."] url: "https://arxiv.org/abs/2403.01977" sections: 20 estimated_tokens: "14.2k" --- ## Contents - I Introduction - II Related Work - II-A The Free Energy Principle in Robotics - II-B Test-time Adaptation - III Methods - III-A Point-Goal Navigation Problem Formulation - III-B End-to-End Visual Navigation Models - III-C FEP-Nav: Implementation - III-D FEP-Nav minimises Variational Free Energy - Assumption 1 (Mean-Field Approximation) . - Assumption 2 (Spatial Ergodicity) . - Proposition 1 . - IV Experiments - IV-A Visual Corruptions - IV-B Baselines for Simulation Test - IV-C Evaluation Metrics for Simulation Test - IV-D Testing FEP-Nav on a real-world robot - IV-E Results - V Conclusion - References ## Abstract Abstract Navigation in the natural world is a feat of adaptive inference, where biological organisms maintain goal-directed behaviour despite noisy and incomplete sensory streams. Central to this ability is the Free Energy Principle (FEP), which posits that perception is a generative process where the brain minimises Variational Free Energy (VFE) to maintain accurate internal models of the world. While Deep Neural Networks (DNNs) have served as powerful analogues for biological brains, they typically lack the real-time plasticity required to handle abrupt sensory shifts. We introduce FEP-Nav, a biologically-inspired framework that implements real-time perceptual adaptation for robust visual navigation. By decomposing VFE into its constituent components—prediction error and Bayesian surprise—we propose a dual-mechanism architecture: a Top-down Decoder that provides an internal expectation of uncorrupted sensory input, and Adaptive Normalisation that dynamically aligns shifted feature distributions with prior beliefs. Theoretically, we demonstrate that this integration of reconstruction and normalisation provides a formal mechanism for minimising VFE during inference without the need for gradient-based updates. Evaluations across a diverse suite of simulated and real-world visual corruptions demonstrate that FEP-Nav facilitates a substantial recovery of navigation performance, consistently exceeding the capabilities of both non-adaptive baselines and strong adaptive methods. We show that bridging machine learning with the brain’s variational principles offers a robust strategy for autonomous behaviour, enabling robots to remain functional under sensory conditions that typically degrade the performance of standard adaptive models. ## I Introduction Navigation is a fundamental capability of living organisms, from bacteria to primates. Humans and other highly intelligent animals show a remarkable ability to navigate under uncertainty. For instance, when raindrops suddenly splash on a person’s eyeglasses, visual input becomes partially corrupted, yet the person can still continue toward the destination without stopping to clear the view. This raises the fundamental question: what mechanisms allow humans to maintain adaptive perception and action when facing noisy and incomplete sensory information? The leading hypothesis in theoretical neuroscience states that perception is a generative process of constructing the most probable representations or embeddings $Z$ of the world. The representations $Z$ are learnt and updated by minimising *Variational Free Energy (VFE)*, formalised within the normative framework known as the *Free Energy Principle (FEP)* [8, 33]. *VFE* essentially contains two terms: prediction error and Bayesian surprise. The first is to minimise the difference between top-down predictions from representation and the actual sensory inputs. The latter is to minimise the divergence between the representational distribution given sensory inputs and the prior belief. *Deep Neural Networks (DNNs)* have been used as the analogue of biological brains at multiple levels, from computational goals to algorithms that construct representations. Previous studies explore *DNNs* trained using the *VFE* objectives to propose computational hypotheses of how brains might work [40, 42, 7, 25, 28]. However, they do not consider how these *DNNs* can continuously update their predictions under real-time constraints and do not demonstrate their practicality to solve real-world tasks, especially under the condition that the sensory distribution is abruptly shifted, requiring instantaneous model updates. Figure: Figure 1: A non-adaptive navigation robot fails to reach the goal position when it faces visual corruption: A robot is tasked to move from start to goal position. Top: The robot fails to navigate from the bedroom to the other room under dim light condition. Bottom: With our method, the robot can successfully navigate to the goal. Refer to caption: 2403.01977v3/x1.png Figure: Figure 2: An overview of our proposed method (FEP-Nav): The pre-trained navigation agent comprises a Visual Encoder ($VE$) and a policy network. Left (Offline Learning): To minimise prediction error, a Top-down Decoder ($TD$) is trained via mean squared error (MSE) to reconstruct observations from $VE$’s extracted features, while the $VE$ remains frozen. Training utilises clean frames from a replay buffer of uncorrupted scenes. Right (Online Adaptation): To minimise Bayesian surprise during inference, the $TD$ is frozen and $VE$ BatchNorm layers dynamically update their normalisation statistics to account for sensory non-stationarity. Refer to caption: 2403.01977v3/x2.png Therefore, we propose a *real-time implementation of the FEP* to deal with perceptual uncertainty in both simulation and the real world. Consider a DNN-based perception model with low-level processing at earlier layers and high-level processing at later layers. *FEP* essentially relies on top-down signals that carry input prediction [18, 36, 23]. Therefore, we propose a *Top-down Decoder (TD)*, which receives the high-level features, reconstructs the inputs, and feeds reconstructed inputs back to the perception model. The perception model incorporates *Adaptive Normalisation (AN)*, which implicitly minimises the Bayesian surprise term of *VFE*. During test time, *AN* adjusts normalisation statistics, denoises input features, and enables the decoder to reconstruct cleaner inputs. Our method enables the robot to navigate using these reconstructions as surrogate inputs instead of the actual ones. *TD* is trained via offline self-supervised learning on uncorrupted inputs to learn representations capable of reconstructing clean observations. Importantly, the decoder can denoise the inputs during test time. The decoder output becomes the internal expectation of the world, allowing the robot to navigate based on what the clean input should look like, despite the corrupted input. To evaluate our proposed method, we created a navigation benchmark that includes eight simulated and four real visual corruptions to assess how navigation performance declines with each type of corruption, as shown in Fig. [3](#S2.F3). We show that the strong *DNN* baseline, trained with Reinforcement Learning (RL) on nearly 3 billion frames from diverse scenes, fails to navigate to the target position under trivial visual corruption (e.g., dimmed lighting), as shown in Fig. [1](#S1.F1), despite the fact that it achieves a near-perfect success rate on unseen indoor scenes without visual corruption. Our method substantially recovers navigation performance from non-adaptive baselines, which cannot adjust visual representation during test time. Our method also outperforms three state-of-the-art adaptive methods: DUA [30], TENT [44], and SHOT-IM [21], which update the visual representation during test time. Moreover, our light-weight method works in real robot navigation in real-time and can greatly recover the navigation performance across four visual corruptions. Our novel contributions are as follows: 1. We propose a real-time implementation of *Free Energy Principle (FEP)* for perception and showcase its practicality in visual navigation tasks in both a simulation and the real world. 2. We mathematically show that our method, which integrates two standard deep learning techniques, i.e., normalisation and reconstruction, minimises *Variational Free Energy (VFE)*, thereby offering theoretical insight into the connection between machine learning, robotics, and neuroscience. ## II Related Work ### II-A The Free Energy Principle in Robotics The *Free Energy Principle (FEP)* in robotics is mainly implemented as the *Active Inference Framework (AIF)* [33], which unifies state estimation [4, 26], control, and learning [2] into a single optimisation problem [6, 20]. As scaling these agents to high-dimensional state-spaces requires *DNNs*, recent studies leverage them to drive more complex behaviours [25, 28, 27, 47]. Fountas et al. [7] mark a seminal advancement in this domain, showing the efficacy of deep active inference in high-dimensional state spaces. Similarly, Fujii and Murata [9] introduce temporally hierarchical world models to enable computationally efficient planning, while Vijayaraghavan et al. develop linguistic and sensorimotor compositionality for generalizing to novel tasks [43]. While *AIF* unifies perception and action, Lanillos and Cheng [19] demonstrated the utility of pure *Perceptual Inference* for robust state estimation, minimising *VFE* only to update internal beliefs. Adopting this stance, we apply *VFE* minimisation to representation adaptation. This decoupling enables us to leverage pretrained high-performance policies from Reinforcement Learning (RL) and Imitation Learning (IL), avoiding the complexity in training deep *AIF* agents from scratch. Importantly, a distinct feature of our method is its reliance on only the feedforward operation of the *DNNs* without gradient updates, enabling light-weight, real-time adaptation. ### II-B Test-time Adaptation The idea of updating the model on incoming test data against the sensory shift is introduced as Test-Time (Domain) Adaptation (TTA). TTA enables models to dynamically adapt to unknown test distributions during inference [44, 21, 30, 22]. Despite its success in other vision domains [45, 22], real-time TTA for visual corruption in navigation is under-explored, as most SOTA methods are task-specific (e.g., assuming categorical outputs) and require fine-tuning time on the test batch. Previous studies also introduce TTA in robot navigation to improve a robot’s trajectories in novel simulated scenes without visual corruption, but their methods are not real-time and require a fine-tuning that needs data collection from a new test episode or feedback from humans [17, 15]. We introduce FEP-Nav, a novel approach with a theoretical connection to neuroscience that achieves superior performance to three TTA methods [44, 21, 30] and works in the real world. Figure: Figure 3: Visual Corruptions: Pairs consecutive images of the same corruption type are presented in temporal order, as indicated by the arrow. The top and middle rows show simulated corruptions in the Habitat simulation, while the bottom row shows real-world corruptions. Refer to caption: 2403.01977v3/x3.png ## III Methods This section explains the problem formulation of the point-goal navigation and the implementation of our proposed method. We then provide a theoretical link between our proposed method and Free-Energy Principle (FEP). ### III-A Point-Goal Navigation Problem Formulation In point-goal navigation, a robot is tasked with moving to the given relative direction: $(\Delta x_{\text{target}},\;\Delta y_{\text{target}})\triangleq(x_{\text{target}}-x_{\text{start}},y_{\text{target}}-y_{\text{start}})$ in a cluttered indoor scene. For instance, given the goal of moving to $(\Delta x_{\text{target}}=0.5m,\Delta y_{\text{target}}=-2m)$, the robot has to move to the target position $(x_{\text{start}}+0.5,y_{\text{start}}-2)$, where $(x_{\text{start}},y_{\text{start}})$ is the initial position. A shorter path leads to a higher reward. In our setting, the robot is equipped with a navigation model trained with clean RGB observations and GPS & Compass $(x_{\text{target}}-x_{t},y_{\text{target}}-y_{t})$. The GPS & Compass is derived from the error-free odometry $(\Delta x_{t},\Delta y_{t})$. Where $\Delta x_{t}=x_{t}-x_{t-1}$, $\Delta y_{t}=y_{t}-y_{t-1}$, and ($x_{t}$, $y_{t}$) is the position of the robot at time $t$. ### III-B End-to-End Visual Navigation Models Generally, an end-to-end visual navigation model can be decomposed into two parts: Visual Encoder (VE) and a policy network. VE gets an RGB observation and extracts useful features for the task. The recurrent policy network is then given the target destination and uses the extracted features along with the target location to output action [51]. The models can be trained with RL and/or IL. The overview architecture of the navigation models we used is in the purple frame in Fig. [2](#S1.F2). In our study, we adopt the SOTA navigation model from  [46], which is trained with 2.5 billion RGB frames on a point-goal navigation task. These RGB frames are obtained from Gibson $2+$ dataset (train) [49]. We will call this model *DD-PPO* in the remaining parts of the text. Our algorithm is based on modifying the batch normalisation (BatchNorm) statistics [13], that has been proven to deal with distribution shifts during test time [37, 30, 14], but the pretrained DD-PPO model has group normalisation (GroupNorm) [48]. Therefore, we replace all GroupNorm layers with BatchNorm and fine-tune the model on Gibson scenes for 120M frames using the same configurations as in [46]. Specifically, we adopt SE-ResNeXt-50 [12] for VE architecture and LSTM [11] with the hidden size of $512$ for the policy network. We use the same reward function as proposed in [46]. We will call this model as Pretrained-Nav. ### III-C FEP-Nav: Implementation Figure: Algorithm 1 FEP-Nav: Training and Inference FEP-Nav introduces *Top-down Decoder (TD)* denoted as $g_{\theta}$ that receives the output of the late layer of VE and predicts the reconstructed image $\hat{o_{t}}$. See Fig. [2](#S1.F2). *TD* comprises residual block modules adapted from architectures proposed in [5, 31]. This decoder produces RGB images with dimensions of $256\times 256$. It has 5.3M trainable parameters. The overall process is described in Algorithm [1](#alg1). Training: During the training of FEP-Nav, we freeze all VE parameters and train *TD* using mean squared error (MSE) reconstruction loss with samples from 72 training scenes of Gibson [49] without visual corruption. *TD* is trained with a very small subset of navigation data: 112k out of 2.5 billion frames. We follow the same optimisation hyperparameters as in [31]. Specifically, we use ADAM optimiser [16] with the learning rate of $2\times 10^{-5}$, and the momentum of $0.9999$. Adaptation: FEP-Nav allows statistics in the BatchNorm layers of the VE to be updated during test time. We call this Adaptive Normalisation (AN). BatchNorm layers normalise each feature map (channel) separately, meaning that pixels in a feature map are normalised by the same estimated moments. Suppose we have a CNN layer with a set of feature maps $Z=[Z^{1},\dots,Z^{C}]\in\mathcal{R}^{H\times W\times C}$ where $H\times W$ refers to width and height and $C$ is the number of feature maps. For simplicity, we consider each pixel $z$ of a feature map $Z^{i}$; where $i\in\{1,\dots,C\}$. Specifically, each batch normalisation layer normalises an incoming sample $z$ by the estimated mean and variance as in ([1](#S3.E1)), $$ $z_{\text{norm}}=\frac{z-\hat{\mu}_{k}}{\hat{\sigma}_{k}}\cdot\beta+\gamma,$ (1) $$ where $\beta$ and $\gamma$ are affine parameters that scale and shift the data distribution respectively. Instead of estimating the mean and variance by averaging samples in a batch, we set VE to update the mean and variance by the moving average equations as in ([2](#S3.E2)) and ([3](#S3.E3)) below, $$ $\displaystyle\hat{\mu}_{k}$ $\displaystyle=(1-\rho)\cdot\hat{\mu}_{k-1}+\rho\cdot\mu_{k},$ (2) $\displaystyle\hat{\sigma}^{2}_{k}$ $\displaystyle=(1-\rho)\cdot\hat{\sigma}^{2}_{k-1}+\rho\cdot\sigma^{2}_{k},$ (3) $$ where $\hat{\mu}_{k}$ is the estimated mean and $\hat{\sigma}^{2}_{k}$ is the estimated variance, $\mu_{k}$ and $\sigma^{2}_{k}$ are the sample mean and variance of the batch data, and $\rho$ is a hyperparameter called momentum, which determines the contribution of the most recent sample estimate ($\mu_{k},\sigma^{2}_{k}$) to the calculation of the average. Here we update $\hat{\mu}_{k}$ and $\hat{\sigma}^{2}_{k}$ every inference step so $\mu_{k}$ and $\sigma^{2}_{k}$ are computed from instance statistics. Our model can adapt to the visual change at every action step and does not have to wait until it has enough samples in the batch, which may be too late for the robot to adjust its behaviour. Finally, the pretrained decoder, acting as the model’s internal expectation of the world, uses the re-normalised representation to predict the image. The decoder remains frozen and is trained to infer the expected appearance based on the given representation. We introduce two variants of FEP-Nav: FEP-Nav Running and FEP-Nav Instance. FEP-Nav Running updates the statistics as in ([2](#S3.E2)) and ([3](#S3.E3)), where adaptive $\rho=f(t)<1$, while FEP-Nav Instance only uses instance statistics by setting $\rho=1$. In other words, FEP-Nav Instance reduces moving average to instance normalisation. ### III-D FEP-Nav minimises Variational Free Energy We ground our FEP-Nav method in the Free Energy Principle (FEP) [8, 3]. In this framework, the robot perceives the world by minimising *Variational Free Energy (VFE)*, which decomposes into prediction error and Bayesian surprise: $$ $\mathcal{L}_{\text{VFE}}=\underbrace{\mathbb{E}_{q_{\phi}(z|o)}[-\ln p_{\theta}(o|Z)]}_{\text{Prediction Error}}+\underbrace{D_{KL}[q_{\phi}(Z|o)\|p(Z)]}_{\text{Bayesian Surprise}}$ (4) $$ *TD* explicitly minimises the first term via offline decoder training. Here, we prove that *AN* implicitly minimises the second term during test time. To ensure the tractability of the probabilistic formulation and the validity of the statistical estimation using spatial dimensions, we adopt two standard assumptions: ###### Assumption 1 (Mean-Field Approximation) . We approximate the complex posterior distribution $q(Z|o)$ with a factorised variational density [32]. Specifically, we treat the feature at distinct spatial locations $i\in\{1,\dots,N\}$ as independent in the variational posterior: $$ $q(Z|o)\approx\prod_{i=1}^{N}q(z^{(i)}|o)$ (5) $$ This standard approximation makes the ELBO tractable. ###### Assumption 2 (Spatial Ergodicity) . The feature process is spatially ergodic [41]. This implies that the spatial average over a single realisation converges in probability to the ensemble expectation as the spatial dimensions $N$ grow: $$ $\displaystyle\hat{\mu}$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}z^{(i)}\xrightarrow{P}\mathbb{E}[z]$ (6) $\displaystyle\hat{\sigma}^{2}$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}(z^{(i)}-\hat{\mu})^{2}\xrightarrow{P}\text{Var}(z)$ (7) $$ This property justifies the use of estimated moments as proxies for the true ones. ###### Proposition 1 . Let $p(z)=\mathcal{N}(0,I)$ be the prior. Let the corrupted feature distribution be $q(z|o)=\mathcal{N}(\mu_{corr},\sigma_{corr}^{2})$, where the corruption induces a significant distribution shift $\Delta$ away from the prior parameters $(0,1)$. Let the normalised distribution be $q(z_{norm})$, computed using empirical statistics $\hat{\mu}$ and $\hat{\sigma}$. Given that the estimation error of the statistics is smaller than the distribution shift caused by corruption ($\epsilon\ll\Delta$), the KL divergence decreases: $$ $D_{KL}(q(z_{norm}|\cdot)\|p(z))Instance (Ours) | | --- | --- | --- | --- | | Clean | 0.9 | 0.8 | 0.87 | | Color | 0.07 | 0.4 | 0.57 | | Light on Camera | 0.27 | 0.53 | 0.57 | | Dirt | 0.27 | 0.3 | 0.8 | | Light on Obstacle | 0.47 | 0.67 | 0.77 | | Average | 0.40 | 0.54 | 0.72 | | Minimum | 0.07 | 0.3 | 0.57 | ###### Assumption 1 (Mean-Field Approximation) . ###### Assumption 2 (Spatial Ergodicity) . ###### Proposition 1 . ## IV Experiments We first delineate the experimental setup, comprising visual corruptions, baseline methods, and evaluation metrics. The results are subsequently presented in the following sequence: (i) an analysis of the internal expectations reconstructed by the *TD*; (ii) a performance benchmark against adaptive baselines; (iii) a comparison between our generative approach and image restoration models; (iv) validation of real-time performance on a physical robot; and (v) an ablation study concerning hierarchical normalisation. Notably, all models are trained exclusively on clean observations to evaluate zero-shot adaptation to out-of-distribution shifts. ### IV-A Visual Corruptions The implementations of Defocus Blur, Motion Blur, Lighting, Spatter, and Speckle Noise are from the Habitat robustness benchmark dataset [35]. The rest of the corruption types are implemented in the following ways: Light Out is implemented by reducing the brightness of each frame with a $50\%$ probability, for example, if we sample 100 frames we will be likely to have around 50 dark frames. Colour Jitter is implemented by randomly adjusting the brightness, contrast, saturation, and hue of a frame. Real-world visual corruptions are implemented as follows. Light on Obstacle is achieved by projecting disco lights onto a black suitcase. Light on Camera is achieved by projecting disco lights onto a camera. Dirt is implemented by applying dirt to a camera. Color is implemented by covering a camera with color films. ### IV-B Baselines for Simulation Test We compare the proposed FEP-Nav methods with five baselines: Pretrained-Nav, DD-PPO, TENT, SHOT-IM, and DUA. Among these baselines, TENT, SHOT-IM, and DUA are task-agnostic and can be seamlessly integrated into our navigation model. All baselines are evaluated with fourteen unseen scenes and 994 episodes [49] used in [29]. We use Habitat platform [24, 39, 34] as an environment. The robot can take 4 actions: move forward (0.25m), turn left (10°), turn right (10°) and stop. An episode ends when the robot takes a stop action. During the evaluation, we allow the robot to take up to 500 environment steps in an episode, i.e., timed-out steps. Parameters of DD-PPO and Pretrained-Nav are frozen during testing, while others have adaptation. Pretrained-Nav is our navigation model that is fine-tuned from DD-PPO as explained in [section III](#S3). The model is trained with 72 training scenes as in [29]. DD-PPO [46] is the SOTA on a point-goal navigation task. See [section III](#S3) for more details. DUA [30] has the same model architecture as Pretrained-Nav, with the difference in that DUA updates the BatchNorm statistics $\hat{\mu}_{k}$ and $\hat{\sigma}_{k}$ during test time with running mean and variance as explained in ([2](#S3.E2)) and ([3](#S3.E3)). TENT [44] is initialised with Pretrained-Nav’s architecture and parameters. The model is then fine-tuned with entropy loss during test time. We follow the original implementation of TENT. SHOT-IM [21] is initialised with the architecture and parameters of Pretrained-Nav. During test time, the model undergoes fine-tuning using the Information Maximisation (IM) loss. ### IV-C Evaluation Metrics for Simulation Test The evaluation metrics below are used for the methods tested on Habitat environment. Success Rate ($SR$) measures the proportion of success episodes among all evaluation episodes. An episode is deemed to be successful if an agent takes stop action within $0.2m$ distance to a target position. Success weighted by Path Length ($SPL$) is proposed in [1]. We follow the same definition. Particularly, $SPL=S\frac{l}{\max(l,p)}$, where $S$ is a binary indicator of success in an episode. $l$ is the shortest path between the start and target of an episode, and $p$ is the length of a path the agent takes. $SPL$ is a strict metric in the sense that in order to achieve $SPL=1$ the agent must make no mistake, successfully arrive at the destination, and follow one of the shortest paths. Regarding the last two rows of [Table I](#S3.T1), Average is the average of SR or SPL over all corruptions. Minimum is the minimum SR or SPL over all corruptions, delineating the lower bound of navigation performance. **TABLE III: Performance Comparison: Image Restoration vs. our method.** | Corruption Type | MPRNet | FEP-Nav (Ours) | | | | --- | --- | --- | --- | --- | | SR | SPL | SR | SPL | | | Defocus Blur | 0.92 | 0.78 | 0.85 | 0.67 | | Motion Blur | 0.96 | 0.85 | 0.86 | 0.70 | | Rain | 0.79 | 0.57 | 0.93 | 0.81 | | Speckle Noise | 0.89 | 0.74 | 0.94 | 0.81 | ### IV-D Testing FEP-Nav on a real-world robot Continuous-action Navigation Model: During data collection, reflective markers are attached to the drone, and motion capture cameras are used to track its position and orientation. Instead of performing actual flights, the drone is manually moved by humans from the starting point to the target point to simulate its movement. During testing, the drone’s velocity is used to estimate its odometry. We collected around 300 episodes of data in a single room, which amounts to approximately 3 hours of data. The navigation model we use has the same architecture as the one implemented in the Habitat simulation. The *Visual Encoder (VE)* is initialised with the Habitat navigation model weights, but the policy is initialised from scratch due to differences in the action space. The output action from the model is $a_{t}=(x_{t+1}-x_{t},y_{t+1}-y_{t},z_{t+1}-z_{t},\theta_{t+1}-\theta_{t})$ [38], where $x,y,z$ are in the drone’s coordinate system, and $\theta$ represents the yaw angle. The navigation model is fine-tuned using behavioural cloning with $L1$ loss between the predicted action and the ground-truth action. We call the finetuned model from Pretrained-Nav weights as Vanilla Policy. During testing, we omit the $z$ action because we do not train it to fly at different altitudes. Evaluation Task: The drone is placed in front of the suitcase at a distance of approximately 0.4 to 1 meter. The drone must navigate to the backside of the suitcase without colliding with it to successfully complete an episode. We use the Success Rate (SR) as the performance metric. **TABLE IV: Ablation Study: Applying *AN* to different parts of the model** | Configuration | Average | Minimum | | | | --- | --- | --- | --- | --- | | SR | SPL | SR | SPL | | | No Adapt | 0.84 | 0.65 | 0.64 | 0.36 | | Block 1 | 0.91 | 0.76 | 0.81 | 0.63 | | Block 2 | 0.90 | 0.76 | 0.81 | 0.62 | | Block 3 | 0.90 | 0.76 | 0.78 | 0.60 | | All | 0.92 | 0.80 | 0.85 | 0.67 | Figure: Figure 4: Top-down Decoder (TD) with Adaptive Normalisation in FEP-Nav can reconstruct images from the unforeseen corrupted images: The top-row images are the inputs of Visual Encoder (VE). The bottom-row images are reconstructions from the same VE’s inputs. Refer to caption: 2403.01977v3/x4.png ### IV-E Results Adaptive Normalisation denoises images: Fig. [4](#S4.F4) shows that the reconstruction from *TD* is brighter, sharper, and clearer than the original inputs which are dimmed, blurred, and noisy, respectively. Additionally, the reconstruction contains fewer water and rain droplets compared to the raw images. Furthermore, the colour in the reconstruction returns to normal when the input image is tinted yellow. FEP-Nav outperforms other methods on navigation under visual corruptions: FEP-Nav Instance shows remarkable improvement over all baseline methods, achieving above $90\%$ SR and above $78\%$ SPL on the four most severe corruptions. Additionally, FEP-Nav Instance achieves the highest average SR and SPL across all types of visual observations(^1^11As shown in [Table I](#S3.T1), Speckle Noise, Lighting, Spatter, Rain, and Defocus Blur can be classified as severe corruptions due to the substantial performance degradation observed in column 4, which is the SOTA navigation model (DD-PPO).). Colour Jitter and Light Out are corruption types where DUA, the second best method, does not improve or minimally improves SR and SPL compared to its non-adaptive counterpart, Pretrained-Nav, while FEP-Nav Instance improves SR and SPL by large margins over Pretrained-Nav. In the case of Colour Jitter, FEP-Nav Instance outperforms DUA by $12\%$ in SR and $19\%$ in SPL. Furthermore, under Light Out corruption, FEP-Nav Instance outperforms DUA by $22\%$ in SR and $34\%$ in SPL. Generally, FEP-Nav Instance either ranks top 1 or 2. Its overall average and minimum performance are the best among all methods. We also observe that TENT performs the worst among all the methods. Moreover, SHOT-IM [21], which closely resembles TENT, also performs poorly. The poor performance of these methods is due to temporal correlation, which is an inherent characteristic of navigation images. A recent study [10] shows that TENT [44] has poorer performance compared to the same model without adaptation when the test data stream exhibits temporal correlation. These results align with the findings of [10]. FEP-Nav helps the robot avoid the obstacle: We apply two TTA methods that work well in the simulation, DUA and FEP-Nav Instance, to the Vanilla Policy, which is fine-tuned from the Pretrained-Nav model using behavioural cloning. *TD* is trained on both 112k Gibson images and additional real-world images, which are used to train the drone navigation model. FEP-Nav Instance outperforms DUA across all types of visual corruption, as shown in Table [II](#S3.T2). Interestingly, DUA barely improves the success rate over the Vanilla Policy when facing dirt on the camera, while FEP-Nav maintains a high success rate of $80\%$. Therefore, FEP-Nav Instance is the best TTA method for handling real-world corruption types. Comparison with Image Restoration: We evaluate FEP-Nav against MPRNet [50], a strong model for image restoration. While MPRNet is explicitly trained on clean-corrupt image pairs to map degraded inputs back to a clean groundtruth, its efficacy remains constrained by the specific noise distributions present in its training data. As shown in Table [III](#S4.T3), MPRNet underperforms our method on *Rain* and *Speckle Noise* corruptions. We attribute this to a distribution shift: although the simulated corruptions may visually resemble the model’s training set, a neural network may perceive the underlying statistical noise differently, leading to suboptimal restoration. In contrast, FEP-Nav does not require paired clean-corrupt data. By training solely on uncorrupted frames, our method treats all sensory shifts as out-of-distribution challenges to be resolved through implicit variational inference. This independence from corruption-specific priors allows FEP-Nav to maintain functional stability across a broader spectrum of unforeseen noise types where specialised restoration models fail to generalise. FEP-Nav allows real-time adaptation: FEP-Nav requires an additional inference stage, which increases computational overhead. However, the encoding and decoding overhead in FEP-Nav only requires $0.045$ s per frame and around $150$ MB of graphics memory on the Nvidia Jetson AGX Orin 32GB. Therefore, FEP-Nav is capable of functioning in real-time systems. Due to the Tello’s limited payload capacity, the computer was placed at a ground station for this experiment. Ablations: We study which layer with adaptation has the most effect on average and minimum SR and SPL scores on Gibson scenes. No Adapt refers to no update on normalisation parameters, Block 1,2 and 3 refer to update normalisation parameters in VE’s block 1,2, and 3, respectively, while freezing the others. All refers to updating all normalisation parameters in the networks. Block definitions follow SE-ResNeXt-50 [12], which is the architecture of our VE. The last row in [Table IV](#S4.T4) suggests that updating BatchNorm statistics across all blocks is the best choice, which is used in [Table I](#S3.T1) and [Table II](#S3.T2). Figure: Figure 5: FEP-Nav helps the robot avoid the obstacle when the drone’s camera is perturbed by lights: (Left) The drone with the non-adaptive navigation policy crashes into the suitcase when the disco light is applied to the drone camera. (Right) The FEP-Nav Instance allows the drone to adapt to the visual corruption and successfully avoid the suitcase. Red curve indicates the drone trajectory. Refer to caption: 2403.01977v3/x5.png ## V Conclusion We present FEP-Nav, a real-time implementation of Free-Energy Principle for robust visual navigation. By partitioning Variational Free Energy (VFE) into prediction error and Bayesian surprise, our architecture utilises a Top-down Decoder and Adaptive Normalisation to mitigate sensory non-stationarity without the requirement for online gradient updates. Evaluations across simulated and real-world corruptions demonstrate that FEP-Nav substantially recovers navigation performance, outperforming existing adaptive methods such as DUA, TENT, and SHOT-IM. The system’s computational efficiency ensures its viability for real-time deployment on embedded robotic platforms, such as NVIDIA Jetson AGX Orin. Our results suggest that grounding robotic perception in these generative formalisms offers a principled pathway towards resilient autonomous agency in the wild. Proof of Proposition [1](#Thmproposition1) Let $J(\mu,\sigma)=D_{KL}(\mathcal{N}(\mu,\sigma^{2})\|\mathcal{N}(0,1))$ denote the divergence from the prior. We analyse the behaviour of $J$ under two conditions: unnormalised (corrupted) and normalised (adapted). Visual corruption induces a domain shift, causing the true moments of $z$ to deviate significantly from the prior. The divergence is: $$ $\text{RHS}=J(\mu_{corr},\sigma_{corr})=\frac{1}{2}\left(\mu_{corr}^{2}+\sigma_{corr}^{2}-1-\ln(\sigma_{corr}^{2})\right)$ (9) $$ We assume the corruption is significant ($|\mu_{corr}|\gg 0$ and $|\sigma_{corr}^{2}-1|\gg 0$), resulting in a large divergence value ($\text{RHS}\gg 0$). We normalise $z$ using empirical statistics $\hat{\mu}$ and $\hat{\sigma}$. Since the spatial resolution $N=H\times W$ is large, the statistical estimation errors $\epsilon$ are negligible relative to unity ($|\epsilon|\ll 1$). We approximate the normalised moments as $\tilde{\mu}\approx-\epsilon_{\mu}$ and $\tilde{\sigma}\approx 1-\epsilon_{\sigma}$. The divergence becomes: $$ LHS $\displaystyle=\frac{1}{2}\left[\tilde{\mu}^{2}+\tilde{\sigma}^{2}-1-\ln(\tilde{\sigma}^{2})\right]$ (10) $\displaystyle\approx\frac{1}{2}\left[(-\epsilon_{\mu})^{2}+(1-\epsilon_{\sigma})^{2}-1-2\ln(1-\epsilon_{\sigma})\right]$ (11) $$ Using the Taylor expansion $\ln(1-x)\approx-x-x^{2}/2$, which is valid given $|x|\ll 1$: $$ LHS $\displaystyle\approx\frac{1}{2}\left[\epsilon_{\mu}^{2}+(1-2\epsilon_{\sigma}+\epsilon_{\sigma}^{2})-1-2(-\epsilon_{\sigma}-\frac{\epsilon_{\sigma}^{2}}{2})\right]$ (12) $\displaystyle=\frac{1}{2}\left[\epsilon_{\mu}^{2}+1-2\epsilon_{\sigma}+\epsilon_{\sigma}^{2}-1+2\epsilon_{\sigma}+\epsilon_{\sigma}^{2}\right]$ (13) $\displaystyle=\frac{1}{2}\left[\epsilon_{\mu}^{2}+2\epsilon_{\sigma}^{2}\right]$ (14) $$ The LHS is of second order in the tiny error term ($\mathcal{O}(\epsilon^{2})$), effectively approaching zero. In contrast, the RHS is of second order in the significant corruption shift ($\mathcal{O}(\Delta^{2})$). Since $|\epsilon|\ll\Delta$, it follows that $\text{LHS}\ll\text{RHS}$. Thus, Adaptive Normalisation reduces the Bayesian surprise. ## References - [1] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, et al. (2018) On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757. Cited by: [§IV-C](#S4.SS3.p2.7). - [2] M. Baioumy, P. Duckworth, B. Lacerda, and N. Hawes (2021) Active inference for integrated state-estimation, control, and learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4665–4671. Cited by: [§II-A](#S2.SS1.p1.1). - [3] M. Baioumy, C. Pezzato, R. Ferrari, and N. Hawes (2022) Unbiased active inference for classical control. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 12787–12794. Cited by: [§III-D](#S3.SS4.p1.1). - [4] F. Bos, A. A. Meera, D. Benders, and M. Wisse (2022) Free energy principle for state and input estimation of a quadcopter flying in wind. In 2022 International Conference on Robotics and Automation (ICRA), pp. 5389–5395. Cited by: [§II-A](#S2.SS1.p1.1). - [5] R. Child (2020) Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, Cited by: [§III-C](#S3.SS3.p1.3). - [6] L. Da Costa, P. Lanillos, N. Sajid, K. Friston, and S. Khan (2022) How active inference could help revolutionise robotics. Entropy 24 (3), pp. 361. Cited by: [§II-A](#S2.SS1.p1.1). - [7] Z. Fountas, N. Sajid, P. Mediano, and K. Friston (2020) Deep active inference agents using monte-carlo methods. Advances in neural information processing systems 33, pp. 11662–11675. Cited by: [§I](#S1.p3.1), [§II-A](#S2.SS1.p1.1). - [8] K. Friston (2010) The free-energy principle: a unified brain theory?. Nature reviews neuroscience 11 (2), pp. 127–138. Cited by: [§I](#S1.p2.2), [§III-D](#S3.SS4.p1.1). - [9] K. Fujii and S. Murata (2025) Real-world robot control by deep active inference with a temporally hierarchical world model. IEEE Robotics and Automation Letters 11 (1), pp. 890–897. Cited by: [§II-A](#S2.SS1.p1.1). - [10] T. Gong, J. Jeong, T. Kim, Y. Kim, J. Shin, and S. Lee (2022) NOTE: robust continual test-time adaptation against temporal correlation. Advances in Neural Information Processing Systems 35, pp. 27253–27266. Cited by: [§IV-E](#S4.SS5.p4.1). - [11] S. Hochreiter and J. Schmidhuber (1997) Long short-term memory. Neural computation 9 (8), pp. 1735–1780. Cited by: [§III-B](#S3.SS2.p2.1). - [12] J. Hu, L. Shen, and G. Sun (2018) Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141. Cited by: [§III-B](#S3.SS2.p2.1), [§IV-E](#S4.SS5.p8.1). - [13] S. Ioffe and C. Szegedy (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448–456. Cited by: [§III-B](#S3.SS2.p2.1). - [14] J. Kang, N. Kim, J. Ok, and S. Kwak (2024) MemBN: robust test-time adaptation via batch norm with statistics memory. In European Conference on Computer Vision, pp. 467–483. Cited by: [§III-B](#S3.SS2.p2.1). - [15] S. Kim, G. Oh, H. Ko, D. Ji, D. Lee, B. Lee, S. Jang, and S. Kim (2025) Test-time adaptation for online vision-language navigation with feedback-based reinforcement learning. In International Conference on Machine Learning, Cited by: [§II-B](#S2.SS2.p1.1). - [16] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: [§III-C](#S3.SS3.p2.2). - [17] H. Ko, S. Kim, G. Oh, J. Yoon, H. Lee, S. Jang, S. Kim, and S. Kim (2025) Active test-time vision-language navigation. In Advances in Neural Information Processing Systems, Cited by: [§II-B](#S2.SS2.p1.1). - [18] G. Kreiman and T. Serre (2020) Beyond the feedforward sweep: feedback computations in the visual cortex. Annals of the New York Academy of Sciences 1464 (1), pp. 222–241. Cited by: [§I](#S1.p4.1). - [19] P. Lanillos and G. Cheng (2018) Adaptive robot body learning and estimation through predictive coding. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4083–4090. Cited by: [§II-A](#S2.SS1.p2.1). - [20] P. Lanillos, C. Meo, C. Pezzato, A. A. Meera, M. Baioumy, W. Ohata, A. Tschantz, B. Millidge, M. Wisse, C. L. Buckley, et al. (2021) Active inference in robotics and artificial agents: survey and challenges. arXiv preprint arXiv:2112.01871. Cited by: [§II-A](#S2.SS1.p1.1). - [21] J. Liang, D. Hu, and J. Feng (2020) Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pp. 6028–6039. Cited by: [§I](#S1.p7.1), [§II-B](#S2.SS2.p1.1), [§IV-B](#S4.SS2.p1.2), [§IV-E](#S4.SS5.p4.1). - [22] W. Lin, M. J. Mirza, M. Kozinski, H. Possegger, H. Kuehne, and H. Bischof (2023) Video test-time adaptation for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22952–22961. Cited by: [§II-B](#S2.SS2.p1.1). - [23] W. Lotter, G. Kreiman, and D. Cox (2017) Deep predictive coding networks for video prediction and unsupervised learning. In International Conference on Learning Representations, Cited by: [§I](#S1.p4.1). - [24] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra (2019) Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: [§IV-B](#S4.SS2.p1.2). - [25] P. Mazzaglia, T. Verbelen, and B. Dhoedt (2021) Contrastive active inference. Advances in neural information processing systems 34, pp. 13870–13882. Cited by: [§I](#S1.p3.1), [§II-A](#S2.SS1.p1.1). - [26] A. A. Meera and P. Lanillos (2023) Adaptive noise covariance estimation under colored noise using dynamic expectation maximization. In 2023 62nd IEEE Conference on Decision and Control (CDC), pp. 165–171. Cited by: [§II-A](#S2.SS1.p1.1). - [27] C. Meo, G. Franzese, C. Pezzato, M. Spahn, and P. Lanillos (2022) Adaptation through prediction: multisensory active inference torque control. IEEE Transactions on Cognitive and Developmental Systems 15 (1), pp. 32–41. Cited by: [§II-A](#S2.SS1.p1.1). - [28] C. Meo and P. Lanillos (2021) Multimodal vae active inference controller. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2693–2699. Cited by: [§I](#S1.p3.1), [§II-A](#S2.SS1.p1.1). - [29] L. Mezghan, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari (2022) Memory-augmented reinforcement learning for image-goal navigation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3316–3323. Cited by: [§IV-B](#S4.SS2.p1.2). - [30] M. J. Mirza, J. Micorek, H. Possegger, and H. Bischof (2022) The norm must go on: dynamic unsupervised domain adaptation by normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14765–14775. Cited by: [§I](#S1.p7.1), [§II-B](#S2.SS2.p1.1), [§III-B](#S3.SS2.p2.1), [§IV-B](#S4.SS2.p1.2). - [31] K. Pandey, A. Mukherjee, P. Rai, and A. Kumar (2022) DiffuseVAE: efficient, controllable and high-fidelity generation from low-dimensional latents. Transactions on Machine Learning Research. Cited by: [§III-C](#S3.SS3.p1.3), [§III-C](#S3.SS3.p2.2). - [32] G. Parisi and R. Shankar (1988) Statistical field theory. Cited by: [Assumption 1](#Thmassumption1.p1.2.2). - [33] T. Parr, G. Pezzulo, and K. J. Friston (2022) Active inference: the free energy principle in mind, brain, and behavior. MIT Press. Cited by: [§I](#S1.p2.2), [§II-A](#S2.SS1.p1.1). - [34] X. Puig, E. Undersander, A. Szot, M. D. Cote, T. Yang, R. Partsey, R. Desai, A. Clegg, M. Hlavac, S. Y. Min, et al. (2023) Habitat 3.0: a co-habitat for humans, avatars, and robots. In The Twelfth International Conference on Learning Representations, Cited by: [§IV-B](#S4.SS2.p1.2). - [35] F. Rajič (2022) Robustness of embodied point navigation agents. In European Conference on Computer Vision, pp. 193–204. Cited by: [§IV-A](#S4.SS1.p1.1). - [36] R. P. Rao and D. H. Ballard (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature neuroscience 2 (1), pp. 79–87. Cited by: [§I](#S1.p4.1). - [37] S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge (2020) Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems 33, pp. 11539–11551. Cited by: [§III-B](#S3.SS2.p2.1). - [38] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine (2023) Gnm: a general navigation model to drive any robot. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7226–7233. Cited by: [§IV-D](#S4.SS4.p1.5). - [39] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al. (2021) Habitat 2.0: training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems 34, pp. 251–266. Cited by: [§IV-B](#S4.SS2.p1.2). - [40] K. Ueltzhöffer (2018) Deep active inference. Biological cybernetics 112 (6), pp. 547–573. Cited by: [§I](#S1.p3.1). - [41] D. Ulyanov, A. Vedaldi, and V. Lempitsky (2016) Instance normalization: the missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022. Cited by: [Assumption 2](#Thmassumption2.p1.1.1). - [42] O. van der Himst and P. Lanillos (2020) Deep active inference for partially observable mdps. In International Workshop on Active Inference, pp. 61–71. Cited by: [§I](#S1.p3.1). - [43] P. Vijayaraghavan, J. F. Queißer, S. V. Flores, and J. Tani (2025) Development of compositionality through interactive learning of language and action of robots. Science Robotics 10 (98), pp. eadp0751. Cited by: [§II-A](#S2.SS1.p1.1). - [44] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell (2020) Tent: fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, Cited by: [§I](#S1.p7.1), [§II-B](#S2.SS2.p1.1), [§IV-B](#S4.SS2.p1.2.4), [§IV-E](#S4.SS5.p4.1). - [45] W. Wang, Z. Zhong, W. Wang, X. Chen, C. Ling, B. Wang, and N. Sebe (2023) Dynamically instance-guided adaptation: a backward-free approach for test-time domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24090–24099. Cited by: [§II-B](#S2.SS2.p1.1). - [46] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra (2019) DD-ppo: learning near-perfect pointgoal navigators from 2.5 billion frames. In International Conference on Learning Representations, Cited by: [§III-B](#S3.SS2.p1.1), [§III-B](#S3.SS2.p2.1), [§IV-B](#S4.SS2.p1.2.2). - [47] N. Wirkuttis and J. Tani (2021) Leading or following? dyadic robot imitative interaction using the active inference framework. IEEE Robotics and Automation Letters 6 (3), pp. 6024–6031. Cited by: [§II-A](#S2.SS1.p1.1). - [48] Y. Wu and K. He (2018) Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3–19. Cited by: [§III-B](#S3.SS2.p2.1). - [49] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese (2018) Gibson Env: real-world perception for embodied agents. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on, Cited by: [§III-B](#S3.SS2.p1.1), [§III-C](#S3.SS3.p2.2), [§IV-B](#S4.SS2.p1.2). - [50] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M. Yang, and L. Shao (2021) Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14821–14831. Cited by: [§IV-E](#S4.SS5.p6.1). - [51] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi (2017) Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3357–3364. Cited by: [§III-B](#S3.SS2.p1.1).