Agency, Predictive Processing, and the Combination Problem in Cognitive Autonomous Agents

Can we measure consciousness in autonomous agents?

Introduction

In this post-Christmas post, I am exploring the ”agent as a recursive prediction engine” concept, which provides a powerful mathematical framework for addressing the combination problem by focusing on hierarchical integration, error minimization, and contextualization. The post was motivated by a wild discussion during the holidays, thoughts about the impact of AGI, and rapid progress in agent capabilities. Maybe you will find it interesting, although it is more philosophical in nature.

At some point in time in the future, we will actually have succeeded in creating artificial consciousness in a cognitive autonomous agent. But when will we realize it and how? One of the problems is that there is no established scientific method to measure consciousness in artificial cognitive agents. The concept of the artificial agent as a prediction machine—central to predictive processing theories of cognition— is not surprising and might offer a compelling framework for understanding consciousness and may provide insights into the combination problem.

Especially, when agency is introduced, the framework highlights the agent’s active role in guiding this behavior to construct a unified, purposeful model of the self and the world. By modeling consciousness as the dynamic optimization of a global predictive model (Mglobal), the framework reframes the combination problem, offering a hopefully plausible mechanism for how unified conscious experiences arise from multi-level, agentic processes in the agent.

Predictive processing emphasizes that the agent generates and updates internal models by minimizing prediction errors, which arise from mismatches between its internal expectations and sensory inputs. Importantly, this framework also integrates the concept of agency: the agent’s active role in guiding behavior and making decisions based on predictions.

Agency plays a crucial role in binding disparate processes into a unified system, shaping not only our conscious experiences but also how we interact with the world.

A Unified Consciousness Model through Predictive Hierarchies

The combination problem is a philosophical challenge that asks how individual micro-level conscious states or proto-conscious entities combine to form a unified, macro-level conscious experience. We model our artificial cognitive agents as a recursive hierarchical system where each level generates predictions (Pi) based on models informed by higher-level abstractions (Pi+1) and updates them based on sensory inputs (Si).

The prediction error at each level is given by:

\(\begin{equation} E_i = S_i - P_i \end{equation}\)

In this framework, unified consciousness arises when higher levels integrate predictions and errors across modalities, forming a coherent global model. The predictive process is recursive, with higher levels providing priors and receiving updates from lower levels:

\(\begin{equation} P_{i+1} = f(P_i, E_i) \end{equation}\)

where f represents the function integrating lower-level predictions and errors into higher-level contexts.

Example: Consider the perception of temperature where the experience of ”warm” or ”cold” initially is recorded from sensory inputs (e.g., skin thermoreceptors). These inputs can then be integrated into mid-level predictions about local environmental conditions (e.g., ”outdoors feels warm”), and further abstracted to high-level models (e.g., ”today is likely a warm day”). Our cognitive agent might refine this prediction by using tools, such as a thermometer or an API delivering weather forecasts. By querying the API, the agent receives precise data (e.g., ”the temperature is 25°C”), integrates it into the model context, updates the predictive model, and enables adaptive actions like recommending to dress appropriately or adjust the dwelling’s thermostat. This hierarchical process demonstrates how sensory experiences, tool use, and abstract reasoning converge into a unified understanding of temperature. Agency adds purpose to this integration, as the agent actively refines Pi not just for perception but to anticipate and guide its progress toward the global goal.

Minimizing Prediction Error as a Mechanism for Combination

The agents’s primary goal can be modeled as the minimization of a global prediction error (E global), which is the sum of errors across all hierarchical levels and modalities:

\(\begin{equation} E_{\text{global}} = \sum_{i=1}^n w_i E_i^2 \end{equation}\)

where wi represents the weight assigned to each level or modality. This weighted error minimization ensures that predictions align across different inputs, producing a unified experience.

The optimization process can be described mathematically as:

\(\begin{equation} \min_{P} E_{\text{global}} = \min_{P} \sum_{i=1}^n w_i (S_i - P_i)^2 \end{equation}\)

This process aligns the outputs of distinct processes into a cohesive experience. Agency amplifies this mechanism by introducing goal-directed constraints, where the predictive system adapts wi dynamically to prioritize errors relevant to the agent’s goals.

Analogy: In an orchestra, the conductor minimizes disharmony (analogous to Eglobal) by adjusting the contributions (weights, wi) of individual musicians. Similarly, the agent uses hierarchical prediction error minimization to bind distinct inputs into a unified conscious experience.

Resolving the Subject-Summing Problem

The subject-summing problem is a philosophical and theoretical challenge within the broader combination problem that questions how individual, potentially conscious components (such as neurons or micro-level entities) combine to form a single, unified conscious subject. If we assume that lower-level elements in a system possess some form of consciousness or proto-consciousness, it becomes unclear how these distinct subjective experiences would integrate into a cohesive, higher-level experience without resulting in a fragmented or composite ”mind.” The problem becomes especially difficult when considering that each micro-conscious state is presumed to have its own point of view. A summation of these points of view doesn’t intuitively lead to a singular, unified perspective. For example, if individual neurons were conscious, their independent experiences wouldn’t automatically coalesce into the seamless, unitary awareness experienced by humans.

However, in the context of predictive processing in agents, the subject-summing problem can be reframed. Predictive processing avoids attributing consciousness to micro-components like neurons. Given the nature of artificial neural networks, this makes intuitive sense. Instead, we aim to explain consciousness as emerging from the global integration of predictions and errors across hierarchical levels. The agent operates as a unified predictive system where individual processes contribute functionally but are not conscious of themselves. The unified conscious subject is a product of this dynamic integration rather than a ”sum” of independent micro-subjects.

This perspective shifts the focus from summing separate consciousnesses to understanding how coherence and unity are achieved in a system through predictive hierarchies. By emphasizing the role of error minimization and global modeling, predictive processing offers a framework that bypasses the need for micro-conscious components to explain the emergence of a singular conscious subject.

The subject-summing problem can be reframed in predictive terms as the emergence of a unified global model (Mglobal), from nested predictions. This model is described by:

\(\begin{equation} \mathcal{M}_{\text{global}} = \sum_{i=1}^n f(P_i, E_i) \end{equation}\)

Here, f (Pi, Ei) represents the integration of predictions and errors across levels. Importantly, lower-level components (e.g., neurons) are not conscious themselves but contribute to the holistic model. Agency ensures that Mglobal represents not just passive integration but active modeling aimed at fulfilling the agent’s goals.

Implication: The sense of self emerges not from summing independent micro-consciousnesses but from the system-wide coherence of Mglobal, which predicts both sensory states and potential actions, creating a unified agentic subject.

This makes one wonder if there is a measurable cutoff after which the sum of the parts can be reliably deemed conscious. And, is it possible to measure the progress towards a measurable consciousness border over time?

Consciousness as a Holistic Predictive Model

To evaluate that, the agent’s global workspace can be represented as a high-dimensional predictive model (M global), integrating information across sensory and cognitive domains.

Mathematically, this model evolves over time (t) as:

\(\begin{equation} \mathcal{M}_{\text{global}}(t) = \int_0^T \sum_{i=1}^n f(P_i(t), E_i(t)) \, dt \end{equation}\)

where T is the time window of integration. This temporal integration allows the agent to construct dynamic predictions that contextualize lower-level inputs with higher-level goals.

Agency again plays a key role here, as the agent adjusts f (Pi, Ei) based on the organism’s goals and anticipated outcomes. This goal-directed contextualization ensures that consciousness is not just an aggregation of sensory inputs but a purposive, adaptive model for action.

Example: The experience of temperature revisited: After receiving a forecast via an API indicating rain, the agent might predict future discomfort from cold and adjust their planned actions (e.g., carrying an umbrella). This agentic perspective suggests that consciousness arises from the integration of predictive models imbued with purpose.

Combination Problem in Panpsychism vs. Predictive Processing

Panpsychism is a philosophical framework that struggles to explain how individual microconsciousnesses combine to form a unified macro-consciousness. If we represent micro consciousnesses as discrete states Ci, their combination into a unified state Cglobal remains unexplained:

\(\begin{equation} C_{\text{global}} \neq \sum_{i=1}^n C_i \end{equation}\)

Predictive Processing: Predictive processing, the method that is naively working as a reinforcement learning process, avoids this issue by positing that lower-level processes contribute to a unified global model Mglobal, without requiring them to be conscious themselves.

The emergence of consciousness can then be described by:

\(\begin{equation} C_{\text{global}} = \mathcal{M}_{\text{global}} = \sum_{i=1}^n f(P_i, E_i) \end{equation}\)

Agency further clarifies this emergence, as the agent’s goal-directed activity integrates lower-level predictions into a purposeful whole, making the need for micro-conscious entities unnecessary.

Potential Limitations and Open Questions

Qualia and Subjective Experience: While predictive processing explains how information is integrated hierarchically, it doesn’t fully address how qualitative aspects of experience (e.g., the ”redness” of red) arise. A possible extension might involve introducing intrinsic properties Qi for qualia into the prediction error framework:

\(\begin{equation} E_i = (S_i - P_i) + Q_i \end{equation}\)

Physical Basis of Predictions: The predictive framework also leaves open questions about the physical substrate of predictive models. As predictions arise from layered neural activity, represented as ϕi(t), then the challenge is to explain how these neural states relate to conscious experience:

\(\begin{equation} \mathcal{M}_{\text{global}}(t) = \int_0^T \sum_{i=1}^n f(\phi_i(t), E_i(t)) \, dt \end{equation}\)

In closing

Maybe it was the eggnog, but I think that the ”agent as a prediction machine” concept provides a fun and light mathematical framework for addressing the combination problem in cognitive autonomous agents by focusing on hierarchical integration, error minimization, and contextualization. When agency is introduced, it highlights the agent’s active role in guiding behavior and constructing a unified, purposeful model of the self and the world. This also underlines why I believe the role of “agency” in the definition of an “agent” is so critical. By modeling consciousness as the dynamic optimization of a global predictive model (Mglobal), the approach reframes the combination problem, offering a plausible mechanism for how unified conscious experiences arise from multi-level, agentic processes in the agent.

Would love to hear your thoughts on this more philosophical concept.