AI has dynamical systems at many levels of granularity
As you are probably guessing, unveiling the dynamics of intelligence is not straightforward. Like many other engineering systems, AI systems exhibit complex and carefully curated architectures that are responsible for the behaviors that we observe. For this reason, we study the dynamical systems that arise at different levels of granularity while accounting for their role in the full AI system's architecture:
AI generative models interact with the world by being embedded in cyber or physical systems, which are themselves dynamical systems: robots, recommendation platforms, chatbots, swarms of agents, etc. We can analyze and control how these systems interact with the world using tools from control theory!
See papers →Zooming in, an entire model's dynamics can also be studied as a closed-loop system. For instance, autoregressive loops or denoising steps can be seen as closed-loop dynamical processes.
See papers →Inside the deep architecture we find a sequence of stacked computing blocks, the layers. This can be treated as an unrolled dynamical system, where depth is seen as time. Treated as an input-output system agnostic to its internals, a stack of layers becomes a trajectory unfolding in time. Steering a model's activations during generation is then a control problem.
See papers →Inside each layer we find the mixer scaffold, a set of components such as normalization layers and skip connections. These components pre- and post-process the signal before and after the token mixer. By studying these components, we can gain important insights about their dynamics, both at inference and training time.
See papers →At the core of every architecture sits the block that mixes information across tokens: attention, the SSM recurrence, and their relatives are examples of this block. Studying how tokens evolve here clarifies what actually distinguishes one architecture from another, and what essential ingredients are needed to generate intelligent behavior.
See papers →Fun fact — Planes also have dynamical systems at different degrees of granularity: (i) the dynamics of an airfoil dictates lift, (ii) airfoils have structural components that provide stability and shape the airflow, (iii) stacking airfoils gives wings, which have complex aeroelastic dynamics, (iv) putting the wings, body, and tail together gives the dynamics of a plane, and (v) once in the sky, air traffic dynamics are also crucially controlled. All of these can be studied with tools from control theory!
Studying AI systems through the lens of dynamical systems holds great promise to achieve (a) principled design strategies, replacing today's trial-and-error approach (compute- and energy-intensive!), and (b) controllable AI architectures with guarantees, enabling the use of AI reliably in critical infrastructure. Classical control theory, however, does not directly translate to AI: most of its tools are not equipped to handle the sheer complexity, massive scale, sparse topologies, and local communication present in current AI systems. In the Control and Intelligent Systems Lab, we also extend control-theoretic tools to address these problems!