Aleksander Kalacun

Abstract

Although evolving algorithms are based on principles of biological evolution, such as reproduction, mutation, and selection, they still obey the von Neumann principle "Program = algorithm + data". These algorithms remain rigid in the sense that their code does not change in computer memory during execution; their evolvability mainly concerns the data they process. In this paper, we introduce a novel variant of differential evolution that can adapt at the code level, overcoming this rigidity. In our proposed method, differential evolution mutation strategies are represented as short symbolic programs. Periodically, the system synthesizes new operators by mutating the top-performing operators through variable swaps, F-scaling, and the addition of difference terms. It merges these operators into a deduplicated library with a size cap and assigns credit based on empirical success using a softmax function. After each phase, the method rewrites its own program by serializing the evolved operator library and the best solution back into its source code and re-executing it, producing heritable, cumulative changes to the program itself. On canonical benchmarks, the optimizer discovers effective, previously absent operators and improves across phases, illustrating a practical path toward structural replication and self-adaptation.

Abstract

This paper presents a new method for dynamic, partially observed multi-agent Markov decision processes with stochastic elements. We use this formulation to define and simulate real-world problems and represent them as graphs. Existing multi-agent reinforcement learning algorithms struggle in changing environments that lack consistency. The core idea of the proposed algorithm is based on the deep autoregressive model GraphRNN, enabling long-term goal memorization and greater resilience to stochastic changes in the environment. Policy decisions are based on graph convolutional network encoders. Using observations and information communicated by other agents through graph attention network encoders, agents decide on the best next state. To demonstrate the method's capabilities, we implemented new graph-based environments. We used these environments to evaluate multi-agent reinforcement learning algorithms on problems resembling real-world situations, providing an effective benchmark.

Abstract

Causal reinforcement learning agents, such as structural causal bandit agents, must not only find a policy that maximizes reward in an uncertain environment but also decide where to intervene: which variables to act on and which to leave to their natural, possibly confounded mechanisms. Causal agents use a three-level hierarchy to interact with their environment: observational, interventional, and counterfactual. Established structural causal bandit methods, such as POMIS, reduce the complexity of searching the intervention set from O(3^k) to O(2^k), where k denotes the number of arms, by using the causal diagram to prune targets that are never optimal. However, the POMIS intervention set deliberately represents the worst case and ignores environment-specific parameter restrictions implied by the available data.

This paper addresses that gap by introducing a second, parameter-dependent intervention-pruning layer. The problem is transformed into a tropical geometry representation that incorporates the parameters into the optimal-arm decision. The proposed pipeline consists of five steps: (1) calculate the POMIS-reduced arm set from the causal graph G; (2) decompose the environment parameterization using c-factorization and identify a reward distribution; (3) tropicalize the reward distribution; (4) apply parameter-dependent tropical pruning to remove only provably suboptimal arms, certified using a linear program; and (5) choose an action using an online policy for structural causal bandits. In the pruning step, arms are represented by a tropical surrogate in which each monomial dominates the others in the observed tropical rational functions containing the reward distributions.

The proposed method is policy-agnostic and introduces two variants of structural causal bandit algorithms: the upper confidence bound method PDTP-UCB and the Thompson sampling method PDTP-TS. It applies to environments that satisfy the stated assumptions. We evaluated the approach on synthetic drug-combination benchmarks with k ∈ {3, 4, 6} arms. Across 100 independent runs with a time horizon of T = 10,000, PDTP-UCB reduced the UCB1-on-POMIS assignment-arm set from 12 arms to 2, from 36 to 4, and from 144 to 5, respectively. PDTP-UCB also reduced final cumulative regret relative to UCB1-on-POMIS by a factor of approximately 11 to 19. Replacing the online policy with Thompson sampling improved the results further.