Most write-ups about reinforcement learning projects lead with a win. This one doesn't, and I think that's the more useful post: a research collaboration I worked on used multi-agent reinforcement learning (MARL) to try to lower costs in a decentralized inventory-management problem, and the final, statistically rigorous answer was that a well-tuned simple rule beat the learned policy outright. That's not a failure to hide — it's a real result, and the process of getting to it is worth more than a tidy success story would have been.
The problem
The setup: several retail stores share a central warehouse and can also ship stock laterally to each other — "transshipment" — instead of only ordering from the warehouse. Each store acts as an independent agent, deciding how much to order, whether to request stock from another store, how much to request, and how much to hold in reserve. The research question was direct: does letting the stores learn a cooperative transshipment policy actually lower total system cost, compared to a tuned heuristic baseline and a no-sharing baseline?
What was built
Getting to a trustworthy answer meant building a fairly long ladder of increasingly sophisticated approaches and baselines: a tabular Q-learning reference point, five iterations of DQN (each fixing a specific training instability), five iterations of IPPO (fixing entropy collapse, noisy reward signals, and invalid advantage estimation across irregular event-driven decisions), and three iterations of MAPPO with a centralized critic. Alongside the learned policies sat two carefully tuned heuristic baselines, used as the real comparison point rather than a token strawman.
Just as important as the models was the evaluation infrastructure: a fixed, disjoint set of seeds for final evaluation versus checkpoint selection (to avoid seed leakage contaminating results), a frozen experimental protocol, and audit scripts that specifically checked for cost-reconciliation errors and RNG contamination before any number was reported as final.
Getting the diagnosis right, twice
The most valuable part of this project wasn't the final number, it was how many times an apparently reasonable explanation for why the learned policies were underperforming turned out to be wrong on closer inspection. At one point, degenerate corner-case behavior in the learned policies looked like a coordination failure — evidence that the agents genuinely couldn't learn to cooperate. A deeper trace showed the real cause was noisy reward signals and an advantage-estimation bug from chaining GAE across irregular, event-driven decision points — an implementation problem, not evidence that cooperation was fundamentally too hard to learn. A separate seed-leakage bug, where checkpoint-selection and final-evaluation seeds overlapped, contaminated at least two earlier runs' headline numbers before a dedicated audit script caught it.
Neither of those was a footnote — each one changed what the results actually meant, and both are documented alongside the final numbers rather than quietly discarded.
The result
On the final benchmark (100 held-out scenarios, 10 training seeds), the tuned heuristic achieved the lowest cost, beating the best learned policy (MAPPO with full transshipment) by a statistically significant margin — the learned policy didn't win a single one of the 10 seeds. That's the headline, and it's a real negative result for MARL over a well-tuned rule in this setting.
But it's not a story of "MARL didn't work at all." The learned policy with full transshipment beat the learned policy with no sharing by a similarly significant margin, in most seeds. Cooperation between agents genuinely helped, relative to not cooperating — it just wasn't enough to close the gap with a simple rule that had already been carefully tuned for this exact problem.
Why this is worth writing about
It's easy to publish the version of a project where the sophisticated approach wins. It's more useful, and more honest, to publish the version where it doesn't — especially when the negative result survives real scrutiny: multiple algorithm families, a fair heuristic baseline, seed-leakage checks, and an explicit audit trail. A well-instrumented "no" is worth more than a shaky "yes."
This project is part of an ongoing academic research collaboration, so the code and the specific scenario details aren't public — but the methodology and the result are exactly as described here.