Data plane intelligence: AI-based optimization for traffic engineering and intrusion mitigation in next-gen networks

This paper explores a novel framework for deploying self-optimizing AI agents designed to enforce real-time security policies across dynamic broadband infrastructures. Given the rise of zero-touch networks, increasing traffic heterogeneity, and growing cyber threats, conventional reactive security methods are no longer sufficient. We propose an architecture that combines reinforcement learning (RL), federated observability, and edge-native threat detection. The paper introduces a scalable agent-based model with proactive anomaly detection and self-adjustment capabilities. Key contributions include a hybrid decision loop, a risk-weighted policy optimizer, and an adaptive trust index. The proposed solution is validated through simulations and real-world telecom KPIs. The results demonstrate enhanced mean time to detect (MTTD), reduced false positives, and improved threat response efficiency.