Sustainable supply chain management through a digital twin-enabled federated deep reinforcement learning framework

Abstract

The need to have sustainable supply chain management demands the decision frameworks being jointly optimal in the economic performance and environmental impact without violating data decentralization across the stakeholders. The proposed study suggests multiple-level (federated) deep reinforcement learning (FedDRL) to the optimization of supply chains to unite inventory management, production strategy, and transportation decisions in conditions of an explicit carbon-emission penalty. The supply chain entities also train a local deep reinforcement learning agent on local operational data, and periodically, the aggregation of the parameters is done via federated averaging to create a global policy without collecting all the data in a central location. The reward is a joint-coding of overall operational cost and emissions through a sustainability weight l which can be tuned allowing the economic-environmental trade-off space to be steered. The framework was tested using large stochastic simulations of a variety of demand scenarios and studied with regards to Order-Up-To heuristics, unreinforced non-federated DRL, and centralized reinforcement learning baselines. Repeated-measures comparisons were used to provide a statistical significance with an expression of means of a significantly lower total cost and variance of the proposed approach compared to baselines (p < 0.01). A monotonic and statistically significant change in the results of cost-emission analysis with different l values was found, which indicated the stability of Pareto-efficiency. Altogether, the findings indicate that federated reinforcement learning is capable of providing almost centralized performance, high operational efficiency, and governability sustainability, which is a scalable and privacy-controlling solution to data-driven supply chain optimization.