Description

This project addresses the need to manage the immense computational resources required by Large-Scale Language Models (LLMs). The central objective is to develop a framework that identifies the most appropriate parallelism configuration to maximize training performance. Using the AstraSim tool and its synthetic LLM generator, the research explores the search space to recommend optimal degrees of data, tensor, sequence, and pipeline parallelism. To increase simulation fidelity, the system integrates an optimized network model that accounts for the effects of congestion, providing critical insights into bottlenecks and design trade-offs for AI researchers and hardware engineers.