Accurate wave simulation in semi-enclosed and shallow environments remains associated with significant uncertainties due to hydrodynamic complexities and limitations in the accuracy of incoming wind data. This study evaluates the performance of the third-generation SWAN model in simulating the wave field in the semi-enclosed Botany Bay, focusing on the enhancement of ERA5 wind data and the calibration of the model's physical parameters. To this end, ERA5 wind data were first assessed using meteorological station observations and subsequently corrected through several bias adjustment methods. The corrected wind data were then used as input for the SWAN model, and the model's sensitivity to three main physical packages—KOMEN, WST, and ST6—was examined with various combinations of wind input, white-capping coefficients, bed friction, and depth-induced breaking. Model performance was evaluated using classical statistical indices and diagnostic plots such as Taylor diagrams, probability density functions, and Q–Q plots at four coastal stations with differing hydrodynamic conditions. Results indicated that the WST package performed best at more open stations influenced by distant waves, whereas at more sheltered stations, targeted adjustment of white-capping coefficients and reduction of distant wave energy contribution in the KOMEN and ST6 packages led to significant improvements. This study demonstrates that combining ERA5 wind data correction via machine learning methods with targeted calibration of SWAN's physical parameters plays a decisive role in enhancing wave simulation accuracy in semi-enclosed environments, and that employing a uniform configuration across the entire bay may result in systematic errors in wave energy estimation.