Abstract：Due to the complex nonlinear characteristics often exhibited in the shear failure behavior of rock mass，the existing theoretical methods face challenges in analyzing the stability of fractured rock slopes. In this study，the differential control functions of the slip surface and its stress are established using the variational method. The calculation formula for the factor of safety(FOS) of the slope and the unknown variable in the stress function for the slip surface are derived by applying the limit equilibrium(LE) principle. To address the interdependent relationship among the shape of the slip surface，the normal stress on the slip surface，and the instantaneous shear strength parameters under the generalized Hoek-Brown(H-B) strength criterion，a discrete calculation approach and a coupled correlation solution strategy are implemented to directly tackle the difficulties introduced by the nonlinear strength criterion. On this basis，the global vraiational extremum problems are treated as multiple deterministic boundary variational extremum problems. With slip surface parameters as optimization variables and the minimum slope FOS along with variational extreme conditions as optimization objectives，a multi-objective optimization genetic algorithm was employed to accurately and efficiently search for the critical slip surface of the slope. This approach eliminates the moving boundary transboundary condition and strictly adheres to the variational control conditions. The feasibility and practicability of the proposed method are verified through comparisons with several examples. This research contributes to the advancement and refinement of the LE theory for slope stability and provides an effective computational framework for the stability analysis of fractured rock slopes under complex conditions.
