Abstract:The instability criterion due to shear localization in rock specimen system composed of shear band and elastic rock outside the band is studied. A few important phenomena observed in test are considered as basic assumption for obtaining closed-form analytical solution on the instability criterion. Whether shear instability occurs or not,some parameters,such the length of specimen,shear elastic modulus,shear softening modulus,internal length parameter (or the width of shear band) and the inclination angle of shear band,play important roles. If the system exhibits class II behavior,the shear instability occurs surely. The proposed instability criterion can be used to analyze the instability of coal pillar,i.e. coal pillar rock burst. The advantage of adopting gradient-dependent plasticity to analyze the problem of uniaxial compressive test subjected to shear failure is that the width of inclined shear band is involved in the present model and the distributed shear strain in shear band can be obtained strictly. The reasonability of theoretical results is also discussed.