Boundary layer edge is usually defined as the position where the velocity is 0.99 times of the main flow. However, the main flow in complicated flow field is often non-uniform, which makes it difficult to accurately identify the boundary layer. To solve this problem, a boundary layer edge detection method is proposed by using “reference mainstream” instead of the actual mainstream. The reference main flow which is not disturbed by the boundary layer is obtained by numerical simulation under zero shear stress slip wall boundary condition, and the actual main flow velocity replaced by velocity at corresponding position in the reference main flow when the boundary layer edge is determined according to the boundary layer definition. Two supersonic compression flow fields including multi-ramp compression and curved surface compression are used to evaluate this detection method. The results show that relative error between the thickness of boundary layer at multi-ramp compression outlet station and the actual mainstream velocity is only 4.1%. According to the identification results of the method, the error between the bending shock height and the inviscid design value is reduced from 2.0% to 0.3%, and the relative error of the end pressure of the compression surface is reduced from 6.6% to 2.3%. This detection method could avoid the arbitrariness when determining the main flow velocity, and the recognition result is accurate.