As an important component of the nuclear power plant, nuclear main pump has high requirement in terms of the operational stability. An important stability problem of the nuclear main pump unit is the unstable flow-rate-head ( Q-H ) hump characteristic, which is a nonmonotonic pump head variation within a specific flow rate range. To understand the unstable Q-H hump characteristic in the nuclear main pump, an engineering case is investigated in this study based on the computational fluid dynamics simulation and experiment in the cold state. In this case, the unstable Q-H hump region is found in a wide design flow rate range from 0.4 to 0.7. According to the hydraulic loss analysis, the Q-H hump is found to be relative to the increase in the hydraulic loss in the stators. When the flow rate decreases below 0.7 design flow rate, vortices occur in the suction chamber. The backflow structure is found near the impeller inlet. Large-scale vortices are also found in the guide vane passage and volute. These undesirable flow structures affect the flow regime, block the flow passage, and cause the increase in the hydraulic loss in all the stators. As a result of the superposition of high hydraulic loss, the increase in the impeller head is strongly counterbalanced. The pump head drops and forms the unstable Q-H hump characteristic. This study reveals the mechanism of the unstable Q-H hump characteristic in the nuclear main pump. The results provide a reference for the no-hump design and operation of the nuclear main pump unit.