Under the policy of “Being Bigger and Stronger” of audit firms in China, the scale of the office business has expanded rapidly. Based on this, this paper takes 2014—2018 China's A-share listed companies as a sample, empirically tests the relationship between the business growth of the office and audit quality, and further examines the impact of the audit firm's internal governance on this. The results show that: The growth of office business is significantly negatively correlated with audit quality, which indicates that the business growth of the office is at the expense of reducing the audit quality. Further analysis indicates these results are driven by the audit firms that have not performed the integrated governance. Moreover, declines in audit quality are more likely to occur in regions with high market concentration and among smaller clients. This paper contributes to the disclosure of the “black box” of the audit firm's internal governance, and it is of great significance to strengthen the supervision of the audit firm and strengthen the internal governance of the audit firm.