Advanced high-strength steel (AHSS) is a highly competitive material for the automobile industry to resolve the challenge of weight reduction and passenger safety. With the increase in strength, defects such as microcrack, local thickness reduction, and large corner radius in the corner section of cold roll-formed steel products become more and more serious. Cold and hot composite roll forming (CHCRF) is a newly developed manufacturing technology that combines the advantages of cold and hot forming to overcome the challenge of forming AHSS. In this paper, the mathematical models that represent the effect of line velocity, heating power, and deformation amount on the yield strength, outer corner radius, and microcrack length of the corner section of cold and hot composite roll-formed AHSS square tube are investigated by the response surface methodology (RSM) with Box-Behnken design (BBD). From the analysis of variance (ANOVA), it is found that all the three process parameters play a significant role in the outer corner radius and microcrack length, while the line velocity and heating power influence the yield strength significantly. Furthermore, an optimization process is proposed, with the outer corner radius and microcrack length as objective functions and the yield strength as a constrained condition, to manufacture the AHSS square tube with a small outer corner radius and no microcrack defect. Experimental verification has been conducted using the optimal process parameters, and the average magnitude of error is about 5%, which implies that the optimization design is accurate and reliable.