I tried to explain it, though I am unsure of the validity of this explanation because I am only a yr 11 student but this does feel logical.
Basically what I'm saying is that whilst from squaring and applying the pythagorean theorem we obtain the following equation for A:
A^2 = 12
it's not the ONLY equation that is concerned with A since there is also :
A cos (alpha) = sqrt (3)
and
A sin (alpha) = 3
Which we get from matching the coefficients of sqrt (3)sinx + 3cos x with the expansion of Rsin(x+alpha)
So you can think of it like three simultaneous equations that all must be satisfied simultaneously. This is only achievable with A=2sqrt(3)
if you try A=-2sqrt(3) you will satisfy A^2=12 BUT you will NOT satisfy A cos (alpha) = sqrt (3) or A sin (alpha) = 3, which are crucial for our expression from the auxiliary angle expansion to actually be equivalent since the coefficients must line up in order for them to be equal.