1) cis5A=(cos(A)+isin(A))^5
Expand the RHS and then equate the reals. It's better if you do it as it'll give you some practice.
2) Let x=cos(A) and cos(5A)=0.
<=> cos(5A)=0=16x^5-20x^3+5x (divide by x here)
<=> cos(5A)=0=16x^4-20x^2+5 (you have a typo here)
=> cos(5A)=0=cos(+-(2n+1)*pi/2) where n E Z+
=> 5A=+-(2n+1)*pi/2
=> A=+-(2n+1)*pi/10
Now, just sub in n values to get x=cos(+-pi/10,...,+-9pi/10)
x=+-cos(+-pi/10,+-3pi/10,+-pi/2,+-pi/10,+-3pi/10) [by symmetry of cos(x) ie. cos(x)=-cos(pi-x)]
x=+-cos(+-pi/10,+-3pi/10,+pi/2) [but x=/=0 and hence x=/=+-cos(+-pi/2)]
x=+-cos(pi/10,3pi/10) [cos(x)=cos(-x)]
3) 16x^4-20x^2+5=0
=> x^2(16x^2-20+5x^-2)=0
=> 16 (x^2-5/8)^2-5/4=0
=> (x^2-5/8)^2=5/64
=> x^2=5/8+-sqrt(5)/8
=> x=+-(1/2)sqrt((1/2)(5+-sqrt(5))
By product of roots;
=> (+-1)^2[cos(pi/10)cos(3pi/10)]^2=[(1/2)sqrt((1/2)(5+sqrt(5))]^2[(1/2)sqrt((1/2)(5-sqrt(5))]^2
=> [cos(pi/10)cos(3pi/10)]^2=(1/4)^2(0.25(25-5))=(1/16)(0.25(20))=5/16
In part 2): it's 16x^4-20x^2+5 as you need a symmetric quartic to obtain the final part