The fact that you have a low ranking amongst a cohort of "really smart" people probably means that you aint too bad at the subject. You need to put things into perspective. If you're at a selective school or have a really talented year then you have to think in terms of the state rather than your school. If you're decent a math then you should benefit from 4U given the scaling and the double up of units it gives to 3U.
When you say 80%, was that your mark or the weighting of the exam? If it was your mark then that's a really good one to be getting. IN my opinion 4U is one of, if not the most interesting HSC courses since it's one of the few which requires thinking as aposed to rote learning. In the end it's up to you, but don't just drop out because you have misconceptions from being in a smart year. Oh, and if you enjoy it... then keep it.