The science handbook for the year you started will provide you with the information you need about requirements to attain the degree.
The 2013 handbook for science is available here:
http://sydney.edu.au/handbooks/science/undergraduate/b_science/b_science.shtml
"Bachelor of Science
(a) To qualify for the award of the Bachelor of Science, a student must successfully complete 144 credit points, comprising:
(i) a major from Science subject areas listed in Table 1; and
(ii) a minimum of 12 credit points from the Science subject areas of Mathematics and Statistics; and
(iii) a
minimum of 24 credit points of Junior units of study from at
least two Science subject areas other than Mathematics and Statistics;
and ensure:
(i) a minimum of 96 credit points from Science subject areas; and
(ii) no more than 60 credit points from junior units of study; and
(iii) no more than 48 credit points from units of study not listed in Table 1, with the approval of the relevant faculty.
It is (a)(iii) that you need to concern yourself with. As long as you get junior units (ie. 1000 level) from two science subject areas and Maths/Statistics then you are fine, and you can leave the physics. Note however although you may not need to repeat the physics unit, you will need to pass another unit in its place to get the required total number of units before you finish your degree.
I would also check with your faculty to be sure (I haven't studied science before, but the handbook usually has all the information - as long as you can understand what it is getting at).
