Yeah your not screwed at all. You'll just be new to some of the concepts, but in terms of solving problems, those who have done just HSC Physics will have no advantage over you
I would say like if you are good at math (like 3 unit probably preferable but 2 unit is still sort of okay) and haven’t done physics you might need to do some extra work but on the most part you should be okay. But if you’re not that good at math and haven’t done physics you might struggle a bit because it’s a lot easier if you have a good grasp of the math involved