Given that none of your year 11 results will contribute towards your HSC results or ATAR, having a casual job while you are in year 11 should not be an issue. Of course, you should consider the number of hours that you will be working each week and ensure that you do not work for too long, i.e. a number of hours that would negatively impact your ability to study and ultimately your performance and results. While your year 11 results will not count towards your HSC results or ATAR, there are university early entry schemes that consider your year 11 results. If you can receive an offer to your chosen degree through such a scheme, then this would be relevant to your situation. Otherwise, you can safely disregard this aspect.
On the other hand, I would personally suggest avoiding casual work in year 12 as this is the most important year of school, where all of your results will contribute towards your HSC results and subsequently your ATAR. If you are not in need to have a job (such as for personal/financial reasons), then it is a good idea to focus all your efforts on ensuring that you are up-to-date with your studies so that you are able to maximise your results and hopefully attain your desired ATAR. After you secure your offer for your chosen degree, you will have the opportunity to work again during your time at university, where you will benefit from the increased flexibility compared to high school.
I hope this helps!