In year 11, none of this matters because none of your year 11 results will contribute to your HSC results or ATAR. In year 12, however, 50% of your HSC mark (final mark) in a particular subject comes from your internal performance. Internal performance refers to your performance in your school-based assessment tasks/exams, i.e. any assessment item that is not an HSC exam. The value of the marks that you achieve in your school-based assessment tasks/exams (for example, if you score 90% in a hand-in assignment) does not matter in itself, rather its purpose is to determine your rank in a particular subject relative to your cohort.
Essentially, the higher your rank, the higher you can expect your Assessment Mark to be. The Assessment Mark is the technical name of the mark that you receive after you complete your HSC, which contributes 50% of your HSC mark for a particular subject. This leads us to the difference between St Marys Senior High School and Rooty Hill High School. SMSHS, being a more academically focused and higher-ranked school than RHHS, suggests that it will be more difficult to rank high in your respective subjects compared to RHHS. At the same time, because SMSHS is more academically focused, you will not need to rank as high at SMSHS as you would in RHHS in order to achieve the same Assessment Mark.
My personal recommendation would be to attend SMSHS. Not only is it a 250+ rank jump from RHHS, but as someone who went to this school, I believe that you will not be disappointed with the non-academic aspects of the school either.
I hope this helps!