Your DM is completely wrong and I suspect just hoping you aren't aware of your entitlements for the sake of making it easier for him/herself.
Correct. In fact, the DM should be requesting for volunteers for the day - all work on a public holiday as per the agreement is voluntary. If you don't...