Yep acceleration increases at an increasing rate. According to F=ma, a=F/m. As the rocket travels upwards it shoots exhaust gases out which reduces its mass (m). When m decreases then a (=F/m) will increase because there is and inverse relationship.
Furthermore as it travels higher up the earths gravitational pull weakens. Therefore it is progressively easier to accelerate to reach higher speeds.