Hints: Just write b – a as vec(AB) for simplicity, recall that the dot product is linear (can expand it), and d•d = ||d||2.
More generally though, for any vectors u, v† in Rn, it is true that u – projv u is orthogonal to v. This is essentially how the vector projection is defined (i.e. note the...