For an effective buffer, you need a combination of species like HA and A- that are present in equilibrium. The buffer is most effective when the two species have equal concentrations. You thus need both to be weak, and the effective range of a buffer will be around the pH at which the two species have equal concentration.
For example, an acetic acid / sodium acetate buffer solution will be most effective at pH of 4.76, whereas an ammonium chloride / ammonia buffer will be most effective at a pH of 9.24. (Both of those are from memory).
If I take (as an example) a buffer made of sodium dihydrogenphosphate (NaH2PO4) and sodium hydrogenphosphate (Na2HPO4), we need significant quantities of the two active species, H2PO4- and HPO42- so that most of any added acid or base is consumed, minimising the change in pH. For example, if acid is added then it is (mostly) consumed by the reaction
HPO42- + H3O+ ----> H2PO4- + H2O
And, if base is added, it is (mostly) consumed by the reaction
H2PO4- + OH- ----> HPO42- + H2O
In each case, the species that would lead to a change in pH is mostly consumed by a process which converts one component of the buffer into the other.
You can also rationalise this by applying Le Chatelier's Principle to the equlibrium between the two buffer species, that is, by looking at
H2PO4- + H2O <----> HPO42- + H3O+
Added acid cases a shift left and so most of the added acid is consumed. Added base reacts with hydronium and then causes the system to shift to the right, replacing most of the lost hydronium and so again the change in concentration of hydronium (and thus pH) is minimised.