yes as Kurosaki said, because of the long chain it is very hard for water to really get in between the polymers and dissolve them. Look at the relative sizes of water and polymers. Most polymers are typically what, 3000-5000 to 10000 monomers? so its yeah.
Also, when coming down to specific examples, e.g. hydrocarbon monomers (e.g. ethylene), the C-H bonds are pretty damn non-polar so yeah, the EN different is about 0.35 from memory, and for it to have a polarity its usually >0.5. Also yeah, polyvinyl chloride is insoluble, but if you think about it most of it is made of hydrocarbons which means it'll probably be insoluble. And for things like cellulose and starch, even tho there is hydroxyl (-OH) groups, these tend to form interpolymer and intrapolymer bonds, as opposed to the water, (i.e. bonds with it self on the cellulose chain, and bonds with the next polymer chains).