The filter which gives you the soft pictures is likely incompatible with that specific lens on that camera. I shoot Olympus and OM System cameras and lenses, and had a problem with soft photos from a 100-400mm lens which turned out to be caused by the very popular brand name UV filter of a type I used on many of my lenses without problem. I changed to my other preferred filter brand and the problem went away. I've read other reports of this problem, so my conclusion is to always test a new lens and filter combination.

That's not what any polarizing filter does. It doesn't block anything based on wavelength. It blocks some light of all wavelengths based on its polarization.