Yes. I’ve noticed this too. Leaving my house, and going downhill and westbound (into the prevailing surface winds), the wind noise is more pronounced. Driving north to LAX in a crosswind, and then home again southeast bound, the wind noise is less.
Maybe we’ll all have to learn to just live...