I've noticed somethig similar with my fob. I have passive unlock turned off.
Sometimes pushing front or rear driver side handles works, other times it does not. I can see the yellow light blink when its going to work. When the driver side fails, I have better luck with the passenger side...