I don't carry a fob either (card in wallet as backup), and I often (more and more) do taker my phone out and hold it near the window. That works, most of the time, but not always and there is sometime a significant delay still. I find it mildly inconvenient, but chalk it up to an early design decision and it is what it is. Does not detract from my enjoyment of the car, just wish it was more consistent or just a bit better.
On a side note, sometimes I wave my iPhone near the door and it gets too close to the NFC area and my phone switches to Apple Pay

Maybe I should pay the car some money and it will work better ;-)