Have had no speed adjustment issues. That said, I grip the toggle in a very specific way to ensure that I can twist it up or down rather than accidentally press it in. The angle at which my hand meets the switch means that attempts to push the switch 'up' can easily result in it being pressed 'in' instead.
HA/AC recognizes changes in speed limits and offers a function where the target speed can be reset to the posted speed limit (even incorrectly interpreted versions of the speed limit, such as 25mph limits in 40mph zones that aren't active outside of school hours) by pressing the toggle in. If you're seeing the target speed making a large change, that might be what's happening. To rule that out, try some tests where you grip the toggle with your index and thumb (I put the thumb underneath the part that sticks out towards the drive, and the index finger above it...so I'm not actually using the 'ledge' on the toggle at all, ensuring the only force I'm exerting is a twisting moment).
Setting the target speed is the only part I truly dislike in the interface...I'm going through odd gymnastics and holding the toggle in non-standard ways because the resistance and feel is otherwise all wrong. It's also a tad laggy imo.