I use vim almost pathologically, but I gave up on set -o vi. Since there's no visual indicator of which mode you're in, it's easy to make confusing mistakes and not the right feedback. It just wasn't worth it.
I use vi mode in my shell all the time, and love it. It only took a day or so to get used to it. But then again, I am a die-hard vim user.
What I'd really like, though, would be a vim mode for the shell, not just a vi mode. Then I could use some of vim's more advanced features, which would make the mode even more useful.
I can only speak for myself. Give it a day or two, and it will quickly become as transparent as your editing sessions. Once you cut over, you can forget about chording and arrow keys. Just one key: <Esc>.