References: Nothing provides you more insight into a vendor's performance than in-depth conversations with peers at other organizations that have been with the vendor where you are now. Ask the vendor for references and follow up with them in an honest conversation.
- Ask them what they like and what they don't like about the vendor.
- Ask them about the vendor's support performance.
- Ask them about the process and effort around upgrades or maintenance.
- Ask them about the learning curve - maybe even arrange to talk to a few users, maybe even facilitate a user to user conversation.
- Ask them for tips on how to most effectively "manage" this vendor.