1) Require a submitted portfolio with references. Five sites with traffic levels doing more than 40k monthly uniques you can verify with independent services like Compete or Quantcast with a good sprinkling of both technical and business contacts.
2) Always have a MSA, SoW, and SLA. The first is the Master Services Agreement (MSA) and contains the general terms of the contract with appropriate indemnification, jurisdiction, and penalties for breach. Second is the Statement of Work (SoW) which serves as the gospel for the project; the full list of all deliverables, their delivery dates, and the delivery acceptance terms. Finally, the Service Level Agreement (SLA) contains the specific terms governing support. These three documents serve as the context for a successful relationship and ensure all the sticking points in a web dev project are clearly defined before it is engaged.
3) The right partner is more important than your tech organization’s favorite scripting language. Java, PHP, Python, and Flash all have standard best practices surrounding their implementation on the modern web and have examples of very high scale. More important than the language is the ability to deliver on time and provide consistent support after launch. A vendor insisting on Ruby on Rails with a 24/7 support line is infinitely more valuable than a vendor with the most elegant PHP yet written who is chronically unavailable on weekend / holiday outages.
We’ll be adding this content to the RFP when we add it to the resources page in a few weeks after all suggestions are in.