Since folks here seem to be in the know, if I were after "parsing as close to the browser as possible", ideally in Java/Scala or Ruby or Python (sorry PHP!) -- any recommendations?
I've done scraping (e.g. w/ BeautifulSoup) but haven't looked to see how true the parses are to what IE/FF/WebKit would produce.
Run headless Firefox+xvfb+Selenium. Selenium has java, ruby, and python clients. This is working really well in production.
You might also have luck with AppleScript/RubyCocoa/MacRuby+Safari. I've tried a couple other options like HTMLUnit (which google uses with GWT), and Mozilla Java Html Parser, which Dapper.net uses. I couldn't get them running, but YMMV.
for Ruby there's Hpricot and Nokogiri. Now I must admit not understanding what "parsing as close to the browser as possible" would mean. These parsers would not be for displaying the HTML, they're not rendering engines like those in browsers, but will help you navigate the DOM of a (X)(HT)ML document programatically. Surely I'm missing your meaning.
What I'm aiming at would mean "given this lump of (malformed) HTML, what DOM would a browser give me?" Maybe Hpricot, BeautifulSoup, et al are already there, but I don't know. :)
I've done scraping (e.g. w/ BeautifulSoup) but haven't looked to see how true the parses are to what IE/FF/WebKit would produce.
(On my list of things to look into: html5lib -- http://code.google.com/p/html5lib/ ... is it any good?)