There can't be any such definitive guide. I've written my own robot that
works the way I want. Nobody else but me can tell you how it reads HTML. Now
multiply that by all the people out there writing robots.
As for the major search engines, you'll have better luck. Many of the big
guns document what their robots look for, and give ways to provide keywords.
However, they differ on how they accomplish this, and you'll need to visit
their sites to find up-to-date information.
Good luck,
Darrin