To steal a horse, simply mount it and start riding. Stealing horses is considered to be a crime.
The only two exceptions to this are Shadowmere and Frost. All horses have the same speed, stamina, and health, varying only in color. Horses can be purchased from stables, located outside each major city.