Horses get spooked easily. It doesn't matter if you are on one, or it is just standing there idly watching the day go by.
Suddenly, something will happen and - whoosh! - off it goes.
Going over fences on horses is particularly dangerous if they're not specially trained and controlled. Even then, it's dangerous. Just like those scenes in "Gone With the Wind" (there were two horse accidents, remember? - they were very realistic and almost common back in the day.
Imagine being on this horse when it suddenly decided to take off. Over the fence it goes - and whammo. You're still there. Well, you're there, under the horse now.
And horses are heavy.
Life is full of risks. That's why insurance companies have so much money.