I've had the same problem with horses. There is a guy in our neighbourhood with zero respect for us with some horses badly fed.His horses wanted to be fed and without anything to eat came to our land in order to eat something. We tried to make him understand the problem but he didn't do anything. After a little my grampa, my dad and I fenced our plot and the problem was resolved. This fence is good both for horses and for wild boars (the most important one in my opinion).
Anyways, fences are expensive, and if your problem is deers I don't think you should build a fence, the solution of hot pepper spray is good the same and less expensive.