Custom farming is being hired by someone else, usually another farmer, to do a certain job on the farm with your own equipment. It can be custom tillage, planting, fertilizer application, spraying, harvest, etc. A farmer will usually hire another farmer to do "custom" work if they are behind on schedule or have other delays that keep them from getting things done when they did to be done. A contract can be made for a set price, or it can be done in trade. You help me out, I'll help you out kind of thing.
It's different from renting farm ground in that the custom farmer usually doesn't do every operation required to grow the crop, and they don't usually make decisions as far as what to plant, what variety, what herbicides to use, etc. The farmer doing the hiring is normally still in charge of the big decisions. Whereas when renting ground to farm, the renter makes all the farming decisions himself, and pays the landowner a flat rate per acre of rent regardless of his costs and profits.