Tiller is the distance from the string to the fade-outs. It is measured at the top fade-out and the bottom fade-out, and the tiller is normally expressed as the difference between the two. For example, if you have a top distance of 6" and a bottom distance of 5 3/4", that means there is a positive tiller of 1/4".
The fade-out is where the riser ends and the working part of the limb begins. It may be the actual end of the riser, in the case of a take-down bow, or where the laminations that make the riser thicker than the limbs come to an end in a one-piece bow.
When a bowyer is making a bow, he tries to get the limbs to bend evenly, with a certain type of draw (split fingers or three under). The tiller of a bow results from his efforts to do that. One limb usually ends up with a higher tiller than the other, because the working length of one limb is shorter than the other.
After all that explanation, you probably don't need to know what "tiller" is to set up a trad bow, since tiller is not adjustible in most cases. You do need to know what brace height is, because you should set it within the bowyer's specifications.