Touching is prohibited on Yom Kippur (Shulchan Aruch Orach Chaim 615:1) and on Tisha B’Av (Mishnah Berurah 37), just as if the wife were niddah. The couple is not allowed to share the same bed even if they do not touch (Orach Chaim 554:1, 8).
On Tisha B’av during the day, there is leniency with the harchakot (Mishnah Berurah 37) and you could, for example, tap your husband’s shoulder to get his attention, but should not engage in affectionate touch such as holding hands. The reason for the difference between day and night on Tisha B’Av is not clearly stated in the sources. It may be because night is more conducive to marital relations and thus one is more likely to need reminders that relations are prohibited.
There is a debate over whether the other harchakot apply as well, particularly on Yom Kippur, and one should ask one’s rabbi how to proceed. Yom Kippur is more stringent because it is a Torah ordained fast day, while Tisha B’Av was instituted by rabbinic decree.