Off-white is considered white in this respect. Tan and beige are considered colored. You are correct that it can be difficult to distinguish between a very pale beige and off-white. When white garments are required (such as during the seven clean days), you should make sure to wear clearly white or off-white underwear. Conversely, if you find a stain on a pale beige and are unsure whether the garment is considered white or colored, you can bring it to a halachic authority for clarification.