Based on what you write, you need not be concerned.
Examining a
bedikah cloth with care is proper; it is also proper to disregard specks on a cloth that do not appear to be blood. Especially given your rav's response to you, it would seem that his experience with you has been that these specks are often ones you can
assess yourself.
In the future, you may want to check the cloth for pre-existing specks before doing a
bedikah. This may help reduce the number of specks you find on
bedikot.