Because a diaphragm is inserted into the vaginal canal, any stain found on it is treated like a stain on a bedikah and we do NOT apply the leniencies of stains found externally. Therefore, regardless of size, if the color is problematic, you would be a niddah. You may clarify whether the brown was a niddah color by contacting a halachic authority.
For the future, it is a good idea not to look at the diaphragm until you have finished cleaning it and rinsing out the sink.