Unsetting the color should happen in the lifecycle of the fragment that sets it in the first place (onStop most likely)