Retention isn’t an issue as the choice that they have is to either apply to modify the permission by having the condition overturned or comply or just try and ignore it.
If they ignore it they are in breach of a planning condition which is more serious as they cannot claim ignorance of the obligation. The problem you face is that it is difficult to pursue as the permission probably still has a valid life and they can argue that they will get around to it until the permission life period expires.