It depends on the kind of watch you own. For quartz watches, yes, with a few exceptions. For mechanical watches, no.
Quartz movement watches are usually safe. If you own a quartz watch where neither the date window nor the hands are moved directly by the crown, then turning the time backwards will not harm the watch. That is usually the case with quartz watches with a digital date display, or in watches where the hands are moved through electronic command. However, rolling the time backwards can harm both quartz or mechanical watches with dates or other complicated functions that are handled by mechanical parts, especially near midnight.
The reason moving the time in reverse is harmful on mechanical watches is because the mechanisms are only designed to move forward, so reversing their movement can cause stress on the parts. Also, once you’ve owned the watch for awhile, reversing the movement rubs against the pattern of wear and can cause additional, unnecessary wear.
That being said, this doesn’t apply to most modern watches. They are generally designed to handle this without problems or wear, but check with your manufacturer first. Here’s a quick way to tell: if your mechanical watch’s second hand does NOT stop when you put the watch in time setting mode, then you should definitely not turn the hands backwards.