I say leave the implant alone unless it starts giving you great trouble. If it does, them have it removed. One doesn't have a great need for wisdom teeth. I almost had some of my wisdom teeth removed years ago when they were not emerging properly. There shouldn't be such great worry about teeth shifting after a wisdom tooth is removed.