Your code is not correct because you don’t synchronize both sides of the bidirectional association. Check out this article for more details.
The same is true for adding and removing the entity. The remove operation will work just fine if you use LAZY fetching strategy for the many-to-one association. Check out this article for more details.