Writer's Revenge? Real People in Your Fiction

I've always used real people whom I know to create my characters. But I don't go so far as to copy them entirely. I just use real people as a sort of base and then the fiction takes over and they become their own character. But in some instances, there have been a person or persons who've crossed me or just went out of his or her way to make my life a living hell. It's possible more of their real, f'd up persona makes it into my novel than not. In a way, you know who you are, but are just too proud to admit it.