If I were taking a grammar test, I'm sure I would mess this up. But stylistically, I've seen it seen it used both ways, depending on the writer. Some off set "of course" with commas: He was, of course, going to die. Some don't. As for whether or not a comma should follow the phrase "of course," it really depends how I want the sentence to be read. The emphasis. The pause.
Of course he would think that. Is read without any breaks; it conveys a lighter emotion, almost amused or slightly annoyed, than if we changed it to...
Of course, [pause] he would think that. Now it reads with a more serious tone. There's more gravity. So depending on the context of the sentence, add or remove the comma.
