If you're like me, you have trouble deciding which of these is the right word to use in a particular situaion. I will offer some thoughts on this issue.
Memoir comes to us from "memoire", the French word for memory. Thus, memoir means one's memory of a particular event. The writer is not claiming to be presenting historical facts, but rather is relating the event as he or she remembers it at the time of writing about it.
"Memoirs", in the plural, would therefore make more sense when referring to an entire book, which presumably consists of a collection of many such remembrances.
It should be noted that memoirs is to be distinguished from autobiography in that an autobiography undertakes to tell about one's entire life, not just a particular group of experiences from that life, as one's memoirs would be. There is also the understanding that in an autobiography the writer has done research and fact-checking to ensure accuracy as much as humanly possible. With one's memoirs the writer is not expected to engage in this sort of research.