Writing 'memoir'

I have come to the conclusion that the 'sort of' memoir that I've been working on for approximately eight years is an unhealthy pursuit. And although I have 400 hundred pages completed I am thinking of shelving it.

To go back a bit, this grand undertaking was what started me writing seriously, the three fantasy  books I've subsequently published, the 'fun' writing to take my mind off the 'more important' project.

The background to this story arrived one day in a box sent from my sister-in-law and brother who had been cleaning out the basement of the farm where my father spent his last years. No one had known about the journal my father kept while held captive, a POW who had been one of the soldiers on Corregidor when it was taken over by the Japanese. I read this missive through twice, crying every night. I had to bring this to the world's attention.

After several false starts, I devised a fictionalized version,  focusing mostly on the difficulties of a marriage between a shy young woman born to wealthy, socialite parents in Washington, D.C. and an ambitious young man who wanted to bring himself out of the blue collar family he'd grown up in. In the interest of family members who might disagree with my version I changed names and dates and many circumstances that I was privy to. And because there was no one left to consult I had to make things up. I've added in letters from my mother to my father and many of his journal entries, but other than that it is pure fiction.

So, you say, this sounds good, why quit? While writing this blog I've felt the urge to continue, the pull of the story I've created but...these are some of the reasons why I think it's a bad idea:
1. I'm not doing justice to the reality of their lives.
2. Family members will be horrified.
3. I would like to divorce the story from reality but I can't get these characters I've created out of my mind. (and they definitely look like the younger version of my parents)
4. It's too glib.
5. I don't fully understand the history of the time.

Have you come up against anything like this in your writing? And if so, how did you solve it?