Spring Garden by Tomoka Shibasaki (author), Polly Barton (translator) 

Taro is divorced, unhappy in his job, and living in a half-empty building that is about to be torn down. One summer morning, he sees a fellow resident climbing over the wall to the next-door house. She says she is called Nishi, and invites herself inside.

It emerges that Nishi's fascination with this pale blue house began in her student days twenty years before, and came from a book of photos called "Spring Garden" from decades earlier. As the summer draws to a close, Nishi, Taro and the new family that has moved into the old house come together and drift apart, leaving the reader with a sense of their whole life in just a few vivid snapshots.