Game of Thrones has to be the most difficult show to follow that I have ever watched. There must be at least 2 different storylines going on right now. You really need to pay attention, and remember things, because they pop back up later on.
A friend of mine always hears me talking about it when it is on, and tried to start watching it a few weeks go. He gave up.