Why spoken English is difficult to understand

• People use different words and phrases in spoken English than they do in writing. They use slang. They say "um", "hmm", "ah", "uh", etc. They skip the word "that" when using relative clauses,... 
• Words are pronounced differently when you say them individually than when you say them together. This is called "connected speech" (pa amb oli). ;)
• People talk about a lot of different topics. Some topics are very specialized and uncommon. Sometimes they talk about people you don't know, or make a joke about a movie that you haven't seen. So it's important to realize that you don't have to understand everything.

The best way to improve your English listening is to listen. A lot. There's no way around it; you have to spend hours and hours listening to people speaking English. Listen to things that interest you. 

Other listening tips:

• Interacting is better than passive listening. In other words, it's better to talk with someone than just to listen to a TV show, radio program, or podcast. Being in a conversation forces you to listen more carefully
• The next thing to watch out for is to make sure that you listen to a variety of different kinds of speech. I know some people who listen to radio news shows every day but don't feel like they're improving. That's because news English is one specialized form of speaking. Only news broadcasters speak that way. So you can't expect to understand everyday conversation if you only listen to the news.
• Captions can help if they're in English. If you watch a movie, video, or TV show with English subtitles, it can help you match the written words to their spoken pronunciation. But subtitles in your native language can be dangerous. They do help you to understand what's going on, but they also keep your brain locked into "native language mode". Try to use native-language subtitles as little as possible.

Extracted and adapted from http://www.phrasemix.com/blog/how-to-improve-your-english-listening-comprehension