Can you listen to ham radio without a licence?

For many people tuning into radio transmissions is a great way to spend time. But are you allowed to listen to ham radio if you don't have a licence?