The first problem with saying 'I just don't have time for it' is its a bare-faced lie! I have to resist the urge to scream 'BUT YOU HAVE TIME TO WATCH EASTENDERS?!' ;)

The second problem is - why not just tell the truth? 'It doesn't really appeal to me, personally.' (talking about neutral subjects here, not requests for favours etc)