The Problem of Infinity

Why do people have a problem with the idea of infinity? Is it because it's counter-intuitive? There is nothing mystical about the idea of infinity. No need to equate infinity with God. No need to try and prove mathematically that infinity is possible. It's really a very simple concept. Jonathan Swift wrote this verse in On Poetry: A Rhapsody (1733):

"So, naturalists observe, a flea
Hath smaller fleas on him that prey,
And these hath smaller fleas to bite 'em,
And so proceed ad infinitum."