Anyone that says that is most likely confusing the man with the office. There is a specific man in the office at a particular time, and that man can make mistakes. That man is a sinner like any of us. (Peter, if you recall, made some huge mistakes, in fact.) The Pope does not speak infallibly when he gives a personal opinion, cheers on a sports team, or orders a pizza.
(Infallibly ordering pizza would be a great gift, though. "The pizza will have sausage and mushrooms." Maybe one day I'll open Papal Pizza. Our slogan will be "Whatever you order on earth shall be delivered from heaven.")
The office of Pope is infallible when teaching. The office is protected by Christ, by the Holy Spirit that came upon the apostles to guide their teaching and their tongues (see Acts 1:8 and Acts 2). When someone in the office performs the duties of the office - the duties set down by Christ in His mandates to Peter and to all the apostles - that performance is infallible.
While many (I think and hope) are just confusing the man with the office, some may be questioning the authority of the office. There are excellent analyses of papal authority on the Internet. Perhaps I'll contribute to them one day, but for now, I would like to imagine that those dissenters are correct.
What happens if the office of Pope is not infallible when teaching? Simply put, the Church is false.
Any clarifications made by any pope are only a man's opinion and, therefore, open to interpretation. (You may accept that the apostles had a mandate but that it ended with the death of the last apostle; in that case, any clarifications made by any pope after the last apostle died would be just a man's opinion. I'll stick to the more broad version.) Either way, we lose teaching on:
- the perpetual virginity of Mary
- the Immaculate Conception
- the canonization of saints
- the Trinity
Oh, yes. No Papal Authority could lead to no Trinity. Show me where in the Bible the word "trinity" appears? It doesn't. It appears in later doctrinal documents - the infallible teaching of bishops and the Pope. If you don't believe in that infallible teaching, you cannot say with certainty that there is a trinity. It is only your opinion, and I can have my opinion that there isn't one.
Without infallible authority, the documents issued to counter incorrect schools of thought - Pelagianism, Manachianism, etc. - are simply opinions. I can still be Catholic yet believe that Jesus was just a vessel for God, a man possessed by a spirit that wasn't his. I can believe that Jesus was only God in a human guise - a very impressive hologram but not a real man. I can believe these things because we're just comparing opinions.
Actually, I'm wrong - I can't be "Catholic". The word Catholic isn't used in the Bible; the term was coined by the same bishop that coined the term "Christian", but only the word "Christian"shows up in the Bible. The term was only written by a fallible man. Oops, I guess we have to stop being Catholic.
We put a lot of store in the teaching of the Church. While the Bible is a Catholic book and the faith will never contrdict the Bible (or vice versa), there is still a richness that comes from apostolic tradition that absolutely hinges on infallibility. Paul tells us to hold to Scripture and Tradition. If tradition - the oral teaching of the Church, passed down through generations with the Pope at the head - can be wrong, then the faith we have now could be utterly wrong.
To be Catholic, is to assume infallibility. I cannot be escaped without abandoning what makes us Catholic.