Cancel Charge Card

I know that if I cancel a credit card it can lower my credit score because my overall credit limit goes down.

What if I cancel an AMEX charge card? A promotion is coming to an end and I will have to start paying the annual fee. I would rather change to a different card that has no fee, but AMEX won't let me. (It can only be another charge card with a lower fee.)

Will canceling it hurt my credit score? And if it will, how long will that last? And is it worth $55 in annual fees to avoid?

Thanks!

Unless you have something big in the works (like a house purchase, etc) it never makes sense to pay good money to keep your score up a few points.