I know what you mean. When it's cold and your hands get cold, even a small snap to the fingers hurt like hell. I think it has something to do with the nerves, but the cold won't get deep enough to affect the bones. And even if it did, I don't think they would break more easily.
Don't take the gloves off if you get in a fight. They protect your hands and keep them warm. Make sure your gloves are warm enough so the hands won't get cold and tender.
premier