I don't know all the circumstances or details on your situation, but in general it sounds like it's best to just walk away from it.
A challenge of your skills is one thing, but when hostiliies are present, now your talking something that is more of a selfdefence situation "Disguised" as a challenge. If you do decide to do it, make sure it's in a ring, at a sanctioned event under the rules put forth by the santioning body. MAKE SURE you don't stray from those rules yourself. Let him do so and get evicted form the ring.