Under normal conditions the impedance itself isn't an issue.
Hang on... This is going to get a little "technical".
The main issue with impedance is
related to frequency response.
The impedance of a headphone (or speaker) isn't constant over the frequency range. Headphones are tested with a low-impedance (constant voltage) source. That is, the voltage out of the headphone amplifier doesn't change when the load impedance changes.
A good headphone amplifier will behave the same way and impedance doesn't have any effect and the frequency response will be "as specified".
If the headphone amplifier source impedance isn't low relative to the headphone impedance you have a
voltage divider where the "top" resistor is the source-resistance the headphone amp and the "bottom" resistor is the headphone impedance.
If these were both pure resistors the only consequence would be a signal loss. If the resistors are equal you'd get a 6dB drop. (Higher headphone impedance or lower amplifier impedance means less voltage drop.)
If the headphone amp has an output capacitor, the voltage divider becomes a high-pass filter.
If the capacitor value is too low, or if the headphone impedance is too low, you'll get a loss of bass.
As the headphone impedance varies over the frequency range, the output varies (if the source impedance is not low relative to the headphone impedance).
An impedance-rise in the mid-bass (which is somewhat common) will create a boost in the mid-bass frequency response.
A higher impedance headphone (and/or a low impedance headphone amplifier) minimizes both of these effects.
Unfortunately, amplifier output-impedance is rarely published. They usually just publish the recommended load impedance. It's the same with power amplifiers... The source impedance of a power amplifier (designed to drive 4 or 8 Ohm loads) is usually far-less than 1 Ohm, but that's not published in the specs. (Sometimes they publish the damping factor which is the ratio of the load impedance to the source impedance.)
Impedance is also related to power (milliwatts in the case of headphones). With a given voltage, half the impedance gives you double the power. If you have a low-voltage headphone amplifier (like what's built-into a cell phone) you may not get enough volume. The volume from laptops or other devices may be limited too.
But although volume/loudness is related to power, some headphones are more efficient than others. Lower impedance headphones
tend to be louder, but that's not always the case.