Dirt on your filter is a good thing, up to a point. After all, this is why you put the filter in your system, i.e. to collect dirt and other contaminants. However, when the dirt accumulation on the filter gets too extreme, the air coming from your register will definitely affect your health.

Mold, mildew, bacteria and other contaminants will grow rapidly on the dirt, on the coils, and on the drain pan. This is because the reduced airflow will not be able to adequately remove humidity. These contaminants love a humid environment.

As the contaminants and pollutants flow into the living space loaded with all these microbes they can cause havoc with your respiratory system leading to infections, and other respiratory illnesses. Some studies have shown that bad quality air can even cause cancer.

Finally, a clogged air filter with reduced airflow will cause your cooling coils to freeze reducing the airflow even more. This condition is very bad for the compressor's health and may permanently damage the compressor, the heart of the system.