One of the most basic concerns of phonology is determining what phonemes constitute the phonological inventory of a given language, i.e., which sounds are used contrastively. Sounds are used contrastively if switching one pronunciation for another could result in a change in meaning (it may not be the case that a new word thus formed actually exists, but I think the point still holds for pairs like brick~blick). I can pronounce the word "atom" with an alveolar flap or with an alveolar stop, but the choice does not produce a difference in meaning. At worst I sound British if I use an alveolar stop rather than a flap. One of the ways phonologists look at contrastive versus non-contrastive sounds is the distribution of sounds. Contrastive sounds will typically have the same distribution. In English, for instance, /t/ and /d/ are contrastive, and we can find them in the same environments. Each one can occur in the onset or coda of a syllable, including before sonorants in a complex onset or after sonorants in a complex coda. As mentioned before, alveolar flaps and stops are non-contrastive in English, and are viewed as variant pronunciations. Another example is aspirated and unaspirated voiceless stops -- we perceive the /p/ in "pit" and "spit" to be somehow the same, even though one is aspirated and one is not.
Problems arise when phonemes are in complementary distribution but don't in any real sense seem to be allophones. An example from English is the case of /h/ and /ŋ/. /h/ occurs only in onset position, and /ŋ/ only in coda position. We might have to do a little bit of hand-waving, but as long as we don't insist on following our rubrics with machine-like strictness, we can say something about the fact that we don't have any external evidence that these two sounds come from the same phoneme, e.g., alternations that we do find in "atom" (with a flap) versus "atomic" (with a stop). We also have evidence from historical English sources (including modern orthography) that explains why /ŋ/ only shows up in coda position -- it arose from place assimilation when /n/ appeared before /g/, which is why we spell /ŋ/ as ng, as in "sing". Another difficulty appears when sounds only contrast in some environments. This is called neutralization. For instance, in Blackfoot, /t/ and /ts/ contrast in most environments. Both can appear in onsets and codas. However, only /ts/ appears before /i/. Thus we can't simply say that /t/ and /ts/ are or aren't contrastive; we have to specify the specific environments in which they are contrastive.