In most cases vaccination relies on the induction of immune memory responses, which are recalled during the early stages of infection. These recall responses are typically faster and stronger and have a higher affinity for the antigen, compared to primary immune responses. For many pathogens these recall immune responses represent a real challenge to their survival. As such they exert significant evolutionary pressures on pathogens resulting in survival advantages for those pathogens able to escape recall immune responses. One way of achieving this is to redirect the immune memory response away from protective immunity towards a type of immune response that is less detrimental to the pathogen. Thus some pathogens have developed immuno-modulatory properties in an attempt to circumvent immune destruction. In turn, one can expect that the immune system will also develop ways to resist immuno-modulation by pathogens. We recently introduced the concept of immune memory resilience, which is defined as the ability of an immune memory response to withstand manipulation by pathogens. In this mini-review we will examine the literature in relation to how pathogens manipulate the immune response with an emphasis on immune memory responses. We will also discuss the implication of the concept of immune memory resilience for the development of protective vaccines against immuno-modulatory pathogens.