Recently, a guideline has been published by the European Medicines Agency (EMA) on setting safe limits, permitted daily exposures (PDE) [also called acceptable daily exposures (ADE)], for medicines manufactured in multi-product facilities. The ADE provides a safe exposure limit for inadvertent exposure of a drug due to cross-contamination in manufacturing. The ADE determination encompasses a standard risk assessment, requiring an understanding of the toxicological and pharmacological effects, the mechanism of action, drug compound class, and the dose-response as well as the pharmacokinetic properties of the compound. While the ADE concept has broad application in pharmaceutical safety there are also nuances and specific challenges associated with some toxicological endpoints or drug product categories. In this manuscript we discuss considerations for setting ADEs when the following specific adverse health endpoints may constitute the critical effect: genotoxicity, developmental and reproductive toxicity (DART), and immune system modulation (immunostimulation or immunosuppression), and for specific drug classes, including antibody drug conjugates (ADCs), emerging medicinal therapeutic compounds, and compounds with limited datasets. These are challenging toxicological scenarios that require a careful evaluation of all of the available information in order to establish a health-based safe level.