Safety and efficacy of therapeutics based on RNA interference, e.g., small interfering RNA (siRNA), are dependent on the optimal engineering of the delivery technology, which is used for intracellular delivery of siRNA to the cytosol of target cells. We investigated the hypothesis that commonly used and poorly tolerated cationic lipids might be replaced with more efficacious and safe lipidoids as the lipid component of siRNA-loaded lipid-polymer hybrid nanoparticles (LPNs) for achieving more efficient gene silencing at lower and safer doses. However, formulation design of such a complex formulation is highly challenging due to a strong interplay between several contributing factors. Hence, critical formulation variables, i.e. the lipidoid content and siRNA:lipidoid ratio, were initially identified, followed by a systematic quality-by-design approach to define the optimal operating space (OOS), eventually resulting in the identification of a robust, highly efficacious and safe formulation. A 17-run design of experiment with an I-optimal approach was performed to systematically assess the effect of selected variables on critical quality attributes (CQAs), i.e. physicochemical properties (hydrodynamic size, zeta potential, siRNA encapsulation/loading) and the biological performance (in vitro gene silencing and cell viability). Model fitting of the obtained data to construct predictive models revealed non-linear relationships for all CQAs, which can be readily overlooked in one-factor-at-a-time optimization approaches. The response surface methodology further enabled the identification of an OOS that met the desired quality target product profile. The optimized lipidoid-modified LPNs revealed more than 50-fold higher in vitro gene silencing at well-tolerated doses and approx. a twofold increase in siRNA loading as compared to reference LPNs modified with the commonly used cationic lipid dioleyltrimethylammonium propane (DOTAP). Thus, lipidoid-modified LPNs show highly promising prospects for efficient and safe intracellular delivery of siRNA.