With the development of advanced imaging technologies, the demand for high quality images in various fields has increased. However, image degradation due to noise, data loss, and other factors persistently hinder image quality. Image restoration (IR) is a critical task in computer vision, aiming to recover original images from degraded observations. Traditional non-learning prior based methods offer flexibility and interpretability but often yield sub-optimal results due to limited representational capacity. In contrast, learning prior based counterparts produce superior performance but suffer from over-fitting and poor generalization to unseen degradations. In this paper, we introduce a novel dual-scale prior (DSP) model that integrates the flexibility strength of non-learning prior with the representation power of learning-based prior. Specifically, the DSP model employs a group-scale physical prior, leveraging non-local self-similarity (NSS) for jointly sparse and low-rank approximation. And an image-scale bias-free deep denoising prior for capturing external characteristics. These dual-scale priors complement each other by effectively preserving edges and removing noise, demonstrating robustness across various types of degradation. We then present DSPIR, an effective IR method by incorporating DSP into existing maximum a posteriori (MAP) principle. DSPIR is solved by alternating minimization and alternating direction method of multipliers. Extensive evaluations on both synthetic and real data demonstrate that DSPIR achieves better performance in image denoising and inpainting compared to state-of-the-art methods.