We propose a novel learning-based approach to detect an imperceptible hand-held needle in ultrasound images using the natural tremor motion. The minute tremor induced on the needle however is also transferred to the tissue in contact with the needle, making the accurate needle detection a challenging task. The proposed learning-based framework is based on temporal analysis of the phase variations of pixels to classify them according to the motion characteristics. In addition to the classification, we also obtain a probability map of the segmented pixels by cross-validation. A Hough transform is then used on the probability map to localize the needle using the segmented needle and posterior probability estimate. The two-step probability-weighted localization on the segmented needle in a learning framework is the key innovation which results in localization improvement and adaptability to specific clinical applications. The method was tested in vivo for a standard 17 gauge needle inserted at 50–80° insertion angles and 40–60 mm depths. The results showed an average accuracy of (2.12°, 1.69 mm) and Symbol for localization and classification, respectively.