It is possible that
some Xiongnu disguised as Han and move to the south with the Han. Many Han aristocrats also had hundreds to thousands of Xiongnu servants and soldiers.
Hakka language was probably spoken by many Han Chinese in central China. A Hun language would not have survived under hostile political conditions.
Even if some Hakkas were Xiongnu, their language, behavior must be totally integrated with Han to survive this era. Culturally speaking, if certain Hakka were Xiongnu decendents, they should be totally indistinguishable from the Hans.
It is not impossible, but unlikely that 400 years can upgrade the identity and class of Xiongnu to equal the Hans.
Source : Asiawind Hakka History