In my opinion Floyd needs this fight for his legacy FAR more than Manny.
I don't know about 'FAR' more.
Being an undefeated champion says something. Floyd is ducking Manny, but unlike what many say, Floyd has beaten some VERY good fighters.
He beat Oscar De La Hoya when Oscar was still a champion.
He beat a heralded and previous undefeated Ricky Hatton.
He beat a revitalized Shane Mosely.
Unfortunately for Floyd, after he demolished those fighters people suddenly said: "Hey those guys were old/bums anyway".
Sadly being undefeated means so much to Floyd that he's ducking Manny. I say 'sadly' because I believe he would win and I KNOW it would be an amazing fight. It would be boxing's first superfight in years. Also, as sjon pointed out, Floyd is giving off the aura of being afraid. NOT a good look in combat sports (boxing, MMA).