If it is necessary for retirement or the kids' college, it may perhaps make additional sense to invest—a payoff is extra particular down the road, even if it doesn't amount to a sexy six-figure check.