Doubt centered on the mathematical unreality behind his investment performance. The trading system he claimed he used could not produce the data he reported. Did the broker-dealer arm of his investment business furnish Madoff with inside information? What was so earth-shattering about his trading technique that he had to keep it a complete secret? That uncertainty was enough to dissuade the Wall Street elite from throwing in with Madoff.
What this touches on is a piece of delicious, if tragic, irony. While most Madoff investors were likely as innocent as an ignorant investor can be, others were surely sophisticated enough to recognize the "mathematical unreality behind his investment performance." Some, too smart for their own good, suspected that Madoff was leveraging his broker dealer business to trade in inside information, and, therefore invested in him to get in on the scam, unsuspecting that they were not eating, but being eaten.