Why can't we perform primary storage dedupe on standard disk drives?

It's not easy. With memory technology, the systems can perform deduplication and/or compression a whole lot faster. There are more effective ways of implementing it in a flash environment when compared with a traditional spinning disk environment where there are a lot of synchronization issues to deal with.

If I was making an investment from a vendor side, I wouldn't invest [much] into primary storage dedupe or compression for spinning disks. I would probably only offer those technologies for flash and embed them into the flash controllers as much as possible, so it wouldn't take processing cycles away from the storage controller. I think the return on the investment [for dedupe and compression] is higher for flash than for spinning disk.

That being said, there are a number of implementations that perform data reduction for spinning disks. Some perform it post-process, which is problematic in some cases, but it does increase the capacity. If the data doesn't have a high-reference usage, that's probably very adequate. Also, there are some implementations that perform data reduction inline, before it's actually written. Those are very effective.