Supercomputing Is a Religion But Clouds Are Mundane

Supercomputer is a specialized device. It can do a lot of math, with precision no other platform can do, but it's about all it can have an edge  for.  Have you ever used one as a webserver? Or at least some sort of middle tier realtime application server? There's no practical benefit in doing it that way. Value/cost ratio is not so good. No reason to use demigod's power for your backyard gardening.

By supercomputers I ofcourse only mean hardware-based NUMA architectures; commodity hardware based  simulations with LAN links (such as Beowulf) IMO are really too slow to be seriously considered NUMA type supercomputers.

Clouds are, on the other hand, are as mundane as a toothbrush. Clouds are as versatile as a minivan. And, you can scale them at a fraction of a cost to have as many cpus as supercomputers do. In that sense, paraphrasing the old saying "today's supercomputer is tomorrow's computer", today's supercomputer is today's cloud. The only problem is there's virtually no shared memory space for those CPUs, and so it takes forever to move data from one CPU to another (compared to NUMA). So you can only harness the power of those cpus if you parallelize work in independent splits big enough so that the cloud could be loaded for long enough before the need for sharing split computations arises again. But wouldn't it be cool if that toothbrush could solve as big problems as its supercomputer counterpart with the same amount of cores might? Wouldn't it be nice for a moment to feel a Demigod's power rushing in one's veins while doing that annoying weeding in your backyard?

So, what can we do about highly interconnected problems that rely a lot on shared memory to parallelize a solution?

Fortunately, many interconnected problems in their traditional solutions are connected much more than necessary just because software abstractions such as MPI make shared memory access look cheap and iterative approach is used more than it needs be. Or at least less iterative approach exists. It's just sometimes you need a Demigod's wisdom to see it. But as they say it in the east, the need is the forefather of all inventions.