In theory shadow could one day support sharding simulations across multiple machines. We had planned on building this feature for this reason - that renting or acquiring multiple smaller machines may be cheaper and more feasible than one huge one - but ultimately prioritized other features instead. It's a big feature that would probably need some kind of funding to get accomplished.

Does mininet or other emulators support sharding across machines like this? Or maybe you're talking about not using emulation at all and just running on a real private network of 5k machines?

Trying to orchestrate 5k physical machines sounds painful to me, though maybe with the right tools (puppet?) it could be manageable.

Depending on the emulator (or if you don't use an emulator at all) you may lose other properties that shadow provides:

• determinism
• the ability to run faster than real-time for idle periods
• the ability to simulate whatever network topology and link characteristics you want

with CPU time simulated as well

Maybe - though it sounds like you'd be "modeling" whatever CPU speed and parallelism happens to be on the physical machines you're renting. You'd either want to rent beefy enough machines that whatever you're doing doesn't become CPU-bound (in which case you're your results won't be so different than shadow's model of "cpu time is free"), or machines that accurately reflects how powerful you expect the "real" machines to be, which may be more expensive than "the cheapest Digital Ocean instance".

As with sharding, modeling CPU time in shadow is a feature that could potentially be added to shadow. I think modeling CPU time for a single-CPU machine wouldn't be terribly difficult, but would add some complexity and performance overhead. Trying to accurately model CPU time of a multi-core machine would be much more difficult though, especially while preserving determinism.

I spot a few problems with this analysis.

• Most obviously, 20x the price for comparable scale is not what I would consider competitive. ;) You would probably be better off using that money to run more (or larger) simulations to tighten your error bars.
• While I've seen some people run Shadow on hosting services, most people I know use it on-site (for some loose definition of that, e.g. a rack in a university data center). Even if you don't have large org resources, you can buy a used server or workstation with enough memory to run experiments yourself for around $1000. If you do that, it's obviously never going to be more efficient to run hosts on VMs or physical machines than as processes in Shadow.
• You say "you have one CPU for each node". This is not true for the Basic tier of Digital Ocean Droplets. You're on a shared host, and they will throttle you based on both your load and the load of other customers. D.O. Droplets with a dedicated CPU thread per host start at $42/mo.
• You're not going to get valid results if you run a "real environment" on its own. You need an emulator to go with it, to ensure that the links are behaving realistically, assuming you're not trying to study the properties of the data center you're running the experiment on. This is going to require setting up custom images, and, since all the network emulators I know of require centralized routing, something considerably beefier than a $4 droplet (CPU, memory, and bandwidth).
• Related to the previous two points, even if you do manage to achieve the scale you're looking for, you still are going to have the same set of problems emulators always have around reproducibility and configurability. Variables as mundane as "what temperature is it" and, especially in a shared hosting environment, "what else is running on the system" can cause wildly different results, and you're always going to be constrained by the performance of the underlying hardware.

There are situations where emulation does beat Shadow, the big ones being if you do have access to lots of small machines and not a large one, you're running software incompatible with Shadow, or you're trying to study a system bottle-necked on compute and not the network (in which case, accurate network properties matter less). But because emulators are subject to more physical constraints, and don't simulate CPU time, the idea of "it's real, therefore more accurate" doesn't actually pan out for network-constrained experiments.

I'm potentially a big fan of Shadow approach because it is deterministic. I.e. if something weird happened in the network anyone may potentially reproduce and investigate exactly this case.

However there are some difficulties:

• Accounting CPU delay. It is sometimes crucial to take it into account considering heavy KZG and BLS operations. (this could basically be solved by using stub KZG and BLS and emulating their CPU time with a sleep call)
• Accounting disk access time. E.g. restoring a states from DB may cause significant lags for a node. BTW is the disk access time considered to be 0 simulated ms (similar to CPU time)?
• Need some client adoption and maintenance to
  • avoid missing syscalls
  • be deterministic