MOE uses more memory than dense model and is slower

[samuelwheeler]

I am training a ~520 M model, but I have found that the megablocks moe version uses substantially more memory and takes longer to train than a dense model of corresponding size. I am using a model embedding dimension of 1536. The moe model has 48 experts with 8 active and and expert size of 128. I set lbl loss weight to 0.001.

[samuelwheeler]

I tried again with the same settings except that I set the number of active experts to 24, still half of the active parameters of the dense model with equal number of total parameters. Got an OOM error.