Sort and unsort the input texts for transformers

[feihugis]

Sort the input text by their length so that the padding for each batch can be as small as possible.

[feihugis]

Without this PR:

Util	Model	Task	Split	BatchSize	Samples	Tokens	Bleu	Rouge	Loss	Perplexity	Runtime(seconds)	Throughput(samples/s)	Throughput(tokens/s)
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	32	1024	NA	NA	34.92|14.95|25.28	NA	NA	179	5.7	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	64	1024	NA	NA	34.96|14.98|25.25	NA	NA	100	10.2	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	128	1024	NA	NA	34.96|14.92|25.28	NA	NA	92	11.1	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	32	1024	NA	NA	34.92|14.96|25.30	NA	NA	136	7.5	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	64	1024	NA	NA	34.95|14.93|25.24	NA	NA	97	10.6	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	128	1024	NA	NA	34.97|14.95|25.28	NA	NA	92	11.1	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	32	1024	NA	NA	34.93|14.95|25.27	NA	NA	135	7.6	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	64	1024	NA	NA	34.93|14.93|25.25	NA	NA	97	10.6	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	128	1024	NA	NA	34.96|14.96|25.25	NA	NA	92	11.1	NA

With this PR:

Util	Model	Task	Split	BatchSize	Samples	Tokens	Bleu	Rouge	Loss	Perplexity	Runtime(seconds)	Throughput(samples/s)	Throughput(tokens/s)
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	32	1024	NA	NA	34.93|14.95|25.26	NA	NA	167	6.1	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	64	1024	NA	NA	34.92|14.90|25.24	NA	NA	98	10.4	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	128	1024	NA	NA	35.02|15.01|25.30	NA	NA	86	11.9	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	32	1024	NA	NA	34.95|15.00|25.26	NA	NA	126	8.1	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	64	1024	NA	NA	34.91|14.88|25.21	NA	NA	94	10.9	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	128	1024	NA	NA	35.01|15.00|25.30	NA	NA	87	11.8	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	32	1024	NA	NA	34.96|15.02|25.29	NA	NA	127	8.1	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	64	1024	NA	NA	34.91|14.87|25.24	NA	NA	94	10.9	NA
transformers_v3.0.2+fastseq_v0.0.4	facebook/bart-large-cnn	cnn_dm.1k/raw	val	128	1024	NA	NA	35.03|15.00|25.32	NA	NA	88	11.6	NA

Run the below command on GPU-1:

sudo docker run --gpus device="2" --network=host -v /datadrive:/datadrive -v /tmp:/tmp --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 adsbrainwestus2.azurecr.io/fastseq:dev-py3 /bin/bash -c "cd /datadrive/fhu/github/fastseq && pip install -e . && cd benchmarks && bash models/hf_bart.sh"

[JiushengChen]

Update doc and benchmark.

[yuyan2do]

Any perf for large data?

[feihugis]

@yuyan2do Do you mean the benchmarking result on a larger dataset than the data in our benchmark script?

[JiushengChen]

I also have similar worry, as it loaded all data into memory, and then sort. Please check perf on a larger dataset.

[feihugis]

Here is the test results on a larger dataset

• 5,242,880 of examples (~ 16GB): sort took ~4.4 seconds; unsort took: 1.4 seconds;
• 15,728,640 of examples (~ 48GB): sort took ~14.0 seconds; unsort took: 4.6 seconds;