Reproduce result

[minhquoc0712]

Hi,
Thank you for your interesting research.
I would like to ask about the Booster finetune GSM8K code. In it, lambda is set to 25. Is that the value you used for GSM8K, or is it actually 5 as mentioned in the paper?

Booster/script/finetune/smooth_gsm8k.sh

Line 19 in 8f14566

lamb=25

In the additional results of GSM8K, why in the default setting the HS score of GSM8K is 9.50, while in Table 3, it is 6.40?

[huangtiansheng]

Hi,

Thanks for raising this issue. The lambda should be set to 25 for GSM8K.

In Table 3, we are using bs=5 and all lambda set to 5 for gsm8k experiment.

In Table 10 11 12 13 14 15, we are using bs=10 and lambda set to 25.

Apology for the confusion. We forgot to mention this subtle difference in the paper. The reason for this difference is that in our later stage of project (when we are producing results for Table 10-15), we found that Booster is performing better in lambda=25, but we forgot to mention this in the paper. You should be able to fully produce the number in Table 10-15 by the script.

Thanks again for the interest.

[minhquoc0712]

Hi,
Thank you for your response.
So for table 3, in alignment phase, lambda is set to 5 and batch size is set to 10, in finetuning phase, batch size is set to 10 for SST2 and AGNEWS, while batch size is set to 5 in GSM8K?

[huangtiansheng]

That is true. For GSM8K, in our initial experiment (table 3), we mistakenly set the bs of gsm8k in fine-tuning stage to be 5.

[minhquoc0712]

Maybe you have some typo or something since I don't see where we use lambda=25?

[huangtiansheng]

In Table 10 11 12 13 14 15, we are using bs=10 (in both alignment and fine-tuning) and lambda set to 25. We forgot to mention this in the paper.

[minhquoc0712]

Thank you. So just for confirmation: lambda=5, batch_size=10, finetune_batch_size=5 (Table 3) give better results HS (HS=6.40) than lambda=25, batch_size=10, finetune_batch_size=10 (Table 10, HS=9.5) right?

[huangtiansheng]

In my understanding, I think is yes. But I suggest to just keep lambda=25, batch size=10 , fine-tune_batch_size=10. This is my final experiment trial and I am pretty sure that you can reproduce this result.

[minhquoc0712]

Hi, for the table 12, Gemma2-9b and Qwen2-7b, you also use lambda=5, batch_size=10, finetune_batch_size=10?

[huangtiansheng]

No, they are using lambda=25, batch_size=10, finetune_batch_size=10

[minhquoc0712]

So this is you final config right?

Model	SST2	AGNEWS	GSM8K
Llama2-7B	λ=5, al_bs=10, ft_bs=10	λ=5, al_bs=10, ft_bs=10	λ=25, al_bs=10, ft_bs=10
Gemma2-9B	λ=25, al_bs=10, ft_bs=10	λ=25, al_bs=10, ft_bs=10	λ=25, al_bs=10, ft_bs=10
Qwen2-7B	λ=25, al_bs=10, ft_bs=10	λ=25, al_bs=10, ft_bs=10	λ=25, al_bs=10, ft_bs=10

[huangtiansheng]

For sst2 and agnews, results of gemma2 and qwen2 is lambda=5, al_bs=10, ft_bs=10