question about pass@1

[zero-peace]

Hello, thank you very much for your excellent work. I have a question regarding your SQL-R1-7B model. When I evaluate it on the bird-dev dataset using pass@1, I get a score of 33%. Could you please let me know if this performance is within the expected range? I would greatly appreciate your guidance.

[MPX0222]

Currently, we mainly use Execution Accuracy (EX) to evaluate the quality of the generated SQL. The pass@k evaluation is not used in our evaluation script. If you need to use this metric, you may need to modify it for NL2SQL tasks.

[zero-peace]

您好，我的表述存在一点问题，我这里使用的指标也是EX，我看到你们的推理方式采用了自一致性，采样次数为4，如果不采用自一致性，采样次数为1的话，结果为0.33属于正常的范围吗？在论文中看到了对于采样次数的分析，但都是从4开始的分析。感谢解答。

[MPX0222]

如果不采用自一致性检查，您可能需要调整 temperature 为 0，以及可能调整更多其他参数，否则可能导致生成稳定性下降。可以参考 BIRD bench的描述，端到端的单个模型允许自一致性推理，因为它反映了模型自身的能力。通常来说，采样大于1的效果会比采样为1的效果更好，因此我们在文中讨论了更多采样数量下的情况，并在采样数为8时观察到较为稳定和高效的生成结果。

[zero-peace]

了解了，非常感谢，祝高分！