How does the inclusion of between-sentence tokens or separate begin- and end-sentence tokens affect the computation of perplexity in language models?

Question

Answer

It decreases perplexity by providing clearer sentence boundaries

Answer

It increases the perplexity by introducing additional tokens in the vocabulary

Answer

We do not know how exactly, it only adjusts the total count of word tokens considered in the perplexity calculation

Answer

It has no impact on the perplexity calculation

Crowdly