Skip to content

Implement log probabilities for chat completions #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

atbe
Copy link

@atbe atbe commented Jun 24, 2025

Log probability support was introduced across Ollama's OpenAI-compatible endpoints.

Key changes include:

  • In api/types.go:
    • A new LogProbs struct was defined to hold token-level log probabilities, token IDs, and top-N log probability maps.
    • ChatResponse and GenerateResponse structs were extended with an optional *LogProbs field.
    • Options struct gained LogProbsEnabled (bool) and TopLogProbs (int) fields to control logprob generation.
  • In llm/server.go:
    • The internal completion struct was updated to parse log probability data from the backend.
    • The Completion function now forwards logprobs and top_logprobs parameters to the core generation request and populates the LogProbs field in CompletionResponse.
  • In openai/openai.go:
    • OpenAI API response models (Choice, ChunkChoice, CompleteChunkChoice) were updated to include Logprobs.
    • Request models (ChatCompletionRequest, CompletionRequest) now accept logprobs and top_logprobs parameters, which are then mapped to the internal api.Options.
    • Conversion functions were updated to correctly map internal api.LogProbs to the OpenAI schema.
  • In server/routes.go:
    • GenerateHandler and ChatHandler were updated to include the LogProbs field in the final API responses.

This enables clients to request and receive per-token log probability information (with configurable top-N) in both normal and streaming modes, maintaining compatibility with the OpenAI chat completions schema.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants