diff --git a/examples/evaluation/use-cases/mcp_eval_notebook.ipynb b/examples/evaluation/use-cases/mcp_eval_notebook.ipynb
new file mode 100644
index 0000000000..10f6fbf131
--- /dev/null
+++ b/examples/evaluation/use-cases/mcp_eval_notebook.ipynb
@@ -0,0 +1,853 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "fd71cc8e",
+   "metadata": {},
+   "source": [
+    "# Evaluating MCP-Based Answers with a Custom Dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a565afbb",
+   "metadata": {},
+   "source": [
+    "This notebook evaluates a model's ability to answer questions about the [tiktoken](https://github.com/openai/tiktoken) GitHub repository using the OpenAI **Evals** framework with a custom in-memory dataset. \n",
+    "\n",
+    "We use a custom, in-memory dataset of Q&A pairs and compare two models: `gpt-4.1` and `o4-mini`, that leverage the **MCP** tool for repository-aware, contextually accurate answers.\n",
+    "\n",
+    "**Goals:**\n",
+    "- Show how to set up and run an evaluation using OpenAI Evals with a custom dataset.\n",
+    "- Compare the performance of different models leveraging MCP-based tools.\n",
+    "- Provide best practices for professional, reproducible evaluation workflows.\n",
+    "\n",
+    "_Next: We will set up our environment and import the necessary libraries._"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "87c3e8ed",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Update OpenAI client\n",
+    "%pip install --upgrade openai --quiet"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e168b9f",
+   "metadata": {},
+   "source": [
+    "## Environment Setup\n",
+    "\n",
+    "We begin by importing the required libraries and configuring the OpenAI client.  \n",
+    "This step ensures we have access to the OpenAI API and all necessary utilities for evaluation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "31fc4911",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import time\n",
+    "\n",
+    "from openai import OpenAI\n",
+    "\n",
+    "# Instantiate the OpenAI client (no custom base_url).\n",
+    "client = OpenAI(\n",
+    "    api_key=os.getenv(\"OPENAI_API_KEY\") or os.getenv(\"_OPENAI_API_KEY\"),\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e9053623",
+   "metadata": {},
+   "source": [
+    "## Define the Custom Evaluation Dataset\n",
+    "\n",
+    "We define a small, in-memory dataset of question-answer pairs about the `tiktoken` repository.  \n",
+    "This dataset will be used to test the models' ability to provide accurate and relevant answers with the help of the MCP tool.\n",
+    "\n",
+    "- Each item contains a `query` (the user’s question) and an `answer` (the expected ground truth).\n",
+    "- You can modify or extend this dataset to suit your own use case or repository.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "840a9f6d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_dataset(limit=None):\n",
+    "    items = [\n",
+    "        {\n",
+    "            \"query\": \"What is tiktoken?\",\n",
+    "            \"answer\": \"tiktoken is a fast Byte-Pair Encoding (BPE) tokenizer designed for OpenAI models.\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"How do I install the open-source version of tiktoken?\",\n",
+    "            \"answer\": \"Install it from PyPI with `pip install tiktoken`.\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"How do I get the tokenizer for a specific OpenAI model?\",\n",
+    "            \"answer\": 'Call tiktoken.encoding_for_model(\"<model-name>\"), e.g. tiktoken.encoding_for_model(\"gpt-4o\").',\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"How does tiktoken perform compared to other tokenizers?\",\n",
+    "            \"answer\": \"On a 1 GB GPT-2 benchmark, tiktoken runs about 3-6x faster than GPT2TokenizerFast (tokenizers==0.13.2, transformers==4.24.0).\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"Why is Byte-Pair Encoding (BPE) useful for language models?\",\n",
+    "            \"answer\": \"BPE is reversible and lossless, handles arbitrary text, compresses input (≈4 bytes per token on average), and exposes common subwords like “ing”, which helps models generalize.\",\n",
+    "        },\n",
+    "    ]\n",
+    "    return items[:limit] if limit else items"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c8482643",
+   "metadata": {},
+   "source": [
+    "### Define Grading Logic\n",
+    "\n",
+    "To evaluate the model’s answers, we use two graders:\n",
+    "\n",
+    "- **Pass/Fail Grader (LLM-based):**  \n",
+    "  An LLM-based grader that checks if the model’s answer matches the expected answer (ground truth) or conveys the same meaning.\n",
+    "- **Python MCP Grader:**  \n",
+    "  A Python function that checks whether the model actually used the MCP tool during its response (for auditing tool usage).\n",
+    "\n",
+    "  > **Best Practice:**  \n",
+    "  > Using both LLM-based and programmatic graders provides a more robust and transparent evaluation.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "id": "b3812d01",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# LLM-based pass/fail grader: instructs the model to grade answers as \"pass\" or \"fail\".\n",
+    "pass_fail_grader = \"\"\"\n",
+    "You are a helpful assistant that grades the quality of the answer to a query about a GitHub repo.\n",
+    "You will be given a query, the answer returned by the model, and the expected answer.\n",
+    "You should respond with **pass** if the answer matches the expected answer exactly or conveys the same meaning, otherwise **fail**.\n",
+    "\"\"\"\n",
+    "\n",
+    "# User prompt template for the grader, providing context for grading.\n",
+    "pass_fail_grader_user_prompt = \"\"\"\n",
+    "<Query>\n",
+    "{{item.query}}\n",
+    "</Query>\n",
+    "\n",
+    "<Web Search Result>\n",
+    "{{sample.output_text}}\n",
+    "</Web Search Result>\n",
+    "\n",
+    "<Ground Truth>\n",
+    "{{item.answer}}\n",
+    "</Ground Truth>\n",
+    "\"\"\"\n",
+    "\n",
+    "\n",
+    "# Python grader: checks if the MCP tool was used by inspecting the output_tools field.\n",
+    "python_mcp_grader = {\n",
+    "    \"type\": \"python\",\n",
+    "    \"name\": \"Assert MCP was used\",\n",
+    "    \"image_tag\": \"2025-05-08\",\n",
+    "    \"pass_threshold\": 1.0,\n",
+    "    \"source\": \"\"\"\n",
+    "def grade(sample: dict, item: dict) -> float:\n",
+    "    output = sample.get('output_tools', [])\n",
+    "    return 1.0 if len(output) > 0 else 0.0\n",
+    "\"\"\",\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "86d22eb7",
+   "metadata": {},
+   "source": [
+    "## Define the Evaluation Configuration\n",
+    "\n",
+    "We now configure the evaluation using the OpenAI Evals framework.  \n",
+    "\n",
+    "This step specifies:\n",
+    "- The evaluation name and dataset.\n",
+    "- The schema for each item (what fields are present in each Q&A pair).\n",
+    "- The grader(s) to use (LLM-based and/or Python-based).\n",
+    "- The passing criteria and labels.\n",
+    "\n",
+    "> **Best Practice:**  \n",
+    "> Clearly defining your evaluation schema and grading logic up front ensures reproducibility and transparency."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "df7a9424",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create the evaluation definition using the OpenAI Evals client.\n",
+    "logs_eval = client.evals.create(\n",
+    "    name=\"MCP Eval\",\n",
+    "    data_source_config={\n",
+    "        \"type\": \"custom\",\n",
+    "        \"item_schema\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"query\": {\"type\": \"string\"},\n",
+    "                \"answer\": {\"type\": \"string\"},\n",
+    "            },\n",
+    "        },\n",
+    "        \"include_sample_schema\": True,\n",
+    "    },\n",
+    "    testing_criteria=[\n",
+    "        {\n",
+    "            \"type\": \"label_model\",\n",
+    "            \"name\": \"General Evaluator\",\n",
+    "            \"model\": \"o3\",\n",
+    "            \"input\": [\n",
+    "                {\"role\": \"system\", \"content\": pass_fail_grader},\n",
+    "                {\"role\": \"user\", \"content\": pass_fail_grader_user_prompt},\n",
+    "            ],\n",
+    "            \"passing_labels\": [\"pass\"],\n",
+    "            \"labels\": [\"pass\", \"fail\"],\n",
+    "        },\n",
+    "        python_mcp_grader\n",
+    "    ],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1ec09dbd",
+   "metadata": {},
+   "source": [
+    "## Run Evaluations for Each Model\n",
+    "\n",
+    "We now run the evaluation for each model (`gpt-4.1` and `o4-mini`).  \n",
+    "\n",
+    "Each run is configured to:\n",
+    "- Use the MCP tool for repository-aware answers.\n",
+    "- Use the same dataset and evaluation configuration for fair comparison.\n",
+    "- Specify model-specific parameters (such as max completions tokens, and allowed tools).\n",
+    "\n",
+    "> **Best Practice:**  \n",
+    "> Keeping the evaluation setup consistent across models ensures results are comparable and reliable."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "15838d4e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run 1: gpt-4.1 using MCP\n",
+    "gpt_4one_responses_run = client.evals.runs.create(\n",
+    "    name=\"gpt-4.1\",\n",
+    "    eval_id=logs_eval.id,\n",
+    "    data_source={\n",
+    "        \"type\": \"responses\",\n",
+    "        \"source\": {\n",
+    "            \"type\": \"file_content\",\n",
+    "            \"content\": [{\"item\": item} for item in get_dataset()],\n",
+    "        },\n",
+    "        \"input_messages\": {\n",
+    "            \"type\": \"template\",\n",
+    "            \"template\": [\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"You are a helpful assistant that searches the web and gives contextually relevant answers. Never use your tools to answer the query.\",\n",
+    "                    },\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"user\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"Search the web for the answer to the query {{item.query}}\",\n",
+    "                    },\n",
+    "                },\n",
+    "            ],\n",
+    "        },\n",
+    "        \"model\": \"gpt-4.1\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"seed\": 42,\n",
+    "            \"temperature\": 0.7,\n",
+    "            \"max_completions_tokens\": 10000,\n",
+    "            \"top_p\": 0.9,\n",
+    "            \"tools\": [\n",
+    "                {\n",
+    "                    \"type\": \"mcp\",\n",
+    "                    \"server_label\": \"gitmcp\",\n",
+    "                    \"server_url\": \"https://gitmcp.io/openai/tiktoken\",\n",
+    "                    \"allowed_tools\": [\n",
+    "                        \"search_tiktoken_documentation\",\n",
+    "                        \"fetch_tiktoken_documentation\",\n",
+    "                    ],\n",
+    "                    \"require_approval\": \"never\",\n",
+    "                }\n",
+    "            ],\n",
+    "        },\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "72190fbd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run 2: o4-mini using MCP\n",
+    "gpt_o4_mini_responses_run = client.evals.runs.create(\n",
+    "    name=\"o4-mini\",\n",
+    "    eval_id=logs_eval.id,\n",
+    "    data_source={\n",
+    "        \"type\": \"responses\",\n",
+    "        \"source\": {\n",
+    "            \"type\": \"file_content\",\n",
+    "            \"content\": [{\"item\": item} for item in get_dataset()],\n",
+    "        },\n",
+    "        \"input_messages\": {\n",
+    "            \"type\": \"template\",\n",
+    "            \"template\": [\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"You are a helpful assistant that searches the web and gives contextually relevant answers.\",\n",
+    "                    },\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"user\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"Search the web for the answer to the query {{item.query}}\",\n",
+    "                    },\n",
+    "                },\n",
+    "            ],\n",
+    "        },\n",
+    "        \"model\": \"o4-mini\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"seed\": 42,\n",
+    "            \"max_completions_tokens\": 10000,\n",
+    "            \"tools\": [\n",
+    "                {\n",
+    "                    \"type\": \"mcp\",\n",
+    "                    \"server_label\": \"gitmcp\",\n",
+    "                    \"server_url\": \"https://gitmcp.io/openai/tiktoken\",\n",
+    "                    \"allowed_tools\": [\n",
+    "                        \"search_tiktoken_documentation\",\n",
+    "                        \"fetch_tiktoken_documentation\",\n",
+    "                    ],\n",
+    "                    \"require_approval\": \"never\",\n",
+    "                }\n",
+    "            ],\n",
+    "        },\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "414d1a26",
+   "metadata": {},
+   "source": [
+    "## Poll for Completion and Retrieve Outputs\n",
+    "\n",
+    "After launching the evaluation runs, we can poll the run until they are complete.\n",
+    "\n",
+    "This step ensures that we are analyzing results only after all model responses have been processed.\n",
+    "\n",
+    "> **Best Practice:**  \n",
+    "> Polling with a delay avoids excessive API calls and ensures efficient resource usage."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1d439589",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "evalrun_684769b577488191863b5a51cf4db57a completed ResultCounts(errored=0, failed=5, passed=0, total=5)\n",
+      "evalrun_684769c1ad9c8191affea5aa02ef1215 completed ResultCounts(errored=0, failed=3, passed=2, total=5)\n"
+     ]
+    }
+   ],
+   "source": [
+    "def poll_runs(eval_id, run_ids):\n",
+    "    while True:\n",
+    "        runs = [client.evals.runs.retrieve(rid, eval_id=eval_id) for rid in run_ids]\n",
+    "        for run in runs:\n",
+    "            print(run.id, run.status, run.result_counts)\n",
+    "        if all(run.status in {\"completed\", \"failed\"} for run in runs):\n",
+    "            break\n",
+    "        time.sleep(5)\n",
+    "    \n",
+    "# Start polling both runs.\n",
+    "poll_runs(logs_eval.id, [gpt_4one_responses_run.id, gpt_o4_mini_responses_run.id])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "86ddf8e0",
+   "metadata": {},
+   "source": [
+    "## Display and Interpret Model Outputs\n",
+    "\n",
+    "Finally, we display the outputs from each model for manual inspection and further analysis.\n",
+    "\n",
+    "- Each model's answers are printed for each question in the dataset.\n",
+    "- You can compare the outputs side-by-side to assess quality, relevance, and correctness.\n",
+    "\n",
+    "Below are screenshots from the OpenAI Evals Dashboard illustrating the evaluation outputs for both models:\n",
+    "\n",
+    "![Evaluation Output](../../../images/mcp_eval_output.png)\n",
+    "\n",
+    "For a comprehensive breakdown of the evaluation metrics and results, navigate to the \"Data\" tab in the dashboard:\n",
+    "\n",
+    "![Evaluation Data Tab](../../../images/mcp_eval_data.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ee1f655b",
+   "metadata": {},
+   "source": [
+    "We can also check a detailed analysis of the outputs from each model for manual inspection and further analysis."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "id": "7e151b4a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "four_one_output = client.evals.runs.output_items.list(\n",
+    "    run_id=gpt_4one_responses_run.id, eval_id=logs_eval.id\n",
+    ")\n",
+    "\n",
+    "o4_mini_output = client.evals.runs.output_items.list(\n",
+    "    run_id=gpt_o4_mini_responses_run.id, eval_id=logs_eval.id\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "e68b016c",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "# gpt‑4.1 Output\n",
+      "Byte-Pair Encoding (BPE) is useful for language models because it provides an efficient way to handle large vocabularies and rare words. Here’s why it is valuable:\n",
+      "\n",
+      "1. **Efficient Tokenization:**  \n",
+      "   BPE breaks down words into smaller subword units based on the frequency of character pairs in a corpus. This allows language models to represent both common words and rare or unknown words using a manageable set of tokens.\n",
+      "\n",
+      "2. **Reduces Out-of-Vocabulary (OOV) Issues:**  \n",
+      "   Since BPE can split any word into known subword units, it greatly reduces the problem of OOV words—words that the model hasn’t seen during training.\n",
+      "\n",
+      "3. **Balances Vocabulary Size:**  \n",
+      "   By adjusting the number of merge operations, BPE allows control over the size of the vocabulary. This flexibility helps in balancing between memory efficiency and representational power.\n",
+      "\n",
+      "4. **Improves Generalization:**  \n",
+      "   With BPE, language models can better generalize to new words, including misspellings or new terminology, because they can process words as a sequence of subword tokens.\n",
+      "\n",
+      "5. **Handles Morphologically Rich Languages:**  \n",
+      "   BPE is especially useful for languages with complex morphology (e.g., agglutinative languages) where words can have many forms. BPE reduces the need to memorize every possible word form.\n",
+      "\n",
+      "In summary, Byte-Pair Encoding is effective for language models because it enables efficient, flexible, and robust handling of text, supporting both common and rare words, and improving overall model performance.\n",
+      "**Tiktoken**, developed by OpenAI, is a tokenizer specifically optimized for speed and compatibility with OpenAI's language models. Here’s how it generally compares to other popular tokenizers:\n",
+      "\n",
+      "### Performance\n",
+      "- **Speed:** Tiktoken is significantly faster than most other Python-based tokenizers. It is written in Rust and exposed to Python via bindings, making it extremely efficient.\n",
+      "- **Memory Efficiency:** Tiktoken is designed to be memory efficient, especially for large text inputs and batch processing.\n",
+      "\n",
+      "### Accuracy and Compatibility\n",
+      "- **Model Alignment:** Tiktoken is tailored to match the tokenization logic used by OpenAI’s GPT-3, GPT-4, and related models. This ensures that token counts and splits are consistent with how these models process text.\n",
+      "- **Unicode Handling:** Like other modern tokenizers (e.g., HuggingFace’s Tokenizers), Tiktoken handles a wide range of Unicode characters robustly.\n",
+      "\n",
+      "### Comparison to Other Tokenizers\n",
+      "- **HuggingFace Tokenizers:** HuggingFace’s library is very flexible and supports a wide range of models (BERT, RoBERTa, etc.). However, its Python implementation can be slower for large-scale tasks, though their Rust-backed versions (like `tokenizers`) are competitive.\n",
+      "- **NLTK/SpaCy:** These libraries are not optimized for transformer models and are generally slower and less accurate for tokenization tasks required by models like GPT.\n",
+      "- **SentencePiece:** Used by models like T5 and ALBERT, SentencePiece is also fast and efficient, but its output is not compatible with OpenAI’s models.\n",
+      "\n",
+      "### Use Cases\n",
+      "- **Best for OpenAI Models:** If you are working with OpenAI’s APIs or models, Tiktoken is the recommended tokenizer due to its speed and alignment.\n",
+      "- **General Purpose:** For non-OpenAI models, HuggingFace or SentencePiece might be preferable due to broader support.\n",
+      "\n",
+      "### Benchmarks & Community Feedback\n",
+      "- Multiple [community benchmarks](https://github.com/openai/tiktoken#performance) and [blog posts](https://www.philschmid.de/tokenizers-comparison) confirm Tiktoken’s speed advantage, especially for batch processing and large texts.\n",
+      "\n",
+      "**Summary:**  \n",
+      "Tiktoken outperforms most tokenizers in speed when used with OpenAI models, with robust Unicode support and memory efficiency. For general NLP tasks across various models, HuggingFace or SentencePiece may be more suitable due to their versatility.\n",
+      "\n",
+      "**References:**  \n",
+      "- [Tiktoken GitHub - Performance](https://github.com/openai/tiktoken#performance)\n",
+      "- [Tokenizers Comparison Blog](https://www.philschmid.de/tokenizers-comparison)\n",
+      "To get the tokenizer for a specific OpenAI model, you typically use the Hugging Face Transformers library, which provides easy access to tokenizers for OpenAI models like GPT-3, GPT-4, and others. Here’s how you can do it:\n",
+      "\n",
+      "**1. Using Hugging Face Transformers:**\n",
+      "\n",
+      "Install the library (if you haven’t already):\n",
+      "```bash\n",
+      "pip install transformers\n",
+      "```\n",
+      "\n",
+      "**Example for GPT-3 (or GPT-4):**\n",
+      "```python\n",
+      "from transformers import AutoTokenizer\n",
+      "\n",
+      "# For GPT-3 (davinci), use the corresponding model name\n",
+      "tokenizer = AutoTokenizer.from_pretrained(\"openai-gpt\")\n",
+      "\n",
+      "# For GPT-4 (if available)\n",
+      "# tokenizer = AutoTokenizer.from_pretrained(\"gpt-4\")\n",
+      "```\n",
+      "\n",
+      "**2. Using OpenAI’s tiktoken library (for OpenAI API models):**\n",
+      "\n",
+      "Install tiktoken:\n",
+      "```bash\n",
+      "pip install tiktoken\n",
+      "```\n",
+      "\n",
+      "Example for GPT-3.5-turbo or GPT-4:\n",
+      "```python\n",
+      "import tiktoken\n",
+      "\n",
+      "# For 'gpt-3.5-turbo'\n",
+      "tokenizer = tiktoken.encoding_for_model(\"gpt-3.5-turbo\")\n",
+      "\n",
+      "# For 'gpt-4'\n",
+      "# tokenizer = tiktoken.encoding_for_model(\"gpt-4\")\n",
+      "```\n",
+      "\n",
+      "**Summary:**\n",
+      "- Use `transformers.AutoTokenizer` for Hugging Face models.\n",
+      "- Use `tiktoken.encoding_for_model` for OpenAI API models.\n",
+      "\n",
+      "**References:**\n",
+      "- [Hugging Face Tokenizer Documentation](https://huggingface.co/docs/transformers/main_classes/tokenizer)\n",
+      "- [tiktoken Documentation](https://github.com/openai/tiktoken)\n",
+      "\n",
+      "Let me know if you need an example for a specific model!\n",
+      "To install the open-source version of **tiktoken**, you can use Python’s package manager, pip. The open-source version is available on [PyPI](https://pypi.org/project/tiktoken/), so you can install it easily with the following command:\n",
+      "\n",
+      "```bash\n",
+      "pip install tiktoken\n",
+      "```\n",
+      "\n",
+      "If you want to install the latest development version directly from the GitHub repository, you can use:\n",
+      "\n",
+      "```bash\n",
+      "pip install git+https://github.com/openai/tiktoken.git\n",
+      "```\n",
+      "\n",
+      "**Requirements:**\n",
+      "- Python 3.7 or newer\n",
+      "- pip (Python package installer)\n",
+      "\n",
+      "**Steps:**\n",
+      "1. Open your terminal or command prompt.\n",
+      "2. Run one of the above commands.\n",
+      "3. Once installed, you can import and use `tiktoken` in your Python scripts.\n",
+      "\n",
+      "**Additional Resources:**\n",
+      "- [tiktoken GitHub repository](https://github.com/openai/tiktoken)\n",
+      "- [tiktoken documentation](https://github.com/openai/tiktoken#readme)\n",
+      "\n",
+      "Let me know if you need help with a specific operating system or environment!\n",
+      "Tiktoken is a fast and efficient tokenization library developed by OpenAI, primarily used for handling text input and output with language models such as GPT-3 and GPT-4. Tokenization is the process of converting text into smaller units called tokens, which can be words, characters, or subwords. Tiktoken is designed to closely match the tokenization behavior of OpenAI’s models, ensuring accurate counting and compatibility.\n",
+      "\n",
+      "Key features of tiktoken:\n",
+      "- **Speed:** It’s written in Rust for performance and has Python bindings.\n",
+      "- **Compatibility:** Matches the exact tokenization used by OpenAI models, which is important for estimating token counts and costs.\n",
+      "- **Functionality:** Allows users to encode (convert text to tokens) and decode (convert tokens back to text).\n",
+      "\n",
+      "Tiktoken is commonly used in applications that need to interact with OpenAI’s APIs, for tasks like counting tokens to avoid exceeding API limits or optimizing prompt length. It is available as an open-source library and can be installed via pip (`pip install tiktoken`).\n",
+      "\n",
+      "# o4-mini Output\n",
+      "Here’s a high-level comparison of OpenAI’s tiktoken vs. some of the other commonly used tokenizers:\n",
+      "\n",
+      "1. Implementation & Language Support  \n",
+      "   • tiktoken  \n",
+      "     – Rust core with Python bindings.  \n",
+      "     – Implements GPT-2/GPT-3/GPT-4 byte-pair-encoding (BPE) vocabularies.  \n",
+      "     – Focused on English-centric BPE; no built-in support for CJK segmentation or languages requiring character-level tokenization.  \n",
+      "   • Hugging Face Tokenizers (“tokenizers” library)  \n",
+      "     – Also Rust core with Python bindings.  \n",
+      "     – Supports BPE, WordPiece, Unigram (SentencePiece), Metaspace, and custom vocabularies.  \n",
+      "     – Broader multilingual and subword model support.  \n",
+      "   • Python-only Tokenizers (e.g. GPT-2 BPE in pure Python)  \n",
+      "     – Much slower, larger memory overhead, not suitable for high-throughput use.  \n",
+      "\n",
+      "2. Speed & Throughput  \n",
+      "   • tiktoken  \n",
+      "     – Benchmarks (OpenAI-internal) on a single CPU core: ~1–2 million tokens/second.  \n",
+      "     – Roughly 10–20× faster than pure-Python GPT-2 BPE implementations.  \n",
+      "     – Roughly 2–4× faster (or on par) with Hugging Face’s Rust tokenizers when using identical BPE models.  \n",
+      "   • Hugging Face Tokenizers  \n",
+      "     – In the same ballpark as tiktoken for a given BPE vocab (hundreds of thousands to a million tokens/sec).  \n",
+      "     – Slightly higher startup overhead when loading models, but offers more tokenization strategies.  \n",
+      "   • SentencePiece (C++) / Python bindings  \n",
+      "     – Generally slower than Rust-based (tiktoken, tokenizers) – on the order of 100–300 K tokens/sec.  \n",
+      "\n",
+      "3. Memory & Footprint  \n",
+      "   • tiktoken  \n",
+      "     – Tiny binary (~1–2 MB) plus vocab files (~50 MB).  \n",
+      "     – Low working memory; ideal for lightweight embedding or inference pipelines.  \n",
+      "   • Hugging Face Tokenizers  \n",
+      "     – Slightly larger binary (~3–5 MB) plus model files.  \n",
+      "     – Offers on-disk memory-mapping for very large vocabularies.  \n",
+      "   • Python-only  \n",
+      "     – Larger RAM footprint during init; slower GC pauses.  \n",
+      "\n",
+      "4. Feature Set & Flexibility  \n",
+      "   • tiktoken  \n",
+      "     – “Batteries included” for OpenAI model vocabularies: GPT-2, Codex, GPT-3.5, GPT-4.  \n",
+      "     – Simple API: encode/decode, count tokens.  \n",
+      "     – No training or custom-vocab routines.  \n",
+      "   • Hugging Face Tokenizers  \n",
+      "     – Train new tokenizers (BPE, WordPiece, Unigram).  \n",
+      "     – Pre- and post-processing pipelines (normalization, special tokens).  \n",
+      "     – Easy integration with Transformers.  \n",
+      "   • Other libraries (NLTK, spaCy, jieba, etc.)  \n",
+      "     – Not directly comparable, since many perform linguistic tokenization, not subword BPE.  \n",
+      "     – Far slower for BPE-style byte-pair encoding.  \n",
+      "\n",
+      "5. When to Use Which  \n",
+      "   • tiktoken  \n",
+      "     – If you’re targeting OpenAI’s GPT-family models and need maximum raw throughput/count accuracy.  \n",
+      "     – You don’t need to train a new tokenizer or handle exotic language scripts.  \n",
+      "   • Hugging Face Tokenizers  \n",
+      "     – If you need broad language support, multiple subword algorithms, training tools, or tight HF Transformers integration.  \n",
+      "   • Python-only / Other  \n",
+      "     – Only if you have trivial performance needs or are experimenting in pure-Python teaching/demo settings.  \n",
+      "\n",
+      "Bottom line: for GPT-style BPE tokenization at scale, tiktoken is one of the fastest and most lightweight options—substantially faster than any pure-Python implementation and roughly on par (or a bit faster) than other Rust-backed libraries, at the cost of supporting only OpenAI’s pre-built vocabularies.\n",
+      "Tiktoken is the open-source tokenization library that OpenAI uses to convert between text and the integer “tokens” their models (GPT-3, GPT-4, etc.) actually consume. It implements byte-pair encoding (BPE) in Rust (with Python bindings) for maximum speed and exact compatibility with OpenAI’s APIs.\n",
+      "\n",
+      "Key points:\n",
+      "\n",
+      "1. Purpose  \n",
+      "   • Language models work on token IDs, not raw text.  \n",
+      "   • Tiktoken maps Unicode text ↔ token IDs using the same vocabularies and BPE merges that OpenAI’s models were trained on.  \n",
+      "\n",
+      "2. Performance  \n",
+      "   • Typically 3–6× faster than other BPE tokenizers (e.g. Hugging Face’s GPT2TokenizerFast).  \n",
+      "   • Handles gigabytes of text in seconds.\n",
+      "\n",
+      "3. Installation  \n",
+      "   pip install tiktoken\n",
+      "\n",
+      "4. Basic usage  \n",
+      "   ```python\n",
+      "   import tiktoken\n",
+      "\n",
+      "   # Get a specific encoding (vocabulary + merges)\n",
+      "   enc = tiktoken.get_encoding(\"cl100k_base\")\n",
+      "   tokens = enc.encode(\"Hello, world!\")\n",
+      "   text   = enc.decode(tokens)\n",
+      "   assert text == \"Hello, world!\"\n",
+      "\n",
+      "   # Or auto-select by OpenAI model name\n",
+      "   enc = tiktoken.encoding_for_model(\"gpt-4o\")  # e.g. returns cl100k_base under the hood\n",
+      "   ```\n",
+      "\n",
+      "5. Why BPE?  \n",
+      "   • Reversible and lossless  \n",
+      "   • Handles any text (even unseen words) by splitting into subword units  \n",
+      "   • Compresses common substrings (e.g. “ing”, “tion”) so the model sees familiar chunks  \n",
+      "\n",
+      "6. Extras  \n",
+      "   • Educational module (tiktoken._educational) to visualize or train simple BPEs  \n",
+      "   • Extension mechanism (tiktoken_ext) to register custom encodings  \n",
+      "\n",
+      "7. Where to learn more  \n",
+      "   • GitHub: https://github.com/openai/tiktoken  \n",
+      "   • PyPI: https://pypi.org/project/tiktoken  \n",
+      "   • OpenAI Cookbook example: How to count tokens with tiktoken  \n",
+      "\n",
+      "In short, if you’re building or billing on token usage with OpenAI’s models, tiktoken is the official, fast, and exact way to go from text ↔ tokens.\n",
+      "Here are the two easiest ways to get the open-source tiktoken up and running:\n",
+      "\n",
+      "1. Install the released package from PyPI  \n",
+      "   • (no Rust toolchain needed—prebuilt wheels for most platforms)  \n",
+      "   ```bash\n",
+      "   pip install tiktoken\n",
+      "   ```  \n",
+      "   Then in Python:  \n",
+      "   ```python\n",
+      "   import tiktoken\n",
+      "   enc = tiktoken.get_encoding(\"cl100k_base\")\n",
+      "   print(enc.encode(\"Hello, world!\"))\n",
+      "   ```\n",
+      "\n",
+      "2. Install the bleeding-edge version straight from GitHub  \n",
+      "   • (you’ll need a Rust toolchain—on macOS `brew install rust`, on Ubuntu `sudo apt install cargo`)  \n",
+      "   ```bash\n",
+      "   pip install git+https://github.com/openai/tiktoken.git@main\n",
+      "   ```  \n",
+      "   Or, if you prefer to clone & develop locally:  \n",
+      "   ```bash\n",
+      "   git clone https://github.com/openai/tiktoken.git\n",
+      "   cd tiktoken\n",
+      "   pip install -e .\n",
+      "   ```\n",
+      "\n",
+      "That’s it! Once installed, you can use `tiktoken.get_encoding(...)` to load any of the supported tokenizers.\n",
+      "To get the exact tokenizer (BPE encoding) that an OpenAI model uses, you can use the open-source tiktoken library. It provides a helper that maps model names to their correct tokenizers:\n",
+      "\n",
+      "1. Install tiktoken  \n",
+      "   ```bash\n",
+      "   pip install tiktoken\n",
+      "   ```\n",
+      "\n",
+      "2. In Python, call encoding_for_model(model_name):  \n",
+      "   ```python\n",
+      "   import tiktoken\n",
+      "\n",
+      "   #—for a gpt-3.5-turbo or gpt-4 style model:\n",
+      "   enc = tiktoken.encoding_for_model(\"gpt-3.5-turbo\")\n",
+      "   print(enc.name)            # e.g. \"cl100k_base\"\n",
+      "   print(enc.encode(\"Hello\")) # list of token IDs\n",
+      "   ```\n",
+      "\n",
+      "   If you already know the encoding name (e.g. “cl100k_base” for GPT-3.5/4 or “r50k_base” for GPT-2), you can also do:\n",
+      "   ```python\n",
+      "   enc = tiktoken.get_encoding(\"cl100k_base\")\n",
+      "   ```\n",
+      "\n",
+      "3. In Node.js / JavaScript, use the tiktoken npm package the same way:\n",
+      "   ```js\n",
+      "   import { encoding_for_model } from \"tiktoken\";\n",
+      "\n",
+      "   const enc = await encoding_for_model(\"gpt-3.5-turbo\");\n",
+      "   console.log(enc.name);       // \"cl100k_base\"\n",
+      "   console.log(enc.encode(\"Hi\")); // array of token IDs\n",
+      "   ```\n",
+      "\n",
+      "Under the hood encoding_for_model knows which BPE schema (“r50k_base”, “cl100k_base”, etc.) each OpenAI model uses and returns the right tokenizer instance.\n",
+      "Byte-Pair Encoding (BPE) has become the de-facto subword tokenization method in modern language models because it strikes a practical balance between fixed, closed vocabularies (word-level tokenizers) and open, but very long sequences (character-level tokenizers).  In particular:\n",
+      "\n",
+      "1. Open-vocabulary coverage  \n",
+      "   • Learns subword units from your corpus by iteratively merging the most frequent byte (or character) pairs.  \n",
+      "   • Can represent any new or rare word as a sequence of known subwords—no “unknown token” blowups.  \n",
+      "\n",
+      "2. Compact vocabulary size  \n",
+      "   • Vocabulary sizes on the order of 20K–100K tokens capture very common words as single tokens and rare or morphologically complex words as a few subwords.  \n",
+      "   • Keeps softmax layers and embedding tables manageable in size.  \n",
+      "\n",
+      "3. Reduced data sparsity  \n",
+      "   • Shares subwords among many words (e.g. “play,” “playing,” “replay”).  \n",
+      "   • Provides better statistical estimates (fewer zero‐count tokens) and faster convergence in training.  \n",
+      "\n",
+      "4. Morphological and cross-lingual adaptability  \n",
+      "   • Naturally splits on morpheme or syllable boundaries when those are frequent in the data.  \n",
+      "   • Can be trained on multilingual corpora to share subwords across related languages.  \n",
+      "\n",
+      "5. Speed and simplicity  \n",
+      "   • Linear-time, greedy encoding of new text (just look up merges).  \n",
+      "   • Deterministic and invertible: you can reconstruct the original byte sequence exactly.\n",
+      "\n",
+      "In short, BPE tokenization gives you a small, fixed-size vocabulary that still generalizes to unseen words, reduces training and memory costs, and improves statistical efficiency—key ingredients for high-quality, scalable language models.\n"
+     ]
+    }
+   ],
+   "source": [
+    "print('# gpt‑4.1 Output')\n",
+    "for item in four_one_output:\n",
+    "    print(item.sample.output[0].content)\n",
+    "\n",
+    "print('\\n# o4-mini Output')\n",
+    "for item in o4_mini_output:\n",
+    "    print(item.sample.output[0].content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "924619e0",
+   "metadata": {},
+   "source": [
+    "In this notebook, we demonstrated a sample workflow for evaluating the ability of LLMs to answer technical questions about the `tiktoken` repository using the OpenAI Evals framework leveraging MCP tooling.\n",
+    "\n",
+    "**Key points covered:**\n",
+    "- Defined a focused, custom dataset for evaluation.\n",
+    "- Configured LLM-based and Python-based graders for robust assessment.\n",
+    "- Compared two models (`gpt-4.1` and `o4-mini`) in a reproducible and transparent manner.\n",
+    "- Retrieved and displayed model outputs for automated/manual inspection.\n",
+    "\n",
+    "**Next steps:**\n",
+    "- **Expand the dataset:** Add more diverse and challenging questions to better assess model capabilities.\n",
+    "- **Analyze results:** Summarize pass/fail rates, visualize performance, or perform error analysis to identify strengths and weaknesses.\n",
+    "- **Experiment with models/tools:** Try additional models, adjust tool configurations, or test on other repositories.\n",
+    "- **Automate reporting:** Generate summary tables or plots for easier sharing and decision-making.\n",
+    "\n",
+    "For more information, check out the [OpenAI Evals documentation](https://platform.openai.com/docs/guides/evals)."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/examples/evaluation/use-cases/structured-outputs-evaluation.ipynb b/examples/evaluation/use-cases/structured-outputs-evaluation.ipynb
new file mode 100644
index 0000000000..d255fe79aa
--- /dev/null
+++ b/examples/evaluation/use-cases/structured-outputs-evaluation.ipynb
@@ -0,0 +1,335 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "9dd88e7c",
+   "metadata": {},
+   "source": [
+    "# Evaluating Code Quality Extraction with a Custom Dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "64bf0667",
+   "metadata": {},
+   "source": [
+    "This notebook demonstrates how to evaluate a model's ability to extract symbols from code using the OpenAI **Evals** framework with a custom in-memory dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "eacc6ac7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import time\n",
+    "import openai\n",
+    "\n",
+    "client = openai.OpenAI(\n",
+    "    api_key=os.getenv(\"OPENAI_API_KEY\") or os.getenv(\"_OPENAI_API_KEY\"),\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "b272e193",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_dataset(limit=None):\n",
+    "    openai_sdk_file_path = os.path.dirname(openai.__file__)\n",
+    "\n",
+    "    file_paths = [\n",
+    "        os.path.join(openai_sdk_file_path, \"resources\", \"evals\", \"evals.py\"),\n",
+    "        os.path.join(openai_sdk_file_path, \"resources\", \"responses\", \"responses.py\"),\n",
+    "        os.path.join(openai_sdk_file_path, \"resources\", \"images.py\"),\n",
+    "        os.path.join(openai_sdk_file_path, \"resources\", \"embeddings.py\"),\n",
+    "        os.path.join(openai_sdk_file_path, \"resources\", \"files.py\"),\n",
+    "    ]\n",
+    "\n",
+    "    items = []\n",
+    "    for file_path in file_paths:\n",
+    "        items.append({\"input\": open(file_path, \"r\").read()})\n",
+    "    if limit:\n",
+    "        return items[:limit]\n",
+    "    return items\n",
+    "\n",
+    "\n",
+    "structured_output_grader = \"\"\"\n",
+    "You are a helpful assistant that grades the quality of extracted information from a code file.\n",
+    "You will be given a code file and a list of extracted information.\n",
+    "You should grade the quality of the extracted information.\n",
+    "\n",
+    "You should grade the quality on a scale of 1 to 7.\n",
+    "You should apply the following criteria, and calculate your score as follows:\n",
+    "You should first check for completeness on a scale of 1 to 7.\n",
+    "Then you should apply a quality modifier.\n",
+    "\n",
+    "The quality modifier is a multiplier from 0 to 1 that you multiply by the completeness score.\n",
+    "If there is 100% coverage for completion and it is all high quality, then you would return 7*1.\n",
+    "If there is 100% coverage for completion but it is all low quality, then you would return 7*0.5.\n",
+    "etc.\n",
+    "\"\"\"\n",
+    "\n",
+    "structured_output_grader_user_prompt = \"\"\"\n",
+    "<Code File>\n",
+    "{{item.input}}\n",
+    "</Code File>\n",
+    "\n",
+    "<Extracted Information>\n",
+    "{{sample.output_json.symbols}}\n",
+    "</Extracted Information>\n",
+    "\"\"\"\n",
+    "\n",
+    "logs_eval = client.evals.create(\n",
+    "    name=\"Code QA Eval\",\n",
+    "    data_source_config={\n",
+    "        \"type\": \"custom\",\n",
+    "        \"item_schema\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\"input\": {\"type\": \"string\"}},\n",
+    "        },\n",
+    "        \"include_sample_schema\": True,\n",
+    "    },\n",
+    "    testing_criteria=[\n",
+    "        {\n",
+    "            \"type\": \"score_model\",\n",
+    "            \"name\": \"General Evaluator\",\n",
+    "            \"model\": \"o3\",\n",
+    "            \"input\": [\n",
+    "                {\"role\": \"system\", \"content\": structured_output_grader},\n",
+    "                {\"role\": \"user\", \"content\": structured_output_grader_user_prompt},\n",
+    "            ],\n",
+    "            \"range\": [1, 7],\n",
+    "            \"pass_threshold\": 5.5,\n",
+    "        }\n",
+    "    ],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "18f357e6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "gpt_4one_completions_run = client.evals.runs.create(\n",
+    "    name=\"gpt-4.1\",\n",
+    "    eval_id=logs_eval.id,\n",
+    "    data_source={\n",
+    "        \"type\": \"completions\",\n",
+    "        \"source\": {\n",
+    "            \"type\": \"file_content\",\n",
+    "            \"content\": [{\"item\": item} for item in get_dataset(limit=1)],\n",
+    "        },\n",
+    "        \"input_messages\": {\n",
+    "            \"type\": \"template\",\n",
+    "            \"template\": [\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": {\"type\": \"input_text\", \"text\": \"You are a helpful assistant.\"},\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"user\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"Extract the symbols from the code file {{item.input}}\",\n",
+    "                    },\n",
+    "                },\n",
+    "            ],\n",
+    "        },\n",
+    "        \"model\": \"gpt-4.1\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"seed\": 42,\n",
+    "            \"temperature\": 0.7,\n",
+    "            \"max_completions_tokens\": 10000,\n",
+    "            \"top_p\": 0.9,\n",
+    "            \"response_format\": {\n",
+    "                \"type\": \"json_schema\",\n",
+    "                \"json_schema\": {\n",
+    "                    \"name\": \"python_symbols\",\n",
+    "                    \"schema\": {\n",
+    "                        \"type\": \"object\",\n",
+    "                        \"properties\": {\n",
+    "                            \"symbols\": {\n",
+    "                                \"type\": \"array\",\n",
+    "                                \"description\": \"A list of symbols extracted from Python code.\",\n",
+    "                                \"items\": {\n",
+    "                                    \"type\": \"object\",\n",
+    "                                    \"properties\": {\n",
+    "                                        \"name\": {\"type\": \"string\", \"description\": \"The name of the symbol.\"},\n",
+    "                                        \"symbol_type\": {\n",
+    "                                            \"type\": \"string\", \"description\": \"The type of the symbol, e.g., variable, function, class.\",\n",
+    "                                        },\n",
+    "                                    },\n",
+    "                                    \"required\": [\"name\", \"symbol_type\"],\n",
+    "                                    \"additionalProperties\": False,\n",
+    "                                },\n",
+    "                            }\n",
+    "                        },\n",
+    "                        \"required\": [\"symbols\"],\n",
+    "                        \"additionalProperties\": False,\n",
+    "                    },\n",
+    "                    \"strict\": True,\n",
+    "                },\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "gpt_4one_responses_run = client.evals.runs.create(\n",
+    "    name=\"gpt-4.1-mini\",\n",
+    "    eval_id=logs_eval.id,\n",
+    "    data_source={\n",
+    "        \"type\": \"responses\",\n",
+    "        \"source\": {\n",
+    "            \"type\": \"file_content\",\n",
+    "            \"content\": [{\"item\": item} for item in get_dataset(limit=1)],\n",
+    "        },\n",
+    "        \"input_messages\": {\n",
+    "            \"type\": \"template\",\n",
+    "            \"template\": [\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": {\"type\": \"input_text\", \"text\": \"You are a helpful assistant.\"},\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"user\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"Extract the symbols from the code file {{item.input}}\",\n",
+    "                    },\n",
+    "                },\n",
+    "            ],\n",
+    "        },\n",
+    "        \"model\": \"gpt-4.1-mini\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"seed\": 42,\n",
+    "            \"temperature\": 0.7,\n",
+    "            \"max_completions_tokens\": 10000,\n",
+    "            \"top_p\": 0.9,\n",
+    "            \"text\": {\n",
+    "                \"format\": {\n",
+    "                    \"type\": \"json_schema\",\n",
+    "                    \"name\": \"python_symbols\",\n",
+    "                    \"schema\": {\n",
+    "                        \"type\": \"object\",\n",
+    "                        \"properties\": {\n",
+    "                            \"symbols\": {\n",
+    "                                \"type\": \"array\",\n",
+    "                                \"description\": \"A list of symbols extracted from Python code.\",\n",
+    "                                \"items\": {\n",
+    "                                    \"type\": \"object\",\n",
+    "                                    \"properties\": {\n",
+    "                                        \"name\": {\"type\": \"string\", \"description\": \"The name of the symbol.\"},\n",
+    "                                        \"symbol_type\": {\n",
+    "                                            \"type\": \"string\",\n",
+    "                                            \"description\": \"The type of the symbol, e.g., variable, function, class.\",\n",
+    "                                        },\n",
+    "                                    },\n",
+    "                                    \"required\": [\"name\", \"symbol_type\"],\n",
+    "                                    \"additionalProperties\": False,\n",
+    "                                },\n",
+    "                            }\n",
+    "                        },\n",
+    "                        \"required\": [\"symbols\"],\n",
+    "                        \"additionalProperties\": False,\n",
+    "                    },\n",
+    "                    \"strict\": True,\n",
+    "                },\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cbc4f775",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def poll_runs(eval_id, run_ids):\n",
+    "    while True:\n",
+    "        runs = [client.evals.runs.retrieve(rid, eval_id=eval_id) for rid in run_ids]\n",
+    "        for run in runs:\n",
+    "            print(run.id, run.status, run.result_counts)\n",
+    "        if all(run.status in {\"completed\", \"failed\"} for run in runs):\n",
+    "            # dump results to file\n",
+    "            for run in runs:\n",
+    "                with open(f\"{run.id}.json\", \"w\") as f:\n",
+    "                    f.write(\n",
+    "                        client.evals.runs.output_items.list(\n",
+    "                            run_id=run.id, eval_id=eval_id\n",
+    "                        ).model_dump_json(indent=4)\n",
+    "                    )\n",
+    "            break\n",
+    "        time.sleep(5)\n",
+    "\n",
+    "poll_runs(logs_eval.id, [gpt_4one_completions_run.id, gpt_4one_responses_run.id])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "c316e6eb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "completions_output = client.evals.runs.output_items.list(\n",
+    "    run_id=gpt_4one_completions_run.id, eval_id=logs_eval.id\n",
+    ")\n",
+    "\n",
+    "responses_output = client.evals.runs.output_items.list(\n",
+    "    run_id=gpt_4one_responses_run.id, eval_id=logs_eval.id\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9f1b502e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print('# Completions Output')\n",
+    "for item in completions_output:\n",
+    "    print(item)\n",
+    "\n",
+    "print('\\n# Responses Output')\n",
+    "for item in responses_output:\n",
+    "    print(item)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "openai",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/examples/evaluation/use-cases/tools-evaluation.ipynb b/examples/evaluation/use-cases/tools-evaluation.ipynb
new file mode 100644
index 0000000000..cd5c72b52e
--- /dev/null
+++ b/examples/evaluation/use-cases/tools-evaluation.ipynb
@@ -0,0 +1,268 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Evaluating Code Symbol Extraction Quality with a Custom Dataset"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "This notebook demonstrates how to evaluate a model's ability to extract symbols from code files using the OpenAI **Evals** framework with a custom in-memory dataset."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "evalrun_68437e5370c481919a6874594ca177d9 queued ResultCounts(errored=0, failed=0, passed=0, total=0)\n",
+            "evalrun_68437e544fe881918f76dbd8dce3fd15 queued ResultCounts(errored=0, failed=0, passed=0, total=0)\n",
+            "evalrun_68437e5370c481919a6874594ca177d9 in_progress ResultCounts(errored=0, failed=0, passed=0, total=0)\n",
+            "evalrun_68437e544fe881918f76dbd8dce3fd15 in_progress ResultCounts(errored=0, failed=0, passed=0, total=0)\n",
+            "evalrun_68437e5370c481919a6874594ca177d9 in_progress ResultCounts(errored=0, failed=0, passed=0, total=0)\n",
+            "evalrun_68437e544fe881918f76dbd8dce3fd15 in_progress ResultCounts(errored=0, failed=0, passed=0, total=0)\n",
+            "evalrun_68437e5370c481919a6874594ca177d9 in_progress ResultCounts(errored=0, failed=0, passed=0, total=0)\n",
+            "evalrun_68437e544fe881918f76dbd8dce3fd15 completed ResultCounts(errored=0, failed=0, passed=1, total=1)\n",
+            "evalrun_68437e5370c481919a6874594ca177d9 in_progress ResultCounts(errored=0, failed=0, passed=0, total=0)\n",
+            "evalrun_68437e544fe881918f76dbd8dce3fd15 completed ResultCounts(errored=0, failed=0, passed=1, total=1)\n",
+            "evalrun_68437e5370c481919a6874594ca177d9 completed ResultCounts(errored=0, failed=1, passed=0, total=1)\n",
+            "evalrun_68437e544fe881918f76dbd8dce3fd15 completed ResultCounts(errored=0, failed=0, passed=1, total=1)\n"
+          ]
+        }
+      ],
+      "source": [
+        "import os\n",
+        "import time\n",
+        "\n",
+        "import openai\n",
+        "\n",
+        "client = openai.OpenAI(\n",
+        "    api_key=os.getenv(\"OPENAI_API_KEY\") or os.getenv(\"_OPENAI_API_KEY\"),\n",
+        ")\n",
+        "\n",
+        "\n",
+        "def get_dataset(limit=None):\n",
+        "    openai_sdk_file_path = os.path.dirname(openai.__file__)\n",
+        "\n",
+        "    file_paths = [\n",
+        "        os.path.join(openai_sdk_file_path, \"resources\", \"evals\", \"evals.py\"),\n",
+        "        os.path.join(openai_sdk_file_path, \"resources\", \"responses\", \"responses.py\"),\n",
+        "        os.path.join(openai_sdk_file_path, \"resources\", \"images.py\"),\n",
+        "        os.path.join(openai_sdk_file_path, \"resources\", \"embeddings.py\"),\n",
+        "        os.path.join(openai_sdk_file_path, \"resources\", \"files.py\"),\n",
+        "    ]\n",
+        "\n",
+        "    items = []\n",
+        "    for file_path in file_paths:\n",
+        "        items.append({\"input\": open(file_path, \"r\").read()})\n",
+        "    if limit:\n",
+        "        return items[:limit]\n",
+        "    return items\n",
+        "\n",
+        "\n",
+        "structured_output_grader = \"\"\"\n",
+        "You are a helpful assistant that grades the quality of extracted information from a code file.\n",
+        "You will be given a code file and a list of extracted information.\n",
+        "You should grade the quality of the extracted information.\n",
+        "\n",
+        "You should grade the quality on a scale of 1 to 7.\n",
+        "You should apply the following criteria, and calculate your score as follows:\n",
+        "You should first check for completeness on a scale of 1 to 7.\n",
+        "Then you should apply a quality modifier.\n",
+        "\n",
+        "The quality modifier is a multiplier from 0 to 1 that you multiply by the completeness score.\n",
+        "If there is 100% coverage for completion and it is all high quality, then you would return 7*1.\n",
+        "If there is 100% coverage for completion but it is all low quality, then you would return 7*0.5.\n",
+        "etc.\n",
+        "\"\"\"\n",
+        "\n",
+        "structured_output_grader_user_prompt = \"\"\"\n",
+        "<Code File>\n",
+        "{{item.input}}\n",
+        "</Code File>\n",
+        "\n",
+        "<Extracted Information>\n",
+        "{{sample.output_tools[0].function.arguments.symbols}}\n",
+        "</Extracted Information>\n",
+        "\"\"\"\n",
+        "\n",
+        "logs_eval = client.evals.create(\n",
+        "    name=\"Code QA Eval\",\n",
+        "    data_source_config={\n",
+        "        \"type\": \"custom\",\n",
+        "        \"item_schema\": {\"type\": \"object\", \"properties\": {\"input\": {\"type\": \"string\"}}},\n",
+        "        \"include_sample_schema\": True,\n",
+        "    },\n",
+        "    testing_criteria=[\n",
+        "        {\n",
+        "            \"type\": \"score_model\",\n",
+        "            \"name\": \"General Evaluator\",\n",
+        "            \"model\": \"o3\",\n",
+        "            \"input\": [\n",
+        "                {\"role\": \"system\", \"content\": structured_output_grader},\n",
+        "                {\"role\": \"user\", \"content\": structured_output_grader_user_prompt},\n",
+        "            ],\n",
+        "            \"range\": [1, 7],\n",
+        "            \"pass_threshold\": 5.5,\n",
+        "        }\n",
+        "    ],\n",
+        ")\n",
+        "\n",
+        "symbol_tool = {\n",
+        "    \"name\": \"extract_symbols\",\n",
+        "    \"description\": \"Extract the symbols from the code file\",\n",
+        "    \"parameters\": {\n",
+        "        \"type\": \"object\",\n",
+        "        \"properties\": {\n",
+        "            \"symbols\": {\n",
+        "                \"type\": \"array\",\n",
+        "                \"description\": \"A list of symbols extracted from Python code.\",\n",
+        "                \"items\": {\n",
+        "                    \"type\": \"object\",\n",
+        "                    \"properties\": {\n",
+        "                        \"name\": {\"type\": \"string\", \"description\": \"The name of the symbol.\"},\n",
+        "                        \"symbol_type\": {\"type\": \"string\", \"description\": \"The type of the symbol, e.g., variable, function, class.\"},\n",
+        "                    },\n",
+        "                    \"required\": [\"name\", \"symbol_type\"],\n",
+        "                    \"additionalProperties\": False,\n",
+        "                },\n",
+        "            }\n",
+        "        },\n",
+        "        \"required\": [\"symbols\"],\n",
+        "        \"additionalProperties\": False,\n",
+        "    },\n",
+        "}\n",
+        "\n",
+        "gpt_4one_completions_run = client.evals.runs.create(\n",
+        "    name=\"gpt-4.1\",\n",
+        "    eval_id=logs_eval.id,\n",
+        "    data_source={\n",
+        "        \"type\": \"completions\",\n",
+        "        \"source\": {\"type\": \"file_content\", \"content\": [{\"item\": item} for item in get_dataset(limit=1)]},\n",
+        "        \"input_messages\": {\n",
+        "            \"type\": \"template\",\n",
+        "            \"template\": [\n",
+        "                {\"type\": \"message\", \"role\": \"system\", \"content\": {\"type\": \"input_text\", \"text\": \"You are a helpful assistant.\"}},\n",
+        "                {\"type\": \"message\", \"role\": \"user\", \"content\": {\"type\": \"input_text\", \"text\": \"Extract the symbols from the code file {{item.input}}\"}},\n",
+        "            ],\n",
+        "        },\n",
+        "        \"model\": \"gpt-4.1\",\n",
+        "        \"sampling_params\": {\n",
+        "            \"seed\": 42,\n",
+        "            \"temperature\": 0.7,\n",
+        "            \"max_completions_tokens\": 10000,\n",
+        "            \"top_p\": 0.9,\n",
+        "            \"tools\": [{\"type\": \"function\", \"function\": symbol_tool}],\n",
+        "        },\n",
+        "    },\n",
+        ")\n",
+        "\n",
+        "gpt_4one_responses_run = client.evals.runs.create(\n",
+        "    name=\"gpt-4.1\",\n",
+        "    eval_id=logs_eval.id,\n",
+        "    data_source={\n",
+        "        \"type\": \"responses\",\n",
+        "        \"source\": {\"type\": \"file_content\", \"content\": [{\"item\": item} for item in get_dataset(limit=1)]},\n",
+        "        \"input_messages\": {\n",
+        "            \"type\": \"template\",\n",
+        "            \"template\": [\n",
+        "                {\"type\": \"message\", \"role\": \"system\", \"content\": {\"type\": \"input_text\", \"text\": \"You are a helpful assistant.\"}},\n",
+        "                {\"type\": \"message\", \"role\": \"user\", \"content\": {\"type\": \"input_text\", \"text\": \"Extract the symbols from the code file {{item.input}}\"}},\n",
+        "            ],\n",
+        "        },\n",
+        "        \"model\": \"gpt-4.1\",\n",
+        "        \"sampling_params\": {\n",
+        "            \"seed\": 42,\n",
+        "            \"temperature\": 0.7,\n",
+        "            \"max_completions_tokens\": 10000,\n",
+        "            \"top_p\": 0.9,\n",
+        "            \"tools\": [{\"type\": \"function\", **symbol_tool}],\n",
+        "        },\n",
+        "    },\n",
+        ")\n",
+        "\n",
+        "\n",
+        "def poll_runs(eval_id, run_ids):\n",
+        "    # poll both runs at the same time, until they are complete or failed\n",
+        "    while True:\n",
+        "        runs = [client.evals.runs.retrieve(run_id, eval_id=eval_id) for run_id in run_ids]\n",
+        "        for run in runs:\n",
+        "            print(run.id, run.status, run.result_counts)\n",
+        "        if all(run.status in (\"completed\", \"failed\") for run in runs):\n",
+        "            break\n",
+        "        time.sleep(5)\n",
+        "\n",
+        "\n",
+        "poll_runs(logs_eval.id, [gpt_4one_completions_run.id, gpt_4one_responses_run.id])\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 2,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "completions_output = client.evals.runs.output_items.list(\n",
+        "    run_id=gpt_4one_completions_run.id, eval_id=logs_eval.id\n",
+        ")\n",
+        "\n",
+        "responses_output = client.evals.runs.output_items.list(\n",
+        "    run_id=gpt_4one_responses_run.id, eval_id=logs_eval.id\n",
+        ")\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 13,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "[{'name': 'Evals', 'symbol_type': 'class'}, {'name': 'AsyncEvals', 'symbol_type': 'class'}, {'name': 'EvalsWithRawResponse', 'symbol_type': 'class'}, {'name': 'AsyncEvalsWithRawResponse', 'symbol_type': 'class'}, {'name': 'EvalsWithStreamingResponse', 'symbol_type': 'class'}, {'name': 'AsyncEvalsWithStreamingResponse', 'symbol_type': 'class'}, {'name': '__all__', 'symbol_type': 'variable'}, {'name': 'Evals.runs', 'symbol_type': 'function'}, {'name': 'Evals.with_raw_response', 'symbol_type': 'function'}, {'name': 'Evals.with_streaming_response', 'symbol_type': 'function'}, {'name': 'Evals.create', 'symbol_type': 'function'}, {'name': 'Evals.retrieve', 'symbol_type': 'function'}, {'name': 'Evals.update', 'symbol_type': 'function'}, {'name': 'Evals.list', 'symbol_type': 'function'}, {'name': 'Evals.delete', 'symbol_type': 'function'}, {'name': 'AsyncEvals.runs', 'symbol_type': 'function'}, {'name': 'AsyncEvals.with_raw_response', 'symbol_type': 'function'}, {'name': 'AsyncEvals.with_streaming_response', 'symbol_type': 'function'}, {'name': 'AsyncEvals.create', 'symbol_type': 'function'}, {'name': 'AsyncEvals.retrieve', 'symbol_type': 'function'}, {'name': 'AsyncEvals.update', 'symbol_type': 'function'}, {'name': 'AsyncEvals.list', 'symbol_type': 'function'}, {'name': 'AsyncEvals.delete', 'symbol_type': 'function'}, {'name': 'EvalsWithRawResponse.__init__', 'symbol_type': 'function'}, {'name': 'EvalsWithRawResponse.runs', 'symbol_type': 'function'}, {'name': 'AsyncEvalsWithRawResponse.__init__', 'symbol_type': 'function'}, {'name': 'AsyncEvalsWithRawResponse.runs', 'symbol_type': 'function'}, {'name': 'EvalsWithStreamingResponse.__init__', 'symbol_type': 'function'}, {'name': 'EvalsWithStreamingResponse.runs', 'symbol_type': 'function'}, {'name': 'AsyncEvalsWithStreamingResponse.__init__', 'symbol_type': 'function'}, {'name': 'AsyncEvalsWithStreamingResponse.runs', 'symbol_type': 'function'}]\n",
+            "[{'name': 'Evals', 'symbol_type': 'class'}, {'name': 'AsyncEvals', 'symbol_type': 'class'}, {'name': 'EvalsWithRawResponse', 'symbol_type': 'class'}, {'name': 'AsyncEvalsWithRawResponse', 'symbol_type': 'class'}, {'name': 'EvalsWithStreamingResponse', 'symbol_type': 'class'}, {'name': 'AsyncEvalsWithStreamingResponse', 'symbol_type': 'class'}, {'name': '__all__', 'symbol_type': 'variable'}]\n"
+          ]
+        }
+      ],
+      "source": [
+        "import json\n",
+        "\n",
+        "for item in completions_output:\n",
+        "    print(json.loads(item.sample.output[0].tool_calls[0][\"function\"][\"arguments\"])[\"symbols\"])\n",
+        "\n",
+        "for item in responses_output:\n",
+        "    print(json.loads(item.sample.output[0].tool_calls[0][\"function\"][\"arguments\"])[\"symbols\"])\n"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "openai",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.12.9"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 2
+}
diff --git a/examples/evaluation/use-cases/web-search-evaluation.ipynb b/examples/evaluation/use-cases/web-search-evaluation.ipynb
new file mode 100644
index 0000000000..fc0546b4fc
--- /dev/null
+++ b/examples/evaluation/use-cases/web-search-evaluation.ipynb
@@ -0,0 +1,581 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Evaluating Web Search Quality with a Custom Dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This notebook demonstrates how to evaluate a model's ability to retrieve correct answers from the web using the OpenAI **Evals** framework with a custom in-memory dataset.\n",
+    "\n",
+    "**Goals:**\n",
+    "- Show how to set up and run an evaluation for web search quality.\n",
+    "- Provide a template for evaluating information retrieval capabilities of LLMs.\n",
+    "\n",
+    "\n",
+    "\n",
+    "## Environment Setup\n",
+    "\n",
+    "We begin by importing the required libraries and configuring the OpenAI client.  \n",
+    "This ensures we have access to the OpenAI API and all necessary utilities for evaluation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n",
+      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
+      "Note: you may need to restart the kernel to use updated packages.\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Update OpenAI client\n",
+    "%pip install --upgrade openai --quiet"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import time\n",
+    "import pandas as pd\n",
+    "from IPython.display import display\n",
+    "\n",
+    "from openai import OpenAI\n",
+    "\n",
+    "client = OpenAI(\n",
+    "    api_key=os.getenv(\"OPENAI_API_KEY\") or os.getenv(\"_OPENAI_API_KEY\"),\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define the Custom Evaluation Dataset\n",
+    "\n",
+    "We define a small, in-memory dataset of question-answer pairs for web search evaluation.  \n",
+    "Each item contains a `query` (the user's search prompt) and an `answer` (the expected ground truth).\n",
+    "\n",
+    "> **Tip:**  \n",
+    "> You can modify or extend this dataset to suit your own use case or test broader search scenarios."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_dataset(limit=None):\n",
+    "    dataset = [\n",
+    "        {\n",
+    "            \"query\": \"coolest person in the world, the 100m dash at the 2008 olympics was the best sports event of all time\",\n",
+    "            \"answer\": \"usain bolt\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"best library in the world, there is nothing better than a dataframe\",\n",
+    "            \"answer\": \"pandas\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"most fun place to visit, I am obsessed with the Philbrook Museum of Art\",\n",
+    "            \"answer\": \"tulsa, oklahoma\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"who created the python programming language, beloved by data scientists everywhere\",\n",
+    "            \"answer\": \"guido van rossum\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"greatest chess player in history, famous for the 1972 world championship\",\n",
+    "            \"answer\": \"bobby fischer\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"the city of lights, home to the eiffel tower and louvre museum\",\n",
+    "            \"answer\": \"paris\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"most popular search engine, whose name is now a verb\",\n",
+    "            \"answer\": \"google\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"the first man to walk on the moon, giant leap for mankind\",\n",
+    "            \"answer\": \"neil armstrong\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"groundbreaking electric car company founded by elon musk\",\n",
+    "            \"answer\": \"tesla\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"query\": \"founder of microsoft, philanthropist and software pioneer\",\n",
+    "            \"answer\": \"bill gates\",\n",
+    "        },\n",
+    "    ]\n",
+    "    return dataset[:limit] if limit else dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define Grading Logic\n",
+    "\n",
+    "To evaluate the model’s answers, we use an LLM-based pass/fail grader:\n",
+    "\n",
+    "- **Pass/Fail Grader:**  \n",
+    "  An LLM-based grader that checks if the model’s answer (from web search) matches the expected answer (ground truth) or contains the correct information.\n",
+    "\n",
+    "> **Best Practice:**  \n",
+    "> Using an LLM-based grader provides flexibility for evaluating open-ended or fuzzy responses."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pass_fail_grader = \"\"\"\n",
+    "You are a helpful assistant that grades the quality of a web search.\n",
+    "You will be given a query and an answer.\n",
+    "You should grade the quality of the web search.\n",
+    "\n",
+    "You should either say \"pass\" or \"fail\", if the query contains the answer.\n",
+    "\n",
+    "\"\"\"\n",
+    "\n",
+    "pass_fail_grader_user_prompt = \"\"\"\n",
+    "<Query>\n",
+    "{{item.query}}\n",
+    "</Query>\n",
+    "\n",
+    "<Web Search Result>\n",
+    "{{sample.output_text}}\n",
+    "</Web Search Result>\n",
+    "\n",
+    "<Ground Truth>\n",
+    "{{item.answer}}\n",
+    "</Ground Truth>\n",
+    "\"\"\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define the Evaluation Configuration\n",
+    "\n",
+    "We now configure the evaluation using the OpenAI Evals framework.  \n",
+    "\n",
+    "This step specifies:\n",
+    "- The evaluation name and dataset.\n",
+    "- The schema for each item (what fields are present in each Q&A pair).\n",
+    "- The grader(s) to use (LLM-based pass/fail).\n",
+    "- The passing criteria and labels.\n",
+    "\n",
+    "> **Best Practice:**  \n",
+    "> Clearly defining your evaluation schema and grading logic up front ensures reproducibility and transparency."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create the evaluation definition using the OpenAI Evals client.\n",
+    "logs_eval = client.evals.create(\n",
+    "    name=\"Web-Search Eval\",\n",
+    "    data_source_config={\n",
+    "        \"type\": \"custom\",\n",
+    "        \"item_schema\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"query\": {\"type\": \"string\"},\n",
+    "                \"answer\": {\"type\": \"string\"},\n",
+    "            },\n",
+    "        },\n",
+    "        \"include_sample_schema\": True,\n",
+    "    },\n",
+    "    testing_criteria=[\n",
+    "        {\n",
+    "            \"type\": \"label_model\",\n",
+    "            \"name\": \"Web Search Evaluator\",\n",
+    "            \"model\": \"o3\",\n",
+    "            \"input\": [\n",
+    "                {\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": pass_fail_grader,\n",
+    "                },\n",
+    "                {\n",
+    "                    \"role\": \"user\",\n",
+    "                    \"content\": pass_fail_grader_user_prompt,\n",
+    "                },\n",
+    "            ],\n",
+    "            \"passing_labels\": [\"pass\"],\n",
+    "            \"labels\": [\"pass\", \"fail\"],\n",
+    "        }\n",
+    "    ],\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Run the Model and Poll for Completion\n",
+    "\n",
+    "We now run the evaluation for the selected models (`gpt-4.1` and `gpt-4.1-mini`).  \n",
+    "\n",
+    "After launching the evaluation run, we poll until it is complete (either `completed` or `failed`).\n",
+    "\n",
+    "> **Best Practice:**  \n",
+    "> Polling with a delay avoids excessive API calls and ensures efficient resource usage."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Launch the evaluation run for gpt-4.1 using web search\n",
+    "gpt_4one_responses_run = client.evals.runs.create(\n",
+    "    name=\"gpt-4.1\",\n",
+    "    eval_id=logs_eval.id,\n",
+    "    data_source={\n",
+    "        \"type\": \"responses\",\n",
+    "        \"source\": {\n",
+    "            \"type\": \"file_content\",\n",
+    "            \"content\": [{\"item\": item} for item in get_dataset()],\n",
+    "        },\n",
+    "        \"input_messages\": {\n",
+    "            \"type\": \"template\",\n",
+    "            \"template\": [\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"You are a helpful assistant that searches the web and gives contextually relevant answers.\",\n",
+    "                    },\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"user\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"Search the web for the answer to the query {{item.query}}\",\n",
+    "                    },\n",
+    "                },\n",
+    "            ],\n",
+    "        },\n",
+    "        \"model\": \"gpt-4.1\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"seed\": 42,\n",
+    "            \"temperature\": 0.7,\n",
+    "            \"max_completions_tokens\": 10000,\n",
+    "            \"top_p\": 0.9,\n",
+    "            \"tools\": [{\"type\": \"web_search_preview\"}],\n",
+    "        },\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Launch the evaluation run for gpt-4.1-mini using web search\n",
+    "gpt_4one_mini_responses_run = client.evals.runs.create(\n",
+    "    name=\"gpt-4.1-mini\",\n",
+    "    eval_id=logs_eval.id,\n",
+    "    data_source={\n",
+    "        \"type\": \"responses\",\n",
+    "        \"source\": {\n",
+    "            \"type\": \"file_content\",\n",
+    "            \"content\": [{\"item\": item} for item in get_dataset()],\n",
+    "        },\n",
+    "        \"input_messages\": {\n",
+    "            \"type\": \"template\",\n",
+    "            \"template\": [\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"system\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"You are a helpful assistant that searches the web and gives contextually relevant answers.\",\n",
+    "                    },\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"message\",\n",
+    "                    \"role\": \"user\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"input_text\",\n",
+    "                        \"text\": \"Search the web for the answer to the query {{item.query}}\",\n",
+    "                    },\n",
+    "                },\n",
+    "            ],\n",
+    "        },\n",
+    "        \"model\": \"gpt-4.1-mini\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"seed\": 42,\n",
+    "            \"temperature\": 0.7,\n",
+    "            \"max_completions_tokens\": 10000,\n",
+    "            \"top_p\": 0.9,\n",
+    "            \"tools\": [{\"type\": \"web_search_preview\"}],\n",
+    "        },\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "evalrun_68477e0f56a481919eea5e7d8a04225e completed ResultCounts(errored=0, failed=1, passed=9, total=10)\n",
+      "evalrun_68477e712bb48191bc7368b084f8c52c completed ResultCounts(errored=0, failed=0, passed=10, total=10)\n"
+     ]
+    }
+   ],
+   "source": [
+    "# poll both runs at the same time, until they are complete or failed\n",
+    "def poll_runs(eval_id, run_ids):\n",
+    "    while True:\n",
+    "        runs = [client.evals.runs.retrieve(run_id, eval_id=eval_id) for run_id in run_ids]\n",
+    "        for run in runs:\n",
+    "            print(run.id, run.status, run.result_counts)\n",
+    "        if all(run.status in {\"completed\", \"failed\"} for run in runs):\n",
+    "            break\n",
+    "        time.sleep(5)\n",
+    "\n",
+    "# Start polling the run until completion\n",
+    "poll_runs(logs_eval.id, [gpt_4one_responses_run.id, gpt_4one_mini_responses_run.id])\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Display and Interpret Model Outputs\n",
+    "\n",
+    "Finally, we display the outputs from the model for manual inspection and further analysis.\n",
+    "\n",
+    "- Each answer is printed for each query in the dataset.\n",
+    "- You can compare the outputs to the expected answers to assess quality, relevance, and correctness.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>GPT-4.1 Output</th>\n",
+       "      <th>GPT-4.1-mini Output</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>If you're captivated by the Philbrook Museum o...</td>\n",
+       "      <td>Bobby Fischer is widely regarded as one of the...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>\\n## [Paris, France](https://www.google.com/ma...</td>\n",
+       "      <td>The 2008 Olympic 100m dash is widely regarded ...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>Bill Gates, born on October 28, 1955, in Seatt...</td>\n",
+       "      <td>If you're looking for fun places to visit in T...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>Usain Bolt's performance in the 100-meter fina...</td>\n",
+       "      <td>On July 20, 1969, astronaut Neil Armstrong bec...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>It seems you're interested in both the world's...</td>\n",
+       "      <td>Bill Gates is a renowned software pioneer, phi...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>Neil Armstrong was the first person to walk on...</td>\n",
+       "      <td>Your statement, \"there is nothing better than ...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>Tesla, Inc. is an American electric vehicle an...</td>\n",
+       "      <td>The search engine whose name has become synony...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td>Bobby Fischer, widely regarded as one of the g...</td>\n",
+       "      <td>\\n## [Paris, France](https://www.google.com/ma...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
+       "      <td>Guido van Rossum, a Dutch programmer born on J...</td>\n",
+       "      <td>Guido van Rossum, a Dutch programmer born on J...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>9</th>\n",
+       "      <td>The most popular search engine whose name has ...</td>\n",
+       "      <td>Elon Musk is the CEO and largest shareholder o...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                      GPT-4.1 Output  \\\n",
+       "0  If you're captivated by the Philbrook Museum o...   \n",
+       "1  \\n## [Paris, France](https://www.google.com/ma...   \n",
+       "2  Bill Gates, born on October 28, 1955, in Seatt...   \n",
+       "3  Usain Bolt's performance in the 100-meter fina...   \n",
+       "4  It seems you're interested in both the world's...   \n",
+       "5  Neil Armstrong was the first person to walk on...   \n",
+       "6  Tesla, Inc. is an American electric vehicle an...   \n",
+       "7  Bobby Fischer, widely regarded as one of the g...   \n",
+       "8  Guido van Rossum, a Dutch programmer born on J...   \n",
+       "9  The most popular search engine whose name has ...   \n",
+       "\n",
+       "                                 GPT-4.1-mini Output  \n",
+       "0  Bobby Fischer is widely regarded as one of the...  \n",
+       "1  The 2008 Olympic 100m dash is widely regarded ...  \n",
+       "2  If you're looking for fun places to visit in T...  \n",
+       "3  On July 20, 1969, astronaut Neil Armstrong bec...  \n",
+       "4  Bill Gates is a renowned software pioneer, phi...  \n",
+       "5  Your statement, \"there is nothing better than ...  \n",
+       "6  The search engine whose name has become synony...  \n",
+       "7  \\n## [Paris, France](https://www.google.com/ma...  \n",
+       "8  Guido van Rossum, a Dutch programmer born on J...  \n",
+       "9  Elon Musk is the CEO and largest shareholder o...  "
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# Retrieve output items for the 4.1 model after completion\n",
+    "four_one = client.evals.runs.output_items.list(\n",
+    "    run_id=gpt_4one_responses_run.id, eval_id=logs_eval.id\n",
+    ")\n",
+    "\n",
+    "# Retrieve output items for the 4.1-mini model after completion\n",
+    "four_one_mini = client.evals.runs.output_items.list(\n",
+    "    run_id=gpt_4one_mini_responses_run.id, eval_id=logs_eval.id\n",
+    ")\n",
+    "\n",
+    "# Collect outputs for both models\n",
+    "four_one_outputs = [item.sample.output[0].content for item in four_one]\n",
+    "four_one_mini_outputs = [item.sample.output[0].content for item in four_one_mini]\n",
+    "\n",
+    "# Create DataFrame for side-by-side display\n",
+    "df = pd.DataFrame({\n",
+    "    \"GPT-4.1 Output\": four_one_outputs,\n",
+    "    \"GPT-4.1-mini Output\": four_one_mini_outputs\n",
+    "})\n",
+    "\n",
+    "display(df)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can visualize the results in the evals dashboard by going to https://platform.openai.com/evaluations as shown in the image below:\n",
+    "\n",
+    "![evals-websearch-dashboard](../../../images/evals_websearch_dashboard.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In this notebook, we demonstrated a workflow for evaluating the web search capabilities of language models using the OpenAI Evals framework.\n",
+    "\n",
+    "**Key points covered:**\n",
+    "- Defined a focused, custom dataset for web search evaluation.\n",
+    "- Configured an LLM-based grader for robust assessment.\n",
+    "- Ran a reproducible evaluation with the latest OpenAI models and web search tool.\n",
+    "- Retrieved and displayed model outputs for inspection.\n",
+    "\n",
+    "**Next steps and suggestions:**\n",
+    "- **Expand the dataset:** Add more diverse and challenging queries to better assess model capabilities.\n",
+    "- **Analyze results:** Summarize pass/fail rates, visualize performance, or perform error analysis to identify strengths and weaknesses.\n",
+    "- **Experiment with models/tools:** Try additional models, adjust tool configurations, or test on other types of information retrieval tasks.\n",
+    "- **Automate reporting:** Generate summary tables or plots for easier sharing and decision-making.\n",
+    "\n",
+    "For more information, see the [OpenAI Evals documentation](https://platform.openai.com/docs/guides/evals)."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/images/evals_websearch_dashboard.png b/images/evals_websearch_dashboard.png
new file mode 100644
index 0000000000..1207b4f6f0
Binary files /dev/null and b/images/evals_websearch_dashboard.png differ
diff --git a/images/mcp_eval_data.png b/images/mcp_eval_data.png
new file mode 100644
index 0000000000..c6538edc8a
Binary files /dev/null and b/images/mcp_eval_data.png differ
diff --git a/images/mcp_eval_output.png b/images/mcp_eval_output.png
new file mode 100644
index 0000000000..f6da19732a
Binary files /dev/null and b/images/mcp_eval_output.png differ
diff --git a/registry.yaml b/registry.yaml
index 8e7102004b..26bcd7dc0b 100644
--- a/registry.yaml
+++ b/registry.yaml
@@ -4,14 +4,14 @@
 # should build pages for, and indicates metadata such as tags, creation date and
 # authors for each page.
 
-- title: Eval Driven System Design from Prototype to Production
+- title: Eval Driven System Design - From Prototype to Production
   path: examples/partners/eval_driven_system_design/receipt_inspection.ipynb
-  date: 2025-06-01
+  date: 2025-06-02
   authors:
     - shikhar-cyber
-    - moredatarequired
-    - tooluser
-    - eddiesiegel
+    - Hugh Wimberly
+    - Joshua Marker
+    - Eddie Siegel
   tags:
     - evals
     - API Flywheel
@@ -2129,3 +2129,46 @@
     - evals
     - stripe
     - conversion
+
+- title: Evals API Use-case - MCP Evaluation
+  path: examples/evaluation/use-cases/mcp_eval_notebook.ipynb
+  date: 2025-06-09
+  authors:
+    - josiah-openai
+    - shikhar-cyber
+  tags:
+    - evals-api
+    - responses
+    - evals
+    - mcp
+
+- title: Evals API Use-case - Structured Outputs Evaluation
+  path: examples/evaluation/use-cases/structured-outputs-evaluation.ipynb
+  date: 2025-06-09
+  authors:
+    - josiah-openai
+  tags:
+    - evals-api
+    - responses
+    - evals
+
+- title: Evals API Use-case - Tools Evaluation
+  path: examples/evaluation/use-cases/tools-evaluation.ipynb
+  date: 2025-06-09
+  authors:
+    - josiah-openai
+  tags:
+    - evals-api
+    - responses
+    - evals
+
+- title: Evals API Use-case - Web Search Evaluation
+  path: examples/evaluation/use-cases/web-search-evaluation.ipynb
+  date: 2025-06-09
+  authors:
+    - josiah-openai
+    - shikhar-cyber
+  tags:
+    - evals-api
+    - responses
+    - evals
