fix(otel): prevent infinite retry loops on unicode hex escape errors #2337
Conversation
|
WalkthroughThe changes update Prisma client initialization and logging in the database server module, enhancing log configuration and forwarding structured log events to a central logger with metadata. Logging behavior is now controlled by environment variables for error and verbose query logs. In the event repository, error handling for batch inserts is refactored: new helper functions classify retriable Prisma errors, extract error details, and detect Unicode-related errors. The retry logic is clarified to bisect batches on certain error types, with improved logging and span attribute reporting. The unit test workflow shard count is reduced from 10 to 8. A comprehensive integration test suite for Estimated code review effort🎯 4 (Complex) | ⏱️ ~40 minutes Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (3)
✅ Files skipped from review due to trivial changes (1)
🧰 Additional context used📓 Path-based instructions (4)**/*.{ts,tsx}📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)
Files:
**/*.test.{ts,tsx}📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)
Files:
{packages/core,apps/webapp}/**/*.{ts,tsx}📄 CodeRabbit Inference Engine (.github/copilot-instructions.md)
Files:
apps/webapp/**/*.{ts,tsx}📄 CodeRabbit Inference Engine (.cursor/rules/webapp.mdc)
Files:
🧠 Learnings (12)📓 Common learnings📚 Learning: in apps/webapp/app/services/runsrepository.server.ts, the in-memory status filtering after fetching ...Applied to files:
📚 Learning: applies to **/*.test.{ts,tsx} : our tests are all vitest...Applied to files:
📚 Learning: applies to apps/webapp/app/**/*.test.{ts,tsx} : tests in the webapp should only import classes and f...Applied to files:
📚 Learning: applies to apps/webapp/app/**/*.test.{ts,tsx} : test files in the webapp should not import `env.serv...Applied to files:
📚 Learning: do not use or add new code to the legacy run engine; focus on using and migrating to run engine 2.0 ...Applied to files:
📚 Learning: in apps/webapp/app/services/environmentmetricsrepository.server.ts, the clickhouse methods (gettaska...Applied to files:
📚 Learning: applies to **/trigger/**/*.{ts,tsx,js,jsx} : when using realtime features, use the `runs.subscribeto...Applied to files:
📚 Learning: follow our tests.md guide for how to write tests in the monorepo...Applied to files:
📚 Learning: applies to apps/webapp/app/services/**/*.server.ts : for testable services, separate service logic a...Applied to files:
📚 Learning: applies to internal-packages/database/**/*.{ts,tsx} : we use prisma in internal-packages/database fo...Applied to files:
📚 Learning: when running tests, it is often better to `cd` into the package directory and then run tests using `...Applied to files:
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (23)
🔇 Additional comments (14)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Summary
Fixed infinite retry loops in OTEL log ingestion caused by Unicode validation errors in PostgreSQL. The issue occurred when malformed Unicode data in telemetry logs triggered "lone leading surrogate in
hex escape" and "unexpected end of hex escape" errors, which were not properly handled by the DynamicFlushScheduler's retry logic.
Changes
Unicode Error Handling
PrismaClientKnownRequestErrorwith Unicode/hex escape error messagesisUnicodeError()helper function for reusable Unicode error detectiongetPrismaErrorDetails()helper for type-safe error detail extractionObservability Improvements
prisma_error_type,prisma_error_codePrisma Logging Improvements
PRISMA_LOG_TO_STDOUTenvironment variable to control error output format. If it equals "1" we log straight to stdout instead of using structured logs, this is helpful when running locally.clientTypefield to distinguish between writer and reader client logsRoot Cause
The recent DynamicFlushScheduler enhancements added retry logic that re-queued failed batches, but didn't account for Unicode serialization errors that are permanent data issues rather than transient
failures. This caused problematic batches to retry indefinitely.
Resolution
Unicode errors now trigger the same bisection strategy used for other data issues, recursively splitting batches until individual problematic events are isolated and dropped, allowing clean events to be
successfully inserted.