Observability — Senior chuqurlik

Collector nima uchun

Ilova OTLP orqali hammasini (trace, metrika, log) Collectorga yuboradi. Collector esa: batch qiladi, filtr/transform qiladi, sampling qiladi (keyingi bo‘limlar) va kerakli backend’larga yo‘naltiradi. Ilova qaysi backend borligini bilmaydi — faqat Collector’ni biladi:

# otel-collector.yaml — receivers -> processors -> exporters
receivers:
  otlp:                                     # ilovadan OTLP qabul qiladi (gRPC/HTTP)
    protocols: { grpc: {}, http: {} }
processors:
  batch: {}                                 # to'plamlab yuboradi (samaradorlik)
  memory_limiter: {}                        # collector'ni OOM'dan himoya qiladi
exporters:
  otlp/tempo:  { endpoint: "tempo:4317" }   # trace -> Tempo
  prometheus:  { endpoint: "0.0.0.0:8889" } # metrika -> Prometheus scrape
  loki:        { endpoint: "http://loki:3100/loki/api/v1/push" }  # log -> Loki
service:
  pipelines:
    traces:  { receivers: [otlp], processors: [batch], exporters: [otlp/tempo] }
    metrics: { receivers: [otlp], processors: [batch], exporters: [prometheus] }
    logs:    { receivers: [otlp], processors: [batch], exporters: [loki] }

W3C Trace Context

Standart — traceparent header. U trace ID, ota-span ID va flag’larni olib yuradi. Har xizmat kelgan traceparentni o‘qib, o‘z span’ini uning bolasi qilib qo‘shadi:

// W3C traceparent header — trace kontekstini SERVISLAR BO'YLAB uzatadi:
// traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
//              |  |________________________________|  |______________| |
//           version        trace-id (32 hex)          span-id (16 hex) flags (sampled=1)

Propagator: inject va extract

Chiquvchi so‘rovga kontekst inject qilinadi (header’ga yoziladi), kiruvchida extract qilinadi (o‘qiladi). Bu HTTP’da avtomatik. Lekin broker (Kafka/RabbitMQ — 3-mavzu) orqali o‘tganda qo‘lda uzatish kerak — aks holda trace uziladi:

// Broker bo'ylab trace: kontekstni MESSAGE HEADER'iga inject qilamiz (3-mavzu)
const headers: Record<string, string> = {};
propagation.inject(context.active(), headers);          // traceparent -> headers
await producer.send({ topic: "orders", messages: [{ value, headers }] });

// consumer tomonda: header'dan extract qilib, trace'ni DAVOM ettiramiz
const ctx = propagation.extract(context.active(), msg.headers);
tracer.startActiveSpan("process-order", { }, ctx, (span) => { /* ... */ });

Head sampling — boshda qaror

Trace boshlanganda (root’da) qaror qilinadi: masalan, "har 100 tadan 1 tasini saqla". Oddiy va arzon. Lekin muammo: qaror boshda qilingani uchun — nodir xato yoki sekin trace ham 99% ehtimol bilan tashlanadi (aynan kerak bo‘lgani!).

Tail sampling — oxirda qaror (aqlliroq)

Qaror trace tugagach, collector’da qilinadi — endi trace’ning natijasini bilamiz. Demak: barcha xatoli va sekin trace’larni saqlaymiz, qolganidan ozini:

# TAIL SAMPLING (collector'da) — trace TUGAGACH qaror qilinadi
processors:
  tail_sampling:
    policies:
      - name: keep-errors                    # BARCHA xatoli trace'larni saqla
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow                       # BARCHA sekin trace'larni saqla
        type: latency
        latency: { threshold_ms: 500 }
      - name: sample-rest                     # qolganidan faqat 5%
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Metrika dizayn qoidalari

• Label qiymatlari chegaralangan bo‘lsin: method (5 ta), status (~10 ta), route (o‘nlab) — mumkin. user_id, order_id, ip — hech qachon.
• Route’ni normallashtir: /orders/:id (shablon), /orders/123 emas.
• Yuqori kardinalli ma’lumot — metrikaga emas, log yoki tracega.

Error budget

Agar SLO = 99.9% bo‘lsa, error budget = 0.1% — ya’ni so‘rovlarning 0.1%i "buzilishga ruxsat etilgan". Bu — muhandislik uchun "byudjet": undan tez sarflasang muammo, sekin sarflasang — hammasi joyida.

Burn rate — budjet qanchalik tez yonyapti

Bitta oddiy "xato > X" alert yomon: ozgina sakrash ham chalintiradi (alert fatigue), yoki sekin oqib turgan muammoni o‘tkazib yuboradi. Multi-window multi-burn-rate yechadi: tez yonish → darhol page; sekin yonish → ticket:

# MULTI-BURN-RATE alert — budjetni TEZ yoqsa page, SEKIN yoqsa ticket
# SLO = 99.9%  ->  error budget = 0.1%
groups:
- name: slo
  rules:
  - alert: ErrorBudgetFastBurn          # 1 soatda budjetning katta qismi yonyapti
    expr: |
      error_ratio_1h > 14.4 * 0.001
      and error_ratio_5m > 14.4 * 0.001
    for: 2m
    labels: { severity: page }          # DARHOL odamni chaqir
  - alert: ErrorBudgetSlowBurn          # 6 soatda sekin yonish
    expr: error_ratio_6h > 6 * 0.001
    for: 15m
    labels: { severity: ticket }        # shoshilinch emas, ticket

Exemplar nima

Histogram bucket’iga namuna trace ID biriktiriladi. Grafana’da kechikish grafigida bir cho‘qqini ko‘rsang — o‘sha nuqtadagi exemplarni bosib, aynan o‘sha sekin so‘rovning trace’iga o‘tasan:

# Prometheus histogram + EXEMPLAR (bucket'ga namuna trace ID biriktiriladi)
http_request_duration_seconds_bucket{le="0.5"} 4283 # {trace_id="4bf92f35..."} 0.48 1620000000
#                                                     |___________________________|
#                                              exemplar: aynan shu sekin so'rovning trace'i

Log masshtabda

• Log sampling: yuqori hajmli, takroriy loglarni tanlab saqlash (masalan, muvaffaqiyatli so‘rovlardan ozini, barcha xatolarni).
• Indekslangan maydonlar vs to‘liq matn: faqat kerakli maydonlarni (trace_id, userId, status) indeksla — hammasini emas (Loki bu falsafada: yorliqlar bo‘yicha, matn bo‘yicha emas).
• Retention qatlamlari: so‘nggi loglar tez (issiq), eskilari arzon arxivda (sovuq).
• Har logda trace_id — trace bilan bog‘lash uchun (6-bo‘lim zanjiri).

Continuous profiling — to‘rtinchi ustun

Metrika/trace "qaysi xizmat/endpoint sekin"ini aytadi. Lekin "kod ichida qaysi funksiya CPU yeyapti?" — buni continuous profiling beradi: productionda doimiy, past overhead bilan CPU/xotira profillari (funksiya/qator darajasida).

To‘liq observability pipeline — yig‘ma ko‘rinish

Senior tayyorlik cheklisti