agent session monitor

AI Community

Skill notes

Overview

Real-time monitoring of Higress access logs, extracting ai_log JSON, grouping multi-turn conversations by session_id, and calculating token costs with visualization.

Core Features

Real-time Log Monitoring: Monitors Higress access log files, parses new ai_log entries in real-time Log Rotation Support: Full logrotate support, automatically tracks access.log.1~5 etc. Incremental Parsing: Inode-based tracking, processes only new content, no duplicates Session Grouping: Associates multi-turn conversations by session_id (each turn is a separate request) Complete Conversation Tracking: Records messages, question, answer, reasoning, tool_calls for each turn

Web UI features: 📊 Overview: View all session statistics and group by model 🔍 Session Details: Click session ID to drill down into complete conversation history 💬 Conversation Log: Display messages, question, answer, reasoning, tool_calls for each turn 💰 Cost Statistics: Real-time token usage and cost calculation

3. Use in Clawdbot Conversations

When users ask about current session token consumption or conversation history: Get current session_id (from runtime or context) Generate web UI URL and return to user Example response:

main.py (Background Monitor)

Parameter · Description · Required · Default --log-path · Higress access log file path · Yes · /var/log/higress/access.log --output-dir · Session data storage directory · No · ./sessions --session-key · Monitor only specified session key · No · Monitor all sessions --state-file · State file path (records read offsets) · No · <output-dir>/.state.json

Fluxly profile Runtime and permissions are inferred by Fluxly static checks, not author declarations or runtime test results.
Domain
AI
Author-declared agents
No explicit declaration found; this is not inferred or tested compatibility
Static check
88 / 100 · heuristic scan, not runtime safety proof
Runtime evidence
Not run; no isolated runtime test evidence is currently recorded
Author / version / license
@higress-group · no license declared
Token usage
Lean
Setup complexity
Guided setup
External API key
Not required
Operating systems
Unspecified (assume cross-platform)
Runtime requirements
No explicit requirements detected
Detected file/system behavior
  • Read-only
  • Write / modify
Network behavior
Local-only
Install commands
None (reference only)

Heads up: Tool permissions are not declared; actual access depends on the runtime.

Author source file SKILL.md

## Overview

Real-time monitoring of Higress access logs, extracting ai_log JSON, grouping multi-turn conversations by session_id, and calculating token costs with visualization.

### Core Features

- **Real-time Log Monitoring**: Monitors Higress access log files, parses new ai_log entries in real-time
- **Log Rotation Support**: Full logrotate support, automatically tracks access.log.1~5 etc.
- **Incremental Parsing**: Inode-based tracking, processes only new content, no duplicates
- **Session Grouping**: Associates multi-turn conversations by session_id (each turn is a separate request)
- **Complete Conversation Tracking**: Records messages, question, answer, reasoning, tool_calls for each turn
- **Token Usage Tracking**: Distinguishes input/output/reasoning/cached tokens
- **Web Visualization**: Browser-based UI with overview and session drill-down
- **Real-time URL Generation**: Clawdbot can generate observation links based on current session ID
- **Background Processing**: Independent process, continuously parses access logs
- **State Persistence**: Maintains parsing progress and session data across runs

## Usage

### 1. Background Monitoring (Continuous)

```bash
# Parse Higress access logs (with log rotation support)
python3 main.py --log-path /var/log/proxy/access.log --output-dir ./sessions

# Filter by session key
python3 main.py --log-path /var/log/proxy/access.log --session-key <session-id>

# Scheduled task (incremental parsing every minute)
* * * * * python3 /path/to/main.py --log-path /var/log/proxy/access.log --output-dir /var/lib/sessions
```

### 2. Start Web UI (Recommended)

```bash
# Start web server
python3 scripts/webserver.py --data-dir ./sessions --port 8888

# Access in browser
open http://localhost:8888
```

Web UI features:
- 📊 Overview: View all session statistics and group by model
- 🔍 Session Details: Click session ID to drill down into complete conversation history
- 💬 Conversation Log: Display messages, question, answer, reasoning, tool_calls for each turn
- 💰 Cost Statistics: Real-time token usage and cost calculation
- 🔄 Auto Refresh: Updates every 30 seconds

### 3. Use in Clawdbot Conversations

When users ask about current session token consumption or conversation history:

1. Get current session_id (from runtime or context)
2. Generate web UI URL and return to user

Example response:

```
Your current session statistics:
- Session ID: agent:main:discord:channel:1465367993012981988
- View details: http://localhost:8888/session?id=agent:main:discord:channel:1465367993012981988

Click the link to see:
✅ Complete conversation history
✅ Token usage breakdown per turn
✅ Tool call records
✅ Cost statistics
```

### 4. CLI Queries (Optional)

```bash
# View specific session details
python3 scripts/cli.py show <session-id>

# List all sessions
python3 scripts/cli.py list --sort-by cost --limit 10

# Statistics by model
python3 scripts/cli.py stats-model

# Statistics by date (last 7 days)
python3 scripts/cli.py stats-date --days 7

# Export reports
python3 scripts/cli.py export finops-report.json
```

## Configuration

### main.py (Background Monitor)

| Parameter | Description | Required | Default |
|-----------|-------------|----------|---------|
| `--log-path` | Higress access log file path | Yes | /var/log/higress/access.log |
| `--output-dir` | Session data storage directory | No | ./sessions |
| `--session-key` | Monitor only specified session key | No | Monitor all sessions |
| `--state-file` | State file path (records read offsets) | No | <output-dir>/.state.json |
| `--refresh-interval` | Log refresh interval (seconds) | No | 1 |

### webserver.py (Web UI)

| Parameter | Description | Required | Default |
|-----------|-------------|----------|---------|
| `--data-dir` | Session data directory | No | ./sessions |
| `--port` | HTTP server port | No | 8888 |
| `--host` | HTTP server address | No | 0.0.0.0 |

## Output Examples

### 1. Real-time Monitor

```
🔍 Session Monitor - Active
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📊 Active Sessions: 3

┌──────────────────────────┬─────────┬──────────┬───────────┐
│ Session ID               │ Msgs    │ Input    │ Output    │
├──────────────────────────┼─────────┼──────────┼───────────┤
│ sess_abc123              │       5 │    1,250 │       800 │
│ sess_xyz789              │       3 │      890 │       650 │
│ sess_def456              │       8 │    2,100 │     1,200 │
└──────────────────────────┴─────────┴──────────┴───────────┘

📈 Token Statistics
  Total Input:   4240 tokens
  Total Output:  2650 tokens
  Total Cached:  0 tokens
  Total Cost:    $0.00127
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

### 2. CLI Session Details

```bash
$ python3 scripts/cli.py show agent:main:discord:channel:1465367993012981988

======================================================================
📊 Session Detail: agent:main:discord:channel:1465367993012981988
======================================================================

🕐 Created:  2026-02-01T09:30:00+08:00
🕑 Updated:  2026-02-01T10:35:12+08:00
🤖 Model:    Qwen3-rerank
💬 Messages: 5

📈 Token Statistics:
   Input:           1,250 tokens
   Output:            800 tokens
   Reasoning:         150 tokens
   Total:           2,200 tokens

💰 Estimated Cost: $0.00126000 USD

📝 Conversation Rounds (5):
──────────────────────────────────────────────────────────────────────

  Round 1 @ 2026-02-01T09:30:15+08:00
    Tokens: 250 in → 160 out
    🔧 Tool calls: Yes
    Messages (2):
      [user] Check Beijing weather
    ❓ Question: Check Beijing weather
    ✅ Answer: Checking Beijing weather for you...
    🧠 Reasoning: User wants to know Beijing weather, I need to call weather API.
    🛠️  Tool Calls:
       - get_weather({"location":"Beijing"})
```

### 3. Statistics by Model

```bash
$ python3 scripts/cli.py stats-model

================================================================================
📊 Statistics by Model
================================================================================

Model                Sessions   Input           Output          Cost (USD)  
────────────────────────────────────────────────────────────────────────────
Qwen3-rerank         12         15,230          9,840           $  0.016800
DeepSeek-R1          5          8,450           6,200           $  0.010600
Qwen-Max             3          4,200           3,100           $  0.008300
GPT-4                2          2,100           1,800           $  0.017100
────────────────────────────────────────────────────────────────────────────
TOTAL                22         29,980          20,940          $  0.052800

================================================================================
```

### 4. Statistics by Date

```bash
$ python3 scripts/cli.py stats-date --days 7

================================================================================
📊 Statistics by Date (Last 7 days)
================================================================================

Date         Sessions   Input           Output          Cost (USD)   Models              
────────────────────────────────────────────────────────────────────────────
2026-01-26   3          2,100           1,450           $  0.0042   Qwen3-rerank
2026-01-27   5          4,850           3,200           $  0.0096   Qwen3-rerank, GPT-4
2026-01-28   4          3,600           2,800           $  0.0078   DeepSeek-R1, Qwen
────────────────────────────────────────────────────────────────────────────
TOTAL        22         29,980          20,940          $  0.0528

================================================================================
```

### 5. Web UI (Recommended)

Access `http://localhost:8888` to see:

**Home Page:**
- 📊 Total sessions, token consumption, cost cards
- 📋 Recent sessions list (clickable for details)
- 📈 Statistics by model table

**Session Detail Page:**
- 💬 Complete conversation log (messages, question, answer, reasoning, tool_calls per turn)
- 🔧 Tool call history
- 💰 Token usage breakdown and costs

**Features:**
- 🔄 Auto-refresh every 30 seconds
- 📱 Responsive design, mobile-friendly
- 🎨 Clean UI, easy to read

## Session Data Structure

Each session is stored as an independent JSON file with complete conversation history and token statistics:

```json
{
  "session_id": "agent:main:discord:channel:1465367993012981988",
  "created_at": "2026-02-01T10:30:00Z",
  "updated_at": "2026-02-01T10:35:12Z",
  "messages_count": 5,
  "total_input_tokens": 1250,
  "total_output_tokens": 800,
  "total_reasoning_tokens": 150,
  "total_cached_tokens": 0,
  "model": "Qwen3-rerank",
  "rounds": [
    {
      "round": 1,
      "timestamp": "2026-02-01T10:30:15Z",
      "input_tokens": 250,
      "output_tokens": 160,
      "reasoning_tokens": 0,
      "cached_tokens": 0,
      "model": "Qwen3-rerank",
      "has_tool_calls": true,
      "response_type": "normal",
      "messages": [
        {
          "role": "system",
          "content": "You are a helpful assistant..."
        },
        {
          "role": "user",
          "content": "Check Beijing weather"
        }
      ],
      "question": "Check Beijing weather",
      "answer": "Checking Beijing weather for you...",
      "reasoning": "User wants to know Beijing weather, need to call weather API.",
      "tool_calls": [
        {
          "index": 0,
          "id": "call_abc123",
          "type": "function",
          "function": {
            "name": "get_weather",
            "arguments": "{\"location\":\"Beijing\"}"
          }
        }
      ],
      "input_token_details": {"cached_tokens": 0},
      "output_token_details": {}
    }
  ]
}
```

### Field Descriptions

**Session Level:**
- `session_id`: Unique session identifier (from ai_log's session_id field)
- `created_at`: Session creation time
- `updated_at`: Last update time
- `messages_count`: Number of conversation turns
- `total_input_tokens`: Cumulative input tokens
- `total_output_tokens`: Cumulative output tokens
- `total_reasoning_tokens`: Cumulative reasoning tokens (DeepSeek, o1, etc.)
- `total_cached_tokens`: Cumulative cached tokens (prompt caching)
- `model`: Current model in use

**Round Level (rounds):**
- `round`: Turn number
- `timestamp`: Current turn timestamp
- `input_tokens`: Input tokens for this turn
- `output_tokens`: Output tokens for this turn
- `reasoning_tokens`: Reasoning tokens (o1, etc.)
- `cached_tokens`: Cached tokens (prompt caching)
- `model`: Model used for this turn
- `has_tool_calls`: Whether includes tool calls
- `response_type`: Response type (normal/error, etc.)
- `messages`: Complete conversation history (OpenAI messages format)
- `question`: User's question for this turn (last user message)
- `answer`: AI's answer for this turn
- `reasoning`: AI's thinking process (if model supports)
- `tool_calls`: Tool call list (if any)
- `input_token_details`: Complete input token details (JSON)
- `output_token_details`: Complete output token details (JSON)

## Log Format Requirements

Higress access logs must include ai_log field (JSON format). Example:

```json
{
  "__file_offset__": "1000",
  "timestamp": "2026-02-01T09:30:15Z",
  "ai_log": "{\"session_id\":\"sess_abc\",\"messages\":[...],\"question\":\"...\",\"answer\":\"...\",\"input_token\":250,\"output_token\":160,\"model\":\"Qwen3-rerank\"}"
}
```

Supported ai_log attributes:
- `session_id`: Session identifier (required)
- `messages`: Complete conversation history
- `question`: Question for current turn
- `answer`: AI answer
- `reasoning`: Thinking process (DeepSeek, o1, etc.)
- `reasoning_tokens`: Reasoning token count (from PR #3424)
- `cached_tokens`: Cached token count (from PR #3424)
- `tool_calls`: Tool call list
- `input_token`: Input token count
- `output_token`: Output token count
- `input_token_details`: Complete input token details (JSON)
- `output_token_details`: Complete output token details (JSON)
- `model`: Model name
- `response_type`: Response type

## Implementation

### Technology Stack

- **Log Parsing**: Direct JSON parsing, no regex needed
- **File Monitoring**: Polling-based (no watchdog dependency)
- **Session Management**: In-memory + disk hybrid storage
- **Token Calculation**: Model-specific pricing for GPT-4, Qwen, Claude, o1, etc.

### Privacy and Security

- ✅ Does not record conversation content in logs, only token statistics
- ✅ Session data stored locally, not uploaded to external services
- ✅ Supports log file path allowlist
- ✅ Session key access control

### Performance Optimization

- Incremental log parsing, avoids full scans
- In-memory session data with periodic persistence
- Optimized log file reading (offset tracking)
- Inode-based file identification (handles rotation efficiently)

Discussion

Powered by GitHub Discussions. Sign in with GitHub to comment, react, or subscribe.