Performance Metrics Collection

The GD node includes a comprehensive performance metrics collection system that tracks various aspects of node execution throughout its lifecycle.

Architecture

        sequenceDiagram
    participant Node
    participant MetricsCollector
    participant Storage

    Note over Node,Storage: Initialization Phase
    Node->>MetricsCollector: Create collector
    Note right of MetricsCollector: Records start time
    Note right of MetricsCollector: Records PID

    Note over Node,Storage: Import Phase
    Node->>MetricsCollector: record_timestamp("imports_done")
    MetricsCollector->>MetricsCollector: Capture memory metrics

    Note over Node,Storage: Initialization Phase
    Node->>MetricsCollector: record_timestamp("init_start")
    MetricsCollector->>MetricsCollector: Capture memory metrics
    Node->>MetricsCollector: Update node info & transports
    Node->>MetricsCollector: record_timestamp("init_done")
    MetricsCollector->>MetricsCollector: Capture memory metrics

    Note over Node,Storage: Runtime Phase
    Node->>MetricsCollector: record_timestamp("runtime_start")
    MetricsCollector->>MetricsCollector: Capture memory metrics

    Note over Node,Storage: Termination Phase
    Node->>MetricsCollector: record_timestamp("terminate_start")
    MetricsCollector->>MetricsCollector: Capture memory metrics
    Node->>MetricsCollector: mark_terminated(exit_code, reason)
    Node->>MetricsCollector: record_timestamp("terminate_done")
    MetricsCollector->>MetricsCollector: Capture memory metrics
    MetricsCollector->>MetricsCollector: Calculate durations
    MetricsCollector->>Storage: save_metrics()

Collection Phases

The metrics collection system tracks five distinct phases of node execution, using the MetricsEvent enum:

Start-up Phase (START)
- Process ID capture
- Start timestamp recording
Import Phase (IMPORTS_DONE)
- Module import completion time
- Memory usage after imports
- Import duration calculation
Initialization Phase (INIT_START, INIT_DONE)
- Initialization start/end times
- Node name and instance name recording
- Active transport configuration
- Memory usage during initialization
- Initialization duration calculation
Runtime Phase (RUNTIME_START)
- Runtime start timestamp
- Memory usage at runtime start
- Continuous runtime duration tracking
Termination Phase (TERMINATE_START, TERMINATE_DONE)
- Termination start/end timestamps
- Exit code recording
- Termination reason logging
- Final memory metrics capture
- Total duration calculations

Node Status Information

The metrics system tracks node state transitions:

State: Progresses through:
- “starting” (initial state)
- “running” (after initialization)
- “terminated” (final state)
Exit Code: Integer value indicating how the node terminated
Termination Reason: String description of why the node terminated

Memory Metrics

At each phase, the following memory metrics are collected:

RSS (Resident Set Size): Actual physical memory used by the process
VMS (Virtual Memory Size): Total virtual memory allocated

These metrics are stored as fields with event-specific prefixes, for example:

memory_imports_done_rss
memory_runtime_start_rss
memory_terminate_done_vms

Duration Calculations

The system automatically calculates several duration metrics in seconds:

Import Time: Duration of module imports
Initialization Time: Time spent in initialization
Runtime Duration: Time between runtime start and termination start (or current time for active nodes). This represents the actual execution time of the node, excluding initialization overhead.
Termination Duration: Time taken to clean up and exit
Process Time: Total time from process start until current time or termination, including all overhead (imports, initialization, runtime, and termination)
Total Process Time: Overall lifetime of the node from start to termination

Key Differences:

Process Time vs Runtime Duration:
- Process Time tracks the entire lifetime of the node process from the very start, including all overhead
- Runtime Duration only measures the actual execution time of the node between RUNTIME_START and TERMINATE_START events
- This distinction is useful for:
  
  Identifying initialization overhead
  
  Debugging startup performance issues
  
  Understanding total resource usage vs active runtime usage

Time Storage

All timestamps and durations are kept in seconds and sent as floating-point values. Timestamps come from time.perf_counter(), so they are only meaningful relative to each other within the same process, not as wall-clock times.

Storage Fields

While the node runs, save_metrics() is called every 60 seconds, and once more when the node terminates. Every save sends one sample in the gdnode_metrics metric group with these fields:

timestamp_{event_name}: Timestamp of each recorded event
memory_{event_name}_{rss|vms}: Memory snapshot at each recorded event, in bytes
duration_{import_time|init_time|runtime|termination|total|process_time}: Calculated durations
state: Node state (starting, running or terminated)
exit_code: Exit code, or -1 if not set yet
termination_reason: Present only once the node has terminated
transports: Comma-separated list of active transports

Each record also carries these tags:

node, instance_name, pid, measurement_tag and robot, set by the collector
metric_group=gdnode_metrics
fleet and service, added by the telemetry client

Storage Backends

MetricsCollector.create() selects the collector from the PLATFORM_METRICS environment variable:

TelemetryMetricsCollector (PLATFORM_METRICS enabled): Sends metrics through MetricsTelemetryClient from movai_core_enterprise, which therefore must be installed.
NullMetricsCollector (PLATFORM_METRICS disabled, the default): Does nothing and does not import any metrics dependencies.

MetricsTelemetryClient does not connect to Mimir or Loki directly. It writes records in Fluent Forward format to the platform telemetry Unix socket, and Fluent Bit forwards them:

Mimir: Every numeric field (timestamps, memory, durations, exit code) is sent as a separate gauge record.
Loki: The whole sample, including the non-numeric state, transports and termination_reason fields, is sent as one metric_record event.

If the socket is unavailable, the client logs the error and the node keeps running.

Configuration via environment variables:

PLATFORM_METRICS: Enables collection when set to 1, true or t (default: disabled)
GD_NODE_METRICS_TAG: Value of the measurement_tag label (default: default)
MOVAI_TELEMETRY_SOCKET: Path of the telemetry socket (default: $MOVAI_IPC_PATH/movai-platform-metrics.sock, i.e. /opt/mov.ai/comm/movai-platform-metrics.sock). PLATFORM_METRICS_SOCKET_PATH is accepted as a fallback name.
TELEMETRY_ENABLE: Set to false to disable sending telemetry altogether (default: enabled)

Usage Examples

Using a tag named “my_test”, with the platform telemetry stack (Fluent Bit, Mimir and Loki) running:

export GD_NODE_METRICS_TAG=my_test
export PLATFORM_METRICS=1
python3 -m gd_node --name node_name --inst instance_name

The samples from the last hour can then be read back from Loki with MetricsTelemetryClient().get_metrics("gdnode_metrics", tags={"measurement_tag": "my_test"}).

Pass from_ and to_ (Unix timestamps) to query a different time range.