Streaming Console Guide

This page provides a complete guide to using the Streaming section of the Namirasoft Infra Console. You will find a detailed explanation of how streaming systems are connected and structured for infrastructure monitoring, along with descriptions of each field used when creating and managing Streaming configurations. Use this guide to understand how Streaming connections allow Namirasoft Infra to collect operational data from data streaming platforms to ensure reliable real-time data processing.

What Is Streaming?

Streaming in Namirasoft Infra represents a connection configuration that allows the platform to access and structure operational data from streaming data platforms. Streaming systems enable continuous, real-time data flow and processing across distributed systems through topics, partitions, and consumer groups.

 

Streaming configurations in Namirasoft Infra define how the platform connects to, monitors, and collects performance metrics from your streaming platforms. These configurations enable the collection of critical operational data including throughput rates, latency metrics, partition health, consumer lag, and replication status. This data is structured and delivered to Namirasoft Expert, where it supports intelligent monitoring, performance analysis, and automated remediation workflows for your streaming data infrastructure.

 

Each Streaming configuration belongs to a specific Project and Environment, ensuring that streaming monitoring data remains organized according to your operational context. Streaming systems can be deployed in various environments (on dedicated servers, within Kubernetes clusters, or as managed cloud services) and Namirasoft Infra provides consistent monitoring through flexible connection methods.

 

The Challenge in Managing Streaming Infrastructure

Streaming data platforms are critical for real-time analytics and event-driven architectures, but they introduce unique monitoring and management complexities. As data volumes and processing requirements grow, maintaining streaming platform performance and reliability becomes increasingly challenging:

 

• Throughput Management: Monitoring and optimizing data ingestion and processing rates

 

• Latency Control: Ensuring minimal delay in data processing across the streaming pipeline

 

• Consumer Lag: Tracking and managing the delay between data production and consumption

 

• Partition Balancing: Ensuring even distribution of data and processing across partitions

 

• Cluster Health: Maintaining the health of distributed streaming platform clusters

 

• Data Loss Prevention: Monitoring for potential data loss in high-throughput environments

 

• Scale Complexity: Managing monitoring across distributed streaming clusters with hundreds of nodes

 

While streaming platforms provide some basic metrics, they typically lack integration with broader infrastructure monitoring and intelligent analysis for proactive problem prevention.

 

How Namirasoft Infra Solves the Problem

Namirasoft Infra addresses streaming monitoring challenges through structured Streaming configurations that provide comprehensive visibility into streaming platform performance and health.

 

Streaming configurations establish a standardized approach to connecting to streaming platforms regardless of their deployment environment. By offering multiple connection types (direct, via server, or via Kubernetes), Namirasoft Infra accommodates various deployment patterns while maintaining consistent monitoring capabilities across different streaming platform implementations.

 

Once configured, Namirasoft Infra automatically collects structured operational data from streaming systems, including throughput metrics, latency measurements, partition statistics, and consumer health indicators. This data flows seamlessly to Namirasoft Expert, where it can be correlated with application and infrastructure metrics to provide complete visibility into how streaming impacts overall system performance and data processing reliability.

 

By treating Streaming as a first-class monitoring entity, Namirasoft Infra enables teams to maintain reliable data flows, detect issues before they impact data processing pipelines, and optimize streaming infrastructure as part of their overall operational strategy.

Difference Between Messaging and Streaming

While both Messaging and Streaming handle data movement between systems, they serve different purposes and have distinct monitoring requirements:

Messaging is about reliable communication between services, while Streaming is about continuous data processing pipelines. For more information about Streaming, visit Messaging Console Guide.

Overview of Streaming Fields and Options

Below is a detailed explanation of the fields available when creating or managing a Streaming configuration. Understanding these fields helps ensure streaming systems are properly connected and structured for monitoring and operational analysis.

• ID (string): This is a unique identifier automatically assigned to the Streaming configuration when it is created. This is used internally to track this specific Streaming connection.

• User ID (Namirasoft Account’s ID): This is the unique identifier assigned to the Namirasoft Account user who owns the Streaming configuration. It is used internally to track who created and has access to this streaming monitoring setup.

• Workspace ID (Namirasoft Workspace’s ID): This refers to a workspace created in the Namirasoft Workspace app, which identifies the workspace where the Streaming configuration is created. 

• Name (String): This is a human-readable label used to identify the Streaming configuration. The name helps teams distinguish between multiple streaming instances.

• Project (Enum): This field specifies which Project the Streaming configuration belongs to. A Project is a logical container that groups related infrastructure components together.

You must create a Project before creating a Messaging configuration. If no Project exists, you can click the “+” icon next to this field, which redirects you to the Project configuration page. For more information about Projects, visit the Project Console Guide.

• Environment (Enum): This field specifies which Environment the Streaming operates within. An Environment represents a stage in your development lifecycle

You must create an Environment before creating a Streaming configuration. If no Environment exists, you can click the “+” icon next to this field, which redirects you to the Environment configuration page. For more information about Environments, visit the Environment Console Guide.

• Log Group (Enum): A Log Group is a logical container that stores and organizes log files from your applications and systems in Namirasoft Log. For Streaming configurations, this Log Group will contain streaming platform logs, processing logs, and performance logs.

You must select an existing Log Group for this field. If no Log Group exists, you can click the “+” icon next to this field to create a new one. For more information about Log Groups and log management, visit Namirasoft Log.

• Streaming Type (Enum): This field specifies which type of streaming platform you are connecting to. Selecting the correct streaming type ensures proper integration with platform-specific monitoring features and metrics collection.

• • Kafka: Kafka is a distributed streaming platform capable of handling trillions of events per day. Kafka is designed as a distributed commit log that allows publishing and subscribing to streams of records, storing streams of records, and processing streams of records as they occur.

• Connect Type (Enum): This field specifies how Namirasoft Infra will connect to the messaging system. The connection type determines what additional configuration will be required and how the monitoring agent accesses the message broker.

• • Direct: Connect directly to the messaging system using its network address (host and port). Use this option when:

• • • The messaging system is publicly accessible with a fixed IP or hostname

• • • Namirasoft Infra has direct network access to the message broker

• • • You don’t need to route through a Server or Kubernetes cluster

• • By Server: Connect to the messaging system through a Server resource. Use this option when:

• • • The messaging system is running on or accessible only through a specific server

• • • You need to use SSH tunneling or agent-based monitoring on the server

• • • The message broker isn’t directly accessible from the Namirasoft Infra network

• • • When you select “By Server,” you will need to select which Server configuration to use. The Server must have network access to the messaging system and may require additional configuration for monitoring.

• • By Kubernetes: Connect to the messaging system through a Kubernetes cluster. Use this option when:

• • • The messaging system is running as a container inside a Kubernetes cluster

• • • The message broker is managed by a Kubernetes operator or deployed via Helm

• • • You need to access the messaging system through Kubernetes service discovery

• • • When you select “By Kubernetes,” you will need to select which Kubernetes configuration to use. The Kubernetes cluster must have the messaging system running and accessible via Kubernetes services.

• Description (String): This field allows you to document the purpose, role, or special characteristics of the messaging system. It helps teams understand the streaming system’s function within your infrastructure.

• Streaming Credential ID (String): This is the unique identifier of the credential set used to connect to the Streaming system. This references securely store authentication information (username, password, certificates, etc.) without exposing the actual credentials. The system creates this automatically when you configure streaming credentials.

• Connect Server ID (String): If the Connect Type is “By Server,” this field contains the unique identifier of the Server resource used for the connection. This links the Streaming configuration to the specific Server entity.

• Connect Kubernetes ID (String): If the Connect Type is “By Kubernetes,” this field contains the unique identifier of the Kubernetes configuration used for the connection. This links the Streaming configuration to the specific Kubernetes entity.

• Created At (DateTime): This shows the date and time when the Streaming configuration was created. This value is automatically generated and cannot be modified.

• Updated At (DateTime): This shows the date and time when the Streaming configuration was last updated. The value changes automatically whenever configuration details are modified or when significant changes to the Streaming connection occur.