Name
Streaming Data Handler - Centinel (A distributed reliable framework for collection, aggregation and analysis of streaming data)
Repo Name
centinel
Description
The Centinel project aims at providing a distributed, reliable framework for efficiently collecting, aggregating and sinking streaming data across Persistence DB and stream analyzers (example: Graylog, Elastic search, Spark, Hive etc.).
This framework enables SDN applications/services to receive events from multiple streaming sources (example: Syslog, Thrift, Avro, AMQP, Log4j, HTTP/REST etc) and execute actions like network configuration/batch processing/real-time analytics.
Core features of Centinel framework are:
- Stream collector - Collecting, aggregating and sinking streaming data
- Log Service - Listen log stream events coming from log analyzer
- Log Service - Enables user to configure rules (example: alerts, diagnostic, health, dashboard etc.)
- Log Service - Performs event processing/analytics
- User Interface - Enable set-rule, search, visualize, alert, diagnostic, dashboard etc.
- Adaptor - Log analyzer plug-in to Graylog and a generic data-model to extend to other stream analyzers (Logstash etc.)
- REST Services - Northbound APIs for Log Service and Steam collector framework
- Leverages - TSDR persistence service, data query, purging and elastic search
The following architecure depicts the core components of the Centinel:
Scope
The scope of this project (Centinel for OpenDaylight Beryllium) is as follows:
- Flume based framework for efficiently collecting, aggregating, and moving streaming data into different storage destinations like ODL Persistence DB, stream analyzer
- Implementation of Log service to listen/processing for log events coming from open source Graylog analyser
- Implements GrayLog plugin extending an abstraction layer for log analyzer
- Integrated User interface for features provided by "log service"(configure-rules, alerts etc)
Functional overview of features provided by "log service" are give below:
Feature | Details |
---|
Streams | - Mechanism to route messages into categories in real time while they are processed like stream for audit logs( install bundle etc.)
- Rule configuration includes message, level, source etc.
- Streams are generated by GrayLog server as per user-defined rules. Event-handler module handles streams from GrayLog server and persist it into Centinel
|
Alerts | - Log alerts :
- Alerts get generated based on specific event matching in real-time
- Alert condition types : Message count condition, Field value condition, Field string value condition
- Alerts get cleared if specified condition does not persist. Alert check interval time is configurable, default is 60 seconds.
- Common alert operations like manual acknowledge , delete, filter , sort etc. will be supported
|
Search and Analyze | - Support for search query language.
- Time range for search can be specified
- Visualization includes histogram
|
Dashboard | - Build pre-defined views on data by adding widgets. Domain expert can define search query and save results on dashboard.
- Search result type : Search result counts, Search result histogram charts, Field value charts,Quick value results
|
Diagnostic | - Enables , user to specify/configure group of log messages (events or specific conditions ) as single rule. Order of messages and condition for each messages can also be configured.
- Notification will be persisted on db if log messages are coming out of order/sequence
- On clicking notification on UI, popup window will open which displays expected order and received order of log messages.
- User will be able to scroll popup window to see individual received log message.
- Event processor has intelligence to verify sequencing of messages and generate diagnostic reports
- Success notification is displayed when all specified conditions are true.
|
Health | - User specify list of messages with conditions for feature.
- Feature health will be changed to Critical, Major, Warn on reception of all messages specified
|
Resources Committed (developers committed to working)
Sumit Kapoor < sumit.kapoor@tcs.com>
Rajender Joshi <rajender.joshi@tcs.com>
Shreshtha Joshi <Shreshtha.Joshi@tcs.com>
Rattenpal Amandeep <Rattenpal.Amandeep@tcs.com>
Abhishek Abhi <Abhishek.Abhi@tcs.com>
Sunaina Khanna <Sunaina.Khanna@tcsin.com>
Himanshu Yadav <Yadav.Himanshu1@tcs.com>
Swati Tyagi <Tyagi.Swati@tcs.com>
Initial Committers
Sumit Kapoor < sumit.kapoor@tcs.com> <ODL ID: sumitkapoor>
Shreshtha Joshi <Shreshtha.Joshi@tcs.com> <ODL ID: shreshthajoshi>
Rajender Joshi <rajender.joshi@tcs.com> <ODL ID: rajenderjoshi1>
Vendor Neutral
The project is made from scratch, no vendor code, logos nor is anything included.
Meets Board Policy (including IPR)
New Project. No Inbound Code Review required
See also