Skip navigation links

Apache Edgent (incubating) v1.1.0

Edgent provides an programming model and micro-service style runtime for executing streaming analytics at the edge.

See: Description

Edgent Providers 
Package Description
org.apache.edgent.providers.development
Execution of a streaming topology in a development environment .
org.apache.edgent.providers.direct
Direct execution of a streaming topology.
org.apache.edgent.providers.iot
Iot provider that allows multiple applications to share an IotDevice.
Edgent API 
Package Description
org.apache.edgent.execution
Execution of Edgent topologies and graphs.
org.apache.edgent.execution.mbeans
Management MBeans for execution.
org.apache.edgent.execution.services
Execution services.
org.apache.edgent.function
Functional interfaces for lambda expressions.
org.apache.edgent.topology
Functional api to build a streaming topology.
org.apache.edgent.topology.json
Utilities for use of JSON in a streaming topology.
org.apache.edgent.topology.mbeans
Controls for executing topologies.
org.apache.edgent.topology.plumbing
Plumbing for a streaming topology.
org.apache.edgent.topology.services
Services for topologies.
Edgent Analytics 
Package Description
org.apache.edgent.analytics.math3.json
JSON analytics using Apache Commons Math.
org.apache.edgent.analytics.math3.stat
Statistical algorithms using Apache Commons Math.
org.apache.edgent.analytics.sensors
Analytics focused on handling sensor data.
Edgent Utilities 
Package Description
org.apache.edgent.metrics
Metric utility methods, oplets, and reporters which allow an application to expose metric values, for example via JMX.
org.apache.edgent.metrics.oplets  
org.apache.edgent.streamscope
Stream Oscilloscope - package for instrumenting streams to capture tuples.
org.apache.edgent.streamscope.mbeans
Stream Oscilloscope ControlService Management Bean interfaces.
org.apache.edgent.streamscope.oplets
Stream Oscilloscope underlying implementation oplets.
Edgent Connectors 
Package Description
org.apache.edgent.connectors.command
Command / OS Process connector.
org.apache.edgent.connectors.csv
Utilities for handling Comma Separated Value strings.
org.apache.edgent.connectors.file
File stream connector.
org.apache.edgent.connectors.http
HTTP stream connector.
org.apache.edgent.connectors.iot
Edgent IoT device and IoT Gateway device connector API to an IoT message hub.
org.apache.edgent.connectors.iotp
IBM Watson IoT Platform stream connector.
org.apache.edgent.connectors.jdbc
JDBC based database stream connector.
org.apache.edgent.connectors.kafka
Apache Kafka enterprise messing hub stream connector.
org.apache.edgent.connectors.mqtt
MQTT (lightweight messaging protocol for small sensors and mobile devices) stream connector.
org.apache.edgent.connectors.mqtt.iot
An MQTT based IotDevice connector.
org.apache.edgent.connectors.pubsub
Publish subscribe model between jobs.
org.apache.edgent.connectors.pubsub.oplets
Oplets supporting publish subscribe service.
org.apache.edgent.connectors.pubsub.service
Publish subscribe service.
org.apache.edgent.connectors.serial
Serial port connector API.
org.apache.edgent.connectors.wsclient
WebSocket Client Connector API for sending and receiving messages to a WebSocket Server.
org.apache.edgent.connectors.wsclient.javax.websocket
WebSocket Client Connector for sending and receiving messages to a WebSocket Server.
Edgent Samples 
Package Description
org.apache.edgent.samples.apps
Support for some more complex Edgent application samples.
org.apache.edgent.samples.apps.mqtt
Base support for Edgent MQTT based application samples.
org.apache.edgent.samples.apps.sensorAnalytics
The Sensor Analytics sample application demonstrates some common continuous sensor analytic application themes.
org.apache.edgent.samples.connectors
General support for connector samples.
org.apache.edgent.samples.connectors.elm327
OBD-II protocol sample using ELM327.
org.apache.edgent.samples.connectors.elm327.runtime  
org.apache.edgent.samples.connectors.file
Samples showing use of the File stream connector.
org.apache.edgent.samples.connectors.iotp
Samples showing device events and commands with IBM Watson IoT Platform.
org.apache.edgent.samples.connectors.jdbc
Samples showing use of the JDBC stream connector.
org.apache.edgent.samples.connectors.kafka
Samples showing use of the Apache Kafka stream connector.
org.apache.edgent.samples.connectors.mqtt
Samples showing use of the MQTT stream connector.
org.apache.edgent.samples.connectors.obd2  
org.apache.edgent.samples.console
Samples showing use of the Console web application.
org.apache.edgent.samples.scenarios.iotp  
org.apache.edgent.samples.topology
Samples showing creating and executing basic topologies .
org.apache.edgent.samples.utils.metrics  
org.apache.edgent.samples.utils.sensor  
Edgent Low-Level API 
Package Description
org.apache.edgent.graph
Low-level graph building API.
org.apache.edgent.graph.spi  
org.apache.edgent.graph.spi.execution  
org.apache.edgent.oplet
Oplets API.
org.apache.edgent.oplet.core
Core primitive oplets.
org.apache.edgent.oplet.functional
Oplets that process tuples using functions.
org.apache.edgent.oplet.plumbing
Oplets that control the flow of tuples.
org.apache.edgent.oplet.window
Oplets using windows.
org.apache.edgent.window
Window API.
Edgent SPI 
Package Description
org.apache.edgent.topology.spi  
org.apache.edgent.topology.spi.functions  
org.apache.edgent.topology.spi.graph  
org.apache.edgent.topology.spi.tester  
Other Packages 
Package Description
org.apache.edgent.apps.iot
Applications for use in an Internet of Things environment.
org.apache.edgent.apps.runtime
Applications which provide monitoring and failure recovery to other Edgent applications.
org.apache.edgent.javax.websocket
Support for working around JSR356 limitations for SSL client container/sockets.
org.apache.edgent.javax.websocket.impl
Support for working around JSR356 limitations for SSL client container/sockets.
org.apache.edgent.runtime.appservice  
org.apache.edgent.runtime.etiao
A runtime for executing an Edgent streaming topology, designed as an embeddable library so that it can be executed in a simple Java application.
org.apache.edgent.runtime.etiao.graph  
org.apache.edgent.runtime.etiao.graph.model  
org.apache.edgent.runtime.etiao.mbeans  
org.apache.edgent.runtime.jmxcontrol  
org.apache.edgent.runtime.jobregistry  
org.apache.edgent.runtime.jsoncontrol
Control service that takes a Json message and invokes an operation on a control service MBean.
org.apache.edgent.test.svt  
org.apache.edgent.test.svt.apps  
org.apache.edgent.test.svt.apps.iotp  
org.apache.edgent.test.svt.utils.sensor.gps  
org.apache.edgent.topology.tester
Testing for a streaming topology.
Edgent provides an programming model and micro-service style runtime for executing streaming analytics at the edge.

Apache Edgent is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

Edgent

  1. Overview
  2. Programming Model
  3. Getting Started

Overview

Edgent provides an programming model and runtime for executing streaming analytics at the edge. Edgent is focusing on two edge cases: In both cases Edgent applications analyze live data and send results of that analytics and/or data intermittently to back-end systems for deeper analysis. An Edgent application can use analytics to decide when to send information to back-end systems, such as when the behaviour of the system is outside normal parameters (e.g. an engine running too hot).
Edgent applications do not send data continually to back-end systems as the cost of communication may be high (e.g. cellular networks) or bandwidth may be limited.

Edgent applications communicate with back-end systems through some form of message hub as there may be millions of edge devices. Edgent supports these message hubs:

Back-end analytic systems are used to perform analysis on information from Edgent applications that cannot be performed at the edge. Such analysis may be:


Back-end systems can interact or control devices based upon their analytics, by sending commands to specific devices, e.g. reduce maximum engine revs to reduce chance of failure before the next scheduled service, or send an alert of an accident ahead.

Programming Model

Edgent applications are streaming applications in which each tuple (data item or event) in a stream of data is processed as it occurs. Additionally, you can process windows (logical subsets) of data. For example, you could analyze the last 90 seconds of data from a sensor to identify trends in the data

Topology functional API

Overview

The primary api is Topology which uses a functional model to build a topology of streams for an application.
TStream is a declaration of a stream of tuples, an application will create streams that source data (e.g. sensor readings) and then apply functions that transform those streams into derived streams, for example simply filtering a stream containg engine temperator readings to a derived stream that only contains readings thar are greater than 100°C.
An application terminates processing for a stream by sinking it. Sinking effectively terminates a stream by applying processing to each tuple on the stream (as it occurs) that does not produce a result. Typically this sinking is transmitting the tuple to an external system, for example the messgae hub to send the data to a back-end system, or locally sending the data to a user interface.

This programming style is typical for streaming systems and similar APIs are supported by systems such as Apache Flink, Apache Spark Streaming, IBM Streams and Java 8 streams.

Functions

Edgent supports Java 8 and it is encouraged to use Java 8 as functions can be easily and clearly written using lambda expressions.

Arbitrary Topology

Simple applications may just be a pipeline of streams, for example, logically:
source --> filter --> transform --> aggregate --> send to MQTT
However Edgent allows arbitrary topologies including:

Graph API

Overview

The graph API is a lower-level API that the topology api is built on top of. A graph consists of oplet invocations connected by streams. The oplet invocations contain the processing applied to each tuple on streams connected to their input ports. Processing by the oplet submits tuples to its output ports for subsequent processing by downstream connected oplet invocations.

Getting Started

Below, <edgent-target> refers to an Edgent release's platform target directory such as .../edgent/java8.

A number of sample Java applications are provided that demonstrate use of Edgent.
The Java code for the samples is under <edgent-target>/samples.

Shell scripts to run the samples are <edgent-target>/scripts. See the README there.

Summary of samples:

SampleDescriptionFocus
HelloEdgent Prints Hello Edgent! to standard output. Basic mechanics of declaring a topology and executing it.
PeriodicSource Polls a random number generator for a new value every second and then prints out the raw value and a filtered and transformed stream. Polling of a data value to create a source stream.
SensorsAggregates Demonstrates partitioned aggregation and filtering of simulated sensors that are bursty in nature, so that only intermittently is the data output to System.out Simulated sensors with windowed aggregation
SimpleFilterTransform
File Write a stream of tuples to files. Watch a directory for new files and create a stream of tuples from the file contents. Use of the File stream connector
IotfSensors, IotfQuickstart Sends simulated sensor readings to an IBM Watson IoT Platform instance as device events. Use of the IBM Watson IoT Platform connector to send device events and receive device commands.
JDBC Write a stream of tuples to an Apache Derby database table. Create a stream of tuples by reading a table. Use of the JDBC stream connector
Kafka Publish a stream of tuples to a Kafka topic. Create a stream of tuples by subscribing to a topic and receiving messages from it. Use of the Kafka stream connector
MQTT Publish a stream of tuples to a MQTT topic. Create a stream of tuples by subscribing to a topic and receiving messages from it. Use of the MQTT stream connector
SensorAnalytics Demonstrates a Sensor Analytics application that includes: configuration control, a device of one or more sensors and some typical analytics, use of MQTT for publishing results and receiving commands, local results logging, conditional stream tracing. A more complete sample application demonstrating common themes.

Other samples are also provided but have not yet been fully documented. Feel free to explore them.

Building Applications

You need to include one or more Edgent jars in your classpath depending on what features your application uses.

Include one or more of the topology providers:

Include the jar of any Edgent connector you use: [*] You also need to include a javax.websocket client implementation if you use the wsclient connector. Include the following to use an Eclipse Jetty based implementation:

Include jars for any Edgent analytic features you use:

Include jars for any Edgent utility features you use: Edgent uses slf4j for logging, leaving the decision of the actual logging implementation to your application (e.g., java.util.logging or log4j). For java.util.logging you can include:
Skip navigation links
Apache Edgent (incubating)

Copyright © 2017 The Apache Software Foundation. All Rights Reserved - 4744f56-20170226-1707