YANCEES: Yet ANother Configurable Extensible Event Service
|
|
YANCEES: Yet ANother Configurable Extensible Event Service |
|
| Concepts: | Principles | | Benefits
| |
Design | | Evaluation |
| System: | Implementation | | Documentation | |
In this project, we study the use of configurable and extensible architectures, and their ability to provide versatility to publish/subscribe infrastructures. Our goal is not only to support the development of application-specific publish/subscribe infrastructures, but also to identify design trade-offs and derive a set of software engineering principles to better support infrastructure producers and consumers in the process of building, evolving and (re)using versatile middleware. We evaluate our approach by comparing YANCEES with more traditional versatility strategies implemented by existing publish/subscribe infrastructures.
Current industrial and research infrastructures strive to support application domain variability and evolution by adopting different versatility strategies (see survey on versatility strategies). We define versatility as the ability of software to be used for different purposes in a usable and useful way. Examples of versatility strategies found in both the literature and industry include: (1) Specialization: Build new publish/subscribe infrastructures from scratch to support novel requirement sets; (2) Generalization: reuse minimal generalized publish/subscribe cores such as Siena, Elvin or Scribe; (3) Variation: reuse one-size-fits-all variable infrastructures as CORBA-NS or READY, or more recently, (4) Flexibility: adopting configurable and extensible compositional approaches, as the case with DREAM, FACET, GREEN and YANCEES itself.
In this context, YANCEES is a flexible component framework that allows the construction of application-specific publish/subscribe infrastructures. It supports the customization and extension of its feature set, along main publish/subscribe design dimensions (i.e. event, publication, subscription, routing, notification and protocol models), through the use of plug-ins and extensible languages. Examples of such extensions include advanced event subscriptions, including event correlation; alternate notification policies (push and pull); infrastructure protocols for mobile applications, and ad-hoc P2P protocols; end-user protocols such as event source location and browsing; event persistence services; fast routing cores, and many others.
YANCEES has been extended and configured to support different applications, in different domains, such as Impromptu (a project to study effective security through visualizatins), Vavoom (a visual java virtual machine), and the IC server, a test-bed for the study of contextual collaboration trade-offs. More recently, we are comparing YANCEES compositional approach with other versatility approaches in an empirical study of middleware versatility trade-offs.
Generality. One way to minimize the impact of dependencies, their configuration management costs, and incompatibility problems is to adopt general and fixed data representation. Central features such as the event format, which changes can virtually impact all other variability dimensions, need to be generalized. As such, In YANCEES, the default event model is represented by a tuple-based (attribute/value) abstraction, and the default subscription model is content-based. Generalization broadens reuse, requiring fewer adaptations of existing plug-ins (for the few cases where these models do not apply). As a rule, we use stability (the ratio between incoming and the total dependencies of a design dimension) as the criteria for selecting the dimension to generalize.
Abstraction. The ability to define domain-specific subscription and notification languages support application developers by reducing the cognitive distance involved in the development of event-driven applications. In other words, the ability to extend and configure the subscription and notification languages, as well as the event, routing and protocol models, frees the application developers from tasks associated to data format adaptation and the complexity of implementing domain-specific features.
Reuse. Finally, YANCEES not only supports the reuse of its many components, but can also the reuse of existing infrastructures such as Siena, Elvin and others as a way to achieve interoperability with existing publish/subscribe networks. Another goal of reuse is to leverage on the features provided by existing generalized infrastructures.
Usability. In order to be useful and fulfill its purpose, software must be usable. From the software consumers perspective, the usability in YANCEES is a consequence of the application of abstraction, separation of concerns and automation. Abstraction comes from the use of high-level subscription and notification languages, and a simple client-side API that hides from the users concerns related to: the runtime selection, initialization and configuration of components that implement the features required by end-users. The ability to hide these concerns from the end-users comes from the application of automation. Finally, separation of extension and usage concerns provide multiple interfaces to support both infrastructure developers and infrastructure users.
Simplicity. Complexity is the major source of problems in software development. Approaches that minimize complexity result in more efficient, evolvable, less prone to errors solution. In order to escape the one-size-fits-all syndom where the addition of many variables in the system increases its complexity, configuration costs and even affect its performance, YANCEES is built as a simple core around which different extensions and optional plug-ins are installed.
The application of these principles, while increasing the infrastructure versatility, come at the expense of extra complexity due to the following facts: 1) The different publish/subscribe concerns are not orthogonal. In other words, they participate in the process of publishing, routing and notifying events, which define implicit semantic data and control dependencies between these components. For example, the use of a global event representation, and assumptions about event frequency, creates oportunities for feature interaction. 2) Moreover, complex features are built from a combination of different components, that need to communicate, defining explicit dependencies between different parts (and components) of the system. 3) The dynamic characteristic of the publish/subscribe model require additional support for run-time allocation of components in response to client's needs. 4) Finally, the reuse of existing infrastructures require support for adaptation. In order to address requirements, the following strategies are adopted in YANCEES:
Automation of user interaction and infrastructure build process (static variability management). Even though the different concerns of publish/subscribe systems can be separated into different variability dimensions. It is usually the case that these concerns are not completely orthogonal. In other words, components are implemented in the context of a predefined configuration. As a result, changes in one dimension may affect other dimensions. For example, content-based subscription is directly dependent on the event format; advanced event processing (for e.g. sequence detection), is dependent on timing assumptions induced by different routing protocols, filters and subscription languages. If not properly enforced and documented, changes in apparently unrelated features may interfere with each order. As a consequence these fundamental problem dependencies must be adequately managed. Two main strategies are adopted to manage these issues: indirection, documentation and automation.
Dynamism (dynamic variability management). Publish/subscribe infrastructures are inherently interactive. Subscriptions are user-defined logical expressions on the order or content of events. As such, they are directly dependent on the features provided by the infrastructure. Subscriptions are also dynamic, being posted and removed at runtime by different users. Hence, In order to support the many possible combinations of features a server may have, and the intrinsic interactivity of the publish/subscribe model, YANCEES support runtime loading and assembling of features. A valid subscription is a word in the subscription language installed in YANCEES, and can be expressed in terms of different commands.Hence, subscriptions are used to orchestrate the composition of plug-ins at runtime. This strategy not only copes with configurability and customization, by also supports the co-existence of different feature sets in the same infrastructure, for example: content-based and fast topic-based subscriptions can be supported, at the same time in YANCEES. Finally, this strategy promotes a more rational allocation of resources: features are loaded on demand, copying with more restricted environments such as PDAs.
Reflexive Architecture. A consequence of the different data and control dependencies between the publish/subscribe variability dimensions is that complex features are usually implemented by the combination of different plug-ins. For example, event persistence, mobility protocols and others. Since most of these plug-ins are dynamically allocated, they need to locate each other, at runtime, in order to communicate. As a consequence, YANCEES supports runtime plug-in location through the use of a plug-in registry. A plug-in registry provides a single point of access to all the active components in the infrastructure. It supports name-based access to active plug-ins, at any given moment in time.
Support for Adaptation. The optional reuse of existing infrastructures such as Siena and Elvin as a way to achieve interoperability or to reduce the development costs associated to the routing model, requires means to convert YANCEES internal model (event representation, subscription and notification languages) to existing infrastructures. In YANCEES this is achieved by the use of adapters in the dispatcher model.
Support for Different Interfaces. Infrastructure users and infrastructure developers have different needs with respect to the way they must interact with the infrastructure. While the users are interested in selecting and utilizing the features provided by the infrastructure. Developers are interested in extending and configuring the infrastructure to meet different user needs. In the support for these different roles, YANCEES provides different interfaces. The goal is to simplify the interaction with the system exposing only the necessary API to each king of user. The supported APIs are: Publication and Subscription API, used by application developers; and the Extension and Configuratoin API used by the develpers. Strategy selection is implicitly supported by the subscription language, at runtime, which allows subscribers to select different commands according to their need. For example: content-based or topic-based filtering.
Internally, YANCEES is provide a set of component frameworks that combine:
A component framework defines standard interfaces, composition mechamisns and common data representations. YANCEES has specific component models for each one of its variability dimensions. Threee major composition mechanisms and interfaces are defined. The communication between plug-ins in the subscription, notification and routing models is achieved by the use of the listener design pattern; by the pipe-and-filter design pattern in the case of input and output filters; and by standard interfaces in the case of services and dispatchers. Subscription and Notification plug-ins are therefore composed with one another by registering themselves as each other's listeners; whereas filters are composed by connecting their standard input with a standar output of other filter. Services do not prescribe any particular interface. Their integration with other components is mediated by the plug-in registry component, that permits different pllug-ins to locate each other by name references. In most of the cases, however, the composition process is automated. For example, subscription and notification plug-ins are composed, at runtime, through a subscription parsing engine, according to the provided subscription/notification expression which is defined according to the grammar rules of subscription and notification languages. Finally, the event model is extensible. Events are generic objects, which methods and/or content must be especialized by YANCEES users.1. Dynamic parsers that allocate plug-ins according to user-provided subscriptions subscription and notification commands.
2. Input and output publication filters as a way to intercept event publication and implement global policies and filters.
3. Special static plug-ins that implement services, for example: mDNS and persistence
4. Configuration managers, dynamic parsers and builders that orchestrate the integration of different components at load time and runtime.
5. Adapters that can support existing publish/subscribe infrastructures, for ex: Siena, Elvin and custom-made cores
6. A set of component frameworks that provides interfaces and composition rules on the different variation points.
In sum, the unifying composition approach adopted by YANCEES component model is the listener pattern. Components are both producers and consumers of event, allowing multiple consumers to listen to existing components. Components also have a management interface that allow their runtime composition into listener hierarchies.
Figure2. General YANCEES architecture with plug-ins (in red) and main extension/variation points and APIs (in blue)
The
YANCEES prototype was
implemented and tested with Java (J2SDK1.4.2 and WSDP1.2 - which
provides XMLSchema validation support for the DOM parser). Currently,
the framework supports
the core mechanisms presented in the previous sections: input and
output filters, subscription, notification and protocol plug-ins, and
shared services, which can be configured at boot time with the use of
configuration description files. The system was ported to work using
Siena and
Elvin. It also supports our own fast topic-based dispatcher. The
communication between client stubs and the
server
is implemented using Java RMI protocol.
The
following plug-ins are
implemented: event sequence
detection, pull and push notification policies, polling protocol, a
simple
persistence service (in memory), a fast multiplexer dispatcher
(topic-based), and many other features.
The
service is multi threaded and
supports many simultaneous subscriptions and publications as
non-blocking operations. Input and output buffers were implemented to
improve overall throughput and performance.
More current source code is available
in the CVS
repository.
Examples,
installation
instructions, user
manual and configurations are under construction at the YANCEES
manual page (under construction).
This
software is distributed
according to the following LICENSE.TXT.
This software was tested with J2SDK1.4.2
and
Java API for XML processing available at SUN
WSDP1.2 (web services pack) or higher.
For your
convenience, we provide some examples of YANCEES bundled in different
configurations. Those bundles include only a YANCEES-SNAPSHOT.jar file
with the publisher and subscriber source codes. windows and
Unix shell
scripts are provided as well as the configuration file used to
customize YANCEES.
Minimal YANCEES:
(August 2006 snapshot)
Configured with a fast switch core, the subscription language has only
one command, <require>,
which filters messages with the specified attribute name.
Siena YANCEES:
(August 2006 snapshot)
Configured with Siena as the routing core, it provides Siena
configuration language with an extra sequence detector plug-in. This is
the version used in our example above.
August 2009 - The fourth stable public release of the YANCEES prototype is available. Previous releases can be downloaded here. The Java implementation of the prototype can be downloaded as follows:
This version supports:
Roberto
Silveira Silva Filho
Irvine, CA. August 15th, 2009.
| 2007 - Silva Filho, R. S. and D. F. Redmiles. Managing Feature Interaction by Documenting and Enforcing Dependencies in Software Product Lines. in proceedings of the 9th International Conference on Feature Interactions in Software and Communication Systems. pp.39-54, Grenoble, France, September 3-5, 2007. |
| 2005 - Silva Filho
R. S., |
| 2005 - Silva Filho R. S., Redmiles D. F. A Survey of Versatility for Publish/Subscribe Infrastructures. Technical Report UCI-ISR-05-8. University of California, Irvine, May 2005. |
| 2004 - Silva Filho R. S., Redmiles D. F. Preserving Versatility in Event-based Middleware. Technical Report UCI-ISR-04-7. University of California, Irvine, October 2004. |
| 2004 - Silva Filho R. S., De Souza C. R. B., Redmiles D. F. Design and Experiments with YANCEES, a Versatile Publish-Subscribe Service. Technical Report UCI-ISR-04-1. University of California, Irvine, April 2004. |
| 2003 - Silva Filho R. S., De Souza C. R. B., Redmiles D. F. The Design of a Configurable, Extensible and Dynamic Notification Service. in Proc. Second International Workshop on Distributed Event-Based Systems (DEBS'03), In conjunction with The ACM SIGMOD/PODS Conference, San Diego, CA, USA, June 8th, 2003. |
Additional materials |
| Slides from the ICFI 2007 conference are available here. |
| |
| Slides from the SEM 2005 workshop are available here. |
| Slides from the DEBS 2003 workshop are available here. |
| |
| Current YANCEES
Brochure is available here. |
| |
| YANCEES Poster presented in the ISR Internal Workshop, June 2004 is available here. |
This research has been sponsored by NSF (grants 0133749, 0205724, 0326105, 0527729,0524033), an IBM Eclipse technology exchange grant, and the Intel Corporation.
