David Pallmann's Blog: ESB

Neuron ESB and Endpoints

Mon, 19 May 2008 01:28:11 GMT

Neuron ESB and Endpoints

Today I’d like to talk about how Neuron ESB connects to physical endpoints in the enterprise. Two blog posts ago I talked about Neuron’s topic network architecture, and in my last blog post I focused on the internal side of the bus, where channels implement pub-sub over a variety of Microsoft technologies ranging from WCF to BizTalk Server. But just how do those publisher and subscriber nodes relate to the physical world of services and clients? And what about non-service applications? We’ll answer that here as we talk about the external side of the bus.

First off, let’s mention in passing that one way you can program against Neuron is to write .NET programs using Neuron’s API. If you do that, you are directly creating a publisher or subscriber node. We’ll discuss the API and when you might choose to use it at another time. For now, we want to focus on scenarios where you don’t want to do that. Rather, you have existing endpoints in the enterprise and you want Neuron to be able to talk to them.

External endpoints fall into 2 distinct categories: they’re either service-oriented or they aren’t.

· Service endpoints are service-oriented applications which of course come in two roles: clients (which initiate messaging) and services (which respond to messages).

· Adapter endpoints are non-service oriented applications such as a legacy system or a line-of-business application. Neuron can access these kinds of endpoints using adapters. Like service endpoints, adapter endpoints can be send-oriented (resembling clients) or receive-oriented (resembling services).

Neuron can communicate with both service and adapter endpoints in request-reply or 1-way messaging patterns.

Service Endpoints

Service endpoints are first- or second-generation service programs. Neuron uses WCF to interact with service endpoints and thus inherits the ability to interconnect to services via a wide variety of transports and security models.

A web client connects to the ESB through a client connector (Figure 1). The client connector is both a WCF endpoint that the client can send messages to and a bridge to a Neuron publisher node. The client connector is mapped to a Neuron publisher Id and topic: when a client sends a message, the client connector receives the message and publishes it over the bus using the assigned publisher and topic. As described in a previous post on service pass-through, the hosting port on the ESB server looks to the client just like its target service, except that the address is different.

Figure 1: Client and Service Connectors

A service is connected to the ESB through a service connector (Figure 1) and acts reciprocally to a client connector as you would expect. The service connector is a Neuron subscriber node and a bridge to a WCF channel by which messages can be sent to the service. The service connector is mapped to a Neuron subscriber Id. When the subscriber receives a message over the bus, the service connector relays it to the service.

The bus gives you the freedom to enjoy a great variety of configuration possibilities that would be impossible if you directly connected your clients and services.

· Loosely-coupled routing. Although Figure 1 only shows 2 endpoints, there could be any number of them communicating together. Doing so through the bus is far more attractive than the alternative, where services become clients of other services; such tightly coupled routing requires every module to know the connection details of every other module. It’s much smoother to communicate by named topics.

· Mediation. There’s no requirement for the client and service endpoints to use the same method of communication or the same service model. For example, you could have a client that uses username/password security interacting with a service that uses X.509 certificate security through the bus. Even programs that have a different idea about message format or version can be made to work together by having the bus perform transformations on messages as they are en route.

· Workload Sharing. You can have more than one instance of the same service, and the bus can perform workload sharing among them.

· Service Policy. What should happen to a message if the target service is not available to receive it? Neuron lets you specify policy to control delivery aspects such as retries and failover to alternate destinations.

· Services and Non-Services Working Together. You’re completely free to have your clients and services interact with non-services. The bus extends the reach of your services and other I.T. assets.

Adapter Endpoints

Enterprises often have line of business applications and/or legacy systems that are core to the business but are not service-oriented. Often these applications are “silos” in the sense that they have little or no interaction with other applications in the enterprise. Neuron provides comprehensive adapter support to bring these other systems onto the bus and make them accessible to the rest of the enterprise.

Neuron supports 2 families of adapters:

· WCF LOB Adapters are the successor to BizTalk adapters. Adapters have already been developed for Oracle database, SAP and Siebel by Microsoft, and many other adapters are in developed by Microsoft and other members of the community. As more and more of these adapters are developed, they benefit not only WCF and BizTalk Server but Neuron ESB as well.

· Neuron .NET Adapters are Neuron’s own adapter model. They have an interface that makes them very easy to write (the design goal was that field engineers could write one on site if necessary).Neuron ships with a versatile collection of these adapters.

For Neuron .NET adapters, an endpoint connector (Figure 2) works analogously to a service connector or a client connector. WCF LOB Adapters are a different matter: because they are designed to look like WCF bindings, they can be used with service connectors or client connectors.

Figure 2: Endpoint Connectors

Just like service endpoints, adapter endpoints can operate in a client/sender role or a service/receive role; and just like service endpoints, adapter endpoints can utilize request-reply or 1-way messaging. An adapter defines one or more modes of behavior that define what it does and which direction and pattern messaging the mode needs.

To talk to a send mode adapter endpoint, the endpoint connector acts like a service connector, relaying bus messages to the adapter. The endpoint connector has an associated Neuron subscriber on which it receives bus messages which are then passed on to the adapter for delivery.

To talk to a receive mode adapter endpoint, the endpoint connector acts like a client connector, relaying adapter messages to the bus. The endpoint connector has an associated Neuron publisher on which incoming messages from the adapter are published onto the bus over a designated topic.

Interaction with adapters is optionally transacted: both Neuron adapters and Neuron pub-sub channels may implement transactions, and if they do the hand-off between them is also linked transactionally. This makes it possible to have transacted end-to-end communication through the bus, as long as the channel and adapter components you select support transactions.

As with service/client connectors, you may set up adapter policies to control the behavior of adapter endpoints in the face of errors. You can determine retry counts and actions to take when a message can’t be delivered.

The adapters that ship with Neuron include ActiveMQ, RFID via BizTalk Server, eConnect (Great Plains accounting), SMTP email, files, MSMQ, SQL Server, and WebSphere MQ) and we’ll be taking more about them individually down the road.

Putting it all together: Service and Adapter Endpoints

Neuron’s strong support for services and adapters lets you break down the walls between your SOA applications, legacy systems, and line of

Neuron’s strong support for services and adapters lets you break down the walls between your SOA applications, legacy systems, and line of business applications, and other groups of systems that up till now have had limited reach.

All of your systems are now connected, but with loosely-coupled routing through named pub-sub topics. Actions such as changing the connection specifics of an endpoint, adding new endpoints, or removing endpoints can be done without disrupting the rest of the enterprise and without the need to make code changes to existing programs.

The Neuron ESB Channel Model

Wed, 14 May 2008 19:43:03 GMT

The Neuron ESB Channel Model

In my previous post I discussed the importance of topics in Neuron and how each topic is implemented as an individual publish-subscribe network, called a topic network. Topic networks are based on a modular channel model and can be powered by technologies ranging from WCF peer networking to BizTalk Server. Today we’ll look at the channel model in more detail.

ESBs have endpoint architecture, meaning there is a common stack all endpoints go through in order to interact with each other. This is conceptually no different than using a WCF client to talk to a WCF service: a common stack on both ends of the connection work together to facilitate communication and cooperative processing. The difference with the ESB stack is the functionality provided, which includes publish-subscribe communication to N parties and the ability to layer integration functionality (transformations, rules, workflow) into your messaging at any point.

How does the ESB stack pull off the publish-subscribe communication between N parties? You might assume all those ESB stacks simply connect to an ESB Server in a hub-and-spoke arrangement, but that’s not the case at all. Or more accurately, it’s up to the channel. A channel is simply a software module that implements publish-subscribe using some base technology. The four channel types included with Neuron implement pub-sub over peer networking, WCF TCP, MSMQ, and BizTalk Server. Each of the channels provide the same functionality but with varying qualities of service. Since each topic is a separate channel choice, you are free to run different qualities of service side by side. You might, for example, favor speed for some business conversations but require reliability for others.

Although the channel’s responsibility is to implement publish-subscribe, applications are not limited to that one messaging pattern. Neuron’s stack also allows request-reply, direct point-to-point connections, and itinerary routing (where the message determines the delivery points). All of these patterns are always available, regardless of which channel you use.

All publish-subscribe systems have to deal with backlog, where published messages come in faster than they can be delivered. A channel’s implementation must provide a strategy for handling backlog.

Peer Networking Channel

Peer Channel uses Windows peer-to-peer networking. In WCF, this technology is packaged up as the NetPeerTcpBinding. With PeerChannel you create a cloud or mesh of messaging nodes. The nodes are interconnected using both TCP and UDP, and will also take advantage of IPv6 on the network if they are able to. Any message sent by a node is delivered to all others nodes on the mesh in a highly parallel fashion that doesn’t require server infrastructure. A mesh is self-organizing, changing its shape in response to nodes joining/leaving and activity. This automatic optimization keeps the number of hops to a minimum and slower nodes are moved to the outside of the mesh.

Peer networking often brings to mind consumer applications such as instant messaging, music and file sharing, or multi-player gaming. However, there are enterprise scenarios where peer networking is ideal. Its great strengths are its high speed, its parallelism, and its scalability.

In Neuron, each topic powered by Peer Channel is a separate mesh. The behavior of a PeerChannel mesh is a natural fit for publish-subscribe messaging. Neuron deployments have been known to operate meshes with as many as 1500 nodes that stretch across the U.S. over multiple geographic locations.

There’s a catch with peer channel, however. The base technology is not reliable enough to suit most enterprises. There are no message delivery assurances: you can lose messages (sometimes large blocks of messages if there’s a lot of CPU contention) nor are messages guaranteed to arrive in order. The good news is that Neuron ESB provides an optional reliability layer that provides delivery assurances through message caching.

In its native form, peer channel is server-less. In fact, you can shut down all of your servers and parties can continue to message. Adding in the reliability layer or enabling certain features such as auditing will require you to keep at least one server up and running however.

The peer networking channel is volatile, meaning there is no durable storage of messages. Performance is very good, but is limited by the slowest node in the mesh.

Peer channel is best suited for scenarios requiring low-latency (high speed) communication and/or the need to broadcast to many recipients simultaneously.

WCF TCP Channel

The WCF TCP channel interconnects parties using WCF TCP with reliable messaging. The channel connects parties to an ESB server via TCP. A massively efficient TCP publishing engine on the server copies incoming published messages out to connected subscribers with matching subscriptions.

The server publishing engine can become backlogged, where published messages are coming in faster than they can be relayed out. The engine will buffer backlogged messages up to a threshold; once that threshold is exceeded, flow control kicks in and publishers will be slowed down as needed.

In a server farm scenario, each server will have a different collection of parties connected to it. The servers share incoming messages with each other laterally so that any incoming messages is relayed to all servers for distribution.

The WCF TCP channel is volatile, meaning there is no durable storage of messages. There are delivery assurances within the scope of a connection, but sudden client or server loss can lose messages.

The TCP channel is a good choice for applications that don’t require durable or transacted messaging.

MSMQ Channel

The MSMQ channel implements publish-subscribe over queues. Messaging is durable, transacted, and reliable. There is a publish queue for a topic, and a recipient queue for each subscriber. An MSMQ publishing engine on the server moves messages between queues.

Parties do not have to be online in order to receive messages from each other because undelivered messages persist in the queues. This characteristic allows you to take systems offline and online without being disruptive.

The MSMQ queues used by the channel are durable, providing reliable store-and-forward messaging. In a backlog situation, messages that cannot be immediately processed accrue in publisher or subscriber queues until they can be processed.

The server’s MSMQ publishing engine is transactional. A delivery either fully completes or the queue reads and writes are rolled back. The engine processes messages in batches when it can for efficiency.

The MSMQ channel is a good choice for transacted, durable communication.

BizTalk TCP Channel

The BizTalk TCP channel performs publish-subscribe over BizTalk Server. The channel connects parties to BizTalk Server via TCP, and BizTalk’s messaging engine performs the publish-subscribe messaging.

Making changes to the Neuron ESB configuration automatically programs BizTalk Server to make matching changes to its configuration. For example, defining new publishers and subscribers in Neuron will cause new send ports and receive ports to be created in BizTalk.

With BizTalk as the messaging engine, you gain the many features of BizTalk including strong retry/recovery capabilities. With a small amount of BizTalk configuration you can link messages to features such as Business Activity Monitoring (BAM),, transformations , the rules engine, and orchestrations.

The BizTalk channel is durable and is well-suited for mission-critical applications that require high reliability and/or the unique characteristics and features of BizTalk Server’s. It is not the best choice for low-latency applications, as BizTalk’s message box database limits performance.

What about Services, Legacy Systems, and Line of Business Apps?

Does this mean there are only 4 protocols you can use to communicate with in Neuron? Not at all--the endpoint architecture and channel model we’ve been describing is how Neuron makes publish-subscribe happen, but this is the internal side of the bus. Externally, you can connect a wide, wide variety of services and non-services to the bus… everything from WCF services to WebSphereMQ. We will talk about the external side of the bus in future articles.

Summary

The Neuron ESB channel model provides a modular system for switching in different technologies for messaging. This not only makes it possible to leverage a wide range of existing technologies, but anticipates new technologies and provides a path for accommodating them.

Topic Networks: How Neuron ESB Implements Publish-Subscribe

Tue, 13 May 2008 23:43:53 GMT

Today I’d like to talk about how Neuron ESB implements publish-subscribe over topics. Topics are important in Neuron both logically and architecturally. Logically, topics model business conversations and are an important tool for organizing information. Architecturally, topics are independent networks that give the bus its distributed nature and scalability.

A Logical View of Topics

Logically speaking, topics represent business conversations and are usually named accordingly. Topics can have simple names like Orders or Inventory, or a topic can have a hierarchy of subtopics, expressed with dotted names such as Orders.New, Orders.Approved, Orders.Shipping.US, Orders.Shipping.International, and so forth. A party expresses interest in a topic with a subscription. Subscriptions may be wildcarded, as in Orders.* which expresses interest in Orders as well as all of its subtopics (Orders.New, etc.). By modeling your solution after actual business conversations and entities, you raise the likelihood that your messaging system will be a successful and long-lived one.

Together, topics and subscriptions define a virtual circuit between publishers (senders) and subscribers (receivers) about some subject of interest. In Figure 1, a publisher named OrderEntry has published a message on the topic Orders.New. Of the 3 other parties, only 2 have matching subscription: (OrderApproval and OrderFulfillment. The other subscriber, PaymentApproval, is only interested Orders.Paid and does not see messages published on Orders.New.

Figure 1: Topical Publish-Subscribe

Although bus communication is pub-sub based, that doesn’t limit you to pub-sub as a messaging pattern. Parties can also use point-to-point messaging, request-reply messaging, and itinerary-based routing.

The ability to easily add, change, and remove parties, topics, and subscriptions—even while running live—results in great flexibility and easy change management.

An Architectural View of Topics

Architecturally speaking, each topic is an individually configured network called a topic network. Figure 2 shows 3 topic networks, named Inventory, Orders, and CallCenter. Each topic network is separately assigned a communication technology to drive it, such as peer networking, WCF TCP, MSMQ, or BizTalk Server. This allows you to run topics with a mix of qualities of service, a refreshing change from messaging systems with a single “one size fits all” QoS. Each topic network can also refer to separate server infrastructure, making topics the primary division for scaling out. The distributed topic network design is very different from the classic EAI hub-and-spoke design. There is no central hub and no central message store that all messages must pass through; instead, each topic network performs its functions independently. While you could run all of your topics over the same technology and off of the same server, you could just as easily run each topic over a different technology and different servers. Moreover, some of the technologies that can power topics require little or no server infrastructure at all.

Figure 2: Topic Networks

If each topic is really a separate network, what sort of network is it, and what are its characteristics? The answer to that is determined by the channel you select. A channel is a modular software driver that implements pub-sub for Neuron ESB using some base technology, such as WCF or BizTalk Server. The channel has complete freedom to implement pub-sub any way it wishes, as long as it provides the expected functionality. Some channels are volatile, emphasizing speed over reliability. Others channels are durable, stressing reliability over speed.

In a future article, we will dig into channels more deeply and compare the various trade-offs between them.

Centralizing your WCF Configuration with Neuron ESB

Tue, 06 May 2008 00:29:57 GMT

Centralizing Your WCF Configuration with Neuron ESB

Today I’d like to talk about how you can store your WCF configuration information centrally with Neuron ESB . While it’s great that WCF allows you to move details such as endpoint, binding, and other information out of code and into config file settings, it can still be inconvenient to work with many individual config files. If you are dealing with large numbers of services, you may desire a way to centralize your configuration information. Neuron ESB provides a service repository for this purpose, plus a neat factory for creating WCF clients. Together, they free you from needing local WCF configuration settings.

There are a number of ways you can program against Neuron, ranging from WCF to JMS to Neuron’s own .NET API. In this case, we’re going to use the Neuron API to retrieve configuration information and create a WCF client; the rest is pure WCF from there.

Adding services to the Neuron service repository is very straightforward. An import wizard can guide you through the steps or you can set things up manually. Service definitions can be imported from metadata at URL, file, or via searching a UDDI directory. The appropriate WSDL, schema, and service endpoints are added to the repository. Neuron employs the same classes for service import that are used by Add Service Reference in Visual Studio and the SvcUtil tool.

Figure 1: Service Definition in Neuron's ESB Explorer

The service repository itself is of course platform independent, making no assumptions about what technologies are used to build services and clients.

For WCF developers, Neuron provides a client factory class that uses the central repository to create auto-configured WCF clients. Here’s how this works. Imagine you are writing a WCF client for an order entry service that has a service contract named IOrderEntry. To develop your WCF client, you go through the usual steps, including generating client code from metadata. However, you don't worry about creating a System.ServiceModel section in your local config file. Instead, you import the target service in Neuron’s service repository.

To instantiate a WCF client in your code, you use Neuron’s Client<T> class, where T is the interface you want a client for. For example:

IOrderEntry oeClient = new Client<IOrderEntry>();

What this does is scan the central service repository for a service to locate a matching interface. If a match is found, a WCF proxy is created, using the address, binding, and other information defined in the repository. From this point forward, your client program is using pure WCF, and accesses the service in the usual manner, via named methods:

IOrderEntry oeClient = new Client<IOrderEntry>();
PriceList pricelist = oeClient.GetPriceList();
int orderId = oeClient.SubmitOrder(order);

In the example above, we created the client simply by specifying an interface to Client<T>. In all but the smallest deployments that’s probably not enough information to make a good match. What environment is the service in? Is it available and in good health? Do you have rights to access it? You can be more specific by using alternative constructors to Client<T>. For example, Client<T>(name) will locate a specific service endpoint in the repository by its unique name. More powerful is Client<T>(criteria), which will match an array of name-value properties against properties in the repository. You can use repository properties to indicate any information you like about a service, such as the environment it serves or the version of its contracts.

One valuable service the Client<T> class provides is sparing a developer from having to learn and keep up with the connection details for a target service. Each time a connection is made, the latest information in the service repository determines address, binding details, security settings, and so on. Similarly, administrators can make changes to service locations and access details without worrying about being disruptive to developers.

Another value provided by the Client<T> class is indirection. When you set up a service definition in the Neuron ESB repository, you can optionally enable pass-through of service traffic through the ESB (a subject we will go into more detail about in a separate article). Pass-through of service traffic is transparent to the client and service, but doing so allows additional parties on the bus to share in the messaging. Pass-through also enables bus-managed scenarios such as workflow sharing among multiple service instances. All of this is transparent to the WCF developer, however. if you invoke Client<T> and the matching service definition enables service traffic pass-through, the WCF client that is created will point to a hosting service on the ESB. If service pass-through is not enabled, the WCF client that is created points directly to the actual service. Either way, the usage is the same and the developer won’t know the difference.

Client<T> and the service repository is an example where Neuron adds value, yet you are still using solid, familiar WCF technology to create your clients and services.

Neuron ESB Management Experience

Fri, 02 May 2008 18:29:38 GMT

Neuron ESB Management Experience

Today I’d like to talk about the Neuron ESB management experience. Neuron takes management seriously and a lot of effort has gone into providing a one-stop administrative experience for configuration, deployment, activity reporting, operations monitoring, change management, and problem response

ESB Explorer

Neuron’s management tool is called ESB Explorer, shown in Figure 1 below. ESB Explorer’s UI is styled after MS CRM, with a navigation bar at left and a workspace area to the right. It should have a familiar feel to any Outlook user.

Figure 1: ESB Explorer (click for larger image)

The navigation bar has 7 sections. We’ll be digging into these areas individually down the road.

· Messaging is where you set up your logical messaging model of publisher, subscribers, topics, and subscriptions.

· Data is a repository for various data artifacts including XML schema, WSDL metadata, XSLT transformations, sample messages, and message templates.

· Connections is where you define service- and adapter-based physical endpoint connections.

· Security is where you define credentials, set up access control lists, set encryption keys, and other security-related functions.

· Integration is where you set up integration functions (schema validation, transformations, business rules, workflows, etc.) and string them together into processing pipelines.

· Deployment is where you define zones and environments, manage resources such as databases and queues, and define deployment configurations.

· Activity is where you monitor activity in real-time, access database-driven activity reports, monitor server health, and take action on failed messages.

The ESB Explorer lets you work in online or offline mode. In online mode, saving your changes causes them to be applied live. In online mode, saved changes only take effect when you restart your ESB Server(s).

Entity Management

Many of the pages in the ESB Explorer are for managing entities such as topics, publishers, subscribers, and services. The Explorer handles entity management consistently with a list/item view. Figure 2 shows the page for managing subscribers.

Figure 2: ESB Explorer (click for larger image)

If you are dealing with large numbers of entities, you can organize them by category. Entity management pages let you filter your view by category, alphabetically, or by search term.

Diagrams

The ESB Explorer provides a number of visualizations, such as communication diagrams and deployment diagrams. The communication diagram in Figure 3 shows the relationship between publishers, subscribers, and topics.

Figure 3: Communication diagram (click for larger image)

Real-time Activity Monitoring

Both the server- and client-side components of Neuron are instrumented to report activity in real-time through WMI and performance counters. You can monitor this activity in ESB Explorer with a variety of real-time views. Figure 4 shows a chart of message counts.

Figure 4:Real-time Activity Chart (click for larger image)

Because Neuron activity information is reported via WMI and performance counters, nearly any operations monitoring tool can also

Reports

In addition to real-time reporting, Neuron can also log activity information to a database. For example, you can enable message auditing for any topic(s) in your ESB configuration, which will record every send and every receive. You can this information from a variety of perspectives, such as message history reports, failed message reports, and business event reports.

Figure 5: Message History Report (click for larger image)

Resource Management

Your ESB configuration can optionally make use of enterprise resources such as databases and queues. Neuron can create, configure, and check these resources. It can also generate SQL and WSH scripts if you need this work done by others such as a DBA.

Figure 6: Database Artifact Creation (click for larger image)

Wizards and Guides

Wizard and guides help you through performing tasks for the first time. There are wizard for setting up pub-sub topics, importing services, searching service directories, and setting up deployment configurations. Guides such as the Getting Started Center (Figure 1, above) and Samples Browser help you find what you need.

Configuration Checking

Neuron ESB is a product with a rich set of communication and integration capabilities, so enforcing a correct configuration is critical. If you have inconsistencies in your configuration, you want to find that out at configuration time, not at runtime. An example of an inconsistency is a subscriber with a subscription to Inventory, where no topic named Inventory has been defined.

The ESB Explorer performs a consistency check any time you attempt to save an updated configuration. Inconsistencies either need to be fixed, or you can mark the offending entities as disabled for the time being. Once your configuration is consistent, you can save it. The Neuron ESB server performs an identical consistency check as part of its power-on self tests.

Administration API

The ESB Explorer is an extremely powerful management tool. Everything it does is accomplished through an administration API, which means you can do all of this in .NET code if you choose to and have appropriate rights.

That does it for our introduction to what management is all about in Neuron ESB. We’ll be frequently referring to ESB Explorer as we discuss how to configure individual feature areas in future articles.

A High Level View of the Neuron ESB Architecture

Wed, 30 Apr 2008 19:19:11 GMT

A High Level View of the Neuron ESB Architecture

Today I’d like to provide a very high level view of the Neuron ESB architecture. Understanding this will help put individual features and concepts in context as I describe them in upcoming articles.

The bus is best thought of as having a distinct outside and inside, as the figure below illustrates.

· The outside of the bus is concerned with connecting to diverse enterprise applications, services, and legacy systems

· The inside of the bus is concerned with interconnecting endpoints via publish-subscribe messaging and performing integration services.

The Outside of the Bus

The outside of the bus has exacty one purpose: to establish connections to any kind of business endpoint in the enterprise. Neuron has strong support for both services and non-services. Applications can talk to the bus in a way that is natural to them in terms of protocol, security model, and message format.

· For service-oriented endpoints, Neuron uses WCF to connect both clients and services.

· For non-service endpoints, Neuron uses adapters to communicate. Neuron supports several types of adapters.

· When you have an endpoint that's neither a service nor has an existing adapter, you can consider wrapping the endpoint with a service, or writing a new adapter (writing a Neuron adapter is nowhere near the daunting task it is in many other products). There is also a .NET API for Neuron that can be helpful in new integrations.

Service support is extensive. You start by defining a service to Neuron. You can locate a service endpoint in a UDDI directory, import a service from its WSDL, or manually define the service as you prefer. You can specify any standard WCF binding or any custom binding you may have. You can use any of the common WCF security scenarios, spanning transport security to message security to federated security.

Adapters supported include WCF LOB adapters, which are superseding the earlier family of BizTalk Adapters. There are already WCF LOB adapters for Oracle, SAP and Siebel available from Microsoft, and many more adapters are available or in development from a variety of sources. Neuron also supports its own .NET adapters, which have a simple interface and are easy to write. Adapters that come with Neuron include ActiveMQ, BizTalk RFID, eConnect (Great Plains), file, MSMQ, SharePoint Server, SMTP Email, SQL Server, and WebSphere MQ.

Anyone who's done a lot of enterprise integration in the EAI space knows that integration jobs can vary from the simple to the complex to the really difficult. We've put a lot into Neuron to make enterprise connectivity as simple and painless as possible, as often as possible.

The Inside of the Bus

The inside of the bus is concerned with interconnecting endpoints through publish-subscribe messaging. Each endpoint that connects to the bus is related to Neuron's internal publish-subscribe messaging model.

Every endpoint is mapped to a party. A party is just a messasging node: we call a party that send messages a publisher, and a party that receives messages a subscriber. Subscriptions link a party to one or more topics of interest. Finally, a topic is a name for a business conversation. So, for example, you might have a publisher named OrderEntry and a subscriber named OrderProcessing, both subscribing to the topic Orders. When OrderEntry publishes a message over the topic Orders, OrderEntry receives a copy of the message because it has a matching subscription. If additional parties want to know about orders, all they have to do is subscribe to the Orders topic. Publish-subscribe is extremely flexible, and Neuron allows you to add parties and change subscriptions live while the system is running.

What makes the communication happen? Neuron supports a varierty of Microsoft technologies to perform the communication, from WCF to BizTalk Server. You can also run a mix of technologies, such that different topics provide different qualities of service. More on that later.

The inside of the bus has another job, which is to provide integration functions. While a message is en route between endpoints, the bus can apply pipelines of processing steps. Functions that can be performed include validating schema, transforming a message, executing rules, executing a workflow, performing additional messaging, and moving message data to/from persistent storage. Integration services are part of what it takes to mediate differences between endpoints: you might for example need to transform a message from one endpoint in order to make it intelligible to another endpoint. They also allow to apply business rules or workflows via the bus instead of having to make code changes to your business programs.

Integration functions can be provided by services. This modularity gives you the freedom to use any technology or product you wish to power them, and to change them out at any time. Not only can you use the rules engine, workflow engine, or transformation technology of your choice, but you can run more than one side by side.

Benefits

What does this design do for you? Why connect everything to the bus? What is the gain?

· You can get programs that could never directly interact to work together through the bus.
We call this mediation, and it's a faster and cheaper alternative to rewriting your apps.

· You create additional value from your existing I.T. assets by extending their reach.
CFOs really like getting new value from previous money spent.

· Changes in routing between parties is now a painless configuration exercise instead of an application code change project.
Easy change management makes I.T. and your entire organization more agile.

· Business rules and workflows can be deployed externally to application code.
And you can deploy them while running live, without having to schedule downtime or code changes.

· Messages can be audited, checked for policy compliance, and replayed for disaster recovery.
Make your enterprise more consistent. Replace a fragmented view of individual applications with an enterprise-wide view of activity.

· Patterns of messages can be linked to business events.
Relate messaging to business events. Get real-time notification about what is transpiring in your organization. Feed BI and analysis tools with the information they need.

· Accelerate SOA adoption.
If you're serious about SOA, at some point you're going to realize that your business services need to be complemented by infrastructure services. Neuron ESB gives you a versatile set of infrastructure services (while preserving your right to use best of breed products to power them) along with a central repository and one-stop management experience.

There's another big benefit of the Neuron ESB architecture that I will be devoting a separate article to: it provides an abstraction layer for your enterprise architecture. This allows you to change the underlying technologies for communication and integration without being disruptive or forcing you to reinvent your architecture. More on that soon.

Neuron ESB 2.0 Released!

Tue, 29 Apr 2008 22:34:10 GMT

I've been guilty of not blogging for quite some time. No, I wasn't held hostage by terrorists... but close to it :) I and my fellow engineers have been heads-down, working around the clock to complete some very cool software--that I can now talk about. I'm very pleased to announce completion of Neuron ESB.

Neudesic has released version 2.0 of its Enterprise Service Bus product, Neuron ESB. Neuron ESB is a full-featured ESB based on the Microsoft platform. It provides a sophisticated communication and integration backbone with easy change management and a one-stop administrative experience. Neuron combines the capabilities and best practices of the SOA and EAI disciplines using proven Microsoft technologies. Neuron ESB accelerates SOA adoption by providing the necessary infrastructure services to complement your business services. Neuron communicates equally well with non-services such as legacy systems and LOB applications via adapters. Intelligent routing interconnects endpoints using publish-subscribe messaging and a named topic hierarchy. Mediation brings together programs that otherwise could not interact due to differences in protocol, security model, or message format. Integration services provide EAI functionality such as schema validation, message transformation, auditing, rules engine processing, and workflow execution. Event-based processing includes real-time notifications, the ability to link message patterns to business events, and complex event processing. ESBs can be zoned by location or administrative control, and zones can be federated. Neuron ESB is suitable for solutions ranging from departmental scope to enterprise-wide, mission-critical applications that span wide geographies.

Now that Neuron ESB is released, you can expect plenty of blogging from me about ESBs in general, Neuron ESB in particular, and how real-world customer challenges are met.

Have an ESB question? Interested in having someone come out and speak about ESBs? Drop me a note.

neuronesb.com

Tue, 05 Feb 2008 16:14:55 GMT

The Neuron ESB now has its own dedicated web site, http://www.neuronesb.com.

What's on the site? It's purpose is to provide easy access to:

Product information
Vertical solutions
News
White papers
Case studies and customer testimonials
Support resources
Discussion forum

Some of that content is up there now, and a lot more is on the way.

WCF Security Hydra

Tue, 29 Jan 2008 08:05:47 GMT

The Hydra was the many-headed serpent of Greek mythology. I have created a WCF security hydra by connecting 28 WCF services and clients that use each of the 14 common WCF security scenarios to the same Neuron ESB. All of the endpoints can operate at the same time, so the resemblance to the hydra is more than superficial. The bus can also mediate between endpoints, allowing clients and services of different security models to interact with each other (try that without an ESB and you won't get very far).

The scenarios are:

Internet unsecured
Intranet unsecured
Anonymous message security
Certificate message security
Issued Token (federated) message security
Kerberos message security
Mutual certificate message security
Username message security
Windows message security
Anonymous transport security
Basic transport security
Certificate transport security
Windows transport security
Trusted subsytem

As anyone who has played with WCF knows, the security capabilities of WCF are extensive but can be a bit overwhelming to master. It definitely feels like a major accomplishment to reach this point. I can now confidently implement each of these scenarios using WCF configuration, using WCF code, or using Neuron's ESB Explorer UI settings. All of this shows how capable WCF is, especially when combined with the Neuron ESB.

ESBs: Physically Decentralized, Logically Centralized

Sat, 20 Oct 2007 02:44:33 GMT

A characteristicsof ESBs is that they are physically decentralized but logically centralized. What exactly does this mean? To some this may sound like a contradiction.

This simply means that a bus is physically more like the point-to-point architecture (spread out, no hub) but functionally more like the hub-and-spoke architecture (pub-sub messaging, centralized configuration and activity tracking, easy change management).

Envision a football team. The players are dispersed over the playing field, but they are executing on a common game plan. Although the players are physically distributed, their actions are in concert. And taking out one player doesn't kill the team.

An ESB has servers that contain infrastructure services, but there is no hub and no single point of failure. Each application connects to the bus through a lightweight client container which interacts with the infrastructure services to perform messaging and integration functions. For those who find this explanation insufficiently technical, understand that there's more than one way to build an ESB. Just as you could use a variety of technologies to build a point-to-point architecture (WCF? .NET Remoting? CORBA?) or a hub-and-spoke architecture (BizTalk? JMS?), there's more than one way to build a bus. Here's how we do it in Neuron, our Microsoft-based ESB.

The Infrastructure Controversy

Fri, 05 Oct 2007 17:32:36 GMT

As hot debates rage over SOA / ESB / REST, the central controversy seems to boil down to a fairly simple question: how much infrastructure is appropriate in an enterprise for communication and integration?

Not only is the question simple, but there are really only 3 possible positions to take and they align with 3 different architectures.

1. No Infrastructure / Point-to-Point architecture

2. A Dominant Infrastructure / Hub-and-Spoke architecture

3. A Thin Infrastructure / Bus architecture

Even though I believe approach #3 works best most often in an enterprise setting, I'll freely admit there's a place for all 3 approaches. Let's look at the consequenecs of choosing each of these approaches.

The No Infrastructure / Point-to-Point Approach

In this approach applications are directly connected to each other. Using SOA best practices and WS-* compatible technologies like WCF, it's fairly easy to create solutions.

Con: You only have point-to-point communication. There's no intelligent routing. Your applications are not loosely coupled, they're tightly coupled.

Con: You have no integration facilities. What happens when you need to introduce a transformation between systems to accomomdate a versioning issue?

Con: There's no facility for change management. Making any kind of change, such as adding a new application to the mix or, always forces you to make code changes to one or more of your existing applications.

The bottom line: This model works okay at a limited level, such as a departmental solution, if your needs are high on communication and low on integration. But it's far too limited and rigid to use at enterprise scale. Change management is hard, not easy.

The Dominant Infrastructure / Hub-and-Spoke Approach

In this approach you likely have a major product at the center of a hub-and-spoke architecture. The product at the hub does a lot of nice things for you, such as intelligent routing, transformation of messages, application of rules, and orchestrated business processing. Change management is significantly easier.

Con: The hub can be a single point of failure.

Con: The hub can be a congestion point (bottleneck).

Con: The hub tends to be a large, complex software product. This goes against the SOA trend towards small, distibuted software.

Con: You are probably eternally tied to a proprietary commercial product. Your freedom of choice is severely curtailed.

Con: Moreover, you have to hire people who specialize in the EAI product.

The bottom line: This model is a big improvement over point-to-point, but overly thick and centralized infrastructures are going out of style. The days where a customer standardizes on a single product for EAI functions like transformations, rules engine, workflow, integration etc. are over. Customers want granular freedom of choice about the platforms and products they use to power EAI functions, and the ability to change their minds. They certainly don't want to be forcibly chained to any particular product.

The Thin Infrastructure / Bus Approach

In this approach you have a thin infrastructure that is physically distributed but logically centralized, and it grows as your needs grow. Business endpoints connect to the bus through lightweight software containers instead of talking to a hub. EAI functionality is provided through integration services, and each service can be powered by any platform/technology/product the customer chooses. For example, a customer could power a transformation service with BizTalk and power a Rules Engine service with some other product. Change management is very easy.

The bottom line: This approach avoids the two extremes of no infrastructure and too much infrastructure, and preserves customer freedom of choice to change out product(s) while the architecture remains constant.

Con: ESBs are new, and often misunderstood. I genuinely believe the reason many people are arguing against ESBs is that they confuse them with the hub-and-spoke product-centric approach.

Con: Some commercial ESBs violate their own principles and tie you to an expensive product. Obviously these are to be avoided.

ESB: Appendix or Skeleton?

Mon, 17 Sep 2007 15:55:14 GMT

Is an ESB about as useful as a human appendix? A recent ZDNet article, ESB: Useless as a Human Appendix?, quotes an article by Bobby Woolf that raises this question.

"ESB-oriented architecture is inherently flawed in that it builds connectivity no one might ever want to use. The business does not derive additional value until systems connect to each other and are working together. Until then, the ESB is just cost with no benefit. It might make the IT department feel good because they have built something, but it will not make the business feel any better, because the business is not accomplishing anything it couldn’t have already accomplished without the ESB. The ESB becomes the equivalent of a human appendix for the IT department, a vestigial organ within the topology of deployed applications"

The point being made in the article is not that ESBs are worthless, but that they shouldn't be built for their own sake. In other words, infrastructure should be built when it is needed, not because it might be needed someday.

I think most people would generally agree with this sentiment. Moreover, there's nothing special about "ESB" in this thought. It has always been a wise practice to build infrastructure you need, not infrastructure you might need. Unfortunately, many ESBs (and ESB-wannabes) are compromised of fat products where you either get everything or nothing. That's a sign of providing ESB capabilities without the right architecture to go with it.

A good ESB design gives you a thinfrastructure™ that you expand with more functionality when you need capabilities like transformation, a rules engine, or adapters. In the case of the Neuron ESB, you add the heavy hitters (rules service, BAM service, transformation service, management auditing, etc..) as you have need for them, in the form of new services. Nothing forces them on a customer in advance, or necessarily ever.

A properly designed ESB isn’t an appendix. It’s the skeleton.

The ESB as a Business Eventing System

Sat, 10 Mar 2007 20:54:07 GMT

In explaining the Enterprise Service Bus to audiences, I often make the point that an ESB is cross-discipline, drawing the best ideas from SOA, EAI, MOM, and EDA. Often, SOA and EAI receive the most attention in these discussions because the people I'm takling to have extensive backgrounds in one or the other. Today, I'd like to focus specifically on the Event Driven Architecture (EDA) aspect of an ESB. One of the most exciting things an ESB gives you is a business-wide eventing system.

Events are a simple idea, which makes them convenient for technical and non-technical people alike. When learning how a business works, business owners, managers, and analysts will often naturally describe business proceses in terms of events and who-does-what in response to those events. Events are doubly nice because they can be technically implemented very simply, as messages.

In an ESB, mapping an event model to how programs interact is a snap. All programs communicate through publish-subscribe topics. For example, an Order Entry program may produce orders, which are sent over the ESB on the topic "orders.new". An Order Processing service receives orders because it has a subscription to "orders.new" or perhaps "orders.*". With a sufficiently granular hierarchy of topic names, it's usually very straightforward to map an event-oriented discussion of business processing to a publish-subscribe model. When more than simple sequencing is needed, orchestration is added to the mix to provide deluxe worfklow and transaction control.

Too often in our industry, when business requirements gathering gives way to technical design and implementation, something gets "lost in translation" and the result is not quite what was intended. This is where the ESB and publish-subscribe gets exciting, because the same model that was used to describe the business non-technically is maintained throughout design and implementation. That is, when everything is said and done the programs in the enterprise are communicating by signaling events over logical business topics, still using the same eventing model that was originally described by the business owners. Keeping the same eventing model from requirements gathering to implementation avoids much of the "lost in translation" problem. And that's a big improvement over the past.

ESB presentation at So. Cal Architecture User Group

Fri, 23 Feb 2007 00:56:19 GMT

Last week I presented on ESB to the Southern California .NET Architecture User Group which met in Santa Ana. About 40 people were present and we had some great discussions. This is a group of very intelligent people.

Making the Case for ESB: Architecturally

Tue, 26 Dec 2006 18:38:26 GMT

The industry continues to debate whether or not it likes the Enterprise Service Bus (ESB) concept. As an ESB advocate, I think the case for an ESB can be made on several levels. Today, I'd like to focus on why the ESB makes sense architecturally.

Figure 1 shows some comparative architectures. Let's consider the first two.

Point to Point architectures (Figure 1, left) started getting a bad name in the last decade. If each system in your enterprise needs to connect to most/all of the other systems in your enterprise, you soon encounter the "n squared" problem where the addition of each new system increasingly adds to your integration and management costs. It's difficult to configure and manage systems in a point-to-point architecture, or to ensure that policies are enforced across the enterprise.
Hub and Spoke architectures (Figure 2, center) integrate everything to a central hub. With centralized configuration, a reduced number of integration points, and centralized tracking, hub-and-spoke is a big improvement over point-to-point architectures. But hub-and-spoke is now falling into disfavor: the hub is undesirable because it can be a single point of failure, because it can be a congestion point, and because it typically ends up being a very complex piece of software that may also tie you eternally to a single EAI product.

It would seem, then, that a point-to-point architecture is overly decentralized while a hub-and-spoke architecture is overly centralized. There's another option, but many people seem unwilling to shift from what they're used to. Many EAI people cling to hub-and-spoke, perhaps because their preferred server products are designed around this architecture and it's all they're familiar with. Many SOA people seem to be suggesting that we go back to point-to-point architectures, but don't explain how the classic problems of point-to-point will be resolved.

Let's examine an alternative architecture, the bus architecture. The idea of the bus architecture is simple but profound: logically centralized, physically decentralized. In other words, the parts that make up the system are distributed, but they cooperatively implement what looks, feels, and acts like a centralized system. This is analogous to how a football team works: the players are spread out over the playing field, but they're working together to execute a common game plan.

In an ESB, the bus architecture (Figure 1, right) provides a set of infrastructure services that provide essential services such as intelligent routing, security, mediation, and configuration. These infrastructure services can be physically distributed but work together to enforce the enterprise's processes and policies. Connected to the infrastructure services are business endpoints and integration endpoints. Business endpoints are business programs, accessed either directly as services or through adapter mechanisms. Integration endpoints provide EAI functionality such as transformations, orchestrations, and rules engines; they are applied to messages en route between business endpoints.

The ESB is consistent with SOA principles. Use of services is highly encouraged. All parts of the ESB are loosely coupled. Business, integration, and infrastructure endpoints are interconnected using SOAP and WS-* standards. The ESB complements all of this with added functionality such as publish-subscribe routing and centralized management.

The ESB is consistent with EAI principles. Native applications and legacy systems are full participants in the ESB, through the use of service wrappers and/or integration adapters. Messages can be routed by topical pub-sub, by payload type (schema), or by content. Transformations provide the expected type of message for every endpoint. Orchestrations provide transacted business process execution. The ESB provides all of this with added value such as a distributed architecture and the ability to choose and combine best-of-breed products.

Ultimately, an ESB is about providing a great deal of additional value through the use of infrastructure services that complement your business services. This should not really seem like a new or unusual idea, as we're already used to having infrastructure systems in our enterprises (domain controllers and active directory servers, for example).

The ESB is not only an architecture that makes a lot of sense, it has a long-term appeal. The ability to switch out underlying implementations of services means an ESB architecture is tolerant of changes in technology over time. Yet, some ESB opponents have concluded just the opposite and argue that the ESB is a temporary phenomenon that will not be needed once additional service standards are developed. To me, that seems as likely as having football teams without coaches and orchestras without conductors.

In Search of Publish-Subscribe

Wed, 04 Oct 2006 18:19:32 GMT

For most of the last year I've been working on an exciting project, one major component of which requires publish-subscribe communication. Pub-sub continues to grow in popularity because it offers many benefits:

The connections between applications are loosely coupled.
Applications no longer need to know the addresses of other applications.
Routing of information is determined by central configuration rather than details embedded in application code, and is easily changed.
New applications can easily be hooked in to a system simply by creating a new subscription for them.

For all these reasons and more, we at Neudesic have been considering the best way to accomplish Publish-Subscribe in the Microsoft world. To find the best way, you have to begin by evaluating your options. These are the options we looked at:

BizTalk Server does have a pub-sub mechanism at the core of its messaging engine, but it's not a general-purpose pub-sub communication facility that .NET applications can get at or use. Its main purpose is to connect senders and receivers with the BizTalk message box across one or more servers.
WCF doesn't ship with any specific support for publish-subscribe, though we might expect to see something along those lines in the future. However, there is WCF PeerChannel, which creates a multi-party cloud or mesh. Any message sent to the cloud goes to all parties. Since you can set up as many clouds as you wish, you could have one cloud per topic. WCF PeerChannel has some attractive features, including low latency (fast) communication and the ability to scale up to 10s of 1000's of nodes without requiring an investment in server infrastructure. However, the information is not transacted or durable and there are no delivery assurances.
MSMQ seemed to be leaning toward a pub-sub set of features in its past but information on this is hard to come by lately. There is MSMQ over UDP, which broadcasts messages to large numbers of recipients. In the enterprises I've worked with where this is in use, the loss rate is often considered unacceptably high. Alternatively, you could make use of MSMQ queues--since most pub-sub systems are based on queues--and use custom code to move messages between queues.
SQL Server could be used as a message repository if you were going to build your own durable publish-subscribe system. SQL Server 2005 also offers communication features through its Service Broker including its own message queue facility. But you'd certainly be writing a lot of custom code to publish and receive messages and to move messages between queues.
WS-Eventing is a standard that allows differnet messaging systems to subscribe to each others messages using web services as a way to receive messages. This standard describes the mechanisms of requesting and managing subscriptions, but doesn't spell out how pub-sub itself should be implemented.

In trying to determine which of the above is "best", it became clear that each possibibility possesses some unique capabilities that some enterprises might find important. For example, the speed, reliability, durability, and recovery story for each of the above varies. Moreover, enterprises vary in which technologies they have invested in, so a BizTalk solution might be most attractive for one enterprise but less attractive to another.

Our decision was to support all of these methods of publish-subscribe with an abstracted interface that can perform the same messaging capabilities across any of them. You can read about how we did this in our ESB White Paper (part1, part2). The benefits of this approach are many:

Customers can take advantage of the technologies and products they have an existing investment in
Customers can run several different qualities of service in parallel
The modular driver approach makes it straightforward to graft in new technologies as they become available