This Sunday I'll be speaking at the San Diego Code Camp http://www.socalcodecamp.com/sessions.aspx

 

I have a 12:30 session on the Neuron ESB and a 3:45 session on SOA Design Patterns and how to implement them using Microsoft technologies.

 

Tonight I'll speaking this evening at the L.A. .NET Users Group in Los Angeles.  http://www.ladotnet.org/default.asp

 

My topic is, The Programmer's Guide to Quality Assurance

 

I've been amassing this content for some time (may become a book someday). If I get a good reception to it, I may start blogging it.

 

 

 

Many people think of Southern Califonia as having day after day of sunny weather. And often enough, that's true: I remember one year in the 90s where we literally had 200 straight days of sunshine in a row (they say the two most boring places to be a weatherman are L.A. and Seattle).

 

But not today. Today was probably the craziest day in California weather history. In the greater L.A. area, all of the following was going on, in various parts, all at the same time:

 

rainstorms

strong winds

mudslides

hail

snow

thunderstoms

tornadoes

 

That's right, snow. In California. In late May.

 

It's the torndaes (plural) that got my attention, because they were about 7 miles from my house for awhile. They were powerful enough to overturn trucks and railroad boxcars. Photos below.

 

 

Did I say today was the craziest day in California weather history? The news just gave tomorrow's forecast: "more of same."

 

Neuron ESB and Endpoints

Today I’d like to talk about how Neuron ESB connects to physical endpoints in the enterprise. Two blog posts ago I talked about Neuron’s topic network architecture, and in my last blog post I focused on the internal side of the bus, where channels implement pub-sub over a variety of Microsoft technologies ranging from WCF to BizTalk Server. But just how do those publisher and subscriber nodes relate to the physical world of services and clients? And what about non-service applications? We’ll answer that here as we talk about the external side of the bus.

First off, let’s mention in passing that one way you can program against Neuron is to write .NET programs using Neuron’s API. If you do that, you are directly creating a publisher or subscriber node. We’ll discuss the API and when you might choose to use it at another time. For now, we want to focus on scenarios where you don’t want to do that. Rather, you have existing endpoints in the enterprise and you want Neuron to be able to talk to them.

External endpoints fall into 2 distinct categories: they’re either service-oriented or they aren’t.

·         Service endpoints are service-oriented applications which of course come in two roles: clients (which initiate messaging) and services (which respond to messages).

·         Adapter endpoints are non-service oriented applications such as a legacy system or a line-of-business application. Neuron can access these kinds of endpoints using adapters. Like service endpoints, adapter endpoints can be send-oriented (resembling clients) or receive-oriented (resembling services).

Neuron can communicate with both service and adapter endpoints in request-reply or 1-way messaging patterns.

Service Endpoints

Service endpoints are first- or second-generation service programs. Neuron uses WCF to interact with service endpoints and thus inherits the ability to interconnect to services via a wide variety of transports and security models.

A web client connects to the ESB through a client connector (Figure 1). The client connector is both a WCF endpoint that the client can send messages to and a bridge to a Neuron publisher node. The client connector is mapped to a Neuron publisher Id and topic: when a client sends a message, the client connector receives the message and publishes it over the bus using the assigned publisher and topic. As described in a previous post on service pass-through, the hosting port on the ESB server looks to the client just like its target service, except that the address is different.  

 

 Figure 1: Client and Service Connectors

A service is connected to the ESB through a service connector (Figure 1) and acts reciprocally to a client connector as you would expect. The service connector is a Neuron subscriber node and a bridge to a WCF channel by which messages can be sent to the service. The service connector is mapped to a Neuron subscriber Id. When the subscriber receives a message over the bus, the service connector relays it to the service.

The bus gives you the freedom to enjoy a great variety of configuration possibilities that would be impossible if you directly connected your clients and services.

·         Loosely-coupled routing. Although Figure 1 only shows 2 endpoints, there could be any number of them communicating together. Doing so through the bus is far more attractive than the alternative, where services become clients of other services; such tightly coupled routing requires every module to know the connection details of every other module. It’s much smoother to communicate by named topics.

·         Mediation. There’s no requirement for the client and service endpoints to use the same method of communication or the same service model. For example, you could have a client that uses username/password security interacting with a service that uses X.509 certificate security through the bus. Even programs that have a different idea about message format or version can be made to work together by having the bus perform transformations on messages as they are en route.

·         Workload Sharing. You can have more than one instance of the same service, and the bus can perform workload sharing among them.

·         Service Policy. What should happen to a message if the target service is not available to receive it? Neuron lets you specify policy to control delivery aspects such as retries and failover to alternate destinations.

·         Services and Non-Services Working Together. You’re completely free to have your clients and services interact with non-services. The bus extends the reach of your services and other I.T. assets.

Adapter Endpoints

Enterprises often have line of business applications and/or legacy systems that are core to the business but are not service-oriented. Often these applications are “silos” in the sense that they have little or no interaction with other applications in the enterprise. Neuron provides comprehensive adapter support to bring these other systems onto the bus and make them accessible to the rest of the enterprise.

Neuron supports 2 families of adapters:

·         WCF LOB Adapters are the successor to BizTalk adapters. Adapters have already been developed for Oracle database, SAP and Siebel by Microsoft, and many other adapters are in developed by Microsoft  and other members of the community. As more and more of these adapters are developed, they benefit not only WCF and BizTalk Server but Neuron ESB as well.

·         Neuron .NET Adapters are Neuron’s own adapter model. They have an interface that makes them very easy to write (the design goal was that field engineers could write one on site if necessary).Neuron ships with a versatile collection of these adapters.

For Neuron .NET adapters, an endpoint connector (Figure 2) works analogously to a service connector or a client connector. WCF LOB Adapters are a different matter: because they are designed to look like WCF bindings, they can be used with service connectors or client connectors.

 

Figure 2: Endpoint Connectors

Just like service endpoints, adapter endpoints can operate in a client/sender role or a service/receive role; and just like service endpoints, adapter endpoints can utilize request-reply or 1-way messaging. An adapter defines one or more modes of behavior that define what it does and which direction and pattern messaging the mode needs.

To talk to a send mode adapter endpoint, the endpoint connector acts like a service connector, relaying bus messages to the adapter. The endpoint connector has an associated Neuron subscriber on which it receives bus messages which are then passed on to the adapter for delivery.

To talk to a receive mode adapter endpoint, the endpoint connector acts like a client connector, relaying adapter messages to the bus. The endpoint connector has an associated Neuron publisher on which incoming messages from the adapter are published onto the bus over a designated topic.

Interaction with adapters is optionally transacted: both Neuron adapters and Neuron pub-sub channels may implement transactions, and if they do the hand-off between them is also linked transactionally. This makes it possible to have transacted end-to-end communication through the bus, as long as the channel and adapter components you select support transactions.

As with service/client connectors, you may set up adapter policies to control the behavior of adapter endpoints in the face of errors. You can determine retry counts and actions to take when a message can’t be delivered.

The adapters that ship with Neuron include ActiveMQ,  RFID via BizTalk Server, eConnect (Great Plains accounting), SMTP email, files, MSMQ, SQL Server, and WebSphere MQ) and we’ll be taking more about them individually down the road.

Putting it all together: Service and Adapter Endpoints

Neuron’s strong support for services and adapters lets you break down the walls between your SOA applications, legacy systems, and line of

Neuron’s strong support for services and adapters lets you break down the walls between your SOA applications, legacy systems, and line of business applications, and other groups of systems that up till now have had limited reach.

All of your systems are now connected, but with loosely-coupled routing through named pub-sub topics. Actions such as changing the connection specifics of an endpoint, adding new endpoints, or removing endpoints can be done without disrupting the rest of the enterprise and without the need to make code changes to existing programs.

 

The Neuron ESB Channel Model

In my previous post I discussed the importance of topics in Neuron and how each topic is implemented as an individual publish-subscribe network, called a topic network. Topic networks are based on a modular channel model and can be powered by technologies ranging from WCF peer networking to BizTalk Server. Today we’ll look at the channel model in more detail.

ESBs have endpoint architecture, meaning there is a common stack all endpoints go through in order to interact with each other. This is conceptually no different than using a WCF client to talk to a WCF service: a common stack on both ends of the connection work together to facilitate communication and cooperative processing. The difference with the ESB stack is the functionality provided, which includes publish-subscribe communication to N parties and the ability to layer integration functionality (transformations, rules, workflow) into your messaging at any point.

How does the ESB stack pull off the publish-subscribe communication between  N parties? You might assume all those ESB stacks simply connect to an ESB Server in a hub-and-spoke arrangement, but that’s not the case at all. Or more accurately, it’s up to the channel. A channel is simply a software module that implements publish-subscribe using some base technology. The four channel types included with Neuron implement pub-sub over peer networking, WCF TCP, MSMQ, and BizTalk Server. Each of the channels provide the same functionality but with varying qualities of service. Since each topic is a separate channel choice, you are free to run different qualities of service side by side. You might, for example, favor speed for some business conversations but require reliability for others.

Although the channel’s responsibility is to implement publish-subscribe, applications are not limited to that one messaging pattern. Neuron’s stack also allows request-reply, direct point-to-point connections, and itinerary routing (where the message determines the delivery points). All of these patterns are always available, regardless of which channel you use.

All publish-subscribe systems have to deal with backlog, where published messages come in faster than they can be delivered. A channel’s implementation must provide a strategy for handling backlog.

Peer Networking Channel

Peer Channel uses Windows peer-to-peer networking. In WCF, this technology is packaged up as the NetPeerTcpBinding. With PeerChannel you create a cloud or mesh of messaging nodes. The nodes are interconnected using both TCP and UDP, and will also take advantage of IPv6 on the network if they are able to. Any message sent by a node is delivered to all others nodes on the mesh in a highly parallel fashion that doesn’t require server infrastructure. A mesh is self-organizing, changing its shape in response to nodes joining/leaving and activity. This automatic optimization keeps the number of hops to a minimum and slower nodes are moved to the outside of the mesh.

Peer networking often brings to mind consumer applications such as instant messaging, music and file sharing, or multi-player gaming. However, there are enterprise scenarios where peer networking is ideal. Its great strengths are its high speed, its parallelism, and its scalability.

In Neuron, each topic powered by Peer Channel is a separate mesh. The behavior of a PeerChannel mesh is a natural fit for publish-subscribe messaging. Neuron deployments have been known to operate meshes with as many as 1500 nodes that stretch across the U.S. over multiple geographic locations.

There’s a catch with peer channel, however. The base technology is not reliable enough to suit most enterprises. There are no message delivery assurances:  you can lose messages (sometimes large blocks of messages if there’s a lot of CPU contention) nor are messages guaranteed to arrive in order. The good news is that Neuron ESB provides an optional reliability layer that provides delivery assurances through message caching.

In its native form, peer channel is server-less. In fact, you can shut down all of your servers and parties can continue to message. Adding in the reliability layer or enabling certain features such as auditing will require you to keep at least one server up and running however.

The peer networking channel  is volatile, meaning there is no durable storage of messages. Performance is very good, but is limited by the slowest node in the mesh.

Peer channel  is best suited for scenarios requiring low-latency (high speed) communication and/or the need to broadcast to many recipients simultaneously.

WCF TCP Channel

The WCF TCP channel interconnects parties using WCF TCP with reliable messaging. The channel connects parties to an ESB server via TCP. A massively efficient TCP publishing engine on the server copies incoming published messages out to connected subscribers with matching subscriptions.

The server publishing engine can become backlogged, where published messages are coming in faster than they can be relayed out. The engine will buffer backlogged messages up to a threshold; once that threshold is exceeded, flow control kicks in and publishers will be slowed down as needed.

In a server farm scenario, each server will have a different collection of parties connected to it. The servers share incoming messages with each other laterally so that any incoming messages is relayed to all servers for distribution.

 The WCF TCP channel is volatile, meaning there is no durable storage of messages. There are delivery assurances within the scope of a connection, but sudden client or server loss can lose messages.

The TCP channel is a good choice for applications that don’t require durable or transacted messaging.

MSMQ Channel

The MSMQ channel implements publish-subscribe over queues. Messaging is durable, transacted, and reliable. There is a publish queue for a topic, and a recipient queue for each subscriber. An MSMQ publishing engine on the server moves messages between queues.

Parties do not have to be online in order to receive messages from each other because undelivered messages persist in the queues. This characteristic allows you to take systems offline and online without being disruptive.

The MSMQ queues used by the channel are durable, providing reliable store-and-forward messaging. In a backlog situation, messages that cannot be immediately processed accrue in publisher or subscriber queues until they can be processed.

The server’s MSMQ publishing engine is transactional. A delivery either fully completes or the queue reads and writes are rolled back. The engine processes messages in batches when it can for efficiency.

The MSMQ channel is a good choice for transacted, durable communication.

BizTalk TCP Channel

The BizTalk TCP channel performs publish-subscribe over BizTalk Server. The channel connects parties to BizTalk Server via TCP, and BizTalk’s messaging engine performs the publish-subscribe messaging.

Making changes to the Neuron ESB  configuration automatically programs BizTalk Server to make matching changes to its configuration. For example, defining new publishers and subscribers in Neuron will cause new send ports and receive ports to be created in BizTalk.

With BizTalk as the messaging engine, you gain the many features of BizTalk including strong retry/recovery capabilities. With a small amount of BizTalk configuration you can link messages to features such as Business Activity Monitoring (BAM),, transformations , the rules engine, and orchestrations.

The BizTalk channel is durable and is well-suited for mission-critical applications that require high reliability and/or the unique characteristics and features of BizTalk Server’s. It is not the best choice for low-latency applications, as BizTalk’s message box database limits performance.

What about Services, Legacy Systems, and Line of Business Apps?

Does this mean there are only 4 protocols you can use to communicate with in Neuron? Not at all--the endpoint architecture and channel model we’ve been describing is how Neuron makes publish-subscribe happen, but this is the internal side of the bus. Externally, you can connect a wide, wide variety of services and non-services to the bus… everything from WCF services to WebSphereMQ. We will talk about the external side of the bus in future articles.

Summary

The Neuron ESB channel model provides a modular system for switching in different technologies for messaging. This not only makes it possible to leverage a wide range of existing technologies, but anticipates new technologies and provides a path for accommodating them.