David Pallmann's Blog

San Diego Code Camp

Fri, 27 Jun 2008 14:37:48 GMT

This Sunday I'll be speaking at the San Diego Code Camp http://www.socalcodecamp.com/sessions.aspx

I have a 12:30 session on the Neuron ESB and a 3:45 session on SOA Design Patterns and how to implement them using Microsoft technologies.

The Programmer's Guide to Quality Assurance

Mon, 02 Jun 2008 19:18:21 GMT

Tonight I'll speaking this evening at the L.A. .NET Users Group in Los Angeles. http://www.ladotnet.org/default.asp

My topic is, The Programmer's Guide to Quality Assurance

I've been amassing this content for some time (may become a book someday). If I get a good reception to it, I may start blogging it.

Licensed to WSDL

Fri, 23 May 2008 20:23:15 GMT

A long, long time ago in a galaxy far away... way back in January... I decided to order a personalized license plate advertising my area of specialization, SOA and ESB. Little did I know this was going to be an extreme exercise in patience. I finally was able to pick up the plate shown below and puit it on my car today (in May), after a grueling 4-month wait.

I originally wanted to have it read NEURON, the name of our ESB, but that was already taken (by a neurologist, no doubt). Interestingly, I have discovered the acronym ESB is also a neurological term (meaning, "Electrical Stimulation of the Brain"). So if you're ever googling for Neuron ESB and get some medical hits, you'll know why.

Now everyone will know what drives me. :) Or more accurately, the computer people will and no one else will have a clue.

Crazy Weather Day in California - Including Tornadoes

Fri, 23 May 2008 03:25:56 GMT

Many people think of Southern Califonia as having day after day of sunny weather. And often enough, that's true: I remember one year in the 90s where we literally had 200 straight days of sunshine in a row (they say the two most boring places to be a weatherman are L.A. and Seattle).

But not today. Today was probably the craziest day in California weather history. In the greater L.A. area, all of the following was going on, in various parts, all at the same time:

rainstorms

strong winds

mudslides

hail

snow

thunderstoms

tornadoes

That's right, snow. In California. In late May.

It's the torndaes (plural) that got my attention, because they were about 7 miles from my house for awhile. They were powerful enough to overturn trucks and railroad boxcars. Photos below.

Did I say today was the craziest day in California weather history? The news just gave tomorrow's forecast: "more of same."

Neuron ESB and Endpoints

Mon, 19 May 2008 01:28:11 GMT

Neuron ESB and Endpoints

Today I’d like to talk about how Neuron ESB connects to physical endpoints in the enterprise. Two blog posts ago I talked about Neuron’s topic network architecture, and in my last blog post I focused on the internal side of the bus, where channels implement pub-sub over a variety of Microsoft technologies ranging from WCF to BizTalk Server. But just how do those publisher and subscriber nodes relate to the physical world of services and clients? And what about non-service applications? We’ll answer that here as we talk about the external side of the bus.

First off, let’s mention in passing that one way you can program against Neuron is to write .NET programs using Neuron’s API. If you do that, you are directly creating a publisher or subscriber node. We’ll discuss the API and when you might choose to use it at another time. For now, we want to focus on scenarios where you don’t want to do that. Rather, you have existing endpoints in the enterprise and you want Neuron to be able to talk to them.

External endpoints fall into 2 distinct categories: they’re either service-oriented or they aren’t.

· Service endpoints are service-oriented applications which of course come in two roles: clients (which initiate messaging) and services (which respond to messages).

· Adapter endpoints are non-service oriented applications such as a legacy system or a line-of-business application. Neuron can access these kinds of endpoints using adapters. Like service endpoints, adapter endpoints can be send-oriented (resembling clients) or receive-oriented (resembling services).

Neuron can communicate with both service and adapter endpoints in request-reply or 1-way messaging patterns.

Service Endpoints

Service endpoints are first- or second-generation service programs. Neuron uses WCF to interact with service endpoints and thus inherits the ability to interconnect to services via a wide variety of transports and security models.

A web client connects to the ESB through a client connector (Figure 1). The client connector is both a WCF endpoint that the client can send messages to and a bridge to a Neuron publisher node. The client connector is mapped to a Neuron publisher Id and topic: when a client sends a message, the client connector receives the message and publishes it over the bus using the assigned publisher and topic. As described in a previous post on service pass-through, the hosting port on the ESB server looks to the client just like its target service, except that the address is different.

Figure 1: Client and Service Connectors

A service is connected to the ESB through a service connector (Figure 1) and acts reciprocally to a client connector as you would expect. The service connector is a Neuron subscriber node and a bridge to a WCF channel by which messages can be sent to the service. The service connector is mapped to a Neuron subscriber Id. When the subscriber receives a message over the bus, the service connector relays it to the service.

The bus gives you the freedom to enjoy a great variety of configuration possibilities that would be impossible if you directly connected your clients and services.

· Loosely-coupled routing. Although Figure 1 only shows 2 endpoints, there could be any number of them communicating together. Doing so through the bus is far more attractive than the alternative, where services become clients of other services; such tightly coupled routing requires every module to know the connection details of every other module. It’s much smoother to communicate by named topics.

· Mediation. There’s no requirement for the client and service endpoints to use the same method of communication or the same service model. For example, you could have a client that uses username/password security interacting with a service that uses X.509 certificate security through the bus. Even programs that have a different idea about message format or version can be made to work together by having the bus perform transformations on messages as they are en route.

· Workload Sharing. You can have more than one instance of the same service, and the bus can perform workload sharing among them.

· Service Policy. What should happen to a message if the target service is not available to receive it? Neuron lets you specify policy to control delivery aspects such as retries and failover to alternate destinations.

· Services and Non-Services Working Together. You’re completely free to have your clients and services interact with non-services. The bus extends the reach of your services and other I.T. assets.

Adapter Endpoints

Enterprises often have line of business applications and/or legacy systems that are core to the business but are not service-oriented. Often these applications are “silos” in the sense that they have little or no interaction with other applications in the enterprise. Neuron provides comprehensive adapter support to bring these other systems onto the bus and make them accessible to the rest of the enterprise.

Neuron supports 2 families of adapters:

· WCF LOB Adapters are the successor to BizTalk adapters. Adapters have already been developed for Oracle database, SAP and Siebel by Microsoft, and many other adapters are in developed by Microsoft and other members of the community. As more and more of these adapters are developed, they benefit not only WCF and BizTalk Server but Neuron ESB as well.

· Neuron .NET Adapters are Neuron’s own adapter model. They have an interface that makes them very easy to write (the design goal was that field engineers could write one on site if necessary).Neuron ships with a versatile collection of these adapters.

For Neuron .NET adapters, an endpoint connector (Figure 2) works analogously to a service connector or a client connector. WCF LOB Adapters are a different matter: because they are designed to look like WCF bindings, they can be used with service connectors or client connectors.

Figure 2: Endpoint Connectors

Just like service endpoints, adapter endpoints can operate in a client/sender role or a service/receive role; and just like service endpoints, adapter endpoints can utilize request-reply or 1-way messaging. An adapter defines one or more modes of behavior that define what it does and which direction and pattern messaging the mode needs.

To talk to a send mode adapter endpoint, the endpoint connector acts like a service connector, relaying bus messages to the adapter. The endpoint connector has an associated Neuron subscriber on which it receives bus messages which are then passed on to the adapter for delivery.

To talk to a receive mode adapter endpoint, the endpoint connector acts like a client connector, relaying adapter messages to the bus. The endpoint connector has an associated Neuron publisher on which incoming messages from the adapter are published onto the bus over a designated topic.

Interaction with adapters is optionally transacted: both Neuron adapters and Neuron pub-sub channels may implement transactions, and if they do the hand-off between them is also linked transactionally. This makes it possible to have transacted end-to-end communication through the bus, as long as the channel and adapter components you select support transactions.

As with service/client connectors, you may set up adapter policies to control the behavior of adapter endpoints in the face of errors. You can determine retry counts and actions to take when a message can’t be delivered.

The adapters that ship with Neuron include ActiveMQ, RFID via BizTalk Server, eConnect (Great Plains accounting), SMTP email, files, MSMQ, SQL Server, and WebSphere MQ) and we’ll be taking more about them individually down the road.

Putting it all together: Service and Adapter Endpoints

Neuron’s strong support for services and adapters lets you break down the walls between your SOA applications, legacy systems, and line of

Neuron’s strong support for services and adapters lets you break down the walls between your SOA applications, legacy systems, and line of business applications, and other groups of systems that up till now have had limited reach.

All of your systems are now connected, but with loosely-coupled routing through named pub-sub topics. Actions such as changing the connection specifics of an endpoint, adding new endpoints, or removing endpoints can be done without disrupting the rest of the enterprise and without the need to make code changes to existing programs.

The Neuron ESB Channel Model

Wed, 14 May 2008 19:43:03 GMT

The Neuron ESB Channel Model

In my previous post I discussed the importance of topics in Neuron and how each topic is implemented as an individual publish-subscribe network, called a topic network. Topic networks are based on a modular channel model and can be powered by technologies ranging from WCF peer networking to BizTalk Server. Today we’ll look at the channel model in more detail.

ESBs have endpoint architecture, meaning there is a common stack all endpoints go through in order to interact with each other. This is conceptually no different than using a WCF client to talk to a WCF service: a common stack on both ends of the connection work together to facilitate communication and cooperative processing. The difference with the ESB stack is the functionality provided, which includes publish-subscribe communication to N parties and the ability to layer integration functionality (transformations, rules, workflow) into your messaging at any point.

How does the ESB stack pull off the publish-subscribe communication between N parties? You might assume all those ESB stacks simply connect to an ESB Server in a hub-and-spoke arrangement, but that’s not the case at all. Or more accurately, it’s up to the channel. A channel is simply a software module that implements publish-subscribe using some base technology. The four channel types included with Neuron implement pub-sub over peer networking, WCF TCP, MSMQ, and BizTalk Server. Each of the channels provide the same functionality but with varying qualities of service. Since each topic is a separate channel choice, you are free to run different qualities of service side by side. You might, for example, favor speed for some business conversations but require reliability for others.

Although the channel’s responsibility is to implement publish-subscribe, applications are not limited to that one messaging pattern. Neuron’s stack also allows request-reply, direct point-to-point connections, and itinerary routing (where the message determines the delivery points). All of these patterns are always available, regardless of which channel you use.

All publish-subscribe systems have to deal with backlog, where published messages come in faster than they can be delivered. A channel’s implementation must provide a strategy for handling backlog.

Peer Networking Channel

Peer Channel uses Windows peer-to-peer networking. In WCF, this technology is packaged up as the NetPeerTcpBinding. With PeerChannel you create a cloud or mesh of messaging nodes. The nodes are interconnected using both TCP and UDP, and will also take advantage of IPv6 on the network if they are able to. Any message sent by a node is delivered to all others nodes on the mesh in a highly parallel fashion that doesn’t require server infrastructure. A mesh is self-organizing, changing its shape in response to nodes joining/leaving and activity. This automatic optimization keeps the number of hops to a minimum and slower nodes are moved to the outside of the mesh.

Peer networking often brings to mind consumer applications such as instant messaging, music and file sharing, or multi-player gaming. However, there are enterprise scenarios where peer networking is ideal. Its great strengths are its high speed, its parallelism, and its scalability.

In Neuron, each topic powered by Peer Channel is a separate mesh. The behavior of a PeerChannel mesh is a natural fit for publish-subscribe messaging. Neuron deployments have been known to operate meshes with as many as 1500 nodes that stretch across the U.S. over multiple geographic locations.

There’s a catch with peer channel, however. The base technology is not reliable enough to suit most enterprises. There are no message delivery assurances: you can lose messages (sometimes large blocks of messages if there’s a lot of CPU contention) nor are messages guaranteed to arrive in order. The good news is that Neuron ESB provides an optional reliability layer that provides delivery assurances through message caching.

In its native form, peer channel is server-less. In fact, you can shut down all of your servers and parties can continue to message. Adding in the reliability layer or enabling certain features such as auditing will require you to keep at least one server up and running however.

The peer networking channel is volatile, meaning there is no durable storage of messages. Performance is very good, but is limited by the slowest node in the mesh.

Peer channel is best suited for scenarios requiring low-latency (high speed) communication and/or the need to broadcast to many recipients simultaneously.

WCF TCP Channel

The WCF TCP channel interconnects parties using WCF TCP with reliable messaging. The channel connects parties to an ESB server via TCP. A massively efficient TCP publishing engine on the server copies incoming published messages out to connected subscribers with matching subscriptions.

The server publishing engine can become backlogged, where published messages are coming in faster than they can be relayed out. The engine will buffer backlogged messages up to a threshold; once that threshold is exceeded, flow control kicks in and publishers will be slowed down as needed.

In a server farm scenario, each server will have a different collection of parties connected to it. The servers share incoming messages with each other laterally so that any incoming messages is relayed to all servers for distribution.

The WCF TCP channel is volatile, meaning there is no durable storage of messages. There are delivery assurances within the scope of a connection, but sudden client or server loss can lose messages.

The TCP channel is a good choice for applications that don’t require durable or transacted messaging.

MSMQ Channel

The MSMQ channel implements publish-subscribe over queues. Messaging is durable, transacted, and reliable. There is a publish queue for a topic, and a recipient queue for each subscriber. An MSMQ publishing engine on the server moves messages between queues.

Parties do not have to be online in order to receive messages from each other because undelivered messages persist in the queues. This characteristic allows you to take systems offline and online without being disruptive.

The MSMQ queues used by the channel are durable, providing reliable store-and-forward messaging. In a backlog situation, messages that cannot be immediately processed accrue in publisher or subscriber queues until they can be processed.

The server’s MSMQ publishing engine is transactional. A delivery either fully completes or the queue reads and writes are rolled back. The engine processes messages in batches when it can for efficiency.

The MSMQ channel is a good choice for transacted, durable communication.

BizTalk TCP Channel

The BizTalk TCP channel performs publish-subscribe over BizTalk Server. The channel connects parties to BizTalk Server via TCP, and BizTalk’s messaging engine performs the publish-subscribe messaging.

Making changes to the Neuron ESB configuration automatically programs BizTalk Server to make matching changes to its configuration. For example, defining new publishers and subscribers in Neuron will cause new send ports and receive ports to be created in BizTalk.

With BizTalk as the messaging engine, you gain the many features of BizTalk including strong retry/recovery capabilities. With a small amount of BizTalk configuration you can link messages to features such as Business Activity Monitoring (BAM),, transformations , the rules engine, and orchestrations.

The BizTalk channel is durable and is well-suited for mission-critical applications that require high reliability and/or the unique characteristics and features of BizTalk Server’s. It is not the best choice for low-latency applications, as BizTalk’s message box database limits performance.

What about Services, Legacy Systems, and Line of Business Apps?

Does this mean there are only 4 protocols you can use to communicate with in Neuron? Not at all--the endpoint architecture and channel model we’ve been describing is how Neuron makes publish-subscribe happen, but this is the internal side of the bus. Externally, you can connect a wide, wide variety of services and non-services to the bus… everything from WCF services to WebSphereMQ. We will talk about the external side of the bus in future articles.

Summary

The Neuron ESB channel model provides a modular system for switching in different technologies for messaging. This not only makes it possible to leverage a wide range of existing technologies, but anticipates new technologies and provides a path for accommodating them.

Topic Networks: How Neuron ESB Implements Publish-Subscribe

Tue, 13 May 2008 23:43:53 GMT

Today I’d like to talk about how Neuron ESB implements publish-subscribe over topics. Topics are important in Neuron both logically and architecturally. Logically, topics model business conversations and are an important tool for organizing information. Architecturally, topics are independent networks that give the bus its distributed nature and scalability.

A Logical View of Topics

Logically speaking, topics represent business conversations and are usually named accordingly. Topics can have simple names like Orders or Inventory, or a topic can have a hierarchy of subtopics, expressed with dotted names such as Orders.New, Orders.Approved, Orders.Shipping.US, Orders.Shipping.International, and so forth. A party expresses interest in a topic with a subscription. Subscriptions may be wildcarded, as in Orders.* which expresses interest in Orders as well as all of its subtopics (Orders.New, etc.). By modeling your solution after actual business conversations and entities, you raise the likelihood that your messaging system will be a successful and long-lived one.

Together, topics and subscriptions define a virtual circuit between publishers (senders) and subscribers (receivers) about some subject of interest. In Figure 1, a publisher named OrderEntry has published a message on the topic Orders.New. Of the 3 other parties, only 2 have matching subscription: (OrderApproval and OrderFulfillment. The other subscriber, PaymentApproval, is only interested Orders.Paid and does not see messages published on Orders.New.

Figure 1: Topical Publish-Subscribe

Although bus communication is pub-sub based, that doesn’t limit you to pub-sub as a messaging pattern. Parties can also use point-to-point messaging, request-reply messaging, and itinerary-based routing.

The ability to easily add, change, and remove parties, topics, and subscriptions—even while running live—results in great flexibility and easy change management.

An Architectural View of Topics

Architecturally speaking, each topic is an individually configured network called a topic network. Figure 2 shows 3 topic networks, named Inventory, Orders, and CallCenter. Each topic network is separately assigned a communication technology to drive it, such as peer networking, WCF TCP, MSMQ, or BizTalk Server. This allows you to run topics with a mix of qualities of service, a refreshing change from messaging systems with a single “one size fits all” QoS. Each topic network can also refer to separate server infrastructure, making topics the primary division for scaling out. The distributed topic network design is very different from the classic EAI hub-and-spoke design. There is no central hub and no central message store that all messages must pass through; instead, each topic network performs its functions independently. While you could run all of your topics over the same technology and off of the same server, you could just as easily run each topic over a different technology and different servers. Moreover, some of the technologies that can power topics require little or no server infrastructure at all.

Figure 2: Topic Networks

If each topic is really a separate network, what sort of network is it, and what are its characteristics? The answer to that is determined by the channel you select. A channel is a modular software driver that implements pub-sub for Neuron ESB using some base technology, such as WCF or BizTalk Server. The channel has complete freedom to implement pub-sub any way it wishes, as long as it provides the expected functionality. Some channels are volatile, emphasizing speed over reliability. Others channels are durable, stressing reliability over speed.

In a future article, we will dig into channels more deeply and compare the various trade-offs between them.

Service Pass-Through in Neuron ESB

Wed, 07 May 2008 01:37:39 GMT

Service Pass-Through in Neuron ESB

Today I’d like to talk about how you can connect your services (and clients) to pass message traffic through an ESB, rather than directly connecting to each other. We call this “service pass-through”.

Why Service Pass-Through?

Before getting into how this is done, let’s explore why you might want to.

Loosely-coupled routing. Imagine you have a client that interacts with a service (Figure 1a), but now you have a requirement to involve a third program that also needs to know about the information the client is exchanging with the service. How do you bring this third system into the mix? A conventional answer is to either make the original service a client of a new service (Figure 1b); or make the original client a client of two services (Figure 1c).

Figure 1: interconnecting services in code is not loose coupling

Neither of these answers is very satisfactory because they force you to tightly couple your routing. If you are a fan of SOA and value loose coupling in other areas of your work, this tight coupling of how your services are linked should offend you.

In contrast, if your client and service are have an ESB interposed between them (Figure 2), the ESB can share information with other messaging parties. This approach helps keeps your services pure, context-free and reusable, as you don’t have to add program code to link services together.

Figure 2: passing service traffic through the ESB easily accommodates additional parties.

Mediation of differences. You might have clients and services that can’t interact directly because they aren’t quite compatible, due to differences in transport, security model, message format, or semantics. You can connect each of them to an ESB, however, and the ESB can then interconnect them, mediating between their differences.

Figure 3: Mediation bridges differences in protocol, security model, message format, semantics

Workload Sharing. The ESB can work with multiple instances of services and share workload across them. You can freely add services instances or take them away while running; workload distribution automatically adjusts.

When Worlds Collide. You can get services and non-services to work together. Neuron interconnects easily to services and clients using WCF. Neuron also interconnects to non-services such as legacy system and line-of-business systems using adapters. Once joined to the ESB, these programs can interact even though there be huge gap between the technologies they are use and when they were created.

Auditing and Compliance Monitoring. You may want to keep an audit of your messages in order to show compliances with state, federal, or industry regulations. If your messaging traffic flows through the bus, the bus can take care of the rest.

Figure 4: Both services and non-services can work cooperatively

The Mechanics of Service Pass-Through

When you define a service to Neuron (usually by importing metadata), one or more service endpoint definition(s) are created. By itself, a service definition is nothing more than an interesting record in a repository. I blogged yesterday about how you can use this repository to create WCF clients based on central configuration.

Things get more interesting when you authorize service-pass through for your service definition. You can enable service pass-through for the client-side and/or the service-side, as shown in Figure 5.

Figure 5: A service definition with client- and service-side pass-through enabled

Enabling client-side pass through causes Neuron to create a client connector, which is a hosting service for clients. This hosting service looks just like the client’s actual target service, with the same binding and security model. The only difference is that you point the client to a different address on the ESB Server. Client-side pass- through maps incoming client messages to an ESB topic and to a publisher. As client messages are received, the ESB publishes those messages on a topic. Any party with a subscription to that topic receives a copy of the message. This works for both 1-way and request-reply messaging.

Enabling service-side pass through causes Neuron to create a service connector, which is a client to the service used by the ESB Server. This client has a matching binding and security model for the service it is connecting to. The service connector is mapped to a topic and a subscriber. As messages are received over the ESB, the service connector passes them on to the service. Replies from the service are passed back over the bus to the originating client.

For true client-service pass-through, you would enable both client-side and service-side pass-through. In some scenarios, you may choose to only enable one side or the other.

Notice that in service pass-through, you aren’t asked to modify your client and service in any way, other than to change the address the client points to.

Service pass-through the ESB avoids tightly-coupled routing and keeps your services more modular and reusable. It allows for mediation between program differences and allows services and non-services to interact.

Centralizing your WCF Configuration with Neuron ESB

Tue, 06 May 2008 00:29:57 GMT

Centralizing Your WCF Configuration with Neuron ESB

Today I’d like to talk about how you can store your WCF configuration information centrally with Neuron ESB . While it’s great that WCF allows you to move details such as endpoint, binding, and other information out of code and into config file settings, it can still be inconvenient to work with many individual config files. If you are dealing with large numbers of services, you may desire a way to centralize your configuration information. Neuron ESB provides a service repository for this purpose, plus a neat factory for creating WCF clients. Together, they free you from needing local WCF configuration settings.

There are a number of ways you can program against Neuron, ranging from WCF to JMS to Neuron’s own .NET API. In this case, we’re going to use the Neuron API to retrieve configuration information and create a WCF client; the rest is pure WCF from there.

Adding services to the Neuron service repository is very straightforward. An import wizard can guide you through the steps or you can set things up manually. Service definitions can be imported from metadata at URL, file, or via searching a UDDI directory. The appropriate WSDL, schema, and service endpoints are added to the repository. Neuron employs the same classes for service import that are used by Add Service Reference in Visual Studio and the SvcUtil tool.

Figure 1: Service Definition in Neuron's ESB Explorer

The service repository itself is of course platform independent, making no assumptions about what technologies are used to build services and clients.

For WCF developers, Neuron provides a client factory class that uses the central repository to create auto-configured WCF clients. Here’s how this works. Imagine you are writing a WCF client for an order entry service that has a service contract named IOrderEntry. To develop your WCF client, you go through the usual steps, including generating client code from metadata. However, you don't worry about creating a System.ServiceModel section in your local config file. Instead, you import the target service in Neuron’s service repository.

To instantiate a WCF client in your code, you use Neuron’s Client<T> class, where T is the interface you want a client for. For example:

IOrderEntry oeClient = new Client<IOrderEntry>();

What this does is scan the central service repository for a service to locate a matching interface. If a match is found, a WCF proxy is created, using the address, binding, and other information defined in the repository. From this point forward, your client program is using pure WCF, and accesses the service in the usual manner, via named methods:

IOrderEntry oeClient = new Client<IOrderEntry>();
PriceList pricelist = oeClient.GetPriceList();
int orderId = oeClient.SubmitOrder(order);

In the example above, we created the client simply by specifying an interface to Client<T>. In all but the smallest deployments that’s probably not enough information to make a good match. What environment is the service in? Is it available and in good health? Do you have rights to access it? You can be more specific by using alternative constructors to Client<T>. For example, Client<T>(name) will locate a specific service endpoint in the repository by its unique name. More powerful is Client<T>(criteria), which will match an array of name-value properties against properties in the repository. You can use repository properties to indicate any information you like about a service, such as the environment it serves or the version of its contracts.

One valuable service the Client<T> class provides is sparing a developer from having to learn and keep up with the connection details for a target service. Each time a connection is made, the latest information in the service repository determines address, binding details, security settings, and so on. Similarly, administrators can make changes to service locations and access details without worrying about being disruptive to developers.

Another value provided by the Client<T> class is indirection. When you set up a service definition in the Neuron ESB repository, you can optionally enable pass-through of service traffic through the ESB (a subject we will go into more detail about in a separate article). Pass-through of service traffic is transparent to the client and service, but doing so allows additional parties on the bus to share in the messaging. Pass-through also enables bus-managed scenarios such as workflow sharing among multiple service instances. All of this is transparent to the WCF developer, however. if you invoke Client<T> and the matching service definition enables service traffic pass-through, the WCF client that is created will point to a hosting service on the ESB. If service pass-through is not enabled, the WCF client that is created points directly to the actual service. Either way, the usage is the same and the developer won’t know the difference.

Client<T> and the service repository is an example where Neuron adds value, yet you are still using solid, familiar WCF technology to create your clients and services.

Neuron ESB Management Experience

Fri, 02 May 2008 18:29:38 GMT

Neuron ESB Management Experience

Today I’d like to talk about the Neuron ESB management experience. Neuron takes management seriously and a lot of effort has gone into providing a one-stop administrative experience for configuration, deployment, activity reporting, operations monitoring, change management, and problem response

ESB Explorer

Neuron’s management tool is called ESB Explorer, shown in Figure 1 below. ESB Explorer’s UI is styled after MS CRM, with a navigation bar at left and a workspace area to the right. It should have a familiar feel to any Outlook user.

Figure 1: ESB Explorer (click for larger image)

The navigation bar has 7 sections. We’ll be digging into these areas individually down the road.

· Messaging is where you set up your logical messaging model of publisher, subscribers, topics, and subscriptions.

· Data is a repository for various data artifacts including XML schema, WSDL metadata, XSLT transformations, sample messages, and message templates.

· Connections is where you define service- and adapter-based physical endpoint connections.

· Security is where you define credentials, set up access control lists, set encryption keys, and other security-related functions.

· Integration is where you set up integration functions (schema validation, transformations, business rules, workflows, etc.) and string them together into processing pipelines.

· Deployment is where you define zones and environments, manage resources such as databases and queues, and define deployment configurations.

· Activity is where you monitor activity in real-time, access database-driven activity reports, monitor server health, and take action on failed messages.

The ESB Explorer lets you work in online or offline mode. In online mode, saving your changes causes them to be applied live. In online mode, saved changes only take effect when you restart your ESB Server(s).

Entity Management

Many of the pages in the ESB Explorer are for managing entities such as topics, publishers, subscribers, and services. The Explorer handles entity management consistently with a list/item view. Figure 2 shows the page for managing subscribers.

Figure 2: ESB Explorer (click for larger image)

If you are dealing with large numbers of entities, you can organize them by category. Entity management pages let you filter your view by category, alphabetically, or by search term.

Diagrams

The ESB Explorer provides a number of visualizations, such as communication diagrams and deployment diagrams. The communication diagram in Figure 3 shows the relationship between publishers, subscribers, and topics.

Figure 3: Communication diagram (click for larger image)

Real-time Activity Monitoring

Both the server- and client-side components of Neuron are instrumented to report activity in real-time through WMI and performance counters. You can monitor this activity in ESB Explorer with a variety of real-time views. Figure 4 shows a chart of message counts.

Figure 4:Real-time Activity Chart (click for larger image)

Because Neuron activity information is reported via WMI and performance counters, nearly any operations monitoring tool can also

Reports

In addition to real-time reporting, Neuron can also log activity information to a database. For example, you can enable message auditing for any topic(s) in your ESB configuration, which will record every send and every receive. You can this information from a variety of perspectives, such as message history reports, failed message reports, and business event reports.

Figure 5: Message History Report (click for larger image)

Resource Management

Your ESB configuration can optionally make use of enterprise resources such as databases and queues. Neuron can create, configure, and check these resources. It can also generate SQL and WSH scripts if you need this work done by others such as a DBA.

Figure 6: Database Artifact Creation (click for larger image)

Wizards and Guides

Wizard and guides help you through performing tasks for the first time. There are wizard for setting up pub-sub topics, importing services, searching service directories, and setting up deployment configurations. Guides such as the Getting Started Center (Figure 1, above) and Samples Browser help you find what you need.

Configuration Checking

Neuron ESB is a product with a rich set of communication and integration capabilities, so enforcing a correct configuration is critical. If you have inconsistencies in your configuration, you want to find that out at configuration time, not at runtime. An example of an inconsistency is a subscriber with a subscription to Inventory, where no topic named Inventory has been defined.

The ESB Explorer performs a consistency check any time you attempt to save an updated configuration. Inconsistencies either need to be fixed, or you can mark the offending entities as disabled for the time being. Once your configuration is consistent, you can save it. The Neuron ESB server performs an identical consistency check as part of its power-on self tests.

Administration API

The ESB Explorer is an extremely powerful management tool. Everything it does is accomplished through an administration API, which means you can do all of this in .NET code if you choose to and have appropriate rights.

That does it for our introduction to what management is all about in Neuron ESB. We’ll be frequently referring to ESB Explorer as we discuss how to configure individual feature areas in future articles.

A High Level View of the Neuron ESB Architecture

Wed, 30 Apr 2008 19:19:11 GMT

A High Level View of the Neuron ESB Architecture

Today I’d like to provide a very high level view of the Neuron ESB architecture. Understanding this will help put individual features and concepts in context as I describe them in upcoming articles.

The bus is best thought of as having a distinct outside and inside, as the figure below illustrates.

· The outside of the bus is concerned with connecting to diverse enterprise applications, services, and legacy systems

· The inside of the bus is concerned with interconnecting endpoints via publish-subscribe messaging and performing integration services.

The Outside of the Bus

The outside of the bus has exacty one purpose: to establish connections to any kind of business endpoint in the enterprise. Neuron has strong support for both services and non-services. Applications can talk to the bus in a way that is natural to them in terms of protocol, security model, and message format.

· For service-oriented endpoints, Neuron uses WCF to connect both clients and services.

· For non-service endpoints, Neuron uses adapters to communicate. Neuron supports several types of adapters.

· When you have an endpoint that's neither a service nor has an existing adapter, you can consider wrapping the endpoint with a service, or writing a new adapter (writing a Neuron adapter is nowhere near the daunting task it is in many other products). There is also a .NET API for Neuron that can be helpful in new integrations.

Service support is extensive. You start by defining a service to Neuron. You can locate a service endpoint in a UDDI directory, import a service from its WSDL, or manually define the service as you prefer. You can specify any standard WCF binding or any custom binding you may have. You can use any of the common WCF security scenarios, spanning transport security to message security to federated security.

Adapters supported include WCF LOB adapters, which are superseding the earlier family of BizTalk Adapters. There are already WCF LOB adapters for Oracle, SAP and Siebel available from Microsoft, and many more adapters are available or in development from a variety of sources. Neuron also supports its own .NET adapters, which have a simple interface and are easy to write. Adapters that come with Neuron include ActiveMQ, BizTalk RFID, eConnect (Great Plains), file, MSMQ, SharePoint Server, SMTP Email, SQL Server, and WebSphere MQ.

Anyone who's done a lot of enterprise integration in the EAI space knows that integration jobs can vary from the simple to the complex to the really difficult. We've put a lot into Neuron to make enterprise connectivity as simple and painless as possible, as often as possible.

The Inside of the Bus

The inside of the bus is concerned with interconnecting endpoints through publish-subscribe messaging. Each endpoint that connects to the bus is related to Neuron's internal publish-subscribe messaging model.

Every endpoint is mapped to a party. A party is just a messasging node: we call a party that send messages a publisher, and a party that receives messages a subscriber. Subscriptions link a party to one or more topics of interest. Finally, a topic is a name for a business conversation. So, for example, you might have a publisher named OrderEntry and a subscriber named OrderProcessing, both subscribing to the topic Orders. When OrderEntry publishes a message over the topic Orders, OrderEntry receives a copy of the message because it has a matching subscription. If additional parties want to know about orders, all they have to do is subscribe to the Orders topic. Publish-subscribe is extremely flexible, and Neuron allows you to add parties and change subscriptions live while the system is running.

What makes the communication happen? Neuron supports a varierty of Microsoft technologies to perform the communication, from WCF to BizTalk Server. You can also run a mix of technologies, such that different topics provide different qualities of service. More on that later.

The inside of the bus has another job, which is to provide integration functions. While a message is en route between endpoints, the bus can apply pipelines of processing steps. Functions that can be performed include validating schema, transforming a message, executing rules, executing a workflow, performing additional messaging, and moving message data to/from persistent storage. Integration services are part of what it takes to mediate differences between endpoints: you might for example need to transform a message from one endpoint in order to make it intelligible to another endpoint. They also allow to apply business rules or workflows via the bus instead of having to make code changes to your business programs.

Integration functions can be provided by services. This modularity gives you the freedom to use any technology or product you wish to power them, and to change them out at any time. Not only can you use the rules engine, workflow engine, or transformation technology of your choice, but you can run more than one side by side.

Benefits

What does this design do for you? Why connect everything to the bus? What is the gain?

· You can get programs that could never directly interact to work together through the bus.
We call this mediation, and it's a faster and cheaper alternative to rewriting your apps.

· You create additional value from your existing I.T. assets by extending their reach.
CFOs really like getting new value from previous money spent.

· Changes in routing between parties is now a painless configuration exercise instead of an application code change project.
Easy change management makes I.T. and your entire organization more agile.

· Business rules and workflows can be deployed externally to application code.
And you can deploy them while running live, without having to schedule downtime or code changes.

· Messages can be audited, checked for policy compliance, and replayed for disaster recovery.
Make your enterprise more consistent. Replace a fragmented view of individual applications with an enterprise-wide view of activity.

· Patterns of messages can be linked to business events.
Relate messaging to business events. Get real-time notification about what is transpiring in your organization. Feed BI and analysis tools with the information they need.

· Accelerate SOA adoption.
If you're serious about SOA, at some point you're going to realize that your business services need to be complemented by infrastructure services. Neuron ESB gives you a versatile set of infrastructure services (while preserving your right to use best of breed products to power them) along with a central repository and one-stop management experience.

There's another big benefit of the Neuron ESB architecture that I will be devoting a separate article to: it provides an abstraction layer for your enterprise architecture. This allows you to change the underlying technologies for communication and integration without being disruptive or forcing you to reinvent your architecture. More on that soon.

Neuron ESB 2.0 Released!

Tue, 29 Apr 2008 22:34:10 GMT

I've been guilty of not blogging for quite some time. No, I wasn't held hostage by terrorists... but close to it :) I and my fellow engineers have been heads-down, working around the clock to complete some very cool software--that I can now talk about. I'm very pleased to announce completion of Neuron ESB.

Neudesic has released version 2.0 of its Enterprise Service Bus product, Neuron ESB. Neuron ESB is a full-featured ESB based on the Microsoft platform. It provides a sophisticated communication and integration backbone with easy change management and a one-stop administrative experience. Neuron combines the capabilities and best practices of the SOA and EAI disciplines using proven Microsoft technologies. Neuron ESB accelerates SOA adoption by providing the necessary infrastructure services to complement your business services. Neuron communicates equally well with non-services such as legacy systems and LOB applications via adapters. Intelligent routing interconnects endpoints using publish-subscribe messaging and a named topic hierarchy. Mediation brings together programs that otherwise could not interact due to differences in protocol, security model, or message format. Integration services provide EAI functionality such as schema validation, message transformation, auditing, rules engine processing, and workflow execution. Event-based processing includes real-time notifications, the ability to link message patterns to business events, and complex event processing. ESBs can be zoned by location or administrative control, and zones can be federated. Neuron ESB is suitable for solutions ranging from departmental scope to enterprise-wide, mission-critical applications that span wide geographies.

Now that Neuron ESB is released, you can expect plenty of blogging from me about ESBs in general, Neuron ESB in particular, and how real-world customer challenges are met.

Have an ESB question? Interested in having someone come out and speak about ESBs? Drop me a note.

Buying a Piano (When Parenting Gets Pricey)

Mon, 10 Mar 2008 01:12:17 GMT

Our family is now the proud owner of a Yamaha T118 upright piano. It is a very nice piano for the money and I'm sure it will serve us well for years to come.

However, I am quite certain that I started the week with no plans or budget for a piano whatsoever, so how did this Coup d'Wallet happen? The answer--and a warning to parents everywhere--is that my wife and I can only blame ourselves for planting the seeds of this expense two years ago, when we encouraged our oldest daughter to start taking piano lessons. You see, we encourage our children to try lots of things--it's important to find out early in life what interests you and where your special abilities are. When we started them on piano lessons, we of course didn't run out and buy a piano; a $400 electronic keyboard gives them something to practice on, and who knows how long they will stay at it?

The problem is, the kids have come through. Not only is our oldest daughter doing well now in her second year, our younger daughter has also started piano and is advancing rapidly. And now they're getting into areas like pedaling technique, where an electronic keyboard just won't do anymore. Ka-Ching, time to buy a piano, no way around it. It's not like you can put it off till later.

There's a picture of a T118 below, though ours is polished mahogany, not black.

As for how we ended up selecting this particular model, we simply decided not to chance a used piano or a questionable brand. Before we knew it we were walking into a local Yamaha dealer, armed with some knowledge of appropriate pricing from the Internet. As it turns out the prices were more than reasonable and no haggling was necessary. Although we knew we'd have to stay near the low end of the product line, we were hoping to get something sturdier than a "spinnet" beginner piano. The first model we saw was too spindly and fragile for our liking, but we knew we had found the right model when we came across the T118: while it's affordably priced, the outer woodwork is sturdy and attractive; and it's insides are near identical to the successful U line of Yamaha pianos that are often used in schools and colleges. It has a 10 year warranty, and if we decide down the road we need something different we can trade up.

An unexpected side effect of ths purchase is that my wife and I keep drifting over to the piano to see what we can remember from our own piano lesson days. Perhaps we'll get back into it...

neuronesb.com

Tue, 05 Feb 2008 16:14:55 GMT

The Neuron ESB now has its own dedicated web site, http://www.neuronesb.com.

What's on the site? It's purpose is to provide easy access to:

Product information
Vertical solutions
News
White papers
Case studies and customer testimonials
Support resources
Discussion forum

Some of that content is up there now, and a lot more is on the way.

WCF Security Hydra

Tue, 29 Jan 2008 08:05:47 GMT

The Hydra was the many-headed serpent of Greek mythology. I have created a WCF security hydra by connecting 28 WCF services and clients that use each of the 14 common WCF security scenarios to the same Neuron ESB. All of the endpoints can operate at the same time, so the resemblance to the hydra is more than superficial. The bus can also mediate between endpoints, allowing clients and services of different security models to interact with each other (try that without an ESB and you won't get very far).

The scenarios are:

Internet unsecured
Intranet unsecured
Anonymous message security
Certificate message security
Issued Token (federated) message security
Kerberos message security
Mutual certificate message security
Username message security
Windows message security
Anonymous transport security
Basic transport security
Certificate transport security
Windows transport security
Trusted subsytem

As anyone who has played with WCF knows, the security capabilities of WCF are extensive but can be a bit overwhelming to master. It definitely feels like a major accomplishment to reach this point. I can now confidently implement each of these scenarios using WCF configuration, using WCF code, or using Neuron's ESB Explorer UI settings. All of this shows how capable WCF is, especially when combined with the Neuron ESB.

Software Etudes and Code Kata

Tue, 22 Jan 2008 20:42:41 GMT

I just became familiar with the idea of Software Etudes (James Shore) and Code Kata (Dave Thomas), thanks to Sam Gentile's blog. As soon as I read these postings I said "Of course! How obvious in hindsight!" to myself and immediately embraced the concept. I absolutely love the idea of repetive exercises for software developers that build strong basic skills.

The etude technique works really well in classical disciplines. If you've ever taken piano lessons, you probably remember etudes, repetitive exercises that do fun things like make you play without using a certain note in the scale. The most [in]famous etudes are by Czerny, Chopin, and Hanon. Some of these etudes are good music in their own right, others less appealing, but their purpose is to develop an area in you through repetitive exercise. For example, one etude might focus on improving your ability to play parallel octaves, whereas another might focus on rapid chromatic thirds. Vocal training also uses these kind if exercises. For example, a vocalist might use exercises which transcend high, medium, and low pitch ranges (which tend to resonate in the head, chest, or abdomen) and strive for a consistent tone.

Code Kata is essentially the same idea as software etudes, borrowing a term from Karate for exercises designed to develop discipline and skills.

I'm going to promote awareness of software etudes/code kata within my own organization. I think their use is critical on the path to becoming a Master Developer.

Transactions in the Real World (continued)

Sat, 22 Dec 2007 01:00:24 GMT

Previously I blogged about my real-world experience with how transactions don't always work, where an ATM told me it couldn't give me $400 cash but happily debited my checking account anyway. This episode is all the more shocking because it had to do with banking. Banking! If transactions work anywhere, surely it's in that context. Or so one would think.

To continue the saga: to get the "compensation logic" going for my failed transaction, I sent my bank a message, explaining that the $402 debit was erroneous and why. I also mentioned that the $102 debit that followed was correct (the ATM had dispensed cash for that second attempt). I heard nothing for a few days, so I repeated my message. Finally, I received a response that an investigation would be requested of the bank that owns the ATM, and I would be getting a form in the mail, and I'd need to return it promptly.

So far the forms haven't come, but something more interesting has already occurred. In reviewing my online banking statement (something I now do almost daily given recent events), I saw my $402 had been credited back, provisionally, to my account. How nice. And right above that... a credit for $102 as well. Aaargh!!! The compensation logic had also failed! The $102 transaction had been fine, whoever read my message didn't read it very carefully. Now I've been overpaid.

I'm pretty sure this is a sign that civilization as we know it is crumbling. Better start keeping your money in a mattress.