BizTalk 2013 Beta: Viewing dependencies

In early November 2012 the Beta release from BizTalk 2013 has been released. That version number is interesting in itself, as the CTP was called BizTalk 2010 R2.
One on the new functionalities in this release, which was not yet available in the CTP is a Dependency Viewer. As the name already suggests, this functionality enables the user of the BizTalk Server Administration Console to view which BizTalk artifacts depend on other artifacts. This article intends to provide more information on how this functionality works.

The problem it solves

Most BizTalk Groups contain multiple BizTalk Applications which might contain hundreds of schemas, (custom) pipelines, maps, orchestrations and endpoints. These artifacts are not autonomous, they depend on other artifacts and those artifacts might also depend on other artifacts.

When it comes to subjects like deployment and troubleshooting, it's handy to know which artifacts rely on each other. That's the point where the Dependency Viewer comes in.

Where to find

The Dependency Viewer can be found in the Administration Console. It can't be missed in the lower part of the screen under the following artifacts:

• Receive Ports
• Receive Locations
• Send Ports
• Send Port Groups
• Schemas
• Maps
• Pipelines

How it works

Whenever you select one of the mentioned categories, the Dependency Viewer appears in the lower part of the pane that shows the artifacts. To find out which dependencies a certain artifact has, you need to hit the 'View Dependencies' button in the right pane. So unfortunately the Admin Console does not automatically show the dependencies, whenever you click on another artifact. Hopefully this will be fixed in a later BizTalk 2013 version.
Once you click that hyperlink the Dependency Viewer shows the dependencies, if any, of the selected artifact. It does this per category.
If dependencies exist, the number of dependencies appears as a hyperlink. The dependencies themselves appear after you have clicked the hyperlink.

The screen above shows the dependencies for a Receive Port called ReceiveSamplePort. This port has 2 Receive Locations which are shown both.
Although the upper Receive Location is selected, no dependencies are shown. If you want to see its dependencies, click View Dependencies in the right pane again and in this case it would show that this Receive Location depends on one Receive Port and one Pipeline.

Conclusion

The way the Dependency Viewer is implemented in this Beta release is not (yet) very impressive. I mention a couple of down sides on the current implementation:

• no auto-retrieve. You need to hit the 'View Dependencies' button
• the Dependency Viewer is always there, it's not possible to change its size or hide/show the viewer
• no dependencies shown in case of Message Based Routing scenarios, for example a Send Port with a filter to a Receive Port
• no dependencies shown relating to Hosts

I'm looking forward to the next BizTalk 2013 release and I am sure the Dependency Viewer will be implemented in a better and richer way.

Real World Scenarios with BizTalk360

During the Bouvet BizTalk Innovation Event, 26/27 September 2012 in Stavanger, Norway, there were 2 presentations about BizTalk360. One was held by Saravana Kumar, the founder of BizTalk360, and I did another presentation about this product. Where Saravana showed many functionalities, including the new V5 functionalities, from BizTalk360 from a Product perspective, I wanted it to turn around and have a look at BizTalk360 from a Customer perspective. Therefore I talked about some Real World scenarios with BizTalk360.

In addition to my presentation I wrote an article on this topic and extended it with describing the most important functionalities in V5, which was released recently.

In the article I explain why having a Shared Infrastructure, like BizTalk, makes monitoring more complex. I also describe how BizTalk360 reduces the number of consoles you need to monitor a BizTalk environment.

To expand the visibility from BizTalk360, I posted the article on CodeProject. You can find it here:
http://www.codeproject.com/Articles/474905/Using-BizTalk360

More information about BizTalk360, can be found here:
http://www.biztalk360.com

Important updates to BTSDecompress

As an BizTalk Administrator/Technical Application Engineer, I regularly need to fix problems with BizTalk that I am not able to fix easily with the out-of-the-box tools. For example, at the moment we experience an issue which leads to suspended messages on a particular receive location. To fix that issue I need easy access to these suspended messages and their content. Off course I can get to the content with the BizTalk Administration Console, but because I need to fix about 50 of those messages per day (awaiting a structural solution), I need easier access to the MessageBoxDb to save time and to prevent myself from RSI, due to so many mouse clicks.

Since a couple of years I maintain BTSDecompress on CodePlex. This tool enables me to query the BizTalk databases and retrieve message content and context. Although this tool works fine for me, the main disadvantage is that, until now, it only supported BizTalk 2006. Therefore only a limited group of BizTalk users could potentially benefit of this software.

Because of the issue I mentioned in the first paragraph, lately I've been working a lot on that tool. It has reached version 1.5 and in this version a number of important improvements have been applied. Here follows a list of the most important changes:

Support of BizTalk releases
- BizTalk 2006
- BizTalk 2010
- BizTalk 2010 R2 CTP

I expect BTSDecompress to work with the other major BizTalk releases as well (2004, 2006 R2, 2009), but I did not (yet) test it, so I don't mention these versions here.

Support of Windows versions
- Windows Server 2003
- Windows Server 2008 R2
- Windows 7

I expect BTSDecompress to work with the other major Windows versions as well (XP, Vista, 2008), but I did not (yet) test it, so I don't mention these versions here.

Query maintenance
It is now possible to Create, Open, Save and Save As queries. This makes it a lot easier to maintain multiple queries.

ebXML support
It was already possible to fire XPath queries on the message content, but you can now do that as well upon ebXML headers and envelopes.

Where to find
BTSDecompress is Open Source software and can be found here:
btsdecompress.codeplex.com

Developing BizTalk 2010 R2 with Visual Studio 2011??

Since a couple of days Microsoft enabled Windows Azure users to create Virtual Machines, which are based on BizTalk 2010 R2 CTP. This is great, because it offers those users to have a preview at that release, combined with some of the new related products, like SQL Server 2012 and .NET 4.5.

However, given the screenshot below, which was taken from the BizTalk 2010 R2 CTP Installation Wizard, it looks like Microsoft expected to release the next version of Visual Studio sooner :-)

 

Bouvet BizTalk Innovation Event

Together with Steef-Jan Wiggers, Sandro Pereira, Nino Crudele, Saravana Kumar and host Tord Glad Nordahl, I will do a presention at the Bouvet BizTalk Innovation Event which will be held on 26th and 27th September at Bouvet's office in Stavanger, Norway!

During my presentation I will tell about the added value of Message Flow Monitoring over Endpoint Monitoring. Further I'll show how Message Flow Monitoring is implemented in BizTalk360.

The program looks as follows:

DAY 1
09:00 Registration
09:30 Introduction
10:30 Pause
10:45 Steef-Jan Wiggers: Adapter Pack Integration Capabilities (BizTalk)
12:00 Lunch
13:00 Sandro Pereira: Introduction to the Azure Service Bus EAI/EDI features.
13:45 Pause
14:00 Tord Glad Nordahl: Throttling/& Thresholds
14:45 Pause
15:00 Saravana Kumar: Using BizTalk360
15:45 Lex Hegt: Message Flow Monitoring BizTalk 360
16:00 Pause
16:15 Nino Crudele: BizTalk assessment and architecture review
17:00 Tord Glad Nordahl: Wrap up – Road map BizTalk Next
17:30 Finish
20:00 Dinner

DAY 2
09:00 Startup
09:30 Labs
11:30 Lunch
12:30 Labs
15:30 Finish

I want to thank Tord and has company Bouvet for their hospitality and for giving me the opportunity to do a presentation.

Links:
- more information about the event
- prices
- registration

Off course this BizTalk event is added on http://www.biztalkevents.com/.

BizTalk Message Based Routing: why is my file not transfered to the location of the Send Port

Inspired by some tweets I had with Sam Vanhoutte (@SamVanhoutte), Mikael Sand (@MikaelSand) and Dean Robertson (@DeanRobertson), I decided to write down a number of our experiences of what can go wrong, or what you could forget, in case of a simple message based routing scenario.

So basically this article contains a checklist, which can be used as a reminder in case your message based routing scenario does not work, like you expected.

Scenario
Imagine you have a message based routing solution in its most simple form.
Your Receive Location listens to a local folder and your Send Port is subscribed to the Receive Port from the Receive Location. The URI of the Send Port is another local folder.
Although this is a very simple solution, still there is enough that can prevent the messages from flowing from A to B.

1. The Receive Location listens to the wrong folder or contains a typo
To be absolutely sure you have configured the correct folder, copy the path from the address bar in Windows Explorer and paste it at the URI field in the Receive Location.

2. The Receive Location is not enabled
If the Receive Location is not enabled, it won't poll the configured folder, so you must enable the Receive Location.

3. A Service Window or Start/Stop Date prevents the Receive Location to poll for incoming messages
On a Receive Location a Service Window can be configured. When a Service Window is set, the Receive Location will only process incoming messages during the configured timeframe. So be sure no Service window is active.
Furthermore you can also configure a Start/Stop Date. When a Start Date is configured (and enabled) which lies in the future, no processing will take place until that date. When a Stop Date is configured (and enabled) which lies in the past, no processing will take place from that date on.

4. BizTalk is not authorized to access the file location of the Receive Location
Make sure that the identity which is configured at the Host which runs the Receive Location, has enough permissions (read/write) on the folder of the Receive Location. Failing to do so, results in the Receive Location disabling itself (after some retries).


5. Files with names longer than 256 characters are encountered in the receive location
Files with names longer than 256 characters are ignored by BizTalk, so they will not picked up by the Receive Location. So make sure your file names are shortes than 256 characters.

6. Files have the System or Read-only attribute
If (one or more of) the files at the Receive Location have the System or Read-only attribute, they will be ignored by BizTalk, so make sure that the files do not have the System or Read-only attribute.


7. Empty (zero byte) files
If an empty (zero byte) file is picked up by the File Receive adapter, the file is deleted and a warning is written to the application log of the BizTalk server. The File Receive adapter deletes zero byte files by design.


8. The Host Instance which runs the Receive Location is not started
Host Instances take care of the actual processing of your Receive Locations, Send Ports and Orchestrations. So if the Host Instance from the Receive Location is not started, no files will be picked up, although your Receive Location is enabled, so start the designated Host Instance.

9. The filter on the Send Port contains the wrong Receive Port or contains a typo
Correct this by copying the name from the Receive Port and paste it in the filter of the Send Port.

10. The name of the Receive Port in the filter of the Send Port is surrounded with quotes
Quotes are not needed, so remove them.

11. The Send Port is not enlisted and started
If the Send Port is not enlisted, there is no active subscription for the Send Port. Further the Send Port must be started to send any files, coming from the Receive Port to the outgoing folder. So Enlist and Start the Send Port.

12. The Host Instance which runs the Send Port is not started
Host Instances take care of the actual processing of your Receive Locations, Send Ports and Orchestrations. So if the Host Instance from the Send Port is not started, the file will be picked up by the Receive Location, stay in the MessageBox, but won't be delivered to the outgoing folder, so start the designated Host Instance.

13. BizTalk is not authorized to access the file location from the Send Port
Make sure that the identity which is configured at the Host which runs the Send Port, has enough permissions (read/write) on the folder of the Send Port. Failing to do so, will result in suspended messages.


14. The File Send Adapter cannot write to the file specified because it has the Read-only or System attribute
If you want to append data to a file, but the file to which is written by the Send Port has the System or Read-only attribute, BizTalk won't be able to write to it, so make sure that the file on the Primary (or Backup) Location do not have the System or Read-only attribute.


15. The Send Port uses the Backup Transport (if configured)
In case anything went wrong while writing to the Primary Transport from the Send Port, BizTalk tries to write the message to the Backup Transport if it is configured. So if a message shows up on on the Backup Transport, apperently went wrong while delivering the message to the Primary Transport.


16. Mismatch of the file mask on the Receive Location
The file that you drop on the folder of the Receive Location does not match the file mask from the Receive Location. If you are dropping a .XML file while the Receive Location expects only *.TXT files, your .XML file won't be picked up by the Receive Location.

17. Tricked by Windows Explorer
You think you are dropping a .XML file, and the file mask expects .XML files, but still the file is not picked up. Perhaps you are tricked by Windows Explorer. Windows Explorer has an option to hide familiar extensions of files. So if you have a file called order.xml.txt, Windows Explorer hides the .txt extension, perhaps giving you the idea that you are dealing with a .XML file, while in fact you are dealing with a .TXT file. In Windows Explorer turn off the option to hide familiar extensions, making sure that you will always deal with the full file name.

Thanks to Mikael Sand for supplying a number of possible problems. Some of the problems are derived from this article.

New website: biztalkevents.com

Because there are so many BizTalk (and integration) related events, I thought it would be nice, to dedicate a website for them. So I created www.biztalkevents.com, added a calendar, an entry form and added a couple of upcoming BizTalk related events.

If you have an event, which other BizTalkers must know about, just fill out the entry form and I will take care that the event will be shown in the calendar.

International BizTalk Events 2012: May is Integration month!

In February 2012 Ordina (by means of Steef-Jan Wiggers) organised the Ordina BizTalk Innovation Event. We had the honour to receive 2 international guests, namely BizTalk MVP Saravana Kumar and Tord Glad Nordahl. Since that event multiple international BizTalk/Integration events are being organised. I'll mention a couple that I know off:

10 May: UK Connected Systems User Group
Location: London, United Kingdom
Speakers: Saravana Kumar, Tord Glad Nordahl
More information can be found here

22-24 May: Integration Masterclass
Location: Brisbane, Melbourne, Sydney, Australia
Speaker: Richard Seroter
More information can be found here

24 May: Overnet BizTalk Innovation Event
Event Location: Milan, Italy
Speakers: Steef-Jan Wiggers, Sandro Pereira, Saravana Kumar, Tord Glad Nordahl
More information can be found here

26-27 September: Bouvet BizTalk Innovation Event
Location: Stavanger, Norway
Speakers are not yet confirmed, but probably: Steef-Jan Wiggers, Saravana Kumar, Tord Glad Nordahl, Lex Hegt
More information: not yet available

So if you are into BizTalk (or integration) and in the neighbourhood of one of those events, join the party!
My pick? Milan and Stavanger! Hope to see you there!

My contributions to TechNet Gallery

As I like to research how BizTalk's interiors work, I often find myself writing queries against BizTalk's databases. Until now I've put literally a handfull of those queries at TechNet Gallery.

Here follows a description of those queries:

View Hosts and Host Instances Information from a SQL Query
This query shows information about the configured Hosts and the Host Instances within your BizTalk Group(s). The query becomes run against the BizTalkMgmtDb database.

It shows the following information:
• The name of the BizTalk Group
• The name of the Host
• Host Type (In-process or Isolated)
• 32-bits only (Yes or No)
• Tracking Host (Yes or No)
• The name of the Server where the Instance exists
• Name of the Host Instance
• Disabled from starting (Yes or No)

The result is sorted on:
• BizTalk Group
• Host Name
• Server Name

Show BizTalk Applications, Orchestrations, Ports and Pipelines with a SQL query
This query queries the BizTalk Management database. In one view it shows the following information:
• The BizTalk Applications from the BizTalk Group at hand
• The Assemblies which are deployed to the Applications
• The Orchestrations which are in the Applications
• The Status of the Orchestrations:

1. = Unenlisted 
2. = Stopped 
3. = Started

• The Ports as defined in the Orchestrations
• The Port Types
• The Port Type Operations
• The Receive Ports as they are bound to the Orchestrations
• The Receive Locations as they belong to the Receive Ports
• The Receive Pipelines as they are configured at the Receive Locations
• The Send Ports as they are bound to the Orchestrations
• The Send Pipelines as they are configured at the Send Ports
• The Send Port Groups as they are bound to the Orchestrations

The output is sorted on:
• Application
• Assembly
• Orchestration

Number of records in the MarkLog table
This query returns the number of records found in the MarkLog table of your BizTalk databases. This table is filled by the Backup BizTalk Server job, but is not cleaned up, unless you use the Terminator tool. Read this article to purge the records in the MarkLog table.

If you have more than 150.000 records in any of your MarkLog tables, you should use Terminator to purge these records.

The query can be easily extendedwith any custom databases which are also backed up by the Backup BizTalk Server job.

Show BizTalk Subscriptions
This download contains a .SQL query, which can be executed against your BizTalk Message Box. The results you get are the current subscriptions in the Message Box. Only the 'Equals Predicates' subscriptions are shown.
 
The following fields are returned:
• Application - Name of the BizTalk Application which contains the orchestration for this subscription
• Orchestration - Name of the orchestration which has the subscription
• Created On - The timestamp when the subscription was created
• Equals Predicate - The Equals Predicate (mostly the promoted property on a schema)
• Value - The value part from the predicate

Get Orchestration Instances, Counts and States
When this script is being run against a MessageBox it gives you the following information:

• Name of orchestration instances
• Number of orchestration instances
• State of the orchestration instances

The view is grouped by the first 2 columns and sorted on all 3 columns. To make it safe to execute this query against the MessageBoxDb, the WITH(NOLOCK) hint is used.

Number of Messages per MessageType in the TrackingDb
This query gives you insight in what information is stored in the BizTalk TrackingDb. When being run against it, the query shows the following columns:

• Schema
• Number of Messages

The result is grouped by and sorted on the field Schema. To make it safe to execute this query against the MessageBoxDb, the WITH(NOLOCK) hint is used. If you query the BizTalk databases yourself, never forget to add the WITH(NOLOCK) hint.


Hope these queries are helpfull for you. I'll certainly add more queries to TechNet Gallery.

Review of Steef-Jan's BizTalk Server 2010 Cookbook

At BizTalkAdminsBlogging I've written a review about Steef-Jan's first book.

You can find it here.

A flying start for BizTalkAdminsblogging.com

BizTalkadminsblogging has had a tremendous start. Within a week and a half 13 posts already and this will be number 14. An average almost two blog posts a day. This reflects the high demand for BizTalk administration related information, the need for sharing it and desire to create more exposure for this aspect of BizTalk Server. Jeroen and Joris have done an excellent job setting up this blog site, running and maintaining it. The recent posts have touched topics like PowerShell, Microsoft Operation Framework, and so on.

There is also a TechNet Wiki, where some of the bloggers and others have written articles on BizTalk Administration. I, Tord G. Nordahl, Sandro Pereira, Howard S. Edidin and Microsoft have written the following articles that are related to BizTalk administration:

• BizTalk Server 2010 - Tools
• BizTalk Server: Disaster Recovery
• BizTalk Monitoring Tools
• Administering BizTalk 2010 - Roles and Responsibilities
• BizTalk Server 2010: MSDTC
• Installing BizTalk Server 2010 in a Basic Multi-Computer Environment
• BizTalk Server 2010 Host Throttling
• BizTalk Server 2010: SQL Agent jobs
• BizTalk Server - Determine Version and Edition
• BizTalk Server: Host architecture
• BizTalk: List of Errors and Warnings, Causes and Solutions
• BizTalk Administrator Guidelines
• BizTalk Server: High Availability
• BizTalk Databases: Survival Guide
• BizTalk Health Check
• Microsoft BizTalk automatic throttling
• BizTalk Roles
• Recommendations for Installing, Sizing, Deploying, and Maintaining a BizTalk Server Solution
• BizTalk Administrator's Checklist Compiled by Microsoft BizTalk Support
• Common Issues and Resolutions with the BizTalk Server Administration Console
• BizTalk Server - Troubleshooting Problems with MSDTC
• Multiple Message Boxes in BizTalk (Scale out message box)

As you can see that is quite an extensive list (20+ articles).

In the near future Jeroen and I want to organize an event with the Dutch BizTalk User Group that is focused on BizTalk administration. A similar event has already taken place last week. The Ordina BizTalk Innovation Event with topic Administration and Monitoring has been a great success. More than 40 attendees were present at the event. This shows the growing commitment of the community towards BizTalk administration.

TechNet Wiki: The number of BizTalk articles is growing …

A couple of weeks ago I wrote a blog post for the Official blog of TechNet Wiki with the topic “Wiki-Ninjas on Technology: BizTalk Server”. In that blog post I wrote a story on contributions of the community and Microsoft employees for the TechNet Wiki. Mid December there were little of 40 BizTalk Wiki Articles and over a month’s time it has grown to little over 70. That is a tremendous growth of articles (around 30). It show increasing popularity of the TechNet Wiki as a channel of information on Microsoft technology and products.

The newly added articles on BizTalk Server have been written by myself, Tord Glad Nordahl, Sandro Pereira, Howard S. Edidin, and Mick Badran. All these articles (existing and new) are being improved every day by people mentioned here and others (Microsoft employees and the community). I expect the number will grow over time to over 100+ articles as more community members will probably join.

With the wealth of information on TechNet Wiki, MSDN and other Microsoft channels combined you can fully leverage the BizTalk Server platform. You can reach all the BizTalk related wiki articles through BizTalk Server Resources on the TechNet Wiki article.

Enjoy reading any of the articles and hopefully you will find them useful.

BizTalk Processing Monitor in Public Beta

Since this week BizTalk Processing Monitor is available as a Public Beta. It can be obtained here.

Why use BizTalk Processing Monitor
Depending on how you setup BizTalk Processing Monitor, it helps you, amongst others, with the following needs:

• Monitoring and notification of any message flow interruptions
• Get insight in how the BizTalk solution is setup
• Adds value when doing stress or load tests
• Informs you when your BizTalk system is throttling and why
• Informs you in how many messages (in number and in KB, MB or GB) are processed
• Informs you what kind of messages (Message types) are processed
• Informs you about transmission failures

Features
The most important features of BizTalk Processing Monitor are:

• Message flow monitoring with notifications
• Throttling monitoring with notifications
• Singleton Backlog monitoring
• Support for multiple environments
• Integration with other monitoring tools

Documentation
The following documentation is available:

• Getting started with BizTalk Processing Monitor (download)
• How to use the Traffic Light Monitoring (download)
• Overview Keyboard Shortcuts (download)
• Exporting and importing Diagrams (download)

Ordina BizTalk Innovation Event: Monitoring and Administration

I am organizing an event at Ordina on BizTalk Innovation with the topic “Monitoring and Administration”. This is the first event of a series of events under name “Ordina BizTalk Innovation” that will take place at my company Ordina. This and future events are open for customers, the community and Ordina professionals. The event on the 1st of February three speakers will do their presentations on BizTalk Monitoring and administration. During the event I will be the host.

Wouter Crooy, Senior BizTalk Consultant, will have a session on:

Custom Monitoring solutions for BizTalk, ESB Toolkit & WCF

Wouter will during his talk provide a number of custom solutions for monitoring a BizTalk solution and the ESB Toolkit . Using the standard tooling of BizTalk will get you a long way, still with some of the custom monitoring solutions you can have more insight in your own custom BizTalk solutions.

Saravana Kumar, BizTalk MVP, CEO of BizTalk360 will talk on:

Manage your BizTalk Server environment efficiently using BizTalk360

BizTalk 360 is a web based (Silverlight RIA) application primarily designed for supporting and monitoring Microsoft BizTalk Server environments. It addresses some of the common challenges organizations face on running the day to day operations of a BizTalk environment. Some of the key capabilities of BizTalk360 includes:

• Fine grained authorization
• Governance/Audit
• Proactive Monitoring/Notification capabilities
• Graphical Message Flow Viewer for Tracking data
• Various dashboards (Environment, Application, BizTalk Server, SQL Server, Host etc)
• Advanced Event Viewer
• Integrated BAM Portal
• Dynamic topology diagram
• Message Box Viewer (MBV) integration
• Knowledge base repository

There are various other features in addition to the above, that makes BizTalk 360 a must have application for any Microsoft BizTalk Server environments.

Lex Hegt, BizTalk Architect, will have a session on:

Lex will talk on BizTalk monitoring in general and provide an overview on existing tooling in context with BizTalk administration. He will also demonstrate the BizTalk Processing Monitor. This is a tool that, among other things, does (near) real-time monitoring of message flows through BizTalk systems enabling the administrator to quickly identifying issues.

You can register for the event here. The talks of Lex and Wouter will be in Dutch and Saravana’s talk in English. Also joining us during this event will be Tord Grad Nordahl BizTalk expert on BizTalk administration from Bouvet ASA (Norway).

Cheers.