Ivan Mitev In The Software Trenches

Technology weblog on .NET development and other things that make the world go round

January 28, 2005

Where is the OLE DB connection pooling?

Well, I am facing a serious problem. I have changed the behaviour of data access layer to open a new connection whenever there is a need to talk to the DB. Before the change there was one static global connection that remained opened in the lifetime of the application. And now I experience significant performance casualties.

I have read in numerous resources that I can count on the automatic connection pooling of my OLE DB .NET Framework Data Provider. I am using Microsoft.Jet.OLEDB.4.0 and a MS Access. So let's quote the documentation:

Pooling connections can significantly enhance the performance and scalability of your application. The .NET Framework Data Provider for OLE DB automatically pools connections using OLE DB session pooling. Connection string arguments can be used to enable or disable OLE DB services including pooling.

All right, so my first guess is that the connection pooling doesn't work for some reason and after the last active connection is released the pool (if there is a pool created at all) is purged. I know that I can force to release the pool by calling OleDbConnection.Release, but I want the opposite effect. To test my hypothesis I made a simple class and executed its methods TestWithActiveConnection() and TestMultipleConnection(), using the TestDriven.Net VS.NET AddIn. So let me show you the code and then the results.



/// <summary>
/// Test the performance of the Ole DB connection pooling for Microsoft.Jet.OLEDB
/// with and without an active connection in the lifetime
/// </summary>
public class ConnectionPoolTest
{
public OleDbConnection activeConnection = null;
public string ConnectionString = @"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\test.mdb";
public int IterationCount = 5;

public void TestWithActiveConnection()
{
DateTime start = DateTime.Now;

//Open a connection with the same connection string, during the test lifetime
activeConnection = new OleDbConnection(ConnectionString);
activeConnection.Open();

DateTime end = DateTime.Now;
TimeSpan span = end - start;
Trace.WriteLine( "The connection that is to be active during the test was opened for: " + span.TotalSeconds.ToString() );

TestMultipleConnection();

activeConnection.Close();
}

public void TestMultipleConnection()
{
//the first call to TestConnection()takes more time
//because the JIT compiler has to prepare the function
//so it will be excluded in the total time statistics
TestConnection();

DateTime start = DateTime.Now;
for(int i = 0; i < IterationCount; i ++)
{
TestConnection();
}

TimeSpan span = DateTime.Now - start;
Trace.WriteLine( IterationCount.ToString() + " connections opened and then closed for a total of: " + span.TotalSeconds.ToString() );
}

public void TestConnection()
{
DateTime start = DateTime.Now;

OleDbConnection conn = new OleDbConnection(ConnectionString);
conn.Open();
conn.Close();

TimeSpan span = DateTime.Now - start;
Trace.WriteLine( "Single connection opening and closing took: " + span.TotalSeconds.ToString() );
}
}



I ran the test multiple times in a different order. But the results are following the same pattern and it can be illustrated by this sample output:

1) Test execution output using an active connection

Active during tests connection was opened for: 0.1201704
Single connection opening and closing took: 0.0600852
Single connection opening and closing took: 0.0300426
Single connection opening and closing took: 0.0200284
Single connection opening and closing took: 0.0300426
Single connection opening and closing took: 0.0300426
Single connection opening and closing took: 0.0200284
5 connections opened and closed for a total of: 0.1802556

2) Test execution output without active connection

Single connection opening and closing took: 0.1802556
Single connection opening and closing took: 0.0700994
Single connection opening and closing took: 0.0600852
Single connection opening and closing took: 0.0600852
Single connection opening and closing took: 0.0600852
Single connection opening and closing took: 0.0600852
5 connections opened and closed for a total of: 0.350497

So what does this mean? It is apparent that that the presence of an actively opened connection made the test with multiple connection closing and opening finish a lot faster (2-3 times). The only possible explanation for me is that the connection pool is released each time there are no active connections. I have to make further investigations and read something like Pooling in the Microsoft Data Access Components. Or maybe hold a single opened connection just for the sake of keeping the pool alive. Pfu, this would be ugly, but still it is a good enough workaround! If anyone has a better idea, please share it.

January 27, 2005

.NET Multi-Threading

Currently I need to do some .NET multithreading, so I searched for some online resources. Just want to point out that the most comprehensive material about threading that I found is Jon's 17 sections detailed study. Look at the resource section if you need to dig even deeper. This guy has some other good stuff on C#, too.

January 25, 2005

Transition of my DAL completed

It has been long after my last post on my Data Access Layer :)

Since then I did quite a good job and it seems like that the transition is finally completed with success. In the transition process I employed some nice practices. The first one is obviously Refactoring and I have to say that I loved Resharper . It is very polished and fast (definitely faster than C# Refactory, though more memory hungry). After a week of using it and playing with its features, I have seen its error dialog only twice and experinced no data corruption or unexpected behaviour. I definitely recommend it to everyone who spends a lot of time with VS.NET. I hope that I will be able to persuade my employer to buy a few licenses, since my trial license is not going to last forever.

The next thing that saved me a lot of trouble were a few NUnit tests. I designed a few simple generic tests intended for my Object-Relational classes. I have a MS Access DB with sample data for test purposes in a network share. So the NUnit tests first make a temporary local copy of the DB and then do what they need to. My first test was to take a single row from each table and try to load its data in the corresponding business object. And then simply call Update() without changing anything. This test failed about 15 times before I got everything right. Imagine how long would it take me if I had to identify each error manually within the application, performing all the steps required.

Oops, I almost forgot! My DAL changed quite a lot and I have put more explanatory names to many functions and properties and I hav added new functions. So, I updated my XML comments, then downloaded the latest NDoc and made a good looking MSDN-like CHM documentation. This is, actually, the best commented project I have done (thanks to GhostDoc, which made it easy). But it looks rather awkward in the sea of uncommented (but, I hope, self-explanatory) code of the solution :)

UPDATE: It seems that Blogger ate a part of my sentence about Resharper. This is the second time when I post something using its web WYSIWYG editor and it cuts a portion of my text. Also some typos are fixed and the story is better told. Argh, I know that I should use more drafts and do a few more reviosions...

January 20, 2005

Ha, funny coincidence

While I was looking at Steve McConnell's Classic Mistakes Enumerated (which is an excellent read), I saw this:

In Case Study 3-1, the team knew that Chip was a bad apple, but the team lead didn't do anything about it. The result--redoing all of Chip's work--was predictable.

The funny think is that my ICQ nickname is chip@ and it has some background. I used to call myself Chip when very, very little. But even nowadays I answer to this nickname. So I decided to find out why Chip was referred to as "a bad apple" (I got personal), so I found the answer here.

OlyMars - SQL Server Centric .NET Code Generator

Today I stumbled upon a project of Microsoft France, called OlyMars (this alien name is derived from OLYmpique de MARSeille). It is currently in Beta, but looks like a promising solution that will save a lot of manual work:

SQL Server Centric .NET Code Generator (code named OlyMars) is both a flexible and powerful code generator based on database modeling.It allows instant generation of both SQL and .NET code providing a complete library of stored procedures, .NET classes and ready-to-use Windows/Web form controls (including associated documentation).
SQL Server Centric .NET Code Generator is also fully extensible to use one's own custom templates and consequently can be adjusted to generate any custom code respecting a homogeneous implementation scheme within the company (can be written either in VB .NET or C# .NET).

Did I forget to tell that it is free? Yeap, and its documentation is quite OK to get you started...

Btw, while searching for additional materials for OlyMars, I found various interesting stuff:
O/R Mapping and all that jazz - some thoughts on Code Generation and OR Mapping
Code Generation Portal - it seems that code generation practice is gaining huge popularity, I can count more than 50 C# code genarators listed and the portal is not only .NET oriented:)

January 19, 2005

My favourite .NET resources

I was initially planning to write here about my favourite WindowsForms.NET resources (well, they are still listed at the bottom of the post). But I took another direction trying to cover the huge topic of .NET resources. The list, of course, is not complete and is not intented to be. It would be a few days job to link to all .NET projects, tools, articles, weblogs that I have stumbled upon.

But still, when I have to search for some .NET stuff, this list hopefully is going to be a reminder where to look for ready-to-use solutions or know-how. Of course, Google is generally a good enough place to search, but sometimes a greater focus is needed. And you know the anient lore: "Two weeks of coding save one day of doing research" [it is based on the even more ancient piece of lore: "One week of die-hard coding saves two hours of reading documentation"].

Alright, let's get started:

General .NET resources
CodeProject - by far my favourite place for .NET articles and sample code
GotDotNet - I wish it was not down so often, since a lot of good stuff is hosted there. Once every few months I just go to browse what's cooking in the workspaces
.NET Framework FAQ and C# for C++ programmers FAQ by Andy McMullan - pretty good stuff
FAQ for the microsoft.public.dotnet.languages.csharp newsgroup - Jon really has some good C# stuff

Listings of .NET Tools
C# Team Tools - essential tools listed by the highest authority :)
SharpToolbox - astonishingly comprehensive, great search capabilities
CSharpFriends Tools - quite a few essential tools listed
MSFT .NET Tools and Utilities - FxCop, CLR Profiler etc.

Other .NET Open Source Tools, Projects and Utility Classes
Lutz Roeder's tools and source code for .NET - we all know that Reflector rocks!
Jeff Key's tools, code and other .NET resources - plenty of very cool stuff
Menatlis classes and complete projects - some useful classes
IDesign classes - tools and utilities developed by IDesign
.NET Monster open source projects - huge list of cateforized open source projects
CSharp-Source.NET open source projects - pretty good list of open source projects

Microsoft .NET Code Samples
101 Visual Basic and C# Code Samples
23 C# Code Samples

WindowsForms.NET Resources
CodeProject C# Windows Forms - the first place, I check when I have to deal with WindowsForms
WindowsFoms' Control Gallery - no wonder that www.windowsforms.net is a great place for WindowsForms stuff, the name says it all
C-SharpCorner/WindowsForms Controls - good stuff
Genghis Project - Chris Sells and other folks do a great job with various of controls and other .NET utility classes
SyncFusion WindowsForms FAQ - very comprehensive FAQ, saved me hours and days of experiments

January 18, 2005

.NET ORMs revisited

Following my post from November, where I listed the leading free and commercial Object-Relational Mappers for .NET, I have to make an update. Today, I read [via Larkware] that Codus has been released for free (though it was intented to be a commercial product). It is developed by Sean McCormack and I am pleased to see how powerful it is (it's much more than ORM). With a few clicks you are able to generate ORM code, XML documentation, unit tests and a VS.NET solution for your database. Unfortunately, I've got an exception of a missing file when trying to do so, but I guess this will be fixed soon.

I also want to point out that another ORM Neo seems going pretty well. I liked very much the rich documentation available and I was pleased to know that it has been deployed in a few ThoughtWorks projects. I should get to know it better!

So looking at those great free products I wonder if it was worth doing my own ORM mapper. Well, I learned a lot about the problems in this domain, but now I have to maintain it :)

UPDATE: I am glad to discover that Sean McCormack has posted a comment to my post, asking me for a description of the problem, that I experienced, while playing for the first time with Codus. So, today, I ran again Codus to see what exactly the exception was. But I could not reproduce the problem. I remember that I got a ResourceManager exception of a missing file in the \template directory. I have verified the file was there and the only possible cause, that I can imagined of, was that it was referred by its relative path and probably the base was wrong.

But now a few days later (and after a restart of my computer), every aspect with the code generation went smooth. I generated the data objects, strongly-typed collections, NUnit 2.2 tests, Zanenbur 1.4 tests and VS.NET solution. And in a less than a minute I got my VS.NET solution, based on 23 MS Access table and 7 queries. Well, the solution in fact did not compile, because there was a table with a column named 2 (no, it's not me, I never imagined that such numeric name was a valid column name in any DBMS). So I skipped the problematic table in the next generation attempt and "yeah, no problem".

January 17, 2005

Making the switch to a new version of my Data Access Layer

In a previous post I explained I am working on a .NET project, which functionality I have to extend. So now I am changing the Data Access Layer, which is basically a Object-Relational Mapping, using ADO.NET (with OleDB Providers) for data retrieval and manipulation. I have blogged about it here and then here (when I was trying to "sell" it to my colleagues).

For my second .NET project I redesigned the DAL. I added new features and made it more customizable. The current version supports transactions, multiple DBMS (tested on MS Access and Oracle). Also a few bugs were fixed and a few issues were resolved . But some of those modifications were compatibility breakers... And that's what makes it hard for me to migrate to the newer version.

The most significant change is that the business objects, which inherit from a specific class, no longer have a member property OleDbConnection. This is IMHO a better design, because it eliminates the tough question "who owns the connection?" (and therefore is responsible for closing it).

So, when an object needs to communicate with a DB, it is given a OleDbConnection and OleDbTransaction by the caller, so the caller should take care of it. There are 3 versions of each DB-related method RetrieveByPrimaryKey(), Insert(), Update(), Delete(). Version A takes no parameters and uses a default connection, version B takes only a OleDbConnection as a a parameter, while version C takes both OleDbConnection and OleDbTransaction. Of course, internally A calls B (with a "using" block around the connection) and B calls C (with null transaction object).

This change affects a lot of functionality and requires tedious refactoring. But I just has to be done, since in the older version it was practically imposible to know when to close a connection, without risking to affect other functions. And you don't want to leave opened connections behind you, because sooner or later the connection pool (that's created by default) won't give you any more. I made a test with MS Access connection: the 64-th attempt to establish a connection without closing any connections was fatal. By the way, the error message I got from Microsoft JET Database Engine was "Unspecified error", which is not very informative :)

UPDATE: Look at the comments section for some further investigation about the OleDbException when running out of connections (MS Access + Oracle).

VS.NET designer tips

Matthew Adams, who appears to be a WindowsForms specialist, has recently started to blog and has posted some very useful tips how to avoid VS.NET designer mischiefs and to repair from them painlessly. I, personally, have had occasionally problems with the designer, which lead to controls disappearing along with some code. Well, if one uses source control, the damage can not be that bad, but still it is unpleasant.

January 13, 2005

Have I written this shitty code?

Recently, I have been involved in some maintenance programming. Fortunately, I am fixing and extending my own code, not someone else's.

I thought that everything would go smooth, since I was sure I wrote high-quality maintainable code, conforming with the best practices. But, it appeared that I was wrong! In not so rare occasions my code was just shitty! It works most of the time as intended, but this does not mean that it does not suck (ref: Why Your Code Sucks).

I have to make it clear I had not used intentionally any evil techniques described here :) . But, nevertheless, the result of my efforts is a highly coupled code with not proper object responsibilities and quite a few duplications, thus making it hard to extend and fix. Not to mention the .NET solution structure, which is far from perfect, and needs reorganization. Well, I am talking here about my first .NET project. I just realized how much I had learned since then. And now I am facing the challange to migrate my knowledge and code from my second .NET project to the previous one in a painless manner. Having just a few NUnit test classes, apparantly, does not make my job easier, but probably I may write some, as I go...

January 12, 2005

Numbers and .NET

Wow, a post by Wesner Moise reminded me how simple things in programming that turn to be false assumptions can break your code. How many of you have bothered to read about the Double structure. I guess not too many (you have been using doubles for ages, right?) and the chances are that you are not aware of NaN, PositiveInfinity, NegativyInfinity etc. And probably in some rare boundary cases your code will not behave as you thought it would. This is another scary discovery after my number rounding enlightment.

Free Widgets

Yesterday I ran into a great free time-limited offer for some WinForms and WebForms controls [via Larkware]. So I ordered online and recieved a serial number to activate the free copy of Graphics Server .NET: Widgets Edition 2.5 that I downloaded. Then I looked at the excellent documentation and tried the controls in VS.NET. There are number of widgets: Meter, Slider, Gauge, StopLight and Thermometer and a Charting control. One of the nice things is that it comes with some designers for VS.NET that are quite helpful for a quick start. I am not sure if those widgets will come of any use in the near future, but who knows. Btw, I think, such free of charge offers are a great way to popularize one's product...

Interview with Peli

I finished listening to an interview of Jonathan de Halleux (known in the blogsphere as Peli) conducted by Roy Osherove. I have been an admirer of Peli's work and his outstanding productivty in variety of projects: MbUnit, some cool Reflector Add-Ins, QuickGraph, TestFu etc. This guy's geekiness is just hard to imagine, he is absolutely passionate about what he is doing. The interview is very good. Peli covers a lot of topics about his past work and what he is doing for Microsoft now. But it seems that it is removed temporarily from the interviews download area. I guess it will reappear soon.

January 07, 2005

The trap of "On Error GoTo"

I thought I got VB6 error handling long ago... I have been programming VB6 for more than 2.5 years. And until today I was sure I knew everything about those ugly On Error blocks:

Resume [0] - Program execution resumes with the statement that caused the error or the most recently executed call out of the procedure containing the error-handling routine. Use it to repeat an operation after correcting the condition that caused the error.
Resume Next - Resumes program execution at the statement immediately following the one that caused the error. If the error occurred outside the procedure that contains the error handler, execution resumes at the statement immediately following the call to the procedure wherein the error occurred, if the called procedure does not have an enabled error handler.
Resume line - Resumes program execution at the label specified by line, where line is a line label (or nonzero line number) that must be in the same procedure as the error handler.

This is an excerpt from Designing an Error Handler which is a part of Debugging Your Code and Handling Errors.

So, everything about "On Error GoTo" seems clear. You, usually, place it in the beginning of a procedure, function or property and add a label "eh" in the end, below which you write the error-handling logic. In 90% of the cases this is enough. But there are more complex scenarios. For example, you might have a loop and want to catch errors, log them and resume with the next iteration (recover from the errors). This seems to be possble to be achieved in two ways (at least I thought so, until I tried):

  1. Use "On Error Resume Next" and check Err.Number in each iteration (and don't forget to call Err.Clear when finished dealing with the error)

  2. Use "On Error Goto eh" and after trapping the error log it and continue looping with Goto statement

I chose to implement the second approach, because my code was more intuitive that way. And when an exception eventually occurs the control immediately goes to the statement after the label. But my program behaved peculiarly and I made a little test to investigate what goes wrong. I was astonished to find that after an error is trapped by "On Error Goto eh" no further error handler could be activated within the same procedure. So the first failure gets handled properly, but a second ican not handled at all. I called Err.Clear afterwards, but with no positive effect! I thought that probably I had to reactivate the handler by writing another "On Error Goto eh" statement. But this again did not work. Even "On Error Resume Next" had no effect any more. So, I am giving-up! I have to reimplement my error-handling code in about 20 procedures to use the "On Error Resume Next" approach. Argh, this is just plain stupid!

One of the sad thing about this is that I found no documentations that warns me about such odd behaviour (I only managed to dig a post by another guy with the same problem here). This is one of the lessons, learned the hard way... :)

January 03, 2005

Selected IT Conversations

Last month, I had a chance to listen to two very good series on IT Conversations.

The fist one was Developer Testing by Kent Beck. Recommended listening! For details read Don's very comprehensive notes on the presentation.

And the last workday of year 2004, just before leaving work, I listened to Hiring Techies and Nerds, Johanna Rothman interviewed by Roy Osherove. A very good talk! If you ever need to conduct a technical interview or participate in hiring decision, you better listen!