May 212011
 

Download the source code for this post (10k)

In the previous installments of this series, we worked completely within the confines of the AutoExp.dat framework which was pretty limiting. That’s not really surprising given that that mechanism dates back at least 15 years as of this writing. That was back before the dot-com bubble happened, I think most people still had non-vestigial tails, and dinosaurs still roamed the earth in the some rural areas. It’s been awhile. In this installment, we’re going to take things up a notch by using mechanisms that are only a mere 10 years old – that’s right, a time after most of you were born. Crazy!

We want to deal with several of the big limitations and unknowns we ran into in the previous two parts. Unfortunately, that will have to wait until part 4. First we need to update the way we hook into the debugger. Microsoft has been developing and expanding the Development Tools Environment framework or ‘DTE’ which is at the heart of Visual Studio extensibility since VS2003. It contains quite a lot of interfaces that allow you to inspect and extend several areas of Visual Studio. The scope of what is available is actually pretty impressive. This is one of the interfaces that make products like Visual Assist X possible, But trust me, there’s a lot more to it!

Within DTE, you can do simple things like add new menus and buttons, or you can write more complex tasks like traversing the solution hierarchy. You can also write code that automates certain coding tasks and even add an entirely new language. Clearly, we’re not going to use most of this stuff. For our part, we really only care about hooking some events in the debugger and then doing some simple expression evaluation.

So what’s the plan here, anyway? First, we want to get a new add-in project up and running that will host our next steps. Thankfully, the new-project wizard has a decent template to accomplish most of that. Then we want to get access to the debugger to tell us when the user is debugging something.

A Brave New Project

If you select File->New->Project and then look through the project templates under “Other Project Types” and then “Extensibility”, you’ll find “Visual Studio Add-in” as an option. There are several options in the ensuing wizard, but most of them are just nice-to-have features. For this project, I selected a C++/ATL project and disabled as many of the add-on features as possible.

Right out of the gate, we get a many-filed project with lots  of inscrutable plumbing. You can download the new project here (10k). Let’s do a quick run-down of what’s in there:

stdafx.h/.cpp – As usual, this is the special header that drives the precompiled headers feature in the compiler. If you look closely, however, you’ll see several #import statements. Depending on your version of Visual Studio, they might contain some messy GUIDs or they might contain references to files like “dte80a.olb”. In either case, these are the large sets of interfaces that we’re going to be using  to integrate with Visual Studio.

Addin.rgs – This odd little file is actually a registry fragment that will be used to register your add-in with Windows. Don’t panic, but we’re actually making a registered component here. The registry is the mechanism that Visual Studio uses to find your add-in.The top section registers your component with Windows while the bottom section registers your add-in with Visual Studio. Don’t worry, this stuff will happen automatically.

Addin.cpp – It might seem like this is where you’ll put the important code in your add-in, but it actually just contains some basic plumbing code that allows your add-in to be a DLL and to register itself as a control.

Addin.idl – IDL stands for interface definition language. It’s used to generate data marshaling code for your control’s type library. Again, don’t worry about this file. We won’t be editing it.

Connect.h – This is where the CConnect class lives. It is the main class for your add-in. It will implement the Visual Studio interfaces that we want to use. Notice that CConnect already derives from something called IDTExtensibility2. This is the main interface that makes a class into a Visual Studio Add-in.

Connect.cpp – THIS is where all your awesome code can go, but notice what’s in there already. It has functions called OnConnection and OnStartupComplete. These functions are part of Visual Studio’s IDTExtensibility2 interface that was mentioned above. These functions will be called at various times as your add-in is loaded. Also note in OnConnection, there is some code to fill in a class variable called m_pDTE. This is the access point to most of the things we care about in Visual Studio.

When you build, two non-standard things will happen. First, a few more files will be generated. This is the IDL file doing its thing and creating its glue code. This code is generated every time you compile, so don’t bother changing it. You don’t even have to add it to the project, so just ignore it unless you’re curious what’s in there. The second thing that will happen was your new control should have been quietly registered with Windows. This is important when you start debugging your code. It doesn’t matter if Visual Studio is pointing to the Debug build profile. It will debug whatever profile was built most recently. Keep that in mind as a possible gotcha when things seem to be going all wrong.

Just like last time, if you run your add-in, you will start a new copy of Visual Studio – it’s like launching an aircraft carrier to test out the galley appliances. Go ahead and look under “Tools->Add-in Manager…”. You should see MT_Addin3 on the list of add-ins and it should be activated. If you put some breakpoints in Connect.cpp before you ran, you should have gotten a break or two in OnConnection, OnStartupComplete, or one of the other IDTExtensibility2 handlers.

Hooking a Debugger

(Easier than debugging a hooker! – hey-o!)

Everything we’ve touched on so far is just to get an add-in working inside of the Visual Studio framework. We won’t get the notification we really want. We want to get the chance to act just before the watch windows are updated. In other words, we want to be told whenever the target program is halted, and the user can inspect variables. That requires that we hook the debugger events interface.

We only need to hang a few new bits of code on out CConnect class in order for it to handle debugger events. For now, we only care about the event called OnEnterBreakMode. This is the event that is fired every time the debugger stops the execution of the target process. The obvious case is when the debugger encounters a breakpoint, but this function will also be called every time the user steps into, out of, or over a line of code.

You can follow along in the code sample for these changes, I’ve denoted each one with the comments

//*** Hooking debugger events ***/

The first thing we need to do is make out CConnect class derive from the debugger events interface. This is as simple as adding the following line of code to the class inheritance in Connect.h:

IDispEventImpl<0, CConnect, &EnvDTE::DIID__dispDebuggerEvents, &EnvDTE::LIBID_EnvDTE, 8, 0>

That’s a bit of a mouthful, so in the sample code, I use a simple typedef to make it easier.

The next thing we need to do is tell CConnect what events it should expect and then what to do with them. For this, we need to add  a sink-map:

BEGIN_SINK_MAP(CConnect)
 SINK_ENTRY_EX(0, EnvDTE::DIID__dispDebuggerEvents, 3, OnEnterBreakMode)
END_SINK_MAP()

This is essentially saying that when the EnvDTE::DIID__dispDebuggerEvents interface sends event #3 our way, we should shunt it to a function called OnEnterBreakMode. But wait, that seems a little voodoo – just a magic number three? Honestly, COM is hardly my strong suit, and Im still trying to track down the origin of this value. Its likely tied up in the definition of the IDispatch interface thats part of the debugger events object.

The other part of this event-sink mechanism is attached and detached at run-time from the Add-in’s OnConnection and OnDisonnection handlers. We use the DTE interface once again to get access to the events interface and then ask for the debugger events interface specifically. Once we have the debugger events interface, we can simply register that we want to be notified about those events.

Finally, we declare the CConnect::OnEnterBreakMode function in Connect.h and make a stub for it in Connect.cpp.

A Quick Test Drive

Set a breakpoint in the OnEnterBreakMode function and run the project. When the other copy of Visual Studio comes up, load up another project in it, set some breakpoints, and then run THAT project. When one of those breakpoints hits, you should get popped all the way back to the first Visual Studio with the Add-in project. Neat, huh? Now we can really do whatever we want within Visual Studio, so take a look at what’s available. Next time we’ll be looking at the GetExpression specifically.

Ok, that’s it for now. My apologies that this posting was mostly just a building block for what’s to come. I promise we’ll tie some stuff together next time out.

 

 Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>