Remotely debugging a Windows Azure webrole via Visual Studio Debugger

UPDATE: Even after 3 years this is still popping up on Google’s first page results, so worth mentioning Visual Studio makes this a LOT easier now. Original post below.

Warning: This is a highly technical post and beyond the first two paragraphs requires a deep understanding of cloud web engineering (especially Microsoft Azure)! This is more for my own journaling than for any broadcasts …

Background

We were facing some issues with our cloud application that runs on Microsoft Azure. Locally the stuff would work just great but on the cloud some minor issues would be seen. Not wanting to push them under the carpet we decided to look deeper. Of course, this meant that we had to connect the debugger straight into the heart of the code and look closely.Feel free to skip the next section to get straight into the engineering nitty gitties.

A moment of admiration

Debugging locally, where the source code and the running program on the same computer system, is usually alright but doing this in the “cloud” is quite challenging. Effectively what we did was run the debugger (Visual Studio 2012) locally and then attached it to the server program in the cloud. If you think about that traversal it’s pretty extraordinary! We’re diving deep down, literally, into a Virtualized Server OS’s bits and bytes living on some ram chip, somewhere in a server rack, somewhere in the data center, over a virtual network spanning hundreds of miles (or even across the planet for US-Australia/India links!!) using a development machine that itself runs a virtualized operating system. All that inside a live ‘real time’ environment, mostly limited by distance latency (speed of light). That is truly remarkable, worthy of a moment’s pause to digest it.

So lets get to it…

No surprises, this is a multi stage process so I’m breaking it down into different steps.

1. Setting up Visual Studio

• Getting the right publishing settings

When you publish your webrole to Azure, the most important part is to

• Enable Remote Desktop
• Build settings should be “Debug” (so symbols are created)

These are shown below (right click azure project in solution explorer => Publish)

Other optional but helpful settings are

• Enabling Web Deploy.
  Web Deploy is GREAT for debugging since you can make small changes and ONLY push that out (within 10 seconds) instead of re-publishing the entire web-role (10-15 minutes). Drawbacks are that changes are ONLY for that specific instance and if the instance is recycled, you lose the changes. Unacceptable in life production but not a problem during debugging.Note: For web deploy, you have to right click the actual underlying .NET project (NOT the Azure project) in solution explorer and then choose “Publish”!? I know it’s weird but that’s how it is.

• Enabling Intellitrace or Profiling
  These can slow execution so enable them if you know you need them

• Enable loading of symbols

During local debugging Visual Studio finds the symbols without a problem. However, for remote debugging it needs to be explicitly told when to find it’s symbols. Type the path to the place where your Webrole project puts all it’s .pdb files (usually the <path to your webrole project>bin folder).

Visual Studio automatically assigns a temp folder to cache the symbols locally too. That is useful IF your symbols are also remote on a windows file server or symbol server. I didn’t care so I left the default Visual Studio filled.

2. Setup a Virtual network

We first create a virtual network between the cloud servers (“webroles” in Azure terminolgy) and your development machine. Note that Azure is currently moving to a new portal (currently beta) and has a bug where it wouldn’t let me create a virtual network under my work Azure subscription. I have two subscriptions and this might be tripping it. Not sure. Anyways, you install one end on your computer and you setup the webrole as the other endpoint so they can talk directly and securely. You could try this over the public network interface too but then the only thing protecting you is your Windows authentication password (which here is your ‘remote desktop’ username and password that remote debugger uses). The Virtual network method is a lot safer for the integrity of your application and IP.

Side note: The virtual network seems IPv6 based on the old portal. Based on the configuration UI on the new portal it’s IPv4 based on the new portal? Weird!

• Setup local computer as endpoint

First go to the portal and download and install the software for the local end point.

• Webrole as the other endpoint

Once done, go back there and click the activation token and copy that (see above). With that handy we give it to Visual Studio. Now when you (re)-deploy your webrole, it will know it’s part of the virtual network. This is shown below

• Now (re)publish your entire webrole.

Do NOT skip this step – almost nothing will work! All the above changes so far need to be reflected in the cloud before moving on. And our quick and dirty web deploy doesn’t work either since role level changes have taken place. So just right click your azure project and publish to Azure (hopefully your staging/testing area). This is your 10 minute break!

• Creating the actual virtual network in between

Now you’ve published the web role – great. We’ve defined two end points and say they belong to “this virtual network”. But we didn’t really define what that network really looks like. Azure does most of the heavy lifting in IPv6 but we do have a simple step. So lets do that now. Go to the old portal => Virtual Network => your subscription => “Create Group”. Azure should now be aware of your two end points – your local computer and your webrole. Add each one, enable the checkbox “Allow connections between …” and hit save. That’s it! Azure takes care of the VPN infrastructure/setup itself!

3. Getting Remote Debugger on webrole

• Remote into the Webrole

In Visual Studio, go to server explorer (usually on the right edge), open “Windows Azure Compute” => your webrole => an instance of your webrole => right click => “Connect using remote desktop”. Alternatively you can go to the portal, navigate to the webrole’s instance and connect from there. You will use your remote desktop username/password you setup in your publishing settings

• Make Internet Explorer ‘usable’

You need to download the Visual Studio 2012 remote tools and install them onto the WebRole instance. The easiest way is to open Internet Explorer and download the file straight from Microsoft. However, all Windows Azure servers have a default policy preventing IE from working normally (“Enhanced Security Configuration”). So we temporarily disable it, get the software and then can enable it again. To do that, remote desktop into the Webrole, launch Server Manager (next to the start menu). Hit “Configure IE ESC” on the left and disable it as shown below.

• Launch IE, download and install tools

While still remoted into the webrole, go to start => launch Internet Explorer (on the webrole, not your local development machine!). Download the Visual Studio 2012 remote tools and install it. As of writing this, the Visual Studio tools live at http://www.microsoft.com/en-us/download/details.aspx?id=30674. You want the 64 bit version since Azure servers (build out of Windows Server 2008 R2) are 64 bit based. No image here. If you can’t download and install a simple piece of software, forget this and go get yourself a nice martini 🙂 !

• Configure Remote Debugger

Run the Remote Debugger Configuration Wizard. I set it up as follows. The screen that shows which networks it will work on basically automatically adds rules to the windows firewall to open it up on those interfaces/networks. I didn’t want any debugging over the actual web-facing interface, so I kept it to domain and private only. Just Private would also work since the VPN we setup earlier is a ‘Private’ network, and there is no Domain interface. I left the authentication mode as the default (“Windows Authentication”) so you’ll need to supply the same credentials as when remoting into the webrole via Remote Desktop. I also left the TCP port as the default – it actually doesn’t matter since the firewall rule the configuration wizard sets gives access to the debugger to open and listen on any port.

• Launch remote Debugger

If you didn’t setup the Remote Debugger as a service you will need to manually launch it before connecting. While remoted into the Webrole, hit start, then Remote Debugger to launch it. You will see the actual name of the server:port (like: RD001122334455:4016) in the window, make note of it, Visual Studio will want it soon.

• Prevent process termination

By default IIS will terminate IIS world wide web server worker processes (w3wp.exe’s) after some inactivity. The inactivity is configurable within IIS and just to be safe, I disabled this behavior so that it doesn’t kill the process during debugging itself. Note that this is undocumented, but setting it to zero work.

4. Connect via Visual Studio

• Spawn an IIS web process (i.e. bring the webrole to life!)

By default, there is no IIS worker process (i.e. no w3wp.exe) until the very first request come in. Also, the web process is also automatically terminated after inactivity (but we fixed that 2d problem above). So fire up your browser and go to your website/webrole.  Do something to trigger your webrole and – voila! IIS will create a w3wp.exe process for you!

• Attach to the  IIS web process (i.e. w3wp.exe)

Now fire up Visual Studio, load up the solution or project holding the WebRole. Now go to DEBUG => Attach to process. Enter the name of the server and hit Enter (on your keyboard). The misleading UI makes it look like you should hit “Find” but that attempts to auto-find it and just fails. Just hit enter. I was prompted to enter valid credentials and gave Visual Studio the same credentials that I usually give Remote Desktop (“rdpUser” for me).

5. Tada! Debug!

After you hit attach above, Visual Studio should be connected to the remote process and you can actually remotely debug! It’s not obvious that “things are working” but you should see the “Play/Continue” buttons greyed out and the “pause/stop” buttons active. So open your source file from the solution explorer and set a breakpoint. When you hit it, you’ll see the debugger similar to the screen below.

Caveats?

• Well, the above IS incredible great. However, given that usually you will be several hundred to thousands of miles away from the actual executing machine, there are speed-of-light and also software processing latencies. Even with broadband, expect 10-20 second delays at times when compared to direct local debugging. I don’t think it’s a network optimized protocol so the protocol probably involves MANY round trips amplifying the latencies. I don’t think it’s a network optimized protocol so the protocol probably involves MANY round trips amplifying the latencies.
• Since stopping at a breakpoint in the debugger freezes the process, you can’t use this in live systems. This should be obvious but still wanted to call this out. You need tracing or logging for those cases or run just create a separate set of testing or debugging servers.

Troubleshooting

• I always run my Visual Studio under admin privileges. In case something doesn’t work, try that.
• If Visual Studio can’t connect to the Webrole’s remote debugger
  • Check the firewall settings on the webrole. The remote debugger configuration wizard (remote to webrole => Start => Remote Debugger Configuration Wizard) usually opens up the firewall fine, you might just have to re-run it.
• If you can connect but no symbols are seen:
  • You might have skipped the step above where you explicitly tell Visual Studio where the symbols are found.
  • Check the above path for typos and then see what folders it’s actually scanning. First attach the debugger to the remote debugger. Now go to Debug (top menu) =>Windows => Modules and you can see what folders were scanned and what symbols were picked up.
  • Make sure you are not using a stale folder or ones with older/out of sync .dlls/.pdbs !
• If it says “The breakpoint will not currently be hit. No symbols have been loaded for this document.” many times it WILL still break at that breakpoint, THEN load the assembly (which loads up the symbols) and that message goes away.