One Azure Web Role, Multiple Websites

by Brian Hitney 17. January 2012 13:27

Windows Azure has been capable of running multiple websites in a single web role for some time now, but I found myself recently in a situation with 2 separate Azure solutions and was looking to combine them to create a single deployment.   Just like in IIS, this is most often done via host headers, so requests coming in can be forwarded to the correct site.

The Goal

The fine folks at infragistics created a really cool Silverlight-based reporting dashboard for Worldmaps.  Until now, each was running as its own Azure hosted service:

image

Options to consolidate included folding the code into the Worldmaps site, which would involve actual work, or converting the site to use IIS instead of the hostable web core (HWC), which was, originally, the only way to host Azure deployments prior to version 1.3 of the SDK.  Under IIS, host headers can be used to direct traffic to desired correct site.

Preconsiderations

Inside the ServiceDefinition file, the <sites> section is used to define the websites and virtual directories, like so:

<Sites>
<Site name="Web" physicalDirectory="..\WorldmapsSite">
<Bindings>
<Binding name="HttpIn" endpointName="HttpIn" />
</Bindings>
</Site>
<Site name="Reporting" physicalDirectory="..\..\igWorldmaps\WorldmapsDemo.Web">
<Bindings>
<Binding name="HttpIn" endpointName="HttpIn" hostHeader="reporting.myworldmaps.net" />
</Bindings>
</Site>
</Sites>

Nothing too crazy in there, but I’ll talk about the paths later.

The first problem is that I was using webrole.cs file in the Worldmaps application, overriding the Run method to do some background work:

public class WebRole : RoleEntryPoint
{
public override void Run()
{
// I'm doing stuff!
}
}

The Run method is called from a different thread, and it did a lot of background processing for the site (logging data, drawing maps, etc.).   This is a great technique, by the way, to add “workers” to your website.  This is, by itself, not a problem to do under IIS or HWC, except, under the HWC version, the thread runs in the same process.   I could write to an in-memory queue via the website, and process that queue in the webrole.cs without problem, provided the usual thread safety rules are obeyed.  Likewise, the worker could read/write to an in memory cache used by the website.  Under IIS, though, the site and role are in a separate process, so it wasn’t possible to do this without re-engineering things a bit.  You don’t need to worry about this if you aren’t doing anything “shared” in your webrole.cs file.

Add the Project

In my existing Worldmaps solution, I added the infragistics “WorldmapsRporting” project by adding the project to the solution (right click the solution, and choose Add Existing Project):

image

Hook it Up

The <sites> tag (seen above) is pretty self-explanatory as it defines each site in the deployment.  For the first and main site, I didn’t provide a host header because I want it respond to pretty much anything (www, etc.).  For the second site, I give it the reporting.myworldmaps.net host header. 

Here’s the tricky part, which in retrospect seems so straightforward.  The physicalDirectory path is the path to the web project, relative to the Cloud project’s directory.   When I first created the Worldmaps solution (WorldmapsCloudApp4 is when I converted it .NET 4), I had the cloud project, the website itself, and a library project in the same directory, like so, with the cloud project highlighted:

image

So, the path the WorldmapsSite is up one level.  To get to the infragistics website, it’s up to levels, the into the igWorldmaps folder and into the WorldmapsDemo.Web folder.  We can ignore the other folders. 

DNS

The project in Windows Azure is hosted at myworldmaps.cloudapp.net, as seen from the Azure dashboard:

image

…but I own the myworldmaps.net domain.  In my DNS, I add the CNAMEs for both www and reporting, both pointing to the Azure myworldmaps.cloudapp.net URL (picture from my DNS dashboard, which will vary depending on who your DNS provider is):

image

Testing Locally

To test host headers on a local machine, you’d need to add the DNS names to your hosts file (C:\Windows\System32\drivers\etc\hosts) , like so:

127.0.0.1    myworldmaps.net
127.0.0.1    www.myworldmaps.net
127.0.0.1    reporting.myworldmaps.net

Overall, a fairly straightforward and easy way to add existing websites to a single deployment.   It can save money and/or increase reliability by running multiple instances of the deployment.

Links

http://www.myworldmaps.net

http://reporting.myworldmaps.net

Tags: ,

Azure | Tech Tips | USCloud | Worldmaps

Music Library Synchronization, Sonos Tips

by Brian Hitney 29. November 2011 18:15

I love Windows Home Server.   I’ve been using Windows Home Server for years, and just purchased a Windows Home Server 2011(WHS) box from Newegg (great deal on a HP Proliant micro server). 

image

Many have asked me why I like WHS so much – it’s NAS, it’s a media server, it’s backup.   It’s a step up from a simple NAS device (although, admittedly, not as plug and play), offers more flexibility and is more cost effective than a Drobo.  A small backup agent can take snapshots of your PC, typically on a daily basis, so they can be restored to a given point in time.  I keep snapshots of my initial installation, for example.  Restoring to those backups is a simple process. 

I’m also a big fan of Sonos, a whole-home music solution that works amazingly well.  What Sonos has done exceedingly well is blend quality hardware, quality software, and reasonable (but not cheap) price points.  I have an extensive music collection, and I point Sonos to a share on my WHS box to index and stream music. 

However, I consider my laptop my “database of record” for my music.  It’s where I download stuff, and I take it with me because I’m often on the road.  The problem I run into is keeping my WHS library in sync with my laptop.   In my case, I want to mirror my library on the WHS exactly as it is on my local collection – and because I’m often reorganizing my collection, adding tags, etc., I need a simple way to do this. 

Enter Robocopy.   Robocopy (Robust file copy) is now built into Windows, and it’s a simple command line tool with a number of options to make this a snap.   For example, if I want to mirror a folder on my laptop with my WHS, this command will do it:

c:\>robocopy "D:\Music" "\\BEAST\Sonos\music" /mir /r:10 /MT:8

D:\Music is my local folder, my server is \\Beast.  The /mir command is for mirror – it’s the same as using /purge and /e:  /purge is to delete files at the target folder that no longer exist in the source, and /e is to copy all subdirectories, including empty ones.   The /r:10 will tell it to retry up to 10 times, in case of some network glitch, and the /MT:8 will have Robocopy use 8 threads to speed things along.   (If you’re familiar with Robocopy, I don’t recommend using /z (restartable) mode as it adds overhead, not needed given the size of files we’re dealing with.)

Now, what if you don’t keep all your music local, and just want to copy it over?   You don’t want to use /mir since it will remove files you otherwise want to keep!   The rest of the command will work fine, but if you move/rename files locally that were previously copied, you’ll have to remember to do that manually on the server.  Once Robocopy does its thing, you’ll get a nice summary:

            Total    Copied   Skipped  Mismatch    FAILED    Extras
Dirs :      1384        29      1355          0         0         0
Files :     15078       381     14697         0         0         0
Bytes : 117.188 g   3.212 g 113.975 g         0         0         0
Times :   0:16:40   0:02:52                       0:00:00   0:00:48

Here, it copied about 30 new folders.  It took about 16 minutes to run, but that’s largely due to new content, having copied some 3.2gb of new files.   Assuming minor changes only, the process typically runs in about 30 seconds.

If you want to get fancy, you could even have Robocopy monitor your folders for changes. 

The next challenge is to have Sonos update its music index once new files are copied over.   Sonos can update its index on a daily basis (or manually via the control software), but I want it done automatically after new files are copied over.  This one is a bit trickier, but thanks to some gurus in the Sonos forums, it’s not impossible.   I’m including the .exe file here for you to use.  Obviously, trusting an exe from someone on the web is not something I’d do, but it’s a .NET assembly which means you can use a tool like JustDecompile to crack it open and look at the source yourself.   Having said that, I’m not responsible if this code causes your computer to blow up, your music collection to vanish, or kills any puppies.

The source code looks like so, and it sends an SOAP packet to a specified Sonos unit to trigger an index rebuild:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Net;

namespace SonosIndexUpdater
{
class Program
{

static void Main(string[] args)
{
string ip;

if (args != null && args.Length > 0)
{
ip = args[0].Trim();
}
else
{
Console.WriteLine("Missing IP Address. Please add IP address for any Sonos unit.");
return;
}

string header1 = @"SOAPACTION: ""urn:schemas-upnp-org:service
:ContentDirectory:1#RefreshShareIndex"
"";
string postData = @"<s:Envelope xmlns:s=""http://schemas.xmlsoap.org/soap/envelope/""
s:encodingStyle="
"http://schemas.xmlsoap.org/soap/encoding/"">
<s:Body>
<u:RefreshShareIndex xmlns:u="
"urn:schemas-upnp-org:service:ContentDirectory:1"">
<AlbumArtistDisplayOption></AlbumArtistDisplayOption></u:RefreshShareIndex>
</s:Body>
</s:Envelope>"
;
string url = string.Format("http://{0}:1400/MediaServer/ContentDirectory/Control", ip);

byte[] byteArray = Encoding.UTF8.GetBytes(postData);

try
{
System.Net.WebRequest req = System.Net.WebRequest.Create(url);
req.Headers.Add(header1);
req.ContentType = "text/xml";
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = byteArray.Length;
req.Timeout = 5000;

Stream dataStream = req.GetRequestStream();

dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();

using (WebResponse response = req.GetResponse())
{
Console.WriteLine("Response from Sonos: {0}",
((HttpWebResponse)response).StatusDescription);

using (dataStream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(dataStream))
{
string responseFromServer = reader.ReadToEnd();
Console.WriteLine("Data: {0}", responseFromServer);
}
}
}
}
catch (Exception ex)
{
Console.WriteLine("Exception occured: {0}", ex.Message);
}

}
}
}
 
To use it, you’d just pass in the IP address of any Sonos unit:
 
c:\>SonosIndexUpdater 192.168.1.100

If you stumbled on this and aren’t a developer but want to try it out, you can build this for free using Visual Studio Express.    Here are some files:
 

EXE file only: Download
VS2010 Solution: Download

Tags: , , ,

Development | Technology | Tech Tips

Use the Windows Azure CDN for Assets

by Brian Hitney 26. September 2011 19:28

The most common response to running stuff in the cloud (Azure, Amazon in particular) is the that it’s too expensive for the little guy.   And generally, hosting VMs when a small shared site of something similar will suffice is a tough argument.

There are aspects to Azure, though, that are very cost effective as they do “micro-scale” very well.  A good example of this is the Azure CDN, or more simply, Azure Blob Storage.   It’s effective to exchange files, it’s effective at fast delivery, and even lightweight security using shared access signatures (links that essentially only work for a period of time).    It’s durable:  not just redundant internally, but externally as well, automatically creating a backup in another datacenter.

For MSDN subscribers, you already have Azure benefits, but even going out of pocket on Blob storage isn’t likely to set you back much:  $0.15/GB of storage per month, $0.01/10,000 transactions, and $0.15/GB outbound bandwidth ($0.20 in Asia; all inbound free).  A transaction is essentially a “hit” on a resource, so each time someone downloads, say, an image file, it’s bandwidth + 1 transaction. 

Because these are micro transactions, for small apps, personal use, etc., it’s quite economical … often adding up to pennies per month.   A few typical examples are using storage to host files for a website, serve content to mobile devices, and to simply offload resources (images/JS files) from website code.

Depending on usage, the Azure Content Delivery Network (CDN) can be a great way to enhance the user experience.  It may not always be the case (and I’ll explain why) but essentially, the CDN has dozens of edge servers around the world.  While your storage account is served from a single datacenter, having the data on the edge servers greatly enhances speed.   Suppose an app on a phone is retrieving documents/text to a worldwide audience … enabling CDN puts the content much closer. 

I created a test storage account in North Europe (one of the Azure datacenters) to test this, using a small graphic from RPA:  http://somedemoaccount.blob.core.windows.net/images/dicelogo.png

Here’s the same element via the CDN (we could be using custom DNS names, but for demo purposes we’re not): http://az32338.vo.msecnd.net/images/dicelogo.png

Here’s a trace to the storage account in the datacenter – from North Carolina, really not that bad all things considered:

image

You can see we’re routed to NY, then on across the pond, and total latency of about 116ms.   And now the CDN:

image

MUCH faster, chosen not only by physical distance but also network congestion.   Of course, I won’t see a 100ms difference between the two, but if you’re serving up large documents/images, multiple images, or streaming content, the difference will be noticeable. 

imageIf you’re new to Azure and have an account, creating a storage account from the dashboard is easy.   You’d just click on your storage accounts, and enter a name/location:

image

You’d typically pick someplace close to you or where most of your users are.   To enable CDN, you’d just click the CDN link on the left nav, and enable it:

image

Once complete, you’ll see if on the main account screen with the HTTP endpoint:

image

So why wouldn’t you do this?

Well, it’s all about cacheability.   If an asset is frequently changing or infrequently used, it’s not a good candidate for caching.   If there is a cache miss at a CDN endpoint, the CDN will retrieve the asset from the base storage account.  This will incur an additional transaction, but more importantly it’s slower than if the user just went straight to the storage account.  So depending on usage, it may or may not be beneficial. 

Tags: , , ,

Azure | USCloud | Tech Tips

Wifi on the Road

by Brian Hitney 30. May 2011 15:02

I spend a lot of time on the road and often need to stay online.  Back in the Windows Mobile days, it was pretty easy to turn a WM6.x device into a personal Wifi hotspot; unfortunately this isn’t so easy on Windows Phone 7.  In fact, most of us now carry Verizon air cards/USB modems for alternative access.   While this has worked great, I often would prefer a hotspot:  I might need to have other devices like a Zune player get online, or might need to share an internet connection. 

There are a couple of free pieces of software out there to turn your PC into a hotspot:  Connectify and Virtual Router are two that come to mind.   It might be specific compatibility issues with my PC or Verizon modem (USB760) but Connectify would blue screen my machine.  

I often don’t want my PC to have to be the gateway, either, so enter a Cradlepoint PHS-300 router.  The idea is that you take your tethered phone/modem and plug it into the USB slot on the Cradlepoint router.  The router then acts as a personal hotspot.  Your PC/devices then connect to it over Wifi.  This has a number of advantages – obviously, it allows you to share the connection.  Secondarily, though, it allows you to place the router someplace where perhaps the signal is better – for example, by a window or across the room.  While it’s not as compact/simple as a Mifi, I like the fact that there’s no added cost to use my existing devices (like any of us need another monthly access fee.)

A nice touch to the router is that it has a battery – so you could use it in a car, or put it in a difficult to access location.   The router is largely plug and play and can use many devices out there.   I’m at the beach for Memorial Day weekend, and I decided to put the PHS-300 through some paces. 

Plugging in my Verizon USB760 was effortless and worked great.  No issues with connectivity on the move, and setting it up was config free… it just worked.   After a series of speed tests, I found that the Verizon modem (3G) was putting around 600k/sec, taking roughly 14 seconds to pull a 1MB file, with a latency of around 200ms. 

Next I decided to plug my WP7 Samsung Focus into the device.  To do this, you need to access the diagnostics of the phone.  Doing this might also violate terms of service, so proceed at your own risk.   If you don’t have the diagnostics app in your app list, dial ##634#.  This adds the diagnostics app – from there, dial *#7284# – this allows you to switch the USB function of the phone; by default, it’s Zune Sync.  To tether, switch to “Modem, Tethered Call” – this will likely require a reboot of the phone.

Next, I simply plugged the phone into the PHS-300.  And it worked.  No configuration (didn’t need to dial *99***1#, connect to the Cingular network, etc. – the PHS-300 took care of all that).   Looking at the modem config page on the router, it recognizes everything correctly:

image

I was curious to test the speed vs. Verizon, so I ran some speed tests.   I have no love of Verizon over AT&T or vice-versa, but I have to say, AT&T spanked Verizon in terms of speed over the weekend at my location.   I wouldn’t draw any specific conclusions, but I’m going to make it a point to test locations as I travel about.   In this case, after many speed tests over the weekend, AT&T 3G averaged about 1.1Mb/s, downloading a 1MB file in about 7 seconds (twice as fast as Verizon) however the latency was higher around 500ms.

As for the router – I’m impressed with the PHS-300.  The inclusion of a battery makes it wonderfully portable, and while not as compact as a Mifi, I have the choice of either my Verizon modem or AT&T tethered phone – both of which (in theory) offer me unlimited data.   Another nice trick, especially if you’re sharing the router with others, is to make use of the QoS/traffic shaping feature to give your machine/devices higher priority.

Tags: , ,

Tech Tips

RPA: Burned by Static Cling

by Brian Hitney 18. April 2011 12:19

In a previous post about locking in Rock, Paper, Azure, I said this somewhat offhand:

In this case, there’s no reason to use such code in a bot. The only time you’d need to is if your bot has a static method/object reference, but that’s a bad idea in a bot because it will lead to unpredictable results. Your bot should have only instance members.

I should’ve called that out more, and in this case, we have a player who lost because of it. It’s especially tough because things seemingly worked fine, until the final tournament round. Here’s why, and here’s some information on static variables (shared in VB) for those who haven’t used them before.

In short, a static modifier on a method or variable makes the member part of the type instead of the class. This is really useful on helper methods in particular, because a static member can be used without instantiating the type to which the member belongs.

Static objects (variables) – in any code – should be a red flag. They have very specific advantages and lower overhead (only 1 is created no matter how many objects of that type are created) – BUT, they can burn you easily if you’re not certain how the object is loaded and used. (Static methods, as a general rule, tend to have less risk than static objects/variables.) Unfortunately, that’s what happened in last Friday’s competition to one of our players.

So how can they burn you? For one, they’re not thread safe. Of course, they _could_ be thread safe, but you’d have to be cognizant of what they are doing to make them thread safe. (Non-statics might not be thread safe either, however, using statics implies global reuse so it heightens the exposure of thread safety issues and can be hard to track down.) One example: we modified the original engine to do multithreading and the original log had a static method that used a StringBuilder to build a log file. This caused problems because the StringBuilder is not thread safe – so we had to add locking. The problem was always there (even if it wasn’t static), but the problem never manifested because the core was single threaded. Another way they can burn you is that two or more objects may be accessing the objects in an nondeterministic way, leading to unpredictable results.

The unfortunate part about this in particular was that the issue didn’t manifest until the main tournament round – so everything appeared fine until the final round. The game engine runs two types of rounds: player rounds, and full rounds. In either case, the engine will spin up many threads to execute the games – but during a player round, the engine loops the new bot against everyone else. As other players submit their bots, the engine will load your bot only once. Because your bot is loaded only once, there’s really no chance of static variables causing a problem, much like the StringBuilder example above.

But during the tournament round, all of a sudden many more games are played. Consider that with 50 players, only 49 games are played when a new bot is uploaded. But in a full round, 1,225 games are played! There is a much stronger chance your bot will have multiple instances loaded concurrently, and modifying static variables will cause the bots to go haywire.

So, the lesson of the week is: don’t use statics in a bot! Question or comment about a bot? Let us know…

Tags: , , ,

Azure | RPA | Tech Tips | USCloud

Getting a Windows Azure account for Rock, Paper, Azure

by Brian Hitney 6. April 2011 18:45

If you’re interested in getting a Windows Azure account to play in Rock, Paper, Azure (RPA), there are a few options available to you, from least painful to most painful (in my opinion, anyway):

Method 1 – Windows Azure Pass

The way most people are getting an account is through the Windows Azure Pass program (using code PLAYRPA).  More details can be found on the Get Started page under step 1.    But, this certainly isn’t the only way to get an account, and – for a few of you – might not be possible.  The Azure Pass is limited to one Live ID, so if you got an account through the Azure Pass program say 6 months ago, you can’t get another one.  (I won’t say anything if you just sign up for another Live ID.)

Method 2 – Windows Azure Trial Account

Sign up for the Windows Azure Free Trial.   This gives you 750 hours of an extra small compute instance, and 25 hours of a small compute instance.  You do need a credit card to cover overages.  Note: the Bot Lab project by default is set up as a small compute instance.  If you go this route, I highly recommend you change the Bot Lab to be an Extra Small instance.  You can do this by double-clicking the role and changing the VM size:

image

Method 3 – MSDN Subscriptions

Have an MSDN Premium or Ultimate subscription?   You already have account hours you can use.  Log into your MSDN account for more information.   This step does require a credit card (or other billing arrangement) to handle overages, but you are not billed as long as you stay within plan.  As of the time of this writing, please note that Extra Small compute instances are beta and not included in the MSDN hours – so be sure to stick with a small instance.  As usual, we recommend taking down deployments once you’re done to avoid wasting compute time.

Method 4: Pay as You Go Specials

Check out the current offers.   There are few different options based on your needs (and some are available specifically for partners).  The introductory special is the best way to get started, but if you’re using Windows Azure consistently, the Windows Azure Core offers a great value.   If you’re just interested in playing the game and willing to pay or aren’t able to receive other offers for some reason, deploying the Bot Lab as an Extra Small instance costs $0.05 per hour.   If you were to play during the week, and leave the Bot Lab deployed 24 hours, you’d be looking at roughly $5.  (If you only code in the evenings for a few hours, pulling down the deployment overnight and not during use will bring that down substantially.)

See you on the battlefield!

Tags: , ,

Azure | RPA | Tech Tips | USCloud

Acer 1420P (PDC Laptop) Service Manual

by Brian Hitney 31. January 2011 18:49

There aren’t too many Acer 1420P netbooks out there – this is the one given out at PDC 2009.  I was recently having some problems with it and needed to disassemble (to the bone).  If you’ve ever disassembled a laptop, you know you pretty much need a service manual to guide you.  I love my laptops, but working on them makes me really appreciate the modular ATX design in many desktops.

The problem was finding the service manual – I’m not exactly sure where I ended up getting this (I was about 20 tabs deep searching for various part numbers), but here it is in case you need it:

1420P Service Manual Download

Tags: , , ,

Tech Tips

Windows Azure Guest OS

by Brian Hitney 18. May 2010 01:05

In a Windows Azure project, you can specify the Guest OS version for your VM.  This is done by setting the osVersion property inside the ServiceConfiguration file:

image

If you don’t specify a version explicitly, the latest Guest OS is chosen for you.  For production applications, it’s probably best to always provide an explicit value, and I have a real world lesson that demonstrates this!

MSDN currently has a Guest OS Version and SDK Compatibility Matrix page that is extremely helpful if you’re trying to figure out which versions offer what features.  I recently ran into a problem when examining some of my performance counters – they were all zero (and shouldn’t have been)!  Curious to find out why, I did some digging (which means someone internal told me what I was doing wrong).

In short, I had specified a performance counter to monitor like so:  "\ASP.NET Applications(__Total__)\Requests/Sec".  This had worked fine, but when I next redeployed some weeks later, the new Guest OS (with .NET Fx 4.0) took this to mean 4.0 Requests/Sec, because I didn’t specify a version.   So, I was getting zero requests/sec because my app was running on the earlier runtime.  This was fixed by changing the performance counter to "\ASP.NET Apps v2.0.50727(__Total__)\Requests/Sec".  

For more information on this, check out this article on MSDN.  And thanks to the guys in the forum for getting me up and running so quickly!

Tags: , ,

Azure | Development | Tech Tips

Ip2Location (and IPinfoDB) Performance Tips

by Brian Hitney 9. March 2010 19:00

I’ve done a number of talks lately on Worldmaps and typically in side conversations/emails, people are curious about the databases and converting IP addresses to geographic locations.   And, often when you dive into using the data, it seems there are a number of performance considerations and I thought I’d share my input on these topics.

First up, the data.  Worldmaps uses two databases for IP resolution.  The primary/production database is Ip2Location.  I’ve found this database to be very accurate.  For development/demo purposes, I use IPinfoDB.  I haven’t had too much time to play with this database yet, but so far seems accurate also.   The latter is free, whereas Ip2Location is not.

In either case, the schema is nearly identical:

image

The BeginIp and EndIp columns are a clustered primary key.  In the case of IPinfoDB, there is no EndIp field (and it’s not really needed).  When performing a resolution, a string IP address is converted into a 64 bit integer and then used in searching the table.  That’s why having a clustered key on the BeginIp (and optionally EndIp) is crucial to performance.

But it doesn’t stop there.   The examples posted in the database’s respective home pages are accurate and simple, but need to be refactored for performance.  For example, to do a simple resolution on Ip2Location, according to their docs:

SELECT * FROM dbo.Ip2Location WHERE @IpNum BETWEEN BeginIp and EndIp

And for IPInfoDB:

SELECT TOP 1 * FROM IPInfoDB where BeginIp <= @IpNum ORDER BY BeginIp DESC

Both of these methods are perfectly fine, particularly for use as generic samples.  The second one is on the right track, but it doesn’t work for joins so if you’re querying over a range, you’d need to refactor.  And in the first example, using a BETWEEN operator forces a clustered index scan when joining, killing the performance.

If we run the first example across my minified Ip2LocationSmall table, we’ll see something like this (and this is running off of SQL Azure – the perf is pretty great compared to localhost!):

image

We can also look at the time:

image

Ouch!  Now, it doesn’t seem too bad, but imagine doing thousands of these requests per minute, or doing large joins. 

The goal then is to provide some hints that will optimize the query, particularly for joins.  Our indexes are correct, so we can rework the query to get rid of the BETWEEN operator – we can sacrifice a little readability and do something like:

SELECT *   
FROM (
    select
        ( select MAX(beginip)
          from dbo.Ip2LocationSmall
          where BeginIp <= @IpNum
        ) as IP_Begin           
    ) as foo
INNER JOIN dbo.Ip2LocationSmall iploc
ON iploc.BeginIp = foo.IP_Begin

The result:

image

And the time shows some improvement:

image

But the REAL benefit comes when we need to join.   Suppose I’d like to get a list of the countries for a given map (which is a parameter called MapId):

SELECT DISTINCT(ip.CountryCode)
FROM MapHits hits
INNER JOIN dbo.Ip2LocationSmall ip
    ON hits.IpNum BETWEEN ip.BeginIp AND ip.EndIp
WHERE MapId = @MapId

The query returns 95 rows, and executes in 16 seconds:

image

image

In this case, we can refactor this using the method above to something like:

SELECT
    DISTINCT(CountryCode)   
FROM (
     select IpNum,
        ( select MAX(beginip)
          from Ip2LocationSmall
          where BeginIp <= IpNum
        ) as IP_Begin
      from dbo.MapHits as hits
      where MapId = @MapId
  ) as foo
INNER JOIN Ip2LocationSmall iploc
ON iploc.BeginIp = foo.IP_Begin

Again, not as pretty looking, but boy what a difference:

image

image

We went from 16,500 milliseconds to 260 – over 60x the performance!  Mike @AngryPets would be proud.   The reason for the perf gain is we were able to eliminate the nested loop, which is (in this case) scanning the entire clustered index for the matching rows.

The second benefit is the ability to switch schemas easily between IP2Location and IPinfoDB, and we can additionally lose the EndIp column which trims the size of the table.

Tags: , , ,

SQL Azure | SQL | Tech Tips | Development

Azure Miniseries #4: Monitoring Applications

by Brian Hitney 26. February 2010 08:58

In this screencast, we'll take a look at monitoring Azure applications by capturing event logs and performance counters. We'll also look at using PowerShell to deploy and configure applications using the management API. Finally, we'll take a sneak peek at Azure Diagnostics Manager, a tool from Cerebrata that allows you explore event logs and look at performance counters visually.

Get Microsoft Silverlight

Here are some links from the screencast:

Finally, let’s get into some code snippets! Watch for wrap on the PowerShell lines, and note the single quote ` character as the line continuation:

Creating a self-signed certificate command:

makecert -r -pe -a sha1 -n "CN=Azure Service Test" -ss My -len 2048 -sp "Microsoft Enhanced RSA and AES Cryptographic Provider" -sy 24 AzureServiceTest.cer

Uploading a new deployment to the staging slot, and starting it (requires Azure CmdLets):

$cert = Get-Item cert:\CurrentUser\My\{thumbprint}
$sub = "{subscription GUID}"
$servicename = "{service name}"

$package = "CloudApp.cspkg"
$config = "ServiceConfiguration.cscfg"

[DateTime]$datelabel = Get-Date
$lbl = $datelabel.ToString("MM-dd-yyyy-HH:mm")

Write-Host "Label for deployment: " $lbl

Add-PSSnapin AzureManagementToolsSnapIn

Get-HostedService $servicename -Certificate $cert -SubscriptionId $sub |
New-Deployment -Slot Staging $package $config -Label $lbl |
Get-OperationStatus -WaitToComplete

Get-Deployment staging -serviceName $servicename -SubscriptionId $sub -Certificate $cert |
Set-DeploymentStatus running |
Get-OperationStatus -WaitToComplete

Increasing the number of instances:

Add-PSSnapin AzureManagementToolsSnapIn

$cert = Get-Item cert:\CurrentUser\My\{thumbprint}
$sub = "{subscription GUID}"
$servicename = "{service name}"
$storage = "{storage name}"

#get storage account name and key
$key = (Get-StorageKeys -ServiceName $storage -Certificate $cert -SubscriptionId $sub).Primary
$deployId = (Get-HostedService $servicename -SubscriptionId $sub -Certificate $cert | Get-Deployment Production).DeploymentId       

Get-HostedService $servicename -Certificate $cert -SubscriptionId $sub |
    Get-Deployment -Slot Staging |
    Set-DeploymentConfiguration {$_.RolesConfiguration["WebRole1"].InstanceCount += 1}

Updating the performance counters – specifically, adding total processor time, ASP.NET req/sec, and memory usage to be polled every 30 seconds, and uploaded every 1 minute:

Add-PSSnapin AzureManagementToolsSnapIn

$cert = Get-Item cert:\CurrentUser\My\{thumbprint}
$sub = "{subscription GUID}"
$servicename = "{service name}"
$storage = "{storage name}"

#get storage account name and key
$key = (Get-StorageKeys -ServiceName $storage -Certificate $cert -SubscriptionId $sub).Primary
$deployId = (Get-HostedService $servicename -SubscriptionId $sub -Certificate $cert | Get-Deployment Production).DeploymentId       

# rate at which counters are polled
$rate = [TimeSpan]::FromSeconds(30)

Get-DiagnosticAwareRoles -StorageAccountName $storage -StorageAccountKey $key -DeploymentId $deployId |
foreach {
    $role = $_
    write-host $role
    Get-DiagnosticAwareRoleInstances $role -DeploymentId $deployId `
        -StorageAccountName $storage -StorageAccountKey $key |

    foreach {
        $instance = $_

        $config = Get-DiagnosticConfiguration -RoleName $role -InstanceId $_ -StorageAccountName $storage `
             -StorageAccountKey $key -BufferName PerformanceCounters -DeploymentId $deployId            
        $processorCounter = New-Object Microsoft.WindowsAzure.Diagnostics.PerformanceCounterConfiguration `
                -Property @{CounterSpecifier='\Processor(_Total)\% Processor Time'; SampleRate=$rate }
        $memoryCounter = New-Object Microsoft.WindowsAzure.Diagnostics.PerformanceCounterConfiguration `
                -Property @{CounterSpecifier='\Memory\Available Mbytes'; SampleRate=$rate }
        $requestsCounter = New-Object Microsoft.WindowsAzure.Diagnostics.PerformanceCounterConfiguration `
                -Property @{CounterSpecifier='\ASP.NET Applications(__Total__)\Requests/Sec'; SampleRate=$rate }
        $config.DataSources.Clear()
        $config.DataSources.Add($processorCounter)
        $config.DataSources.Add($memoryCounter)
        $config.DataSources.Add($requestsCounter)
        Set-PerformanceCounter -PerformanceCounters $config.DataSources -RoleName $role `
             -InstanceId $instance -DeploymentId $deployId `
             -TransferPeriod 1 `
             -StorageAccountName $storage -StorageAccountKey $key                     
    }  
}

And finally, the webrole.cs class from the screencast:

public class WebRole : RoleEntryPoint
    {
        public override bool OnStart()
        {
            DiagnosticMonitorConfiguration diagConfig =
                DiagnosticMonitor.GetDefaultInitialConfiguration();

            diagConfig.PerformanceCounters.DataSources.Add(
                new PerformanceCounterConfiguration()
                {
                    CounterSpecifier = @"\Processor(_Total)\% Processor Time",
                    SampleRate = TimeSpan.FromSeconds(5)
                });

            diagConfig.PerformanceCounters.DataSources.Add(
                new PerformanceCounterConfiguration()
                {
                    CounterSpecifier = @"\Memory\Available Mbytes",
                    SampleRate = TimeSpan.FromSeconds(5)
                });

            diagConfig.PerformanceCounters.ScheduledTransferPeriod =
                TimeSpan.FromMinutes(1);

            diagConfig.Logs.ScheduledTransferLogLevelFilter = LogLevel.Information;
            diagConfig.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);

            DiagnosticMonitor.Start("DiagnosticsConnectionString", diagConfig);

            System.Diagnostics.Trace.TraceInformation("Done configuring diagnostics.");

            // For information on handling configuration changes
            // see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
            RoleEnvironment.Changing += RoleEnvironmentChanging;

            return base.OnStart();
        }

        public override void OnStop()
        {
            System.Diagnostics.Trace.TraceWarning("Onstop called.");
            base.OnStop();
        }

        private void RoleEnvironmentChanging(object sender, RoleEnvironmentChangingEventArgs e)
        {
            // If a configuration setting is changing
            if (e.Changes.Any(change => change is RoleEnvironmentConfigurationSettingChange))
            {
                // Set e.Cancel to true to restart this role instance
                e.Cancel = true;
            }
        }
    }

Tags: , , ,

Azure | Microsoft | Tech Tips

your host...

Brian Hitney
Developer Evangelist
Microsoft Corp.

About Me

My Worldmap