Cool IEnumerable Tricks Part 1: GroupBy

IEnumerable, or what most people refer to as LINQ, has changed the way I program. If you love LINQ and you already fully understand this blog post, leave a comment with a topic you'd like me to post about. If you're not so comfortable with LINQ, try to follow along. This post will only take a few minutes, but may change the way you program for life.

While we go through this blog post (and future posts in this series) make sure you understand how every function works, if you don't understand a function, you certainly won't understand the next function. Some of this stuff is hard to grasp and I find the best way to understand the code is to try it in a debugger. It took me a few tries to wrap my head around what is going on, so expect to spend some time experimenting with this code.

I learn best when I'm trying to solve a problem, so lets define a simple problem: How to determine if the random number generator equally distributes integers mod 10.

Step 1: Generate a stream of random numbers:

var seed = new Random();
var countSamples = 10*1000
var randomStream = Enumerable.Range(0,countSamples).Select(r=>seed.Next());

In the above sample, we generate countSamples random numbers. We do this by producing countSamples integers via Enumerables.Range, than for each of those integers we apply a transformation. In this case the transformation ignores the input value, and returns a random number.

Step 2: Compute how many of each random number have each modulus (0..9).

randomStream.GroupBy(r=>r%10).Select(x=>new {x.Key,count=x.Count()}).OrderBy(r=>r.Key);

In the above sample, we use GroupBy to split the random numbers into groups based on their %10 value. For each group, we return a new object containing the modulus (x.Key) and the count of each group. Finally we sort based on the modulus.  The output is below:

Key	count
0	959
1	938
2	1037
3	1028
4	951
5	1009
6	1029
7	1011
8	1038
9	1000

In a future post I'll describe how to create our own IEnumerable functions.

Read More

Leading by praise: How to use a carrot when you really want a stick.

Many naive leaders believe they have two ways to enact change in their followers.  The first way is "the carrot" aka praising good behavior. The second way is "the stick"  aka criticizing bad behavior.  In my experience, the stick is extremely ineffective. I'd say the only time the stick has any positive impact is when the follower doesn't realize she's doing something wrong, or how serious that wrong thing is.  

The goal of this post is how to use a carrot when you want a stick.  First lets recall how to use a carrot effectively in general: Keep it timely, short, specific and often (the one minute manager is my hero).  Any time you see a new good behavior praise it.  When you praise,  be as specific as possible so the follower knows both the behavior you are praising and the valuable impact of that behavior.  It's easy to praise when things go well, but how do we praise when things are going poorly?

When things are going poorly, there's only one thing you can do - put your followers in a position where they perform good behaviors.  Scope their work so they have no choice but to perform good behaviors.  From that good behavior, apply praise.  Even if that behavior is accidental, still give praise.   Slowly, sometimes painfully slowly, you'll get more opportunities to give praise, and you'll get more good behaviors.  Keep giving praise, eventually the fruit of your labor will pay off.

Praising weak performers can be a time consuming delicate process, but it's like lighting a fire: at first you have to work really hard to spark the tinder, then pay close attention while the kindling catches to make sure you don't lose your delicate flame, finally you'll have a fire raging, and you can lean back and relax knowing not only have you lit a fire, but that fire can be used to light other fires.

Read More

Google killed the video store

As a younger man I remember arguing with a co-worker about the death of local storage. 



Me: Who wants to store data themselves?  I can store all data in the cloud, where it will be preserved forever!

Wiser co-worker: What incentive does this cloud have to keep your data? I'll keep my data on my local hard drive thank you very much!

 

8 years later my co-worker was  correct. I got the following letter from Google today:



Later this month, hosted video content on Google Video will no longer be available for playback. <snip> We’ve added a Download button to the video status page, so you can download any video content you want to save. If you don’t want to download your content, you don’t need to do anything. (The Download feature will be disabled after May 13, 2011)

Ouch for being wrong,  and double ouch for having to download all my videos to my hard drive.

Notes:

• Astute readers will note, I actually got a decent deal since I got 8 years of free storage and $/GB has plummeted in that time.


• This isn’t the first time my “bet on a cloud service” has failed. I lost all my face tags when polar rose was bought by apple and shutdown.


• I’m a slow learner, and still use the cloud for the primary copy of all my media.

Read More

Software Engineering “The Soft Skills”

To date, I've focused my blog on the "computer" parts of software engineering. However, as I've grown as a software engineer, one of my largest aha moments is the realization that software engineering is all about people. As such, I'll start posting articles dealing not just with computers, but also with communication, people leadership and management. These "people skills" are as important to a software engineer as the ability to crank out a design specification. For the hardcore among you that think people skills have nothing to do with software, consider this.

Read More

Impedance MisMatch: IEnumerable<T> to DataTable

I had to work with a class that consumed a System.DataTable today. DataTable is an old class that pre-dates generics in the CLR. DataTable carries typed data as an array of System.Object - double plus yuck! As you can imagine the code I use these days has compiler verified type safety by using IEnumerable everywhere. So how did I convert from my beloved IEnumerable to the yucky DataTable? With reflection of course! This simple method solves our problems:

public static DataTable ToDataTable(IEnumerable rows)
{
    var tableToReturn = new DataTable();

    // get properties of T
    var properties = typeof (T).GetProperties().Where(p => p.MemberType == MemberTypes.Property);

    // Convert T's properties to columns in the DataTable
    foreach (var p in properties)
    {
        tableToReturn.Columns.Add(p.Name, p.PropertyType);
    }

    //populate rows
    foreach (var row in rows)
    {
        T row1 = row;
        var propertyValuesAsEnumerable = properties.Select(p => p.GetValue(row1, null));
        tableToReturn.Rows.Add(propertyValuesAsEnumerable.ToArray());
    }
    return tableToReturn;
}

Read More

Powershell script to enable windows to capture localhost traffic in wireshark

If you want to understand why the following scripts work read this post. Otherwise just paste the following into an elevated powershell window:

Setup windows networking to allow localhost capturing in wireshark:

# Find the network configuration that has the default gateway.
$defaultAdapter = Get-WMIObject Win32_NetworkAdapterConfiguration | ? {$_.DefaultIPGateway}
if (@($defaultAdapter).Length -ne 1) {throw "You don't have 1 default gateway, your network configuration is not supported" } 
# Route local IP address via the default gateway
route add $defaultAdapter.IPAddress[0] $defaultAdapter.DefaultIPGateway
Write-Host "Start capturing on localhost by connecting to $($defaultAdapter.IPAddress[0])" 

Return windows networking to normal configuration:

# Find the network configuration that has the default gateway.
$defaultAdapter = Get-WMIObject Win32_NetworkAdapterConfiguration | ? {$_.DefaultIPGateway}
if (@($defaultAdapter).Length -ne 1) {throw "You don't have 1 default gateway, your network configuration is not supported" } 

# Stop routing localhost traffic to the router.
route delete $defaultAdapter.IPAddress[0] 

Remember, you won’t see traffic to localhost (127.0.0.1) but traffic to your network adapter’s IP address as listed in the script.

Read More

How did we get a 53 byte packet size in ATM?

I'll be honest, I don't know squat about ATM, but I was having lunch with this fellow, and he told me the story of the 53 byte ATM packet. 

You can find more details on Wikipedia, but here’s the synopsis:

(Disclaimer: I’m not an expert in ATM; nor am I trying to teach you technical details about ATM networks; so I’ll hand wave and trade off accuracy for simplicity. For example, ATM does have variable sized packets which it divides into cells, and it is the cells which are 53 bytes long. However, since the closest thing to a cell in common networks is an Ethernet packet, I’ll simply refer to cells as packets.)

ATM is designed to be shared between data network applications, and voice network applications(+).

In data networks we want large packets because this gives maximum efficiency.  This is because each packet has a fixed size header and thus the more data you can transmit per packet , the higher your ‘real’ throughput.

For voice networks we want to reduce latency.  Latency is the fancy word for delay.

If voice packets are delayed, you’ll hear them at the receiver as an echo; this is un-acceptable and thus to mitigate delays you’d need to install an echo canceller (++).
In the USA, the country was big enough (+++) that no matter how small the packet size was chosen to be, you’d always need echo cancellers – therefore America proposed an optimal packet size of 64 bytes.

In France, which was much smaller then America, if the packet size was chosen to be 32 bytes the packet could get from one end of the country to the other without enough delay to require echo cancellers. Thus France pushed for a 32 byte packet size.

To achieve consensus, a compromise was required and a 48 byte payload size was chosen.  This did not meet France’s need, nor was it optimal for the US; however, this is what happens when you require consensus.
But wait you ask, where do the last 5 bytes come from? Ah, the committee was very concerned about the overhead of the packet header, so the committee decided the packet header could not exceed 10% of the packet size, thus the committee chose a 5 byte header.  48 + 5 == the 53 byte ATM packet size; which is the optimal solution for no one.

Notes:
(+) At the time ATM was designed, people made long distance calls over voice networks.
(++) If you’re old enough to remember the early days of skype, you needed to use headphones otherwise the echo was unbearable. Today computers are powerful enough to do echo cancellation without blinking.
(+++) Compute big enough by calculating time required for light to traverse the USA.

Read More