Saturday, January 9, 2010

Powershell is dynamically scoped, and that will confuse you.

Lets start with an example, as the concept of dynamic scoping is a big string for most programmers.

Python Program
x = 5
def printX():
print x
def setAndprintX():
x=7
printX()
printX()
setAndPrintX()
printX()





Output From Python




5
5
5

Powershell Program




$x = 5
function printX() { echo $x }
function setAndprintX()
{
$x=7
printX
}
printX
setAndprintX
printX





Output From Powershell




5
7
5








What is this dynamic scoping?


Most programs use static, also called lexical, scoping because it's easy to understand. You figure out what is in scope by looking at the source code. In the python example, the only value of x in scope is the global value of x.







By contrast, powershell uses dynamic scoping, in this model, you lookup up variables at runtime based on a scope stack. Each time you call a function you create a new scope, and copy all values from the parent scope into it. In the powershell example, when printX is called from setAndprintX we get the value of $x that was set in setAndprintX scope.






Why would you want dynamic scoping?



I can't come up with a good explanation of why you'd pick dynamic scoping over lexical scoping. My hunch was this is historical as it's how batch files and shell scripts work. Interestingly, Perl supports both dynamic scoping and lexical scoping. You can read a good article about it here. My synopsis of why Perl has dynamically scoped variables from the article:









  • In the beginning perl only supported global variables and that was painful.


  • Since it was cheap to implement Perl authors added the ability to created dynamically scoped variables (via:local keyword).


  • However, when perl authors got time, they added lexical scoped variables (via: my keyword).


  • Now people are told *not* to use dynamically scoped variables since they're weird.






Do you get other language features to make support for this easier:


Yes, you can write to a different scope explicitly:




$x = 3 # Write to local scope
$global:x = 3 # Write to global scope.





You can also execute your function without creating a new scope via '.' aka sourcing:




. func() # runs the function in local scope, and variables created are visible.





This is a fascinating , but why are we having this conversation?


Because I wanted to write:




function getPrintDogFunction()
{
function Nested1(){echo"dog"}
function Nested2(){Nested1}
Get-Command Nested2
}

$printDog = getPrintDogFunction
# Call PrintDog
& $printDog

# This Call fails saying can't find Nested1 - which makes sense it's not in scope.

So, I changed my code to the following:




# source getPrintDogFunction, which causes Nested1 and Nested2 to be created in my scope, and thus
# Nested1 is in scope when I call printDog

$printDog = . getPrintDogFunction
# Call $printDog
& $printDog





A few days pass, and I add a new feature, printCat. Via the power of cut and paste our code becomes:




function getPrintDogFunction()
{
function Nested1(){echo "dog"}
function Nested2(){Nested1}
Get-Command Nested2
}

function getPrintCatFunction()
{
function Nested1(){echo "cat"}
function Nested2(){Nested1}
Get-Command Nested2
}

$printDog = . getPrintDogFunction
$printCat = . getPrintCatFunction

# Print all the animals
& $printDog
& $printCat

# Grrr - this is printing cat cat.
# Worse yet depending on the order of these calls, the behavior changes.





To fix I use the following pattern:




function getPrintCatFunction()
{
function Nested2()
{
function Nested1(){echo "cat"}
Nested1
}
Get-Command Nested2
}

$printDog = getPrintDogFunction




And now the world makes sense.



Notes:


(1) If you've never heard of dynamic scoping, any language you pick will act the same.

Sunday, January 3, 2010

Syntax Highlighting Take 2

Readers of my blog complained that they can't see the code I was syntax highlighted in RSS readers like Google reader. The reason is moderately interesting so I'll explain it:



HTML likes to gobble up white space, so if you're pasting in source code you use the PRE (preformatted text) tag. PRE shows up in fixed width font and preserves spacing; however you can't place < or > in PRE tags since they denote HTML tags, instead you need to use &lt or &gt (character entity references if you speak techno babble). This is annoying, especially given the blogger text editor gets confused when you edit PRE tags with nested < and > signs.



Speaking of < and >, source code, and HTML files, you often want to write javascript in an HTML file, and you then hit the similar problem that you want to write x < 99, but can't since '<' isn't valid HTML. To solve this problem you can put a CDATA(*) section in your script blocks, which allows you to use write x < 99.



So, to avoid specifying > and <, the author of SyntaxHighlight came up with a clever solution: Put the source code you'd like to display in a non-javascript script block using CDATA, and then use javascript on the page to convert these blocks to tables for the web browser to display.



This works great in your web browser, but RSS readers don't run the javascript that converts the dummy script blocks into tables and thus you can't see the code.  



Luckily, SyntaxHighlighter also supports an usage where you use specify your code in PRE blocks.  It's more annoying for me (for the above reasons), but it's better for my beloved readers since the PRE elements 'degrade' to non syntax highlighted pre blocks (instead of nothing) when in an RSS reader.





For all my RSS reader friends here is the snippet from the last blog post:



Console.Writeline("Hello World!")
Python:

print "Hello World"




NOTES:



(*) CDATA,aka: character data, is not interpreted as part of the document.  In CDATA,  >  means greater than and does not have to be written as character entity references. Unfortunately CDATA can not be placed in PRE blocks.

Thursday, December 31, 2009

Syntax highlighting for code blocks.

My blog used to have ugly display of source code, and I've always been a bit jealous of sites with pretty syntax highlighting for their source code. Well jealous no longer, I've now started using SyntaxHighlighter in my blog. You can find howto directions here. Here' a small sample:



C Sharp:



Python:





If there is demand I'll go through some of the old posts and re-highlight them.

Tuesday, December 29, 2009

Using wireshark to trace localhost traffic on windows.

(If you don’t care why this works and just need a recipe, switch to this post)

Capturing network packets on localhost doesn't work on windows. The reason is windows doesn't send loopback traffic far enough down the networking stack for wireshark to see it. To make sniffing work on localhost you can route your ip traffic to your default gateway. I'll walk you through this, and along the way you'll see:

  • netcat - telnet on steroids (nc.exe)
  • tshark - command line network sniffer from the wireshark package.
  • powershell jobs - background jobs from the shell!

Step 1 - launch the server as a background job (Woohoo powershell)

PS C:\Users\igord> $server = start-job { \bin_drop\nc -L -p 8082 } 

Step 2 - Make client connection:

PS C:\Users\igord> \bin_drop\nc.exe 127.0.0.1 8082
Hello
You can see me

Step 3: See if we can see anything in tshark on port 8082.

C:\Program Files (x86)\Wireshark>tshark -i 4 -R "tcp.port == 8082"
Capturing on Microsoft

Step 4: Point netcat at our ip address that's external:

PS C:\Users\igord> ipconfig
Windows IP Configuration
Wireless LAN adapter Wireless Network Connection:
Connection-specific DNS Suffix . : hsd1.state.comcast.net
Link-local IPv6 Address . . . . . : fe80::49a:2ea6:7757:db5%14
IPv4 Address. . . . . . . . . . . : 192.168.1.100
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.1.1PS

C:\Users\igord> \bin_drop\nc.exe 192.168.1.100 8082
Hello can you see me in tshark?


(Still nothing in netcat)



Step 5: Add a route for our local address to the router:



PS C:\Users\igord> route add 192.168.1.100 192.168.1.1
OK!

Step 6: Run netcat again - and check tshark:

PS C:\Users\igord> \bin_drop\nc.exe 192.168.1.100 8082
Hello Do you see me

Now we get our packets in tshark!

C:\Program Files (x86)\Wireshark>tshark -i 4 -R "tcp.port == 8082"
Capturing on Microsoft
107.838518 192.168.1.100 -> 192.168.1.100 TCP 60080 > us-cli [SYN] Seq=0 Win=8192 Len=0 MSS=1460
107.840456 192.168.1.100 -> 192.168.1.100 TCP 60080 > us-cli [SYN] Seq=0 Win=8192 Len=0 MSS=1460
107.841013 192.168.1.100 -> 192.168.1.100 TCP us-cli > 60080 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460
107.841988 192.168.1.100 -> 192.168.1.100 TCP us-cli > 60080 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460
107.842291 192.168.1.100 -> 192.168.1.100 TCP 60080 > us-cli [ACK] Seq=1 Ack=1 Win=64240 Len=0
107.844181 192.168.1.100 -> 192.168.1.100 TCP [TCP Dup ACK 449#1] 60080 > us-cli [ACK] Seq=1 Ack=1 Win=64240 Len=0
110.528557 192.168.1.100 -> 192.168.1.100 TCP 60080 > us-cli [PSH, ACK] Seq=1 Ack=1 Win=64240 Len=20
110.530030 192.168.1.1 -> 192.168.1.100 ICMP Redirect (Redirect for host)
110.530139 192.168.1.100 -> 192.168.1.100 TCP [TCP Out-Of-Order] 60080 > us-cli [PSH, ACK] Seq=1 Ack=1 Win=64240 Len=20
110.730222 192.168.1.100 -> 192.168.1.100 TCP us-cli > 60080 [ACK] Seq=1 Ack=21 Win=64240 Len=0
110.731258 192.168.1.1 -> 192.168.1.100 ICMP Redirect (Redirect for host)
110.731797 192.168.1.100 -> 192.168.1.100 TCP [TCP Dup ACK 480#1] us-cli > 60080 [ACK] Seq=1 Ack=21 Win=64240 Len=0
116.982412 192.168.1.100 -> 192.168.1.100 TCP 60080 > us-cli [RST, ACK] Seq=21 Ack=1 Win=0 Len=0
116.984259 192.168.1.1 -> 192.168.1.100 ICMP Redirect (Redirect for host)
116.984390 192.168.1.100 -> 192.168.1.100 TCP 60080 > us-cli [RST, ACK] Seq=21 Ack=1 Win=0 Len=0


For bonus points, why do you see 2 of each packet?



Step 7: Cleanup



PS C:\Users\igord> route delete 192.168.1.100 OK!

Sunday, December 27, 2009

That's fast

Geeks born before the 80's had wet dreams about getting a T1 line. A T1 was 1.5 Mbps ~ 0.19 MB/s. Here's a picture of a large download on my home wireless network which connects to the world via cable:


Gotta love technology. How long before 1 MB/s is slow?

Friday, December 25, 2009

Why twitter can be awesome

If you were born before the 80's (heck maybe before the 90's) you have some idea what twitter is but can't understand what it is for.   Well, I've been thinking about twitter and I have a few ideas.



A tweet (a message on twitter) is a short (140 char max) thought you broadcast to the world.  Now the 'magic' of twitter is this: people can decide to listen to your message, only if they want to.  If you think a tweet is stupid, it is because it isn't written for you, instead of complaining about the tweet, you ignore it '.' This should sounds familiar, it is how broadcast works in ethernet networks, and even how conversations work on public transit.



Since tweets are public, and arguably represent what people are thinking/doing, you can do something you could never do before: see what the world is thinking.  From that starting point, you can join a conversation, or simply learn something interesting.



That said, as I'm looking at twitter, and I'm not finding any conversations I'm interested in joining, or even overhearing. Oh well I'll keep my eye on it.

Thursday, December 17, 2009

Pretty Printing XML on Windows.

It happens to the best of us, we get some ugly XML string with no formatting, and we need to view it, ideally in a formatted fashion. XMLLint is the answer. Ugly XML is the input, nicely formatted XML is the output. Finding a version of XMLLint for windows was a challenge till I found this project: http://code.google.com/p/xmllint. Unfortunately this xmllint requires an xml filename as input, and I wanted a version that takes xml on stdin, and produces pretty xml on stdout(*). Luckily this project had source code available and the pretty printing is trivial, here's the full code listing for 'my version' of XMLLint:



Example Usage(+):









NOTES:

(*) StdIn -> Filter -> StdOut is a common trick for VI folks. the syntax is: :%!

(+) Any guesses why I needed a screen shot for example usage?