All Your Base Are Belong To Us : RandomThoughts

Using Something You Can't Implement Yourself

Sasha Goldshtein — Wed, 30 Jan 2013 10:01:00 GMT

I have found that the biggest obstacle I face when adopting a new language, technology, or framework is using something I don't fully understand how to implement myself. I read hundreds of blog posts every week talking about language extensions to JavaScript, cool new iOS application frameworks, and brand-new SaaS offerings on top of Windows Azure -- just as a small sample. Obviously, just using some piece of technology or adapting a sample to my needs is usually not that hard. The thing is, I can't bring myself to adding code to my arsenal if I don't understand how it works and how it has been implemented. It's somewhat akin to the Not Invented Here syndrome, except I don't actually write all my own frameworks; I just want to be able to write them. Here are a few recent examples.

In late 2011 I started getting my feet wet with Node.js, and in 2012 implemented several personal and commercial projects on top of Node, Express, and many other Node modules. I was very reluctant to start working with Node until I grasped the fundamental concepts -- the event loop and the asynchronous nature of everything -- which gave me the foundation to implement "something like Node" should I want to. There was even a time when I was considering to implement a Node-like HTTP framework with the new C# support for async/await, but gave up because there are many like it, ASP.NET MVC async controllers notwithstanding.

Next, I always was -- and to some extent, still am -- quite "afraid" of WPF. I can't say that I love client development in general, but nearly everything feels, on a purely psychological level, like a more appealing target for me than XAML-based frameworks. It's not that WPF is very hard for me to use: I understand the fundamental concepts such as data binding, styles, resources, and data templates, enough to implement simple desktop applications or sample Windows 8 and Windows Phone apps. It's the depth and breadth of WPF that always keeps me wondering: Is there a better way to do what I just did? How is this XAML expression working under the hood with this data context and that dependency property? Should I have moved this entire chunk of code into a behavior or a separate control?... I can't say I haven't tried: I read at least three books on WPF, with a total of more than 1500 pages, and they had had zero effect on my state of mind. The result, at least for me, is that on a subconscious level I steer clear of XAML-based frameworks, because I am not sure how to implement my own. The funny thing is that I'm fairly comfortable with several "thick" client technologies, including MFC, Windows Forms, Android, and iOS -- but it's as though XAML, with all its objective advantages, had developed a conditional gag reflex in my mind.

Moving on to something I am very comfortable with, I always felt at home with Reflection-based approaches to virtually anything, from serialization through validation all the way to code generation. Attributes and Reflection have somehow grown very strong roots in my mind ten years ago, when I was first working on a large .NET application, and have been a very powerful tool for me ever since. I attribute this mostly to the fact that I understand how managed types are laid out in memory, how .NET metadata is organized, and thus how Reflection can provide access, at runtime, to this information. In fact, I immediately feel at home with Reflection in other languages and frameworks as well: for instance, my first instinct when working on the Windows Azure Mobile Services unofficial Android SDK was to write my own JSON serializer in Java instead of relying on a third party library. After the fact, it might not have been the best decision, but it took me less than 2 hours to have a working serializer that supports most of the types WAMS requires.

As a final example, I have a very big problem adopting new languages, especially if they're not merely compiled to another language. In other words, I wouldn't mind using something like TypeScript or CoffeeScript, where I can easily see how the source is compiled down to JavaScript, and how a new piece of syntax is merely a mapping or syntactic sugar on top of existing syntax. But a "brand new" language, such as Objective C, poses huge problems for my mind. It's not the brackety syntax I have trouble with; it's the "how" of the language. How are Objective C objects laid out in memory? How are methods dispatched, considering that you can override them and even dispatch them dynamically by name? How does the compiler manage memory automatically (yes, reference counting -- but that's just a buzzword)? The same applies to a language like Python: I can use Python to write scripts, create modules, and even interact with C-style DLLs, but I don't have a clear picture of where attributes are stored and how typing works in a dynamically-typed language.

To conclude, I wish that every tutorial in the world were followed by a "how it works" section that would tell me how I can implement that language, technology, or framework -- on my own. Until such time, I hope this post explains why I tend to over-analyze how stuff works, to the extend of implementing it myself. The root cause of this might be that I'm used to systems, frameworks, and languages that I fully understand; it might have something to do with debugging or performance concerns; but at the end of the day, I need to know how it works: otherwise, I face a psychological barrier that gets more and more painful to overcome every day.

I am posting short links and updates on Twitter as well as on this blog. You can follow me: @goldshtn

Why App Stores Are a Necessary Evil

Sasha Goldshtein — Mon, 01 Oct 2012 15:37:05 GMT

I’ve just read an article on International Digital Times that laments the impossibility of distributing a Windows Store app externally, without using the Windows Store. Setting aside the oxymoron for a moment (distributing a Windows Store app through something that is not the Windows Store :-)), this is not as bad as the author thinks it is.

In fact, I sometimes get the feeling that people are bashing Windows 8 and the Windows Store not for its merits or disadvantages, but out of a “someone moved my cheese” feeling.

The Installation Model
The author doesn’t like the fact that you can’t get your favorite app from download.com, and have to go to the Windows Store instead. Frankly, most users will find it much more convenient to use the consistent Store interface, which requires no additional instruction. No need to download an installer file, run it, make sure you run it elevated, make sure you’re using a suitable version of the operating system, make sure you have whatever dependencies that application needs, and so on and so forth.

The beauty of the WinRAR download page

The author also complains that instead of Googling the app name, going to the website, and installing it directly, users will have to open the Store application and enter their Microsoft credentials. This is simply inaccurate. First of all, you can (and should) link Windows and the Windows Store to your Microsoft account. You won’t have to enter your Microsoft credentials again. Moreover, the author is overlooking the way users are going to look for apps. They will not Google the app and then go to its website. They will look up the app in the Store interface, which makes it dead easy to find, review, buy, and install any app you want.

Another objection could be that users aren’t used to installing apps through the Store, and will feel more comfortable with Google as the search interface. This may be true of users who have been living under a rock for the past 5 years, but not for anyone with an iPhone, Android phone, or Windows Phone. I’d say most users would be surprised that they have to go to a website to obtain an app instead of the consistent, trusted interface of an application store.

Monetization Opportunities
Users don’t have to know about it, but app developers have a much better monetization story with the Windows Store than with other distribution mechanisms. You have a built-in trial licensing mode which can expire automatically or limit the app’s functionality until it is purchased. You can embed ads in your apps from a number of provides. You can provide in-app purchases of game levels, periodic magazine subscriptions, and many other kinds of virtual goods. And of course, Microsoft takes care of the payment processing, in exchange for the 30% or 20% fee.

Compare this to commercial desktop software. Developers need to worry about trial licensing, application piracy, payment processing, subscriptions, additional purchases – and none of this is covered by any standard framework.

The Update Model
Consider Chrome, Firefox, Adobe Reader, iTunes – large applications that needs some means of automatic updating. The method of choice for these applications is to ship with a special service that runs in the background, checks for updates, and perhaps even installs them automatically. I can tell you with complete honesty that I’ve seen the Adobe Reader and iTunes auto-updaters more frequently than the respective applications. This is not an advantage of out-of-store distribution. This is a huge disadvantage that bloats the users’ PCs with a bunch of background services, which often run with elevated security privileges, consume battery power unnecessarily and expose the system to additional risks.

While we’re at it, the author seems to think that pushing an update to the application’s website is somehow better than pushing the update through the Windows Store. One thing is certain: passing the certification process for the update in the Store will take more time than pushing an exe file to your website. However, getting the update to your application’s users is definitely going to take much longer if you just push the exe to your website. You can’t seriously expect users to periodically check for updates. And if you pop a huge message in their face saying that the app needs updating and they have to go to the website, how is this any better than using the Store’s automatic update mechanism?

In Conclusion
If you feel so bad about your cheese being moved, go ahead an install desktop applications on your Windows 8 machine. I know I did. I spend most of my time on the desktop, with a choice of desktop apps like Visual Studio, Outlook, Word, Chrome, Internet Explorer, Windows Live Writer, Evernote, OneNote, and many others. In fact, I don’t even particularly like the built-in Windows 8 apps right now – the Mail app is crippled, the People app is slow, the Messaging app has limited support for my favorite chat services. But for me, that’s part of the value Windows 8 brings to the table. It is still a desktop operating system, with another face. A face that might – just might – become the bread and butter of Windows developers for years to come.

I am posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn

Why Asking Google-able Questions at Interviews Is Just Fine, Really

Sasha Goldshtein — Thu, 13 Sep 2012 20:17:12 GMT

I’ve recently encountered the position that it is bad form to ask interview questions which can be answered easily by using Google. Some examples include traversing a linked list, naming some important API, or describing the architecture of a popular framework.

I could not disagree more. The litmus test for interview questions should not be whether you can find the answer by researching the subject online. If that were the criteria, interviews would be ten hours long and consist of very difficult algorithmic questions – and of course you’d have to come up with these questions yourself!

There are perfectly legitimate reasons for asking questions that can be answered by researching online. Here are some examples:

Traversing/reversing/otherwise manipulating a linked list demonstrates basic understanding of data structures, runtime complexity, and in unmanaged languages – pointers, which are misunderstood by too many developers, not only on the junior level.
Naming important API demonstrates that you are working with it on a daily basis, and that you really know what you’re talking about. If you are a C# developer and cannot write a C# program that reads a few lines from a text file, you’re probably embraced too tightly by a framework (or three) and need to go back to basics once in a while.
Describing the architecture or design of a particular framework you’re using illustrates that you don’t accept things at face value, and are willing to invest the time to understand a subject in depth.

Yes, these things can be researched online, and I don’t care. I will ask you for the answers, and I won’t let you Google while answering. This is not about the fact we all use Google, StackOverflow, and a myriad of other online resources in our day jobs all the time. This is not about being able to research a topic online and come up with a great implementation. This is about being a Developer, and not mashing up pieces of code with no understanding what they do.

I am posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn

Honored to be Renominated as Microsoft MVP for 2012

Sasha Goldshtein — Wed, 04 Apr 2012 14:34:30 GMT

On April 1 (yes, it has the potential of being an April Fool’s every time!) I received the renomination letter—I am honored to receive the Microsoft MVP Award in Visual C# for 2012.

As I wrote a year ago, 2011 was an exciting year for C#—I had several opportunities to talk about async methods and the parallelism revolution last year, and 2012—the year of Windows 8—promises to be even more interesting for C# as we tackle the development of Metro-style apps.

I wouldn’t be writing this post if it weren’t for the help and support of my colleagues, friends, and managers at SELA and our business partners and friends at Microsoft Israel. Specifically, David Bassa and Ishai Ram are still the best managers and friends a tech professional can have, and Guy Burstein of Microsoft DPE continuously amazes me by shaping the Israeli developer community with MSDN events and user group meetings.

Thanks for reading—I am looking forward for the rest of 2012 and hope to continue providing you with interesting information on gory CLR internals and debugging problems :-)

I am posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn

Analysis of a Mobile Redirection Framework and Obfuscated Regular Expressions

Sasha Goldshtein — Tue, 31 Jan 2012 14:17:48 GMT

I don’t often read Haaretz, but there are from time to time articles that friends share on Facebook or come up in search results – and I find myself on the Haaretz website. Often enough, it happens on my mobile phone – and every time I find myself redirected to a very primitive version of the website. Compare for yourself:

(Screenshot on the right obtained by changing the user agent in the Chrome Canary build. Very nice built-in feature.)

I was curious what were the criteria used by the Haaretz website to do this redirect, and started sniffing around the traffic with Fiddler. After most of the Haaretz front page has been downloaded, the browser suddenly issued a request for g.watap.net/w2w/haaretz, which issues not one, but two 302 redirects and eventually lands on the crippled mobile version.

Interestingly, I’ve tried more than one mobile user agent, and the resulting mobile website was pretty much the same (so I am getting the same experience with an iPhone, Android, or a feature-phone). I believe this is a poor choice on Haaretz’s behalf, so I started investigating a little.

I started by running a whois query on watap.net, and found that it’s registered through Go Daddy for PassCall Advanced Technologies. Then I turned my attention to PassCall, and found on their website that they are providing a platform that adapts existing websites to mobile browsing. Indeed, I find Haaretz in their list of customers. From what I could tell, all the customers are Israeli companies, and the g.watap.net host resolves to an Israeli IP address, probably hosted by NetVision, a major Israeli ISP.

What is the precise process used by PassCall to determine whether or not to redirect my browser to the dumbed-down mobile version? I was brave enough to start reading through the ~8500 lines of HTML and script that is the Haaretz front page. Very close to the beginning there’s a copyright notice by PassCall with a minified script. I won’t paste the whole thing, but here’s a start:

I took this beauty to jsbeautifier.org where it got a much prettier shape. Here’s the first part, beautified:

Okay, this is obviously an unpacker – it even says function (p, a, c, k, e, d) right there. Thanks for the hint. So the first part is a slightly minified unpacker, and the hex-encoded strings (not shown here) are probably the actual code. Instead of trying to run the unpacking algorithm with a pen and paper, I simply put a breakpoint in the beginning of the script and started stepping in and out until I got this beautiful function, called f, which does the interesting part:

Note how this is no longer obfuscated, and perfectly readable. The script starts by checking if there is a cookie instructing it whether to do the mobile redirect or not. Marked in bold are the interesting parts – this is what we get if we have to make a new decision – and then the redirect itself is simply replacing location.href with a new location. The whole redirect-or-not logic boils down to three regular expressions (r1, r2, r3). Let’s take a look at these regular expressions. Here is r1:

^(((A|3|Q)l(v|5|c)a(t|3|2)e(l|3|x))|((X|6|E)Z(O|2|j)S)|((9|H|6)D(q|2|h)_(h|3|T))|((H|7|1)D_m(i|0|j)n)|((9|I|6)C(v|1|o)p(q|8|p)i(q|e|j)d(v|8|F)r(v|o|q)m(4|P|9)a(s|2|0)s(c|4|X)a(v|7|l)l(q|7|C)o(q|6|d)e(j|4|0)h(a|6|0)a(r|0|q)e(t|5|x)z)|((v|8|L)G(E|7|3)?[-/_])|((0|M|6)a(u|3|Q)i B(r|7|x)o(w|4|0)s(q|6|e)r)|((P|0|h)C(L|2|q)[4-6][4-6])|((q|6|S)E(h|C|Q)-)|((v|S|j)G(H|2|3)-)|((S|0|2)I(E|X|h)-)|((j|4|S)K_)|((q|6|S)O(j|4|N)I(M|x|q))|((8|S|4)e(0|n|4)d(j|o|Q))|((j|8|T)e(j|l|X)i(4|t|6))|((h|p|q)o(q|r|5)t(q|a|h)l(q|m|v)m(h|8|m)))

Looks like a bad-ass regular expression? Not at all. In fact, this is just a light attempt at obfuscating the regular expression without changing its meaning too much. Note that the whole thing is just a big disjunction over a bunch of strings. Here’s the first component:

((A|3|Q)l(v|5|c)a(t|3|2)e(l|3|x))

What could it possibly be? Obviously, it’s “Alcatel”:

((A|3|Q)l(v|5|c)a(t|3|2)e(l|3|x))

How about this guy?

((h|p|q)o(q|r|5)t(q|a|h)l(q|m|v)m(h|8|m))

This one is “portalmmm”, which apparently is a mobile user agent used by i-mode mobile browsers. Finally, what is this:

((9|I|6)C(v|1|o)p(q|8|p)i(q|e|j)d(v|8|F)r(v|o|q)m(4|P|9)a(s|2|0)s(c|4|X)a(v|7|l)l(q|7|C)o(q|6|d)e(j|4|0)h(a|6|0)a(r|0|q)e(t|5|x)z)

Fairly long to be a mobile user agent. Indeed, it becomes ICopiedFromPasscallCode?haaretz – which is a rudimentary copy-protection mechanism.

I can tell you right away that r2 is no different:

((a|3|Q)n(v|5|d)r(o|3|2)i(d|3|x))|((X|6|b)l(a|2|j)c(k|X|q)B(4|e|8)r(r|Q|0)y)|((G|x|h)T-P(v|9|1)0(q|0|x)0)|((H|7|Q)T(q|6|C))|((Q|6|H)u(6|a|5)w(X|5|e)i[u/-])|((i|1|0)p(h|5|a)d)|((q|i|v)p(h|9|0)o(h|6|n)e)|((j|2|m)o(5|t|8)o(4|r|8)o(l|2|q)a)|((4|M|7)O(T|7|0)[-_])|((8|n|6)o(k|3|x)i(h|2|a))|((s|Q|1)o(x|4|n)y(x|8|e)r(q|8|i)c(s|5|Q)s(o|3|Q)n)|((h|6|s)a(4|m|5)s(h|u|x)n(g|6|3))|((j|3|P)a(l|X|h)m)|((x|5|p)h(h|7|i)l(i|7|2)p(3|s|5))|((v|U|Q)P.(v|6|B)r(o|0|j)w(s|7|Q)e(r|3|q))|((w|5|2)i(7|n|5)d(q|3|o)w(6|s|4) (((9|p|5)h(o|X|q)n(e|x|q))|((h|c|q)e)))|((8|I|7)E(X|8|M)o(h|5|b)i(h|6|l)e)|((9|V|7)o(d|Q|j)a(f|8|Q)o(X|4|n)e)|((X|9|o)p(q|e|j)r(h|a|j) (v|m|q)o(v|b|x)i)|((4|o|5)p(e|0|2)r(v|4|a) (2|m|5)i(v|9|n)i)|((q|7|s)y(q|m|0)b(q|6|i)a(n|0|q))|((X|8|1)o(o|X|0)m)|( P(q|r|h)e[/])

This yields stuff like “android”, “ipad”, “iphone”, “windows phone”, “Xoom”, and many others. And then there’s r3, which will disqualify a user agent from redirecting to the mobile site:

((i|3|Q)p(v|5|a)d)|((q|9|V)i(7|e|8)w(j|P|X)a(Q|8|d))|((M|7|Q)Z(q|6|9)0(1|X|q))|((q|8|G)T-P(q|2|1)0(3|0|8)0)|((q|6|G)T-P(v|7|Q)5(v|8|0)0)|((j|5|X)o(3|o|5)m)

…and here we have “ipad” and “Xoom” again, which is quite silly because we just saw them in r2. Probably the obfuscation layer makes it hard for the PassCall developers to make changes :-)

All in all, here is a (partial) set of user agents that PassCall will redirect to the mobile website and a (partial) set of user agents that they won’t (the list is based on what I’ve seen on Haaretz’s website on January 31, 2012):

Will redirect if starts with: Alcatel, ICopiedFromPasscallCode?haaretz, LGE-, Maui Browser, SEC-, SGH-, SIE-, SK-, SONIM-, Sendo-, Telit-, portalmmm

Will redirect if contains: android, blackBerry, GTP-?, HTC, Huawei, ipad, iphone, motorola, MOT-, nokia, sonyericcson, samsung, Palm, philips, UP.Browser, windows phone, windows ce, IEMobile, Vodafone, opera mobi, opera mini, symbian, Xoom, Pre

Will not redirect if contains: ipad, ViewPad, MZ601, Xoom

I am curious about other tablets, such as the Samsung Galaxy Tab, meeting the criteria for a mobile device. Indeed, with the Galaxy Tab user agent (“Mozilla/5.0 (Linux; U; Android 3.0; xx-xx; GT-P7100 Build/HRI83) AppleWebkit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13”) we are getting the mobile version. And the most annoying thing? I don’t see a way on the mobile version to switch back to the desktop version if I’d like. And that’s the number one fallback you should have if you’re using blunt regular expressions to determine which website to show me.

To summarize:

Haaretz is using PassCall Advanced Technologies to redirect its mobile visitors to a crippled mobile version of the website
PassCall is using a set of regular expressions to determine whether a user’s user agent represents a mobile device and performs an unconditional redirect
PassCall provides the same mobile experience for a 2011 iPhone 4S and a 2005 Nokia feature-phone
There doesn’t seem to be a way to get back to the desktop version from the dumbed-down mobile one

I would greatly appreciate any comments and corrections. This research has been performed for personal purposes and does not represent the position of my employer.

I am posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn

Things Learned in 2011 and Plans for 2012

Sasha Goldshtein — Sun, 01 Jan 2012 15:22:03 GMT

I keep telling junior software developers that the only way to maintain their expertise and to become better developers is a continuous learning process. 2011 has been a very productive year for me (and many others at SELA!), and I am looking forward to 2012, the year of Windows 8, in eager anticipation. Below are some of the things I learned in 2011 and some of the things planned for 2012.

Learned in 2011

Most of Q1 2011 I was working on the brand-new Parallel Programming in .NET 4.0 course with Bnaya Eshet. I’ve been using the TPL for a while before that, but writing slide decks, labs, and demos – culminating in actually teaching the course – is definitely the best way to learn a subject of this magnitude.

Later during the year I updated the Windows Concurrent Programming course (for C++/Win32 developers), including brand new content on ConcRT, synchronization and threading internals, and a bunch of new hands-on labs.

Back in 2010 I delivered a user group presentation on C++0x, which emerged as a final standard in late September. At the December SDP, Noam Sheffer and I delivered a whole day on the new C++11 language standard and standard library, which involved getting all the nitty-gritty details of the new language syntax (such as perfect forwarding, which I find the hardest “feature” to teach).

Next up is JavaScript. Yes, seriously. If you ever talked to me for more than a few minutes you know that I hate UI development in general, and Web development in particular. However, this was a gaping hole in my understanding of Web-related performance problems and bugs, and a gaping hole in the “general education” of any software developer. So I forced myself to relearn HTML, CSS, and JavaScript, and used the opportunity to learn the new HTML5 JavaScript APIs, jQuery, and even node.js (which should be the subject of another post).

Another thing I was always very interested in but rarely had the time to invest in professionally is security research. Although my day job usually does not involve any serious security research, I like to keep my general knowledge up to date by subscribing to vulnerability disclosure lists, following notorious security researchers, practicing simple exploitation and reverse-engineering scenarios with modern tools, and so on. In 2011 I invested in my reverse engineering skills and some modern exploitation techniques such as ROP.

Any list summarizing 2011 will be woefully incomplete without mentioning Windows 8. At //build we saw a glimpse of what’s to come in 2012, but the Windows 8 Developer Preview is already an exciting consumer experience and an exciting target for software development. I wrote an article on Windows 8 security aspects [pdf, Hebrew] and presented Windows 8 at the December SDP keynote, but there’s obviously much more to learn here, and I’ll leave it to 2012.

Finally, here’s a grocery list of some smaller things I learned in 2011 (some of which I don’t understand yet in a professional capacity):

Windows Phone
GPGPU, specifically C++ AMP
Fundamentals of hardware architecture
Refreshment of Windows Internals for Windows 7
DBX debugger on Solaris
CLR Debugging API

Planned for 2012

2012 is going to be remembered as the year of Windows 8. With a sharp turn to a new runtime, development framework, UI style, and form factors – it’s inevitably going to take quite a while to learn and practice. From what I’ve seen so far, it will be fun :-)

General-purpose GPU computing (GPGPU) is becoming mainstream, especially with an awesome framework like C++ AMP behind it. In 2012, I hope to find opportunities to use C++ AMP in a real project, and expand my understanding of the underlying GPU concepts.

Finally, on a less professional level, I plan to expand my horizons in Web application security, specifically new advances in HTML injection, smart XSS attacks, crypto weaknesses, and similar topics.

Now, if I had to guess in 2010 whether I will be spending time on Solaris or JavaScript in 2011, there’s no way I’d say yes. I’m really looking forward to January 1, 2013 to see what actually happens in 2012 :-)

I am posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn

Tracking Engagement Time Using 302-Moved Temporarily Redirects

Sasha Goldshtein — Sun, 06 Nov 2011 13:31:14 GMT

Suppose you are sending mass emails (legitimately, no doubt) and want to know which % of recipients actually viewed the email. The standard trick here is to embed a 1x1 image into your email’s HTML source, with the <img src= pointing to a location on your Web server with part of the URL unique to the user (e.g., <img src="http://example.com/track/12345" /> where your mailing system knows that 12345 is associated with john@example.org). When the user opens your email, most email clients will send your server a request for that image*, and voila—you know that the recipient opened it. It’s also quite easy to determine which operating system + mail client combination was used to open the email, by inspecting the HTTP request headers.

But what if you want to track engagement time, i.e. the time the user actually spent reading your email**? This is a valuable metric that can tell you how your messages are working across different groups of users and how engagement times are translated into monetized actions.

This sounds more scary—but here’s another simple trick I’ve seen as recently as yesterday. Embed the same 1x1 image into your HTML source, but this time your Web server needs to be somewhat smarter. When it receives a request for the image, it should return a 302—Moved Temporarily response, with a new URL for the image. When the browser hits that new URL, your Web server returns yet another 302 with yet another URL. This continues until the client gives up—which is usually when the user closes your email.

Client (browser, email client)	Server
GET /track/12345	302 Moved Temporarily Location: …/track/12345?n=1
GET /track/12345?n=1	302 Moved Temporarily Location: …/track/12345?n=2
GET /track/12345?n=2	302 Moved Temporarily Location: …/track/12345?n=3
…you get the idea…

The server, on its end, knows the exact times of the requests received from the client and can track not only the fact that the email was opened, but also the engagement time, as defined above.

Is this a legitimate tracking mechanism? In the era of persistent cookies, cookies that stay with you even if you’re logged off the website, tracking that relies on ETags even if you have turned of all cookies and cleared your Flash and HTML5 local storage—it is probably a little naïve to talk about whether a simple image embedded in an email is a legitimate tracking technique, from a morality standpoint.

However, I think it makes sense to discuss the cost of this mechanism, from the client’s perspective. Unless the Web server throttles the request rate by delaying the response as much as possible, the client will issue requests as quickly as it can. Suppose that the RTT is 300ms, and suppose that each request-response exchange takes 1.5KB. That’s a transfer rate of 4.5KB/s, or 270KB per minute. Over a broadband connection at home this is not something to worry about. However, over a GPRS data connection with a limited data plan, or—heavens forbid—a roaming connection, this is a non-negligible amount of data.

Other considerations that come to mind are the CPU and network utilization from a power management perspective. When the email client keeps issuing requests, the network card is probably unable to enter a low power state even if everything else on the system is idle. On mobile devices, tablets, or laptops this may have an effect on battery life (although not as bad as playing Angry Birds :-)).

* Well, actually many email clients have a setting that blocks remote images. And of course there are plain-text email clients that don’t render HTML emails at all. But let’s focus on the users you can track.

** To be accurate, we will see how to determine the time elapsed from the moment the user opened your email to the moment she closed it or switched to another email. Tracking the time the user was actually reading the email requires more sophistication—perhaps eye-movement tracking through Webcam?—that is left as an exercise :-)

9/11

Sasha Goldshtein — Sun, 11 Sep 2011 12:46:52 GMT

A couple of days ago I got a phone call from my dad at 7:30AM. He asked “Are you guys OK?” and immediately scenarios of terrorist acts or missile launches started running through my head. It scared me – this unnatural but immediate reaction to an innocent question.

The reality of living in Israel during the last 20 years has conditioned all of us to think in terms of surviving yesterday’s bombing or tomorrow’s war, and the periods of relative peace sometimes spanning months in a row aren’t enough to wipe away the instincts and fears.

Whenever I use public transportation, I “evaluate” all the passengers for terrorist threats and examine every new face boarding the car. Whenever I hear the siren of an ambulance I wonder if it’s someone having a heart attack or a bombing in the middle of a shopping center. I know from conversations with my friends and family that they are wired that way, too.

Ten years ago it was 3:45PM in Israel and I was watching a show on TV with a friend. A few minutes later, every news channel was broadcasting the unbelievable images of the burning twin towers, and my friend had difficulty believing that it wasn’t footage from a poorly-executed action flick.

We will never forget 9/11. But ten years later, today more than ever, with the very real threat of terror engrained deeply into our instincts, we should refuse to be terrorized.

Dear Team Lead, You Are Not Doing Agile If…

Sasha Goldshtein — Tue, 30 Aug 2011 18:05:54 GMT

…Your sprint planning meeting begins with a condescending description of what The Methodology looks like, and ends with “meet me here at 4PM – you will be assigned tasks and pairs”.

…You switch task management tools every week, never failing to surprise your developers and upper management. (Hint: more tools is not necessarily better. Bugs on the whiteboard, tasks in Excel, projects in TFS, and resource scheduling in a custom tool is confusing.)

…You have a heterogeneous team with young developers and you don’t do code reviews. (And still, you find the audacity to complain about bugs introduced due to lack of experience.)

…You do testing in production. (Oh, I mentioned this one before.)

…You have continuous integration and nightly builds, but the nightly builds never run and the CI build always fails. (“Only these 200 tests keep failing every time” – not an excuse. Commenting out failing tests – not an excuse.)

What do you say, dear readers? What are your favorite “You are not doing agile if…” moments?

I have been recently posting short updates and links on Twitter as well as on this blog. You can follow me: @goldshtn

Dropbox, Instapaper, and the Cloud: Entrusting Your Data

Sasha Goldshtein — Tue, 12 Jul 2011 20:59:16 GMT

I don't typically rant about security or "The Cloud", but as an avid Dropbox and Instapaper user I've had some comments building up inside for the past few weeks.

Dropbox is a simple private file sharing service which gives you access to your files from a variety of devices (I use it on my Windows laptop, Windows desktop, MacBook Air, iPhone, and iPad). Instapaper is a tool for saving web pages for later viewing – when I don't have time to read a long blog post or interesting article, I click a bookmark in my browser and the text gets saved to my Instapaper archive (I use it on all my PCs, iPhone, iPad, and Kindle).

Recently both services have hit the headlines with unfortunate security-related stories. A brief recap of what I'm referring to:

Dropbox rolled out an update that enabled you to log in without the correct password. This update was live for over four hours until it was detected and fixed. (The obvious question of "how on earth does this happen" is left as an exercise for the reader.)
Instapaper's database server was captured by the FBI in a raid on Instapaper's Web hosting provider. It was later discovered that the FBI did not target the specific server, and did not capture the hard disk which was stored in a separate enclosure. The server was subsequently returned.

These two seemingly-unrelated stories finally made me understand that I trust service providers with my data, having not much more than anecdotal information about how the data is stored, how it is secured, and what happens to it along the way. In fact, I have no idea where in the world my Dropbox files and Instapaper bookmarks are stored, how employee access to them is regulated, which governments can capture them given a court order, and what backups are in place in case the whole datacenter goes up in flames.

Am I supposed to perform this investigation every time I entrust my data to a service provider? What do you do?

Microsoft MVP, Third Time’s a Charm

Sasha Goldshtein — Sat, 02 Apr 2011 02:56:31 GMT

I just received my renomination letter for the Microsoft MVP award for Visual C#, 2011. This is the third year in a row, and what a blast these years have been!

I firmly believe that 2011 could be an amazing year for C#; what with the introduction of async methods in C# 5, which is a clear demonstration of how frameworks and fluent interfaces pale in comparison to the convenience and power of language keywords.

This time I am humbly receiving the MVP award as the CTO of Sela Group, where I am constantly amazed by the number and talent of technical folks who are taking us to new challenges in Israel and abroad. So in conclusion, I’d like to thank some good friends and colleagues:

David Bassa and Ishai Ram, my managers and friends at Sela, who work with me at easy times and hard, and let me pursue whatever direction I consider important at the time;
Guy Burstein, who leads the local Microsoft community by example and continues to be, after a few years with DPE, a role model of a technical evangelist;
My colleagues, Sela’s top technical experts—too numerous to list here, who never fail to surprise with yet another area of expertise and yet another great answer.

Finally, to my readers—thanks, and stay tuned for 2011, 2012, and many more years to come :-)

I’m a Microsoft MVP (again)!

Sasha Goldshtein — Thu, 01 Apr 2010 22:24:04 GMT

I just received a letter notifying me that I was awarded the Microsoft MVP award for Visual C# for 2010. I’m very honored to receive the award (the second time). Here’s to hoping that 2010 will be a good year for C# and anything else technological that I’ve been talking about on this blog, in online communities, at UG meetings, and at various conferences!

I would never have made it without the help and support of several people I would like to call out here, as well as many friends and colleagues not listed here:

David Bassa, Erez Fliess, and Ishai Ram, my managers and friends at Sela whose endless patience, support, and willingness to cooperate with my craziness have been vital to my overall happiness and my technical achievements;
Alon Fliess, who continues to be my role model as a technical leader, CTO, well-rounded guru, and good friend;
Guy Burstein, a good friend from Microsoft who kept the Israeli developer community alive and kicking during the past few (difficult) years, and who is constantly coming up with great new ways to involve more and more developers in online and offline activities.

And last but not least, thank you, dear readers, for following me during the last year and for your emails and comments.

Beware of Evil Wizards

Sasha Goldshtein — Sat, 09 Jan 2010 23:18:37 GMT

Ten years ago, Andrew Hunt and David Thomas published the incredible book, “The Pragmatic Programmer: From Journeyman to Master”. It’s not that I like this book because it has taught me so many things; I mainly like this book because of what it has not taught me.

And one thing this book has not taught me is to use Add New Item –> “Some Amazing Technology” Class Template and then just adjust the grid on the control ever so slightly, and bam – there’s your user interface in all its glory.

Here’s what “The Pragmatic Programmer” has to say about it:

[…] Tool makers and infrastructure vendors have come up with a magic bullet, the wizard. Wizards are great. […] But using a wizard designed by a guru does not automatically make Joe developer equally expert. […] Don’t Use Wizard Code You Don’t Fully Understand.

Why is that that we developers should not rely on wizard-generated code, but it’s OK for us to rely on the operating system’s code or on the .NET Framework’s code?

[…] We’d feel the same about wizards if they were simply a set of library calls […] that developers could rely on. But they’re not. Wizards generate code that becomes an integral part of Joe’s application. […] Eventually, it stops being the wizard’s code and starts being Joe’s.

It’s funny how these words never ceased to be true. Despite the ten-year-old warning against Evil Wizards, code generation inside the development environment has become an increasingly popular technique, and not just for spiffy user interface generation. Here are two wizards I use all the time, in Visual Studio:

Adding a WCF service reference generates a whole bunch of proxy code into your project, and not just the bare minimum service contracts and data contracts that you would otherwise write by hand;
Dragging a table across a LINQ to SQL design surface generates a strongly-typed data context, an entity class for each table, relationships between entities, identity constraints and whatnot.

Ten years after “The Pragmatic Programmer”, I’m not saying that you should stop using the Visual Studio wizards to generate code for you. But if you don’t understand the code generated by the wizard even though you’re using and adapting it to your needs, you’re in for some great trouble. (Unfortunately, presenters at conferences and class teachers are often guilty of spreading the trouble.*)

The telling sign is this: If a technology were easy enough so that you didn’t have to thoroughly learn it before applying it, why would there be a wizard for generating code associated with that technology?

* The most unfortunate thing is that sometimes even we, the public speakers who show other people the power of those Evil Wizards, don’t fully understand the code being generated by them. I had the chance to witness, time and again, a presenter showing a bunch of code effected by a simple click of a button, and when something goes wrong the only resort seems to be to regenerate the whole code from scratch. Is this something you would recommend to your fellow developers?

What Gets Me Through the PDC: Conference Gear

Sasha Goldshtein — Wed, 18 Nov 2009 07:30:43 GMT

What does every geek need to get through the PDC, or any technological conference for that matter? I have the simple answer that works for me and that you might be interested in :-)

First and foremost, I need a laptop. It has to be lightweight but usable, so a high-end netbook could do it but a mid-range notebook does a better job. My choice for this conference was the Dell Latitude XT, a multitouch 12.1” laptop that served me well for Windows 7 courses and demos in the past. It has a decent form factor, isn’t too heavy, and its battery can last the few hours if I lose the occasional fight-for-an-outlet.

Next, if I have to get through an entire day without a charge, I need an external battery. Laptop replacement batteries are OK but you need to hibernate which takes a lot of time, and you need to carry them around and charge them inside the laptop. Instead, I use an external battery (the EL1901) that weights just under 0.8kg, and provides my notebook with well over 8 additional hours of battery time.

I couldn’t imagine walking around with the laptop and external battery and the laptop charger and all the other tools without a decent trolley. For the PDC, the trolley question was easily solved – my company, Sela, provided custom-tailored trolleys for the group of experts that are attending the PDC.

When you’re coordinating a group of almost twenty experts in four rental cars, you really need the proper means of communication. Sela bought prepaid cell phones from various wireless providers so that we can all talk to each other and coordinate our activities. (A lesson learned from last year’s PDC, when there was a much smaller group of us around – and yet it was extremely difficult to keep everyone informed.)

Other than that, there are the obvious: business cards, USB flash drive, a couple of USB cables, an extra charger, a power adapter, a jacket for the air-conditioned conference halls, etc.

What do you carry around at conferences?

Learning from Feedback as a Public Speaker

Sasha Goldshtein — Fri, 02 Oct 2009 02:26:37 GMT

Over the last three years, I’ve had lots of experiences as a speaker at conferences, courses, private presentations and other opportunities. Among them, I had the chance to present at the Microsoft Developers Academy, TechEd and IDCC; I taught 54 courses for Sela, including Windows Internals, .NET Debugging, .NET Performance, C++/CLI and many others; and I lectured at short half-day MSDN events on Windows 7, Windows Server 2008, performance, debugging and concurrency.

Almost every training session, every course, every conference concluded with audience feedback. (If you ever filled an evaluation form for my presentation or course, thank you for your time and feedback!) I’ve been meticulously collecting this feedback for the past three years, and figured that now would be a good time to look back and share with you how you think I’ve been doing, and what I’ve been doing with this information.

They say that positive feedback is all very similar; negative feedback is varied and interesting. Here are some of the things I’ve noticed during the past three years.

“Less Code, More Theory”

This is something I’ve been hearing for a long time during the first couple of years, and always wanted to work on. I guess from a certain perspective it’s just that much easier for me to focus on what I’m really good at—showing deep technical demos, writing lots of code on the stage, typing commands into the debugger, and so on. It was much harder for me to talk when I didn’t have any code or live demo to back me up.

During the last year, I focused more and more on subjects such as design for concurrency, performance testing methodology and case studies, architecture of distributed system and other topics that often can’t be accompanied by a simple code demo and must be explained using hand-waving. This is certainly not my comfort zone but I’m pretty happy with the results so far, and will keep trying to get better at it.

“Way Over My Head”

Almost every one of my conference presentations, courses, half-day sessions has been tagged level-400. Often enough it was even more hardcore than many other level-400 sessions at the same conference. So it shouldn’t be a big surprise to my audience that my lectures often require deep technical knowledge, lots of practical experience and the ability to focus on small details as well as the big pictures for a couple of hours straight (which I know is not easy—I always find it exceptionally difficult to keep myself focused on the speaker when sitting in the audience at a conference).

Nonetheless, often enough I get feedback indicating that I was talking too fast, or going over too many details, or diving deep into a piece of technology and assuming it’s just familiar to everyone. This is something I do way too often. Over time I managed to create a certain distance from the subject matter of my lectures, and through this distance I understand how the audience is absorbing my presentation. Still, predicting how easy or hard it’s going to be for other people to take in a particular piece of information is something I need to work on. And after all, everyone would be much better off if I simply cut the material in half but presented it thoroughly and slowly rather than lose half of the audience half-way through the session.

“Introduction Too Long”, or “There Wasn’t Enough Time to Talk about X”

This is a piece of feedback I’ve been getting recently, and I suspect it has something to do with the way I’m handling “Less Code, More Theory”. As I gain familiarity with a subject and observe it from different angles, there’s obviously more for me to rant about in the introductory part of my sessions. For example, when speaking about concurrency and parallelism I can often find myself thirty minutes in without even mentioning a specific technology yet—I can just go on and on talking about how parallelism is vital to modern applications.

On the other hand, if I make the introductions too short I’m pretty sure we’ll be getting back to “Way Over My Head” and “Less Code, More Theory”. I’ll have to see how I can incorporate this piece of feedback into my shorter presentations, where there’s little time to spend on a stretched introduction.

“More Code / Demos, Less Slides”

I never believed I’d see this feedback on a session of mine, but apparently my efforts to get out of the live coding comfort zone might have gone a little over the edge for some folks. I know for certain that seeing 30 slides in the row is the most boring thing known to mankind, and I profoundly apologize if my storytelling skills are not up to par with the boredom-inflicting nature of a demo-less presentation. However, I also feel that sometimes not focusing on a particular technology and not driving the point home with a demo is more beneficial if there’s a general-purpose message to deliver.

I can’t promise you that my future presentations are going to have more slides or more pictures, less code or less live demos—but I can tell you this: I greatly appreciate your feedback over these years, and I’m striving to improve with every bit of it.