Who Accessed My File?

A few years ago, I recall needing to know (programmatically) which user has accessed a particular file.  As part of a legacy system, this couldn't make the code base any worse.  The idea was that there's a configuration file sitting on a network share; access to it is granted to only a single user at any given time.  However, other users requesting access want to know why access is being denied - i.e., which user is holding them from accessing the file.  This information would be displayed to them, and then they were able to go and (physically) kick that person off the file if they really needed it.

Apparently, I wasn't the first person on this planet to ever need something like that.  Alik Levin outlines a very easy and elegant solution using the excellent Process Monitor tool from the SysInternals suite.  In a previous post, he also mentions that you can use Group Policy MMC snap-in (or Local Security Policy) to enable auditing.  After that, you can use the Event Viewer to see audit messages.

Nonetheless, a few years ago I found myself in a position when I needed to do all that programmatically.  Equipped with nothing but Win32 and C++, I effectively needed to perform the following:

• Enable the SeSecurityPrivilege in the current process' token if it wasn't already enabled;
• Add a SACL ACE (System Access Control List Access Control Entry) representing the audit entry to the file being audited on the remote machine;
• Enable auditing on the remote machine (one-time manual procedure, didn't find a programmatic way to do it at the time);
• Traverse the event log on the remote machine whenever necessary to find the audit trail for the file access, and see the user on whose behalf it was being performed.

I vaguely remember that it took me the good part of two weeks to get everything right and tested (especially since I never was, and still am not a Windows security expert by any means).  And after all, it was such a fragile solution.

Out of curiosity, I wanted to check how long a .NET implementation of the above will take, given that the framework encapsulates everything related to file system auditing and event log access in easy-to-use, well-documented types.

Well . . . it took me exactly 2 minutes to write the code that sets up a SACL ACE for a file of my choice, using FileSecurity and File.SetAccessControl.  Which was beautiful - the sores inflicted upon me by that Win32 code from years ago have actually started to heal.

Next, I looked into retrieving the event records from the Vista event log.  At long last (in .NET 3.5) we have a managed model to access Windows Eventing 6.0 functionality - my Vista event log has the audit train glaring at me with all its XML beauty, and I have my good friends in the System.Diagnostics.Eventing.Reader namespace to read it!

So, here's the query we should pass: *[System/EventID=4663 and EventData/Data='D:\Temp\1.txt'].  After that, it's just about getting the Event/EventData/Data[@SubjectUserName] value from the event record XML.  In code, the whole thing boils down to setting up an EventLogWatcher:

. . . and then handling the EventLogWatcher.EventRecordWritten event:

So this leaves us with two things to do: enable the SeSecurityPrivilege if it's not enabled already, and turn on "Audit object access" in the Local Security Policy.  Enabling the privilege is a task best left to native code, with OpenProcessToken, LookupPrivilegeValue and AdjustTokenPrivileges.  So is changing the audit policy - LsaSetInformationPolicy with POLICY_AUDIT_EVENT_INFO hand you this on a silver platter.  This is not more complicated than interpreting MSDN documentation correctly - look at the source code if you're interested.

As always, you can download the sample code (Visual Studio 2008 solution).

Wonders with WCF's DataContractSerializer

Serialization possibilities in the .NET Framework have been significantly augmented with the advent of .NET 3.0.  Tradionally, you had the choice between using the BinaryFormatter used by .NET Remoting (or its slow and deprecated companion, the SoapFormatter) - producing binary data in a completely proprietary format using Reflection as the underlying engine for serialization discovery - and using the XmlSerializer - employing code generation to produce well-formed plain-text XML, as long as your object graph didn't have cycles, didn't try anything out of the ordinary, exposed every serializable piece of data through a public get/set property, and didn't use derived types where base types were expected.  (My bitterness about the XmlSerializer boiled down to implementing a SerializableDictionary<K,V> and SerializableList<T> to allow derived types where base types were expected...)

The DataContractSerializer introduced with WCF is designed to address the compatibility issues of the BinaryFormatter, while still allowing for plain-text XML or binary output, for cycles in the object graph, for data not explicitly exposed through a get/set property, etc.  As a side note, it also features significantly better serialization performance.

This is all fun and games, so why not measure it?  And while we're at it, why measure serialization when we can measure the entire WCF communication stack as opposed to .NET Remoting?  So let's get going.  What we're going to measure is WCF as opposed to .NET Remoting, on two parameters: the amount of data transferred, and the kind of data transferred.  As far as size is concerned, we'll stick to powers of 2 between, say, a tiny 256 bytes packet, and a gigantic 4MB chunk.  As for kinds of data, we'll try a simple array of bytes (byte[]), a simple data structure, and a complex data structure aggregating additional data structures (introducing a relatively complex object graph).  This simple benchmark produces the following (surprising?) results with 100 invocations per each data size tick.  (Note that the scale is logarithmic on both axes.)

So... what on earth is that?  WCF's promising amazing optimizations, while beating Remoting easily with an array of bytes or a simple data structure, fail so badly with a slightly more complicated object graph?  (You might argue that proper data contracts should not exhibit complicated object graphs.  Tough luck, welcome to the real world.)

Now, what's so complex about that complex data structure?  Here's the definition (data contract) of what we're transmitting over the wire:

And here's the way it's initialized:

Hmm, so what we're effectively creating is an object graph along the following lines:

Everyone's pointing at the same data, plus there are lots of references to the same grandchild container ("Kiddo Container").  Let's try something else entirely - why don't we change the graph so that every container gets its own copy of the data:

The object graph looks much better now that everyone has their own copy of the data, and children aren't duplicated across the graph.  By the way, it's clearly larger in size because there are more copies of the data.  Let's see how this affects performance:

Back to what we used to see!  So what could possibly be so wrong with WCF to fail in the previous scenario so badly?  It must have to do with serialization, because it's fairly unlikely for the transport mechanism to behave differently with respect to the data being transferred.

There are several bloggers who have previously mentioned the merits of using the little-known "preserveObjectReferences" flag when constructing the DataContractSerializer.  This flag has the effect of the DCS using non-interoperable extensions to serialize object references as references, without reproducing the entire object in the serialized output.  The BinaryFormatter has this behavior built-in because it doesn't care about producing interoperable XML; WCF, designed for interoperability, doesn't give you this non-standard extension as a default.  Which means that in our ill-behaved scenario, each "Data" object is being laid out separately in the serialized stream, causing a terrible bloat compared to the BinaryFormatter.  Compare and contrast the stream sizes produced by the two serializers for objects of various sizes:

How about the "normal" scenario, where objects in the graph are distinct and the BinaryFormatter doesn't have the advantage of properly serializing references?

Much more legitimate, and also explains the results!

If you've read until this point, you're probably wondering what good is all this stuff.  After all, you can't always modify the object graph so that it's suitable for the DataContractSerializer.  And on the other hand, you can't really control the DataContractSerializer used by WCF.  Or can't you?..  Of course you can!

We can apply an operation behavior derived from DataContractSerializerOperationBehavior to our WCF operation contract.   In that custom behavior we will specify the "preserveObjectReferences" flag for the DCS that we create.  Once we do that, WCF will use our DCS and produce a significantly more compact serialization in the ill-behaved scenario.  The code here is mostly boiler-plate - it's attached to this post as part of the benchmark solution.  But here are the results:

WCF wins again!  (Bear in mind that we've adopted a solution that kills any chance of interoperability with non-.NET code, but then again that's what .NET Remoting is all about.)

As a final note, if you're looking to minimize the message footprint and enjoying WCF nonetheless, there's another option not measured here - the NetDataContractSerializer.  It behaves similarly to the BinaryFormatter, but can be used by WCF's serialization mechanism (so it's possible to change the operation behavior in the benchmark to return a new instance of the NetDataContractSerializer).  I haven't performed extensive measurements, but WCF with NetDataContractSerializer seems to outperform .NET Remoting in every scenario out of the box.

(You can download the benchmark sources from here, as a Visual Studio 2008 solution.)

User Account Control Helpers Library

I've just released a preliminary version of the User Account Control Helpers Library, a bunch of code meant to assist developers targeting Windows Vista with the inevitable adaptation to UAC.

This preliminary release addresses two major hurdles in porting applications to Vista:

• Embedding an application manifest in each and every executable to specify a requested execution level;
• Interacting with the UAC mechanism for application compatibility purposes.

UAC in a Nutshell

The primary goal of the UAC mechanism is improving security in the Windows operating system.  This worthy objective is achieved on two axes:

• Making more actions in the operating system standard user-friendly (i.e. not requiring administrative privileges);
• Filtering the user token for users who have administrative privileges on the machine so that most of their operations are not performed under the full administrative token.

An application (EXE) may require elevation to administrative privileges if the user is a standard user or if the user is an administrator but runs under a filtered token.  This elevation presents a UAC consent prompt if the user is an admin with a filtered token, or presents a credential prompt if the user is a standard user.  An application does that by embedding a manifest into the executable, which specifies the requested execution level for that application.  This also prevents virtualization for the application, which is usually undesirable if you are porting to Vista and have the ability to modify the application configuration and installation.

Download and Examples

The preliminary release is available for download in 32-bit and 64-bit flavors from CodePlex.  The source code is also available.  You're more than welcome to contribute ideas or code to the project by signing up on CodePlex or leaving a comment on the contact form.

Some usage examples to whet your appetite:

More Information

For more information on UAC and manifests, consult the following resources:

• TechNet: Understanding and Configuring User Account Control in Windows Vista
• C. Heller: How to Embed a Manifest in an Assembly: Let Me Count the Ways
• KB: How to Disable the User Account Control Prompt for Certain Application
• Daniel Moth: Blog, UAC Category

Dynamic Reply Endpoints in a Duplex Workflow Service

The Workflow Services paradigm in .NET 3.5 gives us the ability to expose a workflow as a WCF service, and to communicate with a WCF service from within a workflow.  These useful scenarios are enabled by the ReceiveActivity and the SendActivity respectively.  (If all of this sounds new, check out Guy Burstein's primer on Workflow Services.)

The built-in SendActivity is great until you need to provide custom endpoint information.  Basically, there are three options, in rising degree of complexity:

Option 1 (Fully static)

Statically configure SendActivity.ServiceOperationInfo (specifying which method on which contract type you want to invoke), SendActivity.ChannelToken and SendActivity.ChannelToken.EndpointName (specifying the endpoint name from your application's configuration file). - EASY

Option 2 (Only endpoint address is dynamic)

Do all the above but dynamically set the SendActivity.CustomAddress property to redirect the invocation to a different endpoint address.  The binding, contract and endpoint behavior all remain the same because they are still specified in the application's configuration file. - RELATIVELY EASY

Option 3 (Endpoint is fully dynamic)

Do not provide an application configuration file.  Configure the workflow runtime so that it (and you) can determine, in run-time, what the corresponding endpoint for a SendActivity should be. - DIFFICULT

So what do you need to do to enable this highly flexible third scenario?  The WorkflowRuntime hosted in your WorkflowServiceHost uses a service called the ChannelManagerService to pool and locate channels whenever a SendActivity needs to perform an invocation outside the workflow.  Therefore, to customize the endpoints, you need to provide your own instance of the ChannelManagerService class before starting the workflow host, and that ChannelManagerService instance will be pre-configured with your endpoints (also called "code endpoints").  By the way, these code endpoints are preferred over endpoints specified in the configuration file, just as you might expect.

Here's how you do it.  In my sample, the workflow contains a ReceiveActivity which waits for an invocation of IMyWorkflow.Do.  After that, there's a SendActivity which invokes the IMyWorkflowCallback.Done method.  So this is the workflow and the interfaces:

Next, we write a callback object which will receive the IMyWorkflowCallback invocation.  Here's the callback object:

Now we bring that callback object up as a service:

Next, we initialize a ChannelManagerService instance to resolve the endpoint to our callback in runtime (note that the endpoint name here has to match the endpoint name from the SendActivity's channel token):

Finally, we bring up a WorkflowServiceHost, find the WorkflowRuntime, add the ChannelManagerService as a workflow runtime service, add the workflow endpoint (for the ReceiveActivity) and start up the host.

All that's left is test the implementation by creating a workflow by sending a message to the service, and seeing that we receive a reply to the callback object created earlier:

After coding this up, I've had another idea which would give me even more flexibility.  Why wouldn't I write a ChannelManagerService class of my own (or derive from the existing one), and provide the actual endpoint dynamically?  I could have an event to be fired whenever an endpoint had to be resolved, and there I have a fully extensible implementation.  Sadly, this is plain impossible.  First of all, the ChannelManagerService class is sealed.  Second, when the SendActivity is looking for a channel, it's not just asking the ActivityExecutionContext to provide a ChannelManagerService instance.  It goes and calls ChannelManagerService.Take, which is an internal method, directly!  (How that is supposed to be compliant with basic guidelines for using workflow runtime services, I do not know.)  So this means I have to derive from SendActivity and completely replace the base class' implementation of Execute, by means of duplication - and I'm certainly not going to do that.

The code demonstrated in this post can also be downloaded from here, as a Visual Studio 2008 solution.

(These steps are also performed, without too detailed a documentation, in the WorkflowServiceUtility MSDN sample.  To check it out, download the samples - a modest 22MB self-extracting archive - extract them to some directory, and navigate to the WCF\Scenario\WorkflowServices\WorkflowServiceUtility\CS sub-folder.  The LocalWorkflowServiceHost.cs file contains a sample implementation which accomplishes the above requirement by deriving from WorkflowServiceHost and providing a ChannelManagerService instance.)

Notes on Teaching Introduction to .NET

Today I've had an interesting experience of teaching a one-day session titled "Introduction to .NET" (roughly based on this Sela course).  I wouldn't find it post-worthy whatsoever (except for maybe the fact that class started at the dumbfounding time of 7:45AM) if not for the audience.

The group consisted of about 15 ultra-Orthodox men (Haredim; Wikipedia: Hebrew, English) who are going through several months of training to eventually practice software testing.  It is fairly unique in the way people without any previous experience with computers are trained to be software developers and software testers.

When working with this kind of group, you cannot assume any previous knowledge most computer-literate people take for granted.  For example, "Open the text file located at C:\Temp" is not accurate enough an instruction.  "Do File->New->Project" is worthless if navigating through menus is not muscle memory for you.  Telling the class to copy some files around has the potential to become an exercise in running around the class in circles, copying the files on every individual's computer and repairing the damage already caused.  Explaining how a property is different from a field, how code reuse is good but copy-paste is bad, how web services and data exchange are not the same as .NET compilation to IL - things trivial to a "normal" developer-in-training audience cease to be trivial any longer.

And I am very glad that it happened today.  It's actually enlightening to descend from the ivory tower to reach into the oh-so-large group (say, 80%) of programmers who really need your training, who really learn from what you have to say, and who do not approach you with inherent snobbism and suspicion whenever you contradict one of the truths they might have read in a blog post the night before.  (Even in this group, when I was talking about .NET compilation to IL, one of the students said that he remembers a lecturer telling him that VB.NET is inherently slower; thus, my saying that VB.NET and C# compile to the same IL made his question the correctness of that statement.)

In this respect, it is far better to teach people who are not yet full of prejudice, reflexes and half-truths on the material you're trying to convey.  However, while sometimes more fun, it is also one hell of a responsibility.  Consider what happens two years from now if I didn't explain correctly what a function call is.  What a .NET language is.  What the JIT is and why is it important.  And we certainly don't want to produce developers who have been taught partially-correct, half-baked truisms about the technology in question; there are enough of these today, as we speak.

And so I inevitably must conclude that trainers who are teaching classes for this kind of audience must be far better trained from all perspectives - the professional, the social and the training perspectives - than "normal" trainers teaching "normal" classes.  Bizarre as it may sound, this is the only way of giving the software development industry any kind of hope.

(By the way, if you're curious what kind of exercises you could possibly give people after less than a day of .NET training, without even covering any single language properly, here's what I did today:

1. WinForm application with two text boxes (for numeric input) and four buttons representing arithmetic operations.  When a button is pressed, the result of the operation is displayed in a message box.  (This gives faster students the chance to practice control layout, text box masking and other "advanced" things.)
2. WinForm application with a list box, a property grid and two buttons (Create and Update).  Whenever you click the Create button, a new object appears in the property grid.  When you click the Update button, that object is added to the list box.  When you change the selection in the list box, the selected item appears in the property grid.  (All in all, about 5 lines of code they actually have to write, but a cool effect nonetheless.)

And yes, they struggled hard with these exercises; and yes, they have thoroughly enjoyed it.  After being taught C for a couple of sessions and staring at a black console window, the opportunity to actually play with shapes and colors in a GUI environment is highly reassuring.)

P/Invoke Signature Generator

If you ever needed to call a Win32 function from .NET code, you know the dreaded feeling.  You fire up MSDN and start looking for the function signature, hoping it doesn't invent a new pattern of parameter passing, returning a value and calling back a function you specify.  And while you're at it, you still have to figure out what the function is expecting at offset 0x20 from the beginning of the input buffer when the value of the dwExtendedFlags parameter is non-zero.

This tedious process has long ago come to an end with the advent of PInvoke.net, a free Wiki-like developer portal which targets the problem of generating P/Invoke signatures (it also features a cool Visual Studio add-in).  For example, here's the signature for CoMarshalInterThreadInterfaceInStream (one of COM's longest API names):

  That's all fun and games, but it won't help you in two particularly annoying cases:

1. The signature for the API you're looking for is not in the database.  (For example, ConvertSecurityDescriptorToStringSecurityDescriptor or AccessCheckAndAuditAlarm);
2. You're not trying to use a Win32 API, but rather a function in a third-party (non MS) or your own DLL.

For these scenarios, you had to either perform the conversions by hand or resort to old third-party commercial conversion software (I was only able to find the P/Invoke wizard, last updated around May 2004).  I admit, I never tried the latter.  Oftentimes I was frustrated enough to just give up the P/Invoke interop approach and go ahead and write a C++/CLI bridge assembly instead (which has the interesting advantages and disadvantages which might be the subject of a future post :-)).

However, these days are over now.  In the January 2008 issue of the MSDN Magazine, Yi Zhang and Xiaoying Guo have published a CLR Inside Out column on marshaling between managed and unmanaged code.  In that column, they introduce the P/Invoke Interop Assistant, an automatic GUI and command-line utility for converting between managed and unmanaged signatures (in both directions).  This conversion, of course, is not limited just to Windows signatures; give the tool a snippet of your own C header files and it will dutifully convert them to pretty-printed C# [DllImport]'s.

This tool has helped me already, and I plan to continue using it in my projects.  It also has the great advantage of working perfectly even if you're not connected to the Internet.

It takes annoying long to load (about 30 seconds on my laptop), but when it's loaded you can see the signature for most Windows APIs:

And of course translate your own code snippets:

Generating Documentation with Sandcastle

Sandcastle is a free toolset released by Microsoft for constructing professional-looking, MSDN-ish documentation for managed code libraries.  It is also used by Microsoft to construct the .NET framework documentation.  The toolset is currently in its CTP phase (for over a year now), but a community is beginning to form.  The latest release is the October 2007 CTP (look for further announcements on the Sandcastle blog or the Sandcastle CodePlex project).

Here's what the results look like for a sample project I've generated (in HTML Help 1.x format, which produces a CHM):

(Full disclosure: in the past, I've used Document! X, a commercial documentation solution by Innovasys Ltd.  It certainly does the job, although at a fairly high cost - $585 per user.  The GUI used to have some issues, but I suppose they have been resolved in the newer versions.)

Generating documentation using Sandcastle is not very trivial at the moment if you only use the released toolset from Microsoft (this will certainly improve as the project progresses).  Fortunately, there are at least three community projects designed to bridge this gap.  I've decided to use all three and see how relevant they might make Sandcastle, even today.  My test harness was a simple solution with three projects, two written in C++/CLI and one in C# (all .NET 2.0, Visual Studio 2005).  So without further ado:

DocProject

DocProject was the last option I chose to evaluate.  The approach is slightly different and interesting here: instead of using a stand-alone tool for generating documentation, you simply insert a special Visual Studio project into your solution.  That Visual Studio project can now be linked to the outputs of other projects in your solution and automatically generate documentation for these outputs.  Adding that special DocProject project was a snap with a convenient graphical wizard, and the generated documentation was great.

Sandcastle GUI

Sandcastle GUI has a simplistic, grayish UI which you use to select an assembly directory and then let the tool generate the documentation for you.  Unfortunately, Sandcastle GUI was unable to generate documentation for my C++/CLI projects.  That it to say, it did generate the classes, but it didn't add the XML documentation comments I've worked so hard to sprinkle all across the project.  The one thing I did like is the fact the tool didn't require any additional installation.  It can also be run from the command line, which gives you hope if you need to integrate it in your build process.

Sandcastle Help File Builder

SFHB has a simplistic, grayish UI (quite similar to Sandcastle GUI), but it gives you more ability to customize the output and worked just fine with my C++/CLI projects.

Attaching a Workflow Instance to a Workflow Service Host

As part of WF and WCF integration in .NET 3.5, Workflow Services give us the ability to expose workflows as WCF services (including activation) and to consume WCF services from within workflows.  Additional interesting features include explicit context management support, conversation support (multiple parallel send-receive pairs in a workflow), durable services and other interesting issues.  (Check out the demos in the Visual Studio 2008 Training Kit, specifically the Demos\03 - WF directory.)

This model seems really cool because it's enough for me to put a ReceiveActivity as the first activity of my WF, set the CanCreateInstance of that activity to true, and create a WorkflowServiceHost for that workflow type.  The details of activation, message dispatching, correlation (context management) are taken care of by the infrastructure.  (By the way, I have used a custom implementation of almost all of these features first-hand just after the initial .NET 3.0 RTM, so I can't be blamed for being biased in favor of Microsoft's implementation.)

One thing that was immediately missing to me is the ability to create a workflow manually (i.e., not in response to a WCF service call) and then associate it with the WorkflowServiceHost instance.  So basically I want to create the workflow myself in response to some external trigger, but from that point on the workflow can use ReceiveActivity without limitations (which means it must be associated with the WorkflowServiceHost).  Consider the following workflow:

Sequential Workflow
    ReceiveActivity (CanCreateInstance = false) from IMyWorkflow.Start
    ReceiveActivity (CanCreateInstance = false) from IMyWorkflow.End

So what I would like to do is create an instance of that workflow manually (using the WorkflowRuntime obtained from the WorkflowRuntimeBehavior on the WorkflowServiceHost), and then provide the endpoint and workflow instance id so that clients (in the broad sense) can now issue the Start and End operations to the workflow.

By the way, the reason for doing something like that is that I'm constructing a framework for executing workflows which I do not control; to do that, I want to construct the service host dynamically* and execute the workflow in response to my trigger and not in response to a WCF service call over an endpoint that initially is known only to me.

After spending lots of time, I started whipping up an implementation of a custom activity that I would dynamically insert into the workflow, and that activity would contain a ReceiveActivity listening on an internal contract I define; this poses problems because I can't dynamically modify a workflow before it's running, unless that workflow is a XOML-only workflow; so this means that my workflow developers will have to embed that activity in their workflow, YUCK!

Fortunately, there is a way to accomplish what I wanted, and it's phenomenally simple.  Unfortunately, I couldn't find any documentation or reference to it, but here it comes.  Create the workflow yourself using the WorkflowRuntime obtained from the host, and start the workflow (WorkflowInstance.Start).  That's it.  The workflow can now be addressed using the workflow instance id, if you set up the context correctly.  (By the way, it's not that I didn't think of the first step.  What I forgot to do is call the Start method, which is so damn stupid.)

Here's what the code looks like (except for the workflow definition itself):

You can find the sample code here.  It demonstrates setting up the workflow outlined above as a workflow service, but creating an instance of that workflow manually and then communicating with it by manually setting up the context to specify that particular instance as a GUID.  (Note that the "instanceId" in the example is case-sensitive.)

* This in itself is interesting because you need to add an endpoint to the host for each service contract your WF is using a ReceiveActivity on.  To do that, I created an instance of the workflow (as a template) and recursed over all composite activities looking for ReceiveActivity instances (from which you can extract the contract information using the ServiceOperationInfo property).