Asaf Shelly

When to use the IsBadXxxPtr Win32 API

AsafShelly — Sun, 17 Jan 2010 14:56:00 GMT

This post is a follow-up for two previous posts. The first is my own called Verifying C++ Buffers which was replied with a second post by Sasha titled IsBadXxxPtr Is Really Harmful – Please Don’t Use These Functions. The first post mentioned the advantages of using IsBadXxxPtr API over verifying pointers by using ASSERT (or comparing to zero). The second post mentioned the problems with this set of API in a multi-threaded application. Just to clarify my comment here is not personal and Sasha and I are having an academic discussion which will benefit us all.

Let me start by saying that the information provided in the first post is valid and works well with a single threaded application. As Sasha explained the API does not consider parallel operations. This is noted on the MSDN page for the API and at the end of my first post. The reason I did not elaborate more on this is because it is not such a clear cut. Sasha has explained reasons not to use the API in a parallel environment and I will try to argue the reasons for using this API in a parallel environment.

Here are the scenarios I will be examining:

Scenario A: The pointer value is zero / NULL. This is usually because we initialized the pointer correctly or because a structure was never initialized in debug compilation

Scenario B: The pointer value is above zero but is very close to zero, for example address 1, 23, or 32K. This is usually because we are accessing a member of a struct which address is NULL. The address of the struct is zero so the address of a member is its offset from zero. Another possible reason for this scenario is an item in an array when the array is NULL.

Scenario C: A random number is used for the pointer. This is because we are using an uninitialized memory (in the stack for example), or because of buffer over-run / buffer under-run which caused memory corruption in the storage of the pointer.

Scenario D: The pointer is initialized correctly but the buffer was already deleted and therefore the pointer is pointing at an invalid buffer.

ASSERT or IsBadXxxPtr?

When we use ASSERT or test the buffer whether it is NULL or not we are actually testing to see whether or not we are using a buffer that was initialized to zero. This only works if the pointer was correctly initialized. Some functions would not expect NULL, such as printf( ) or puts( ), while other functions may consider this as a valid value for example the UserContext parameter when creating a thread - if it is NULL then there is no context.

Going over the scenarios comparing the pointer to zero or using ASSERT: Scenario A works fine. Scenario B will not find the bug and your function will try to access the buffer. Scenario C: same as B, and it is also the same for Scenario D.

Testing the same scenarios with IsBadXxxPtr: Scenario A works fine. Same for Scenario B. These two scenarios happen even if pointers are correctly initialized! Scenario C will work if the random value happens to fall on a Heap page out of the 4GB virtual space (pages are usually 4K for 32 bit systems). Scenario D will work if the buffer was the last on that page. When the last page on the buffer is deleted the page is marked as inaccessible.

When do we see problems with multi-threading?

Unmapping a page in a second thread: This means that one thread deleted an object while another thread was working on it. This is because you shared the resource (buffer) but forgot to lock or used the locks incorrectly. ASSERT will never detect that because the pointer has an address that it more than zero and ASSERT has no way of telling if the page is valid.

IsBadWritePtr corruption of memory: is because a resource was shared but the thread used the lock only after calling IsBadWritePtr because the programmer did not know that IsBadWritePtr is accessing the memory. This is a problem and you should know that calling IsBadWritePtr will modify memory even though the name suggests that it is only testing a pointer.

Guard Pages are used in an internal implementation which does not expect IsBadXxxPtr API to access these pages. This may happen for example when a thread is sharing a buffer which is located on its own stack, which is a terrible bug. Such a bug will randomly damage Guard Pages, or corrupt the other thread's stack, or cause the other thread to access bad memory (by overwriting pointers located in the other thread's stack). I can't tell you which is better, but if your thread found an invalid pointer of an internal buffer then something is very wrong and you can't find this using ASSERT.

There is a long and painful argument whether exceptions are good or bad. In a multithreading environment I consider exceptions to be always bad because an exception is a bursting Event that will be initiated even when you are in some execution flow. Multi-threading environments are all about synchronizing execution and flow control. The possibility of a flow breaking because of an event is really bad. It is a system wide problem if a thread is using a Semaphore object and then stopping in the middle of work because of an exception because some other thread did something wrong 5 minutes ago. It is simpler to verify pointers than to check all possible exceptions. An exception is something special, not a return value. When exceptions became return values functions started swallowing them and hide the real problems.

For conclusion, with all problems that come with using the IsBadXxxPtr Win32 API, I would still prefer using this API set than using ASSERT which allows bugs to happen. If the code has no bugs then IsBadXxxPtr causes no damage. If your code has bugs then using IsBadXxxPtr will solve more problems than cause them, unless you are going to use this to hide the problems.

If my application supports plug-ins then I would prefer to verify interface buffers than to let buffers from an external module damage my application.

Visual C++ Compiler ERROR C2016

AsafShelly — Tue, 29 Dec 2009 15:58:00 GMT

Compiling a C file with Visual Studio 2008 compiler I got this error:

"error C2016: C requires that a struct or union has at least one member"

I am writing this post because it is not documented, and C2016 on MSDN is defined as "A newline character was found before the closing single quotation mark of a character constant". Either it is not well documented or I don't know where to look for it. Whatever the reason I can only guess that someone else will find the same problem.

This is the cause:

struct
{
   int    u8Type;
   int    u8LenExtended;
   int   u16Length;
} Base_Request;

struct Request_Ex
{
   Base_Request Request;
   int Address;
   int Length;
};

This is the fix:

typedef struct
{
   int    u8Type;
   int    u8LenExtended;
   int   u16Length;
} Base_Request;

struct Request_Ex
{
   Base_Request Request;
   int Address;
   int Length;
};

This is over simplifies intentionally.

Asaf

WinUSB .Net Component

AsafShelly — Tue, 29 Dec 2009 07:43:00 GMT

I have uploaded a WinUSB Component for .Net with full source code here: WinUSB .Net Component in C#.

The component comes with full source code and has the following features:

Pacific Software & HPC

AsafShelly — Sun, 27 Dec 2009 14:02:00 GMT

מזה מספר שנים שחברת פסיפיק תוכנה מובילה בתחום הפיתוח המקבילי בארץ. בקרוב מיקרוסופט מוציאה את גרסת Server 2008 HPC R2 ופסיפיק מחפשת מומחי IT שמתעניינים בתחום הקמה ותחזוקה של מערכות HPC

אסף

Verifying C++ Buffers

AsafShelly — Sun, 27 Dec 2009 08:12:00 GMT

This post is following this post: Verifying Pointers in C++ which demonstrated the problem in testing a pointer for NULL after allocation like this:

char* ptr=new char[1];
if (NULL == ptr) // fail

There is another issue with verifying pointers for NULL. The value NULL is #defined as 0 (zero) which is agreed to be an invalid memory address. Testing a pointer to see if it is NULL means testing to see if the pointer is pointing to address zero or not. As we all know, when we allocate a buffer and then delete it we must zero the pointer. Verifying a pointer is relying on the fact that every pointer that does not point to a legal buffer is manually reset. Like this:

main()
{
   char* ptr = NULL;
   MyFunc(ptr);
   ptr = new char[20];   MyFunc(ptr);
   delete[] ptr; ptr = NULL;
   MyFunc(ptr);
}
void MyFunc(char* ptr)
{
   if (NULL == ptr) return;
   // ...do something...
}

You may assume that verifying the pointer in MyFunc is fine because the code is verified to make sure that every pointer is correctly initialized to zero and so if a NULL pointer is used the function is protected from it. This is not really the case. To demonstrate the problem we can look at the following scenario.

We have a class called Label. This class has several members including the text string to display. Here is the class:

class Label
{
protected:
   int X;
   int Y;
   char* text;
public:
   char* Text();
};

void UpdateDisplay()

{
// ... more
Label* label1 = GetSomeLabel();

   char* ptr = label1->Text();
   if (NULL == ptr) // ERROR
   // ... more
}

Now, here is the problem:

class Label
{
protected:
   int X;
   int Y;
   char text[256];
public:
   char* Text();
};

In the first version of the class the string was a pointer to a memory buffer. In this version the string is stored as part of the object. This means that in the original version the call to GetSomeLabel() in function UpdateDisplay() can return NULL if there is no label and calling the member label1->Text() will also return NULL. When we use the second version of the class we find a problem. If the call to GetSomeLabel() in function UpdateDisplay() returns NULL then the calling the member label1->Text() will return 8. Verifying the pointer for NULL will find no problem and using the macro ASSERT will also no find the bug!

void UpdateDisplay()
{
// ... more
Label* label1 = GetSomeLabel(); // Got label1 = NULL // label1 = 0

   char* ptr = label1->Text();     // ptr = label1 + 8 // ptr = 8
   if (NULL == ptr) // ERROR       // can't see the problem, ignoring
   // ... more
}

When label1 is NULL the data of label1 starts at address zero; X starts at address zero, Y at address 4 which is sizeof(X), after X, and text starts at address 8. Making sure that the string text is not zero is a bug.

When we try to access the buffer text the system will get an exception. This is a CPU exception for accessing the invalid page stating on address zero. If the page size is 4KB then accessing anywhere within the range is invalid and not just address zero, so address 8 is also invalid because it is in the same page. Actually the entire block of memory ranging from address 0 and up to 64KB is marked as inaccessible for this exact reason.

The way to do this correctly is not to verify that the address is under 64K. Instead you can use the Win32 API IsBadxxxPtr. Just like checking the pointer has to be thread-safe, so does calling these API functions. It is possible that between the verification and the usage some other thread deleted the buffer if you do not make sure that this will not happen. Also this does not replace the need to reset pointers to zero after deleting them. If there is another valid buffer on the same page then the entire page is valid.

* Note that this API is not recommended by Microsoft for a parallel environment.

For code example see the wrapper code as it is used internally by the cpp files in this sample code.

WinUSB Read Problem

AsafShelly — Tue, 22 Dec 2009 13:35:00 GMT

Hi All,

I have decided to move forward from my implementation of BulkUSB.sys driver to using the formal implementation called WinUSB. It is a package of a kernel driver and a user-mode API documented as part of the MSDN library.

When I tried to do asynchronous reads I discovered two undocumented / confusing failures:

The first is GetLastError returns 997 - ERROR_IO_PENDING. This is fine for overlapped IO. The function fails and the error code means that there is more data to follow.

The second is somewhat confusing: GetLastError returns 121 - ERROR_SEM_TIMEOUT "The semaphore timeout period has expired". This means that the timeout has expired before the data was ready. The confusing part is that it looks like the call to WinUsb_ReadPipe will also fail with error code 121 if there is data in the buffer.

The read function will wait for the buffer to be full or timeout to expire. This means that the call to read will fail with error code 121 even if some data was read from the device and you must check the number of bytes read to know if read really succeeded or not. The logic behind this is that you don't call GetLastError if the function returns true.

* to set enpoint timeout use WinUsb_SetPipePolicy

Parallel Computing Tutorial

AsafShelly — Tue, 22 Dec 2009 09:18:00 GMT

With some delay I am publishing a video about parallel programming. This tutorial is loaded with advanced concepts mainly dealing with parallel software design and architecture.

As some of you may know I have presented "Parallel Programming for Embedded" TechEd 2009 Europe. This is the video. There are no code samples in this presentations because there are too many languages and operating systems that it is irrelevant to show any code. Instead this presentation shows how the hardware has always been parallel and therefore it is a good source for parallel design. Windows Kernel has always been working in parallel as well, but Windows NT kernel is an embedded operating system and has its own compiler.

The concepts in this video are covered in over 20 days of lectures that I have so it is packed with many advanced concepts.

The video is currently at the home page of http://AsyncOp.com. If you can't see it there then look for it under the Video section, or under Events.

Asaf

Verifying Pointers in C++

AsafShelly — Tue, 22 Dec 2009 08:49:00 GMT

It is well known that C++ is having a few problems with pointers...

When we used C style malloc( ) and free( ) we verified the returned pointer. If the pointer was NULL (zero) then it was invalid and should not be used.

Going to C++ we have constructors. A constructor does not have a return value so the only way to notify the caller that the construction failed was using an exception. If my object is using an internal buffer and the system is out of memory then my constructor has to throw an out-of-memory exception.

The caller who created my object using the new operator will have to catch the exception. This is the only way to know that the creation of the object had failed. Constructors cannot return NULL because an object can also be created in the local scope and then there is no return value.

All this means that the following is a logical error:

MyClass *ptr = new MyClass(12);

if (!ptr) return(ERROR_CODE_FAIL);

There is an error because C++ will make sure that an exception is thrown in case of an error so the code flow will break before the second line and therefore the second line will only execute if the pointer has a valid value.

Here is one way to do this correctly:

MyClass *ptr = null;try { ptr = new MyClass(12); } catch(...) { ptr = NULL; }if (!ptr) return(ERROR_CODE_FAIL);

You can also look for the reserved word nothrow which is a Microsoft Visual C++ extension.

http://AsyncOp.com

Open Source Libraries for windows

AsafShelly — Sun, 20 Dec 2009 07:51:00 GMT

Hi All,

As you can see I have upgraded the shared-source engine (see my previous post http://blogs.microsoft.co.il/blogs/asafshelly/archive/2009/04/21/sharing-my-source.aspx)

The Open-Source section in www.AsyncOp.com now has the complete collection of modules and C++ classes that I have been using for Windows. These include threading and shared memory classes, string classes, communications, graphics and more. Not exactly MFC style classes but will work with MFC.

The code is compatible with MFC, Borland C++, Managed C++, and will also work with native Win32 API.

For example the text class can open a web-page and read from it using >> operators. It can also find an HTML title by using wPage("<head>","</head>")[1]("<title>","</title>")[1]

You are welcome to use the code, parts of the code, and the dll as a library.

http://www.asyncop.com/MTnPDirEnum.aspx?treeviewPath=%5bo%5d+Open-Source%5cWinModules

Asaf

TechEd 2009 Europe

AsafShelly — Thu, 05 Nov 2009 21:22:00 GMT

Hi All,

If you are coming to this year's TechEd in Europe let me know about it.

You are also welcome to join my session on Friday titled "Parallel Programming for Embedded". The session covers parallel computing in general and in particular for embedded systems. The most important part is that it is using a language that infrastructure people can relate to. If you ever used DOS, Device Drivers, low level C and C++, and basic Win32 API then you should come. The session uses this to your advantage. On the other hand web developers will fins this eye opening.

The hardware was always parallel and therefore the kernel environment has always been fully parallel, and guess what, It is working.

Best Regards,

Asaf

A book about R&D - Need your help

AsafShelly — Thu, 08 Oct 2009 19:25:00 GMT

Hi all,

There are many books about C\++, C#, VB, Java and other languages. There are also books about MFC, WPF, Windows Networking, and many other technologies.

All these books teach us the language features and what the different technologies have to offer.

I haven't seen a single good book that will teach me how to effectively do my job in R&D. I still remember my first job in a hi-tech company: I was given a collection of technical books. After I read all these books I still didn't really know what I was expected to do and how to do my job. I went to the big book stores in Israel looking for a book about R&D, and guess what... I am still looking for that book.

Don't you agree with me? I would really like to read such a book and especially be able to give a book about R&D to new employees.

What do you think?

תכנות מקבילי - אירוע חד יומי

AsafShelly — Thu, 03 Sep 2009 05:25:00 GMT

באירוע ה- TechEd האחרון העברתי הרצאה בנושא עבודה מקבילית עם C#. בתקופה ההיא הנושא עוד היה חדש וזאת היתה ההרצאה היחידה. ההענות היתה טובה מאד ולמרות שההרצאה היתה בשעה 8 בבוקר האולם היה מלא עד אפס מקום ובתמונות ניתן לראות את המסדרון מחוץ לאולם מלא עד שאין מקום עמידה (צפיה בתמונות). מכיוון שהיתה היענות כל כך חיובית היו מספר אירועים נוספים שעקבו בניהם יום פתוח בחסות מיקרוסופט ומספר רב של אירועים אחרים. עם הזמן עברתי להוביל את התחום בחברת פסיפיק תוכנה, הקמנו שישה קורסים שונים עם סך הכל של 20 ימי הדרכה בנושאים שונים.

האירוע הבא יתקיים ב- 24 לחודש ספטמבר. מדובר בכנס חד יומי שמטרתו מתחלקת לשני חלקים: 1. לאפשר למתכנת להתחיל לעבוד בעולם המקבילי בצורה נכונה ובאופן מיידי, ו- 2. להבין את התחום, מה כדאי למקבל ומה לא, להבין את השפה שבה משתמשים בעולם המקבילי, ולדעת מה בעתיד - למשל המעבד החדש של אינטל (שעוד לא יצא) ימקבל אלגוריתמים בצורה שונה לחלוטין וכך למשל חיפוש ומיון יעבדו משמעותית יותר מהר מאשר היום, אם הקוד כתוב נכון.

הידע שמועבר בכנס מבוסס על הקורסים שאנחנו מעבירים, על בלוגים (בעיקר של אינטל) וחומר מהקהילה, ועל מצגות שיווקיות וטכניות של חברות בתחום. ההרצאות עונות על אוסף השאלות שעלו במסגרת הייעוץ וההדרכות שהעברנו לחברות בארץ בשנים האחרונות, החל בשאלות של מוטורולה בנושא עבודה מקבילית של בקרי תקשורת חומרתיים, הצורך של צ'ק פוינט להתאים שכבות פיתוח מעל מערכות הפעלה שונות, ועד HPC וארכיטקטורה תומכת SOA. אותן השאלות חוזרות שוב ושוב בתחומים השונים. עבודה מקבילית היא לא תחום בתכנות אלא שיטת עבודה שמשפיעה על כל תחומי הפיתוח.

פרטים נוספים בקישור: http://www.pacificsoft.co.il/course.asp?p=671&s=1255

חלק מעקרונות המפתח בהרצאות מבוסס על חומרים שפרסמתי בבלוג של אינטל: http://software.intel.com/en-us/profile/326676/

נשמח לראותכם שם.

אסף

ותודה רבה לאורי

AsafShelly — Tue, 21 Jul 2009 20:35:00 GMT

בין כל הפוסטים הטכניים הכבדים היה נראה לי ראוי ונכון לדבר על דברים קצת יותר שמחים.מי שהגיע לחתונה שלנו בערב ל"ג בעומר הופתע לשמוע את הזמר אורי פיינמן שר את הסלואו ומוסיף כמה מילים אישיות. מה שלא ידעתם הוא שהשיר שלצליליו צעדנו לחופה היה שיר חדש שעוד לא פורסם, או בעצם יותר נכון פורסם רק השבוע. דואט מרהיב שמשלב את אורי פיינמן ומורן אהרוני בשיר אחד.

ליאור צורף לימד אותי שפוסט טוב כולל תוכן עשיר כמו וידאו, אז הנה קטע מהחופה:

למי שרוצה לשמוע את כל השיר - קיבלתי לכך אישור מיוחד "מההפקה":

תהנו. ואה, כן, אורי - תודה רבה על הכל. אני יודע שהתודה קצת מאוחרת אבל אפילו על התמונות של החתונה עוד לא עברנו...

פרטים נוספים על השיר והדיסק החדש תוכלו למצא באתר הרשמי של אורי פיינמן http://uri-fineman.com/ ותתפלאו לשמוע שאת ה- Flash באתר אורי תכנת בעצמו! (ליאור צורף, האם בשנים שאתם מכירים ידעת את זה על אורי?)

Sharing My Source

AsafShelly — Tue, 21 Apr 2009 13:24:00 GMT

Hi all,

As some of you may know I have a huge collection of applications and code samples. I don't have a degree so all my studies were by designing and writing tools and projects. It was clear to me for some while that I should publish this code. I even tried to do this once before but it is not so simple. The code looks bad when it is viewed as a simple text file over the web. Coloring is problematic because you need to deal with it over and over again whenever you have a bug fixed.

Bottom line is that I decided to finally do it. It took me less than 12 hour of work to produce a text parser that produces HTML files for c/++/# files. It is only appropriate (and simple :) to share the source code of this generator as the first project that I share.

http://asyncop.com/Link.aspx?Open-Source

I will eventually find the time to produce some management system so I can add new projects without too much trouble.

The project publisher tool has three basic elements:

A C# library with ProjectPublisher.cs which is the engine, a C# WebService that uses the library with the source file ProjectPublisherWSvc.asmx.cs, and a simple web page demonstrating the use of the WebService called Default use of ProjectPublisherWS.aspx.cs.

The source files are parsed on demand so the original file on the server is the clean source file. This way I can update the sample by copying the new C\++\# file using the FTP folder. I am assuming that there won't be more than one request per second and if there is then I can always turn on the cache on the firewall.

The WebService is open to the public so you can simply send a string read from a source file and the server will return an HTML file. The reserved words are in arrays and the color tagging are members to a class. It should be easy to add API that receives the list of reserved words and color schemes.

WebService URL is: http://services.asyncop.com/ProjectPublisher/ProjectPublisherWSvc.asmx

Let me know if you find any bugs. This is still only 12 hours of work for coding, designing, debugging, testing, and integration with the AsyncOp website...

Regards,

Asaf

World of Parallel Computing

AsafShelly — Sun, 05 Apr 2009 12:46:00 GMT

I was reading Sasha's blog post about parallel computing and went over the presentation slides.

http://blogs.Microsoft.co.il/blogs/sasha/archive/2009/04/04/concurrent-programming-msdn-event.aspx

It looks somewhat different than mine, even though I can only assume that the goals and the audience were pretty much the same.

http://blogs.Microsoft.co.il/blogs/asafshelly/archive/2009/04/05/video-introduction-to-multicore-programming.aspx

This is usually because 'we are not there yet'. Meaning, we don't really have all the answers. If you ask someone about UML you will usually receive the same answers. This is because everyone is in agreement about the fundamental issues.

This is not the case with parallel computing.

The first thing that comes to mind is that most people either do Procedural C programming or Object Oriented programming and design. Sasha mentions that working in parallel takes another big leap, just like the one from C to C++. These are two completely different ways of thinking.

I take this concept one step further to say that: If we need to move forward from what we know - Procedural and Object Oriented Programming, then this means that both are old and out-of-date. If we move forward then this has to mean that what we have today is wrong! Doing Task Oriented Programming with OOD is as bad as doing OOP with C.

Most architects today are used to working with C++ and C#. Most Realtime programmers are used to C. OOP is very good for managing large fragments of code where as Procedural C is very good in managing flow. In Procedural programming a function is used to implement a Procedure. A Procedure is a process or something that happens and has a flow. In OOP classes are used to represent objects.

What we are looking for is something completely different.

We will still be using classes and functions but now it will not be for procedures and objects. It would be for tasks.

Welcome to the new world of programming.

Asaf