Dysfunctional Pointers, Part 3: Why Can’t They Just Get Along?

In most languages the question is whether functions are first class objects or not. In C, they aren’t objects at all.

We’ve already seen that, in C, pointers to functions and pointers to objects (including void *) are different animals, and trying to convert one to the other results in undefined behavior. And we’ve seen how this can sometimes be mildly inconvenient. But why is this peculiar rule in place, anyway? At the hardware level, code and data are just bytes in memory, right? That’s why things like stack smashing are a problem, because a pointer to data bytes can be interpreted as a pointer to code bytes.

It is true that on the computers most people1 typically encounter today code and data bytes live side by side in the same address space. But it isn’t always so. Some computers expect separate address spaces for code and data. And we don’t have to look very hard to find an example: an x86 processor running in one of its segmented modes will do nicely.

In real mode and 16-bit protected mode the x86 treats memory as segments up to 64KB in size.2 It’s possible to run an entire program out of a single segment if it’s code and data are small enough to fit in 64KB.

But many programs are bigger than that and, at the very least, have to keep their code and data in separate segments. The processor makes this division into code and data very natural, since it keeps the reference to the current code segment in a register, and provides registers to refer to other segments for data manipulation. I already gave a rundown on the conventional memory models used for 16 bit programming on the x86 and the important ways that they differ from one another, here.

The important points for the matter at hand are that:

  1. In the mixed memory models (medium and compact) code pointers are 16-bits by default while data pointers are 32-bits, or vice-versa.
  2. In the small memory model, where both code and data pointers are 16-bits by default, the pointers are relative to different segments. This means that code and data pointers in general point to different things, even if they have the same bit pattern.

Translating this into C-speak, this means that sometimes a pointer to a function and a pointer to an object have different sizes in memory (16-bits versus 32 bits). It also means that, even if function and object pointers have the same size, their values may refer to different address spaces and so can’t usefully be converted back and forth or compared.

It’s hard to see exactly what a C compiler could sensibly do in a situation like this if asked to convert a function pointer to an object pointer. And that’s precisely why the standard says that performing that conversion results in undefined behavior (Harbison and Steele, 2002, pp. 185-187).

Undefined behavior leaves the door wide open: on a machine where code and data share an address space, it may be possible to perform the conversion without any drama. But the standard acknowledges that this cannot hope to be completely portable, because there are platforms out there where the conversion doesn’t even make sense and may result in a warning, an error, a crash, or worse.

References
Harbison, Samuel P. III, and Guy L. Steele, Jr. 2002. C: A Reference Manual. New Jersey: Prentice Hall.

Dysfunctional Pointers, Part 2: Counting the Cost

Last time we discovered that in standard C you aren’t allowed to convert a pointer to a function into a pointer to an object.

The example I gave there didn’t do anything, but here’s an example where you really want to be able to do that, and the C standard leaves you hanging.1

Suppose you have a data structure that stores a pointer to some ancillary data. To make it more concrete, let’s suppose you are writing an interpreter for a programming language. You might have a data structure something like this to represent procedures.

typedef struct PROC {
   char *strName;       /*The name of the procedure.*/
   int nParms;          /*The number of parameters the procedure accepts.*/
   LINE *ptrCode;       /*A pointer to a linked list of data structures, each of which holds a line of the body of the procedure.*/
} PROC;

This structure works fine until you realize that you can get rid of that big switch statement in the main loop of your interpreter that handles primitives, and instead dispatch primitive procedures via a C function pointer in the PROC structure. All you need is a way to tell a PROC with an interpreted body from one with a primitive implementation, and a place to store the function pointer. And to save space you just change ptrCode into a void pointer, since void pointers can point at anything. No need to change the existing code that deals with ptrCode!

typedef struct PROC {
   char *strName;       /*The name of the procedure.*/
   int nParms;          /*The number of parameters the procedure accepts.*/
   enum {PROC_INTERPRETED,PROC_PRIMITIVE} type;
   void *ptrCode;   /*Either a LINE * or an int (*)(CONTEXT *ctx), depending on the value of type.*/
} PROC;

(Code in italics is wrong.)

But of course we’ve blundered straight into the undefined behavior, because a void * is a pointer to a data object, so we can’t stash the pointer to a function there.

Of course there are ways around this. You could just add a separate pointer.

typedef struct PROC {
   char *strName;                  /*The name of the procedure.*/
   int nParms;                     /*The number of parameters the procedure accepts.*/
   enum {PROC_INTERPRETED,PROC_PRIMITIVE} type;
   LINE *ptrCode;                  /*A pointer to a linked list of data structures, or NULL if type != PROC_INTERPRETED.*/
   int (*ptrPrim)(CONTEXT *ctx);   /*A pointer to a function that implements a primitive, or NULL if type != PROC_PRIMITIVE*/
} PROC;

But that wastes a little space and introduces the risk of dereferencing one of those unused pointers by accident. Alternatively, you could use a union.

typedef struct PROC {
   char *strName;                  /*The name of the procedure.*/
   int nParms;                     /*The number of parameters the procedure accepts.*/
   enum {PROC_INTERPRETED,PROC_PRIMITIVE} type;
   union {
      LINE *ptrCode;                  /*A pointer to a linked list of data structures.*/
      int (*ptrPrim)(CONTEXT *ctx);   /*A pointer to a function that implements a primitive.*/
   } impl;
} PROC;

That’s certainly safer from a typing point of view, and expresses exactly what you want to do, but the code you already wrote that deals with ptrCode will have to be changed. (Actually, since C11, even standard C has anonymous unions, so you could make the type declaration without naming the union and the existing code would still work.)

So the problem isn’t insurmountable, but it still seems more than a little quirky that there’s even a problem to surmount. There’s nothing logically wrong with wanting to store two different kinds of pointers in the same piece of memory, as the union alternative shows. We just can’t do it with a void * in this case. But why not — why can’t a void * just point to anything? All pointers are the same, when it comes down to it, right?

Wrong. Next time we’ll see the flaw in that argument, and the reason for the segregation of function and object pointers.

Dysfunctional Pointers, Part 1: Quietly Undefined Behavior

It is hardly an original observation that C is quite a strange language. Apart from the usual suspects, such as the slightly peculiar operator precedence, or the fact that array[index] has the same meaning has index[array], there’s that huge morass of undefined behavior that everyone just tries to forget is there because, y’know, it’s easier to get on with programming without a sense of dread dogging your every move.

And unlike many other languages, which bound along with you like puppies and save you from peril, C often does nothing to save you from running headlong in to the muck. The nasty thing about undefined behavior is that typically compilers don’t have to warn you about, even if they can easily detect it. The C standard (ISO, 1990/93)1 only requires compilers to complain about syntax errors and certain other situations that it explicitly calls out (known as constraints). Otherwise the compiler doesn’t have to do a thing — it can print out a message if it likes, but it doesn’t have to, that’s part of what makes the behavior undefined!

Here’s a case in point that many programmers don’t even know about. C programmers get very cozy with the cast operator (type) to convert values of one type to another. You can even cast pointers of one type to another, something which occasionally comes in handy. For example, if you have one structure that serves as a header in another structure, you’re allowed to a pointer to one structure into a pointer to the other. This is useful if to implement a basic kind of polymorphism

typedef struct HEADER {
   enum {C_DERIVED_1, C_DERIVED_2 }type;
   int sig;
} HEADER;

typedef struct DERIVED_1 {
   HEADER hdr;
   int extra_field;
} DERIVED_1;

typedef struct DERIVED_2 {
   HEADER hdr;
   char extra_string[64];
};
⋮
   {
      HEADER *hptr = getThing();
      if(hptr->type == C_DERIVED_2){
         DERIVED_2 *dptr=(DERIVED_2 *)hptr;
         printf("Your derived_2 would like you to know that %s.\n", dptr->extra_string);
         ⋮
      }
      ⋮
   }

In fact it’s always legal to cast a pointer to one type of object into another type of object, though the result may not make much sense if you try to use it (Harbison and Steele, 2002).

But how about this case.

int everything(void){
   return 42;
}

int main(void){
   char *cptr = (char *) &everything;  /*Dreaded undefined behavior!*/
}

This program, doesn’t really do anything. It should assign a pointer to the function everything (suitably converted via the cast) to the variable cptr. (If it was optimized even that assignment could happily disappear, in principle) But it results in undefined behavior. The reason is that the cast is not allowed. But wait, didn’t I just say that it’s legal to cast a pointer to one type of object into another type of object? I did, indeed. But here’s the thing. A function is not an object.

There are two kinds of “things” in standard C: functions, and objects.2 As a result there are two kinds of pointer to things, and never the twain shall meet. Converting a pointer to one to a pointer to the other results in undefined behavior, one of those undefined behaviors that compilers don’t usually warn you about, even though they could.

I just tried a bunch of versions of gcc and couldn’t coax them to tell me off about this. Visual C++ 2015 issued

Warning	C4054	'type cast': from function pointer 'int (*)(void)' to data pointer 'char *'

but only if I told it to compile as C and disable extensions.

Next time, we’ll a look at why it’s annoying that function and object pointers are segregated in this way, and after that we’ll see why it is that standard C has this seemingly strange rule in the first place.

References
Harbison, Samuel P. III and Guy L. Steele Jr. C: A Reference Manual. 5th ed. Upper Saddle River, NJ: Prentice Hall, 2002.
International Organization for Standardization. The Annotated ANSI C Standard: American National Standard for Programming Languages — C; ANSI/ISO 9899-1990. 1990. Facsimile of the standard, annotated by Herbert Schildt. Berkeley: Osborne McGraw-Hill, 1993.

Getting Files to OS/2 Warp through VirtualBox

There are only really 2 aspects of the VirtualBox Guest Additions that I miss in OS/2 Warp: the mouse pointer integration, and the folder sharing.1

The mouse pointer integration I can live with. I usually run an OS/2 VM full screen on a second monitor, so it already looks quite a lot like I’m running two different physical systems side by side. That makes it seem less onerous to have to press Ctrl if i want to switch from OS/2 back to the host system.

But the lack of folder sharing between host and guest is harder to ignore. If you’re not familiar with this feature of VirtualBox, it allows you to designate certain folders on the host that should be accessible to the guest, which the guest operating system accesses with special drivers. This makes it very easy to transfer files between host and guest.

So what are the alternatives? Well, about the most basic is to use virtual disks, a technique which works with a wide range of operating systems that don’t support guest extensions. To get a few files into an OS/2 VM it’s often enough to create a floppy disk image, transfer files to it (a tool like Mtools may come in handy for this), and then insert the image in the virtual drive. Just reverse the steps to get files out again. For larger or more permanent sharing it is possible to create a virtual hard disk file. This can be a bit more unwieldy to use than a floppy image, but in Windows or Linux it’s possible to mount a virtual disk image in the host to access files while the VM is offline, then access the files in the guest directly when the VM is online.

These solutions are okay, especially when getting a VM up and running. But the constant inserting and ejecting (for a virtual floppy disk; for a hard disk image it’s even more annoying) gets old pretty quick for anything except extremely light use.

So, what are the other options to get an OS/2 Warp VM communicate with the outside world? I’ve got a few ideas that I’ll outline in future posts. If you’ve got any, feel free to share them.

The Long and the Short of Windows

Many of those who have programmed the 16-bit modes of x86 processors often want to forget the experience. Others take it out on the world in the form of in-depth articles on antique programming techniques. But signs of 16-bit programming are still with us, even on 64-bit platforms.

Take the Win32 API. Despite Microsoft’s attempts to brush it under the rug, Win32 is still there. And scattered throughout the Windows header files you’ll find curious type names like LPCSTR. This at-first indecipherable string isn’t (just) the result of pounding the keyboard in frustration. It encodes its meaning carefully. The STR part means string. The C part means constant. And the P means pointer. So what about the L?

L P C STR
? pointer (to) constant string

It turns out that the L stands for long, not in the sense of a C long, but in the sense of a long pointer, or a far pointer, the far pointer of 16-bit x86 modes. For reasons that aren’t clear the Windows headers use the term long where most other sources use large or  far.

In the original 16-bit versions of Windows the headers had to make clear the distinction between far and near pointers. Most pointers passed to Windows were far, since the system had to be clear about which segment a program meant by a given pointer – a short pointer with just the offset within a segment wouldn’t be enough. But the headers helpfully defined just a pointer (which was near), denoted by the type name prefix P and a long pointer (which was far), denoted by the prefix LP.

When the Win32 API was created from the 16-bit API, the L came along for the ride in the name of source compatibility, though the binary interface no longer made any distinction between the two types. And the same thing happened for the 64-bit API. In fact it’s still in there, if you care to look in the SDK header files, complete with definitions in terms of the long-defunct keywords far and near, which in turn are given empty definitions.

And so it lingers on, that little L, long after most have forgotten its meaning. Just another example of what can happen if you apply stringent backwards compatibility and wait 30 or 40 years.

VirtualBox Guest Additions Versus OS/2 Warp

If you’ve been paying attention during the x86 huge pointer series1, you will have noticed that the examples I provided were all run on OS/2 Warp, a.k.a. OS/2 3.0.

OS/2 was a good option in this case, because it was easy to demonstrate the differences, and similarities, of huge pointer handling in a both protected mode and real mode environments (well, virtual real mode, anyway). And version 3 just happens to be the version I have. I run it in a VirtualBox virtual machine, which works quite nicely. The only annoying thing is that you can’t use the fancier guest extensions that VirtualBox provides to other operating systems, like mouse pointer integration, video integration, and shared folders.

Or so I thought, until I actually tried inserting the guest additions CD into the OS/2 machine and, lo and behold, there were some guest additions for OS/2. I decided to give them a shot, and followed along with the manual installation instructions.

Bad idea. It turns out these guest additions are intended for Warp 4 or greater. They rely on a module called KEE, which stands for Kernel Execution Extensions, and isn’t available for regular old Warp. But it was no good trying to explain that to my now-broken OS/2 system. Thank goodness it was just a virtual machine, and could be rolled back to an earlier good state.2

The moral of the story: the VirtualBox Guest Additions for OS/2 don’t work in OS/2 Warp, below version 4, at least as of writing.

A Huge Coincidence

We’ve almost reached the end of the epic tale of huge pointers in 16 bit x86 programs. Last time we saw the parallels between the way Microsoft-style huge pointers work in real mode and 16-bit protected mode.

To recap, suppose that we have a huge pointer seg:off and we want to increment the pointer by n bytes (to access the nth byte, of the object pointed to, for example). Then here’s how to to compute the result, in real mode, and in 16-bit protected mode.

Real mode (e.g. DOS) seg:off + n = (seg + q × 4,096):(r)
Protected mode (e.g. OS/2 1.x) seg:off + n = (seg + q × 8):(r)

Where in both cases q is the quotient, and r the remainder after dividing o + n by 65,536, the size of a segment.

As you can see, there’s really only one formula:

seg:off + n = (seg + q × f):(r)

Where f is a factor that has the value 4,096 in real mode and 81 in protected mode.

This means, as I mentioned last time, that it’s possible to compile code into a form that works correctly in both real mode or protected mode, by plugging in the right value for f at runtime.

Microsoft’s systems take advantage of this in two different ways.

First, let’s look at 16-bit Windows. It isn’t that well known, but versions of Windows up to 3.0 were capable of running in real mode, multitasking and all. This required a lot of cooperation from programs, since there was no hardware memory management, but it really worked.

Since programs written for these early versions of Windows have to work in both real mode and protected mode2 they have to be able to configure their huge pointer factor f at runtime based on the mode the system is using. This all happens behind the scenes for programmers working in high level languages, but you can find out more about how the factor is obtained here.

Sixteen-bit OS/2 also contains an application of the parallel huge pointer computation. The system supports dual mode (also known as family mode) programs (Letwin, 1988), ones written to a subset of the OS/2 API, a subset that was easy to map to regular old DOS functionality.

The idea is that you only need a single executable to run on either DOS or OS/2. When run on OS/2 the program runs as regular 16-bit protected mode program. When run on DOS a miniature abstraction layer built in to the executable file loads the OS/2 program, “links” it to itself, and runs it, with OS/2 system calls mapped to DOS system calls. Again, since the program can run under real and protected mode there has to be a way for it to discover the huge pointer factor f. In this case a function called DosGetHugeIncr does the job.

Dual mode OS/2 programs are nifty, but to be honest I’m more impressed by the application of the technique in early Windows. It’s just one of a number of ways that the design3 attempted to smooth over the (generally vast) difference between real and protected mode.

When we return to huge pointers, I’ll wrap up a few loose ends.

References
Gordon Letwin. 1988.Inside OS/2, Redmond, WA: Microsoft Press.
Matt Pietrek. 1993. Windows Internals, Reading, MA: Addison-Wesley.

Hugely Unreal

Last time we saw how Watcom C handles pointer arithmetic on huge pointers in DOS.

In summary, seg:off + n = (seg + q × 4,096):(r), where q is the quotient, and r the remainder after dividing o + n by 65,536, the size of a segment.

As I said last time, this treats memory like a series of contiguous 64KB segments, with each segment number 4,096 greater than the number of the segment before it. In 8086 real mode that 4,096 is the result of simple mathematics, since seg:0000 points to memory location seg × 16, while (seg + 4,096):0000 points to memory location (seg + 4,096) × 16 = seg × 16 + (4,096 × 16) = seg + 65,536.

But we saw some time ago that the segmented memory model is not limited to programs running real mode on the 8086, as DOS programs do. There’s also a 16 bit protected mode, where a segment numbers are no longer related mathematically to the memory location where the corresponding segment begins.

Instead there is a level of indirection. The segment numbers (known as selectors in Intel parlance) are indices into one of two tables, the local descriptor table, or the global descriptor table (the least significant bit of the selector tells the processor which table to look in). The descriptor associated with the selector tells the processor where the segment begins in memory, and other information, including the size of the segment which still can be no larger than 64KB.1

Now, programs written for 16 bit protected mode have exactly the same problem as those written for real mode when it comes to addressing “huge” objects that won’t fit in a single 64KB segment. At first glance it may seem that the carefully worked out scheme for huge pointers in real mode would not be compatible with protected mode where segments can start wherever they want. But with a little care you can make the scheme above work in protected mode, too.

It doesn’t happen automatically — it requires support from the operating system. The key thing to note is that selectors act as indices into a descriptor table, so there is still serial aspect to the segmented address translation. If we can still arrange to treat (at least a part of) memory as a series of contiguous 64KB segments, then our formula above can still work.

Here’s how to do it. To support a huge object N bytes long, create Q segments, each 65,536 bytes long, where N ÷ 65,536 = Q r R. If R > 0 then create an extra segment R bytes long. Assign the descriptors for these segments to a contiguous range of indices in the descriptor table (it doesn’t matter which one, as long as they’re all in the same table). Doing that ensures that selectors for consecutive segments of our huge object have consecutive indices in the descriptor table. Suppose the index of the descriptor in the table for the first segment of our object is d. Then the descriptor for the second segment of the object will be at d + 1, and so on.

A 16 bit selector contains the descriptor index in its 13 most significant bits, while the least significant 3 bits contain extra information that isn’t important here (including the local/global table indicator). So the selector for the first segment of our object is s = d × 8 + b, where b is whatever those 3 uninteresting bits contain. The huge pointer to the first byte of the object is s:0000. And if we assume that b is the same for all selectors (which is reasonable in the situation we are setting up) then selectors for consecutive indices differ by 8.

Now, put this information together. The byte at offset n in the huge object will be located at offset r in the segment s + q × 8, where n ÷ 65,536 = q r r. Or put a slightly different way, the address of the byte at offset n is (s + 8 × q):r.

Notice the similarity between this formula and the one above for the adding an offset to a huge pointer in real mode.  In fact the only real difference is the scaling factor for the segment modifier: 4,096 in real mode, and 8 in protected mode.2

It turns out that this similarity is why Watcom C (and Microsoft C, more importantly) uses this representation for huge pointers in 16 bit 8086 programs.  Sixteen-bit Windows (Pietrek, 1993) and 16-bit OS/2 (Letwin, 1988), both developed by Microsoft, support this scheme for huge objects natively.  We can confirm this for ourselves. Here’s the program that we used to show how huge pointers work under DOS:

#include <stdio.h>
#include <malloc.h>  /*For halloc.*/
#include <dos.h>       /*For FP_SEG and FP_OFF pointer manipulation macros.*/

int main(int argc, char **argv){
        long huge *startptr = halloc(70000, sizeof(long));
        long huge *midptr;
        long huge *nextptr = halloc(500, 1);

        if(startptr == NULL){
                printf("Not enough memory.\n");
                return 1;
        }

        midptr = startptr + 35000;

        printf("Start pointer: %04X:%04X. Mid pointer: %04X:%04X.\n",FP_SEG(startptr), FP_OFF(startptr), FP_SEG(midptr), FP_OFF(midptr));
        printf("Next pointer: %04X:%04X.\n",FP_SEG(nextptr), FP_OFF(nextptr));

        return 0;                
}

If we compile this again, unchanged, targeting 16-bit OS/2, then here’s the result (run from OS/2 Warp 3.0).

[C:\myprogs\hugeptr]hugeos2
Start pointer: 0047:0000. Mid pointer: 0057:22e0.
Next pointer: 006F:0000.

Notice the small segment parts (compared to the DOS version): these are selectors, not real mode segment numbers. Also notice now the relationship between the start and mid pointers: 0047:0000 + 140,000 = (71):(0) + 140,000 = (71 + 2 × 8):(8,928) = (87):(8,928) = 0057:22E0, because (0 + 140,000) ÷ 65,536 = 2 r 8,928.

But being able to use such a similar representation for huge pointers in real and protected mode is about more than just simplifying compilers that target both modes. It turns out that once you’ve established the similarity you can exploit it to write programs that run, unchanged, in either mode.

References
Gordon Letwin. 1988.Inside OS/2, Redmond, WA: Microsoft Press.
Matt Pietrek. 1993. Windows Internals, Reading, MA: Addison-Wesley.

The Other Representation For Huge Pointers

Last time we saw that huge pointers in a DOS program compiled with Watcom C are not normalized like they are in programs compiled with Turbo C. Turbo C makes sure that pointer arithmetic on a huge pointer results in new pointer with as much of the pointer in the segment part as possible, limiting the offset part to the range 0 to 15. It uses this representation to make pointer comparisons simpler, as we saw the other day.

But normalizing pointers the way Turbo C does it is not the only way to make comparisons simpler. The easiest kind of comparisons to do on an 8086 are integer comparisons, and we want to take advantage of that when comparing huge pointers.  But huge pointers are also, well, pointers, and we want to make sure they still work as such.

Put a bit more clearly, the conditions we want our representation to meet are that:

  1. when a huge pointer is interpreted as a segment:offset, it points to the object we expect it to; and
  2. when two huge pointers a and b are treated as 32 bit integers A and B, A < B if and only if a points to a memory location that comes before b, and A = B if and only if a points the same memory location as b.

The Turbo C-style normalization meets the conditions. But let’s consider another way of doing pointer arithmetic on huge pointers that also satisfies the conditions. To increment a huge pointer, just add one to its offset part. If the offset part wraps around to zero, (i.e. it overflows), then add enough to the segment part to make sure the result points to the byte after the byte that the original pointer pointed to. On the 8086 this value is 4,096, or 0x1000. For example, suppose we add one to the huge pointer 1234:FFFF. 1234:FFFF + 1 = 2234:0000.

Adding n to a huge pointer has the same result as incrementing it n times. But you can work that out more quickly with some division. If you want to work out seg:off + n then the result is (seg + q × 4,096):(r), where 65,536q + r = o + n and r < 65,536. That is, q is the quotient, and r the remainder after dividing o + n by 65,536, the size of a segment.

If you take the output of the Watcom C program from last time, you can see this at work. Recall that the program adds 35,000 to a huge long pointer. A long is four bytes long, so the effect is advance the pointer 140,000 byte locations.

C:\MYPROGS\HUGEPTR>hugedos
Start pointer: 21C0:0000. Mid pointer: 41C0:22E0.
Next pointer: 661D:0000.

If we start with 21C0:0000 and add 140,000, then by the formula above we should end up with 21C0:0000 + 140,000 = (8,640):(0) + 140,000 = (8,640 + 2 × 4,096):(8,928) = (16,382):(8,928) = 41C0:22E0, because (0 + 140,000) ÷ 65,536 = 2 r 8,928.

You can define a similar procedure for subtracting from a pointer.

This representation for huge pointers treats memory as a series of contiguous 65,536 byte segments, where each segment’s number is 4,096 higher than the number of the previous segment. Note that the result of pointer arithmetic is highly dependent on the pointer you start with. Next time, we’ll see how that can be turned to our advantage if we venture beyond real mode.

Hugely Abnormal Pointers

Last time, we saw how how a DOS C program using huge pointers behaved, at least when compiled with Turbo C 2.01. In particular we saw the result of arithmetic on a huge pointer results in a normalized pointer, one where the offset part is in the range 0-15, and everything else is in the segment. part.

C:\MYPROGS\HUGETC>hugetc
Start pointer: 0FD1:0008. Mid pointer: 31FF:0008.
Next pointer: 542E:0008.

Yet if we wrangle the program from yesterday a little to switch one set of non-portable functions and macros for another, very similar set, we get the equivalent program suitable for an appropriate Microsoft C compiler.

#include <stdio.h>
#include <malloc.h>  /*For halloc.*/
#include <dos.h>       /*For FP_SEG and FP_OFF pointer manipulation macros.*/

int main(int argc, char **argv){
        long huge *startptr = halloc(70000, sizeof(long));
        long huge *midptr;
        long huge *nextptr = halloc(500, 1);

        if(startptr == NULL){
                printf("Not enough memory.\n");
                return 1;
        }

        midptr = startptr + 35000;

        printf("Start pointer: %04X:%04X. Mid pointer: %04X:%04X.\n",FP_SEG(startptr), FP_OFF(startptr), FP_SEG(midptr), FP_OFF(midptr));
        printf("Next pointer: %04X:%04X.\n",FP_SEG(nextptr), FP_OFF(nextptr));

        return 0;                
}

I don’t happen to have a Microsoft compiler of that vintage around, but it turns out to compile quite happily with Open Watcom 2.0 as well, which suits my purposes almost as well. Here’s the result of running it, again in a DOS box in OS/2 Warp 3.0.

C:\MYPROGS\HUGEPTR>hugedos
Start pointer: 21C0:0000. Mid pointer: 41C0:22E0.
Next pointer: 661D:0000.

Okay, so the memory blocks are in different places compared to the Turbo C version, and the offsets returned by halloc are both zero, rather than eight, like the ones from farmalloc in the Turbo C version. But that’s just irrelevant detail. 1 The important thing to notice is that the value of midptr is not normalized like the Turbo C version. Like everyone2 says that it should be.

Notice that it’s not wrong. Each long takes up 4 bytes, so adding 35,000 to a pointer to a long should advance the pointer by 35,000 × 4 = 140,000 bytes. And if we convert the printed values of midptr and startptr to linear memory location numbers and find their difference, it turns out to be (0x41C0 × 16 + 0x22E0) – (0x21C0 × 16 + 0x0000) = (0x43EE0) – (0x21C00) = 0x222E0 = 140,000, which is what it should be.

So it’s not wrong, it’s just not normalized — not the same way Turbo C does it, at least. But if Watcom C is not representing huge pointers the way that Turbo C does it, how is it representing them, and why? Stay tuned to find out.