When we last left off, we were able to cook up a service name of our choosing, and resolve it from a client. If we hooked up launchd stuff, we could also make it demand-launch (maybe that’s for another day). But we didn’t actually do anything with that resolution. There are several reasons for that, first, because actually crafting a message and sending/receiving it was covered by a prior article. But even more than that, there are actually a substantial number of design decisions around this task, and plowing ahead requires a lot of text.
I’ll start with the previous server/client, and flesh it out in small steps. In between the steps, I’ll outline design decisions, maybe some history, and pitfalls/sharp edges to be aware of. If we take the previous example and add some extra bits, we can get simple message passing up and running (note that I don’t put anything in the message, it’s just to show that it gets there — we’ll get to jamming in bits later).
#include <bootstrap.h> #include <mach/mach.h> #include <stdio.h> #include <unistd.h> int main(const int argc, char **argv) { printf("bootstrap port: %d\n", bootstrap_port); if (argc == 1) { // "server" mode mach_port_t service_port = MACH_PORT_NULL; kern_return_t kr = 0; // should fail kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); kr = bootstrap_check_in(bootstrap_port, "com.example.test", &service_port); printf("service_port: %d (%x)\n", service_port, kr); // should succeed kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); printf("server loop\n"); while(1) { struct message { mach_msg_header_t header; mach_msg_body_t body; mach_msg_type_descriptor_t type; mach_msg_trailer_t trailer; } message; kr = mach_msg((mach_msg_header_t*)&message, MACH_RCV_MSG, 0, sizeof(message), service_port, MACH_MSG_TIMEOUT_NONE, MACH_PORT_NULL); printf("we get message: (%x) - size: %d\n", kr, message.header.msgh_size); } } else { // "client" mode mach_port_t service_port = MACH_PORT_NULL; kern_return_t kr = 0; kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); struct { mach_msg_header_t header; mach_msg_body_t body; } message = {0}; message.header.msgh_remote_port = service_port; message.header.msgh_size = sizeof(message); message.header.msgh_bits = MACH_MSG_TYPE_COPY_SEND; kr = mach_msg((mach_msg_header_t*)&message, MACH_SEND_MSG, sizeof(message), 0, service_port, MACH_MSG_TIMEOUT_NONE, MACH_PORT_NULL); printf("message send %x\n", kr); } return 0; }
Server says:
$ ./service bootstrap port: 1799 looked-up service_port: 0 (44e) service_port: 2819 (0) looked-up service_port: 2819 (0) server loop
Client says:
$ ./service -client bootstrap port: 1799 looked-up service_port: 4611 (0) message send 0
Server responds:
we get message: (0) - size: 28
Ok, so simple message sending/receiving is up and running. And if we try the client when the server isn’t running, we get an error:
$ ./service -client bootstrap port: 1799 looked-up service_port: 0 (44e) message send 10000003
Where 10000003 means MACH_SEND_INVALID_DEST, as we expect (the looked-up service port failure means we don’t have a port to send the message to).
This is all well and good, but it suffers from a pretty serious problem, and that problem is scalability.
The server is parked waiting for this one port to get messages. We could specify a timeout to cycle through a set of ports, or we could spin up a thread per port of interest, but both of these scale poorly too – timeouts wake the CPU, which eats a surprising amount of power, and lots of threads consume various kernel resources (more ports, stacks, thread states, scheduler overhead). Fortunately, there’s a better way, and that better way is with a port set.
A port set is basically Mach’s approach to select(2), or the more modern equivalents, poll, epoll, or kevents or kqueues. You stuff in all the ports you want, and mach_msg will wake up whenever any of them does anything. No need for timeouts, threads, etc.
Making a port set is pretty simple - it’s done by making a mach port of a particular flavor (a port set is just a port) and then using the impenetrably named mach_port_move_member to add the port to the port (yo dawg).
Here’s the slightly modified example, where the server loop parks on a port set with our server port inside. It’s functionally identical, but it will allow us to do some better scaling in a future article.
#include <bootstrap.h> #include <mach/mach.h> #include <stdio.h> #include <unistd.h> int main(const int argc, char **argv) { printf("bootstrap port: %d\n", bootstrap_port); if (argc == 1) { // "server" mode mach_port_t service_port = MACH_PORT_NULL; kern_return_t kr = 0; // should fail kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); kr = bootstrap_check_in(bootstrap_port, "com.example.test", &service_port); printf("service_port: %d (%x)\n", service_port, kr); // should succeed kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); // Allocate our service port set, using MACH_PORT_RIGHT_PORT_SET mach_port_t service_set = MACH_PORT_NULL; kr = mach_port_allocate(mach_task_self(), MACH_PORT_RIGHT_PORT_SET, &service_set); printf("service set allocation: %x (port set: %d)\n", kr, service_set); // Add service_port to service_set. We can add or remove other ports from the set in the future. kr = mach_port_move_member(mach_task_self(), service_port, service_set); printf("move_member: %x\n", kr); printf("server loop\n"); while(1) { struct message { mach_msg_header_t header; mach_msg_body_t body; mach_msg_type_descriptor_t type; mach_msg_trailer_t trailer; } message; kr = mach_msg((mach_msg_header_t*)&message, MACH_RCV_MSG, 0, sizeof(message), service_set, MACH_MSG_TIMEOUT_NONE, MACH_PORT_NULL); printf("we get message: (%x) - size: %d\n", kr, message.header.msgh_size); } } else { // "client" mode mach_port_t service_port = MACH_PORT_NULL; kern_return_t kr = 0; kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); struct { mach_msg_header_t header; mach_msg_body_t body; } message = {0}; message.header.msgh_remote_port = service_port; message.header.msgh_size = sizeof(message); message.header.msgh_bits = MACH_MSG_TYPE_COPY_SEND; kr = mach_msg((mach_msg_header_t*)&message, MACH_SEND_MSG, sizeof(message), 0, service_port, MACH_MSG_TIMEOUT_NONE, MACH_PORT_NULL); printf("message send %x\n", kr); } return 0; }
Now we’re sending nothing at all still, but we could send nothing to as many services as we want with a single server loop. Neat!
At this point, I’ll take a wild diversion for a tool called mig (Mach Interface Generator), which we can use to simplify marshaling data in both directions, and which will handle a lot of the boilerplate stuff for crafting and decoding messages. This will also simplify sending mach ports and out-of-line data, which can be tricky to do manually. Source code for mig is typically in a .defs file, and mig parses it to generate C and H files that contain the necessary code. You can take a look inside these generated files, but there’s a lot of stuff, so I’ll skip the commentary. Be warned, they’re dense, and there’s a lot of cool stuff in them if you dig around some.
You can learn more about mig in the Mach Server Writer’s Guide, specifically Chapter 3, “Mach Interface Generator (MIG).” More esoteric information on mig is found in the appendices of the same document.
A .defs file is similar to C, but some of the syntax is wonky. It’s primarily intended to define interfaces, similar to a Header file, so don’t expect flow control or fancy pants stuff like that. You can find some .defs files floating around in /usr/include/mach/ and elsewhere with the normal macOS SDK. These are kind of a historical curiosity at this point; Normally, you don’t ever want clients crafting mach messages directly, using the interfaces defined in these files. Doing so is extremely low-level, and it makes binary compatibility a nightmare. Almost all modern frameworks I observed wrapped the client side in normal C interface calls, and handled the mig entrypoints themselves. That said, even if your interface is entirely encapsulated (which you should absolutely do), you still have to be prepared for malicious or errant clients to craft malformed messages. Message size sanity checking is handled for you, but once things get exotic (passing data out of line, for example), you need to be extremely careful. Fuzzing these interfaces is still relatively new, though some researchers have tools that can exercise it. I’m not aware of any that are public or easy to use though.
There are a couple things that you must never, ever, ever put into a mach message. I’m not joking - bugs here can ruin or end someone’s life. These are the mistakes that rootkits are made of.
First, your task port (from mach_task_self()). Giving this to someone is giving them the ability to map or unmap pages in your address space, create or destroy threads in your address space, and access to read or modify anything in your address space. With the exception of the kernel, which can already do all that stuff, this is simply too much power to give to someone else. If they need a unique identifier, use your task name port. You can get this with task_get_special_port with TASK_NAME_PORT. Even if you trust the other side, it’s possible that it was compromised. Never Send Your Task Port.
Second, you must never provide pointers. Round-tripping pointers (where you send an address to the client, and they send it back, and you dereference it) is extraordinarily dangerous - they can pass in any address they want, and you will die. At least one major iOS jailbreak was because someone did this with a kernel interface. Even if it’s not dereferenced, providing pointers gives them an idea of how your address space looks. With enough pointers, ASLR can be defeated, or libraries can be located in memory to craft shell code. All resources that are present in both the client and the server should be referenced by a unique identifier (I was a fan of uint64_t’s, so that you don’t have to deal with rollover under normal situations, usually monotonically increasing), or, in some cases, a mach port (IOSurfaces can be easily passed around via mach ports using IOSurfaceCreateMachPort(), for example). Don’t Ever Pass Pointers.
Now, on to a simple .defs file to get a feel for what’s going on. We’ll call this file small.defs.
// 400 is the initial message ID number - this generally doesn't matter, but overlapping message IDs is troublesome. subsystem small 400; userprefix USER; // client-side RPC entry points will be USER* form serverprefix SERVER; // server-side RPC entry points will be SERVER* form // include some standard types (these also work in C) #include <mach/std_types.defs> #include <mach/mach_types.defs>
As you can see, it looks a lot like C. Comments work similarly, there are includes (which are generally just for types, so they work in C as well). But there are some distinct keywords too.
subsystem tells mig what our interface name is (small in this case), and what the first message ID number is (400). As a general practice, each interface should have a distinct subsystem name, and should have non-overlapping message IDs. Since these are for ports that we’ll be controlling, we don’t have to worry about colliding with other subsystems that may be in use for other ports, but if we have multiple subsystems that we’ve made for ourselves then we should be careful.
The other two, userprefix and serverprefix define what our Server and Client (user) entry points will be named. In this case, I chose the completely unoriginal “USER” and “SERVER” names, so we’d have something like USERDoStuff called from the client, and SERVERDoStuff as server-side handler. Note that this example doesn’t actually define any RPCs, so we don’t actually get any entry points for sending or handling messages - we’ll get to that in the next example. There is one special entry point that the server gets, and we’ll get to that in a moment.
To process this file, we invoke mig thusly:
$ mig -user smallUser.c -server smallServer.c -header smallUser.h -sheader smallServer.h small.defs
This tells mig to generate smallUser.c, smallServer.c, smallUser.h, and smallServer.h from small.defs. If you don’t specify all those arguments, it will default to using the subsystem name, and create similarly named files, though it won’t make the server header I believe.
The one special entry point we get is in smallServer.c/h, and it’s
boolean_t small_server( mach_msg_header_t *InHeadP, mach_msg_header_t *OutHeadP);
Which takes a mach_msg_header_t for input. Unsurprisingly, this is what we get from mach_msg. So, we can feed that in, and see what happens. OutHeadP* must not alias InHeadP or else everything will go crazy. Under the hood, the output is initialized before the input is processed, so it’ll end up smashing the message ID, and it’ll either make the ID go out of bounds (good, because that’s a sensible failure), or alias another message ID (bad, now all bets are off on correctly interpreting the rest of the message). Don’t chance it, just use another buffer.
Now we’ll add the headers, call the server entry point, and print some simple status.
#include <bootstrap.h> #include <mach/mach.h> #include <stdio.h> #include <unistd.h> // include the autogenerated mig bits #include "smallUser.h" #include "smallServer.h" int main(const int argc, char **argv) { printf("bootstrap port: %d\n", bootstrap_port); if (argc == 1) { // "server" mode mach_port_t service_port = MACH_PORT_NULL; kern_return_t kr = 0; // should fail kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); kr = bootstrap_check_in(bootstrap_port, "com.example.test", &service_port); printf("service_port: %d (%x)\n", service_port, kr); // should succeed kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); /*mach_port_t service_set = MACH_PORT_NULL; kr = mach_port_allocate(mach_task_self(), MACH_PORT_RIGHT_PORT_SET, &service_set); printf("service set allocation: %x (port set: %d)\n", kr, service_set); kr = mach_port_move_member(mach_task_self(), service_port, service_set); printf("move_member: %x\n", kr);*/ printf("server loop\n"); while(1) { typedef struct { mach_msg_header_t header; mach_msg_body_t body; mach_msg_type_descriptor_t type; mach_msg_trailer_t trailer; } message; message received = {0}, reply = {0}; kr = mach_msg((mach_msg_header_t*)&received, MACH_RCV_MSG, 0, sizeof(message), service_port, MACH_MSG_TIMEOUT_NONE, MACH_PORT_NULL); printf("we get message: (%x) - size: %d\n", kr, received.header.msgh_size); if (small_server((mach_msg_header_t*)&received, (mach_msg_header_t*)&reply)) printf("mig handled it!\n"); else printf("mig didn't handle it\n"); } } else { // "client" mode mach_port_t service_port = MACH_PORT_NULL; kern_return_t kr = 0; kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); struct { mach_msg_header_t header; mach_msg_body_t body; } message = {0}; message.header.msgh_remote_port = service_port; message.header.msgh_size = sizeof(message); message.header.msgh_bits = MACH_MSG_TYPE_COPY_SEND; kr = mach_msg((mach_msg_header_t*)&message, MACH_SEND_MSG, sizeof(message), 0, service_port, MACH_MSG_TIMEOUT_NONE, MACH_PORT_NULL); printf("message send %x\n", kr); } return 0; }
We now compile like this (to get all the mig bits):
$ clang service.c smallServer.c smallUser.c -o service
And it should compile without warnings.
Let’s fire it up (I’ll skip copy/pasting the client invocation for brevity):
$ ./service bootstrap port: 1799 looked-up service_port: 0 (44e) service_port: 2819 (0) looked-up service_port: 2819 (0) server loop we get message: (0) - size: 28 mig didn't handle it
This is expected - we didn’t define any entry points, so there’s no possible way for the mig server to handle it.
Let’s now add a message to our .defs file:
simpleroutine DoStuff( server_port : mach_port_t; an_integer : uint32_t);
This will define a “simple routine” (an asynchronous RPC - the caller won’t block), and it passes along a single integer. It will be called USERDoStuff in the client, and SERVERDoStuff in the server, per the prefixes in the .defs file, and it will have a message ID of 400. The syntax is kind of goofy, but the element after the colon is the type, and the element before the colon is essentially just for humans to read to know what the thing is.
Add this, re-run mig, and let’s try to recompile.
$ clang service.c smallServer.c smallUser.c -o service
Undefined symbols for architecture x86_64:
"_SERVERDoStuff", referenced from:
__XDoStuff in smallServer-af34d9.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)Yay, something bad! You can see our SERVERDoStuff in there (mig uses __X* form for its own name spacing, so you might see symbols like that if you’re in the habit of debugging on checking out your symbol table. More importantly, it’s complaining that SERVERDoStuff is undefined. I guess we should go and add it.
kern_return_t SERVERDoStuff(mach_port_t server_port, uint32_t an_int) { printf("we got a message, and the integer is %d\n", an_int); return KERN_SUCCESS; }
And while we’re at it, let’s make the client actually use the client side instead of creating a raw mach message.
else { // "client" mode mach_port_t service_port = MACH_PORT_NULL; kern_return_t kr = 0; kr = bootstrap_look_up(bootstrap_port, "com.example.test", &service_port); printf("looked-up service_port: %d (%x)\n", service_port, kr); kr = USERDoStuff(service_port, 42); printf("USERDoStuff: %d\n", kr); /*struct { mach_msg_header_t header; mach_msg_body_t body; } message = {0}; message.header.msgh_remote_port = service_port; message.header.msgh_size = sizeof(message); message.header.msgh_bits = MACH_MSG_TYPE_COPY_SEND; kr = mach_msg((mach_msg_header_t*)&message, MACH_SEND_MSG, sizeof(message), 0, service_port, MACH_MSG_TIMEOUT_NONE, MACH_PORT_NULL); printf("message send %x\n", kr);*/ }
Check out all that code savings! I left the old cruft in there for comparison, but the new way looks much more approachable.
Let’s see what blows up!
$ ./service bootstrap port: 1799 looked-up service_port: 0 (44e) service_port: 4355 (0) looked-up service_port: 4355 (0) server loop we get message: (0) - size: 36 we got a message, and the integer is 42 mig handled it!
Alright! We can now pass integers between client and server.
Wow, this has become stupendously long. Probably too long. So I’ll leave it at this, but rest assured, there’s still a lot to cover. To whet your appetites, I’ll list 3 problems that are outstanding.
First, we’re not able to get any information back from the server, and we’re not able to block the client on the call to get an answer if it was able to provide it.
Second, the server can’t disambiguate multiple clients.
And Third, the server can’t detect when clients exit or crash to do any relevant cleanup.
We’ll tackle all those in the next issue. We’ll also cover some of the more complex options available through mig, including sending memory objects, structures, and out-of-line data if there’s space for them all.