OS X Server and Heartbleed

Heartbleed OS X

If you keep up with technical news you would’ve seen an abundance of articles on something called Heartbleed (or CVE-2014-0160). Essentially, it’s a protocol implementation bug that affects newer versions of OpenSSL, which is used in a majority of Linux installations and can affect many services like Apache, nginx and even IMAP (just to name a few). I’m not going to go into the details because I’ll leave that to the experts, but I highly recommend taking a look at the Heartbleed website to learn more.

Now, when I saw this I got immediately worried about all my clients’ (not to mention my own) installation of OS X Server. Thankfully, OS X has an older implementation of OpenSSL (on 10.9.2 it’s 0.9.8y – found by doing this command in Terminal: openssl version) and according to the Heartbleed website, is not vulnerable. Just to make sure, I ran a Python script found online written by Jared Stafford (download) and when against my server at home, I get the following:

Connecting...
Sending Client Hello...
Waiting for Server Hello...
... received message: type = 22, ver = 0301, length = 53
... received message: type = 22, ver = 0301, length = 2551
... received message: type = 22, ver = 0301, length = 525
... received message: type = 22, ver = 0301, length = 4
Sending heartbeat request...
... received message: type = 21, ver = 0302, length = 2
Received alert:
0000: 02 46                  .F

Server returned error, likely not vulnerable

Further testing on OS X has revealed that Heartbleed won’t be exposing anything, which is a huge relief for me. Having said that, as this was undiscovered for 2 years, theoretically there’s nothing stopping there being another vulnerability in the wild that could be causing the same (or more or less) damage as Heartbleed.

If you run the Python script and find you are affected, I urge you to patch your OpenSSL installation and regenerate your SSL certificates from scratch. As this vulnerability grabs 64KB worth of data from memory, it’s possible your private key could be in that 64KB. This means you need more than just a CSR, you’ll need to start from the beginning. You can check yourself using an excellent service by Filippo Valsorda.

Good luck out there.


Adding Users to Open Directory Programmatically

Recently I had someone contact me because they were looking for a script that would pull users from a CSV, and then import them into Open Directory on OS X Mavericks Server. Always up for a challenge I wrote a bash script to grab the users then send them to the Open Directory domain.

Check out the code on GitHub Gist!

This script brings in the first name, surname, student ID and password from a CSV (the path can be specified in the only argument the script requires, which is the path to the CSV). The shortname is derived from the first name, surname and student ID. For example, Steve Miller with the Student ID of 654321 becomes sm654321.

To run the script, you would enter something like this:

./import_users.sh Users.csv

And, using the example Users.csv provided in the Gist (please bear in mind that the Gist automatically removes the trailing blank line, please add it before running the script), you would expect something like below:

2013-12-11 21:49:03: Added Joe Smith (js123456) to /LDAPv3/127.0.0.1.
2013-12-11 21:49:06: Added Bill Jones (bj987654) to /LDAPv3/127.0.0.1.
2013-12-11 21:49:09: Added Steve Miller (sm654321) to /LDAPv3/127.0.0.1.

For each user, the commands take between 3 to 4 seconds. That includes creating the user along with adding them to the groups. As noted in the bash script, you must ensure your CSV has Unix (CRLF) line endings, and a blank line at the end of the script. If it doesn’t have the correct line endings, it’ll fail completely. If you don’t have a blank line at the end, the last person in the list won’t be added.

Thankfully, adding this user through dscl creates the proper AuthenticationAuthority so the user is set up with Kerberos v5 and Apple Password Server. By default, the user will be added to the local groups com.apple.access_radius, com.apple.access_afp and com.apple.access_addressbook, and the network group workgroup.

Here’s a dump of a user created through this script:

dsAttrTypeNative:objectClass: person inetOrgPerson organizationalPerson posixAccount shadowAccount top extensibleObject apple-user
AltSecurityIdentities: Kerberos:sm654321@MAVERICKS.PRETENDCO.COM
AppleMetaNodeLocation: /LDAPv3/127.0.0.1
AppleMetaRecordName: uid=sm654321,cn=users,dc=mavericks,dc=pretendco,dc=com
AuthenticationAuthority:
 ;ApplePasswordServer;0xdb945916625111e39e62000c2928b48d,1024 65537 128972542829193592982355741981221062100920762016712038819623846465209128342622049783124389838185059988320142773235291480753225648977678597461748953848863734839600213928074142175413820927534135441280785829108224574601521657224863604777924988844508041132576614047193318182335513084715122081757952216834576233343 root@mavericks.pretendco.com:10.1.125.120
 ;Kerberosv5;;sm654321@MAVERICKS.PRETENDCO.COM;MAVERICKS.PRETENDCO.COM;
Comment:
 Student ID: 654321
EMailAddress: sm654321@pretendco.com
FirstName: Steve
GeneratedUID: FECF06E0-6654-4583-8471-B88AC4AC7D41
Keywords: students
LastName: Miller
NFSHomeDirectory: /dev/null
Password: ********
PrimaryGroupID: 20
RealName:
 Steve Miller
RecordName: sm654321
RecordType: dsRecTypeStandard:Users
UniqueID: 1052
UserShell: /usr/bin/false

Check out the code on GitHub Gist!

Note: I’ve tested this script a fair bit, but don’t blame me if your Open Directory screws up! Always ensure you have a known-good backup before running a script you got off the internet. I’ve tried to get this script as perfect as possible, but there’s always bugs because not every environment is the same.


Caching Server Nagios Script Updated With Port Checking

I’ve just updated the Nagios script for OS X Server’s Caching Server with support for ensuring Caching Server is running on a specific port.

In a normal setup of Caching Server, it has the ability to change ports during reboots, and if you have a restrictive firewall, you’ll have to specify a port for Caching Server via the command line. You can specify the port via the serveradmin command (make sure you stop the service first):

sudo serveradmin settings caching:Port = xxx

The xxx is the port you want to use for Caching Server. Normally, Caching Server will use 69010, but depending on your environment, that port may be taken by another service and so Caching Server will pick another port. Using that command above will get Caching Server to attempt to map to the specified port, but if it can’t it will fall back to finding another available port.

The script checks the port as defined in serveradmin settings caching:Port and cross-references it with serveradmin fullstatus caching:Port. If they are the same, we’ll continue through the script, but if it doesn’t I’ll throw a warning. Naturally, if serveradmin settings caching:Port returns 0, I skip any further port checks because a 0 port means randomly assign a port.

Check out the code on GitHub


Caching Server 2 and iOS 7 Confusion

So it appears there is some confusion in the Apple community about whether Caching Server 2 actually caches iOS 7 app downloads. Numerous posts have been popping up saying “No, you’re wrong, it doesn’t cache iOS 7 downloads!”. I’m here to say, no, you’re wrong. It does. After some extensive testing, I will now elaborate.

First off, I’m going to install a program called Charles that acts as a proxy server. Install Charles, set it up then have your iOS device route requests through Charles so we can see exactly what’s happening. Now, before I continue, I must let you know that the app I am about to download was one that I had previously downloaded a week ago, but then deleted (my partner and I were using it for her university project) so it has already been cached on my Mavericks Caching Server. This article isn’t about the initial caching process, more about proof that it does in fact serve cached iOS App Store downloads. I have a standard OS X Mavericks Server running on a 2010 Mac mini with healthy DNS and 1 subnet, and my iPhone is running iOS 7.0.3.

The App Download In Charles
The App Download In Charles.

Right, so I load up the Purchases tab on my iPhone and tap on the app to download it. Now this app in question is Metronome, a small app that is 18.7 MB in size. Make sure you remember the size because it’s important. Just before I downloaded the app I opened up Charles and started a recording session. The app downloads pretty quickly and I check out what Charles has captured.

Caching Server Proxy URI Tree
The Caching Server’s Proxy Tree.

So far, it looks good. As you can sort of see by the image, my caching server at 10.1.125.254 on port 61090 responded to an App Store download request. How is that possible when Caching Server 2 doesn’t support iOS 7 app downloads? Oh wait, it does. My bad. Let’s investigate further:

Caching Server's Response Size
My Caching Server’s Response Size — Go Figure!

Basically, that image shows that the response from my caching server was 18.72 MB in size. Wow! Didn’t I say that the app was 18.7 MB in size? Yep, I did. But wait, that’s not the only piece of evidence I’m going to use to prove that I’m right. Next, I’m going straight to the source. Caching Server 2 stores all of the cached metadata in a database called AssetInfo.db. I’m going to copy that to the Desktop using Terminal (the folder it resides in isn’t accessible by standard Finder means).

Okay, I’ve copied the AssetInfo.db from /Library/Server/Caching/Data/ and opened it up in Base, which is a program to work with SQLite 3 databases. After opening AssetInfo.db I loaded up the ZASSET table and export all rows as a CSV. I then opened up the CSV in Excel (Oh Numbers? We all know Numbers has excellent CSV support…). Next, copying the UUID from the URI in Charles, I pasted that into a search field in Excel, and to my surprise, a row was returned! (I lie, it wasn’t a surprise. I expected it to return a result). You could also do this in Terminal, like this:

sqlite3 /Library/Server/Caching/Data/AssetInfo.db 'SELECT * FROM ZASSET WHERE ZURI LIKE "%8bbfacac-c1ed-b397-e6cd-9110eb87001b%"'

Which gets me this beautiful result:

177|1|5|0|19624091||404215540.806059|daa32fb0fc47da467fd693e12262c318|6FCE511E-C2B2-4449-99E2-0B5EC9F919B4|Fri, 07 Dec 2012 03:15:24 GMT|/au/r1000/070/Purple/v4/8b/bf/ac/8bbfacac-c1ed-b397-e6cd-9110eb87001b/mzps1125351636230933757.D2.dpkg.ipa|

Now, one of those fields is the ZURI (aka the URI that Caching Server uses). When examining the URI in Charles, you can see the URI is the same:

http://10.1.125.254:61090/au/r1000/070/Purple/v4/8b/bf/ac/8bbfacac-c1ed-b397-e6cd-9110eb87001b/mzps1125351636230933757.D2.dpkg.ipa?source=a954.phobos.apple.com

Of course you exclude the IP address as it’s the caching server address on my LAN. The URI matches the AssetInfo.db’s ZURI field and guess what? The ZLASTMODIFIEDSTRING is the exact same date that the last update for Metronome was published (7th December 2012). I also found that the ZTOTALBYTES field, when converted to MB in Base-2, equals… 18.7 MB (which in itself is unusual, the rest of caching server calculates bytes in Base-10). Also, doing a Get Info on the binary file reveals the exact same byte count as ZTOTALBYTES.

Further to that, when I navigate to the actual binary (stored at /Library/Server/Caching/Data/<GUID>/0 by default) and enter md5 /path/to/file I get the same hash that is stored in ZCHECKSUM. For example, the hash for Metronome in the ZCHECKSUM row is daa32fb0fc47da467fd693e12262c318. Then, in Terminal I entered:

md5 /Library/Server/Caching/Data/6FCE511E-C2B2-4449-99E2-0B5EC9F919B4/0

Now, my result was daa32fb0fc47da467fd693e12262c318. Let’s compare them:

daa32fb0fc47da467fd693e12262c318 # ZCHECKSUM
daa32fb0fc47da467fd693e12262c318 # md5 Result in Terminal

Yep, they’re identical. Fact-based proof that OS X Mavericks Server’s Caching Server provides cached iOS App Store downloads to iOS 7 devices.

Now, this article has come off very douchey, but I felt it was necessary to prove to people that Caching Server 2 in OS X Mavericks Server does cache iOS app downloads.


FreeRADIUS Monitoring Script Updated for Mavericks

I’ve updated the FreeRADIUS monitoring script for Nagios with support for OS X Mavericks Server. Mavericks changed the way the FreeRADIUS server is started, along with the paths of execution and storage.

Like the update to the Caching Server 2 monitoring script, I had to write a check to see if the current OS is running 10.9, and if it is, perform a Mavericks-specific check. Like I mentioned in the other article, doing a version check and comparison isn’t particularly easy in Bash, but thankfully, I only have the compare the major and minor release numbers which is nice given that it’s also a float. Using bc (the arbitrary precision calculator language – I should get that on a shirt!) I can quite easily calculate the difference between 10.8 and 10.9. Anyway, check the code below for how I get the version number.

sw_vers -productVersion | grep -E -o "[0-9]+\.[0-9]"

Next, the comparison is performed to see whether the current OS is less than 10.9. If the current OS is less than 10.9, 1 is returned. If it’s the same (or greater), the result is 0. This code is below:

echo $osVersion '< 10.9' | bc -l

Note that the above example requires the variable $osVersion. If you were hard coding the values, you could do something like below:

echo '10.8 < 10.9' | bc -l

The major difference in my script for OS X Mavericks is now there's actually a process running called radiusd! Using ps and grep I now check to see if the FreeRADIUS server is running by doing this:

ps -ef | grep radiusd

Which, if the FreeRADIUS (or radiusd) is running, will return a non-empty string. If you run that and get an empty string (or nothing) back then your FreeRADIUS server isn't running. Shit. I recommend doing radiusd -X to start your RADIUS server in debug mode. That or you forgot to get RADIUS added to launchd by entering radiusconfig -start. Anyway, that's enough chit-chat, just get the damn code from the link below:

Check out the code on GitHub!