Caching Server Nagios Script Updated With Port Checking

I’ve just updated the Nagios script for OS X Server’s Caching Server with support for ensuring Caching Server is running on a specific port.

In a normal setup of Caching Server, it has the ability to change ports during reboots, and if you have a restrictive firewall, you’ll have to specify a port for Caching Server via the command line. You can specify the port via the serveradmin command (make sure you stop the service first):

sudo serveradmin settings caching:Port = xxx

The xxx is the port you want to use for Caching Server. Normally, Caching Server will use 69010, but depending on your environment, that port may be taken by another service and so Caching Server will pick another port. Using that command above will get Caching Server to attempt to map to the specified port, but if it can’t it will fall back to finding another available port.

The script checks the port as defined in serveradmin settings caching:Port and cross-references it with serveradmin fullstatus caching:Port. If they are the same, we’ll continue through the script, but if it doesn’t I’ll throw a warning. Naturally, if serveradmin settings caching:Port returns 0, I skip any further port checks because a 0 port means randomly assign a port.

Check out the code on GitHub


Caching Server 2 and iOS 7 Confusion

So it appears there is some confusion in the Apple community about whether Caching Server 2 actually caches iOS 7 app downloads. Numerous posts have been popping up saying “No, you’re wrong, it doesn’t cache iOS 7 downloads!”. I’m here to say, no, you’re wrong. It does. After some extensive testing, I will now elaborate.

First off, I’m going to install a program called Charles that acts as a proxy server. Install Charles, set it up then have your iOS device route requests through Charles so we can see exactly what’s happening. Now, before I continue, I must let you know that the app I am about to download was one that I had previously downloaded a week ago, but then deleted (my partner and I were using it for her university project) so it has already been cached on my Mavericks Caching Server. This article isn’t about the initial caching process, more about proof that it does in fact serve cached iOS App Store downloads. I have a standard OS X Mavericks Server running on a 2010 Mac mini with healthy DNS and 1 subnet, and my iPhone is running iOS 7.0.3.

The App Download In Charles
The App Download In Charles.

Right, so I load up the Purchases tab on my iPhone and tap on the app to download it. Now this app in question is Metronome, a small app that is 18.7 MB in size. Make sure you remember the size because it’s important. Just before I downloaded the app I opened up Charles and started a recording session. The app downloads pretty quickly and I check out what Charles has captured.

Caching Server Proxy URI Tree
The Caching Server’s Proxy Tree.

So far, it looks good. As you can sort of see by the image, my caching server at 10.1.125.254 on port 61090 responded to an App Store download request. How is that possible when Caching Server 2 doesn’t support iOS 7 app downloads? Oh wait, it does. My bad. Let’s investigate further:

Caching Server's Response Size
My Caching Server’s Response Size — Go Figure!

Basically, that image shows that the response from my caching server was 18.72 MB in size. Wow! Didn’t I say that the app was 18.7 MB in size? Yep, I did. But wait, that’s not the only piece of evidence I’m going to use to prove that I’m right. Next, I’m going straight to the source. Caching Server 2 stores all of the cached metadata in a database called AssetInfo.db. I’m going to copy that to the Desktop using Terminal (the folder it resides in isn’t accessible by standard Finder means).

Okay, I’ve copied the AssetInfo.db from /Library/Server/Caching/Data/ and opened it up in Base, which is a program to work with SQLite 3 databases. After opening AssetInfo.db I loaded up the ZASSET table and export all rows as a CSV. I then opened up the CSV in Excel (Oh Numbers? We all know Numbers has excellent CSV support…). Next, copying the UUID from the URI in Charles, I pasted that into a search field in Excel, and to my surprise, a row was returned! (I lie, it wasn’t a surprise. I expected it to return a result). You could also do this in Terminal, like this:

sqlite3 /Library/Server/Caching/Data/AssetInfo.db 'SELECT * FROM ZASSET WHERE ZURI LIKE "%8bbfacac-c1ed-b397-e6cd-9110eb87001b%"'

Which gets me this beautiful result:

177|1|5|0|19624091||404215540.806059|daa32fb0fc47da467fd693e12262c318|6FCE511E-C2B2-4449-99E2-0B5EC9F919B4|Fri, 07 Dec 2012 03:15:24 GMT|/au/r1000/070/Purple/v4/8b/bf/ac/8bbfacac-c1ed-b397-e6cd-9110eb87001b/mzps1125351636230933757.D2.dpkg.ipa|

Now, one of those fields is the ZURI (aka the URI that Caching Server uses). When examining the URI in Charles, you can see the URI is the same:

http://10.1.125.254:61090/au/r1000/070/Purple/v4/8b/bf/ac/8bbfacac-c1ed-b397-e6cd-9110eb87001b/mzps1125351636230933757.D2.dpkg.ipa?source=a954.phobos.apple.com

Of course you exclude the IP address as it’s the caching server address on my LAN. The URI matches the AssetInfo.db’s ZURI field and guess what? The ZLASTMODIFIEDSTRING is the exact same date that the last update for Metronome was published (7th December 2012). I also found that the ZTOTALBYTES field, when converted to MB in Base-2, equals… 18.7 MB (which in itself is unusual, the rest of caching server calculates bytes in Base-10). Also, doing a Get Info on the binary file reveals the exact same byte count as ZTOTALBYTES.

Further to that, when I navigate to the actual binary (stored at /Library/Server/Caching/Data/<GUID>/0 by default) and enter md5 /path/to/file I get the same hash that is stored in ZCHECKSUM. For example, the hash for Metronome in the ZCHECKSUM row is daa32fb0fc47da467fd693e12262c318. Then, in Terminal I entered:

md5 /Library/Server/Caching/Data/6FCE511E-C2B2-4449-99E2-0B5EC9F919B4/0

Now, my result was daa32fb0fc47da467fd693e12262c318. Let’s compare them:

daa32fb0fc47da467fd693e12262c318 # ZCHECKSUM
daa32fb0fc47da467fd693e12262c318 # md5 Result in Terminal

Yep, they’re identical. Fact-based proof that OS X Mavericks Server’s Caching Server provides cached iOS App Store downloads to iOS 7 devices.

Now, this article has come off very douchey, but I felt it was necessary to prove to people that Caching Server 2 in OS X Mavericks Server does cache iOS app downloads.


Caching Server Nagios Script Updated for Mavericks

Just a quick heads up that Jedda and I’s check_osx_caching monitoring script for Nagios and OS X has been updated to support OS X Mavericks Server. Given that Caching Server 2 in 10.9 adds more verbosity to what content is taking up what space, I’ve added support for each content type and has been added to the performance data that is returned in the script.

As content usage is not tracked in Mountain Lion’s Caching Server I had to do a check of the operating system to see what version the script is being run on. To do this, I used the function sw_vers and added the flag productVersion to only have the product version returned. Now, on a first release of the OS you’ll get a clean float like 10.9 or 10.8 which makes it really easy to do a comparison using bc (a precision calculator language). Where it becomes a problem is when you get numbers like 10.8.5 which, because they have two periods, mean they don’t work with bc. I decided I could either write a function that does a full comparison of the version numbers, or just get the major and minor release and do a standard mathematical float comparison. Using grep I can pull the major and minor release information and assign it to a variable. See below for the code:

sw_vers -productVersion | grep -E -o "[0-9]+\.[0-9]"

Next, the comparison is performed to see whether the current OS is less than 10.9. If the current OS is less than 10.9, 1 is returned. If it’s the same (or greater), the result is 0. This code is below:

echo $osVersion '< 10.9' | bc -l

Note that the above example requires the variable $osVersion. If you were hard coding the values, you could do something like below:

echo '10.8 < 10.9' | bc -l

Now, if the value of the expression is 0, we will now grab all the Mavericks Caching Server usage data, and assign it to a variable called mavericksPerfData which is appended to the end of the final return printf.

I'll be updating our RRDtool graph for the Caching Server 2 and will post the code soon! Stay tuned.

Check out the code on GitHub!


OS X Mavericks Server – Setting Up Caching Server

The Caching Server in OS X Mavericks Server is used to speed up downloads of content distributed by Apple over the internet. Caching Server caches all updates, App Store and iBooks downloads, and Internet Recovery software that your OS X v10.8.2+ and iOS 7 devices download.

Contents

Caching Server - Hasn't Been Started Yet
Caching Server – Hasn't Been Started Yet.

Caching Server 2 which came out with OS X Mavericks Server raised the bar from Caching Server in OS X Mountain Lion Server by adding software updates, iOS app updates, iBooks and Internet Recovery downloads. Also added is the support for multiple Caching Server 2 servers on the same public IP network. If there are multiple Caching Servers on the same network and one doesn’t contain a requested download, the Caching Server will consult with its peer servers on the same network to see if they have the requested download. This information is then passed onto the client, that will then select the right server automatically.

One important thing to note is that from OS X v10.8.2, Caching Server will do all software updates for clients, this doesn’t necessarily mean it can replace the Software Update Server that has been with OS X Server for a long time. Even though Software Update Server does require more maintenance and client configuration (you have to configure each client), you still maintain control over which clients get which updates. Caching Server 2 does not have the ability to deny updates for any devices. Maybe that will be added in the future, but for now if you need to control who gets what update, Caching Server 2 is not for you.

Okay, enough chit-chatter, let’s get Caching Server 2 running! Open the Server app then click on the section named Caching. Depending on your requirements, you might want to store all the downloads on a separate drive that’s connected to the server. I usually configure servers with small SSD boot drives, so storing Caching Server 2 downloads on a small SSD is not an option. Once you’ve chosen your storage device, you can now select the maximum cache size for Caching Server 2. There’s only one rule with Caching Server 2 storage, the minimum storage size is 25GB. Caching Server 2 has a buffer of 25GB where the least-used content is deleted to make way for more popular and requested downloads. We’re now ready to start the caching server! Hit the On button in the top-right corner.

Caching Server After Some Content Has Been Cached
Caching Server After Some Content Has Been Cached.

After Caching Server 2 has started, try going to the App Store and downloading a few apps, and iBooks to download a few books. Check back in a few minutes to see your Usage graph, it should now be populated with details of the usage split, with detailed numbers of how much each cache category is using. Another way to check out how Caching Server 2 is going is by using Terminal! Open up Terminal and enter the following command:

sudo serveradmin fullstatus caching

You should see something like below:

caching:Active = yes
caching:state = "RUNNING"
caching:CacheUsed = 125027720
caching:Port = 61090
caching:TotalBytesRequested = 124886938
caching:RegistrationStatus = 1
caching:CacheLimit = 25000000000
caching:CacheFree = 4025427456
caching:Peers = _empty_array
caching:TotalBytesFromPeers = 0
caching:StartupStatus = "OK"
caching:TotalBytesFromOrigin = 124886938
caching:CacheStatus = "OK"
caching:TotalBytesReturned = 124886938
caching:CacheDetails:_array_index:0:BytesUsed = 60543656
caching:CacheDetails:_array_index:0:LocalizedType = "Mac Apps"
caching:CacheDetails:_array_index:0:MediaType = "Mac Apps"
caching:CacheDetails:_array_index:0:Language = "en"
caching:CacheDetails:_array_index:1:BytesUsed = 53730125
caching:CacheDetails:_array_index:1:LocalizedType = "iOS Apps"
caching:CacheDetails:_array_index:1:MediaType = "iOS Apps"
caching:CacheDetails:_array_index:1:Language = "en"
caching:CacheDetails:_array_index:2:BytesUsed = 10753939
caching:CacheDetails:_array_index:2:LocalizedType = "Books"
caching:CacheDetails:_array_index:2:MediaType = "Books"
caching:CacheDetails:_array_index:2:Language = "en"
caching:CacheDetails:_array_index:3:BytesUsed = 0
caching:CacheDetails:_array_index:3:LocalizedType = "Movies"
caching:CacheDetails:_array_index:3:MediaType = "Movies"
caching:CacheDetails:_array_index:3:Language = "en"
caching:CacheDetails:_array_index:4:BytesUsed = 0
caching:CacheDetails:_array_index:4:LocalizedType = "Music"
caching:CacheDetails:_array_index:4:MediaType = "Music"
caching:CacheDetails:_array_index:4:Language = "en"
caching:CacheDetails:_array_index:5:BytesUsed = 0
caching:CacheDetails:_array_index:5:LocalizedType = "Other"
caching:CacheDetails:_array_index:5:MediaType = "Other"
caching:CacheDetails:_array_index:5:Language = "en"

When your caching server first connects to Apple’s servers to let them know we have set up a caching server, it is assigned a GUID that other peers and clients can use to identify the server. To obtain this GUID you can enter the following into Terminal:

sudo serveradmin settings caching:ServerGUID

Shared Caching Over Subnets

A nifty feature is the ability for the caching server to traverse across subnets to share cached downloads. This feature is enabled by unticking the “Only cache content for local networks”. By default, Caching Server will only cache content for the current subnet (in my case, 10.1.125.0-10.1.125.255). An alternative way to enable multi-subnet caching through Terminal is to alter the LocalSubnetsOnly option, do this command to enable subnet traversal:

sudo serveradmin settings caching:LocalSubnetsOnly = no

This change doesn’t require a reboot of the Caching Server, but reboot just to make sure. If you want to disable subnet traversal, do this:

sudo serveradmin settings caching:LocalSubnetsOnly = yes

Those changes that you make will be reflected in the logs. If you take a look in the Debug.log (located at /Library/Server/Caching/Logs/Debug.log) you’ll see something like this:

2013/10/31 18:09:36:979 Registering with local address: 10.1.125.254; local subnet range only: 10.1.125.0-10.1.125.255
2013/10/31 18:09:38:284 Request for registration from https://lcdn-registration.apple.com/lcdn/register succeeded
2013/10/31 18:09:38:284 Got back public IP xx.xx.xxx.xx
2013/10/31 18:09:38:288 This server has 0 peer(s)
Note: I’ve obscured my public IP address for my own safety.

This feature requires you to be using Network Address Translation (NAT) and the same public IP address – both requirements of Apple. If you have multiple subnets if your office or workplace but only have the resources for one Caching Server, it would be wise to enable this feature on the server. Also make sure your firewall will allow traffic over port 61090/TCP for internal traffic only (this port may need to be changed if you specified a custom port when configuring caching server, or if you have multiple caching servers). If you don’t know what port your Caching Server is available at, you can enter this command to find out:

serveradmin fullstatus caching | grep "Port"

In most cases you should have 61090 returned. Generally, the only time this will be different is if you have multiple caching servers (see below) in the one workplace.

If you have a look at the settings for Caching Server in Terminal, you’ll see that the Caching Server port is set to 0. This means that when it starts up it’ll check for an available port on startup and choose an appropriate one (I presume this is defined when it registers with Apple). For example: if Caching Server was hard-wired to start on 61090, some of your machines or firewall may detect conflicts. Otherwise, you can hard-code a port for it to run on by firstly shutting down the Caching Server in the Server app, or in Terminal by entering sudo serveradmin stop caching. Next, decide on a port that you know to be available. As far as I can tell any port is okay, as long as it’s not already in use on the LAN. If I wanted to specify the Caching Server to run on 61000 I would enter this into Terminal:

sudo serveradmin settings caching:Port = 61000

Hit return then start Caching Server either in the Server app or by entering sudo serveradmin start caching. Give the Caching Server a few seconds to start, then enter sudo serveradmin fullstatus caching. I’ll trim the output, but the line you’re looking for should now show this:

caching:Port = 61000

Huzzah! Your Caching Server is now running on your custom port. Change it back to 0 if you want it to randomise the port on registration. Note that changing it back to default may not take into effect straight away, you may have to reboot Caching Server after a little while as Caching Server will cache registration settings, along with its last port (found with defaults read /Library/Server/Caching/Config/Config.plist LastPort). I’ve also updated the Nagios script for checking Caching Server to include port-specific checks, check it out!.

Peer Review

I really like the idea of a peer-based system with Caching Server 2 so I decided to spin up another OS X Mavericks Server instance and get it set up with Caching Server 2 also. After getting them both registered with Apple’s backend system (an automated process by the way, and should only take 2 minutes for each machine), I ran sudo serveradmin fullstatus caching on both machines so I could check out what the caching:Peers array would hold. Here are my results:

mavericks.pretendco.com — 10.1.125.120:

caching:Peers:_array_index:0:guid = "7B4E17C1-14E2-48E0-8062-34660B4C041B"
caching:Peers:_array_index:0:version = "73"
caching:Peers:_array_index:0:healthy = yes
caching:Peers:_array_index:0:address = "10.1.125.121"
caching:Peers:_array_index:0:port = 50294

replica.pretendco.com — 10.1.125.121:

caching:Peers:_array_index:0:guid = "8594F0EF-E810-4993-A74F-A1473C6B099D"
caching:Peers:_array_index:0:version = "73"
caching:Peers:_array_index:0:healthy = yes
caching:Peers:_array_index:0:address = "10.1.125.120"
caching:Peers:_array_index:0:port = 49486

On each machine you can also enter sudo serveradmin fullstatus caching | grep TotalBytesFromPeers and sudo serveradmin fullstatus caching | grep TotalBytesFromOrigin to get the number of bytes transferred from peers and origin respectively. I noticed a huge speed increase when downloading a purchased app from the App Store when it had already been cached by the other caching server – as you would expect. After downloading about 15 different apps across my two OS X Mavericks Server instances, I check out the cached totals via the command line. Here’s what I got:

mavericks.pretendco.com — 10.1.125.120:

caching:TotalBytesFromPeers = 4957598
caching:TotalBytesFromOrigin = 15591778

Which equates to 4.96 MB from peers and 15.59 MB from itself.

replica.pretendco.com — 10.1.125.121:

caching:TotalBytesFromPeers = 17628454
caching:TotalBytesFromOrigin = 167213809

Which equates to 17.63 MB from peers and 167.21 MB from itself.

Note: Caching Server reports all its usage in the SI format, not the standard base 2 format, so make sure you factor that in to any calculations you may make.

Monitoring

Along with the original Caching Server, Caching Server 2 has a whole host of options to check the health of the service via the command line. Here’s a few commands you can use the ensure Caching Server 2 is alive and happy!

sudo serveradmin fullstatus caching | grep "Active"

This will return no if the service is in “STARTING” mode, and will not return anything if the service is in “STOPPED” mode.

sudo serveradmin fullstatus caching | grep "StartupStatus"

This should return “OK” if the service is running OK (d’uh, that’s obvious). If the service is in “STARTING” mode, this will return “PENDING”.

sudo serveradmin fullstatus caching | grep "CacheStatus"

This should return “OK” if the cache status is healthy. I’ve tried to break this to get it to return something other than OK but I haven’t been successful. Even deleting the entire /Library/Server/Caching directory doesn’t cause this to fail.

sudo serveradmin fullstatus caching | grep "CacheFree"

Theoretically Caching Server 2 should never let this dip down to anywhere near zero, but doesn’t mean you can’t monitor it.

sudo serveradmin fullstatus caching | grep "RegistrationStatus"

Typically this only equals 0 when Caching Server 2 has been started for the first time. While 0 (zero), the caching server is talking to Apple to get a GUID and register itself. This process generally takes no longer than two minutes, and will change to a 1 (one) once it has completed registration.

defaults read /Library/Server/Caching/Config/Config.plist LastRegOrFlush

This will tell you the last time Caching Server 2 checked in with Apple and retrieved an updated list of peers (and their status).

Finally, there’s a great debug log that is really verbose in what is happening during any stage of Caching Server’s life. The location of the log is /Library/Server/Caching/Logs/Debug.log, so tail -f it when you’re playing around to see what it does.

Hopefully this article has helped you get familiar with Caching Server 2 and set it up correctly.


Working with OS X Caching Server’s Max Cache Size Limits

Caching Server
The Caching Server pane in Server.app

Added in a recent version of OS X Server was the ability to provide an advanced software update feature called Caching Server. This enhanced the previous Software Update server features by including content from iTunes (i.e. iOS app updates and iBooks), along with providing an easy way for automatic server update selection (for those who are unfamiliar with Caching Server’s nifty features, check out the PDF on this page).

During setup you are required to select a volume for the cached updates to be stored. Depending on the size of this volume, the maximum size used slider can be rather useless. For example, when configuring my RAID enclosure (which has 8.8TB of useable space), the slider went from 30GB to 580GB in one step! This kind of limited selection is very frustrating. I want more than 30GB of cached updates, but I don’t want 580GB! Unfortunately, the current GUI slider steps in very high increments. Thankfully for us, using serveradmin through the terminal will save us.

If you want to see what limit Caching Server has set via the command line, you can do sudo serveradmin fullstatus caching then check out the line caching:CacheLimit. This number is shown in bytes and will convert nicely to a base 10 unit (otherwise known as SI), rather than the normal base 2.

For example, if you wanted to set a limit of 100GB, you would use the number 100000000000. That’s a lot of zeroes. Alternatively, 250GB would be 250000000000. To work out your storage conversion needs, I’d recommend checking out this converter here and use the Byte SI Decimal Prefix to convert to bytes.

Now that we’ve decided on how much space we’d like to use for Caching Server, lets tell it what we want. Open up a new Terminal window and using the command serveradmin settings caching:CacheLimit we can specify (in bytes) how much space we want to use. In my case, I wanted to use a maximum of 100GB for Caching Server. To do this, I enter the following:

sudo serveradmin settings caching:CacheLimit = 100000000000

After pressing return and running fullstatus on Caching Server (or opening the Caching tab in the Server app), you can now see that a maximum of 100GB will be used. Naturally, entering a different number will yield a different result in Server app (or the Terminal), but you get the picture.