Tuesday, June 24, 2014

Compiling GnuPG on OS X

First make a directory to hold the compiled files:

sudo mkdir /usr/local/gpg

Create a text file /etc/paths.d/gpg with the following contents:

/usr/local/gpg/bin

Open a new terminal for the PATH changes to take effect, or just append /usr/local/gpg/bin to your $PATH.

Download the latest GnuPG from https://www.gnupg.org/download/index.html. The latest version is 2.0.24 as of today. Download the signature file and verify the signature. Extract the archive:

tar xjf ~/Downloads/gnupg-2.0.24.tar.bz2
cd gnupg-2.0.24

In the README file you'll see that you need the following dependencies

GnuPG 2.0 depends on the following packages:

libgpg-error     (ftp://ftp.gnupg.org/gcrypt/libgpg-error/)
libgcrypt        (ftp://ftp.gnupg.org/gcrypt/libgcrypt/)
libksba          (ftp://ftp.gnupg.org/gcrypt/libksba/)
libassuan >= 2.0 (ftp://ftp.gnupg.org/gcrypt/libassuan/)

You also need the Pinentry package for most function of GnuPG; however
it is not a build requirement.  Pinentry is available at
ftp://ftp.gnupg.org/gcrypt/pinentry/ .

Download the latest libgpg-error, along with its signature file, and verify the signature. Extract the archive, configure, make and install:

tar xzf ~/Downloads/libgpg-error-1.13.tar.gz
cd libgpg-error-1.13
./configure --prefix /usr/local/gpg
make
sudo make install

Download the latest libgcrypt, along with its signature file. Extract the archive, configure, make and install:

tar xzf ~/Downloads/libgcrypt-1.6.1.tar.gz
cd libgcrypt-1.6.1
./configure --prefix /usr/local/gpg --disable-aesni-support --disable-asm
make
make check
sudo make install

As you can see, I disabled AESNI support, because while I can examine the source code of this software to get some confidence that there are no backdoors, I can't examine the implementation of the AESNI instructions in the CPU. Therefore I can't trust that hardware AESNI doesn't have a backdoor. I also disabled assembly, because otherwise it fails to compile on OS X.

Download the latest libksba, along with its signature file. Extract the archive, configure, make and install:

tar xjf ~/Downloads/libksba-1.3.0.tar.bz2
cd libksba-1.3.0
./configure --prefix /usr/local/gpg
make
sudo make install

Download the latest libassuan, along with its signature file, and verify the signature. Extract the archive, configure, make and install:

tar xjf ~/Downloads/libassuan-2.1.1.tar.bz2
cd libassuan-2.1.1
./configure --prefix /usr/local/gpg
make
sudo make install

Download the latest pinentry, along with its signature file, and verify the signature. Extract the archive, configure, make and install:

tar xjf ~/Downloads/pinentry-0.8.3.tar.bz2
cd pinentry-0.8.3
./configure --prefix /usr/local/gpg --disable-pinentry-gtk --disable-pinentry-gtk2 --disable-pinentry-qt --disable-pinentry-qt4
make
make install

If you don't disable gtk and qt options, it will complain about not finding X.

You also need to download the latest pth, along with its signature file. Extract the archive, configure, make and install:

tar xzf ~/Downloads/pth-2.0.7.tar.gz
cd pth-2.0.7
./configure --prefix /usr/local/gpg
make
make test
make install

Back to gnupg:

cd gnupg-2.0.24
./configure --prefix /usr/local/gpg
make

You run into a problem here, the compilation fails:

...
Making all in gl
...
gcc -DHAVE_CONFIG_H -I. -I..     -g -O2 -Wall -Wno-pointer-sign -Wpointer-arith -MT allocsa.o -MD -MP -MF .deps/allocsa.Tpo -c -o allocsa.o allocsa.c
In file included from allocsa.c:21:
In file included from ./allocsa.h:23:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/stdlib.h:65:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/sys/wait.h:110:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/sys/resource.h:72:
In file included from ./stdint.h:66:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/inttypes.h:235:8: error: 
      unknown type name 'intmax_t'
extern intmax_t
...

If you look at /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/sys/resource.h:72, you can see that it says

#include <stdint.h>

It's meant to include /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/stdint.h, which would get intmax_t defined by including _types/_intmax_t.h. Unfortunately instead of including the correct stdint.h, it ends up including gl/stdint.h from GnuPG. If you look at gl/stdint.h on line 46, you see:

# include "///Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/5.1/include/stdint.h"

If you replace that line with

# include "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/stdint.h"

and then type make, everything compiles just fine. Then you run sudo make install and everything's done.

Tuesday, November 26, 2013

Tomcat SSL Hardening

If you are using SSL with your Java web application, and you are using Tomcat, the default configuration could be made more secure by by disallowing weak cipher suites. You can test your server configuration using Qualis SSL Server Test. With APR, you can use the cipher string as suggested in Hardening Your Web Server’s SSL Ciphers. Note that RC4 is not in that list, because it is broken. Standard Diffie-Hellman is slow, so I would remove those cipher suites from the string, making it

ECDH+AESGCM:ECDH+AES256:ECDH+AES128:ECDH+3DES:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

If you are not using APR, you need slightly different directives. SSL is configured in conf/server.xml, and there's a default configuration commented out:

<!--
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
           maxThreads="150" scheme="https" secure="true"
           clientAuth="false" sslProtocol="TLS" />
-->

After uncommenting it, we need to change a few things. Setting the certificate is beyond the scope of this post, but you'll probably need at least the keystorePass and keyAlias options. Change the protocol option to protocol="org.apache.coyote.http11.Http11NioProtocol", which will enable the NIO connector. The NIO connector is non-blocking for better performance with slow clients, and doesn't support client-initiated renegotiation, which mitigates some DOS attacks. If you don't care about IE6 and Java 6, you can remove SSLv3 and only leave the TLS protocols enabled using sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1".

Let's convert the cipher string into something Tomcat can understand. The option to configure is ciphers, using the JSSE naming convention, as specified in the documentation. In order to convert the OpenSSL cipher suite names to JSSE names, we can use the standard TLS codes for each cipher suite. The full list can be downloaded from the IANA registry, where you can download a tls-parameters-4.csv file. Now we can use a little bit of shell to convert the list:

STRING='ECDH+AESGCM:ECDH+AES256:ECDH+AES128:ECDH+3DES:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS'
CODES=`/usr/local/ssl/bin/openssl ciphers -V "$STRING" | cut -d- -f1`
CIPHERS=`for CODE in $CODES; do grep "^\"$CODE\"" tls-parameters-4.csv | cut -d, -f3; done`
echo $CIPHERS | sed 's/ /,/g'

Which gives us the string TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA.

Set your ciphers parameter to that. Note that even though the list contains GCM (Galois/Counter Mode) suites, the default Java 7 Sun JSSE provider does not implement them, and so they won't be used under Java 7. GCM suites should be implemented for Java 8 - see JEP 115. Also, if you are in a country that does not prohibit strong encryption, you need to install the JCE Unlimited Strength Jurisdiction Policy Files, otherwise you won't have any 256 bit ciphers.

With the above cipher string and Oracle Java 7, the SSL Server Test provides a score of 90 for key exchange and 90 for cipher strength. It also says that BEAST is not mitigated server-side, but that attack is mitigated on the client side for most clients - see BEAST information. Apple has also finally fixed this in Mavericks - see this post. However, Tomcat does not enforce the order of the ciphers in the list, and so for some reason all version of IE that are tested choose the RSA key exchange, even though they support ECDHE. Of course, RSA key exchange does not provide forward secrecy. We can further improve this by removing RSA key exchange from the cipher string:

ECDH+AESGCM:ECDH+AES256:ECDH+AES128:ECDH+3DES:!aNULL:!MD5:!DSS

Which results in this list of ciphers:

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA.

With this list, IE is forced to choose ECDHE for key exchange, which provides forward secrecy. IE6/XP, IE8/XP, and Java6u45 still fail because SSLv3 is disabled. However, Bing Oct 2013 handshake now also fails, as well as OpenSSL 0.9.8y (released February 5 2013). OpenSSL 1.0.1e works fine. If you are ok with forcing some clients to upgrade, leave out the RSA key exchange.

Tuesday, May 14, 2013

Encrypting Database Passwords In Ruby

Most web applications connect to a database, which means that they need to store the database password in a file. While that file should only be readable by the user running the application server, security can be further improved by encrypting the password in the file. The encryption passphrase is then stored in the code of your application. This means that an attacker would not only have to get access to the password file, but also to the application code to decrypt the password. Here's a simple implementation of AES 256 encryption/decryption in Ruby. I've run it in JRuby using Java 1.7, and it requires you to install "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files" for your JRE. On Mac OS, after extracting the zip file I had to copy US_export_policy.jar and local_policy.jar to /Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home/jre/lib/security/.

# file_crypto.rb

require 'openssl'

class FileCrypto
  def initialize(passphrase, iv_passphrase, salt)
    @key = OpenSSL::PKCS5.pbkdf2_hmac_sha1(passphrase, salt, 2000, 32)
    @iv = OpenSSL::PKCS5.pbkdf2_hmac_sha1(iv_passphrase, salt, 2000, 16)
  end

  def read(filepath)
    decrypt(IO.read(filepath))
  end

  def write(filepath, plaintext)
    IO.write(filepath, encrypt(plaintext))
  end

  def encrypt(data)
    cipher = OpenSSL::Cipher::AES256.new(:CBC)
    cipher.encrypt
    cipher.key = @key
    cipher.iv = @iv
    cipher.update(data) + cipher.final
  end

  def decrypt(data)
    cipher = OpenSSL::Cipher::AES256.new(:CBC)
    cipher.decrypt
    cipher.key = @key
    cipher.iv = @iv
    cipher.update(data) + cipher.final
  end
end

Then, you can encrypt test password like this:
require_relative 'file_crypto'

fc = FileCrypto.new(
  'j7U2 k92M%qc mw}f;T@6cAlH',
  'cZd !W7Xf2PMQY#Xprt)n',
  '28Ca kDl^vYuO'
)

fc.write('/tmp/encpassword', 'test password')

And decrypt it back like this:
require_relative 'file_crypto'

fc = FileCrypto.new(
  'j7U2 k92M%qc mw}f;T@6cAlH',
  'cZd !W7Xf2PMQY#Xprt)n',
  '28Ca kDl^vYuO'
)

puts fc.read('/tmp/encpassword')

Concurrent Tasks In JRuby

Sometimes you need to execute long running tasks (like database queries on multiple shards) concurrently. JRuby can easily make use of Java's built-in Executor class for this purpose.

# Since java.util.concurrent.ThreadPoolExecutor.submit has multiple signatures,
# we force JRuby to use the one signature we want, and stop a warning from being emitted.
java.util.concurrent.ThreadPoolExecutor.class_eval do
  java_alias :submit, :submit, [java.util.concurrent.Callable.java_class]
end

# Start the executor service.  Note that this won't create any threads right away,
# but will create new threads as the jobs are submitted, until it gets to 5 threads.
# Then it will reuse the threads.
executor = java.util.concurrent.Executors.new_fixed_thread_pool 5

# Submit jobs, get back java.util.concurrent.FutureTask objects.
tasks = [1, 3, 4, 6].map do |s|
  executor.submit do
    # run database query on shard s
    # return query result
  end
end

# We get control back immediately, while the queries are being executed by the thread pool.
# We can do some other things here while the queries run.

# Get the results of all queries and combine them into an array.  FutureTask.get will wait
# for the task to complete if it's not completed yet, so it's safe to call this right away.
# If for some reason we wanted to check if a task is complete, we can call t.done? .
results = tasks.map do |t|
  begin
    t.get
  rescue java.util.concurrent.ExecutionException => e
    # Convert the exception to one actually thrown inside the task.
    raise e.cause.exception
  end
end

# Shut down the thread pool.
executor.shutdown_now

Tuesday, October 16, 2012

PHP vs Node.js vs Native Speed Comparison

A quick speed comparison between PHP, Node.js and native code. This is obviously an artificial benchmark. The code generates four random numbers between 0 and 100 and uses them as coordinates of two points on a plane, and then calculates the distance between them. This is repeated 10 million times. Only the loop is timed, so process startup time is not part of the timing. The results are:

PHP - 12.692 seconds
Node.js - 0.328 seconds
Native - 0.308 seconds

All tests were run on a MacBook Pro. PHP is almost 40 times slower than Node.js. Node.js is almost as fast as native code, which makes sense since it does just-in-time compilation.

PHP code:
<?php

function distance($x1, $y1, $x2, $y2) {
  $dx = $x2 - $x1;
  $dy = $y2 - $y1;
  return sqrt($dx * $dx + $dy * $dy);
}

$start = microtime(TRUE);
for($i = 0; $i < 10000000; $i++) {
  $x1 = rand(0, 100);
  $x2 = rand(0, 100);
  $y1 = rand(0, 100);
  $y2 = rand(0, 100);
  distance($x1, $y1, $x2, $y2);
}
$stop = microtime(TRUE);

echo "Elapsed time: " . ($stop - $start) . " s\n";

Node.js code:
function distance(x1, y1, x2, y2) {
  var dx = x2 - x1;
  var dy = y2 - y1;
  return Math.sqrt(dx * dx + dy * dy);
}

var start = Date.now()/1000;
for(var i = 0; i < 10000000; i++) {
  x1 = Math.random() * 100;
  x2 = Math.random() * 100;
  y1 = Math.random() * 100;
  y2 = Math.random() * 100;
  distance(x1, y1, x2, y2);
}
var stop = Date.now()/1000;

console.log("Elapsed time: " + (stop - start) + " s\n");

C code, compiled with gcc -Wall:
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <sys/time.h>
#include <math.h>

double distance(double x1, double y1, double x2, double y2) {
  double dx, dy;
  dx = x2 - x1;
  dy = y2 - y1;
  return sqrt(dx * dx + dy * dy);
}

int main(void) {
  int i;
  double x1, x2, y1, y2;
  struct timeval start, stop;

  srand(time(NULL));
  gettimeofday(&start, NULL);
  for(i = 0; i < 10000000; i++) {
    x1 = rand() * 100;
    x2 = rand() * 100;
    y1 = rand() * 100;
    y2 = rand() * 100;
    distance(x1, y1, x2, y2);
  }
  gettimeofday(&stop, NULL);
  printf("Elapsed time: %.3f s\n", (stop.tv_sec-start.tv_sec) + (stop.tv_usec-start.tv_usec) / 1000000.0);
  return 0;
}

Sunday, October 7, 2012

MongoDB 2.2 Aggregation And MapReduce Performance

TL;DR - 1 million documents with readings across 20 sensors (1 reading per document), find the number of readings and their average per sensor. Aggregation framework does it in 4.1 seconds, map/reduce in 67.2 seconds, map/reduce with jsMode=true in 45.5 seconds, and reading all documents into PHP and doing it there takes 3.3 seconds. Conclusion - MongoDB's map/reduce has horrible performance for some reason.

Load the initial data with
time mongo load_initial.js
which on my laptop took 1m32.271s.

// load_initial.js
//
// Create a collection of documents where each document is
// { "sensor_id": 4, "ts": 123456.78, "reading": 2.3 }

for(var i=0; i < 1000000; i++) {
  sensor_id = Math.floor(Math.random() * 20 + 1); // Random int 1-20.
  ts = Math.random() * 3000000; // Random float from 0-2999999.9999
  reading = Math.random() * 100; // Random float 0.0-99.9999
  obj = {"sensor_id": sensor_id, "ts": ts, "reading": reading};
  db.sensors.save(obj);
}
Run
mongo --eval 'db.sensors.count()'
as a sanity check, which on my laptop printed
MongoDB shell version: 2.2.0
connecting to: localhost:27017/test
1000000
For each sensor, let's count the number of readings and find the average reading. First let's do it using the aggregation framework:
// count_avg_by_sensor_aggregate.js
//
// For each sensor, find the number of readings, and their average value.

res = db.sensors.aggregate(
  {
    $group: {
      _id: "$sensor_id",
      "readings": {$sum: 1},
      "average": {$avg: "$reading"}
    }
  }
).result.sort(
  function(a, b) {
    return a._id - b._id;
  }
);

printjson(res);
Run it with
time mongo count_avg_by_sensor_aggregate.js
I omitted the output, but this took 0m4.111s. Now let's do this with a map/reduce:
// count_avg_by_sensor_mr.js
//
// For each sensor, find the number of readings, and their average value.

if(typeof jsMode === "undefined") {
  jsMode = false;
}

res = db.sensors.mapReduce(
  function() {
    emit(this.sensor_id, { "readings": 1, "total": this.reading });
  },
  function(key, values) {
    var result = { "readings": 0, "total": 0 };
    for(var i = 0; i < values.length; i++) {
      result.readings += values[i].readings;
      result.total += values[i].total;
    }
    return result;
  },
  {
    "out": { "inline": 1 },
    "finalize": function(key, value) {
      return {
        "readings": value.readings,
        "average": value.total / value.readings
      };
    },
    "jsMode": jsMode
  }
).results;

newres = [];
for(var i = 0; i < res.length; i++) {
  newres.push(
    {
      // When jsMode is true, _id becomes a string for some reason.
      "_id": parseInt(res[i]._id),
      "readings": res[i].value.readings,
      "average": res[i].value.average
    }
  );
}

newres = newres.sort(
  function(a, b) {
    return a._id - b._id;
  }
);

printjson(newres);
Run it with
time mongo count_avg_by_sensor_mr.js
This took 1m7.171s . Now do the same thing, but use jsMode=true:
time mongo --eval jsMode=true count_avg_by_sensor_mr.js
This took 0m45.459s . And finally, let's do the same calculation in PHP:
<?php

// count_avg_by_sensor.php
//
// For each sensor, find the number of readings, and their average value.

$mongo = new Mongo('mongodb://localhost:27017', array(
  'db' => 'test'
));
$db = $mongo->selectDB('test');
$sensors = $db->selectCollection('sensors');

$cursor = $sensors->find();
$result = array();
foreach($cursor as $doc) {
  $_id = (int)$doc['sensor_id'];
  if(!array_key_exists($_id, $result)) {
    $result[$_id] = array("readings" => 0, "total" => 0);
  }
  $result[$_id]["readings"] += 1;
  $result[$_id]["total"] += $doc['reading'];
}
ksort($result);

$newresult = array();
foreach($result as $_id => $value) {
  $newresult[] = array(
    "_id" => $_id,
    "readings" => $value["readings"],
    "average" => $value["total"] / $value["readings"]
  );
}

echo json_encode($newresult, JSON_PRETTY_PRINT);
Run it:
time php count_avg_by_sensor.php
This took 0m3.313s .

Wednesday, September 26, 2012

Encrypted RAID Disk on OS X Mountain Lion and Mavericks

UPDATE: This works on Mavericks as well

The Disk Utility application does not allow you to create an encrypted filesystem on a RAID volume. However, it is possible from the command line. WARNING: this will erase everything on those disks - do a backup if you want any of that data. The basic method is to create an Apple RAID volume, create a coreStorage logical volume group on it, and then create an encrypted logical volume on the logical volume group.

These are the unformatted disks before RAID.



You can see them from the command line as disk1 and disk2.

$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *121.3 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:          Apple_CoreStorage                         120.5 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                                                   *2.0 TB     disk1
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                                                   *2.0 TB     disk2
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *120.2 GB   disk3

The coreStorage subsystem knows nothing about them yet, and only shows my Macintosh HD.

$ diskutil cs list
CoreStorage logical volume groups (1 found)
|
+-- Logical Volume Group B42959AC-207C-45CE-AC5B-A3B9E5289368
    =========================================================
    Name:         Macintosh HD
    Size:         120473067520 B (120.5 GB)
    Free Space:   0 B (0 B)
    |
    +-< Physical Volume 33B112ED-10BF-452E-BC96-1761AE2FFDC7
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk0s2
    |   Status:   Online
    |   Size:     120473067520 B (120.5 GB)
    |
    +-> Logical Volume Family BE76718E-765A-4797-B7FD-9B743B6E28E9
        ----------------------------------------------------------
        Encryption Status:       Unlocked
        Encryption Type:         AES-XTS
        Conversion Status:       Complete
        Conversion Direction:    -none-
        Has Encrypted Extents:   Yes
        Fully Secure:            Yes
        Passphrase Required:     Yes
        |
        +-> Logical Volume 46D952CD-311E-476E-8C19-CE2392FBABCE
            ---------------------------------------------------
            Disk:               disk3
            Status:             Online
            Size (Total):       120154296320 B (120.2 GB)
            Size (Converted):   -none-
            Revertible:         Yes (unlock and decryption required)
            LV Name:            Macintosh HD
            Volume Name:        Macintosh HD
            Content Hint:       Apple_HFS

Create a RAID volume from the disks by dragging them into the RAID set and giving it a name (StorageRAID).



Under Options, tell it to automatically rebuild RAID sets if you want to.



Click Create, and then Create again to confirm.



It will create the RAID volume and mount it.



Now back to the command line.

$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *121.3 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:          Apple_CoreStorage                         120.5 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:                 Apple_RAID                         2.0 TB     disk1s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk1s3
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk2
   1:                        EFI                         209.7 MB   disk2s1
   2:                 Apple_RAID                         2.0 TB     disk2s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk2s3
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *120.2 GB   disk3
/dev/disk4
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS StorageRAID           *2.0 TB     disk4

You can see that it created a disk4 as the RAID volume. Create a logical volume group named StorageLVG on disk4.

$ sudo diskutil cs createLVG StorageLVG disk4
Password:
Started CoreStorage operation
Unmounting AppleRAID set at disk4
Adding disk4 to Logical Volume Group
Creating Core Storage Logical Volume Group
Switching disk4 to Core Storage
Waiting for Logical Volume Group to appear
Discovered new Logical Volume Group "20D5D037-F88C-4F05-AD28-E569E9564FC0"
Core Storage LVG UUID: 20D5D037-F88C-4F05-AD28-E569E9564FC0
Finished CoreStorage operation

If you do a diskutil cs list now, you'll see the new LVG with the same UUID as above.

$ diskutil cs list
CoreStorage logical volume groups (2 found)
|
+-- Logical Volume Group B42959AC-207C-45CE-AC5B-A3B9E5289368
|   =========================================================
|   Name:         Macintosh HD
|   Size:         120473067520 B (120.5 GB)
|   Free Space:   0 B (0 B)
|   |
|   +-< Physical Volume 33B112ED-10BF-452E-BC96-1761AE2FFDC7
|   |   ----------------------------------------------------
|   |   Index:    0
|   |   Disk:     disk0s2
|   |   Status:   Online
|   |   Size:     120473067520 B (120.5 GB)
|   |
|   +-> Logical Volume Family BE76718E-765A-4797-B7FD-9B743B6E28E9
|       ----------------------------------------------------------
|       Encryption Status:       Unlocked
|       Encryption Type:         AES-XTS
|       Conversion Status:       Complete
|       Conversion Direction:    -none-
|       Has Encrypted Extents:   Yes
|       Fully Secure:            Yes
|       Passphrase Required:     Yes
|       |
|       +-> Logical Volume 46D952CD-311E-476E-8C19-CE2392FBABCE
|           ---------------------------------------------------
|           Disk:               disk3
|           Status:             Online
|           Size (Total):       120154296320 B (120.2 GB)
|           Size (Converted):   -none-
|           Revertible:         Yes (unlock and decryption required)
|           LV Name:            Macintosh HD
|           Volume Name:        Macintosh HD
|           Content Hint:       Apple_HFS
|
+-- Logical Volume Group 20D5D037-F88C-4F05-AD28-E569E9564FC0
    =========================================================
    Name:         StorageLVG
    Size:         2000054943744 B (2.0 TB)
    Free Space:   1999736168448 B (2.0 TB)
    |
    +-< Physical Volume AB193FA5-822F-479B-9D74-AAEC1BC22632
        ----------------------------------------------------
        Index:    0
        Disk:     disk4
        Status:   Online
        Size:     2000054943744 B (2.0 TB)

In diskutil list you can see that it changed the type of StorageRAID from Apple_HFS to Apple_CoreStorage.

$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *121.3 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:          Apple_CoreStorage                         120.5 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:                 Apple_RAID                         2.0 TB     disk1s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk1s3
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk2
   1:                        EFI                         209.7 MB   disk2s1
   2:                 Apple_RAID                         2.0 TB     disk2s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk2s3
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *120.2 GB   disk3
/dev/disk4
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:         Apple_CoreStorage StorageRAID            *2.0 TB     disk4

Create an encrypted logical volume on this new LVG.

$ sudo diskutil cs createLV 20D5D037-F88C-4F05-AD28-E569E9564FC0 jhfs+ Storage 100% -stdinpassphrase
Passphrase for new volume:
Started CoreStorage operation
Waiting for Logical Volume to appear
Formatting file system for Logical Volume
Initialized /dev/rdisk5 as a 2 TB HFS Plus volume with a 155648k journal
Mounting disk
Core Storage LV UUID: F490C159-4CAB-463A-BAB8-3A6468CF1FE5
Core Storage disk: disk5
Finished CoreStorage operation

If you look at diskutil cs list now, you'll see the new volume.

$ diskutil cs list
CoreStorage logical volume groups (2 found)
|
+-- Logical Volume Group B42959AC-207C-45CE-AC5B-A3B9E5289368
|   =========================================================
|   Name:         Macintosh HD
|   Size:         120473067520 B (120.5 GB)
|   Free Space:   0 B (0 B)
|   |
|   +-< Physical Volume 33B112ED-10BF-452E-BC96-1761AE2FFDC7
|   |   ----------------------------------------------------
|   |   Index:    0
|   |   Disk:     disk0s2
|   |   Status:   Online
|   |   Size:     120473067520 B (120.5 GB)
|   |
|   +-> Logical Volume Family BE76718E-765A-4797-B7FD-9B743B6E28E9
|       ----------------------------------------------------------
|       Encryption Status:       Unlocked
|       Encryption Type:         AES-XTS
|       Conversion Status:       Complete
|       Conversion Direction:    -none-
|       Has Encrypted Extents:   Yes
|       Fully Secure:            Yes
|       Passphrase Required:     Yes
|       |
|       +-> Logical Volume 46D952CD-311E-476E-8C19-CE2392FBABCE
|           ---------------------------------------------------
|           Disk:               disk3
|           Status:             Online
|           Size (Total):       120154296320 B (120.2 GB)
|           Size (Converted):   -none-
|           Revertible:         Yes (unlock and decryption required)
|           LV Name:            Macintosh HD
|           Volume Name:        Macintosh HD
|           Content Hint:       Apple_HFS
|
+-- Logical Volume Group 20D5D037-F88C-4F05-AD28-E569E9564FC0
    =========================================================
    Name:         StorageLVG
    Size:         2000054943744 B (2.0 TB)
    Free Space:   0 B (0 B)
    |
    +-< Physical Volume AB193FA5-822F-479B-9D74-AAEC1BC22632
    |   ----------------------------------------------------
    |   Index:    0
    |   Disk:     disk4
    |   Status:   Online
    |   Size:     2000054943744 B (2.0 TB)
    |
    +-> Logical Volume Family AC7F549F-1D6F-4E22-B050-34791ABF53FB
        ----------------------------------------------------------
        Encryption Status:       Unlocked
        Encryption Type:         AES-XTS
        Conversion Status:       Complete
        Conversion Direction:    -none-
        Has Encrypted Extents:   Yes
        Fully Secure:            Yes
        Passphrase Required:     Yes
        |
        +-> Logical Volume F490C159-4CAB-463A-BAB8-3A6468CF1FE5
            ---------------------------------------------------
            Disk:               disk5
            Status:             Online
            Size (Total):       1999736168448 B (2.0 TB)
            Size (Converted):   -none-
            Revertible:         No
            LV Name:            Storage
            Volume Name:        Storage
            Content Hint:       Apple_HFS

And in diskutil list.

$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *121.3 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:          Apple_CoreStorage                         120.5 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk1
   1:                        EFI                         209.7 MB   disk1s1
   2:                 Apple_RAID                         2.0 TB     disk1s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk1s3
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 TB     disk2
   1:                        EFI                         209.7 MB   disk2s1
   2:                 Apple_RAID                         2.0 TB     disk2s2
   3:                 Apple_Boot Boot OS X               134.2 MB   disk2s3
/dev/disk3
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *120.2 GB   disk3
/dev/disk4
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:          Apple_CoreStorage StorageRAID            *2.0 TB     disk4
/dev/disk5
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Storage                *2.0 TB     disk5

This is what it looks like in Disk Utility after everything is done.