Hashcat in AWS EC2

Intro

During my OSCP studies, I realized I needed a more efficient system for cracking password hashes. The screaming CPU fans and high CPU usage became a problem. I first tried using hashcat and the GPU on my MacBook Pro in OS X. There are some bugs and problems with hashcat on OS X that would make it crash in the middle of cracking a hash. Also, I was not interested in investing a server with a bunch of GPUs, the high costs to do this would outweigh the amount of time I need the system. All of this lead me to do a little research and found the instructions in the following link to build an AWS instance for password cracking.

https://medium.com/@iraklis/running-hashcat-v4-0-0-in-amazons-aws-new-p3-16xlarge-instance-e8fab4541e9b

Since that post was created there have been some changes to the offerings in AWS EC2 leading me write this post.

If you wish to skip ahead I have created scripts to automate the processes in the rest of this post. They are both in my github and can be downloaded at the following links.

https://github.com/suidroot/AWSScripts/blob/master/aws-ec2-create-kracker.sh
https://github.com/suidroot/AWSScripts/blob/master/configure-kracker.sh

For the rest of the article I will cover some of the instance options in EC2, installation of the needed Linux packages, the basic setup of Hashcat, running Hashcat, and finally monitoring and benchmarks of an EC2 instance.

AWS EC2 Options

There are many options for EC2 instances, they have a huge range in cost and scale.

• G3 – https://aws.amazon.com/ec2/instance-types/g3/
• P2 – https://aws.amazon.com/ec2/instance-types/p2/

I found the g3 instances to be the more cost effective tier. For my testing I opted to use the g3.4xlarge tier. Next to choose the AMI image, appropriate the appropriate operating system.

AMI images

There are two options that are I tested hashcat on they are both Ubuntu based. I’m sure there are many other available options that will work too, but I am familiar with Ubuntu systems. The first option is a standard Ubuntu image, there is nothing special about this image and it requires configuration to add the GPU drivers and a little more work.

The next option is a Deep Learning image, this image is preconfigured with the GPU drivers and was originally designed for machine learning applications. I found the the pre-configuration allowed for me skip a few steps in building out a new system.

Instance Build and config

Once you have the instance deployed there are a few steps to get the Instance prepared for hashcat, the steps are a little bit different between a Standard and a Deep Learning Ubuntu instance.

An apt cronjob may already be running and you will have to wait it out.

Prepare Machine (Standard Ubuntu)

This script will install all the required packages and the Nvidia GPU drivers on a vanilla Ubuntu installation.

#!/bin/bash

# mostly copied from: https://medium.com/@iraklis/running-hashcat-v4-0-0-in-amazons-aws-new-p3-16xlarge-instance-e8fab4541e9b
#
sudo apt-get update -yq
sudo apt-get install -yq build-essential linux-headers-$(uname -r) unzip p7zip-full linux-image-extra-virtual
sudo apt-get install -yq ocl-icd-libopencl1 opencl-headers clinfo
#sudo apt-get install -yq libhwloc-plugins libhwloc5 libltdl7 libpciaccess0 libpocl2 libpocl2-common ocl-icd-libopencl1 pocl-opencl-icd
sudo apt-get install -yq python3-pip 
pip3 install psutil

sudo touch /etc/modprobe.d/blacklist-nouveau.conf
sudo bash -c "echo 'blacklist nouveau' >> /etc/modprobe.d/blacklist-nouveau.conf"
sudo bash -c "echo 'blacklist lbm-nouveau' >> /etc/modprobe.d/blacklist-nouveau.conf"
sudo bash -c "echo 'options nouveau modeset=0' >> /etc/modprobe.d/blacklist-nouveau.conf"
sudo bash -c "echo 'alias nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf"
sudo bash -c "echo 'alias lbm-nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf"

sudo touch /etc/modprobe.d/nouveau-kms.conf
sudo bash -c "echo 'options nouveau modeset=0' >>  /etc/modprobe.d/nouveau-kms.conf"
sudo update-initramfs -u
sudo reboot

### Install nVidia Drivers
wget http://us.download.nvidia.com/tesla/410.104/NVIDIA-Linux-x86_64-410.104.run
sudo /bin/bash NVIDIA-Linux-x86_64-410.104.run --ui=none --no-questions --silent -X

Prepare Machine (Deep Learning Ubuntu)

In comparison the previous script there is a much simpler script to prepare the Deep Learning instance. The main focus is installing the needed archive extraction tools.

#!/bin/bash

sudo apt update
sudo apt upgrade
sudo apt install clinfo unzip p7zip-full
sudo apt install build-essential linux-headers-$(uname -r) # Optional 
sudo apt-get install -yq python3-pip 
pip3 install psutil

Hashcat Setup

Now we need to download and extract the star of the show Hashcat. The link in the wget below points to the the most recent version as of writing however you might want to check to see if there is a more recent version at the main site: https://hashcat.net/hashcat/

wget https://hashcat.net/files/hashcat-5.1.0.7z
7z x hashcat-5.1.0.7z

Download wordlists

You will need some wordlists for hashcat to use to crack passwords, he commands listed are for some wordlists I like to use when cracking. You should however add whichever lists are your favories.

mkdir ~/wordlists
git clone https://github.com/danielmiessler/SecLists.git ~/wordlists/seclists
wget -nH http://downloads.skullsecurity.org/passwords/rockyou.txt.bz2 -O ~/wordlists/rockyou.txt.bz2
cd ~/wordlists
bunzip2 ./rockyou.txt.bz2
cd ~

Running hashcat

Now it is time to run hashcat and crack some passwords. When running hashcat I had the best performance with the arguments-O -w 3. Below is an example command line I’ve used inclusing a rules file.

./hashcat-5.1.0/hashcat64.bin --username -m 1800 ./megashadow256.txt wordlists/rockyou.txt -r hashcat-5.1.0/rules/best64.rule -O -w 3

Monitoring the Nvidia GPU

The nvidia-smi utility can be used to show the GPU processor usage and what processes are utilizing the GPU(s). The first example is is showing an idle GPU.

ubuntu@ip-172-31-17-6:~$ sudo nvidia-smi
Fri Apr 26 14:43:49 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   37C    P0    42W / 150W |      0MiB /  7618MiB |     97%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

This example shows a GPU being used by hashcat.

ubuntu@ip-172-31-17-6:~$ sudo nvidia-smi
Fri Apr 26 14:44:44 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   46C    P0   141W / 150W |    828MiB /  7618MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     11739      C   ./hashcat-5.1.0/hashcat64.bin                817MiB |
+-----------------------------------------------------------------------------+

Conclusion and Benchmarks

Finally here is a benchmark I ran on a g3.4xlarge instance. This instance type contains 1 GPU. These results give an idea of performance for this AWS EC2 instance type.

ubuntu@ip-172-31-17-6:~$ ./hashcat-5.1.0/hashcat64.bin -O -w 3 -b
hashcat (v5.1.0) starting in benchmark mode...

* Device #2: Not a native Intel OpenCL runtime. Expect massive speed loss.
             You can use --force to override, but do not report related errors.
nvmlDeviceGetFanSpeed(): Not Supported

OpenCL Platform #1: NVIDIA Corporation
======================================
* Device #1: Tesla M60, 1904/7618 MB allocatable, 16MCU

OpenCL Platform #2: The pocl project
====================================
* Device #2: pthread-Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz, skipped.

Benchmark relevant options:
===========================
* --optimized-kernel-enable
* --workload-profile=3

Hashmode: 0 - MD5

Speed.#1.........: 11611.6 MH/s (90.74ms) @ Accel:512 Loops:512 Thr:256 Vec:4

Hashmode: 100 - SHA1

Speed.#1.........:  4050.2 MH/s (65.01ms) @ Accel:512 Loops:128 Thr:256 Vec:2

Hashmode: 1400 - SHA2-256

Speed.#1.........:  1444.5 MH/s (91.98ms) @ Accel:256 Loops:128 Thr:256 Vec:1

Hashmode: 1700 - SHA2-512

Speed.#1.........:   499.4 MH/s (66.78ms) @ Accel:128 Loops:64 Thr:256 Vec:1

Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4096)

Speed.#1.........:   189.8 kH/s (42.76ms) @ Accel:128 Loops:64 Thr:256 Vec:1

Hashmode: 1000 - NTLM

Speed.#1.........: 18678.1 MH/s (56.58ms) @ Accel:512 Loops:512 Thr:256 Vec:2

Hashmode: 3000 - LM

Speed.#1.........: 10529.6 MH/s (50.60ms) @ Accel:128 Loops:1024 Thr:256 Vec:1

Hashmode: 5500 - NetNTLMv1 / NetNTLMv1+ESS

Speed.#1.........: 10650.8 MH/s (49.60ms) @ Accel:512 Loops:256 Thr:256 Vec:1

Hashmode: 5600 - NetNTLMv2

Speed.#1.........:   829.3 MH/s (80.24ms) @ Accel:256 Loops:64 Thr:256 Vec:1

Hashmode: 1500 - descrypt, DES (Unix), Traditional DES

Speed.#1.........:   442.0 MH/s (37.81ms) @ Accel:4 Loops:1024 Thr:256 Vec:1

Hashmode: 500 - md5crypt, MD5 (Unix), Cisco-IOS $1$ (MD5) (Iterations: 1000)

Speed.#1.........:  4209.1 kH/s (51.39ms) @ Accel:1024 Loops:500 Thr:32 Vec:1

Hashmode: 3200 - bcrypt $2*$, Blowfish (Unix) (Iterations: 32)

Speed.#1.........:     7572 H/s (33.02ms) @ Accel:16 Loops:4 Thr:8 Vec:1

Hashmode: 1800 - sha512crypt $6$, SHA512 (Unix) (Iterations: 5000)

Speed.#1.........:    76958 H/s (83.99ms) @ Accel:512 Loops:128 Thr:32 Vec:1

Hashmode: 7500 - Kerberos 5 AS-REQ Pre-Auth etype 23

Speed.#1.........:   149.4 MH/s (56.00ms) @ Accel:128 Loops:64 Thr:64 Vec:1

Hashmode: 13100 - Kerberos 5 TGS-REP etype 23

Speed.#1.........:   152.1 MH/s (55.00ms) @ Accel:128 Loops:64 Thr:64 Vec:1

Hashmode: 15300 - DPAPI masterkey file v1 (Iterations: 23999)

Speed.#1.........:    32703 H/s (84.02ms) @ Accel:256 Loops:64 Thr:256 Vec:1

Hashmode: 15900 - DPAPI masterkey file v2 (Iterations: 7999)

Speed.#1.........:    21692 H/s (96.24ms) @ Accel:256 Loops:128 Thr:32 Vec:1

Hashmode: 7100 - macOS v10.8+ (PBKDF2-SHA512) (Iterations: 35000)

Speed.#1.........:     5940 H/s (40.09ms) @ Accel:64 Loops:32 Thr:256 Vec:1

Hashmode: 11600 - 7-Zip (Iterations: 524288)

Speed.#1.........:     4522 H/s (55.87ms) @ Accel:256 Loops:128 Thr:256 Vec:1

Hashmode: 12500 - RAR3-hp (Iterations: 262144)

Speed.#1.........:    18001 H/s (56.74ms) @ Accel:4 Loops:16384 Thr:256 Vec:1

Hashmode: 13000 - RAR5 (Iterations: 32767)

Speed.#1.........:    18135 H/s (55.93ms) @ Accel:128 Loops:64 Thr:256 Vec:1

Hashmode: 6211 - TrueCrypt PBKDF2-HMAC-RIPEMD160 + XTS 512 bit (Iterations: 2000)

Speed.#1.........:   121.7 kH/s (59.39ms) @ Accel:128 Loops:32 Thr:256 Vec:1

Hashmode: 13400 - KeePass 1 (AES/Twofish) and KeePass 2 (AES) (Iterations: 6000)

Speed.#1.........:    68380 H/s (158.89ms) @ Accel:512 Loops:256 Thr:32 Vec:1

Hashmode: 6800 - LastPass + LastPass sniffed (Iterations: 500)

Speed.#1.........:  1088.7 kH/s (48.51ms) @ Accel:128 Loops:62 Thr:256 Vec:1

Hashmode: 11300 - Bitcoin/Litecoin wallet.dat (Iterations: 199999)

Speed.#1.........:     2107 H/s (78.97ms) @ Accel:128 Loops:64 Thr:256 Vec:1

Started: Fri Apr 26 14:36:56 2019
Stopped: Fri Apr 26 14:42:03 2019

If you’ve made it this far congratulation and happy cracking!