So... what's the problem?
I am currently working with some Elastic Stack clustering when I quickly realized that, if two or more nodes traverse a security boundary, they may be subject to tampering if an evil man (or woman) in the middle were to intercept and modify the data sent between them. Yes, there is a solution from Elastic, called X-Pack Security, that can provide SSL between the nodes, but that's one of the rare things that Elastic charges for. The reason I was even pondering the use of Elastic was to replace a certain, well-known data aggregation solution that is eating up quite a bit of budget (and even more memory)
Below shows the traffic between two nodes (10.1.2.230 and 10.1.2.231). This data sent over TCP port 9300 is used for Elasticsearch cluster communication as is simply REST over HTTP. What could possibly go wrong? This communication could be things such as node to node conversations, replication of data, or other important cluster information. If this data were to be captured, attackers could get some valuable intelligence regarding the systems supported by this log aggregation service. Worse yet, if this communication were to be poisoned via a man-in-the-middle attack, the entire log aggregation service could be deemed useless.
Below shows the traffic between two nodes (10.1.2.230 and 10.1.2.231). This data sent over TCP port 9300 is used for Elasticsearch cluster communication as is simply REST over HTTP. What could possibly go wrong? This communication could be things such as node to node conversations, replication of data, or other important cluster information. If this data were to be captured, attackers could get some valuable intelligence regarding the systems supported by this log aggregation service. Worse yet, if this communication were to be poisoned via a man-in-the-middle attack, the entire log aggregation service could be deemed useless.
What as the solution?
After a ton of Googling, I stumbled upon a solution that has been very well known in the Red Hat community for some time now that, honestly, I should have been aware of - LibreSwan. LibreSwan is a free, open-source package that allows for host-to-host, host-to-network, and network-to-network IPSEC tunneling. This would be perfect! After some trial and error, I decided to use the Pre-Shared Key implementation to test it out. Here's the very simple setup and configuration of LibreSwan:
1) Install Libreswan on each node:
1) Install Libreswan on each node:
# yum -y install libreswan
2) Configure a tunnel (in this case, it is named mytunnel and the config file is located at /etc/ipsec.d/my_host-to-host.conf):
# vi /etc/ipsec.d/my_host-to-host.conf
conn mytunnel
left=10.1.2.230
right=10.1.2.231
authby=secret
auto=start
3) Create a random, base64 key of 48 total bytes (only need to do this on one machine as the keys MUST match on both sides. Doing this twice should not yield the same key):
# openssl rand -base64 48
n8+ef4PA4VAtqd1iX7QzC3sLmxlLi30LzOTgg7JBmNXQ7Wsi8SnweO+hjlXNK/rE
4) Create a "secrets" file with the above command output as the pre-shared key (Note: this is one, continuous line):
# vi /etc/ipsec.d/es.secrets
10.1.2.230 10.1.2.231 : PSK "n8+ef4PA4VAtqd1iX7QzC3sLmxlLi30LzOTgg7JBmNXQ7Wsi8SnweO+hjlXNK/rE"
5) Enable and start the Libreswan service:
# systemctl enable ipsec
# systemctl start ipsec
2) Configure a tunnel (in this case, it is named mytunnel and the config file is located at /etc/ipsec.d/my_host-to-host.conf):
# vi /etc/ipsec.d/my_host-to-host.conf
conn mytunnel
left=10.1.2.230
right=10.1.2.231
authby=secret
auto=start
3) Create a random, base64 key of 48 total bytes (only need to do this on one machine as the keys MUST match on both sides. Doing this twice should not yield the same key):
# openssl rand -base64 48
n8+ef4PA4VAtqd1iX7QzC3sLmxlLi30LzOTgg7JBmNXQ7Wsi8SnweO+hjlXNK/rE
4) Create a "secrets" file with the above command output as the pre-shared key (Note: this is one, continuous line):
# vi /etc/ipsec.d/es.secrets
10.1.2.230 10.1.2.231 : PSK "n8+ef4PA4VAtqd1iX7QzC3sLmxlLi30LzOTgg7JBmNXQ7Wsi8SnweO+hjlXNK/rE"
5) Enable and start the Libreswan service:
# systemctl enable ipsec
# systemctl start ipsec
As you can see below, the Wireshark output is now showing ESP communication between the two endpoints! As long as that pre-shared key is kept secret, we're good (although it may be a good idea to rotate this key occasionally so offline brute force attacks would be less successful).
What's next?
I should probably move away from the pre-shared key implementation and onto getting this to work with public/private key so I don't need to worry about secure transmission of the pre-shared key in production. Other than that, this solution seems pretty solid as I've been sending quite a bit of data at my Elastic Stack implementation and it's replicating across the cluster rather seamlessly!
