Automating Importing Let's Encrypt Certificates into Payara/Glassfish

Well, I already made a blogpost on letsencrypt certificates in payara¹.

In this blogpost I mentioned a python script that automatically renews certificates for payara². Unfortunately, it's not quite a perfect fit for me, as I do not run payara on port 80, which the script requires.

My setup has an nginx running on port 80, that basically forwards all requests to the https version of the website.

So I could simply turn off nginx, re-certify and turn nginx back on.

This simple bash script is as follows:

Running it will give you:

$ source renew_letsencrypt_certificates.sh
---- Starting.
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.mrbear.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.mrbear.org
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.mrbear.org/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/www.mrbear.org/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
---- Adding certificates...
/home/payara/payara5/glassfish/domains/production/config/keystore.jks
Private key with alias [s1as] added to keystore /home/payara/payara5/glassfish/domains/production/config/keystore.jks.
Command add-pkcs8 executed successfully.
---- Disabling http listener...
server.network-config.network-listeners.network-listener.http-listener-2.enabled=false
Command set executed successfully.
---- Enabling http listener...
server.network-config.network-listeners.network-listener.http-listener-2.enabled=true
Command set executed successfully.
---- Done.

And your certificate is renewed. I am most pleased that this is soo easy in Payara/Glassfish.

In order to properly automate this, the master password must be stored somehow, and it's possible to do this via the --passwordfile=passwordfile.txt commandline parameter.

I found a good blogpost on how to do this at .Lost in Coding³.

And also the blogpost on [4] to get me started on how to do this.

Verifying that the keystore contains the appropriate stuff, can be done as follows:

# keytool -list -keystore /home/payara/payara5/glassfish/domains/production/config/keystore.jks
Enter keystore password:
Keystore type: jks
Keystore provider: SUN

Your keystore contains 2 entries

glassfish-instance, Jan 22, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): WZ:B6:4N:8V:AT:YP:QC:9N:VT:HA:WI:NQ:B[:V8:0W:YT:B8:AW:YT:VQ
s1as, Sep 25, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): N8:5Y:AN:SV:N8:9W:4T:Y7:9W:V9:W4:VT:8Y:WV:4N:W4:VT:FB:8S:4E

To get more information, try:

keytool -list -v -keystore /home/payara/payara5/glassfish/domains/production/config/keystore.jks

Or for a specific alias:

keytool -list -v -keystore /home/payara/payara5/glassfish/domains/production/config/keystore.jks -alias s1as

References

[1] Enabling SSL in Payara with certificate from Let's Encrypt.

https://randomthoughtsonjavaprogramming.blogspot.com/2019/01/enabling-ssl-in-payara-with-certificate.html

[2] github - Let's encrypt script to retrieve and upload keys/certificates to Payara

https://gist.github.com/ratcashdev/1b09877d37e02ef5170bf9e60c377f34

[3] .Lost in Coding - Configure Payara Server passwords in Docker

https://ondro.inginea.eu/index.php/configure-passwords-in-payara-server-and-glassfish/

[4] Payara Blog - TLS certificates with Payara Server and Let's Encrypt

https://blog.payara.fish/configuring-ssl/tls-certificates-with-payara-server-and-lets-encrypt

Nginx - Redirecting to HTTPS

For automatic http->https redirection it is recommended to use Apache or Nginx in front of Payara to do the redirection, see [1].

I decided on using nginx for this^{2 3 4}.

Before nginx, it looks like this:

$ curl -s -D - http://www.mrbear.org -o /dev/null
HTTP/1.1 200 OK
Server: Payara Server  5.184 #badassfish
X-Powered-By: Servlet/4.0 JSP/2.3 (Payara Server  5.184 #badassfish Java/Oracle Corporation/1.8)
Content-Type: text/html;charset=ISO-8859-1
Transfer-Encoding: chunked
X-Frame-Options: SAMEORIGIN

Configuration is found in /etc/nginx/nginx.conf.

I changed it into something very simple:

server {
      listen      80 default;
      access_log off;
      error_log off;
      return      301 https://$server_name$request_uri;
}

Double checking:

# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Enabling automatic start and also starting nginx.

# systemctl enable nginx
# systemctl start nginx

So, let's try seeing what the webserver now responds with:

$ curl -s -D - http://www.mrbear.org -o /dev/null
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.2
Date: Mon, 28 Jan 2019 16:28:59 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://www.mrbear.org/

$ curl -s -D - http://www.mrbear.org/this/is/some/url.jsf -o /dev/null
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.2
Date: Mon, 28 Jan 2019 16:31:06 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://www.mrbear.org/this/is/some/url.jsf

References

[1] Payara - How to Secure Payara Server with Apache

https://blog.payara.fish/how-to-secure-payara-server-with-apache

[2] How To Nginx Redirect All HTTP Request To HTTPS Rewrite 301 Rules

https://www.cyberciti.biz/faq/linux-unix-nginx-redirect-all-http-to-https/

[3] How to do an Nginx redirect

https://bjornjohansen.no/nginx-redirect

[4] Creating NGINX Rewrite Rules

https://www.nginx.com/blog/creating-nginx-rewrite-rules/

DigitalOcean - How To Install Nginx on CentOS 7

https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-centos-7

Running Glassfish on port 80

Well, in short. Linux does not allow non-root programs to listen/occupy ports below 1024. It is a security feature.

So what can I do to make a server that is non-root (because of security) able to receive/send messages through a privileged port? Luckily the question has already been asked multiple times on the Internet.

To summarize:

• Run the program as root anyway
• Have a root daemon run the program as root, then change the rootrights automatically to something less
• Kernel param CAP_NET_BIND_SERVICE
• Run a proxy server in front of it
• Software that performs what you want
• Port-forwarding using IPTables

For the matter of this blog, I am going to assume that I wish to make my Glassfish instance available via both port 80 (the default web port) and port 443 (the default ssl web port). For that is the use case I was faced with.

Run the program as root anyway

Well, that should work, but is a huge security risk in most cases.

We would just need to change the ports of the HTTP Listeners in Glassfish administration console. It can be found in the default-config option.

Have a root daemon run the program as root, then change the rootrights automatically to something less

This seems to be one of the functions of the inetd daemon.

Kernel param CAP_NET_BIND_SERVICE

sudo setcap cap_net_bind_service=ep some-binary

In Linux, the things root can do have been broken up into a set of capabilities. CAP_NET_BIND_SERVICE is the ability to bind to ports <= 1024.

Unfortunately, we would need to set the binary to "java". This automatically means that any java program will have the ability to access any ports. This is not what we want.

Run a proxy server in front of it

That's what I did initially. There are lots of web servers that are perfectly capable of redirecting traffic to another port. For example the Apache HTTP server and Nginx. I choose Nginx, as it seems to be quite easy to setup and doesn't come with a lot of extras, extras I do not need.

The excellent part of this solution is that we can use Nginx to serve static webpages/images/files and not bother the Glassfish with it.

Tricky is that I need the original originating IP address of the client, and that is lost without some magic configuration.

The bad part is that a proxy server is another link in the chain that needs to be supported and maintained etc.

Software that performs what you want

For example authbind seems mentioned a lot.

Port-forwarding using IPTables

This is the solution I went for eventually.

iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080
iptables -t nat -A PREROUTING -p udp -m udp --dport 80 -j REDIRECT --to-ports 8080 

# Do the same thing for ssl:
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8181
iptables -t nat -A PREROUTING -p udp -m udp --dport 443 -j REDIRECT --to-ports 8181

Removing all the rules can be done with:

echo "Stopping firewall and allowing everyone..."

iptables -F

iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

To save the configuration, you can use iptables-save and iptables-restore. Or try something like:

]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]

References

StackOverflow - Is there a way for non-root processes to bind to “privileged” ports (<1024) on Linux?

http://stackoverflow.com/questions/413807/is-there-a-way-for-non-root-processes-to-bind-to-privileged-ports-1024-on-l#414258

Debian Administration - Running network services as a non-root user

http://www.debian-administration.org/article/386/Running_network_services_as_a_non-root_user

Ubuntu Help - Iptables HowTo

https://help.ubuntu.com/community/IptablesHowTo

Oracle Blogs - Byron's Pointless Ramblings - Run GlassFish V3 as a non-root Service on Linux Ubuntu/Debian

https://blogs.oracle.com/foo/entry/run_glassfish_v3_as_a

The JavaDude Weblog - Glassfish and https running secure applications

https://javadude.wordpress.com/2013/03/25/glassfish-and-https-running-secure-applications/