How to prevent Jetty, deployed via Docker on AWS ECS, behind an AWS ELB, redirecting from secure HTTPS to insecure HTTP
So this is your setup. You've using Jetty to serve your web application, and you're deploying it via Docker to AWS Elastic Container Service (ECS), and you place it behind an AWS Elastic Load Balancer (ELB). You aren't living in the past so you enable HTTPS on the ELB. Your clients connect via HTTPS to the ELB, and the ELB connects to your application via HTTP internally to AWS.
So the last thing you want is for users to receive a redirect from secure HTTPS to insecure HTTP. But that's exactly what you're seeing happening. Unfortunately HTTPS, AWS ELB, together with Apache or nginx or Tomcat or Jetty is broken by default.
In the past, when I used root servers, I used Apache or nginx as a load balancer, to terminate the HTTPS connection, in place of the ELB that I use when using AWS. Those load balancers work slightly differently to ELB (and this confused me for a long time; I expected ELB to work the same.)
Apache will terminate the HTTPS connection, and send the request to your backend application web server, using HTTP. The backend web server thinks its address is e.g.
http://192.168.0.23:80/. If the backend web server needs to do a redirect it will emit a redirect to e.g.http://192.168.0.23:80/foo. Apache will accept this response, and rewrite it intohttps://mysite.com/foo.ELB also terminates the HTTPS connection, sends the request to the backend server via HTTP. However, whatever redirects the backend emits, it passes those one-to-one on to the client. If the backend redirects to
http://192.168.0.23:80/foo, the user's browser will see a redirect tohttp://192.168.0.23:80/foo, which obviously won't work.
What ELB expects to happen, is that it puts the original domain name and protocol in extra HTTP headers. The backend is supposed to understand these headers, and emit redirects to the correct domain and protocol. So the backend server, even though it accepts a request at http://192.168.0.23:80/, should parse headers saying the original request from the user was to the domain mysite.com and the protocol was https, and not emit redirects to http://192.168.0.23:80/foo but to https://mysite.com/foo.
The bad news is, that neither Jetty, Tomcat, Apache, nginx support this out of the box.
Jetty does support accepting the domain name over HTTP headers, but not the protocol. So if you go to https://mysite.com/webapp it will redirect to http://mysite.com/webapp/ i.e. the redirect is to put the extra slash on, the domain name is right, but the protocol has changed from HTTPS to insecure HTTP.
I think one reason this issue isn't more well-known is that most people have an automatic redirect from HTTP to HTTPS. So, I think, for most people, their users make a request to HTTPS and get redirected to another location but with insecure HTTP, and then get redirected again to (secure) HTTPS. Unless you have the Chrome inspector open, you don't realize this extra redirect is happening. You think your application is working. But actually, it's redirecting via an insecure protocol. And that's important. If you didn't need security, you could just host the entire application with HTTP.
curl -v https://support.coinbase.com/customer/portal/articles
< HTTP/1.1 301 Moved Permanently
< ...
< Location: http://support.coinbase.com/
< Server: nginx
I found this issue out because I had an application that only ran on HTTPS. For security, we didn't do any redirects from HTTP to HTTPS, or accept HTTP traffic in any way.
If you're using the standard jetty Docker container at https://hub.docker.com/_/jetty/, the bad news is that you are not "root" in the Docker container. So you get "permission denied" when you try to alter certain configuration entries. You can't edit the XML configuration entries, instead all you can edit is the INI configuration entries which wrap and reference those XML configuration entries.
So this is what your Jetty Dockerfile needs to look like:
FROM jetty:9.4.18-alpine
ADD myapp.war /var/lib/jetty/webapps/ROOT.war
...
USER root
RUN echo "--module=http-forwarded" > /var/lib/jetty/start.d/http-forwarded.ini
USER jetty
...