How to Configure Jenkins Behind NGINX with SSL

For a production server, I have been trying to get jenkins running behind NGINX a reverse proxy with SSL enabled. What I faced is that there are many configurations available in wiki.jenkins.io but mostly those configurations worked for the authors with different configurations according to their need.

I have found a couple of useful instructions but they led me nowhere. Because I got the server running behind nginx proxy and the whole css seems to be broken. Then I added some rules (again searching the internet) to route the static contents but it didn’t help me much while trying to keep the ssl up. Those rules work just fine without SSL.

So what was the catch? Most of the online instructions that worked was using a sub-directory to run jenkins, instruction uses --prefix=/jenkins directive that leads to a configuration where the jenkins service is running in a sub-directory. This will lead to an URL like below

https://your-super-awesome-domain.com/jenkins/

But my problem was, that my Jenkins instance was already up and running at the root of the sub-domain. e.g.

http://jenkins.my-awesome-domain.com

Here is the working Nginx configuration and explanation

Goal

https must be enabled to serve a secure instance of jenkins at the following FQDN

https://jenkins.domain.tld

Binding Jenkins to Loopback IP Address

Considering that jenkins service is running at 127.0.0.1:8080

to enable – httplistenAddress to 127.0.0.1 edit the following file(s)

/etc/default/jenkins for Ubuntu/Debian based system
/etc/sysconfig/jenkins for RedHat based system

if the default jenkins file is not found in one of these locations then please consult Jenkins’s installation guide.

change the file and add the following argument to JENKINS_ARGS, the line should read as below

HTTP_HOST=127.0.0.1
JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --httpListenAddress=$HTTP_HOST"

Now, let’s first take a look at a running instance of jenkins behind nginx proxy service without ssl

Configuration for HTTP (Without SSL)

upstream jenkins {
  keepalive 64; # keepalive connections
  server 127.0.0.1:8080; # jenkins ip and port
}

server {
  listen          80;       # Listen on port 80 for IPv4 requests

  server_name     jenkins.domain.tld;

  #this is the jenkins web root directory (mentioned in the /etc/default/jenkins file)
  root            /var/cache/jenkins/war/;

  access_log      /var/log/nginx/jenkins/access.log;
  error_log       /var/log/nginx/jenkins/error.log;

  #pass through headers from Jenkins which are considered invalid by Nginx server.
  ignore_invalid_headers off;

  location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
    #rewrite all static files into requests to the root
    #E.g /static/12345678/css/something.css will become /css/something.css
    rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
  }

  location /userContent {
    #have nginx handle all the static requests to the userContent folder files
    #note : This is the $JENKINS_HOME dir
        root /var/lib/jenkins/;
    if (!-f $request_filename){
      #this file does not exist, might be a directory or a /**view** url
      rewrite (.*) /$1 last;
          break;
    }
        sendfile on;
  }

  # keep an eye on root location directive,
  # because the reverse proxy is being served from the root of the server block
  location / {
      sendfile off;
      proxy_pass         http://jenkins;
      proxy_redirect     default;  # For http, it's set to default
      proxy_http_version 1.1;

      proxy_set_header   Host              $host;
      proxy_set_header   X-Real-IP         $remote_addr;
      proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header   X-Forwarded-Proto $scheme;
      proxy_max_temp_file_size 0;

      #this is the maximum upload size
      client_max_body_size       10m;
      client_body_buffer_size    128k;

      proxy_connect_timeout      90;
      proxy_send_timeout         90;
      proxy_read_timeout         90;
      proxy_buffering            off;
      proxy_request_buffering    off; # Required for HTTP CLI commands in Jenkins > 2.54
      proxy_set_header Connection ""; # Clear for keepalive
  }

}

Configuration for HTTPS

For the above configuration to work exactly as it should with SSL enabled, a few changes are required –

SSL key and certificates location has been added
The listening port has been changed
Change in location root (/) directive declaration

NOTE

A few configuration snippets is added with include directives in the next configuration. Those configuration snippets are also provided below.

IMPORTANT

Carefully consider the changes in the following code snippet. Especially the root location / {......} location directive declararion.

upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

server {
    listen 443 ssl;

    server_name jenkinsx.domain.tld;
    root /var/cache/jenkins/war/;

    # SSL
    ssl_certificate /etc/nginx/ssl/jenkins.domain.tld.crt;
    ssl_certificate_key /etc/nginx/ssl/jenkins.domain.tld.key;

    # security
    include nginxconfig/security.conf;

    # logging
    access_log /var/log/nginx/jenkins.domain.tld.access.log;
    error_log /var/log/nginx/jenkins.domain.tld.error.log warn;

    ignore_invalid_headers off;

    location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
        #rewrite all static files into requests to the root
        #E.g /static/12345678/css/something.css will become /css/something.css
        rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
    }

    location /userContent {
        #have nginx handle all the static requests to the userContent folder files
        #note : This is the $JENKINS_HOME dir
        root /var/lib/jenkins/;
        if (!-f $request_filename){
            #this file does not exist, might be a directory or a /**view** url
            rewrite (.*) /$1 last;
            break;
        }
        sendfile on;
    }

    # reverse proxy -> https://wiki.jenkins.io/display/JENKINS/Jenkins+behind+an+nginx+reverse+proxy
    # NOTE :: the regular expression is necessay to make sure that
    # jenkins is being served form the root of the server block

    location ~* ^/ {
        sendfile off;
        proxy_pass         http://jenkins;

        proxy_set_header   Host              $host:$server_port;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        # For `https` the value changed, it will prevent broken proxy errot message in jenkins GUI
        proxy_redirect     http:// https://;
        proxy_set_header   X-Forwarded-Host  $host;
        proxy_max_temp_file_size 0;

        #this is the maximum upload size
        client_max_body_size       10m;
        client_body_buffer_size    128k;

        proxy_connect_timeout      90;
        proxy_send_timeout         90;
        proxy_read_timeout         90;
        # New HTTP-based CLI
        proxy_http_version         1.1;
        proxy_buffering            off;
        proxy_request_buffering    off; # Required for HTTP CLI commands in Jenkins > 2.54
        proxy_set_header Connection ""; # Clear for keepalive
        # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
        add_header 'X-SSH-Endpoint' 'jenkins.domain.tld:50022' always;
    }

    # additional config
    include nginxconfig/general.conf;
}

# subdomains redirect
server {
    listen 443 ssl;

    server_name *.jenkins.domain.tld;

    # SSL
    ssl_certificate /etc/nginx/ssl/jenkins.domain.tld.crt;
    ssl_certificate_key /etc/nginx/ssl/jenkins.domain.tld.key;

    return 301 https://jenkins.domain.tld$request_uri;
}

# HTTP redirect
server {
    listen 80;

    server_name .jenkins.domain.tld;

    return 301 https://jenkins.domain.tld$request_uri;
}

General Configuration Snippet

# favicon.ico
location = /favicon.ico {
    log_not_found off;
    access_log off;
}

# robots.txt
location = /robots.txt {
    log_not_found off;
    access_log off;
}

# assets, media
location ~* \.(?:css(\.map)?|js(\.map)?|jpe?g|png|gif|ico|cur|heic|webp|tiff?|mp3|m4a|aac|ogg|midi?|wav|mp4|mov|webm|mpe?g|avi|ogv|flv|wmv)$ {
    expires 7d;
    access_log off;
}

# svg, fonts
location ~* \.(?:svgz?|ttf|ttc|otf|eot|woff2?)$ {
    add_header Access-Control-Allow-Origin "*";
    expires 7d;
    access_log off;
}

# gzip
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;

Security Configuration Snippet

# security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

# . files
location ~ /\.(?!well-known) {
    deny all;
}

The global nginx (e.g. /etc/nginx/nginx.conf) configuration is not provided here.

Now with these configurations, the SSL is enabled and CSS is working fine.

Enjoy!

June 27, 2020 Dalwar Hossain