Setting Up Nginx With Upload Module

So I’ve been trying to get reliable uploads for large files working for my Rails application (Nginx with Unicorn/Rails running on Ubuntu 12.04 LTS 64-bit as a small EC2 instance).

My initial setup with nginx+unicorn was to simply pass the upload into unicorn and just let Rails handle it from there. This is fine for small 500kb files but proved to be problematic if the upload took too long – unicorn would simply time out after 30 sec. Increasing the timeout somewhat works, but that’s not very reliable or elegant now is it?

Since I was storing my user uploaded content on AWS S3, I decided to use direct to S3 uploads with CORS. This lets the upload bypass my server entirely, letting S3 handle the grunt work. I’m not going to go into details of how to implement this, as this is outside the scope of this post, plus there’s plenty of tutorials out there (side note – I was using Plupload in the front-end. Highly recommended if you need a reliable cross-browser solution with fallbacks to multiple web technologies).

Direct to S3 worked great for a while until I needed to encrypt user uploaded files. I didn’t want to upload to S3, download to my server, encrypt/process, and then reupload to S3. So…

My Current Setup

In the end, I needed my server to handle the file upload after all (so that the flow ends up being user upload to server > encrypt on server > store on S3). To do this, we’ll use the nginx upload module.

The difference between this and my first setup is that unicorn/rails does not even see the file before it’s already on the disk. The nginx upload module will be fully responsible for the heavy lifting, and pass the file to rails when it’s done. Let’s get started…

First, install some libraries we’ll need later:

sudo apt-get install libpcre3 libpcre3-dev

Then, download nginx and the nginx upload module (I’ve also installed the nginx upload progress module). Note that I am using nginx 1.3.8 as later versions are no longer compatible with the upload module (edit: there seems to be a patch for 1.5.1+). See this for reference, as there seems to be a fix.

wget https://launchpad.net/nginx/1.3/1.3.8/+download/nginx-1.3.8.tar.gz
wget https://github.com/vkholodkov/nginx-upload-module/archive/2.2.zip
wget https://github.com/masterzen/nginx-upload-progress-module/archive/master.zip
unzip 2.2.zip
unzip master.zip
tar -xzvf nginx-1.3.8.tar.gz

Now, let’s compile nginx with the modules. If you need to add any other modules, add them in the configure step. Replace YOURPATH with where you just unzipped.

cd nginx-1.3.8
./configure --add-module=/YOURPATH/nginx-upload-module-2.2 --add-module=/YOURPATH/nginx-upload-progress-module-master --with-http_ssl_module --with-http_gzip_static_module
 
make
sudo make install

This will install nginx in the /usr/local/nginx/ directory. Now, edit the /usr/local/nginx/conf/nginx.conf file with your favorite editor. I used the following (nothing special in this file, just make sure to include the last line for the next config file):

user YOUR_NGINX_USER;
worker_processes 4;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
timer_resolution 500ms;
events {
    use epoll;
    worker_connections 1024;
}
 
http {
    include /usr/local/nginx/conf/mime.types;
    default_type application/octet-stream;
 
    server_name_in_redirect off;
    server_tokens off;
 
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log /var/log/nginx/access.log main;
 
    sendfile on;
    tcp_nopush on;
 
    keepalive_timeout 30;
    tcp_nodelay off;
 
    client_body_timeout 10;
    client_header_timeout 10;
    client_header_buffer_size 128;
    client_max_body_size 1000m;   # change this to max file size
 
    open_file_cache max=1000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;
    open_file_cache_errors on;
 
    proxy_connect_timeout 300;
    proxy_send_timeout 300;
    proxy_read_timeout 300;
    proxy_buffer_size 32k;
    proxy_buffers 4 32k;
    proxy_busy_buffers_size 32k;
    proxy_temp_file_write_size 32k;
 
    gzip on;
    gzip_http_version 1.0;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    gzip_buffers 16 8k;
    gzip_comp_level 6;
    gzip_min_length 0;
    gzip_vary on;
    gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json;
    gzip_proxied any;
 
    include /usr/local/nginx/conf/conf.d/*.conf;
}

More importantly, my /usr/local/nginx/conf/conf.d/default.conf is as follows (stuff relevant to upload handling in lines 20 to 38):

upstream unicorn_myapp {
    server unix:/tmp/my_site.socket fail_timeout=0;
}
 
server {
    listen 80;
    server_name YOUR_SERVER_NAME;
    rewrite ^ https://YOUR_URL$request_uri? permanent; # force premanent redirect to https
}
 
server {
    listen 443;
    server_name YOUR_SERVER_NAME;
    ssl on;
    ssl_certificate /usr/local/nginx/ssl/ssl-unified.crt;
    ssl_certificate_key /usr/local/nginx/ssl/ssl.nopass.key;
 
    root PATH_TO_RAILS_APPLICATION;
 
    location = /upload_file {
        set $upload_field_name "file";
        upload_pass /upload_file_rails; # the route to POST to when the file is uploaded
        upload_store /tmp/uploads 1;  # you need to create 10 folders in /tmp/uploads/X where X is 0 to 9
        upload_resumable on;
        upload_store_access user:rw group:rw all:r;
        upload_max_file_size 0; # 0 for unlimited
 
        # define form fields to be passed to Rails
        upload_set_form_field $upload_field_name[original_filename] "$upload_file_name";
        upload_set_form_field $upload_field_name[tempfile] "$upload_tmp_path";
        upload_set_form_field $upload_field_name[content_type] "$upload_content_type";
        upload_aggregate_form_field $upload_field_name[size] "$upload_file_size";
 
        upload_pass_form_field "^folder_id$|^authenticity_token$|^format$"; # add params that you want to keep and pass on to your application
 
        upload_cleanup 400 404 499 500-505;
 
    }
 
    try_files $uri @app;
 
    location @app {
        proxy_pass http://unicorn_myapp;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect off;
    }
 
    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }
 
    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }
 
    location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ { # static assets. I'm using CloudFront.
        root /RAILS_APPLICATION_PATH/public;
        gzip_static on;
        gzip_http_version 1.0;
        expires max;
        add_header Cache-Control public;
        add_header Last-Modified "";
        log_not_found off;
    }
 
    location ~* \.(eot|svg|ttf|woff|swf|xap)$ { # this is for fonts and flash upload
        root /home/dev/web/current/public;
        expires max;
        add_header Cache-Control public;
        add_header Last-Modified "";
        add_header Access-Control-Allow-Origin *; # need to allow this for firefox
        log_not_found off;
    }
}

You may need to create the “upload_store” tmp directories and/or set the correct permissions. Restart nginx, and now direct your front-end to POST upload to “/upload_file”. When the file is done uploading, the file and params you specified earlier will automatically be passed to “/upload_file_rails”, where you should handle any file processing/saving logic.

In my Rails controller, I save the metadata in the database, encrypt the file, and upload it to a private bucket in S3. I also delete the temp file from the “upload_store” directory (as far as I know nginx won’t delete it automatically, so I remove the temp files manually to save some space). Note – since encrypting and uploading the file can take some time, I run this in the background using Spawnling so that the user upload page is still responsive.

Here’s part of my Rails controller to help get started.

#this is where /upload_file_rails directs here in my routes.
def upload_file_rails 
    # we get the file and other parameters from nginx 
    # file is already local on the file system
    file_param_size = params[:file][:size].to_s
    file_original_filename = params[:file][:original_filename].to_s #you may want to sanitize this for security if you're storing this anywhere
    file_tempfile = params[:file][:tempfile].to_s # location of the file locally
    file_content_type = params[:file][:content_type].to_s
 
  
    # process file, store on S3, save entry to db, whatever you want
    File.open(file_tempfile, "rb") do |inf|
        # do stuff with file here
    end
 
 
    #remove uploaded tmp file
    File.delete(file_tempfile)
end

Let me know if improvements can be made. Comments and feedback are welcome.

Tags:

0 Comments

Disclaimer: In order to provide you with ongoing content, this website participates in affiliate advertising programs, such as Viglink, which are designed to provide a means for sites to earn advertising fees by advertising and linking to commercial websites. This means that we may sometimes get paid if you click one of those links and purchase a product or service.