small_light didn't works well with proxy_pass directive #73

draskolnikova · 2016-10-18T16:11:04Z

Hello!

Got curious, when I deploy small_light module combine with proxy_pass directive, my goal is to optimize images from external resource. My nginx + small_light + proxy_pass configuration looks like :

upstream awsdist {
        server ip.ad.dr.es:80;
        keepalive 128;
}

server {
... common nginx configuration ...
        small_light_pattern_define webpi of=webp,q=80;
        small_light_buffer 16M;

        location ~ \.(jpe?g|png|gif|webp)$ {
                small_light on;
                small_light_getparam_mode on;

                proxy_pass      http://external;
                proxy_cache_use_stale error timeout updating http_500 http_503 http_504;
                proxy_cache_valid 200 302 301 24h;
                proxy_cache     images;
                proxy_cache_valid any 3s;
                proxy_cache_lock on;
                proxy_cache_revalidate on;
                proxy_cache_min_uses 1;
                expires max;
        }

I tried to check external resource file size using cURL:

external resource

$ curl -I http://ip.ad.dr.es/2015-12/0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
HTTP/1.1 200 OK
Content-Type: image/jpeg
Content-Length: 592651
Connection: keep-alive
Date: Tue, 18 Oct 2016 15:59:19 GMT
Last-Modified: Thu, 10 Dec 2015 02:40:24 GMT
ETag: "[removed]"
Cache-Control: max-age=604800, public
Accept-Ranges: bytes
Server: AmazonS3
X-Cache: Miss from cloudfront
Via: 1.1 [removed].cloudfront.net (CloudFront)
X-Amz-Cf-Id: [removed]

Proxied using nginx + small_light + proxy_pass

$ curl -I https://proxy-pass-with-small-light-domain/2015-12/0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg?p=webpi
HTTP/1.1 200 OK
Date: Tue, 18 Oct 2016 16:13:10 GMT
Content-Type: image/webp
Content-Length: 595062
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Thu, 10 Dec 2015 02:40:24 GMT
Cache-Control: max-age=315360000
Server: AmazonS3
Age: 1269
X-Amz-Cf-Id: MISS by id02
Expires: Thu, 31 Dec 2037 23:55:55 GMT

Nginx + small_light

The images self-hosted on it's server, not proxied to upstream.

$ curl -I https://small-light.domain/0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg?p=webpi
HTTP/1.1 200 OK
Date: Tue, 18 Oct 2016 15:44:05 GMT
Content-Type: image/webp
Content-Length: 33698
Last-Modified: Thu, 10 Dec 2015 02:40:24 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5668e618-91476"
Server: unyil

My question is, the size of file didn't optimized when proxied to upstream, see content-length's overhead on external resource (592651 bytes) with nginx + small_light + proxy_pass (595062 bytes). We got ~3KBytes overhead if small_light active on proxy_pass directive.

Then, we got only 33698 bytes on non-proxied images.

Is it normal?

Update:
Bugs appeared when using pattern define, but when I directly invoke using of=webp&q=80, it's works.

The text was updated successfully, but these errors were encountered:

cubicdaiya · 2016-10-21T08:12:12Z

You cannot use the directives such assmall_light and proxy_pass and proxy_cache_xxx in same virtual host. Divide them into different virtual servers.

draskolnikova · 2016-10-21T08:14:33Z

@cubicdaiya divide them into different virtual servers? I still didn't get it.

cubicdaiya · 2016-10-21T08:19:33Z

Show your configuration example for reproducing.

draskolnikova · 2016-10-21T08:29:10Z

@cubicdaiya there's some NDA on my configuration, can we talk using private communication (eg. email).

cubicdaiya · 2016-10-21T09:39:25Z

can we talk using private communication (eg. email).

No.

cubicdaiya · 2016-10-21T10:04:38Z

I tried with the configuration below. But It seems to work fine in my environment.

server {
    small_light on;
    small_light_getparam_mode on;
    small_light_pattern_define hoge dw=200,dh=200;
    location / {
        proxy_pass http://example.com;
    }
}

If you cannot provide the real configuration, please provide the minimal example configuraiton for reproducing.

draskolnikova · 2016-10-21T10:28:56Z

@cubicdaiya here is the configuration, i remove FQDN from external resource.

upstream awsdist {
        server ip.ad.dr.es:80;
        keepalive 128;
}

map $http_user_agent $img_mode {
        $default default;
        ~*chrome of=webp&q=70;
        ~*opera of=webp&q=70;
}

map $http_user_agent $pattern_mode {
        $default default;
        ~*chrome p=optimized;
        ~*opera p=optimized;
}

server {
    listen 80;
    server_name example.com;
    access_log  /var/log/nginx/example.com-access.log;
    error_log   /var/log/nginx/example.com-error.log;

        location = /favicon.ico {
                return 204;
                access_log     off;
                log_not_found  off;
        location = /robots.txt {
                return 204;
                access_log     off;
                log_not_found  off;
        }

    small_light_pattern_define optimized of=webp,q=70;
    small_light_buffer 16M;
    small_light on;
    small_light_getparam_mode on;

    location ~ \.(jpe?g|png|gif|webp)$ {
                proxy_pass  http://awsdist;
                proxy_cache_use_stale error timeout updating http_500 http_503 http_504;
                proxy_cache_valid 200 302 301 24h;
                proxy_cache     images;
                proxy_cache_valid any 3s;
                proxy_cache_lock on;
                proxy_cache_revalidate on;
                proxy_cache_min_uses 1;
                proxy_cache_path /var/cache/nginx/proxy_cache levels=1:2 keys_zone=images:512m inactive=60m use_temp_path=off max_size=512m;
                proxy_cache_key "$host$uri$is_args$args";
                proxy_ignore_headers Cache-Control Expires;
                proxy_hide_header X-Cache;
                proxy_hide_header Via;
                proxy_hide_header ETag;
                # this directive, works
                #rewrite (?<capt>.*) $capt?$args$img_mode break;
                # this directive, not works
                rewrite (?<capt>.*) $capt?$args$pattern_mode break;
    }
}

If I use rewrite (?<capt>.*) $capt?$args$pattern_mode break; the images size didn't optimized, but I use rewrite (?<capt>.*) $capt?$args$img_mode break;, it's works. So the URL would be http://example.com/images.jpg (didn't need add additional parameter(s) after *.jpg)

Update:
Change from 80 to 70 on pattern_define.

cubicdaiya · 2016-10-22T01:21:16Z

Why don't you decrease quality? ( Maybe 80 is too high for WebP.)

draskolnikova · 2016-10-22T03:35:26Z

@cubicdaiya my bad, the value on pattern_define should be 70. I was test it using $pattern_mode, nothing was change. Don't know is it bug or miss configuration, any hints?

cubicdaiya · 2016-10-22T07:09:06Z

The cause is the value of q is too large? In my environment, the converted-webp-image size is larger than original-image when q is large. (e.g. q=80)

Maybe converted-webp-image is smaller than original if q is more low or eliminating q in parameters.

draskolnikova · 2016-10-23T11:50:41Z

@cubicdaiya

As your request, I remove parameter q on rewrite rules, here's the result and use $img_mode directive.

Original Upstream:

accept-ranges:bytes
age:4
cache-control:max-age=604800, public
content-length:68563
content-type:image/jpeg
date:Sun, 23 Oct 2016 11:39:17 GMT
etag:"[removed]"
last-modified:Sat, 25 Jun 2016 23:06:35 GMT
server:AmazonS3
status:200
via:1.1 [removed].cloudfront.net (CloudFront)
x-amz-cf-id:[removed]
x-cache:Hit from cloudfront

Result, I got 67,4KB on chrome

Proxied images:

cache-control:max-age=604800, public
content-encoding:gzip
content-type:image/webp
date:Sun, 23 Oct 2016 11:39:23 GMT
last-modified:Sat, 25 Jun 2016 23:06:35 GMT
server:AmazonS3
status:200
vary:Accept-Encoding
x-amz-cf-id:HIT from id02

On chrome, I got 66,4KB.

Tried add proxy_pass_header Content-Length; on configuration, but no luck :D

cubicdaiya · 2016-10-23T13:01:47Z

content-encoding:gzip

Your WebP image looks gzip-compressed. As a image such as WebP and JPEG is already compressed, they should not be compressed by gzip. ( In most case, image size will be expanded. )

draskolnikova · 2016-10-23T13:21:43Z

@cubicdaiya

Ok, I was set webp are not compressed by gzip.

Upstream Images: 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg

accept-ranges:bytes
age:3
cache-control:max-age=604800, public
content-length:592651
content-type:image/jpeg
date:Sun, 23 Oct 2016 13:08:53 GMT
etag:"[removed]"
last-modified:Thu, 10 Dec 2015 02:40:24 GMT
server:AmazonS3
status:200
via:1.1 [removed].cloudfront.net (CloudFront)
x-amz-cf-id:[removed]
x-cache:Hit from cloudfront

Proxied Images: 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg

cache-control:max-age=604800, public
content-length:595062
content-type:image/webp
date:Sun, 23 Oct 2016 13:09:01 GMT
last-modified:Thu, 10 Dec 2015 02:40:24 GMT
server:AmazonS3
status:200
x-amz-cf-id:HIT from id01

Why proxied image file larger than upstream image file? Still using $img_mode parameter without q. If you need the file, you can go to this link http://103.52.2.19/0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg and check the header, this is the original file before converted to webp.

FYI, this is my system environment.

nginx version: nginx/1.11.4
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) 
built with OpenSSL 1.0.2j  26 Sep 2016
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-openssl=openssl-1.0.2j --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_stub_status_module --with-http_auth_request_module --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_perl_module=dynamic --with-http_xslt_module=dynamic --add-dynamic-module=ngx_cache_purge --add-dynamic-module=nginx-module-vts --add-dynamic-module=headers-more-nginx-module --add-dynamic-module=ngx_small_light --add-dynamic-module=ngx_brotli --add-module=nginx_upstream_check_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_slice_module --with-mail=dynamic --with-mail_ssl_module --with-file-aio --with-ipv6 --with-http_v2_module --with-cc-opt='-g -Ofast -march=native -ffast-math -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2'

uname -ar

Linux redirector-engineer.dist01-jkt3d.xtremenitro.org 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

ImageMagick

ImageMagick-6.7.8.9-15.el7_2.x86_64
Version: ImageMagick 6.7.8-9 2016-06-16 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2012 ImageMagick Studio LLC
Features: OpenMP

cubicdaiya · 2016-10-23T15:10:35Z

Why don't you try to convert your image by convert?

$ convert -format webp image.jpg image.webp

draskolnikova · 2016-10-23T15:13:51Z

@cubicdaiya Done, convert it to test the size is lower than original images?

[root@phantom html]# ls -lah 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.*
-rw-r--r--. 1 root root 582K Dec 10  2015 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
-rw-r--r--. 1 root root  33K Oct 23 21:50 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp

cubicdaiya · 2016-10-23T15:20:44Z

In my envinronment, convert without quality assignment inflates image. But image is shrinked when quality is 50.

$ convert -format webp 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp
$ convert -quality 50 -format webp 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp
$ ls -l | grep 0_0
-rw-r--r--    1 bokko  staff    595062 Dec 10  2015 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
-rw-r--r--    1 bokko  staff    674310 Oct 24 00:17 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp
-rw-r--r--    1 bokko  staff     26386 Oct 24 00:17 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp

cubicdaiya · 2016-10-23T15:27:15Z

Why proxied image file larger than upstream image file?

This phenomenon is not reproduced at all. So I have no idea. Is your image quality just high?

draskolnikova · 2016-10-23T15:33:18Z

@cubicdaiya I can give you a sandbox and replicate my existing environment. So you can login and reproduce there. I thought the configuration above is same with existing environment, the difference is the FQDN upstream.

How about that?

FYI,
I just do and replicate the file like yours, here is the result.

[root@phantom html]# convert -quality 50 -format webp 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp
[root@phantom html]# ls -l | grep 0_0
-rw-r--r--. 1 root root  595062 Dec 10  2015 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
-rw-r--r--. 1 root root   33698 Oct 23 22:04 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp
-rw-r--r--. 1 root root   33698 Oct 23 21:50 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp
[root@phantom html]# convert -quality 10 -format webp 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q10.webp
[root@phantom html]# !ls
ls -l | grep 0_0
-rw-r--r--. 1 root root  595062 Dec 10  2015 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.jpg
-rw-r--r--. 1 root root   33698 Oct 23 22:04 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q10.webp
-rw-r--r--. 1 root root   33698 Oct 23 22:04 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q50.webp
-rw-r--r--. 1 root root   33698 Oct 23 21:50 0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508.webp

You can access it through http://103.52.2.19/0_0_1000_666_ecf9dfd112be52fd0e81f4522f5d8abe60905508_q10.webp.

Dunno how to reproduce and start from where :D

cubicdaiya · 2016-10-23T15:39:35Z

I'm sorry, but I'm not going so far as to investigate this problem.

draskolnikova · 2016-10-23T15:41:27Z

@cubicdaiya it's okey bro :) I just give feedback to your hobby then. 👍

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

small_light didn't works well with proxy_pass directive #73

small_light didn't works well with proxy_pass directive #73

draskolnikova commented Oct 18, 2016 •

edited

Loading

cubicdaiya commented Oct 21, 2016

draskolnikova commented Oct 21, 2016

cubicdaiya commented Oct 21, 2016

draskolnikova commented Oct 21, 2016

cubicdaiya commented Oct 21, 2016

cubicdaiya commented Oct 21, 2016 •

edited

Loading

draskolnikova commented Oct 21, 2016 •

edited

Loading

cubicdaiya commented Oct 22, 2016

draskolnikova commented Oct 22, 2016

cubicdaiya commented Oct 22, 2016

draskolnikova commented Oct 23, 2016

cubicdaiya commented Oct 23, 2016 •

edited

Loading

draskolnikova commented Oct 23, 2016

cubicdaiya commented Oct 23, 2016

draskolnikova commented Oct 23, 2016

cubicdaiya commented Oct 23, 2016 •

edited

Loading

cubicdaiya commented Oct 23, 2016 •

edited

Loading

draskolnikova commented Oct 23, 2016

cubicdaiya commented Oct 23, 2016 •

edited

Loading

draskolnikova commented Oct 23, 2016

small_light didn't works well with proxy_pass directive #73

small_light didn't works well with proxy_pass directive #73

Comments

draskolnikova commented Oct 18, 2016 • edited Loading

external resource

Proxied using nginx + small_light + proxy_pass

Nginx + small_light

cubicdaiya commented Oct 21, 2016

draskolnikova commented Oct 21, 2016

cubicdaiya commented Oct 21, 2016

draskolnikova commented Oct 21, 2016

cubicdaiya commented Oct 21, 2016

cubicdaiya commented Oct 21, 2016 • edited Loading

draskolnikova commented Oct 21, 2016 • edited Loading

cubicdaiya commented Oct 22, 2016

draskolnikova commented Oct 22, 2016

cubicdaiya commented Oct 22, 2016

draskolnikova commented Oct 23, 2016

cubicdaiya commented Oct 23, 2016 • edited Loading

draskolnikova commented Oct 23, 2016

uname -ar

ImageMagick

cubicdaiya commented Oct 23, 2016

draskolnikova commented Oct 23, 2016

cubicdaiya commented Oct 23, 2016 • edited Loading

cubicdaiya commented Oct 23, 2016 • edited Loading

draskolnikova commented Oct 23, 2016

cubicdaiya commented Oct 23, 2016 • edited Loading

draskolnikova commented Oct 23, 2016

draskolnikova commented Oct 18, 2016 •

edited

Loading

cubicdaiya commented Oct 21, 2016 •

edited

Loading

draskolnikova commented Oct 21, 2016 •

edited

Loading

cubicdaiya commented Oct 23, 2016 •

edited

Loading

cubicdaiya commented Oct 23, 2016 •

edited

Loading

cubicdaiya commented Oct 23, 2016 •

edited

Loading

cubicdaiya commented Oct 23, 2016 •

edited

Loading