camo-1.3.0/000077500000000000000000000000001224374550400124405ustar00rootroot00000000000000camo-1.3.0/.gitignore000066400000000000000000000000551224374550400144300ustar00rootroot00000000000000node_modules tmp/camouflage.pid tmp/camo.pid camo-1.3.0/AUTHORS000066400000000000000000000001271224374550400135100ustar00rootroot00000000000000Rick Olson: https://github.com/technoweenie Corey Donohoe: https://github.com/atmos camo-1.3.0/CHANGELOG.md000066400000000000000000000015361224374550400142560ustar00rootroot000000000000001.1.3 ===== * [Address ddos](https://groups.google.com/forum/#!msg/nodejs/NEbweYB0ei0/gWvyzCunYjsJ?mkt_tok=3RkMMJWWfF9wsRonuavPZKXonjHpfsX54%2B8tXaO3lMI%2F0ER3fOvrPUfGjI4ASMFrI%2BSLDwEYGJlv6SgFQrjAMapmyLgLUhE%3D) in earlier versions of node. 1.1.1 ===== * Use pipe() to pause buffers when streaming to slow clients * Fixup tests and Gemfile related stuff * Workaround recent heroku changes that now detect camo as a ruby app due to Gemfile presence * Ensure a location header is present before following redirects, fixes a crash 1.0.5 ===== * Fixup redirect loops where following redirects goes back to camo * Add Fallback Accept headers for type `image/*` * Fixup issues with chunked encoding responses * Explicitly set User-Agent headers when proxying 1.0.2 ===== * Follow 303s and 307s now too 0.5.0 ===== * Follow redirects to a configurable depth camo-1.3.0/LICENSE000066400000000000000000000020551224374550400134470ustar00rootroot00000000000000Copyright (c) 2010 Corey Donohoe, Rick Olson Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. camo-1.3.0/Procfile000066400000000000000000000000241224374550400141220ustar00rootroot00000000000000web: node server.js camo-1.3.0/README.md000066400000000000000000000045721224374550400137270ustar00rootroot00000000000000![camo](http://farm5.static.flickr.com/4116/4857328881_fefb8e2134_z.jpg) Camo is all about making insecure assets look secure. This is an SSL image proxy to prevent mixed content warnings on secure pages served from [GitHub](https://github.com). We want to allow people to keep embedding images in comments/issues/READMEs/google charting. [There's more info on the GitHub blog](https://github.com/blog/743-sidejack-prevention-phase-3-ssl-proxied-assets). Using a shared key, proxy URLs are encrypted with [hmac](http://en.wikipedia.org/wiki/HMAC) so we can bust caches/ban/rate limit if needed. Camo currently runs on node version 0.10.13 at GitHub on [heroku](http://heroku.com). Features -------- * Proxy google charts * Proxy images under 5 MB * Follow redirects to a configurable depth * Proxy remote images with a content-type of `image/*` * 404s for anything other than a 200, 301, 302, 303, 304 or 307 HTTP response * Disallows proxying to private IP ranges At GitHub we render markdown and replace all of the `src` attributes on the `img` tags with the appropriate URL to hit the proxies. There's example code for creating URLs in [the tests](https://github.com/atmos/camo/blob/master/test/proxy_test.rb). ## URL Formats Camo supports two distinct URL formats: http://example.org/?url= http://example.org// The `` is a 40 character hex encoded HMAC digest generated with a shared secret key and the unescaped `` value. The `` is the absolute URL locating an image. In the first format, the `` should be URL escaped aggressively to ensure the original value isn't mangled in transit. In the second format, each byte of the `` should be hex encoded such that the resulting value includes only characters `[0-9a-f]`. ## Testing Functionality ### Bundle Everything % rake bundle ### Start the server % coffee server.coffee ### In another shell % rake ### Debugging To see the full URL restclient is hitting etc, try this. % RESTCLIENT_LOG=stdout rake ### Deployment You can see an example [god config](https://gist.github.com/675038) here. To enable useful line numbers in stacktraces you probably want to compile the server.coffee file to native javascript when deploying. % coffee -c server.coffee % /usr/bin/env PORT=9090 CAMO_KEY="" node server.js camo-1.3.0/Rakefile000066400000000000000000000013111224374550400141010ustar00rootroot00000000000000file 'server.js' => 'server.coffee' do sh "coffee -c -o . server.coffee" end task :build => 'server.js' task :bundle do system("bundle install --gemfile test.gemfile") end namespace :test do desc "Start test server" task :server do |t| $SERVER_PID = Process.spawn("ruby test/proxy_test_server.rb") end desc "Run the tests against localhost" task :check do |t| system("BUNDLE_GEMFILE=test.gemfile bundle exec ruby test/proxy_test.rb") end desc "Kill test server" task :kill_server do |t| Process.kill(:QUIT, $SERVER_PID) && Process.wait end end task :default => [:build, :bundle, "test:server", "test:check", "test:kill_server"] Dir["tasks/*.rake"].each do |f| load f end camo-1.3.0/log/000077500000000000000000000000001224374550400132215ustar00rootroot00000000000000camo-1.3.0/log/.gitignore000066400000000000000000000000001224374550400151770ustar00rootroot00000000000000camo-1.3.0/package.json000066400000000000000000000001601224374550400147230ustar00rootroot00000000000000{ "name": "camo", "version": "1.3.0", "dependencies": { }, "engines": { "node": ">=0.10.21" } } camo-1.3.0/server.coffee000066400000000000000000000206071224374550400151240ustar00rootroot00000000000000Fs = require 'fs' Dns = require 'dns' Url = require 'url' Http = require 'http' Crypto = require 'crypto' QueryString = require 'querystring' port = parseInt process.env.PORT || 8081 version = "1.3.0" shared_key = process.env.CAMO_KEY || '0x24FEEDFACEDEADBEEFCAFE' max_redirects = process.env.CAMO_MAX_REDIRECTS || 4 camo_hostname = process.env.CAMO_HOSTNAME || "unknown" socket_timeout = process.env.CAMO_SOCKET_TIMEOUT || 10 logging_enabled = process.env.CAMO_LOGGING_ENABLED || "disabled" content_length_limit = parseInt(process.env.CAMO_LENGTH_LIMIT || 5242880, 10) debug_log = (msg) -> if logging_enabled == "debug" console.log("--------------------------------------------") console.log(msg) console.log("--------------------------------------------") error_log = (msg) -> unless logging_enabled == "disabled" console.error("[#{new Date().toISOString()}] #{msg}") RESTRICTED_IPS = /^((10\.)|(127\.)|(169\.254)|(192\.168)|(172\.((1[6-9])|(2[0-9])|(3[0-1]))))/ total_connections = 0 current_connections = 0 started_at = new Date four_oh_four = (resp, msg, url) -> error_log "#{msg}: #{url?.format() or 'unknown'}" resp.writeHead 404 finish resp, "Not Found" finish = (resp, str) -> current_connections -= 1 current_connections = 0 if current_connections < 1 resp.connection && resp.end str # A Transform Stream that limits the piped data to the specified length Stream = require('stream') class LimitStream extends Stream.Transform constructor: (length) -> super() @remaining = length _transform: (chunk, encoding, cb) -> if @remaining > 0 if @remaining < chunk.length chunk = chunk.slice(0, @remaining) @push(chunk) @remaining -= chunk.length if @remaining <= 0 @emit('length_limited') @end() cb() write: (chunk, encoding, cb) -> if @remaining > 0 super else false process_url = (url, transferred_headers, resp, remaining_redirects) -> if !url.host? return four_oh_four(resp, "Invalid host", url) if url.protocol == 'https:' error_log("Redirecting https URL to origin: #{url.format()}") resp.writeHead 301, {'Location': url.format()} finish resp return else if url.protocol != 'http:' four_oh_four(resp, "Unknown protocol", url) return Dns.lookup url.hostname, (err, address, family) -> if err return four_oh_four(resp, "No host found: #{err}", url) if address.match(RESTRICTED_IPS) return four_oh_four(resp, "Hitting excluded IP", url) fetch_url address, url, transferred_headers, resp, remaining_redirects fetch_url = (ip_address, url, transferred_headers, resp, remaining_redirects) -> src = Http.createClient url.port || 80, url.hostname src.on 'error', (error) -> four_oh_four(resp, "Client Request error #{error.stack}", url) query_path = url.pathname if url.query? query_path += "?#{url.query}" transferred_headers.host = url.host debug_log transferred_headers srcReq = src.request 'GET', query_path, transferred_headers srcReq.setTimeout (socket_timeout * 1000), ()-> srcReq.abort() four_oh_four resp, "Socket timeout", url srcReq.on 'response', (srcResp) -> is_finished = true debug_log srcResp.headers content_length = srcResp.headers['content-length'] if content_length > content_length_limit srcResp.destroy() four_oh_four(resp, "Content-Length exceeded", url) else newHeaders = 'content-type' : srcResp.headers['content-type'] 'cache-control' : srcResp.headers['cache-control'] || 'public, max-age=31536000' 'Camo-Host' : camo_hostname 'X-Content-Type-Options' : 'nosniff' # Handle chunked responses properly if content_length? newHeaders['content-length'] = content_length if srcResp.headers['transfer-encoding'] newHeaders['transfer-encoding'] = srcResp.headers['transfer-encoding'] if srcResp.headers['content-encoding'] newHeaders['content-encoding'] = srcResp.headers['content-encoding'] srcResp.on 'end', -> if is_finished finish resp srcResp.on 'error', -> if is_finished finish resp switch srcResp.statusCode when 200 if newHeaders['content-type'] && newHeaders['content-type'].slice(0, 5) != 'image' srcResp.destroy() four_oh_four(resp, "Non-Image content-type returned", url) return debug_log newHeaders resp.writeHead srcResp.statusCode, newHeaders limit = new LimitStream(content_length_limit) srcResp.pipe(limit) limit.pipe(resp) limit.on 'length_limited', -> srcResp.destroy() error_log("Killed connection at content_length_limit: #{url.format()}") when 301, 302, 303, 307 srcResp.destroy() if remaining_redirects <= 0 four_oh_four(resp, "Exceeded max depth", url) else if !srcResp.headers['location'] four_oh_four(resp, "Redirect with no location", url) else is_finished = false newUrl = Url.parse srcResp.headers['location'] unless newUrl.host? and newUrl.hostname? newUrl.host = newUrl.hostname = url.hostname newUrl.protocol = url.protocol debug_log "Redirected to #{newUrl.format()}" process_url newUrl, transferred_headers, resp, remaining_redirects - 1 when 304 srcResp.destroy() resp.writeHead srcResp.statusCode, newHeaders else srcResp.destroy() four_oh_four(resp, "Origin responded with #{srcResp.statusCode}", url) srcReq.on 'error', -> finish resp srcReq.end() resp.on 'close', -> error_log("Request aborted") srcReq.abort() resp.on 'error', (e) -> error_log("Request error: #{e}") srcReq.abort() # decode a string of two char hex digits hexdec = (str) -> if str and str.length > 0 and str.length % 2 == 0 and not str.match(/[^0-9a-f]/) buf = new Buffer(str.length / 2) for i in [0...str.length] by 2 buf[i/2] = parseInt(str[i..i+1], 16) buf.toString() server = Http.createServer (req, resp) -> if req.method != 'GET' || req.url == '/' resp.writeHead 200 resp.end 'hwhat' else if req.url == '/favicon.ico' resp.writeHead 200 resp.end 'ok' else if req.url == '/status' resp.writeHead 200 resp.end "ok #{current_connections}/#{total_connections} since #{started_at.toString()}" else total_connections += 1 current_connections += 1 url = Url.parse req.url user_agent = process.env.CAMO_HEADER_VIA or= "Camo Asset Proxy #{version}" transferred_headers = 'Via' : user_agent 'User-Agent' : user_agent 'Accept' : req.headers.accept ? 'image/*' 'Accept-Encoding' : req.headers['accept-encoding'] 'x-forwarded-for' : req.headers['x-forwarded-for'] 'x-content-type-options' : 'nosniff' delete(req.headers.cookie) [query_digest, encoded_url] = url.pathname.replace(/^\//, '').split("/", 2) if encoded_url = hexdec(encoded_url) url_type = 'path' dest_url = encoded_url else url_type = 'query' dest_url = QueryString.parse(url.query).url debug_log({ type: url_type url: req.url headers: req.headers dest: dest_url digest: query_digest }) if req.headers['via'] && req.headers['via'].indexOf(user_agent) != -1 return four_oh_four(resp, "Requesting from self") if url.pathname? && dest_url hmac = Crypto.createHmac("sha1", shared_key) hmac.update(dest_url, 'utf8') hmac_digest = hmac.digest('hex') if hmac_digest == query_digest url = Url.parse dest_url process_url url, transferred_headers, resp, max_redirects else four_oh_four(resp, "checksum mismatch #{hmac_digest}:#{query_digest}") else four_oh_four(resp, "No pathname provided on the server") console.log "SSL-Proxy running on #{port} with pid:#{process.pid}." console.log "Using the secret key #{shared_key}" server.listen port camo-1.3.0/tasks/000077500000000000000000000000001224374550400135655ustar00rootroot00000000000000camo-1.3.0/tasks/.gitignore000066400000000000000000000000001224374550400155430ustar00rootroot00000000000000camo-1.3.0/test.gemfile000066400000000000000000000001361224374550400147510ustar00rootroot00000000000000source 'https://rubygems.org' gem 'rest-client', '~>1.3' gem 'addressable', '~>2.3' gem 'thin'camo-1.3.0/test.gemfile.lock000066400000000000000000000006001224374550400156740ustar00rootroot00000000000000GEM remote: https://rubygems.org/ specs: addressable (2.3.4) daemons (1.1.9) eventmachine (1.0.3) mime-types (1.23) rack (1.5.2) rest-client (1.6.7) mime-types (>= 1.16) thin (1.5.1) daemons (>= 1.0.9) eventmachine (>= 0.12.6) rack (>= 1.0.0) PLATFORMS ruby DEPENDENCIES addressable (~> 2.3) rest-client (~> 1.3) thin camo-1.3.0/test/000077500000000000000000000000001224374550400134175ustar00rootroot00000000000000camo-1.3.0/test/proxy_test.rb000066400000000000000000000120641224374550400161670ustar00rootroot00000000000000require 'rubygems' require 'json' require 'base64' require 'openssl' require 'rest_client' require 'addressable/uri' require 'thin' require 'test/unit' module CamoProxyTests def config { 'key' => ENV['CAMO_KEY'] || "0x24FEEDFACEDEADBEEFCAFE", 'host' => ENV['CAMO_HOST'] || "http://localhost:8081" } end def test_proxy_survives_redirect_without_location assert_raise RestClient::ResourceNotFound do request('http://localhost:9292') end response = request('http://media.ebaumsworld.com/picture/Mincemeat/Pimp.jpg') assert_equal(200, response.code) end def test_follows_https_redirect_for_image_links response = request('http://dl.dropbox.com/u/602885/github/soldier-squirrel.jpg') assert_equal(200, response.code) end def test_proxy_valid_image_url response = request('http://media.ebaumsworld.com/picture/Mincemeat/Pimp.jpg') assert_equal(200, response.code) end def test_proxy_valid_image_url_with_crazy_subdomain response = request('http://27.media.tumblr.com/tumblr_lkp6rdDfRi1qce6mto1_500.jpg') assert_equal(200, response.code) end def test_proxy_valid_google_chart_url response = request('http://chart.apis.google.com/chart?chs=920x200&chxl=0:%7C2010-08-13%7C2010-09-12%7C2010-10-12%7C2010-11-11%7C1:%7C0%7C0%7C0%7C0%7C0%7C0&chm=B,EBF5FB,0,0,0&chco=008Cd6&chls=3,1,0&chg=8.3,20,1,4&chd=s:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA&chxt=x,y&cht=lc') assert_equal(200, response.code) end def test_proxy_valid_chunked_image_file response = request('http://www.igvita.com/posts/12/spdyproxy-diagram.png') assert_equal(200, response.code) assert_nil(response.headers[:content_length]) end def test_follows_redirects response = request('http://cl.ly/1K0X2Y2F1P0o3z140p0d/boom-headshot.gif') assert_equal(200, response.code) end def test_follows_redirects_formatted_strangely response = request('http://cl.ly/DPcp/Screen%20Shot%202012-01-17%20at%203.42.32%20PM.png') assert_equal(200, response.code) end def test_follows_redirects_with_path_only_location_headers assert_nothing_raised do request('http://blogs.msdn.com/photos/noahric/images/9948044/425x286.aspx') end end def test_404s_on_infinidirect assert_raise RestClient::ResourceNotFound do request('http://modeselektor.herokuapp.com/') end end def test_404s_on_urls_without_an_http_host assert_raise RestClient::ResourceNotFound do request('/picture/Mincemeat/Pimp.jpg') end end def test_404s_on_images_greater_than_5_megabytes assert_raise RestClient::ResourceNotFound do request('http://apod.nasa.gov/apod/image/0505/larryslookout_spirit_big.jpg') end end def test_404s_on_host_not_found assert_raise RestClient::ResourceNotFound do request('http://flabergasted.cx') end end def test_404s_on_non_image_content_type assert_raise RestClient::ResourceNotFound do request('https://github.com/atmos/cinderella/raw/master/bootstrap.sh') end end def test_404s_on_10_0_ip_range assert_raise RestClient::ResourceNotFound do request('http://10.0.0.1/foo.cgi') end end 16.upto(31) do |i| define_method :"test_404s_on_172_#{i}_ip_range" do assert_raise RestClient::ResourceNotFound do request("http://172.#{i}.0.1/foo.cgi") end end end def test_404s_on_169_254_ip_range assert_raise RestClient::ResourceNotFound do request('http://169.254.0.1/foo.cgi') end end def test_404s_on_192_168_ip_range assert_raise RestClient::ResourceNotFound do request('http://192.168.0.1/foo.cgi') end end def test_404s_on_environmental_excludes assert_raise RestClient::ResourceNotFound do request('http://iphone.internal.example.org/foo.cgi') end end def test_follows_temporary_redirects response = request('http://d.pr/i/rr7F+') assert_equal(200, response.code) end def test_request_from_self assert_raise RestClient::ResourceNotFound do uri = request_uri("http://camo-localhost-test.herokuapp.com") response = request( uri ) end end end class CamoProxyQueryStringTest < Test::Unit::TestCase include CamoProxyTests def request_uri(image_url) hexdigest = OpenSSL::HMAC.hexdigest( OpenSSL::Digest::Digest.new('sha1'), config['key'], image_url) uri = Addressable::URI.parse("#{config['host']}/#{hexdigest}") uri.query_values = { 'url' => image_url, 'repo' => '', 'path' => '' } uri.to_s end def request(image_url) RestClient.get(request_uri(image_url)) end end class CamoProxyPathTest < Test::Unit::TestCase include CamoProxyTests def hexenc(image_url) image_url.to_enum(:each_byte).map { |byte| "%02x" % byte }.join end def request_uri(image_url) hexdigest = OpenSSL::HMAC.hexdigest( OpenSSL::Digest::Digest.new('sha1'), config['key'], image_url) encoded_image_url = hexenc(image_url) "#{config['host']}/#{hexdigest}/#{encoded_image_url}" end def request(image_url) RestClient.get(request_uri(image_url)) end end camo-1.3.0/test/proxy_test_server.rb000066400000000000000000000002711224374550400175520ustar00rootroot00000000000000require 'thin' class ProxyTestServer def call(env) [302, {"Content-Type" => "image/foo"}, "test"] end end Thin::Server.start('127.0.0.1', 9292) do run ProxyTestServer.new endcamo-1.3.0/tmp/000077500000000000000000000000001224374550400132405ustar00rootroot00000000000000camo-1.3.0/tmp/.gitignore000066400000000000000000000000001224374550400152160ustar00rootroot00000000000000