Monkey Snatch Banana

In Search of Twitter Client Nirvana

2010-09-02T00:00:00-04:00

Why I Need A New Twitter Client

I’ve gone through a number of twitter clients, switching as I’ve found new features that I consider ‘must have’ items. Currently I use echofon. Until today I was happy with it however, I just hit a very large snag. There are many user streams I’d like to ‘sample’ without adding noise to my main stream; following these streams is out of the question as I’m only interested in small percentage of their tweets. Lists seem like the perfect mechanism to deal with this problem. I’ve created a private list called ‘dip into’ that I can jump over and scan from time to time without cluttering my main stream of tweets I’m always interested in. In addition to ‘dip into’, I have a number of other lists for things that I’m interested in but don’t consider time sensitive¹. Unfortunately, Echofon’s handling of lists is horribly lacking. It only tracks new tweets/last read for the last list you looked at in the last account you had open. This means, I have to remember where I was in most lists I’ve created; an extreme non-starter. I have come to the conclusion that I either have to stop using lists or replace echofon.

Must Have Features

Mac OS X: I use Mac OS X.
Multiple Accounts: I have multiple accounts and need to be able to easily switch between them.
Threaded Conversations: Twitter became far more useful to me when I started using a client that supported threading conversations.
Remember Last Read Tweet Per View: If the client doesn’t support remembering the last read tweet and returning you back there, it isn’t useful it me. It needs to remember the last read tweet in each view I’m in: main stream by account, lists etc.
Account Control: Echofon has a feature I’ve found to be a massive time saver: Click on an account name and it opens up a side window with a their tweet stream plus controls change lists the account is in along with follow, unfollow, block and report as spam options.

Nice To Have Features

Built in Translation: I follow non-english speakers and it would be nice to be able to press a translate button to see what their non-english tweets are. Bonus points for being able to set specific users to be auto-translated.
Muting: Ability to mute a user or hashtag for a period of time
Saved Searches: Access to commonly used searches via the client is nice. I tend to do a lot of searches for smalltalk and clojure looking for new people to follow.

Can You Help?

If you know of a twitter client that meets my needs, please drop me a line via twitter or email.

¹ e.g. @SeanTAllen/services

Using FastCGI with Nginx and Seaside

2010-08-17T00:00:00-04:00

Why FastCGI a.k.a Why Not Reverse Proxy?

I’ve previously written about why I prefer reverse proxying to FastCGI. I still believe all those points, however there are certain Seaside deployment situations where FastCGI is currently preferable to reverse proxying. If you are deploying your Seaside application to run on Gemstone’s GLASS then you should give FastCGI serious consideration. The current swazoo adapter that you use when proxying to GLASS is nowhere near as battle tested as the FastCGI adapter. If you want a rock solid deployment, FastCGI will give you much better results than Swazoo.

Our Bare Bones Nginx FastCGI configuration

 1 worker_processes 1;
 2 
 3 events
 4 {
 5   worker_connections  1024;
 6 }
 7 
 8 http
 9 {
10   include       mime.types;
11   default_type  application/octet-stream;
12   
13   upstream seaside 
14   {
15     server localhost:9001;
16     server localhost:9002;
17     server localhost:9003;
18   }
19 
20   server
21   {
22     root /var/www/glass/;
23 
24     location /
25     { 
26       error_page 403 404 = @seaside;
27     }
28     
29     location @seaside
30     {
31       include fastcgi_params;
32       fastcgi_pass seaside;
33     }
34   }
35 }

The preceeding bit of code is a mostly bare bones Nginx to GLASS FastCGI configuration. There isn’t a whole lot to it, so I will quickly hit the points of interest:

Lines 13-18

upstream seaside 
{
  server localhost:9001;
  server localhost:9002;
  server localhost:9003;
}

Define a named upstream pool called seaside. The default GLASS installation sets up 3 FastCGI listeners on ports 9001 to 9003. You would just add more server entries in the upstream block to have Nginx start using any additional FastCGI listeners that you might setup in GLASS.

Line 22

root /var/www/glass/;

Defines where our static assets like images, css and html are served from. You need to change the definition to a directory you create or create /var/www/glass/ and populate it with your various assets.

Lines 24-27

location /
{ 
  error_page 403 404 = @seaside;
}

Define a location handler¹ that encompasses everything that matches ‘/’. Any standard uri will match against this location so, you can consider it to be “global” in scope. Line 26 states that any uri request that would result in an http 403 or 404 response code should be routed to the named location @seaside. N.B. This named location is not the same as our upstream block. Nginx will process a request as follows:

Check to see if a static asset for the uri exists. If it does, return it.
If a static asset doesn’t exist then either a 403 or 404 error will be generated resulting in handing the uri off to the @seaside location.

This could also be done as:

location /
{ 
  try_files $uri @seaside;
}

Getting into the details of why you would prefer try_files or error_page is the subject for another post.

Lines 29-33

location @seaside
{
  include fastcgi_params;
  fastcgi_pass seaside;
}

Define our named @seaside location that is responsible for handling FastCGI. The include fastcgi_params directive has Nginx include its standard FastCGI protocol definitions that are installed at /etc/nginx/fastcgi_params. The fastcgi_pass directive simply says: ‘pass all uris from this to the servers defined in the seaside upstream block’.

Are We Done?

Yes. Yes we are. There is plenty more configuration that can be added to make the FastCGI handling more robust ( failovers, timeouts etc ), but that is all you need to get started. I would suggest visiting the Nginx wiki to learn about Nginx so you can flesh out your configuration. If you run into issues or need any help, feel free to drop me an email or catch up with me on Twitter.

¹ See the Nginx wiki for more information on the location directive.

² Directory indexes’ that don’t exist like ‘/’ will result in a 403; regular files that don’t exist like ‘/welcome’ will result in a 404.

Reverse Proxying to Seaside with Nginx

2010-06-23T00:00:00-04:00

Why Reverse Proxy a.k.a Why Not FastCGI?

Transparency.

FastCGI setups are opaque. You can’t easily test all aspects of it. Unless you write your own FastCGI client, debugging configuration problems can be very difficult. Lets imagine for a moment that are trying to debug a problem with your application— the issue might be in the front end server or it might be in the Seaside application server. When you are using FastCGI, all your interactions with your application server are routed through the front end server which acts as a client for you. If you are reverse proxying to your Seaside application running at http://127.0.0.1:8080/, you can fire up lynx¹ and hit the application directly. No error from the application? Probably a front end server issue. It might be slightly hyperbolic, but I can’t count the number of hours of head scratching this has saved me.

Reverse Proxying to Seaside Options

Now that you have decided that you want to use reverse proxying to your Seaside application, you need to decide between a couple of basic setup options. You can either setup each application on its own subdomain of your domain or you can run multiple applications off the same domain.

One Application Per Subdomain

Our first example, shows a basic setup for running the counter and multicounter example applications off their own subdomains. With this configuration, you would be able to access the counter example via the uri http://counter.example.com/ and the multicounter example via http://multicounter.example.com/.

 1 server
 2 {
 3   server_name counter.example.com;
 4   
 5   rewrite ^/$ /examples/counter;
 6   proxy_redirect /examples/counter /;
 7 
 8   location /examples/counter
 9   {
10     proxy_pass http://127.0.0.1:8080;
11   }
12 }
13 
14 server
15 {
16   server_name multicounter.example.com;
17   
18   proxy_redirect /examples/multicounter /;
19   rewrite ^/$ /examples/multicounter;
20 
21   location /examples/multicounter
22   {
23     proxy_pass http://127.0.0.1:8080;
24   }
25 }

If you’ve never seen a nginx configuration file before that might be quite a bit to absorb, so lets break it down:

We setup two virtual servers, one for each application. This is done via the server {} directives that start on lines 1 and 14. Each virtual server is almost identical except counter appears in one and multicounter in the other.
Line 3 tells our first server to respond to requests for the host counter.example.com.
Line 5 rewrites http://counter.example.com/ to http://counter.example.com/examples/counter. This change in URI isn’t seen by the client browser.
Line 6 removes /examples/counter from URIs being returned by the counter application.
Line 8 sets up an nginx location. It passes the URI /examples/counter to a our application server listening on port 8080 of localhost.

Multiple Applications Per Domain

For our multiple applications per domain example, we are going to setup example.com to server any of the example Seaside applications. All the example Seaside applications start with their URI path with /examples— a nicety we will take advantage of.

1 server
2 {
3   server_name example.com;
4 
5   location /examples
6   {
7     proxy_pass http://127.0.0.1:8080/examples;
8   }
9 }

The nginx location we setup on line 5, routes anything under /examples to the same path on our application server. Under this setup, we could access the counter application at http://example.com/examples/counter and multi counter application at http://example.com/examples/multicounter.

Complete Nginx Reverse Proxy Server Configuration²

Building on the ‘One Application Per Subdomain’ setup above, lets setup the serving a hypothetical To Do application. Our application will be responsible for serving the site’s homepage. Nginx will serve any static assets ( css files, images etc ).

 1 server
 2 {
 3   server_name todo.example.com;
 4  
 5   root /var/www/todo.example.com/;
 6   
 7   location /
 8   {
 9     try_files $uri $uri/;
10   }
11   
12   proxy_redirect /todo /;
13   rewrite ^/$ /todo;
14 
15   location /todo
16   {
17     proxy_pass http://127.0.0.1:8080;
18   }
19 }

A quick breakdown of the new elements we’ve introduced:

Line 5 tells nginx that any static attributes should be located by looking in /var/www/todo.example.com/. This is equivalent to Apache’s DocumentRoot directive.
The location directive that starts on line 7 defines how nginx should locate static assets³. The try_files directive on line 9 tells nginx to try to means to satisfy a URI. If our incoming request has a path of /about then, nginx will first look for a file called about in the server root. If that isn’t found nginx will look for a directory about/ in the server root. If neither rule can be satisfied, nginx will return a 404 error.
Line 15 defines a location rule that is more specific than the one on line 7. It will only be triggered if the URI path is /todo. This is the same setup that we saw in both of our previous examples.

Putting It All Together

Now all we have to do is drop our reverse proxy server configuration into our nginx configuration and we end up with something like⁴…

worker_processes 1;

events
{
  worker_connections  1024;
}

http
{
  include       mime.types;
  default_type  application/octet-stream;
      
  server
  {
    server_name todo.example.com;
 
    root /var/www/todo.example.com/;
  
    location /
    {
      try_files $uri $uri/;
    }
  
    proxy_redirect /todo /;
    rewrite ^/$ /todo;

    location /todo
    {
      proxy_pass http://127.0.0.1:8080;
    }
  }
}

¹ Or any other http client; one of my favorites is http console.

² See http://wiki.nginx.org/NginxHttpProxyModule for complete nginx proxy module documentation. If you are running multiple Seaside application instances ( on GLASS for instance ), you should also check out the upstream, fair upstream and upstream_hash modules.

³ Location / is the most general location rule possible with nginx. It will used if a more specific location can’t be found.

⁴ Yours may look radically different. This is just a basic example of a full nginx configuration file.

Faking a https client for GLASS

2010-06-22T00:00:00-04:00

So, you are humming along developing your GLASS hosted Seaside application. If you are following the proscribed development path, your application is running in Pharo and you’ve started porting it to GLASS when a problem arises:

Your application needs to make calls out to a secure web service ( merchant processor etc ) but, GLASS doesn’t have a native https client. The Pharo “cryptography package:”http://www.squeaksource.com/Cryptography.html implements an https client which you could port to Gemstone, but the process model it uses doesn’t map nicely and you have better things to do with your time. You could wrap up the C OpenSSL library and access it from Smalltalk— also a large amount of work or… you could go for the quick, dirty and effective approach: open a regular http connection and run it through a https proxy. This article covers the last option.

Http to Https Proxying

A proxy server is a server (a computer system or an application program) that acts as an intermediary for requests from clients seeking resources from other servers. In our current scenario, our Seaside application will make an http connection to a local proxy that will in turn open a https connection to a remote server passing our info along in the process. In order to accomplish this, we will give a hostname to our proxy and connect to it from our smalltalk code. Lets say you want to use a resource on the server api.secure_service.com, you need to setup a name for the proxy that does the actual connection to api.secure_service.com. If proxy is running on your local machine and you want to name it secure_service, add the following entry to your machine’s /etc/hosts file¹:

127.0.0.1   secure_service

From our smalltalk code instead of opening a connection to https://api.secure_service.com/apiLocation, we will open a connection to http://secure_service/apiLocation. With this simple code change out of the way, we can move on to setting up the proxy. There are two primary options for doing http to https proxying. Using stunnel or a proxying web server:

Stunnel

I’m not going to give an in depth coverage to setting up stunnel as a means of doing outgoing https connections from GLASS. Why? For any production GLASS system, we are probably going to be running a web server in front of the swazoo, hyper or fastcgi adapter that is serving your Seaside content; that web server is all you need. If we use stunnel to as a proxy for our http to https connections, we are just adding another point of failure to our setup. By using our existing web server as the proxy, there is less to worry about. If our web server goes down, we aren’t going to need the https proxying as we have bigger problems.

Proxying Web Server

We could use apache, lighttpd or any other proxying web server to implement the following solution. I’ll use nginx ³ as it is the server I’m most comfortable with. We already added a name for our /etc/hosts and now it is time to add an additional server entry to our nginx.conf file².

server
{
  server_name secure_service;
  
  location /
  {
    proxy_pass https://api.secure_service.com;
  }
}

Restart nginx and we are good to go.

Using test resources

If we use a build system and the remote service offers production and testing versions, we can get an added bonus from the proxy setup: easy switching at build time between production and testing services.

Let’s assume that secure_service is a merchant processing provider that provides a test interface where no billing is actually done as well as a production system where transactions are charged to the consumer. Instead of having testing and production objects that change the url we are using, we can have a build system insert the correct url in our nginx.conf file. If we are using a standard c preprocessor ⁴, the following setup and slight nginx.conf change allows us to switch from production to testing at build time rather than relying on switching objects in our smalltalk code:

#ifdef PRODUCTION
#define SECURE_SERVICE_URL  production-api.secure_service.com
#else
#define SECURE_SERVICE_URL  testing-api.secure_service.com
#endif

server
{
  server_name secure_service;
  
  location /
  {
    proxy_pass https://SECURE_SERVICE_URL;
  }
}

¹ If we need to access your proxy from multiple machines, using real dns entries for our domain will make maintenance much easier.

² Normally located in /etc/nginx.conf.

³ You need to configure nginx at build time to include https support.

⁴ Any preprocessor would do.

Compress images to speed up your website

2010-02-01T00:00:00-05:00

Everyone has heard that you need to compress images on your website to get the best performance, but not everyone knows how. There are a number of programs that can help. If you use linux, I would suggest:

pngcrush for png
jpegtran for jpeg
gifsicle for gif

Debian & Ubuntu both have packages for all 3. Combine with the following script and you are moments away from better site performance:

#!/usr/bin/ruby
require 'find'

Find.find( ARGV[0] ) do | file |
  if file.match( '\.gif$' )
    command = sprintf( 'gifsicle --batch -O2 %s', file )
    puts command
    system( command )
  end
  if file.match( '\.jpe?g$' )
    command = sprintf( 'jpegtran -copy none -optimize -outfile %s %s', file, file )
    puts command
    system( command )
  end
  if file.match( '\.png$' )
    new_file = "#{file}-crushed"
    command = sprintf( 'pngcrush -q -rem alla -brute -reduce %s %s', file, new_file )
    puts command
    system( command )
    if File.exists?( new_file )
      if File.size( new_file) < File.size( file )
        command = "mv #{new_file} #{file}"
      else
        command = "rm #{new_file}"
      end
      puts command
      system( command )
    end
  end
end

Just pass the local filesystem root of your website to the script and it will compress all gif, jpeg and png images that it finds within the website hierarchy.