Realtime passenger statistics

run this in a bash console:
watch -n1 /usr/local/lib/ruby/gems/1.8/gems/passenger-2.2.15/bin/passenger-status

using curl to test rails app session management

I am building a rails app that communicates json back and forth to a client. I needed to test the session creation mechanisms and also some posting of objects. Here are some handy curl calls:

Post a json object { "FishID":"greatwhite" } while also passing the Content-Type and Accept headers as "application/json" so rails will autmatically turn the body of the request into params in the params[] array the below would be translated into [:FishID=>"greatwhite"]

curl -H "Content-Type:application/json" -H "Accept:application/json" -d "{\"FishID\":\"greatwhite\"}" http://74.116.250.34/createFish


Post a token to a session start action that will return the session id to the client then save the cookies that are sent back to a file that can be re-used in subsequent curl calls thus simulating a longer running session

curl -H "Content-Type:application/json" -H "Accept:application/json" -d "{\"token\":\"ja9er4fn9\"}" -c cookies.txt http://localhost/beginSession

Now put the two together:

Re-use a text file with cookies saved by a previous curl session to re-use the old session just use the -b switch again
curl -H "Content-Type:application/json" -H "Accept:application/json" -d "{\"FishID\":\"greatwhite\"}" -b cookies.txt http://localhost/createFish

That last call should be able to instantiate a fish for the account of the user based on the session from the session cookie.

A quick and dirty bash script to periodically clean out a static html mirror of a dynamic site

I use nginx to flatten my site to html buffers on disk just in case varnish ever crashes i can warm up the cache with the last known good coppies of stuff. However to make sure that we don't constantly serve old pages i move the files from a "fresh" folder to a "stale" folder every so often. I do some tricky stuff in my nginx config to check the upstream server first. If that fails (is overloaded) then check the "fresh" folder and THEN if worst comes to worst check the "stale" server and serve from there. At least the user doesn't see an error page. Just slightly older content.

Anyway I will post the nginx config soon. for now here the new pure bash version of the cleanup script. I posted a perl version a little while back. I think this version is faster.

#!/bin/bash
MINS=30 #the age threshold in minutes at which point the file is moved to stale
DIR=/cache/fresh #starting directory
NEWDIR=/cache/stale #directory to move to

cd $DIR
for file in `find . -type f -mmin +$MINS`
do

#we need to get the directory name holding our file and remove the leading . so that it is just /foo/bar
backup_dir=$(dirname $file| sed 's/^\.\///')

#we don't actually need to use the basename of the file unless we are going to move it to a different directory inside the stale directory than it was inside the fresh directory
#file_name=$(basename $file)

# the following just makes sure to skip the . directory if the find command picks it up
dot="."
if [ $backup_dir = $dot ] ; then
backup_dir=""
fi

#lets check to see if the supposed new directory is already in the stale folder as a "file" rather than a directory. sometimes when using pretty urls or a combination of pretty urls and non pretty urls we can end up with files that should be directories. just because a files ends in .php on disk doesnt mean that it doesnt have virtual "subdirectories" under it when viewed via the web ex: /page.php/1/ which in our flat html version should be /page.php/1/index.html so lets delete the file and replace it with a directory.
if [ -f $NEWDIR/$backup_dir ] ; then
rm -f $NEWDIR/$backup_dir
fi
mkdir -p $NEWDIR/$backup_dir

#if we were moving our stale cache to another server we could create the directory by issuing remote ssh commands ex
#ssh testaccount@192.168.10.15 mkdir -p $DIR/$backup_dir

#mv $file $NEWDIR/$backup_dir #mv is coughing on files with spaces in the name. I could spend the time to regex escape the special characters in the filename but why bother when rsync works just fine
rsync --stats -auvz --remove-sent-files --times -og $file $NEWDIR/$backup_dir
rm -f $file
done

bash oneliner to clear out apache httpd semaphores

If your seeing the following error message in your apache error logs

[emerg] (28)No space left on device: Couldn't create accept lock

You probably need to clear out some stale httpd semaphores. The following oneliner will do that for you.

ipcs -s | grep apache | perl -e 'while () { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

log log log

Log stand and error outs to syslog and prefix it

#!/bin/bash
/usr/local/bin/something 2>&1 | logger -p daemon.notice -t ${0##*/}[$$]

Reload nginx config

First make your changes to nginx.conf.

Then run the following command to test the new configuration:

# nginx -t -c /etc/nginx/nginx.conf
2007/10/18 20:55:07 [info] 3125#0: the configuration file /etc/nginx/nginx.conf syntax is ok
2007/10/18 20:55:07 [info] 3125#0: the configuration file /etc/nginx/nginx.conf was tested successfully
Next, look for the process id of the master nginx process:

# ps -ef|grep nginx
root 1911 1 0 18:00 ? 00:00:00 nginx: master process /usr/sbin/nginx
www-data 1912 1911 0 18:00 ? 00:00:00 nginx: worker process
Lastly, tell nginx to reload the configuration and restart the worker processes:

# kill -HUP 1911

taken from:
http://snippets.aktagon.com/snippets/93-Change-nginx-configuration-on-the-fly

bash one liner (essentially) to loop through list of files and upload using rsync

for f in `cat file-includes.txt`; do rsync -avz $f user@server.ip:/base-path/$f; done

you could also do the -e 'ssh -i /path/to/preshared/key' to avoid the rsync password prompt on each connect

for f in `cat file-includes.txt`; do rsync -avz -e 'ssh -i /path/to/preshared/key' $f user@server.ip:/base-path/$f; done

I find that this works better for only syncing a short list of files than trying to use the rsync switch --include-from=/path/to/file-includes.txt along with the --exclude-from=/path/to/file-excludes.txt or the --exclude=*

for some reason i couldnt get it to exclude everything BUT what was in my includes-from txt file.


Here is an even more expanded version that creates the white list of files for you containing all visible files in the current directory. You will probably want to modify it so that it only includes a subset of that. Otherwise it is pretty pointless (because it just does what rsync normally does uploads all changed files). You can hand create your white list of files to upload or use svn commit messages even.

ls -1 ./ > ./file-includes.txt; for file in `cat ./file-includes.txt`; do rsync -avz -e 'ssh -i /path/to/preshared/key' $( readlink -f "$( dirname "$file" )" )/$( basename "$file" ) user@server.ip:/base-path/$( readlink -f "$( dirname "$file" )" )/$( basename "$file" ); done; rm ./file-includes.txt

mysqldump to comma delimited file

mysqldump -u username --password=pass -h hostip -t -T./ dbname tablename --fields-enclosed-by=\" --fields-terminated-by=,

syntax reminder on how to send mail from the command line

I always forget the params for this stuff

mail -s "subject" -c "person.to.cc.to@blah.com another.person.to.cc.to@blah.com" person.for.to.address@blah.com < /path/to/message/contents/text/file.txt

page up down in vi in mac os terminal

Move up or down in “vi” from Mac OS X terminal:

Then CTRL-B and CTRL-F should do the trick.

.vimrc settings

These are my current vim settings

let mysyntaxfile = "/Users/jessesanford/.vim/go.vim"
syntax on
set nu
set ai
set tabstop=4
set ruler
set laststatus=2
set showmode
set expandtab
let loaded_matchparen=1

How to execute remote bash commands using ssh

"...

ssh -t YOURHOST "bash --rcfile PATH_TO_RCFILE_ON_REMOTE_HOST_HOME_DIR_SHORTCUTS_WORK_FOR_AUTHED_USER"

bash will be executed on the remote host, and bash will execute the specfied RCFILE at startup, and connection will remain open. -t is to have the current terminal forwarded to the ssh session so that you have a real terminal.

You can have some variant on the same kind, still use ssh -t. Like if screen is installed, you can do:

ssh -t YOURHOST screen

..."

Alternative:

"...

ssh YOURHOST bash --rcfile YOUR_RC_FILE -i

But then you don't have a real terminal, and some stuff will not work correctly (like tab auto-completion).

..."

from:

http://www.linuxforums.org/forum/linux-networking/102713-how-execute-remote-shell-commands-via-ssh.html

my current bash_profile

export PS1="\\u@\\h:\\w\\$ "
export EC2_HOME=~/.ec2/ec2-api-current
export ELB_HOME=~/.ec2/ec2-ElasticLoadBalancing-current
export M2_HOME=~/apache-maven-2.2.0
export M2=$M2_HOME/bin
export EC2_CERT=~/.ec2/cert-R4SBVFO3FBH7TLS27NS7GEL5FG345ZBJ.pem
export EC2_PRIVATE_KEY=~/.ec2/pk-R4SBVFO3FBH7TLS27NS7GEL5FG345ZBJ.pem
export AWS_X509_CERT=~/.ec2/jessesanford.pem

export JAVA_HOME=/Library/Java/Home

export EDITOR=/usr/bin/vim

export PATH=/opt/local/bin:/opt/local/sbin:$PATH
export PATH=/opt/local/apache2/bin:/opt/local/subversion/bin:$PATH
export PATH=~/zero:~/apache-maven-2.2.0/bin:/usr/local/zend/share/ZendFramework/bin:$PATH
export PATH=${PATH}:~/.ec2/ec2-api-current/bin:~/.ec2/ec2-ami-current/bin:~/.ec2/ec2-ElasticLoadBalancing-current/bin:~/.ec2/ec2-CloudWatch-current/bin:~/.ec2/ec2-AutoScaling-current/bin

test -r /sw/bin/init.sh && . /sw/bin/init.sh

alias mysqlstart='sudo /opt/local/bin/mysqld_safe5 &'
alias mysqlstop='/opt/local/bin/mysqladmin5 -u root -p shutdown'

alias apachestart='sudo /opt/local/apache2/bin/apachectl start'
alias apachestop='sudo /opt/local/apache2/bin/apachectl stop'
alias apacherestart='sudo /opt/local/apache2/bin/apachectl restart'

if [ -f /opt/local/etc/bash_completion ]; then
. /opt/local/etc/bash_completion
fi

iterating through lines in a file in bash

quick bash script to allow you to iterate through lines in a file

#!/bin/bash

cat filename | while read line; do
echo $line
done

another way:

#!/bin/bash

IFS=$'\n'
for line in $(cat filename); do
echo $line
done

article on symfony svn externals

for projects i use subversion as the scm for symfony source

http://echodittolabs.org/blog/2009/09/symfony-and-svnexternals-super-slick-easy-way

pressflow a drupal branch for scaling

basicly these guys rebranded drupal with some core hacks that allow it to scale out database reads. i did a diff of the two files a while back and it looked very similar to the following drupal patch.

http://fourkitchens.com/pressflow-makes-drupal-scale/downloads

//database.inc
function db_query($query) {
$args = func_get_args();
array_shift($args);
$query = db_prefix_tables($query);
if (isset($args[0]) and is_array($args[0])) { // 'All arguments in one array' syntax
$args = $args[0];
}
_db_query_callback($args, TRUE);
$query = preg_replace_callback(DB_QUERY_REGEXP, '_db_query_callback', $query);

//load balancing
if(strpos(strtolower($_GET['q']),"admin") !== false)
db_set_active('write'); //its important that all admin gets access to the most recent data
else
if(strpos(strtolower($query),"select") === 0){
db_set_active('read'); //this will not contain any data from the master (write) database untill replication happens
}
else {
db_set_active('write');
}

return _db_query($query);
}

//sites/default/settings.php
//$db_url = 'mysql://username:password@localhost/databasename';

$db_url = array(
'default' => 'mysql://username:password@localhost/databasename',
'read' => 'mysql://username:password@localhost/databasename',
'write' =>'mysql://username:password@localhost/databasename'
);
?>

I got that patch from the following post on drupal.org:

http://groups.drupal.org/node/2147

2bits posts a lot on drupal performance

need to read some more of these. most stuff is known material but im sure there are some gems in there:

http://2bits.com/articles/drupal-performance-tuning-and-optimization-for-large-web-sites.html

article on performace monitoring lamp stack

this article could be extrapolated to things other than lamp. missing are mentions of things like nagios and zabbix both of which are the equivalents of what they are using cacti for in this article. i prefer nagios.

http://2bits.com/articles/tools-for-performance-tuning-and-optimization.html

Drupal Uploading files to folder outside of the webroot

for security reasons you shouldn't allow users to upload files to web accessible directories. I compiled these links after doing some research on how to change drupal's default behavior to allow files to be uploaded to a web inaccesible directory.

Kind of step by step:
http://www.vmtllc.com/drupal-as-an-intranet

Description of drupal public/private filesystem
http://drupal.org/node/230984

Ebook on files in drupal:
http://11heavens.com/files-in-Drupal

List of all file upload modules
http://groups.drupal.org/node/20291

using nginx as reverse proxy with http-acceleration (caching)

I had to use nginx on a project recently rather than varnish (needed ssl, virtual hosts etc) but I also needed to accelerate the drupal app behind the proxy because drupal can be soooo slow. here are some methods that i researched. i think i am going to use the first because it is easiest and most familiar (similar to what varnish does)

Proxy_cache methods (most like varnish):

http://www.ruby-forum.com/topic/183590

Nginx docs for proxy_cache:
http://wiki.nginx.org/NginxHttpProxyModule#proxy_cache

Proxy_store method:

Another older type of nginx caching (well its not really caching more like mirroring flat file copies of content) that just stores a copy of the requested uri's output from the upstream server to disk and then serves that copy indefinitley as long as it exists: (great for sites with content that does not change much)

http://lucasforge.2bopen.org/2009/09/caching-dynamic-content-using-nginx/

More info on using something like the above but this post includes a cron script to delete files from the cache after a certain length of time.

http://mark.ossdl.de/2009/07/nginx-to-create-static-files-from-dynamic-content/

The above 2 posts could be combined with a post like this (on nginx and memcachce) to store the repsonses in memcache rather than on disk:
http://www.igvita.com/2008/02/11/nginx-and-memcached-a-400-boost/

The question would be how to create propper key's into the memcache. I think that you could crc the output from the upstream servers and use that concatenated to the full uri for the object as the key. That way when the content changes the crc will change thus invalidating the old key and the old content that was stored with the old key.

Here are some posts about how to create keys for memcached content that changes and invalidating stale content:

http://blog.leetsoft.com/2007/5/22/the-secret-to-memcached

http://nubyonrails.com/articles/about-this-blog-memcached

Nginx docs for proxy_store:
http://wiki.nginx.org/NginxHttpProxyModule#proxy_store


Alternate caching plugin for nginx:

This project is interesting and would be awesome but looks too experimental and I don't read chinese :(

http://code.google.com/p/ncache/