random toughts about geek stuff

Displaying directories first in Mac OSX


I’m a linux user which recently moved to Mac. Most of the time I’m using the Terminal application and something I really miss in the commad ls of Mac is that it does not have the option --group-directories-first.

$ ls --group-directories-first
/bin/ls: illegal option -- -

Installing commands from coreutils

First of all, you need to install the command ls from coreutils, what is the same package that runs in all common linux distributions. You can use brew you install coreutils:

$ brew install coreutils

By default, all the commands will be installed using the prefix “g” so now you have the following:

$ gls --group-directories-first

If you, like me, want to always use the standard ls command from coreutils, you can add the following alias at your $HOME/.bashrc file:

alias ls='gls --color=auto --group-directories-first'

Outgoing traffic filetered by DigitalOcean!?


I usually host my machines on Linode but today I helped a friend of mine to configure his brand new server on DigitalOcean.

We were using Configr to provide the services deployment and just after the initial configuration, we setup Mandrill Add-On in order to improve the email deliverability rate. Everything configured and properly deployed, so it’s time to test the environment… and we got no emails delivered. =/

The error message from exim log was:

routing defer (-51): retry time not reached

Going deeper on the problem, we double checked that we had no firewall rules being applied in our server but we noticed that we still weren’t able to connect with Mandill servers:

$ telnet smtp.mandrillapp.com 587
telnet: Unable to connect to remote host: Connection timed out

How is it possible? Every other well-known network ports was working correctly… So looks like someone is filtering our outgoing traffic.

After some time spent with support tickets on Digital Ocean and more tests asked by support team, we got this final answer from them:

Greetings, It appears that your account was mistakenly blocked by our backend system after some automated abuse triggers. The abuse triggers erroneously marked your account as a spam account. I apologise deeply for this inconvenience. The block has been removed and full connectivity should be restored. Please let me know if you have any more questions.

That said, our emails started to be delivery automagically… What great initial user experience huh!?

Converting plaintext files using VIM


I work with mixed environments: some people using Windows, some people using Linux (or Mac). If you are in the same situation, a common scenario is to be working on a remote linux server and to find a file in DOS format, or running a git pull command and receiving a file in this format from someone who uses Windows.

Converting the file format

If you have the tofrodos package installed, this problem is easily solved with fromdos command … but if you don’t, you can use vim to convert the format by simply openning the file, setting the file format and then saving it:

:set ff=unix

and that’s it, now the file os converted to unix format.

Converting the file encoding

Another great feature of vim and which also saves lots of time is: encoding conversion. As I live in Brazil, it’s common to find ISO-8859-1 files and when it happens, converting it back to UTF-8 is as simple as openning the file on vim, setting the file encoding and saving it:

:set fenc=utf-8

Actually, I’m used to convert file encodings using vim, but the file format I just discovered this week and it saved a lot of time. It’s nice to see how usefull one single tool can be! :)

Lets get back to blogging


Sometimes you need a place where you can save those notes that are neither important enough to justify a repository on github nor simple enough to be just forgotten. For these cases there is nothing better than a blog… so here we go. :)

As I’m intended to write most of the time only small pieces of code and its relevant comments, I decided to use octopress since it allows me to create new posts using vim and publish them with one single command.

Clearing the screen with IPython


Some time ago I changed my default shell to use IPython instead of Bash. The IPython is a really great as a shell and IMHO has great advantages but for those who, like me, are completely addicted to the clear command, the IPython’s clear is a huge disadvantage.

In IPython, the clear command clear varios data like input, output and directory histories but it doesn’t clear the screen! You can take a look at the documentation of the default function for the clear command in IPython below and see how it works:

$ from IPython.Extensions.clearcmd import clear_f
$ print clear_f.__doc__
 Clear various data (e.g. stored history data)

    %clear out - clear output history
    %clear in  - clear input history
    %clear shadow_compress - Compresses shadow history (to speed up IPython)
    %clear shadow_nuke - permanently erase all entries in shadow history
    %clear dhist - clear dir history

So I created a new function to extend the first one by adding the option to clear the screen when no args were passed. If you’d like to do the same, you just need to add the following lines in your ~/.IPython/ipy_user_conf.py:

def new_clear_f(*args, **kwargs):
    ''' Extend the default clear function adding the option to
    clear the screen when no args were passed.'''
    if not args[1]:
        from IPython.Extensions.clearcmd import clear_f
        clear_f(*args, **kwargs)
        ip.expose_magic('clear', new_clear_f)
ip.expose_magic('clear', new_clear_f)

And now it all seems to be the standard clear again. :)

Using IPython as your default shell


If you are a shell addicted person and like Python, then the IPython was made just for you. The IPython provides a very usefull feature (that may look a bit strange at first) of mixing Python and Shell commands in a easy and intuitive way. I don’t pretend to say why you should use IPython instead of your current shell but I believe that if you know Python, you should prefer to write Python code instead of Bash code.

I’ll assume that you already have the IPython installed, but if you don’t take a look at IPython’s documentation to see how to install it. I’m running Debian, so this setup is focused on it. It should work for the most of the others linux distros but if you are running a different system, I’m sorry but you may have to adapt some commands and/or paths. :)

Configuring the IPython’s prompt

The first time you run IPython, it displays a warning about the creation of your personal configuration directory. One of the most important parts of this warning is about the new configuration file: ~/.ipython/ipy_user_conf.py. So let’s configure this file and the IPython’s prompt to make it looks exactly like the Debian’s prompt. Edit your configuration file and change the following settings as below:

import ipy_profile_sh
o.prompt_in1 = r'\C_Normal\u@\H:\Y2$ '
o.prompt_in2 = r'\C_Normal... '

The first option, import ipy_profile_sh, avoid you to have to escape all the Bash commands. The second and third options, o.prompt_in1 and o.prompt_in2, change the IPython’s prompt. And now start the IPython as follow:

afurlan@wetzel:~$ ipython -nosep -nobanner -noconfirm_exit
afurlan@wetzel:~$ print 'hello python!'
hello python!
afurlan@wetzel:~$ echo 'hello bash!'
hello bash!

Great, now you’re able to run Python commands in your iPython shell:

afurlan@wetzel:~$ for line in file('domains.txt'):
              ...     print line


And a mixed of Python and Bash commands:

afurlan@wetzel:~$ for line in file('domains.txt'):
              ...     echo "$line"


afurlan@wetzel:~$ for line in file('domains.txt'):
          ...     echo "$line.strip()"

Setting the IPython as your default shell

Actually I don’t like to set the IPython as my default shell… I use to run screen so I prefer to configure it to open five IPython sessions by default, but if you really like to set it as your default shell, you can add the following line at the end of your ~/.bashrc file:

exec ipython -nosep -nobanner -noconfirm_exit

If you’re a screen user like me, you can add the following lines at your ~/.screenrc:

## open 5 ipython sessions by default
screen -t p 1 ipython -nosep -nobanner -noconfirm_exit
screen -t p 2 ipython -nosep -nobanner -noconfirm_exit
screen -t p 3 ipython -nosep -nobanner -noconfirm_exit
screen -t p 4 ipython -nosep -nobanner -noconfirm_exit
screen -t p 5 ipython -nosep -nobanner -noconfirm_exit

Or, if you like, you can also get my current .screenrc file.

Playing with Python and GeoIP in Debian


Sometime ago I needed to find out the origin country of an hostname (actually a list of hostnames). As Debian has a GeoIP database available via aptitude, I could accomplish that using this database and the Python GeoIP module.

First of all I installed the required packages:

$ aptitude install geoip-database python-geoip

and then I used the following small script that configures the Python’s GeoIP module to use the Debian’s GeoIP database:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# written by Arthur Furlan <afurlan@afurlan.org>

import re
import sys
import GeoIP

## oops, check the usage! :)
if len(sys.argv) != 2:
    print 'Usage: %s ADDRESS' % sys.argv[0]

## geoip database from "geoip-database" debian package
GEOIP_DATABASE = '/usr/share/GeoIP/GeoIP.dat'

## get the country name based on the given address
if re.match('^[0-9]{1,3}(\.[0-9]{1,3}){3}$', sys.argv[1]):
    response = geoip.country_name_by_addr(sys.argv[1])
    response = geoip.country_name_by_name(sys.argv[1])

## print the response if there is any response
if response:
    print response

Some tests to check if it is really working… and looks like it is. :)

$ python address_country.py brasil.gov.br
$ python address_country.py configr.com
United States
$ python address_country.py debian.com
$ python address_country.py mandriva.com
$ python address_country.py some-nonexistent-domain.org