Szkoda czasu

niedziela, 18 września 2011

How to secure your webpage with lighttpd mod_auth

Here's the setup:
There's a web page served by lighttpd by my OpenWRT router. Unfortunately the content shouldn't be available for everyone. Here's the solution:

  1. Install mod_auth

    You can do it with:

    opkg install lighttpd-mod-auth
  2. Create password file
    I first used plain authentication just to see if it's that easy as it's written in documentation. It turned out it is! Better solution is to use htdigest. You just need to create a file that looks like this:
    user1:realm:passwordhash
    user2:realm:passwordhash
    user3:another realm:passwordhash
    Realm is a name of a section of your website that you declare in configuration file. I just use one realm called "download", because I don't need anything more at the moment.
    Password hash is md5 sum calculated from a following string:
    username:realm:password
    e. g. john:website:qwerty would result in hash: 329516de44fe7cf1216194bb02348284.
    Entry in the password file would look like this:
    john:website:329516de44fe7cf1216194bb02348284

    Put your file somewhere where it's not accessible from outside of your server. You don't want anyone to be able to download this file and get to know all users' passwords.
  3. Update configuration
    Uncomment mod_auth in server.modules, for example:
    server.modules = (
    #       "mod_rewrite",
            "mod_redirect",
            "mod_alias",
            "mod_auth",
    #       "mod_status",
    #       "mod_setenv",
    #       "mod_fastcgi",
    #       "mod_proxy",
    #       "mod_simple_vhost",
    #       "mod_cgi",
    #       "mod_ssi",
    #       "mod_usertrack",
    #       "mod_expire",
    #       "mod_webdav"
    )

    Then add this somewhere down the line:
    ## AUTH
    auth.debug = 0
    auth.backend = "htdigest"
    auth.backend.htdigest.userfile = "/path/to/your/file/called/for/example/htdigest.user"
    
    When you want to secure access to your website with a password add:
    auth.require = ( "" =>
                            (
                                    "method" => "digest",
                                    "realm" => "website",
                                    "require" => "valid-user"
                            )
                           )
    
  4. Restart lighttpd
    You can do it with:
    /etc/init.d/lighttpd restart

Now try to access your webpage. You should be greeted with a dialogbox asking for user name and password. In case it's not working increase debug level to 1 or 2 and check out /var/log/lighttpd/error.log to find out what's wrong. Maybe wrong file name, file permissions or password hash? Good luck.

środa, 6 kwietnia 2011

Automatyczne pobieranie na napisów na OpenWRT

Opis

Poniższa instrukcja opisuje w jaki sposób zaplanować automatyczne poszukiwanie napisów dla filmów i odcinków seriali na routerze z systemem OpenWRT w wersji Backfire 10.03.

Instalacja

Potrzebny będzie pakiet napiprojekt, który znajduje się tutaj: napiprojekt. Instalacja następuje za pomocą:

opkg install http://eko.one.pl/openwrt/backfire/10.03/napiprojekt_6_ar71xx.ipk
Opkg zainstaluje potrzebne zależności z wyjątkiem p7zip, który można dostać stąd: p7zip. Instalujemy go podobnie:
opkg install http://ecco.selfip.net/backfire/packages/p7zip_9.13-3_ar71xx.ipk

Skrypt

Tworzymy nowy skrypt wydając polecenie:
nano /usr/bin/napisy.sh
Dzięki umieszczeniu w katalogu /usr/bin, który znajduje się w zmiennej środowiskowej $PATH, plik będzie można uruchomić z dowolnego miejsca w systemie. Następnie w treści pliku powinno znaleźć się:
#!/bin/sh
# Przejrzyj po kolei wymienione katalogi
for katalog in /srv/filmy /srv/seriale
do
 # Znajdz wszystkie pliki avi, wypisz pelna sciezke: "1 linia = 1 plik", wczytuj po jednej linii do zmiennej
 # Uzycie read zabezpiecza przed sytuacją w której w nazwie pliku występują spacje
 find $katalog -name "*.avi" | while read plik_z_filmem
 do
  # Zamien rozszerzenie avi na txt, zeby utworzyc nazwe pliku z napisami do danego filmu
  # Otaczamy nazwe zmiennej cudzyslowem na wypadek gdyby nazwa pliku zawierala spacje
  plik_z_napisami=$(dirname "$plik_z_filmem")/$(basename "$plik_z_filmem" .avi).txt
  # Jesli nie istnieje plik z napisami
  if [ ! -e "$plik_z_napisami" ]
  then
   # Szukaj napisow dla danego pliku
   napiprojekt "$plik_z_filmem"
  fi
 done
done
Plikowi należy nadać pozwolenie na wykonywanie:
chmod +x /usr/bin/napisy.sh

Cron

Dzięki temu, że skrypt pomija filmy i odcinki dla których już ściągnął napisy, można spokojnie zaplanować jego cogodzinne wykonywanie. W tym celu należy dodać następującą linijkę do pliku /etc/crontabs/root:
* */1 * * * napisy.sh
Przystępne wyjaśnienie i wygodny "kalkulator" dla crona można znaleźć na http://eko.one.pl/?p=openwrt-cron

poniedziałek, 21 marca 2011

Kalendarz Pyrkon 2011

Calendar

Code

This code helped me to automate most of the work. Last year I used gdata directly, but this year I got so frustrated that I decided I need something simpler. The tradeoff is that the events are not described in details as they used to be. This year I'm not making a mistake of not keeping the code, so here it is. There are basically three steps:
  1. Scrape the website to gather information
  2. Build a shell command to call googlecl - google command line interface that provides some ability to use Google services from command line.
  3. Run the command in the terminal. Keep retrying if it failed for any reason.
There were some issues:
  • Doubled entries that migrated from calendar "Naukowa 2" to "Naukowa", but showed up in their proper place too.
  • Sometimes starting times where off by an hour or two. For example by 1 hour on Friday, 1 hour on Saturday, 2 hours on Sunday. WTF? I noticed it happened on calendars that have numbers in their names, but most of them do, so I might be just imagining it.
  • Some problems with encoding. I could never understand when I need to decode/encode a string.
#!/usr/bin/python
# -*- coding:utf8 -*-

from BeautifulSoup import BeautifulSoup
import urllib2
import re
import subprocess

# Download the page
# You may want to save the page in the browser and use a local copy
# for example: 'file:///home/daniel/Pobrane/pyrkon.html'
page = urllib2.urlopen('http://www.pyrkon.pl/2011/index.php?go2=program')
soup = BeautifulSoup(page)

# Find div with the content
content = soup.find('div', id='content')
# Get all his children which are divs too
divs = content.findAll('div')

# Set starting index in case you wanted to start in the middle after some interruption
start_from = 0
i = 0
l = len(divs) - start_from

for div in divs[start_from:]:
 # Name and lecturer are easy
 tytul = div.contents[1].b.string
 prowadzacy = div.contents[1].i.string
 # I can never understand when I need to decode/encode from/to utf-8.
 # This was done by trial and error.
 
 # Madafaking new lines are contents too, so
 #  div.contents[2] == u'\n'
 
 # Place
 miejsce = re.search('^<b>miejsce: </b>(?P<miejsce>.+?)<br />', div.contents[3].renderContents(), re.M).group('miejsce')
 miejsce = miejsce.decode('utf-8')

 # Show some progress information
 i += 1
 print '[%d/%d] %s: %s' % (i, l, miejsce, tytul)

 # Event start time
 czas = re.search('^<b>termin: </b> (?P<dzien>pią|sob|nd)(\s*)(?P<godzina>\d{2}):(?P<minuta>\d{2})', div.contents[3].renderContents(), re.M)
 dzien, godzina, minuta = czas.group('dzien', 'godzina', 'minuta')
 godzina = int(godzina)
 minuta = int(minuta)
 
 # Conversion from name of the day to number of the day
 if dzien == 'pią':
  dzien = 25
 elif dzien == 'sob':
  dzien = 26
 elif dzien == 'nd':
  dzien = 27
 else:
  raise ValueError('Błędny dzień')
 
 # How long it lasts
 dlugosc = re.search('^<b>czas trwania: </b>(?P<godzin>\d+):(?P<minut>\d{2}) h<br />', div.contents[3].renderContents(), re.M)
 godzin, minut = dlugosc.group('godzin', 'minut')
 godzin = int(godzin)
 minut = int(minut)
 
 # Build shell command for googlecl - google command line interface (available at code.google.com)
 # uses "Quick Add" syntax
 polecenie = '''google calendar add --cal='%s' '%s - %s on %d/03/2011 %d:%02d for %d minutes in %s' ''' % (miejsce, tytul, prowadzacy, dzien, godzina, minuta, godzin * 60 + minut, miejsce)

 # Keep calling shell command until it succeeds
 # Sometimes it throws gdata.service.RequestError with status 302 and reason 'Redirect received, but redirects_remaining <= 0'
 return_code = 1
 while return_code != 0:
  print polecenie
  return_code = subprocess.call(polecenie, shell=True)

poniedziałek, 21 lutego 2011

"97 things every programmer should know" is going to be translated to Polish

I volunteered to help translating "97 things programmers should know". Here's our progress: You can find it at 97rzeczy.devblogi.pl

niedziela, 19 grudnia 2010

OpenWRT + Lighttpd + Fastcgi + Django

Introduction

Here's something I was fighting with for about 3 days. At first I wanted to use virtualenv too, but it all resulted in a headache. I'm a novice in these matters, so forgive me.

Recently I bought a router TP-Link WR-1043ND. Not the best choice now, I'm aware: TP-Link MR3420 has higher quality to price ratio. Anyway, my router got upgraded to OpenWRT, version Backfire 10.3-RC4, thanks to great work of Cezary Jackiewicz aka Obsy.

Installation

Opkg packages

First packages that we can install with OpenWRT package manager:

  • lighttpd, lighttpd-mod-alias, lighttpd-mod-rewrite, lighttpd-mod-fcgi - packages needed for http server called lighty
  • pysqlite, libsqlite3 - the database
  • python wasn't installed at first, so we'll need this too
opkg install lighttpd lighttpd-mod-alias lighttpd-mod-rewrite lighttpd-mod-fcgi
opkg python libsqlite3 pysqlite

Python packages

These will be installed from Python Package Index (PyPI). We will need:

  • setuptools - Go to pypi.python.org, search for setuptools, follow the instructions.
  • pip - We should be able to call: easy_install pip to obtain it.
  • django - With pip in place getting django is as easy as pip install django. Pip, contrary to easy_install, has a nice uninstall option.
  • flup - For FastCGI related stuff
There we go. It's all installed.

Configuration

3 ways of doing this

First of all, let me briefly explain how FastCGI works. Actually this might not be a good idea to believe me, because, as I said, I'm a layman in this matters. There are 3 entities in this process: the browser, the server (Lighttpd), the FastCGI process (Django).
First step is forwarding the http request for a webpage to FastCGI process:
Browser => Lighttpd => Django FCGI
Request is processed and Django returns html source code of a webpage. Shove it back to your browser and render it.
Browser <= Lighttpd <= Django FCGI
    I came to conclusion that there are 3 ways of setting this up.
  1. By bin-path
  2. By host and port
  3. By socket

My app's directory: /www/apps/homepage. It doesn't have to be anything fancy. It's ok to use brand new project created with django-admin.py startproject homepage.

Bin-path

This method is my favourite so far. We tell lighttpd where to find executable file .fcgi and it starts FCGI processes by itself. All we need to take care of now is that the server is started after reboot. Here are the details:

  • Contents of .fcgi file. It sits at the django project's directory.
    #!/usr/bin/env python
    import sys, os
    
    # Add a custom Python path.
    sys.path.insert(0, "/www/apps")
    
    # Set the DJANGO_SETTINGS_MODULE environment variable.
    os.environ['DJANGO_SETTINGS_MODULE'] = "homepage.settings"
    
    from django.core.servers.fastcgi import runfastcgi
    # maxspare=2 is minimum from what i saw
    runfastcgi(method="threaded", daemonize="false", maxspare=2)
  • Lighttpd config file from /etc/lighttpd/lighttpd.conf. It is edited version of default config file. I just omitted most of commented out lines.
    # lighttpd configuration file
    # 
    ## modules to load
    # all other module should only be loaded if really neccesary
    # - saves some time
    # - saves memory
    server.modules = ( 
     "mod_rewrite", 
    # "mod_redirect", 
     "mod_alias", 
    # "mod_auth", 
    # "mod_status", 
    # "mod_setenv",
     "mod_fastcgi",
    # "mod_proxy",
    # "mod_simple_vhost",
    # "mod_cgi",
    # "mod_ssi",
    # "mod_usertrack",
    # "mod_expire",
    # "mod_webdav"
    )
    
    # force use of the "write" backend (closes: #2401)
    server.network-backend = "write"
    
    ## a static document-root, for virtual-hosting take look at the 
    ## server.virtual-* options
    server.document-root = "/www/public"
    
    ## where to send error-messages to
    server.errorlog = "/var/log/lighttpd/error.log"
    
    ## files to check for if .../ is requested
    index-file.names = ( "index.html", "default.html", "index.htm", "default.htm" )
    
    ## mimetype mapping
    mimetype.assign = (  
     ".pdf"   => "application/pdf",
     ".class" => "application/octet-stream",
     ".pac"   => "application/x-ns-proxy-autoconfig",
     ".swf"   => "application/x-shockwave-flash",
     ".wav"   => "audio/x-wav",
     ".gif"   => "image/gif",
     ".jpg"   => "image/jpeg",
     ".jpeg"  => "image/jpeg",
     ".png"   => "image/png",
     ".css"   => "text/css",
     ".html"  => "text/html",
     ".htm"   => "text/html",
     ".js"    => "text/javascript",
     ".txt"   => "text/plain",
     ".dtd"   => "text/xml",
     ".xml"   => "text/xml"
     )
    
    $HTTP["url"] =~ "\.pdf$" {
     server.range-requests = "disable"
    }
    
    ##
    # which extensions should not be handle via static-file transfer
    #
    # .php, .pl, .fcgi are most often handled by mod_fastcgi or mod_cgi
    static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
    
    ## to help the rc.scripts
    server.pid-file = "/var/run/lighttpd.pid"
    
    #### fastcgi module
    ## read fastcgi.txt for more info
    # this line may help with finding what's wrong, check out errorlog file
    # fastcgi.debug=1
    fastcgi.server = (
     "/homepage.fcgi" => (
      "main" => (
       "host" => "127.0.0.1",
       "port" => 3033,
       "check-local" => "disable",
       "max-procs" => 1,
       "bin-path" => "/www/apps/homepage/homepage.fcgi"
      )
     )
    )
    
    alias.url = (
                    "/media" => "/www/apps/homepage/media",
            )
    
    url.rewrite-once = (
                    "^(/media.*)$" => "$1",
                    "^/favicon\.ico$" => "/media/favicon.ico",
                    "^(/.*)$" => "/homepage.fcgi$1",
            )
    

Host:port

You don't need .fcgi file in this method. Remove line with "bin-path" in fastcgi.server section of lighttpd config. Now you need to make sure that you run this at every boot (maybe you can use start-stop-daemon tool for this):

/www/apps/homepage/manage.py runfcgi method=threaded host="127.0.0.1" port=3033
This will start FastCGI threads. Lighttpd will expect them to be there when he needs a page. Otherwise you'll get 503 or 500. This method is said to be easier because there is no need to set permissions on TCP socket.

Socket

In this method FCGI process and HTTP Server communicate through a socket (aka named pipe). Proper permissions must be in place. FCGI file is also not needed here. You must make sure that you run similar command as in the second method:

/www/apps/homepage/manage.py runfcgi method=threaded socket=/www/apps/homepage/homepage.socket
Lines in fastcgi.server with port and host get removed. Instead add
"socket" => "/www/apps/homepage/homepage.socket"

Conclusion

I use a lot of bad practices here:

  • do everything as root
  • not using virtualenv
  • keeping my app in /www (but I changed my document-root, so maybe I'm ok?)
This is an example of work done by someone who came from "no idea" to "it started working" and all this happens on a router that won't be under a lot of load or work in a cloud of virtual machines. Certainly it's not state of the art, but gets the job done.

There's quite a lot of this kind of tutorials, but to someone without proper knowledge all these terms and blindly followed instructions are no good if something doesn't work as expected. I hope someone will find the wording I used useful. I hope will when I'll need to do everything all over again.

czwartek, 12 sierpnia 2010

Kernel log spammed with rt2500 errors

My wifi card kept spamming kernel log with messages like this: "phy0 -> rt2500pci_set_device_state: Error - Device failed to enter state 1 (-16)". The solution is easily found among launchpad bug reports. Power management needs to be turned off for the card. It's easy to do:

iwconfig wlan0 power off

To make it permanent edit /etc/rc.local adding this line before exit 0. Reboot and you're done! Confirm that with calling iwconfig:

lo        no wireless extensions.

eth0      no wireless extensions.

wlan0     IEEE 802.11bg  ESSID:"TP69"  
          Mode:Managed  Frequency:2.422 GHz  Access Point: 00:23:CD:15:F2:A0   
          Bit Rate=48 Mb/s   Tx-Power=20 dBm   
          Retry  long limit:7   RTS thr:off   Fragment thr:off
          Power Management:off
          Link Quality=60/70  Signal level=-50 dBm  
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:0   Missed beacon:0

wtorek, 1 czerwca 2010

First success with RTAI

Recently I've been trying to grok Real Time Application Interface (more on www.rtai.org). It certainly wasn't a walk in the park, but still far far easier than RTLinux. This time at least I had a clue what was going on. Following this tutorial worked for me: http://qrtailab.sourceforge.net/rtai_installation.html.

Finally when everything is set up you can do: sudo start_rtai on command line. This starts RTAI.

Now for some programming. Here's a snippet:

/* led1.c, an LXRT LED blinker */

#include <rtai_lxrt.h>
#include <pthread.h>
#include <sys/mman.h>
#include <sys/io.h>

// delay in nanoseconds
#define TICKS 500000000 

main()
{
        RT_TASK *task;
        int priority = 0, i;
        int stack_size = 4096;
        int msg_size = 0; // use default

        // get enough privilege to 
        // access the I/O ports.

        iopl(3);

        task = rt_task_init(nam2num("main"), priority, stack_size, msg_size);
        if(task == 0)
        {
         printf("Task could not be initialized. Are you root?");
         exit(1);
        }
        rt_set_oneshot_mode();
        start_rt_timer(0);
        mlockall(MCL_CURRENT|MCL_FUTURE);
        rt_make_hard_real_time();

        for(i = 0; i < 10; i++) {
                outb(0xff, 0x378);
                rt_sleep(nano2count(TICKS));
                outb(0x0, 0x378);
                rt_sleep(nano2count(TICKS));
        }
        // back to non-rt land!
        rt_make_soft_real_time();
        stop_rt_timer();
        rt_task_delete(task);
        return 0;
}
It's a program that comes from this article with added troubleshooting comment.

To compile this program it's necessary to run:

CFLAGS=$(rtai-config --lxrt-cflags
LDFLAGS=$(rtai-config --lxrt-ldflags
gcc $(CFLAGS) $(LDFLAGS) main.c -o blinker
Where main.c is the name of the source file and blinker is the name of output executable.
To run it you should write sudo ./blinker. Sudo is needed because of the direct hardware access. You should see that the leds hooked up to lpt port are blinking.