Archive | VPS RSS for this section

Download torrent files automatically with flexget

Since I started renting a VPS, my goal has been to automate as much tasks as possible. I watch TV shows, and it had become a pain to manually download torrent files every week. In this post, I will explain how those torrent files are automatically downloaded on my VPS. Note that I’m only talking about the torrent files, not the actual files.

First off, you are going to need a RSS feed to download the torrent files from. If you don’t already know one, just head on to the EZTV RSS feed and search for your favorite TV show. You’ll be able to get the RSS feed from the “Search-based RSS Feed” link. For example, for Breaking Bad, the link look like this: http://www.ezrss.it/search/index.php?simple&show_name=Breaking+Bad&mode=rss. Don’t concern yourself with the ‘Date’ and ‘Quality’ fields, we’ll deal with that later.

Seconly, let’s install flexget, which will fetch the torrent files for us. Flexget is an automation tool, that you can install easily. (If you know what pip is, just run sudo pip install flexget and you’re done)

Now, we need to hook these two parts, by telling flexget what to do. In your home directory, create a .flexget directory and create an empty file in it called config.yml. This file is the configuration file flexget will read everytime it’s run. Here is an example config file:

tasks:
  GameOfThrones-torrents:
    rss: http://www.ezrss.it/search/index.php?show_name=Game+of+Thrones&date=&quality=&release_group=&mode=rss
    series:
      - Game of Thrones:
          begin: S04E01
          qualities: [hdtv+, 720p]
    download: /home/downloads/Watch/

  MadMen-torrents:
    rss: http://www.ezrss.it/search/index.php?show_name=Mad+Men&date=&quality=&release_group=&mode=rss
    series:
      - Mad Men:
          begin: S07E01
          quality: "<720p"
    download: /home/downloads/Watch/

When called, flexget runs every ‘task’. Here we have two tasks, fetching torrent files for Game of Thrones and Mad Men. In each task, put a rss field with the link you acquired in the first step. Then add a series field, put the name of the show (‘Game of Thrones’ for example). The begin field tells flexget to only download torrent starting from the specified episode (the specified episode is included), the quality or qualities fields are self explanatory. Lastly, the download field tells flexget where to put the downloaded file.
To test your configuration file, run flexget --test. If your config file is not properly formatted, you’ll instantly know. You can also check which torrent files are retrieved. Don’t hesitate to remove the begin field to confirm that flexget downloads files.

Once you made sure that flexget does what you expect him to, let’s automate the process by adding a cron. Run crontab -e and add the following line to your crontab:

0,30 * * * * /usr/local/bin/flexget --cron

This will create a cron entry that will run flexget every 30 minutes. You can choose another timeframe, don’t decrease it too much though, as some feeds won’t like you pinging them every five seconds.

Let’s recap: we installed flexget, told him to fetch torrent files from a RSS feed and made sure he does it regularly. All done!

Extra credit: Mail and customization.
You can further customize flexget, the flexget Wiki is really understandable and full of examples, I invite you to look into it. For example, using presets could make your config file more DRY.
As a final note, I customized flexget so that it sends me an email when a torrent file is fetched. If you want something like this, just add the following to your config file:

email:
  from: mybox@gmail.com
  to: pmourlanne@gmail.com
  smtp_host: smtp.gmail.com
  smtp_port: 587
  smtp_username: mybox
  smtp_password: qwerty
  smtp_tls: yes
  template: accepted

Edit: Scale better
If you put too much shows in your flexget config, you might start to have problems, such as torrents not downloading. I had this problem, it was because every time flexget was run, it would poll the rss feed for each show. Meaning that if I was waiting on 10 shows, I would make 10 requests to EZRSS every half-hour.
A better way to poll the website is to ask once for the rss feed and then find the shows in it. I did this and the torrents downloaded way better. Here is an excerpt of my config.yml:

tasks:
  Shows-torrents:
    rss: https://ezrss.it/feed/
    download: /home/downloads/Watch/
    series:
      - Game of Thrones:
          begin: S04E01
          qualities: [hdtv+, 720p]

      - Mad Men:
          begin: S07E01
          quality: "<720p"

This way I retrieve the “Latest torrent releases” feed once and try to find each of my shows in it.

Advertisements

Bash custom functions and autocompletion

To easily upload torrents files to my VPS I wrote a custom function I put in my .bashrc :

upload_torrents () {
    for file; do
        filename=$(basename "$file")
        extension="${filename##*.}"
        if [ $extension = 'torrent' ]
            then
                rsync -a --partial --progress "$file" mybox:/home/downloads/torrents/watch/
        else
            echo "$file"" is not a torrent"
        fi
    done
}

This function is pretty straightforward, it uploads all the torrent files passed in arguments. Using it I realized it was lacking something: I wanted the bash autocompletion to only propose the torrent files instead of all the files in the directory. I added the following snippet to my .bashrc and it did the trick (found here):

complete -f -X '!*.@(torrent)' upload_torrents

Now that we’ve uploaded the torrents, we need to download the files from the remote server to our local machine. Here is another custom function I wrote:

download_from_mybox () {
    for file; do
        rsync -a --partial --progress --bwlimit=500 mybox:/home/downloads/torrents/"$file" .
    done
}

Even simpler than the first one. How about adding autocompletion to this command? Obviously we need to complete using the list of files located on the remote server in /home/downloads/torrents/. I had quite a lot of difficulties trying to figure this one out and I finally asked the fine people of unix.stackexchange. Here is how I finally did it:

In /etc/bash_completion.d/, I created a download_from_my_box file containing the following:

#!/bin/bash

_get_rsync_file_list()
{
    ssh mybox "cd /home/downloads/torrents/pierre && find *"
}

_GetOptSSH()
{
    local cur

    COMPREPLY=()
    cur=${COMP_WORDS[COMP_CWORD]}

    case "$cur" in
    -*)
        COMPREPLY=( $( compgen -W "-h -l --help --list --" -- "$cur" ) );;
    *)
        local -a filelist=$(_get_rsync_file_list)
        COMPREPLY=( $( compgen -W "${filelist[@]}" -- "$cur" ) );;
    esac

    return 0
}

complete -F _GetOptSSH download_from_mybox

The case on $cur isn’t strictly necessary since I didn’t implement any option in my command but it’s good to know how to do it.

So that’s how you do custom remote autocompletion in bash! Pretty rad I know.

If you want to learn more on the matter, here is a very good article in two parts hosted on http://www.debian-administration.org

Quick edit before I post an article with the complete solution: this doesn’t work if you have spaces in your filenames/directories. I asked for help on the UnixStackExchange and got an awesome answer. I’m trying to find the time to implement a fully working solution and post it here.

My VPS@Hetzner

For the last 3 years or so I’ve been using a VPN service.

It started out when I entered my engineering school and lived on campus. The Internet connection the students shared was really poor. To top it off, peer-to-peer was deactivated. So I had to use a VPN to be able to do P2P and I subscribed to VPNFacile. It was perfect for a first timer, everything was well explained and the offer was only 5€ per month. The only complain I would have is that the people behind the website cannot write French without making a mistake every other sentence (and I’m being kind).
The next year, fed up with living on campus, a couple of friends and I rented an apartment in the city and we suscribed to a French ISP, Free. I stopped using my VPN until I received an email from HADOPI, telling me that I was downloading illegal material (a ludicrous accusation, needless to say).
This year I started using a VPN because I get a piss poor connection on Youtube and any other semi bandwith heavy american websites (twitch.tv, blip.tv etc.). This is because of my ISP, still Free, who refuses to get a better infrastructure, arguing that Google should participate in this project (for more details, see here). The French government is starting to question this action, Net neutrality and all that. In the meantime, consumers suffer but who cares really ?

A couple of months ago, I decided to rent a remote server. I wanted to try setting things up from scratch: configuring the ssh server, an OpenVPN server, rtorrent and so on.
I looked into hosting services and went with Hetzner. I subscribed to their 20€/month VPS offer. Their offer is pretty good, the only downside is that their FAQ in English is almost impossible to navigate. They offer a simple recovery system which allows you to mount your disks on a rescue computer, if you lock yourself out of your VPS. This kind of situation will probably happen when you play with the ssh configuration (or the firewall, or both…).

I’m renting this server with a friend, we already set up a couple of services, including OpenVPN which I’m using by default on my machine. I discovered that having my exit point in Germany was a poor decision: a lot of Youtube videos are banned in Germany because of copyright issues and Grooveshark is flat-out banned. I also cannot post on 4chan because my IP is flagged. Furthermore, I doubt they are very keen on people downloading from peer-to-peer networks. That’s why I might leave Hetzner to go somewhere else.

In the meantime we are setting up services on the VPS and figuring stuff as we go, it’s very instructive. I’ll try to share interesting gists for others to learn from.

rtorrent: Receiving an email when a download is complete

Since I had to get the information from different websites, I thought I would make a post compiling all the info necessary.
I’m using rtorrent on a remote server (running Ubuntu) and I wanted to receive a mail when rtorrent is done downloading a file, so that I wouldn’t have to check it by hand. I still have to download the files manually but that will be my next assignment!

First things first, you need a valid email address from which you will send these notifications. I created a gmail email address because it was the most convenient solution at the time. Let’s call it mybox@gmail.com and let’s say I wasn’t really concerned with security and used ‘qwerty’ as a password.

Next, you’ll need to install and configure a package to send email from the terminal. I’ve adapted this guide, which helped me greatly, to Debian based distros.

Install ssmtp with something like: sudo apt-get install ssmtp
Configure it by editing /etc/ssmtp/ssmtp.conf and change it to:

root=mybox@gmail.com
mailhub=smtp.gmail.com:587
rewriteDomain=
hostname=smtp.gmail.com:587
UseSTARTTLS=YES
UseTLS=YES
AuthUser=mybox
AuthPass=qwerty
AuthMethod=LOGIN

And in /etc/ssmtp/revaliases:

root:mybox@gmail.com:smtp.gmail.com:587
mainuser:mybox@gmail.com:smtp.gmail.com:587

Obviously, change the fields to match your own login credentials.

To test your configuration try this: echo test | mail -s "Hey this actually works!" your_personal@email.com
You should receive a mail from mybox@gmail.com. For more details on the mail command, see the man page.

Let’s now write a script that sends email to your personal address. Later on, we’ll tell rtorrent to execute this script when a download is completed.
Create a rtorrent_mail.sh file wherever you want and edit it like so:

#!/bin/sh
echo "$(date) : $1 - Download completed." | mail -s "[rtorrent] - Download completed : $1" your_personal@email.com

Don’t forget to make it executable (chmod +x rtorrent_mail.sh) and to give it the right permissions (the user launching rtorrent needs to be able to execute it).

Finally, we need to hook this script to rtorrent. In your .rtorrent.rc, add the following line:

system.method.set_key = event.download.finished,notify_me,"execute=/path/to/script/rtorrent_mail.sh,$d.get_name="

If I’m not mistaken, this hooks the notify_me receiver to the event.download.finished signal (or event). The last part between quotes is the content of the notify_me method.
Once again, put the path corresponding to where you put your script. You might need to restart rtorrent for these settings to be active.

Here you go! rtorrent will automatically send you emails when a download is completed.
This post doesn’t cover everything I did with rtorrent. I’m actually sharing my VPS with a very good friend of mine and we both get notified when a download is completed. This is great for tv shows we both watch. What if you want rtorrent to send an email to different users depending on the files it downloaded? (practical example: you don’t want your dear friend to know that you are very much into videos displaying women and horses in a lewd manner)
Well, that will be the object of a future post !