07 Jul 2015

Movistar 300 Mbps

This was my connection speed yesterday:

Movistar 30 Mbps

And this is my connection speed today:

Movistar 300 Mbps

I still remember my exciment the first time I got my US Robtics 56K connected. At that time, it really felt fast.

US Robotics 56K

27 Mar 2015

Links for the weekend 13 - 2015

Interesting links I’ve bookmarked this week:

23 Mar 2015

Links for the weekend 12 - 2015

Slow week due local holidays.

16 Mar 2015

How to keep your web app logs clean using nginx

I’m sure that if you take a look at your internet exposed web app production logs, you’ll find lots of entries like this one:

I, [2015-03-02T15:43:38.103037 #13610]  INFO -- : Started GET
"/pma/scripts/setup.php" for at 2015-03-02 15:43:38 -0500
F, [2015-03-02T15:43:38.108401 #13610] FATAL -- : ActionController::RoutingError (No route matches [GET] "/pma/scripts/setup.php"):
    actionpack (4.2.0) lib/action_dispatch/middleware/debug_exceptions.rb:21:in `call'
    actionpack (4.2.0) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
    railties (4.2.0) lib/rails/rack/logger.rb:38:in `call_app'
    railties (4.2.0) lib/rails/rack/logger.rb:20:in `block in call'
    activesupport (4.2.0) lib/active_support/tagged_logging.rb:68:in `block in tagged'
    activesupport (4.2.0) lib/active_support/tagged_logging.rb:26:in `tagged'
    activesupport (4.2.0) lib/active_support/tagged_logging.rb:68:in `tagged'
    railties (4.2.0) lib/rails/rack/logger.rb:20:in `call'
    request_store (1.1.0) lib/request_store/middleware.rb:8:in `call'
    actionpack (4.2.0)
    lib/action_dispatch/middleware/request_id.rb:21:in `call'
    rack (1.6.0) lib/rack/methodoverride.rb:22:in `call'
    rack (1.6.0) lib/rack/runtime.rb:18:in `call'
    activesupport (4.2.0)
    lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
    rack (1.6.0) lib/rack/sendfile.rb:113:in `call'
    railties (4.2.0) lib/rails/engine.rb:518:in `call'
    railties (4.2.0)
    lib/rails/application.rb:164:in `call'
    /usr/local/rvm/gems/ruby-2.1.2/gems/passenger-4.0.56/lib/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
    /usr/local/rvm/gems/ruby-2.1.2/gems/passenger-4.0.56/lib/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
    /usr/local/rvm/gems/ruby-2.1.2/gems/passenger-4.0.56/lib/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
    /usr/local/rvm/gems/ruby-2.1.2/gems/passenger-4.0.56/lib/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'

This is from a Rails app. This is a bot trying to find vulnerabilities in webpages. There are hundreds of attempts like this daily. Usually they don’t cause any harm but they are annoying because they pollute your logs.

Try to run this against your log:

<pre><code>cat production.log | grep php|cgi | wc -l

It will show you how many requests contains php or cgi (assuming you are not using those technologies). Also, for each one of those requests you get the full call stack.

One way to partially mitigate this entries, is to use nginx location directive.

    location ~ php|cgi { return 444; log_not_found off; }

Put this line on your server section and it will block all requests which contains the strings php or cgi. Returning 444 instead of 404, nginx won’t even respond, it will just drop the connection.

You can reload nginx without stopping the service with:

sudo service nginx reload

You might want to customize the regex to match your scenario or add multiple filters depending on your scenario. I’ve found the following samples on forums to protect a Rails log:

location ~ ^/cgi-bin { return 444; log_not_found off; }
location ~ \.(?:php|aspx|asp)$ { return 444; log_not_found off; }
location ~ php\.cgi$ { return 444; log_not_found off; }
location ~ myadmin { return 444; log_not_found off; }

13 Mar 2015

Links for the weekend 11 - 2015

Interesting links I’ve bookmarked this week:

06 Mar 2015

Links for the weekend 10 - 2015

Just one link this time. Busy week I guess…

04 Mar 2015

PostgreSQL, playing with triggers, functions and sequences

I have a very special use case where I want Postgres to automatically generate an identifier for one column when a new row is inserted. This identifier has the following properties:

  • It must be unique.
  • It must have two parts.
    • The first one is the value of another field in the sampe table.
    • The second one must be sequence number padded with zeros.

Basically I want the identifier to have this format:


These are the tables I’ll be using:

<pre><code>CREATE TABLE warehouses (
    id      serial PRIMARY KEY,
    name    varchar(255)

CREATE TABLE incomings (
    id              serial PRIMARY KEY,
    warehouse_id    integer,
    identifier      varchar(50),
    data            text

Each time a new warehouse is created I want PostgreSQL to create a customized SEQUENCE to keep track of the next identifier for that warehouse.

<pre><code>CREATE OR REPLACE FUNCTION make_seq_for_warehouse() RETURNS TRIGGER AS $$
    sql varchar := 'CREATE SEQUENCE seq_for_warehouse_' || NEW.id;
    EXECUTE sql;
    return NEW;
$$ LANGUAGE plpgsql;

And here is the trigger for the warehouses table that will call the previous function:

<pre><code>CREATE TRIGGER generate_next_identifier_function
ON warehouses
EXECUTE PROCEDURE make_seq_for_warehouse();

If we now insert a new warehouse, it should create a new sequence for us:

test=# INSERT INTO warehouses (name) VALUES ('Warehouse1');
test=# \d
List of relations
Schema |        Name          |   Type   | Owner
public | incomings            | table    | jose
public | incomings_id_seq     | sequence | jose
public | seq_for_warehouse_1  | sequence | jose
public | warehouses           | table    | jose
public | warehouses_id_seq    | sequence | jose

Now I want the column identifier to be filled in every time a new incoming is created. First we create the function to do the work:

<pre><code>CREATE OR REPLACE FUNCTION update_incoming_identifier()
    NEW.identifier := NEW.warehouse_id || '-' || lpad(nextval('seq_for_warehouse_' || NEW.warehouse_id)::varchar, 8, '0');
$$ LANGUAGE plpgsql;

Finally we link this function with a trigger on the incomings table:

<pre><code>CREATE TRIGGER generate_next_identifier
ON incomings
EXECUTE PROCEDURE update_incoming_identifier();

Note that now we use BEFORE TRIGGER so we get the new identifier before the row is saved.

Now we can create a new warehouse, so we have two and create some incomings:

<pre><code>test=# INSERT INTO warehouses (name) VALUES ('Warehouse 2');
test=# INSERT INTO incomings  (warehouse_id, data) VALUES (1, 'data on warehouse 1');
test=# INSERT INTO incomings  (warehouse_id, data) VALUES (1, 'more data on warehouse 1');
test=# INSERT INTO incomings  (warehouse_id, data) VALUES (2, 'data on warehouse 2');
test=# INSERT INTO incomings  (warehouse_id, data) VALUES (2, 'more data on warehouse 2');
test=# INSERT INTO incomings  (warehouse_id, data) VALUES (2, 'even more data on warehouse 2');

And this is the result:

test=# SELECT * FROM incomings;
id | warehouse_id | identifier  |              more_data
1  |            1 | 1-00000001  | data on warehouse 1
2  |            1 | 1-00000002  | more data on warehouse 1
3  |            2 | 2-00000001  | data on warehouse 2
4  |            2 | 2-00000002  | more data on warehouse 2
5  |            2 | 2-00000003  | even more data on warehouse 2

Another way to achive the same result would have been to use a SELECT MAX() + 1 query on the trigger and use an additional column on the incomings table.

DISCLAIMER: Bear in mind that we are creating a new sequence for each warehouse. This is not a problem in my scenario because I expect to have between 2 and 8 warehouses and most of them are created when I setup the application. Do not use this as a pattern. This is just me playing with PostgresSQL.

27 Feb 2015

Links for the weekend 09 - 2015

Interesting links I’ve bookmarked this week:

21 Feb 2015

Linux, display local IP before login prompt

I might need to ship one of my applications to a customer as a VM. This application is a web app, so in order to use the application they’ll need to run the VM and just point their browser to the VM IP address.

I’d like to show the local IP address without login in the VM first.

It turns out that the file I’m looking for is /etc/issue (there’s another one for remote logins called /etc/issue.net, but I don’t think I need this).

Whatever you put in this file, it will be displayed before the login prompt.

Now we just need to update this file every time the system starts. For that I added this code to /etc/rc.local:

<pre><code>#!/bin/sh -e

IP=$(hostname -I)
echo "Hey, navigate to http://$IP to use my awesome app!" > /etc/issue

20 Feb 2015

Links for the weekend 08 - 2015

Interesting links I’ve bookmarked this week:

Older posts...