Free to traverse software for linux

RIR to DNS converter is a tool to convert Regional Internet Registry data to a DNS country lookup zone. You can use it to build your own DNS zone for looking up country codes from IP addresses.

It uses data directly from RIPE, ARIN, APNIC, LACNIC, and AFRINIC. The data can be updated on a schedule of your choosing.

The input data comes from:

ftp://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-latest
ftp://ftp.apnic.net/pub/stats/apnic/delegated-apnic-latest
ftp://ftp.arin.net/pub/stats/arin/delegated-arin-latest
ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-latest
ftp://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-latest

The input data format is described in:

http://www.apnic.net/db/rir-stats-format.html

The output is a BIND 9 zone file that can be used to look up country codes
in a similar fashion to in-addr.arpa. For example, to find out what country
203.30.47.58 is:

host 58.47.30.203.rir.example.com
58.47.30.203.rir.example.com has address 127.0.65.86

where 65 and 85 are ASCII for A and U, which means 203.30.47.58 is
in Australia (AU).

HOW TO USE IT

Just feed it the above delegated- -latest files into stdin and it will
spit out the zone file to stdout. The zone file will only have the IP addresses,
so you could $INCLUDE it into a zone file that contains NS records, SOA, $ORIGIN,
etc.

WHY USE IT

You dont need the resolution of MaxMinds GeoIP database, but you do want
something that is free and you want it kept up to date on a schedule that
you decide.

You could use this to block or tag email based on countries, block or redirect
visitors to your website based on end-user country, and so on. Be very
careful about blocking mail this way, though, as you may block legitimate
email. Instead of blocking outright, use it in a SpamAssassin rule to add
something to the spam level, based on where the email comes from.

HOW IT WORKS

The RIR files contain ranges of IP addresses, and indicate what CC each range is allocated to. At the simplest level, rir2dns just sorts the ranges then iterates
through the IPs in each range and generates a reverse-dns-style A record that
represents the country code.

HOW IT WORKS - IN DETAIL

Rather than iterate through each IP address, the program tries to skip through
entire classes at a time (256 IPs, 65536 IPs, etc). Rather than iterate
through each IP, the loop iterates through classes or IP ranges (whichever are
smaller at the loop control), using control-breaks to accummulate neighbouring
ranges where possible so that entire classes that are in the same country dont
generate huge numbers of records.

Firstly, IPs are considered to be 4-digit numbers, but in base-256. In other
words, each octet is dealt with as if it were a single base-256 digit. This
turns out to be convenient because optimisations of large chunks of IP space can be done by looking for places where least-significant base-256 digits are zero.

Next, IP ranges are broken down into the following sub-ranges:

Optional individual IP addresses (ie: 4 octets)
Optional A-class ranges (ie: 3 octets)
Optional B-class ranges (ie: 2 octets)
Optional C-class ranges (ie: 1 octet)
Optional B-class ranges (ie: 2 octets)
Optional A-class ranges (ie: 3 octets)
Optional individual IP addresses (ie: 4 octets)

Considering that there is a pattern here, Im sure theres an elegant way to
handle breaking this down into two loops (one reducing the octets and one
increasing the octets), but I cant be bothered, so Ill break it down into
seven loops. Kind of hard-coded, but at least its simple.

For ease of processing, the IP addresses are actually converted to 32-bit numbers, then back again. This simplifies mathematics and looping through ranges.

Thats pretty much it, really...

Note that currently there are about 80,000 RIR records between all five
registries. This takes about 35 seconds on a 2.4GHz P4 to process, and
generates a 26MB file with around 3/4 million lines (RRs). This causes BIND
to use about 100MB or so of memory, and on a slow machine will probably cause it to take too long to reply, while it searches the zone. That size zone can
take a minute or two to load, which is quite a while.

Basic algorithm:

Read & process RIR data:

Read RIR ranges
Sort RIR ranges by start IP address
Glue together contiguous ranges of the same country

For each range

Generate the IPs at the start of the range

Generate the A-classes at the start of the range

Generate the B-classes at the start of the range

Generate the C-classes in the middle of the range

Generate the B-classes at the end of the range

Generate the A-classes at the end of the range

Generate the IPs at the end of the range

OO Text To Speech is a text-to speech macro for OpenOffice.org.

Its a syllable analyzer: using a reading motor, it reads a document and translates it into a vocal message.

About OpenOffice

OpenOffice.org is a multiplatform and multilingual office suite and an open-source project. Compatible with all other major office suites, the product is free to download, use, and distribute.

It is an international office suite that will run on all major platforms and provide access to all functionality and data through open-component based APIs and an XML- based file format.

The Wonder Shaper is a very special network shaper script with a lot of features. Works on Linux 2.4 & higher.

Goals

I attempted to create the holy grail:

* Maintain low latency for interfactive traffic at all times.

This means that downloading or uploading files should not disturb SSH or even telnet. These are the most important things, even 200ms latency is sluggish to work over.

* Allow surfing at reasonable speeds while up or downloading

Even though http is bulk traffic, other traffic should not drown it out too much.

* Make sure uploads dont harm downloads, and the other way around

This is a much observed phenomenon where upstream traffic simply destroys download speed. It turns out that all this is possible, at the cost of a tiny bit of bandwidth. The reason that uploads, downloads and ssh hurt eachother is the presence of large queues in many domestic access devices like cable or DSL modems.

Why it doesnt work well by default

ISPs know that they are benchmarked solely on how fast people can download. Besides available bandwidth, download speed is influenced heavily by packet loss, which seriously hampers TCP/IP performance. Large queues can help prevent packetloss, and speed up downloads. So ISPs configure large queues.

These large queues however damage interactivity. A keystroke must first travel the upstream queue, which may be seconds (!) long and go to your remote host. It is then displayed, which leads to a packet coming back, which must then traverse the downstream queue, located at your ISP, before it appears on your screen.

This HOWTO teaches you how to mangle and process the queue in many ways, but sadly, not all queues are accessible to us. The queue over at the ISP is completely off-limits, whereas the upstream queue probably lives inside your cable modem or DSL device. You may or may not be able to configure it. Most probably not.

So, what next? As we cant control either of those queues, they must be eliminated, and moved to your Linux router. Luckily this is possible.

Limit upload speed somewhat

By limiting our upload speed to slightly less than the truly available rate, no queues are built up in our modem. The queue is now moved to Linux.

Limit download speed

This is slightly trickier as we cant really influence how fast the internet ships us data. We can however drop packets that are coming in too fast, which causes TCP/IP to slow down to just the rate we want. Because we dont want to drop traffic unnecessarily, we configure a burst size we allow at higher speed.

Now, once we have done this, we have eliminated the downstream queue totally (except for short bursts), and gain the ability to manage the upstream queue with all the power Linux offers.

Let interactive traffic skip the queue

What remains to be done is to make sure interactive traffic jumps to the front of the upstream queue. To make sure that uploads dont hurt downloads, we also move ACK packets to the front of the queue. This is what normally causes the huge slowdown observed when generating bulk traffic both ways. The ACKnowledgements for downstream traffic must compete with upstream traffic, and get delayed in the process.

We also move other small packets to the front of the queue - this helps operating systems which do not set TOS bits, like everything from Microsoft.

Allow the user to specify low priority traffic (new in 1.1!)

Sometimes you may notice low priority OUTGOING traffic slowing down important traffic. In that case, the following options may help you:

NOPRIOHOSTSRC
Set this to hosts or netmasks in your network that should have low priority

NOPRIOHOSTDST
Set this to hosts or netmasks on the internet that should have low priority

NOPRIOPORTSRC
Set this to source ports that should have low priority. If you have an unimportant webserver on your traffic, set this to 80

NOPRIOPORTDST
Set this to destination ports that should have low priority.

See the start of wshaper and wshaper.htb

Results

If we do all this we get the following measurements using an excellent ADSL connection from xs4all in the Netherlands:

Baseline latency:
round-trip min/avg/max = 14.4/17.1/21.7 ms

Without traffic conditioner, while downloading:
round-trip min/avg/max = 560.9/573.6/586.4 ms

Without traffic conditioner, while uploading:
round-trip min/avg/max = 2041.4/2332.1/2427.6 ms

With conditioner, during 220kbit/s upload:
round-trip min/avg/max = 15.7/51.8/79.9 ms

With conditioner, during 850kbit/s download:
round-trip min/avg/max = 20.4/46.9/74.0 ms

When uploading, downloads proceed at ~80% of the available speed. Uploads at around 90%. Latency then jumps to 850 ms, still figuring out why.

What you can expect from this script depends a lot on your actual uplink speed. When uploading at full speed, there will always be a single packet ahead of your keystroke. That is the lower limit to the latency you can achieve - divide your MTU by your upstream speed to calculate. Typical values will be somewhat higher than that. Lower your MTU for better effects!

A small table:

Uplink speed | Expected latency due to upload
--------------------------------------------------
32 | 234ms
64 | 117ms
128 | 58ms
256 | 29ms

So to calculate your effective latency, take a baseline measurement (ping on an unloaded link), and look up the number in the table, and add it. That is about the best you can expect. This number comes from a calculation that assumes that your upstream keystroke will have at most half a full sized packet ahead of it.

This boils down to:

mtu * 0.5 * 10
-------------- + baseline_latency
kbit

The factor 10 is not quite correct but works well in practice.

Your kernel

If you run a recent distribution, everything should be ok. You need 2.4 with QoS options turned on.

If you compile your own kernel, it must have some options enabled. Most notably, in the Networking Options menu, QoS and/or Fair Queueing, turn at least CBQ, PRIO, SFQ, Ingress, Traffic Policing, QoS support, Rate Estimator, QoS classifier, U32 classifier, fwmark classifier.

In practice, I (and most distributions) just turn on everything.

The scripts

The script comes in two versions, one which works on standard kernels and is implemented using CBQ. The other one uses the excellent HTB qdisc which is not in the default kernel. The CBQ version is more tested than the HTB one!

See wshaper and wshaper.htb.

Tuning

These scripts need to know the real rate of your ISP connection. This is hard to determine upfront as different ISPs use different kinds of bits it appears. People report success using the following technique:

Estimate both your upstream and downstream at half the rate your ISP specifies. Now verify if the script is functioning - check interactivity while uploading and while downloading. This should deliver the latency as calculated above. If not, check if the script executed without errors.

Now slowly increase the upstream & downstream numbers in the script until the latency comes back. This way you can find optimum values for your connection. If you are happy, please report to me so I can make a list of numbers that work well. Please let me know which ISP you use and the name of your subscription, and its reputed specifications, so I can list you here and save others the trouble.

Installation

If you dial in, you can copy the script to /etc/ppp/ip-up.d and it will be run at each connect.

If you want to remove the shaper from an interface, run wshaper stop. To see status information, run wshaper status.

KNOWN PROBLEMS

If you get errors, add an -x to the first line, as follows:

#!/bin/bash -x

And retry. This will show you which line gives an error. Before contacting me, make sure that you are running a recent version of iproute!

Recent versions can be found at your Linux distributor, or if you prefer compiling, here:
ftp://ftp.inr.ac.ru/ip-routing/iproute2-current.tar.gz

MySQL PHP to PostgreSQL is an application that takes a php page that uses mysql calls and changes them into PostgreSQL calls.
This allows any website (if it converts properly) that was written in PHP for MySQL to run as a website written to run on PostgreSQL.
- Step 1: Download and uncompress
To uncompress simply type: tar xvzf mysqlphp2postgres.tar.gz
If youre on a non-GNU system you might have to type: gzip -dc mysqlphp2postgres.tar.gz | tar xvf -
- Step 2: Compile
To compile simply type: make
If youre on a non-GNU system youll have to edit the make file to change the compiler to cc instead of gcc
- Step 3: Install
Type: make install
This will simply copy the binary file to /usr/local/bin/
- Step 4: Convert
Type: mysqlphp2postgres inputfile.php outputfile.php
The input file is the php page that uses mysql. The outfile is the page you want to create. You will probably need to tell mysqlphp2postgres the name of the postgreSQL database since mysql_select_db doesnt translate. You can do this by using a -d switch. So if, for example, the database youre connecting to is called blah, you could type: mysqlphp2postgres -d blah inputfile.php outputfile.php
Problems:
I have had the following problems from using this program:
MySQL has a lot more functions in PHP than PostgreSQL that do more things. I havent had a problem with any unsupported functions in my programs, but someone probably will. If you want to help write code for this to support those functions (if thats even possible) feel free to help.
Enhancements:
- Fixed some stuff.
- Cleaned up some code.
- This program now produces code optomized for PHP 4.1.0 and above.

UTIN Firewall script project is a script for Linux 2.4.x and iptables.

###########
# Configuration options, these will speed you up getting this script to
# work with your own setup.

#
# your LANs IP range and localhost IP. /24 means to only use the first 24
# bits of the 32 bit IP adress. the same as netmask 255.255.255.0
#
# INET_IP is used by me to allow myself to do anything to myself, might
# be a security risc but sometimes I want this. If you dont have a static
# IP, I suggest not using this option at all for now but its still
# enabled per default and will add some really nifty security bugs for all
# those who skips reading the documentation=)

LAN_IP="192.168.0.2"
LAN_BCAST_ADRESS="192.168.0.255"
LAN_IFACE="eth1"

LO_IFACE="lo"
LO_IP="127.0.0.1"

INET_IP="194.236.50.155"
INET_IFACE="eth0"

IPTABLES="/usr/local/sbin/iptables"

#########
# Load all required IPTables modules
#

#
# Needed to initially load modules
#
/sbin/depmod -a

#
# Adds some iptables targets like LOG, REJECT and MASQUARADE.
#
/sbin/modprobe ipt_LOG
#/sbin/modprobe ipt_REJECT
/sbin/modprobe ipt_MASQUERADE

#
# Support for owner matching
#
#/sbin/modprobe ipt_owner

#
# Support for connection tracking of FTP and IRC.
#
#/sbin/modprobe ip_conntrack_ftp
#/sbin/modprobe ip_conntrack_irc

#
# Enable ip_forward, this is critical since it is turned off as defaul in
# Linux.
#

echo "1" > /proc/sys/net/ipv4/ip_forward

#
# Dynamic IP users:
#
#echo "1" > /proc/sys/net/ipv4/ip_dynaddr

#
# Enable simple IP Forwarding and Network Address Translation
#

$IPTABLES -t nat -A POSTROUTING -o $INET_IFACE -j SNAT --to-source $INET_IP

#
# Set default policies for the INPUT, FORWARD and OUTPUT chains
#

$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT DROP
$IPTABLES -P FORWARD DROP

#
# bad_tcp_packets chain
#
# Take care of bad TCP packets that we dont want.
#

$IPTABLES -N bad_tcp_packets
$IPTABLES -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j LOG
--log-prefix "New not syn:"
$IPTABLES -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j DROP

#
# Do some checks for obviously spoofed IPs
#

$IPTABLES -A bad_tcp_packets -i $INET_IFACE -s 192.168.0.0/16 -j DROP
$IPTABLES -A bad_tcp_packets -i $INET_IFACE -s 10.0.0.0/8 -j DROP
$IPTABLES -A bad_tcp_packets -i $INET_IFACE -s 172.16.0.0/12 -j DROP
$IPTABLES -A bad_tcp_packets -i $LAN_IFACE ! -s 192.168.0.0/16 -j DROP

#
# Bad TCP packets we dont want
#

$IPTABLES -A FORWARD -p tcp -j bad_tcp_packets

#
# Accept the packets we actually want to forward between interfaces.
#

$IPTABLES -A FORWARD -p tcp --dport 21 -i $LAN_IFACE -j ACCEPT
$IPTABLES -A FORWARD -p tcp --dport 80 -i $LAN_IFACE -j ACCEPT
$IPTABLES -A FORWARD -p tcp --dport 110 -i $LAN_IFACE -j ACCEPT

$IPTABLES -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -m limit --limit 3/minute --limit-burst 3 -j LOG
--log-level DEBUG --log-prefix "IPT FORWARD packet died: "


#
# Create separate chains for ICMP, TCP and UDP to traverse
#

$IPTABLES -N icmp_packets
$IPTABLES -N tcp_packets
$IPTABLES -N udpincoming_packets

#
# The allowed chain for TCP connections
#

$IPTABLES -N allowed
$IPTABLES -A allowed -p TCP --syn -j ACCEPT
$IPTABLES -A allowed -p TCP -m state --state ESTABLISHED,RELATED
-j ACCEPT
$IPTABLES -A allowed -p TCP -j DROP

#
# ICMP rules
#

# Changed rules totally
$IPTABLES -A icmp_packets -p ICMP -s 0/0 --icmp-type 8 -j ACCEPT
$IPTABLES -A icmp_packets -p ICMP -s 0/0 --icmp-type 11 -j ACCEPT

#
# TCP rules
#

$IPTABLES -A tcp_packets -p TCP -s 0/0 --dport 21 -j allowed
$IPTABLES -A tcp_packets -p TCP -s 0/0 --dport 22 -j allowed
$IPTABLES -A tcp_packets -p TCP -s 0/0 --dport 80 -j allowed
$IPTABLES -A tcp_packets -p TCP -s 0/0 --dport 113 -j allowed

#
# UDP ports
#

$IPTABLES -A udpincoming_packets -p UDP -s 0/0 --source-port 53 -j ACCEPT
$IPTABLES -A udpincoming_packets -p UDP -s 0/0 --source-port 123 -j ACCEPT
$IPTABLES -A udpincoming_packets -p UDP -s 0/0 --source-port 2074 -j ACCEPT
$IPTABLES -A udpincoming_packets -p UDP -s 0/0 --source-port 4000 -j ACCEPT

#
# INPUT chain
#
# Bad TCP packets we dont want
#

$IPTABLES -A INPUT -p tcp -j bad_tcp_packets

#
# Rules for incoming packets from anywhere
#

$IPTABLES -A INPUT -p ICMP -j icmp_packets
$IPTABLES -A INPUT -p TCP -j tcp_packets
$IPTABLES -A INPUT -p UDP -j udpincoming_packets

#
# Rules for special networks not part of the Internet
#

$IPTABLES -A INPUT -p ALL -i $LO_IFACE -s $LO_IP -j ACCEPT
$IPTABLES -A INPUT -p ALL -i $LO_IFACE -s $LAN_IP -j ACCEPT
$IPTABLES -A INPUT -p ALL -i $LO_IFACE -s $INET_IP -j ACCEPT
$IPTABLES -A INPUT -p ALL -d $INET_IP -m state --state ESTABLISHED,RELATED
-j ACCEPT
$IPTABLES -A INPUT -m limit --limit 3/minute --limit-burst 3
-j LOG --log-level DEBUG --log-prefix "IPT INPUT packet died: "

#
# OUTPUT chain
#
#
# Bad TCP packets we dont want
#

$IPTABLES -A OUTPUT -p tcp -j bad_tcp_packets


$IPTABLES -A OUTPUT -p ALL -s $LO_IP -j ACCEPT
$IPTABLES -A OUTPUT -p ALL -s $LAN_IP -j ACCEPT
$IPTABLES -A OUTPUT -p ALL -s $INET_IP -j ACCEPT
$IPTABLES -A OUTPUT -m limit --limit 3/minute --limit-burst 3
-j LOG --log-level DEBUG --log-prefix "IPT OUTPUT packet died: "

The Introspector enables the programming tools that deal with source code such as the compiler to communicate in a standard and neutral manner reducing the accidental cost of programming.

Like a telephone switchboard connects many parties who might wish to talk to each other, the Introspector allows multiple consumers and producers of data about software to quickly and painlessly transfer information to each other.

This data about software, or meta-data is read and written in and out of existing software tools via a standardized plug-in interface. Each producer has its own flavour of data and format of data that it stores about your software, and the introspector allows for each software to speak its native language.

The Introspector plug ins act analogous to network cards in an ethernet allowing for broadcasting of the meta-data to the network of consumers.

The usage of RDF or Resources Description Format as the underlying representation is of essential importance. RDF is the foundation of the next generation of the internet, the Semantic Web where hypertext links is replaced by meaningful references to resources of type and quality.

The introspector uses the excellent Redland RDF Application Framework for parsing/serializing, storing/retrieving and querying/traversing the RDF data. The Berkley DB storage mechanism provides an efficent indexing system, and the SWIG Simplified Wrapper Interface Generator provides native langauge interfaces for all major programming systems.

The GCC interface uses Redland to create repositories of data that can be processed by any tool needed. Experiments have been made with compiling this graph data into arrays, so called "ICE Cubes" that can be traversed even quicker than the redland database.

This new technology when available will be able to be used from the same introspector API that gives access to Redland.

Java Properties provides an efficient way to access bean-like properties of Java objects.

In a nutshell, provides an efficient way to access bean-like properties of java objects. Unlike java bean properties, a chain of properties can be specified efficiently, allowing convenient access to properties of nested domain objects.

Runtime class-generation and caching can be used very easily to obviate the need for reflection, without losing the flexibility it provides.

Class Properties It also provides classes which manipulate objects at runtime by using strings to address conceptual variables (think bean properties) in a similar fashion to the java reflection mechanisms.

Class Property example Say we have the following classes:

public class Order {
...
public Customer getCustomer() { ... }
...
}
public class Customer {
...
public Address getAddress() { ... }
...
}
public class Address {
...
public String getLine1() { ... }
public String getLine2() { ... }
...
}

Now, we could do this to access line 1 of the address:

result = order.getCustomer().getAddress().getLine1();

But what if there are nulls? What if we want to compare the first line of the address from two different orders? Whith properties this is really simple:

ClassProperty p = PropertyManager.getProperty(Order.class,"Customer:Address:Line1",true);
result = p.getValue(order);

The idea is simple enough - you use the string "Customer:Address:Line1" to specify a series of getXXX calls. Importantly, it the library handles nulls for you, returning null if any of the objects in the chain are null.

Note, this is not simply a wrapper around java reflection, since it includes non-reflective optimizations by making use of the cojen library to generate class files at runtime to acces these properties.

The real benefit lies in being able to specify which properties you wish to access at runtime. You can even pass this ability on to the users of your library or application. Unlike reflection, class files are generated for the properties, and these files are cached - which means once you have specified a property once, further uses of the same property dont use reflection and are very fast.

Property API vs Java Bean Properties. The property API models accessor/mutator method pairs as conceptual variables called properties. The approach used is similar to, but more flexible than java bean properties, and more performant than reflection if runtime class generation is used. Unlike java bean properties, a chain or path of properties can be specified (and more importantly, turned into bytecode using runtime generation) to traverse a complex tree of objects.
It is simple to create applications portable between different security environments - using runtime generation where allowed, and falling back to reflection where security is tighter.

Dynamic Class Management. The properties package also provides classes to handle the loading (and unloading/reloading) of classes at runtime. It provides a framework useful for dynamically loading runtime-generated classes, for example.
While the property API can happily ignore the dynamic loading framework, it can also make use of it to enable runtime class generation.