Free upstream software for linux

Upstream is a collection of tools that allow users to send in system specific log and troubleshooting data to support personnel. The project can be easily modified to output data to any pastebin or support tracker.

Upstream aims to simplify the support cycle and make it more efficient. One of the main problems that support staff face when helping people via IRC or mailing lists is getting user log files across.

Often this process is time consuming, and many new users may even be unable to find / copy these logs. Upstream does this for them by finding and transmitting the log files relevant to a specific issue.

Usage:

We hope to turn this,

< newuser >hi, something is wrong with my resolution, how can i fix it?
< support_person >newuser, what is your video card, and which driver is xorg using?
< newuser >err... how can i find out?
< support_person > cat /var/log/Xorg.0.log and pastebin it
< newuser >pastebin?
< support_person >http://pastebin.ca
< newuser >how do i copy? it wont copy!

In to this,

< newuser >hi, something is wrong with my resolution, how can i fix it?
< support_person >newuser, please run Canoe (System -> Help -> Canoe) and choose Video support. Give me the url you get once its done.

MP3 Streaming DownSampler for PHP is written with cable modem/DSL users with low upstream caps in mind. Using LAME, it dynamically downsamples a requested MP3 to a bitrate which can be streamed.

It doesnt use temporary files, so you dont have to take the time to do any batching, and you just give it a directory with symlinks to your music in a place the Web server can reach. The interface is simplistic and fast, meant for immediate plug and play access.

dnshistory project provide a means for storing a history of DNS and Name changes for the IP Addresses extracted from web log files.
The major target being that multiple analyses of older log files do not require re-lookups of IP Address to FQDNs, and additionally maintain the accuracy of the lookup as it was then and not as it is now.
Main features:
- Do Lookups. The default mode. Given a web log file, dnshistory will perform DNS reverse lookups on each unique IP Address and store the results in a history database.
- Do Translations. Given a raw web log file, dnshistory will make use of a previously created history database and send to STDOUT the same web log but with addresses replaced by the Fully Qualified Domain Name as previously looked up.
- Do Recombining. Given two web log files, one raw and one previously translated (eg. by using dnstran): Create a history database from the values in these separate log files.
- Do Dump. Dump a given history database to STDOUT.
- Show History. Given one or more IP Addresses on the command line, display their history from the database.
Its quite possible that most users would only ever use the first two modes.
The lookups make use of threads for near maximum speed, and use the standard resolution libraries on a system. Thus hosts files, NIS, LDAP and other name resolution methods should work transparently. Unfortunately most other tools ignore local name resolution methods in favour of DNS lookups only.
It is strongly recommended that for massive raw lookups a DNS server is "nearby". Preferably not a forwarding server, or your upstream provider will not like you.
dnshistory can read .gz files. Any input sent via STDIN is currently assumed to not be gz encoded.
dnshistory assumes that the logs being sent are already sorted into oldest --> most_recent date/time order.
A Berkeley Database is used to store the history; as well as possibly reducing the memory footprint within a run.
dnshistory is released under the General Public License.
Enhancements:
- This is practically identical to 1.3-beta1.

Darwin is the core of Mac OS X. The Darwin kernel is based on FreeBSD and Mach 3.0 technologies and provides protected memory and pre-emptive multitasking. Darwin runs on PowerPC-based Macintosh computers and a version is also available for x86-compatible computers.
Apples open source projects allow developers to customize and enhance key Apple software. Through the open source model, Apple engineers and the open source community collaborate to create better, faster and more reliable products for our users.
Beneath the appealing, easy-to-use interface of Mac OS X is a rock-solid foundation that is engineered for stability, reliability, and performance. This foundation is a core operating system commonly known as Darwin.
Darwin integrates a number of technologies, most importantly Mach 3.0, operating-system services based on 4.4BSD (Berkeley Software Distribution), high-performance networking facilities, and support for multiple integrated file systems.
Since it was first released in March 1999, Darwin has been the open-source OS technology underlying Apples Mac OS X operating system, with all development being managed and hosted by Apple at http://developer.apple.com/darwin/.
Since Apples Mac OS X releases are based directly on the live Darwin CVS repository, it has been necessary to have a fairly comprehensive procedural framework in place for registering and managing Darwin developers to ensure a good level of quality control. While this system has served its intended purpose quite well, it is desirable to further increase the collaboration between Apple and the open source community beyond the current model.
OpenDarwin.org, jointly founded in April 2002 by Internet Systems Consortium, Inc. (ISC) and Apple, is an attempt to take cooperative Darwin development to the next level. Membership in the OpenDarwin project and access to its works are open to everyone.
The project is also fully independent, with control over its own web site, project news, bug tracking information and CVS repository, as well as any other services that the community owners may wish to provide. Neither Apple nor ISC take any responsibility for, or exercise any editorial control over, the OpenDarwin project.
Many OpenDarwin members are either Apple employees or Darwin Committers, who have an active interest in merging technologies from OpenDarwin.org into Darwin and Mac OS X releases. With OpenDarwin, project members have greater latitude in producing incremental updates or interim releases of Darwin.
The mission of the OpenDarwin project is to innovate and explore new technologies while still remaining relevant, through its informal connection to www.opensource.apple.com, to the mainstream computing environments that Apple provides. It complements Apples infrastructure by allowing increased participation by the community.
Enhancements:
- Install script has been enhanced to allow user creating, setting root password
- AuthenticAMD cpus are now properly detected
- cvs from upstream feature releases
- locate from NetBSD
- at/atrun synced with FreeBSD-5
- various unix software has been updated.
- ODTulipDriver, a driver for tulip-based ethernet controllers, has been added.
- ncutil, a network configuration tool for Darwin, has been updated to version 2.1

eTktab is an ASCII tablature editor for 4/5/6 stringed instruments. he code is based on TkTab by Giovanni Chierico. Many of the ideas for the user interface came from tablature mode written for the Unix editor emacs. Mac and Windows binaries were created with freely available tk library.
The program makes 30 fretboard positions available for one-keypress entry (5 on each string.) The following is an example keyboard layout for United States keyboard (first row is numbers, second row QWERTY, etc.) "Base" refers to the movable position of the players hand.
Enhancements:
- Lyrics/tab mode now switches automatically, in response to mouse clicks in tab scores or textboxes
- Can now click on end of textbox symbol to go into that textbox
- In lyrics mode, "current position" color now only added to the textbox that contains the cursor
- Focus now follows cursor for left/right cursor key
- Menubuttons now keep consistent size, where possible
- Fixed bug in internationalization code, regarding Save in File menu
- Return key uses new feature, listed below to insert whitespace, rather than blank tab positions to end of line
- Macintosh Classic and OS X now have File, Edit, and Help menus exclusively in the Mac Menubar (rather than the window statusbar)
- Many OS X bugfixes, due to upstream cleanup in tcl/tk 8.4.5
- Blank spaces may now be added to tablature (using whitespace in tab makes files unreadable by versions older than 3.2)

bzr-hudson 1.0 Beta 1 offers you a perfect and helpful plugin which can allow the use of Bazaar as a build SCM in Hudson. With this plugin, you can designate a Bazaar branch as the "upstream" repository. Every build will update the local branch. Polling is also supported, it'll check if the upstream branch have any new revision and use this as the build triggering condition.

Requirements:

• Bazaar Revision Control System: bzr >= 1.9
• xmloutput >= 0.8.2

Dnsmasq is lightweight, easy to configure DNS forwarder and DHCP server. Dnsmasq is designed to provide DNS and, optionally, DHCP, to a small network. It can serve the names of local machines which are not in the global DNS.
The DHCP server integrates with the DNS server and allows machines with DHCP-allocated addresses to appear in the DNS with names configured either in each host or in a central configuration file. Dnsmasq supports static and dynamic DHCP leases and BOOTP for network booting of diskless machines.
Dnsmasq is targeted at home networks using NAT and connected to the internet via a modem, cable-modem or ADSL connection but would be a good choice for any small network where low resource use and ease of configuration are important.
Supported platforms include Linux (with glibc and uclibc), *BSD and Mac OS X. Dnsmasq is included in at least the following Linux distributions: Gentoo, Debian, Slackware, Suse, Smoothwall, IP-Cop, floppyfw, Firebox, LEAF, Freesco, CoyoteLinux and Clarkconnect. It is also available as a FreeBSD port and is used in Linksys wireless routers and the m0n0wall project.
Main features:
- The DNS configuration of machines behind the firewall is simple and doesnt depend on the details of the ISPs dns servers
- Clients which try to do DNS lookups while a modem link to the internet is down will time out immediately.
- Dnsmasq will serve names from the /etc/hosts file on the firewall machine: If the names of local machines are there, then they can all be addressed without having to maintain /etc/hosts on each machine.
- The integrated DHCP server supports static and dynamic DHCP leases and multiple networks and IP ranges. It works across BOOTP relays and supports DHCP options including RFC3397 DNS search lists. Machines which are configured by DHCP have their names automatically included in the DNS and the names can specified by each machine or centrally by associating a name with a MAC address in the dnsmasq config file.
- Dnsmasq caches internet addresses (A records and AAAA records) and address-to-name mappings (PTR records), reducing the load on upstream servers and improving performance (especially on modem connections).
- Dnsmasq can be configured to automatically pick up the addresses of its upstream nameservers from ppp or dhcp configuration. It will automatically reload this information if it changes. This facility will be of particular interest to maintainers of Linux firewall distributions since it allows dns configuration to be made automatic.
- On IPv6-enabled boxes, dnsmasq can both talk to upstream servers via IPv6 and offer DNS service via IPv6. On dual-stack (IPv4 and IPv6) boxes it talks both protocols and can even act as IPv6-to-IPv4 or IPv4-to-IPv6 forwarder.
- Dnsmasq can be configured to send queries for certain domains to upstream servers handling only those domains. This makes integration with private DNS systems easy.
- Dnsmasq supports MX records and can be configured to return MX records for any or all local machines.
Enhancements:
- This release adds some features to the log subsystem: it is now possible to log directly to a file, and to log asynchronously to syslog.
- More detailed logging of DHCP transactions is available.
- DHCP options can now be specified by name instead of by number.
- DHCP configuration can now use information from a DHCP relay agent.
- There are a few minor bugfixes (notably failure of TFTP with --listen-address).
- Some translations have been updated.

The Wonder Shaper is a very special network shaper script with a lot of features. Works on Linux 2.4 & higher.

Goals

I attempted to create the holy grail:

* Maintain low latency for interfactive traffic at all times.

This means that downloading or uploading files should not disturb SSH or even telnet. These are the most important things, even 200ms latency is sluggish to work over.

* Allow surfing at reasonable speeds while up or downloading

Even though http is bulk traffic, other traffic should not drown it out too much.

* Make sure uploads dont harm downloads, and the other way around

This is a much observed phenomenon where upstream traffic simply destroys download speed. It turns out that all this is possible, at the cost of a tiny bit of bandwidth. The reason that uploads, downloads and ssh hurt eachother is the presence of large queues in many domestic access devices like cable or DSL modems.

Why it doesnt work well by default

ISPs know that they are benchmarked solely on how fast people can download. Besides available bandwidth, download speed is influenced heavily by packet loss, which seriously hampers TCP/IP performance. Large queues can help prevent packetloss, and speed up downloads. So ISPs configure large queues.

These large queues however damage interactivity. A keystroke must first travel the upstream queue, which may be seconds (!) long and go to your remote host. It is then displayed, which leads to a packet coming back, which must then traverse the downstream queue, located at your ISP, before it appears on your screen.

This HOWTO teaches you how to mangle and process the queue in many ways, but sadly, not all queues are accessible to us. The queue over at the ISP is completely off-limits, whereas the upstream queue probably lives inside your cable modem or DSL device. You may or may not be able to configure it. Most probably not.

So, what next? As we cant control either of those queues, they must be eliminated, and moved to your Linux router. Luckily this is possible.

Limit upload speed somewhat

By limiting our upload speed to slightly less than the truly available rate, no queues are built up in our modem. The queue is now moved to Linux.

Limit download speed

This is slightly trickier as we cant really influence how fast the internet ships us data. We can however drop packets that are coming in too fast, which causes TCP/IP to slow down to just the rate we want. Because we dont want to drop traffic unnecessarily, we configure a burst size we allow at higher speed.

Now, once we have done this, we have eliminated the downstream queue totally (except for short bursts), and gain the ability to manage the upstream queue with all the power Linux offers.

Let interactive traffic skip the queue

What remains to be done is to make sure interactive traffic jumps to the front of the upstream queue. To make sure that uploads dont hurt downloads, we also move ACK packets to the front of the queue. This is what normally causes the huge slowdown observed when generating bulk traffic both ways. The ACKnowledgements for downstream traffic must compete with upstream traffic, and get delayed in the process.

We also move other small packets to the front of the queue - this helps operating systems which do not set TOS bits, like everything from Microsoft.

Allow the user to specify low priority traffic (new in 1.1!)

Sometimes you may notice low priority OUTGOING traffic slowing down important traffic. In that case, the following options may help you:

NOPRIOHOSTSRC
Set this to hosts or netmasks in your network that should have low priority

NOPRIOHOSTDST
Set this to hosts or netmasks on the internet that should have low priority

NOPRIOPORTSRC
Set this to source ports that should have low priority. If you have an unimportant webserver on your traffic, set this to 80

NOPRIOPORTDST
Set this to destination ports that should have low priority.

See the start of wshaper and wshaper.htb

Results

If we do all this we get the following measurements using an excellent ADSL connection from xs4all in the Netherlands:

Baseline latency:
round-trip min/avg/max = 14.4/17.1/21.7 ms

Without traffic conditioner, while downloading:
round-trip min/avg/max = 560.9/573.6/586.4 ms

Without traffic conditioner, while uploading:
round-trip min/avg/max = 2041.4/2332.1/2427.6 ms

With conditioner, during 220kbit/s upload:
round-trip min/avg/max = 15.7/51.8/79.9 ms

With conditioner, during 850kbit/s download:
round-trip min/avg/max = 20.4/46.9/74.0 ms

When uploading, downloads proceed at ~80% of the available speed. Uploads at around 90%. Latency then jumps to 850 ms, still figuring out why.

What you can expect from this script depends a lot on your actual uplink speed. When uploading at full speed, there will always be a single packet ahead of your keystroke. That is the lower limit to the latency you can achieve - divide your MTU by your upstream speed to calculate. Typical values will be somewhat higher than that. Lower your MTU for better effects!

A small table:

Uplink speed | Expected latency due to upload
--------------------------------------------------
32 | 234ms
64 | 117ms
128 | 58ms
256 | 29ms

So to calculate your effective latency, take a baseline measurement (ping on an unloaded link), and look up the number in the table, and add it. That is about the best you can expect. This number comes from a calculation that assumes that your upstream keystroke will have at most half a full sized packet ahead of it.

This boils down to:

mtu * 0.5 * 10
-------------- + baseline_latency
kbit

The factor 10 is not quite correct but works well in practice.

Your kernel

If you run a recent distribution, everything should be ok. You need 2.4 with QoS options turned on.

If you compile your own kernel, it must have some options enabled. Most notably, in the Networking Options menu, QoS and/or Fair Queueing, turn at least CBQ, PRIO, SFQ, Ingress, Traffic Policing, QoS support, Rate Estimator, QoS classifier, U32 classifier, fwmark classifier.

In practice, I (and most distributions) just turn on everything.

The scripts

The script comes in two versions, one which works on standard kernels and is implemented using CBQ. The other one uses the excellent HTB qdisc which is not in the default kernel. The CBQ version is more tested than the HTB one!

See wshaper and wshaper.htb.

Tuning

These scripts need to know the real rate of your ISP connection. This is hard to determine upfront as different ISPs use different kinds of bits it appears. People report success using the following technique:

Estimate both your upstream and downstream at half the rate your ISP specifies. Now verify if the script is functioning - check interactivity while uploading and while downloading. This should deliver the latency as calculated above. If not, check if the script executed without errors.

Now slowly increase the upstream & downstream numbers in the script until the latency comes back. This way you can find optimum values for your connection. If you are happy, please report to me so I can make a list of numbers that work well. Please let me know which ISP you use and the name of your subscription, and its reputed specifications, so I can list you here and save others the trouble.

Installation

If you dial in, you can copy the script to /etc/ppp/ip-up.d and it will be run at each connect.

If you want to remove the shaper from an interface, run wshaper stop. To see status information, run wshaper status.

KNOWN PROBLEMS

If you get errors, add an -x to the first line, as follows:

#!/bin/bash -x

And retry. This will show you which line gives an error. Before contacting me, make sure that you are running a recent version of iproute!

Recent versions can be found at your Linux distributor, or if you prefer compiling, here:
ftp://ftp.inr.ac.ru/ip-routing/iproute2-current.tar.gz