hackyourworld.com

Prerequisets

Linux/Unix host
Linux/Unix destination with SSH server

Ready, Set, Go!

Getting right down to it, here is a quick simple snippet I use quite often to transfer batches of files (often text, or other highly compress-able files) :

tar cz file.foo folder foo/* | ssh user@destination.host.com ‘cat > ~/file.tar.gz’

Today I found myself wanting to transfer an entire CD to a remote host, normally I’d just find a place to download it, but being a proprietary disc, that wasn’t an option.  This was too easy, but keep in mind if you don’t have a good upload speed, this can take awhile:

dd if=/dev/cdrom | ssh user@destination.host.com ‘cat > cd.iso’

Now of course you can mount it on the remote server:

mount /home/user/cd.iso -o loop /mount/path

Any questions?

Prerequisets

This article assumes you’re already familiar with Mercurial, if you’re not, there are plenty of places to find a quick and basic tutorial.  Chances are if you’ve ever used version control, you’ll be able to follow.  Some of the Mercurial commands used don’t show up until later versions of Mercurial (1.6), so if you’re running an old version (shoutout to all those Lenny users), there are ways to accomplish the same, but the syntax and ease-of-use is a little more difficult.

What is the patch queue?

Mercurial’s patch queue is probably seen differently depending on the environment it’s used in.  Personally, I like to think of it as “flexible changesets”, meaning until I’ve finalized my queue and pushed it upstream I can work on any revision in the queue and “refresh” the revision to include changes I make.

What does that mean?

If you’re familiar with version control, you know that generally speaking, changesets are immutable, if you wanted to add some extra comments to a file, you’d have to make them in a new revision, separate from the changes that make the comments relevant.  I like to keep my version control history clean and concise, and that is where the patch queue really shines.

The patch queue in the workplace

I work with many various Mercurial repositories on a daily basis and can have projects in-progress across multiple repositories.  Normally, I’m not going to finish a project in a single sitting, so the very first thing I do when I start working is create a patch, named for set of changes I’m planning to implement, but the name doesn’t really matter.

Example: hg qnew patch_queue_example

This initializes the patch queue and puts the “patch_queue_example” patch on top of the stack.  I am now free to make changes, and can then save them by “refreshing” the patch I’m in.

Example:
touch newfile
hg add newfile
echo “Test” >> newfile
hg qrefresh

Now, if you look at the head of the repository (hg heads), you’ll see a new changeset, with the patch name.

changeset:   0:6627f1eb79a5
tag:         patch_queue_example
tag:         qbase
tag:         qtip
tag:         tip
user:        Tim <example@example.com>
date:        Wed Oct 12 09:14:59 2011 -0400
summary:     [mq]: patch_queue_example

This changeset includes my changes to “newfile”, which you can see by running “hg export tip”

Now, that I’ve made those changes, I want to work on changes to the same repository, but they don’t relate to my previous changeset, so I create a new patch.

hg qnew non_related_changes

Looking at “hg heads” again, I see a new revision:

changeset:   1:52cfaec4afa1
tag:         non_related_changes
tag:         qtip
tag:         tip
user:        Tim <example@example.com>
date:        Wed Oct 12 09:19:38 2011 -0400
summary:     [mq]: non_related_changes

Now, I’ll go about making new changes:

echo “These changes are unrelated to my previous changes, so I’ve created a new patch” > unrelated_changes
hg add unrelated_changes
hg qrefresh (again, this is rolls the current changes into the current patch)
hg heads

You’ll notice the short revision id has stayed the same (1), but the node has changed

changeset:   1:52cfaec4afa1
tag:         non_related_changes
tag:         qtip
tag:         tip
user:        Tim <example@example.com>
date:        Wed Oct 12 09:19:38 2011 -0400
summary:     [mq]: non_related_changes

You can see the changes by running “hg export tip”, you’ll see the “unrelated_changes” with the text we’ve echoed into it.

Great, get to the useful stuff

Now, let’s say I wanted to go back and edit the “patch_queue_example” changes, I decided I wanted to add some more files in that changeset.  Normally, you’d just add the files, and make a new commit, providing a commit message stating you forgot to do something in the previous commit, or something similar.
With the patch queue I can simple “pop-off” the current changeset and go back to editing “patch_queue_example”

hg qpop

popping non_related_changes
now at: patch_queue_example

So we’ll add a few files

echo “foo” > example1
echo “bar” > example2
echo “baz” > example3

Now, add them to Mercurial

hg add

adding example1
adding example2
adding example3

Looking at a “hg status”, you’ll see the files have been added:

A example1
A example2
A example3

Now, let’s save the changes and go back to the “unrelated_changes” patch

hg qrefresh
hg status

You’ll notice the files are no longer listed as added, because the changes have been “commited” ( != hg commit).

Let’s move back to the “unrelated_changes”, by “pushing” it back onto the top of the stack

hg qpush

applying non_related_changes
now at: non_related_changes

At this point, I hope you see where this is going, you can continue doing this until you’ve finished all the relevant work.
Now it’s time to convert them into “real changesets” so they can be pushed upstream, which I haven’t mentioned until now, you don’t want to try pushing patches upstream, namely because they should be considered unfinished until you’ve converted them, but also because each “hg qrefresh” you do, the node will be updated (Mercurial generates the node id based on changes/history, it will change if you change anything).

To convert these into real changes I want to do two things
1) Edit the commit messages
2) Remove them from the patch queue

Editing commit messages:
hg qpop (to move back to the first patch)
hg qrefresh -e (should drop you into a text editor, whatever you put here is the commit message)
hg qpush (to move to the second patch)
hg qrefresh -e (edit second patch commit message)

Removing from the patch queue:
Note: You can remove them one at a time using ‘hg qfinish $rev’, but we’re going to do it all at once.
hg qfinish -a

That’s it!  Now take a look at your log:

hg log

changeset:   1:1bb3052c69fc
tag:         tip
user:        Tim <example@example.com>
date:        Wed Oct 12 09:42:47 2011 -0400
summary:     This was the second commit in the Mercurial patch queue example

changeset:   0:5c616b8e4745
user:        Tim <example@example.com>
date:        Wed Oct 12 09:41:56 2011 -0400
summary:     This was the first patch in the Mercurial patch queue examples

These changes are ready to be pushed upstream.

This article of course does not cover extensive use and some of the neat things you can do while working in the patch queue, for the brave, here are some other commands I use frequently, maybe you can figure out where they’re good for.

Extra commands:
hg qnew -f new_patch_name
hg qpush –move patch_name

Every geek should have at least a few tips or tricks that they don’t have to reference documentation or google for, here’s a few of my favorites that I use daily.

1) SSH Tunnels – Each and every time I’ve used this to cut half the time off a task I remember just how great SSH is.

When to use: Anytime you need internal access to a remote network. Or perhaps your company filters your internet usage and you need to check your gmail, this allows you to TUNNEL to your internet back at the house and ask it to do all the talking for you.

Example: I’m working from home, but I really need to access a ‘local’ resource on the remote network, it could be something as simple as connecting to the router on-site that doesn’t have remote access setup.

Howto Linux: If you’re a linux user, use the “-D” option with your standard ssh command. IE: ssh -l username -D 54321 sub.domain.com

Howto Windows: Get PuTTY. You’ll likely want to save this configuration so, open PuTTY, type in the hostname, give the session a name, on the left hand panel navigate to Connection->SSH->Tunnels. Under “Add New Forwarded Port” pick a port number, I like to use something high in the range, 54321 or you can pick your own. There are two rows of radio buttons, on the top choose “Dynamic” and leave the other as “Auto”, click “Add”. Now before connecting, on the left hand panel goto Sessions and click “Save”, now click “Open”

Utilizing the tunnel: What this essentially does is creates a secure SOCKS proxy for you to use. In Firefox, goto Preferences->Advanced->Network->Settings and set the SOCKS proxy address to “127.0.0.1″, that’s right, localhost (but don’t use ‘localhost’), then enter the port you chose earlier. Click “Ok” and you’re now surfing securely through SSH.

2) Converting movies to pocket size (linux only):
I have a simple Creative Zen 8GB Mp3 Player, it was cheap and works exactly as expected. It plays video of the same quality popular cell phones use, occasionally I want to watch a TV episode so I pieced together a simple little alias to make it quick and painless, you’ll need mencoder for this, if your on debian/ubuntu it’s as simple as ‘apt-get install mencoder’

Code (I put in ~/.bashrc): alias encode=’mencoder -ovc xvid -oac mp3lame -vf scale=320:240 -xvidencopts fixed_quant=4 -o ${args[0]} ${args[1]}’

Usage: encode outputFile.avi /path/to/file/to/convert.flv|avi|mpg

3) Using axel for super fast downloads
Situation: You’re remotely connected to a server you adminster, you’re crunched for time and need to download a file, maybe an ISO or perhaps a service pack, but you need it fast. You have no problem finding links to get it, but which do you use, which will be faster? The solution, grab 3 or 4 of those links and use axel.

Usage: axel -n X url1 url2 url3…
Replacing ‘X’ with the number of connections you want to use, passing as many URLs in as you’d like. In most cases, 3 or 4 seem to be enough to max out my 2MB/s.

For the past year and a half I’ve been doing developement in a linux based enviroment, these tools and particularly these commands have become used frequently. If anyone has something useful, post it up!

Mercurial is a cross-platform, distributed revision control tool for software developers.

Easy way to view most recent changes:
Command: hg export tip | colordiff | less -R

Useful for: Quick code reviews, or just to peek at what you’ve recently done

Manipulating text files:

SED: (stream editor) is a Unix utility that (a) parses text files and (b) implements a programming language which can apply textual transformations to such files.

Command: sed ‘s/\t/”,”/g;s/^/”/;s/$/”/;s/\n//g’ > filename.csv
In action: mysql -uuser -ppass DBNAME -e “SELECT foo FROM bar WHERE baz = ‘world’” | sed .. > output.csv

Explanation:

• “s/\t/”,”/g;” – Search for tabs, replace with commas
• “s/$/”/;” – Replace last of line with quote
• “s/\n//g” – Remove newlines

Useful for: This is specifically used for converting MySQL results to csv. I build lots of mailing lists for my employer.

AWK: is a programming language that is designed for processing text-based data, either in files or data streams, and was created at Bell Labs in the 1970s

Command: awk ‘BEGIN {OFS=”|” FS=”|”}; {print $1, $3, $5}’ file.csv
Explanation:

• OFS – Output field separator
• FS – Field separator
• Printing fields 1, 3, and 5

Useful For: Pulling text out of delimited list and re-organizing the data, quickly and gui-free