Archive for the ‘Linux Kernel’ Category

Remote kgdb target debugging via the Cyclades TS-3000 Terminal Server

Friday, February 12th, 2010

So I’ve been poking at Jason Wessel’s kgdb patches recently (specifically, the ones in kgdb-next – you do believe in kernel debuggers, right? Good). They came in very handy when trying to track down an obscure netfilter brokenness last week that was causing Fedora kernels to fall over reproducibly when running KVM. That particular issue was caused by libvirt’s namespace code that attempts to create additional network namespaces on startup, just to see if it’s possible (for optional containers support). After a very long weekend, I pointed out a number of bugs that got fixed. But it got me thinking about kgdb and being able to easily debug stuff that rolls over and plays dead.

Traditionally, I have used a (somewhat loud, and sometimes therefore unfortunately annoying) PC attached to my debugging target via a serial crossover cable. Actually, it’s the inverse of the usual setup in which said other PC is intended to be the target of experimental test kernels, with my desktop generally not being anticipated to fall over with kernel bugs (as it has been doing increasingly of late). In any case, it’s not optimal to leave that PC running and I prefer it being used for evil test experiments. An opportunity to buy random crap on eBay presented itself in the form of an awesome Cyclades (now some other random company) terminal server. I bought a TS-3000 for $115, which is less than a tenth of what they used to go for retail. 48 ports of serial terminal server goodness for the home.

Photo: My Cyclades TS-3000 sitting atop an APC Masterswitch Plus

I was never very good at waiting for santa. I was tracking this damned thing several times a day for the two days it was in transit. And when it arrived – shock! – it might not have the latest firmware! Quick! Time to fix that. I hadn’t even used it in anger before I managed to brick the thing with an update not intended for this model. Cursing myself, I figured I would just rescue it via TFTP. But that requires a special console cable (not quite the same as some others) in order to interrupt the standard boot. Obviously I had none of these cables, and all of the ones here were useless. And I wasn’t prepared to wait ten minutes to order another one. So I went to Microcenter, and bought two RJ45-DB9 generic converters you can click together to wire yourself.

I followed a diagram online to make the RJ45-DB9 cable for the Cyclades – twice. But all of the posted diagrams were incorrect (this is nothing like a Cisco cable, even if you’re a moron and think that it is when you incorrectly make a website with the wrong pinouts, especially if you’re Cyclades and write a manual with the wrong information contained within it…thanks a bunch!). Not to be discouraged, the soldering iron came out, and I rummaged around in a box of parts to find some serial connectors. Fortunately, I had a female DB9 and plenty of old crappyish network cables. I soldered, desoldered, and resoldered this thing about 4 times before finding the correct Cyclades console cable pinout (ADB0036 female DB9) (repeated below, for the benefit of others who read this). Finally, I reflashed the unit with the same firmware it had had when it arrived (zImage_ts_140-3.bin) – the “new” firmware was only for specific other units of which mine was not one thereof, there is a newer “GPL” kit I will poke at sometime – and booted it up.

Photo: A homebrew Cyclades ADB0036 Cable

RJ45 pin DB9 pin
1 8 (CTS)
2 1 (DCD) and 6 (DSR)
3 2 (RD)
4 5 (SGND)
5 7 (RTS)
6 3 (TD)
7 4 (DTR)
8 4 (DTR)

Figure: The correct pinout for a Cyclades ADB0036 console cable (RJ45 to Female DB9 connector)

Cyclades made good (fanless) hardware, but they were hardly the most adept at making configuration straightforward. Sure, you can configure the network easily (this one is called “morse” after the inventor – in the US – of the coding used for telegraphs, which are an ancient precursor to the RS232 standard used on modern serial ports), but when it comes to the port setup…what you want to know is that you’re looking for the “Socket SSH” option, set to increment (e.g. from “1″ – no need to use the “7001″ example, you’re not directly sshing into the port anyway, as with telnet), and based upon a simple “CAS profile” with local authentication (make sure you add a new “system” user for those SSH logins), unless you want to use RADIUS (I have home KRB5, but haven’t deployed RADIUS at the moment). Always make sure you “Run Configuration” before flashing – it seems the former writes to the actual config files that the latter will use, so you cannot necessarily flash and then “Run Configuration” that way around, depending upon the particular operation you are performing.

Once you have the terminal server running, you can talk to it:

$ ssh user_name:port_number@terminal_server.address

More importantly perhaps, you can use the gdb remote target:

(gdb) target remote | ssh -t -t user_name:port_number@terminal_server.address

Remember to tell ssh not to ruin the day (fail to allocate a pty for your friendly conversation) by specifying the “-t -t”, then you can talk to Jason’s kgdb stub.

Next steps? I need to make some more of these damned ADB0036 cables (or find some more on eBay – anyone want some useless Cisco cables I bought thinking they were the same?) and hook them up to all of my systems at home. They will then constantly log via the awesomeness of GNU screen to a remote VM, and I can jump in if something rolls over and catch it so I won’t miss panic/debug opportunities.

Kernel debuggers FTW.


Remote fencing with an APC Masterswitch Plus (with an AP9606)

Sunday, June 28th, 2009

Photo: APC Masterswitch Plus (with an AP9606)

As I mentioned before, I’ve been fencing most of my home/office systems (and even lights) these days. The problem is that cheaper power switches like the IP Power 9258 can be damaged quite easily. Two of mine have failed under a particular load element and I’m not saying in that case that it’s not my fault (I still like those units), but it’s clear that having something more “household name” can be a good idea. So I looked on ebay and discovered that old APC Masterswitches now often go for similar money to other more expensive kit.

I bought an 8-port Masterswitch Plus (with an AP9606) this week. Previously these went for up to $1000, but can now be had for even a tenth of that much. And they do telnet/SNMP (and ssh, if you upgrade them – not so much of a concern in this particular out-of-band configuration). I looked around for fencing scripts and obviously found the Red Hat Cluster Suite fence_apc stuff but I don’t want to install lots of stuff, and I don’t want to talk over telnet if I’ve got a private SNMP community configured and am reasonably comfortable with that. So I updated my previous script to talk to APC Masterswitch units.

APC Masterswitch Plus (with an AP9606) fencing script.



Tuesday, June 16th, 2009

I recently registered and setup, where you can find links to the RSS feeds (podcast, and the transcripts), comment, and a lot more besides.


Linux Kernel Podcast

Monday, May 4th, 2009

So I’ll freely admit, I’m obsessed with Podcasts. I love listening to news – the New York Times Front Page, Wall St. Journal Today, NPR News, APM’s Marketplace, On Point, etc. I particularly like those Podcasts that provide a quick summary of what’s going on, since we don’t all have time to be subject experts on everything in life.

I recently pondered whether it would be useful to have a similar podcast for the Linux Kernel Mailing List. I was actually pretty surprised at the lack of existing podcasts (other than the excellent one from TimeSys), especially considering folks are often travelling and away from email – and then there are many people who are interested but don’t have a reason to follow the list so closely. Since I do already read the list, I decided it wouldn’t be much additional effort to make a quick recording based on what I had read that day. Look upon this as an experiment to gauge interest – and to see if there are enough volunteers to help make this a regular thing.

Today’s episode took 15 minutes to prepare and record, and 1 hour to get staged and setup with the right XML, etc. That’s hopefully 15-20 minutes per day on an ongoing basis, which isn’t much, and might help someone. I would especially like it if there were enough people who were interested in helping out that we could farm this out between a few of us on a weekly basis – but that’s predicated on this being useful. I’ll watch the logs, listen for feedback, and this will last as long as it provides something useful to the wider community.

Linux Kernel Podcast

Remote fencing with the IP Power 9258

Friday, May 1st, 2009

Photo: IP Power 9258

A little while ago, I picked up one of these units on eBay for around $125USD. I needed something that was slightly cheaper than the regular range of higher-end APC Masterswitch and had heard that this was an embedded Linux device. It is (though that doesn’t excuse the horrific web UI). This afternoon, I setup a new Real Time kernel test box and decided to get the fencing done right this time around – so I wrote a simple script. It allows one to do a few simple things: power on, power off, and report status:

[jcm@perihelion ~]$ ippower
Usage: ippower <status> | <target> <command>

TARGETS: perihelion apohelion power5 light
COMMANDS: on off status

[jcm@perihelion ~]$ ippower status
perihelion: on
apohelion: off
power5: off
light: off
[jcm@perihelion ~]$ ippower apohelion on
[jcm@perihelion ~]$ ippower apohelion on
[jcm@perihelion ~]$ ippower apohelion off
[jcm@perihelion ~]$ ippower status
perihelion: on
apohelion: off
power5: off
light: off

You can download my ippower fencing script if you would like to use it with a similar device.


Sunday, July 9th, 2006

I’m now the official maintainer of module-init-tools and am looking to get more folks involved with future development. There’s a mailing list, there’s a wiki, there’s a lot of interest in getting various patches moving forward.

One of my first key interests is in adding configuration file support to tools like depmod so we can dynamically alter the priority of “extra”, “updates”, “weak-updates” and other directories under /lib without having to have lots of per-distro patchsets in place. We also need to get some of the non-intrusive vendor patches upstream and generally have a code cleanup (there are a few obvious memory leaks from just looking at the list handling code in depmod alone).

Further out, there’s an interest in allowing greater integration with udev and expanding the functionality of modinfo. If you have useful ideas, please check out that website and mail me suggestions. If you have a kernel project that you’d like to host on, please also drop me an email – the wiki is easy enough for you to just add projects anyway.