On buying components

January 22nd, 2011

So, this is a plea for help. Back in the day, in the UK, it used to be possible to physically go to Maplin Electronics stores (now but a mere husk of its former self, barely worth the time of day) and get random components. They had huge stashes, and you could easily go buy an inexpensive pack of assorted resistors, a bunch of diodes, transistors, whatever. Building a home hacking kit was pretty easy. Then the Internet slowly killed them and hurt weekend hackers everywhere.

When I moved to the US, I brought a large quantity of passive, semi, and other components with me, and since I’ve not been doing much electronic hackery those have lasted a number of years now. But I’m in need of replenishing, and particularly in need of just a bunch of inexpensive bags of assorted components – nothing in particular, just a range of values to sit in my component boxes. I looked at DigiKey (whose website is always atrociously bad), Mouser, Newark, and all the rest (none of whom seem to know how to do good site design), but as usual the best option for the weekend hacker is to go with Sparkfun or Adafruit, and neither really does the “assorted parts” kit thing. Great if you want funky devices though.

I’m at a loss. Radioshack stores suck ass compared with even a few years ago, nobody else does decent retail components, and I’m not interested in navigating one-at-a-time expensive ordering with Digi. So where the heck am I supposed to go? I’m sure this is one of these things where I’m supposed to give up and watch MTV or enjoy Football (or another total waste of time), but I’d prefer not to. I’m more down with random intellectual pursuits instead. Recommendations for actually useful places to get what I want are quite welcome.


Fun with Arduino

January 22nd, 2011

So I decided to finally poke a little at my Arduino Mega 2560 last night, mostly as a random thing to do on the couch while watching Netflix. I wrote some simple programs to poke at IO ports – nothing that exciting. But I do now see the value of Arduino as a platform, especially if you just want to do something fun and not have to build an entire universe to get there. It’s not a full Linux environment, but that can be a good thing.

With Arduino, you don’t need to worry about lots of the usual microcontroller stuff since the AVR chip comes with a bootloader that interprets the “Sketch” you have compiled from Wiring (C-like) source code. You also don’t need to worry about flashing and driving serial lines because the board has one of those standard FTDI USB parts that provides both the serial programming interface and easily usable serial output, too. In other words, it’s a simple design, but also complex enough to be useful, and easy to use if you just want to make a simple gadget. At the same time, there’s a huge amount of potential there too.

As it was late, I ordered some parts on Sparkfun to replace a few in my trusty component box. I used to enjoy going to Maplin back in the day, but like Radioshack, their component selection shrank and I never really found a good place to buy small numbers of parts. Sparkfun isn’t perfect, but like Adafruit, it does seem to be one of the better places for hobbyist purchases these days. I picked up one of their FTDI based generic USB interfaces, some solid state relays, and a bunch of other bits for playing with Arduino some more as time permits. Maybe I’ll build a remote control power switch or whatever.


Another car incident

January 21st, 2011

So, sometime over the last 36 hours, someone saw fit to drive into my car and injure her in exactly the same spot she had been hit last time, way back in…last month. It seems that it happened in the private lot at the rear of my apartment, which only a certain number of people use, in addition to various snow removal contractor equipment. The damage doesn’t seem as severe, but of course, there’s no note or phone call, or “oh, sorry dude, did I just cause that giant gash in your car?”. Maybe they were too busy on their cellphone at the time to apply some level of concentration to the task of driving a heavy vehicle.

I first noticed this when driving Katherine somewhere and my car was making a crunching sound (from the wheel impacting the panel at the rear) as we hit rough parts of the road. I thought it was snow stuck in the wheel well, but sadly found after stopping that someone had swiped the panel above the wheel with some kind of heavy machinery. After some prodding, a contractor seems very willing to accept liability, in the kind of way that comes with a realization that you’re responsible for something you should have owned up to properly before. But we’ll see. Maybe they’ll cover the loss, or maybe my insurance will ding me again for another $500 deductible. Because I just love paying those. Oh yes.

Unlike too many other MA drivers, I enjoy taking care of my car, and following the rules of the road (Chapter 90, M.G.L,, and others) to the absolute letter of the law. I signal, observe proper breaking distances, check my rear view mirror every time I adjust my speed or location, and I constantly assess every other driver for risk assessment of stupidity. Yet, with all of the care and attention, I am no match for the stupidity and incompetence that I see on the roads all around me every single day. People don’t signal, they don’t look, they text and drive, they eat and drive, they text, eat, and drive while on their cell phone. They don’t think “gee, I’m driving a 2 ton killing machine that has actual safety consequences”. It’s just disgusting, and it’s no wonder some people give up owning cars or go off driving them. It’s like there’s a reward for the lowest possible driving standard, and it doesn’t matter if you whack into someone in a parking lot either, because it’s assumed everyone likes dings and dents (those being so common people have given up realizing they are neither inevitable nor necessary consequences of car ownership).

Photo: Proposed MX-5 Carputer design

I’m not taking this problem lying down. I’m going to write to the governor, requesting that some actual sanity be restored in this state’s driving regulation, and enforcement thereof, and I’ve a particularly nasty patent idea for an automated means to deal with bad MA drivers (I’m filing that one privately). Meanwhile, I’m going to install a (temporary) wireless night vision camera to monitor my car while I wait for time to build out a complete solution. That will involve installing front and rear “back up” cameras on the license plates, and other cameras, attached to a BeagleBoard installed in the car with a 3G radio. I need live, streaming video of everything that happens, from every angle, complete with location data. I want it so well done that the next person who hits me and fails to do the decent thing will regret it in court. Because if people don’t understand how to drive, they hopefully do understand being sued. That seems to work a lot better in the US sometimes, and if threat of legal action is the only way to get through to some people, then so be it. So, so done with crappy drivers.


Why Enron happened

January 15th, 2011

So every now and again, I watch documentaries I know are going to upset me. Yesterday, it was on how the bottled water industry is destroying this country’s ground water supplies to local municipalities (but I had to turn that one off because of how angry it made me – and boycott yet another corporation). Today, I watched a bit of “The Smartest Guys in the Room”, which I’ve seen many times before. Let me share some of my reasons for why Enron happened, because it angers me and I feel like ranting. Nothing will change, but I’ll feel better.

First, this country (the US) is too often regulation-averse. Reagan (the religious leader of modern Conservatism) told people “government is not the solution to our problems; government is the problem”. This, coupled with “free market” thinking tells people everything will be magically ok, that the government is the big bad wolf, and that if we’d only let people decide their own fate without any rules whatsoever then the “invisible hand” will take care of everything. This is the biggest bunch of miss-guided horse excrement I’ve ever heard in my life. What’s worse, it’s somehow (historically) blurred with confusing critique of the current implementation of the capitalist system with being a commie or whatever other nonsense totally unrelated to the discussion (I’m a “Capitalist”, but of the sensible and sane variety).

Without any regulation, people with a high moral compass and sense of ethics (perhaps the kind of people Reagan imagined the world outside the confines of the bubble of the Oval Office to be filled with), coupled with wider thinking about the world might act only for the greater good. But don’t count on it. In reality, in the real world that actually exists (outside of Washington and Hollywood, as well as within), there are bad actors, people out only for themselves (who don’t give a crap about destroying the country in the process – and I don’t mean this as an indirect political slur, people abusing the system are from many different political backgrounds), and the rest. Regulation employs rules and principles that are designed to prevent abuse. It allows people to play games, but within the bounds of having some actual upper limits on how insane they are allowed to be with other people’s lives. Regulation is also far from perfect. Situations arise that were never considered, ways are found to abuse any system, etc. But when regulation fails, is insufficient, or excessive, the solution is to fix the regulation, not remove it entirely.

Second, we’re too reactionary. Rather than thinking ahead, “gee, we haven’t done anything with our infrastructure in 50 years, do you think that bridge is going to fail any time soon? What about that tunnel?”, we just wait until bad stuff happens and then spend billions of dollars more than necessary to correct for it afterward (see also, Healthcare). We tend to consider all kinds of (irrational) new rules and regulations in the wake of giant scandals, rather than introducing a few simple rules beforehand, and empowering our government and regulators to track down these impending disasters years before they happen. In a similar vain, we will probably wait until the water is around our feet before we wake up to climate change, because it’s not such a huge giant scandal yet (unlike bridges and tunnels, billions of dollars won’t be sufficient this time around, nor all the hot air Congress can generate).

There are other problems, such as an unwillingness to introduce additional necessary taxes in order to pay for new infrastructure, or government programs that can actually make people’s lives better, but the main problems are a lack of regulation and a need to react to everything years after it happens, rather than years before. I want to make it clear, I love this country, which is why I’m very saddened by these realities. I’m also a believer in Capitalism, and of the value in market-driven innovation, etc. But I’m a more touchy-feely, friendly kind Capitalist in which everyone has an even playing field, starts at the same point, and plays by the same (sane) rules.


New England Driving Q&A

January 13th, 2011

Living in Boston (well, Cambridge), one eventually comes to realize that there are some issues with driver education in this State. Here are a few Q&As to help those facing intellectual challenges on the road. I wish I could hand these out to people I see violating the rules of the road (yes, I’ve read Chapter 90 of the M.G.L.s and the drivers manual, as well as the rules for my own town…have you?). All of these Q&As would have applied in the space of the 2 hours I was on the road (on and off) today, except for the cop with the radar gun, which was a couple of weeks ago (at night, with his lights off, in the middle of the Mass. Ave bridge…like that’s not going to end horribly badly as some distracted driver fails to notice the obstruction one day).

Q). I love reading people’s bumper stickers and I can’t read at a distance.
A). Buy some glasses, or have a passenger read the bumper stickers for you. When you drive less than a safe distance from someone, you create a safety hazard and cause undue stress for other drivers around you.

Q). Why do people get angry when I use my horn? It’s the New England way!
A). Your horn is intended to convey safety concerns, and indicate your presence. Yes, even in the US. It is not a generic signal used to harass other drivers. When you use it at other times, it says “I’m a giant moron”. In CAPS.

Q). What are these sticks or levers on my steering wheel intended for?
A). These are called turn signals. You use them to indicate your intended direction of travel before making a manoeuvre. It allows other vehicles to avoid colliding with you, and is not a sign of weakness.

Q). Is it ever acceptable to wave someone across the street at a green light?
A). No. This is incredibly dangerous. It pre-supposes that oncoming traffic will not plough straight into the person crossing. Any decent driver education program would teach this, but Massachusetts fails horribly.

Q). I’m in law enforcement and like obstructing traffic so I can catch people speeding with my radar gun! Yeah!
A). You also create a safety hazard and should be suspended, and prosecuted.

Q). I don’t like concentrating on driving, I prefer to use my phone, read a magazine or a book, etc.
A). You should have your license revoked and be banned from driving.

Q). I think everyone should drive in the bike lane, it’s a whole new lane!
A). You should have your license revoked pending re-education. You should learn that you are a danger to others on the road, especially cyclists.

Q). I don’t like finding parking. I prefer to use the magic lights with the orange triangle symbol.
A). These are intended for emergency use to indicate your vehicle is disabled or that you are in need of assistance. When you use them to park in the middle of the road, you create a safety hazard and should have your license revoked.

I love my car, and I love driving. But driving in Massachusetts is a constant struggle of outwitting intellectually challenged morons who should not have a driver’s license to begin with, let alone be allowed to drive an SUV on the road, in winter. Driving (even in the US) is not a fundamental right. It’s a privilege that you get for following the rules. That’s not to say everyone is a bad driver, but it is saying that MA has a problem and does not do enough to solve it. Have a few cops out there actually enforcing the rules…and don’t forget to tell them that signals, traffic signals, and the rules of the road in general apply to them and everyone else equally in civilized society, not just to you and me.

Q). Hey Jon, didn’t you have an accident recently?
A). Yes. Someone hit me as I was making a left turn (at a left turn signal). Under MA rules, I am presumed to be at fault even though I was not the one in the wrong. The other driver had just had his car repaired from the previous accident, and was sitting on the plastic sheeting following an interior detailing. So, sorry, I’m not a bad driving example in this situation. I’m just the stiff who gets lumped with the insurance hike. And this post isn’t even about that, it’s about people not engaging their brains. There is also a difference between an accident and willingly ignoring sensible rules of the road, as in the above.


Free Software Predictions for 2011

January 12th, 2011

So I was thinking over the holiday about what I expect to see in the coming year. I’ll write a more optimistic post about some of the great improvements I expect to see another time, but meanwhile here are some of the things I think will continue to suck this year, in many of the non-Enterprise Linux distributions.

1). People will care even less about UNIX and where we came from.

2). Churn rate of intrusive software updates will be too high.

3). Standards will be shunned and sidelined too often as unimportant.

4). The trend toward dumbed-down “User Experiences” will continue.

5). The wheel will be re-invented at least twice, probably more.

Meanwhile, my Mac running OS X will (sadly) remain compatible with itself, retain a similar user interface that works well for technical users, and will only receive tested, appropriate updates that work well under all situations.


The BeagleBoard [part 1]

December 31st, 2010

So I’ve been dissecting the BeagleBoard-xM OMAP3 port for a book project I’m working on in my spare time. This has also seen me brush up on my ARM assembler (PowerPC, POWER, and so forth are my first loves), in particular stuff that happens in “SVC” (Supervisor) mode. Here are some notes that might be of interest to others getting into Beagle or ARM porting (part 1).

ARM essentials

First, you should know that ARM (as it stands, in v7 of the architecture, we’ve yet to see the new 64-bit ISA released in v8 or whatever that will be) is a 32-bit processor IP core that uses fixed-width 32-bit instructions with various forms of instruction encoding. The ISA is designed to be simple, but it does include some very flexible (simple) instructions. For example, almost every instruction can be conditionalized based upon 4 bits of condition state (and that state is typically not updated automatically within the ALU unless it is specifically encoded into the affecting instruction) and some instructions (additions, etc.) can include shifts of one of the operands (giving you DSP-ish instructions for free). Using conditionalized instructions keeps the pipeline full as opposed to having a lot of branching, and can render very small and tight sequences. Modern ARM cores implement a modified Harvard architecture, as indeed are most “Harvard architecture” processors these days.

There are 30 general purpose registers, but not all of these are available at any one time. In fact, it is easier to think of ARM as having 15 “general purpose” registers with some banking being used in certain modes (such as FIQ or IRQ – Fast or regular IRQs) to replace a subset of the registers with a context-specific special set. Even better, think of it as 12 general purpose registers, with r13 used as the stack pointer (sp), r14 as the link register (lr), and r15 used as the program counter (pc). Even even better, think of according to the ABI calling convention, with r0-r3 used for argument passing, r4-r11 used for local variables, r12 as a function call scratch register, r13 as sp, r14 as lr, and r15 as pc. Some instructions assume the use of sp, lr, and pc (write something into r15 and that will be taken as a branch address), but generally register use is flexible. Of the 4 different stack modes supported, ARM conventionally uses a “Full Descending” stack in which the sp points to the last value written, but you can use any of the “store multiple” or “load multiple” instructions either with their “stack” friendly names (e.g. stmfd), or directly (stmdb). Processor state is encoded in the CPSR (Current Processor Status Register), and saved in the SPSR (Saved Processor Status Register) when handling various exceptions (traps).

The ARM ISA is intentionally extensible. Either ARM, certain third parties (in limited situations), or co-processors may extend it. Co-processors are a means for the ARM core to remain simple while offloading certain functions to additional cores (that might actually be part of the same die). For example, the original FPA (Floating Point Accelerator) was implemented as a co-processor, and has long since been deprecated by the VFP (Vector Floating Point) co-processor (currently at version 3 thereof). Another co-processor you might care about is cp15, the MMU or VMSA (Virtual Memory Systems Architecture). When an instruction the ARM doesn’t recognize is seen, it will try to find one of 16 possible co-processors that can implement it (each has a special pre-determined number, such as cp15 for the MMU) before hitting the illegal instruction handler. The latter allows some instructions to be implemented in software, via traps, such as the older floating point emulation in the Linux kernel. There are special instructions for copying data to/from co-processors, and each has its own self-contained/self-defined register set.

Modern ARM cores implement a very long pipeline internally, but the original design was three stage. And this leaks out into the visible value of, for example, the pc (which is generally 8 bytes away from where you think it should be – different pipeline stage). This is the reason why PC relative addressing can be confusing (though assemblers help to deal with this). Another issue is that the ARM fixed-width instructions didn’t traditionally have a direct parallel to the PPC “li” “ori” address loading sequence. Instead, a 4K pc-relative address could be easily loaded in one instruction, a 64K pc-relative in two, and certain other addresses could be loaded by means of the built-in barrel shifter used in (e.g.) the addition instruction or by using a “literal pool”, depending upon the assembler. This means that traditionally, only certain addresses could be loaded with a “LDR” “meta” assembly instruction, and that the assembler would try to replace your LDR with appropriate shifts (in possibly several instructions), etc. If it were unable to do so, it would generate a warning. You could also load addresses directly from memory and jump to those that way by means of loading them in to the pc. ARM “bl” relative branches (which use a direct label address) have a 32-MB limit, traditionally requiring trampolines to handle larger jumps[0]. Since ARMv6T2, there is a movt instruction for loading the top 16 bits of a register (leaving the lower alone), and since ARMv7 there is even a movw (load a value into lower 16 bits and zero the upper part), which now allows a movw/movt combination akin to the li/ori combination on PowerPC. A mov/movt combo is used by, e.g. GCC when generating longer jumps (in which it loads the pc directly).

The ABI for ARM was changed a few years ago from the OABI (old ABI) to the EABI (not to be confused with the PowerPC ABI of the same name). In particular, system calls are no longer implemented by passing the system call number in the “comment” field of the system call instruction, but are instead passed by means of register (faster than having the kernel poke at the memory to find what number was contained within that instruction). Also, 64-bit values are passed using sequential registers, beginning on an evenly aligned register number. The ARM implements a limited number of trap vectors, traditionally located at 0×0000_0000 but today possibly relocated high at 0xffff_0000 (this is the preference of the kernel, if it is possible, since it avoids having the NULL physical page over-used).

Memory alignment is extremely important to ARM. Not only did it not have things like floating point for the longest time, but to keep the design simple, unaligned memory accesses weren’t really supported historically. Modern processors do have some support for handling unaligned access, but you generally want to avoid that. ARM deals in Bytes, Half-Words (16-bit), and Words (32-bit), none of this IA32 historical nonsense of “double words”. 64-bit values can be handled by means of multiple word instructions, and also manipulated in user code on some systems by use of the VFP, and so forth.

ARM don’t make processors directly. They define the ISA (currently v7, though processors still implement v4 and v5, etc.) and provide reference designs in the form of soft and hard (netlists, gates, etc.) cores to third parties. Depending upon the licensing (whether it is a foundry or merely a regular licensee), that third party might only have the right to implement what they are given. But if they have special foundry access, they can actually change the design. In any case, certain CPU functionality is optional. In the past, these were designated by letters after the CPU (e.g. “TDMI” for older cores implementing the then-optional Thumb instruction set, supporting virtual memory, debug, etc.), but today profiles are generally used instead. Cortex (the latest generation) provides for A, R, and M profiles, of which the Application profile (A) is what we care about.

ARM kernel

Execution of the ARM kernel begins in the inferred standard location of arch/arm/kernel/head.S, at the place very obvious labeled with “Kernel startup entry point”. At this point, the MMU must be off, the D-cache must be off, I-cache can be on or off, r0 must contain 0, r1 must contain the “machine number” (an ARM Linux standard assigned number, one per machine port, passed from the bootloader code), and r2 must contain the “ATAGS” pointer (a flexible data structure precursor to things like fdt and device trees that allows a bootloader to pass parameters). First, the processor mode is quickly set to ensure interrupts (FIQ and IRQ) are off, and that the processor is properly in Supervisor (SVC) mode. Then, MMU co-processor register c0 is copied into ARM register r9 to obtain the processor ID. This is followed by a call to __lookup_processor_type (contained within head-common.S, the common file for both MMU-enabled and non-MMU enabled ARM kernels – the latter are not covered by this document).

__lookup_processor_type behaves like a number of other functions in the early kernel setup. First, you should note that it has a C-ABI companion (used later, in higher level code) called lookup_processor_type that also preserves stack and calling semantics for use from C-code, then calls into __lookup_processor_type. Like so many other of these functions, __lookup_processor_type has a __lookup_processor_type_data friend that contains three data items:

  • The address of the structure
  • A pointer to the beginning of an array of “processor info” structures
  • A pointer to the end of an array of “processor info” structures

The function loads the address of __lookup_processor_type_data into r3, then uses an ldmia instruction to load the members of that structure into r4-r6. Since the first member of the structure contains the (relocated) virtual address of the kernel, and r3 contains the (currently) physical address, a simple subtraction is used to store into r3 the offset between the linked address of __lookup_processor_type_data and the actual address. This offset is then applied to registers r5 and r6 (containing the second and third members of that structure – __proc_info_begin and __proc_info_end). A loop is then entered to iterate over the proc info structure array and find a matching known-processor. If a matching processor is found, the physical address is preserved in r5 on return, otherwise #0 (NULL) is written into it so that the calling code can determine whether a processor was found.

Once the processor type has been determined, a very similar function called __lookup_machine_type is called to find the machine type passed by the bootloader, and get the “machinfo” structure describing the machine in question. If either of these functions fails, a call is made to __error_p, which can be compiled (in debug mode) to output an error ASCII message to the UART (byte by byte) but at least gives a point to pick up in a hardware debugger. Next, a call is made to __vet_atags (which checks that the ATAGS in r2 begin with the ATAG_CORE magic marker[1], and so forth), then sometime later we call __create_page _tables to fudge up entries in cp15 that cover just the kernel code and data. After that, we call through a few functions beginning with __enable_mmu that actually cause the MMU to get enabled and for the return to be to __mmap_switched (also in the head-common.S file). __mmap_switched also has an __mmap_switched_data, which is used to store various global values (such as processor_id, __machine_arch_type, __atags_pointer, and so forth). After zeroing out the BSS, __mmap_switched causes the high-level kernel start_kernel function to be entered as usual.

start_kernel (init/main.c) does a lot of things on every architecture/platform. These include calls to setup_arch, which on ARM calls setup_machine. This latter function returns an mdesc (machine descriptor) which is used in setting up any ATAGS, and for paging_init (which calls devicemaps_init that uses the mdesc to see if any special device IO maps are needed early on, for example for debugging and so forth). If ATAGS are not specified, a default init_tags set is used that defaults the machine to e.g. MEMSIZE of 16MB RAM, and other limited (but sane) defaults. After running through other generic early startup code, start_kernel causes the kernel_init thread to run, which calls do_basic_setup. do_basic_setup calls various initcalls, including the special customize_machine initcall. Customize machine calls init_machine (e.g. omap3_beagle_init), which adds various platform devices and does other board specific setup.

When the board specific init function is run, it will typically call platform_device_register for each known platform device (you can see these in /sys/devices/platform for example). Some of these platform entries include a driver_name field, which will cause the actual device driver to be loaded, but others contain more limited data. For example, in the case of EHCI (USB) on the Beagle, an ehci platform device is registered, but what causes the actual USB driver to load (if not built-in) is that the OMAP EHCI driver contains a MODULE_ALIAS(“platform:omap-ehci”) that udev and modprobe will cause to load on boot. In the case of OMAP, it might seem like a lot of hardware addresses and so forth are missing from the platform registration, but this is because the hardware is standardized (EHCI is always in the same place for a given OMAP, and hard-coded into the USB driver). One thing that does differ is GPIO MUX assignment, and these pins/values are poked in the init_machine code for the Beagle.

That’s all for now. More later.


[0] Modern ARM also implements the Thumb instruction “BLX” (Branch with Link and optional eXchange) that can be used in combination with a series of assembler-generated mov/orr sequences to load an address into a register and branch to it. The older “bl” requires a pc-relative label address to jump to. In fact, when you use a sequence like “ldr reg, address” and “blx reg” the assembler may in fact generate 5 of more actual instructions using mov, orr, and bx or blx to perform the actual load and branch. Rather than switching to Thumb mode, the modern mov/movt combination is preferable for loading arbitrary addresses.

[1] I do wonder what significance this particular number has to rmk. If anyone knows why it was chosen – a birthday? or other date? etc. please do let me know.