biais.org

Introduction

This article explain how to trace OpenGL ES function calls of your iPhone application but you can trace any other kind of function call using this technique. Note that I'm not sure it's the best way to do that but I didn't found any other tool able to trace all function calls (Instruments can sample process usage, but I didn't manage to trace all function calls with it).

The technique is based on a base gdb script:

1. break on glSomething functions,

2. for each break, use commands to print the frame #1 (where the glSomething has been called in your source code).

Sample translation for the glEnableClientState function:

set breakpoint pending on # Enable breakpoints on symbols not yet loaded

break glEnableClientState
commands      # start command when the break is 
silent
frame 1       # print the frame #1
continue      # continue to the next break
end           # end the command list

Following scripts and iphonesim source code is available on my iphone-opengl-profiler github project page

Note: if you want to execute the following guide, you should download the projet and compile the iphonesim binary.

Example use

Generate gdb script

I wrote a small script to generate the list of glFunctions you use in your program, for example for the OpenGLES sample project:

$ gdbscripts/listGlFunctions.sh "/Users/max/work/iphone apps/OpenGLES/build/Debug-iphonesimulator/OpenGLES.app/OpenGLES" \
  > glFunctions.txt

Then run the following script to generate the gdb script from your function list:

$ gdbscripts/generate-gdb-trace.sh glFunctions.txt > gdbscript.txt

Now you're ready to launch your application and attach gdb.

Run your iphone app in the simulator

Run the iphonesim binary that will launch your application in the simulator, run gdb and attach it to your process.

$ iphonesim launch "/Users/max/work/iphone apps/OpenGLES/build/Debug-iphonesimulator/OpenGLES.app/OpenGLES" \
  -debug YES -sdkVersion "3.0" -gdbcommands gdbscripts.txt > gdb-trace.log

Ctrl-C when you decide to stop your applicaton (when you think you gathered all profiling data you need).

Analyze gathered data

I wrote a last script to filter glSomething commands, you can run it:

$ cat gdb-trace.log | gdbscripts/opengl-state-tracer.sh | head
glGenFramebuffersOES(1, &ViewFramebuffer);
glGenRenderbuffersOES(1, &ViewRenderbuffer);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, ViewRenderbuffer);
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_WIDTH_OES, &backingWidth);
glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_HEIGHT_OES, &backingHeight);
glGenRenderbuffersOES(1, &ViewDepthRenderbuffer);
glBindRenderbufferOES(GL_RENDERBUFFER_OES, ViewDepthRenderbuffer);
glBindFramebufferOES(GL_FRAMEBUFFER_OES, ViewFramebuffer);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES, GL_RENDERBUFFER_OES, ViewRenderbuffer);
glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_DEPTH_ATTACHMENT_OES, GL_RENDERBUFFER_OES, ViewDepthRenderbuffer);

Get basic statistics:

$ cat gdb-trace.log | gdbscripts/opengl-state-tracer.sh | sort | uniq -c | sort -n -r
461 glMatrixMode(GL_TEXTURE);
461 glLoadIdentity();
249 glMatrixMode(GL_MODELVIEW);
198 glMatrixMode(GL_PROJECTION);
...

Have fun profiling you're OpenGL ES iPhone applications !

I was looking for statistics about iPhone applications and raw data to extract my own statistical data. After a quick googling, I didn't found any way to crawl the App Store easily, so I tried to do it quickly.

Each application data is accessible via HTTP on this url: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=FIXME-ID where FIXME-ID is a number indetifying an application. To make an HTTP request on this url, you have to fake the User-Agent (iTunes/9.0.2 (Macintosh; Intel Mac OS X 10.5.8) AppleWebKit/531.21.8 works).

With theses informations you can write this kind of shell script to brute crawl the App Store:

OUT=downloaded
ENDID=337489918
CURDIR=0
WGETMAX=40
mkdir -p $OUT/$CURDIR
cd $OUT/$CURDIR

i=$ENDID
while [ $i -ge 0 ]; do
    if [ $((i % 25)) -eq 0 ]; then
	CURDIR=$(($i / 10000))
	if [ ! -d ../$CURDIR ]; then
	    mkdir  ../$CURDIR
	fi
	cd ../$CURDIR
	# Limit the maximum number of wget process
	nwget=$(ps auxxwww|grep "wget"|grep apple|grep -v grep|wc -l)
	if [ $nwget -ge $WGETMAX ]; then
	    sleep 1
	    continue
	fi
    fi
    i=$(($i-1))
    echo $i
    wget "http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=$i&mt=8" \
            -U "User-Agent: iTunes/9.0.2 (Macintosh; Intel Mac OS X 10.5.8) AppleWebKit/531.21.8"\
            > /dev/null 2> /dev/null && if [ $(\ls -l "viewSoftware?id=$i&mt=8"|tr -s " "|cut -d " " -f 5) -le 5000 ];\
            then rm -f "viewSoftware?id=$i&mt=8"; fi  &
done

My first iPhone game has been accepted in the Appstore. I worked on it during february 2009 and I stopped because of lack of motivation and lack of graphic design talents ;). Recently, I took 2 full days to work on it, I polished some things (sounds and online high scores) and decided to submit it for free. A few days ago, Apple sent me an email telling me that the game was downloadable in the Appstore.

Here is a short video of the game:

For french readers, I just started a blog in french about the Scala language. Some translations from the original english documentation, but also articles and news about scala. The name is quite original : http://www.scala-fr.org/ ;).

I like testing piece of code I write. Because I often rewrite things, optimize and try to make things better (maybe i'm bad in doing the best thing at first time ?). When I'm working, I have constraints : time, coworkers, boss. In some of the companies I worked for, tests were only an option on particular code. I'm trying to convince that testing everything is good (In my current job, I'm actualy happy about that, because I'm not the only one who want to write tests ). I never had the idea to do that, but I think this is something I should have done before : calculating how much time you spend testing.

The estimation done by the author reveals a 10% of his time writing tests and 90% writing the actual software :

The number of hours spent on the project is my best estimate, as I have not kept track of these numbers. Also, the 10% breakdown comes from keeping track of my coding habits for the last two weeks of coding. But, these are my best guesses.

I started to learn Scala. I started by reading some code snippets and then write my own program. Scala integrates features of object-oriented and functional languages. When reading code example I saw that a reserved keyword was often used, it's the Scala wildcard: the underscore (_). And it's used in different ways.

Underscore and anonymous parameters

I read some examples that use the Scala wildcard character: _ (the underscore). Here is one:

val sum = List(1, 2, 3).reduceLeft(_+_)

Note that the reduceLeft method apply a function of two arguments cumulatively to the items of the List. That means _+_ is a function of t two arguments. This example can be translated to:

val sum = List(1, 2, 3).reduceLeft((x, y) => x + y)

(x) => x is the Scala notation to write anonymous functions. For pythoners, this is the equivalent to lambda x: x.

As described in the Scala language specification. Implicit anonymous functions using underscores in parameter position. Some examples below. The expressions in the left column are each function values which expand to the anonymous functions on their right:

_ + 1                 x => x + 1 
_ * _                 (x1, x2) => x1 * x2 
(_: int) * 2          (x: int) => (x: int) * 2 
if (_) x else y       z => if (z) x else y 
_.map(f)              x => x.map(f) 
_.map(_ + 1)          x => x.map(y => y + 1)

Another example, I think, a bit harder to see where the anonymous function is hidden (note that foreach takes a function of one argument):

object Test extends Application {
    private def printAddition(a: Int, b: Int)  {
        println(a + b)
    }
 
    List(1, 2, 3).foreach(printAddition(5, _))
}

is equivalent to:

object Test extends Application {
    private def printAddition(a: Int, b: Int)  {
        println(a + b)
    }
 
    List(1, 2, 3).foreach((x) => printAddition(5, x))
}

Underscore and partially applied function

Let's write a simple function that return a function:

object Test extends Application {
    private def createAddFunc(a: Int) = {
      (x: Int) => a + x
    }
    
    val add100 = createAddFunc(100) // createAddFunc return a function, 
                                // so add100 references a function
    add100(3) // call add100 method (returns 100 + 3)
 
    // We can also write this:
    createAddFunc(100)(3)
 
    // Or affect createAddFunc to another name (note the trailing _)
    var createAddFuncCopy = createAddFunc _
}

In the last line of code of this example, the trailing underscore tells the compiler to look at createAddFunc as a functional, and not invoke it.

Underscore and pattern matching

Scala has a pattern matching mechanism (first-match policy). In this case, the underscore matches everything that has not be matched before:

object Test extends Application {
    private def cardColor(symbol: String) : String = symbol match {
      case "heart" => "red"
      case "diamond" => "red"
      case "spade" => "black"
      case "club" => "black"
      case _ => "typo ?" 
    }
    
    println(cardColor("club"))
    println(cardColor("clubb"))
}

prints:

black
typo ?

In this case, the Scala underscore is the equivalent to the underscore in OCaml, it matches everything that has not be matched:

let cardColor = function
      "heart" | "diamond" -> "red"
    | "spade" | "club"    -> "black"
    | _       -> "typo ?";;

print_endline (cardColor "club");;
print_endline (cardColor "clubb");;

Note the previous Scala code can also be written à la OCaml using the "|" symbol:

private def cardColor(symbol: String) : String = symbol match {
    case "heart" | "diamond" => "red"
    case "spade" | "club" => "black"
    case _ => "typo ?" 
}

Underscore and package imports

Another use of the _ is to import all tje classes of a package:

import scala.io._ 

THe equivalent of the java syntax:

import java.util.*; 

I'm currently working on an iPhone game during a part of my spare time. The project name is OrangeSlice. It's based on the very good 2D graphics library cocos2d-iphone

Here is a video I captured from the iPhone Simulator (that doesn't support accelerometer, so the all the game features are not showed):

Maker Faire Africa (MFA), a celebration of African ingenuity, innovation and invention, will take place August 13-15 at the Ghana-India Kofi Annan Centre of Excellence in ICT in Ghana's capital, Accra.

You also should check out the Afrigadget website:

dedicated to showcasing African ingenuity. A team of bloggers and readers contribute their pictures, videos and stories from around the continent. The stories of innovation are inspiring. It is a testament to Africans bending the little they have to their will, using creativity to overcome life’s challenges.

In an old post I wrote about a metaheuristic: particle swarm optimization (PSO). It was the simplest code to demonstrate what PSO looks like. Today I'm writing about genetic algorithm, another metaheuristic inspired by evolution of species. You can read the description of the algorithm on the genetic algorithm article on Wikipedia.

I applied the algorithm to a problem that is not really one: trying to help lazy programmers to write file converters. I had to write file converters to unify all (more or less) formated input files into one kind of CSV file. Each of these converters is made using the right combination of filtering / regexp matching / line splitting.

So I wrote a program based on genetic algorithm where an individual is composed by 3 "genes":

• a list of filters (remove consecutive spaces, replace tabs by spaces, ...)
• a list of basics regexp (int, float, date, line separator, ...)
• a list of cleaning functions (remove empty cell, merge cells, ...)

The fitness of an individual is calculated using a sample file to parse that goes through the genes (filters / regexp / cleaning functions). The result of this process is a CSV file that can be compared to the expected sample result (that I wrote manually). The comparison uses the SequenceMatcher of the difflib Python stantard module, it returns the fitness: a float. A fitness close to 1.0 means the results is close to the expected CSV format. When the fitness is exaclty 1.0, that means the converter works perfectly for the sample.

Here is a sample file I wanted to convert:

08/12   	 Hello world 06/12
Paris 121231231 	  	- 22,29
08/12 	Something something ... 04/12
1111 1111 1111 1111 	  	- 14,35
08/12 	something else
	  	- 12,96
26/11 	Vir AAAAA
AAAAAA 2008	  	264,51

into this csv file:

"08/12","Hello world 06/12","Paris 121231231","-22,29"
"08/12","Something something ... 04/12","1111 1111 1111 1111","-14,35"
"08/12","something else","","-12,96"
"26/11","Vir AAAAA","AAAAAA 2008","264,51"

The program prints for each generation the 5 bests individual (I'm using elitism, so the best individual is always kept from a generation to the next one). A sample run:

$ python ga.py tests/sample1.txt tests/sample1.expected result1.pickled
Generation: 0 (mutation rate=10)
0.53772
0.42507
0.30406
0.28598
0.26389
Generation: 1 (mutation rate=10)
0.53772
0.42507
0.32684
0.30406
0.26799

...

Generation: 271 (mutation rate=15)
1.0
0.96025
0.95107
0.91779
0.87886
"08/12","Hello world 06/12","Paris 121231231","-22,29"
"08/12","Something something ... 04/12","1111 1111 1111 1111","-14,35"
"08/12","something else","","-12,96"
"26/11","Vir AAAAA","AAAAAA 2008","264,51"

filters: str_remove_consecutive_spaces, str_remove_somespaces, str_strip, str_remove_consecutive_spaces
regex: ([0-9]{2}/[0-9]{2})(.+?)(
)(.+?)(-?[0-9]+[.,]+[0-9]+)
cleaners: clean_strip, _in, _in, _in

Conclusion

The good:

• It's fun to see your program evolve
• You are lazy and don't want to write _many_ of this kind of file converters
• With a good interface and if the basic functions and genes size are well defined, a non-programmer can create his own parser (I think this is the only argument to use it).

The bad:

• The resulted parsers are not optimal

The ugly:

• Of course it's quite easy to write such genes by hand
• It may never converge to 1.0 and generated parsers of fitness != 1.0 are useless (this may not be the case for other kind of GA applications)
• You need many well chosen primitives to cover a wide set of solutions

Download

gaparser-0.1.tar.bz2: source code and examples.

OpenCalais is a free web service that can perform semantic analysis on any English text. It processes the text sent in your request and respond with extracted concepts and relationships. It's a great tool if you want to play with semantics and if you want to add some nice features to your website / blog.

As an example, I tried to send the text from a this small article about Ruby and Python. Note : For readability I kept only interesting data from the response :

<!-- 
Relations: 
ProgrammingLanguage: Python, Ruby
--> 
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:c="http://s.opencalais.com/1/pred/">
  <rdf:Description rdf:about="...">
    <!-- ProgrammingLanguage: Python; --> 
    <c:detection>[similarities and differences between Ruby and ]Python[ but I didn't find any idioms list in Ruby, so if]</c:detection> 
    <c:prefix>similarities and differences between Ruby and</c:prefix> 
    <c:exact>Python</c:exact> 
    <c:suffix>but I didn't find any idioms list in Ruby, so if</c:suffix> 
    <c:relevance>0.543</c:relevance> 
  </rdf:Description>
 
  <rdf:Description rdf:about="...">
    <!-- ProgrammingLanguage: Ruby; --> 
    <c:detection>[ list in Ruby, so if you know one or if you are a ]Ruby[ programmer, please post a]</c:detection> 
    <c:prefix>list in Ruby, so if you know one or if you are a</c:prefix> 
    <c:exact>Ruby</c:exact> 
    <c:suffix>programmer, please post a</c:suffix> 
    <c:relevance>0.386</c:relevance> 
  </rdf:Description>
</rdf:RDF>

The analyzed text is quite small but the results seems OK : 2 programming languages detected here, no animal, no gemtone...