LoudHush Blog

OrbView-3 imagery freely available from USGS

2012-01-25T21:06:00.000+02:00

geo-spatial.org's mailing list had a link to this little gem: High Resolution OrbView - 3 Images Now Available from USGS.

The resolution itself is not spectacular when compared to the Bing imagery displayed in OSM's Potlatch and the coverage of Romania's territory is rather sparse, but the license lets you do whatever you want with the imagery (public domain).

A preview of the OSM street network on top of OrbView-3 imagery for Bucharest, Aviatiei:

geo-spatial.org are working on a project to make the processed data available through their WMS service.

Line of sight analysis using Grass GIS and SRTM elevation data

2011-11-08T13:27:00.000+02:00

Grass GIS has built-in support for line of sight analysis (the command is called r.los).

I'm revisiting an older post (Mountain Hiking Tracks in Făgăraș) that I've ended with a line of sight diagram I've built by hand based on the description given by a fellow hiker who described to us what we were seeing (peak names).This time I've used Grass GIS to build the line of sight raster from Vânătoarea lui Buteanu peak with a 20 Km radius.The background is based on the freely available Landsat 7 image data.

Note there are NULL values (failures to read altitudes in certain raster cells) in the SRTM data (represented in black in the picture below) so the line of sight analysis above was carried on an interpolated version of the data.

As far as I can tell, the diagram is fairly accurate. I'm not sure about "Turnul Paltinului" and the small ridge following it, but the rest of the features fall into place pretty nicely.This is a photo taken from Vânătoarea lui Buteanu towards Negoiu Peak:

And this is a mash-up of the line of sight diagram over-imposed on the photograph:

Using OpenStreetMap osm2pgsql data in Grass GIS

2011-11-07T14:39:00.000+02:00

These are my notes on using the OSM data as imported by osm2pgsql in a PostgreSQL database with Grass GIS.

While Grass is a very powerful tool, it's Unixy feel is a little disconcerting.If you're a programmer like me, it helps to realize the Grass command line shell is in fact a preconfigured bash session:


GRASS 6.4.1 (OSM_900913):~ > ps ax | grep "$$" | grep -v grep
98804 s000  S      0:00.06 /bin/bash

The things you get from Grass are a set of binaries that interact with the data you have (organized in locations that are part of a mapset stored inside your filesystem).The other thing that is somewhat unexpected for a PostGIS user is that, within a Grass location, all the data has to share a single projection.

Creating a location for your OSM data

The first step required when using OSM data is to create a Grass location for the OSM data.I've used the same projection osm2pgsql uses when importing, the "fake" projection with SRID 900913. Grass doesn't know it, so you can define it as a custom PROJ4 string based on the definition you get from PostGIS:


osm=# select proj4text from spatial_ref_sys where srid = 900913;
                                                    proj4text                                                     
------------------------------------------------------------------------------------------------------------------
 +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +units=m +k=1.0 +nadgrids=@null +no_defs
(1 row)

Here are the relevant screen shots when defining the location using Grass Python WXGUI:

Creating temporary tables in OSM

To reduce the size of the data set, I've created a couple of tables in PostGIS by selecting only those features I planned to use for my map by spatially selecting those features within a certain administrative boundary:


create table bucharest_s1_streets as
select * from planet_osm_line where st_contains((select way from planet_osm_polygon where boundary = 'administrative' and name ='Sector 1' limit 1), way)
and highway is not null;

create table bucharest_s1_pois as
select * from planet_osm_point where st_contains((select way from planet_osm_polygon where boundary = 'administrative' and name ='Sector 1' limit 1), way);

I then defined these tables in PostGIS' geometry_tables (otherwise Grass will refuse to import them):


insert into geometry_columns(f_table_catalog, f_table_schema, f_table_name, f_geometry_column, coord_dimension, srid, type) values('','public','bucharest_s1_pois','way',2,900913,'POINT');
insert into geometry_columns(f_table_catalog, f_table_schema, f_table_name, f_geometry_column, coord_dimension, srid, type) values('','public','bucharest_s1_streets','way',2,900913,'LINESTRING');

Importing the data in Grass

For some reason, my Grass setup fails to set the environment variables below, so I have to re-export them:


GRASS 6.4.1 (OSM_900913):~ > export GISDBASE=/Users/diciu/grassdata
GRASS 6.4.1 (OSM_900913):~ > export LOCATION_NAME=OSM_900913
GRASS 6.4.1 (OSM_900913):~ > export MAPSET=PERMANENT

The PostGIS database parameters have to be defined next:


export HOST=localhost
export DB=osm
export USER=gisuser
export PASS=somepassword
export DSN="PG:host=${HOST} dbname=${DB} user=${USER} password=${PASS}"

You can now run the v.ogr.in commands to import your data (I'm importing roads and pharmacies):


v.in.ogr dsn="${DSN}" layer=bucharest_s1_streets out=streets
v.in.ogr dsn="${DSN}" layer=bucharest_s1_pois out=pharmacy where="amenity='pharmacy'"

If everything went well, not you have two vector layers in your location:


GRASS 6.4.1 (OSM_900913):~ > g.list type=vect
----------------------------------------------
vector files available in mapset :
pharmacy                  streets
----------------------------------------------

Computing distance to pharmacies

The Vector networking tutorial describes a number of very nice examples of processing tools available in Grass.I am using v.net.iso below to compute the distance from pharmacies:


v.distance -p from=pharmacy to=streets output=pharmacy_conn_streets upload=dist column=dist
v.patch in=pharmacy,streets,pharmacy_conn_streets out=pharmacy_net
v.clean in=pharmacy_net out=pharmacy_net_clean tool=snap,break thresh=1
v.net.iso input=pharmacy_net_clean output=pharmacy_iso ccats=1-100 costs=200,500,1000 nlayer=1

The command's output are a number of vector layers, pharmacy_iso being the one that stores our road network categorized by distance to pharmacies (less than 200 meters, between 200 and 500 meters and so on).

Rendering a map using the data

It's much easier and the results are probably better if you use QGIS instead of Grass' UI.There's a plugin for QGIS that asks you to specify the Grass mapset, location and loads a given layer. As an added bonus, QGIS is able to re-project on the fly so you can mix layers in different projections.

Here's a map of pharmacies in my neighborhood as rendered by QGIS' export to PNG feature:

Using pgRouting on an OSM database imported by osm2pgsql

2011-10-12T12:57:00.000+03:00

If you've tried to use pgRouting on an OSM database imported by osm2pgsql you've probably noticed it doesn't work.

It doesn't work because one of the requirements for pgRouting is that streets are split at intersections, meaning it expects each osm_id from planet_osm_line to only contain street segments.
However, in planet_osm_line, a street is represented by a LINESTRING, most of the times the intersection with other streets being just another POINT in the street's LINESTRING.

A visual representation of the long explanation above:

The good news is that it's fairly easy to split the OSM streets into constituent segments using PLPGSQL.
One way to achieve that is the one listed below (note the filtering on "Sector 1" - I am restricting table entries within the administrative boundary of the first sector of Bucharest because things get really slow on the entire OSM road network):



drop table if exists network;
create table network(gid serial, osm_id integer, name varchar, the_geom geometry, source integer, target integer, length float);


CREATE OR REPLACE FUNCTION compute_network() RETURNS text as $$
DECLARE
streetRecord record;
wayRecord record;
pointCount integer;
pointIndex integer;
geomFragment record;
BEGIN
-- for each street
FOR streetRecord in select way, osm_id, name from planet_osm_line where highway is not null and st_contains((select way from planet_osm_polygon where boundary = 'administrative' and name like 'Sector 1' limit 1), way) LOOP
 -- for each street in the region of interest
 SELECT * from planet_osm_ways where id = streetRecord.osm_id into wayRecord; 
 FOR pointIndex in array_lower(wayRecord.nodes, 1)..array_upper(wayRecord.nodes,1)-1 LOOP
  RAISE NOTICE 'Inserting name % source %, target %', streetRecord.name, wayRecord.nodes[pointIndex], wayRecord.nodes[pointIndex+1];
  select st_makeline(st_pointn(streetRecord.way, pointIndex), st_pointn(streetRecord.way, pointIndex+1)) as way into geomFragment;
  insert into network(osm_id, name, the_geom, source, target, length) values(streetRecord.osm_id, streetRecord.name, geomFragment.way, wayRecord.nodes[pointIndex], wayRecord.nodes[pointIndex+1], st_length(geomFragment.way));
 END LOOP;
END LOOP;
return 'Done';
END;
$$ LANGUAGE 'plpgsql';

select * from compute_network();
select assign_vertex_id('network', 1, 'the_geom', 'gid');

The PLPGSQL procedure works by creating a street segment for each pair of nodes from planet_osm_ways; the geometry is created by simply creating a new line from the points corresponding to the nodes (see the st_makeline call).

Once assign_vertex_id has been run, you can start using shortest_path:



osm=# select n.name, p.edge_id, o.highway from network n, (select edge_id from shortest_path('
osm'# SELECT gid as id,
osm'# source,
osm'# target,
osm'# length as cost
osm'# FROM network',
osm(# (select source from network where name='Intrarea Jugului' limit 1),(select source from network where name='Strada Sofia' limit 1),false,false)) p,
osm-# planet_osm_line o
osm-# where p.edge_id = n.gid and n.osm_id = o.osm_id;
             name             | edge_id |   highway   
------------------------------+---------+-------------
 Intrarea Jugului             |    2330 | residential
 Strada Elena Caragiani       |     808 | residential
 Strada Borșa                 |    2016 | residential
 Strada Borșa                 |    2015 | residential
 Strada Borșa                 |    2014 | residential
 Strada Borșa                 |    2013 | residential
                              |    1992 | residential
 Șoseaua Pipera               |    8098 | secondary
 Strada Nicolae Caramfil      |    1828 | secondary
 Strada Nicolae Caramfil      |    1827 | secondary
 Strada Nicolae Caramfil      |    1826 | secondary
 Strada Nicolae Caramfil      |    1825 | secondary
 Strada Nicolae Caramfil      |    1824 | secondary
 Strada Nicolae Caramfil      |    1823 | secondary
 Strada Nicolae Caramfil      |    1822 | secondary
 Strada Nicolae Caramfil      |    1821 | secondary
 Bulevardul Beijing           |    1035 | secondary
 Bulevardul Beijing           |    1036 | secondary
 Bulevardul Beijing           |    7883 | secondary
 Bulevardul Beijing           |    7766 | secondary
 Bulevardul Beijing           |    1474 | residential
 Bulevardul Beijing           |    1473 | residential
 Bulevardul Beijing           |    1472 | residential
 Bulevardul Beijing           |    1471 | residential
 Bulevardul Beijing           |    1470 | residential
 Bulevardul Beijing           |    1469 | residential
 Bulevardul Beijing           |    1468 | residential
 Bulevardul Beijing           |    1467 | residential
 Piața Charles de Gaulle      |     859 | residential
 Piața Charles de Gaulle      |     858 | residential
 Piața Charles de Gaulle      |    8918 | residential
 Piața Charles de Gaulle      |    8917 | residential
 Piața Charles de Gaulle      |    8916 | residential
 Piața Charles de Gaulle      |    8915 | residential
 Piața Charles de Gaulle      |    8914 | residential
 Piața Charles de Gaulle      |    8913 | residential
 Piața Charles de Gaulle      |    8912 | residential
 Calea Dorobanților           |     264 | primary
                              |    9999 | footway
 Calea Dorobanților           |    8116 | primary
 Calea Dorobanților           |    8117 | primary
 Calea Dorobanților           |    8118 | primary
 Calea Dorobanților           |    8119 | primary
 Strada Emil Pangrati         |    3489 | residential
 Strada Emil Pangrati         |    3412 | residential
 Strada Emil Pangrati         |    3413 | residential
 Strada Emil Pangrati         |    3414 | residential
 Cpt. Av. Demetriade Gheorghe |    3446 | residential
 Strada Andrei Muresanu       |    8086 | residential
(49 rows)

Time: 77.807 ms

If you render the above (I used QGIS), the (start of) the route looks like this:

Note how pgRouting has found multiple edges (2016, 2015, 2014, 2013) alongside "Strada Borșa"; that's because we split the street using the PLPGSQL procedure described at the beginning of the post:


osm=# select osm_id, name from network where gid in (2016, 2015, 2014, 2013);
  osm_id  |     name     
----------+--------------
 23730998 | Strada Borșa
 23730998 | Strada Borșa
 23730998 | Strada Borșa
 23730998 | Strada Borșa
(4 rows)

Installing the Fast SQL Layer plugin on Mac OS X 10.7

2011-10-04T10:02:00.004+03:00

If you're running QGis version 1.7 (trunk) from the KyngChaos site and you want to install the Fast SQL Layer plugin, you need to solve some dependencies first.

The paths below may not be the same on your machine if you've not installed GEOS and PROJ4 from the KyngChaos packages.


sudo easy_install-2.6 Pygments
sudo C_INCLUDE_PATH=/usr/local/pgsql/include/ easy_install-2.6  psycopg2
sudo LIBRARY_PATH=/Library/Frameworks/GEOS.framework/Versions/3/unix/lib/:/Library/Frameworks/PROJ.framework/Versions/4/unix/lib/ C_INCLUDE_PATH=/Library/Frameworks/GEOS.framework/Versions/3/unix/include/:/Library/Frameworks/PROJ.framework/Versions/4/Headers/ easy_install-2.6  pyspatialite

If the installation has been successful, you should not get an error when importing the modules, i.e.:


cristi:~ diciu$ python2.6
Python 2.6.6 (r266:84292, Jun 16 2011, 16:59:16) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygments, psycopg2, pyspatialite
>>>

Once you've installed the dependencies, you can go on installing the plugin using the QGIS GUI.

Using pgRouting to compute routes on OSM hiking trails

2011-09-01T14:14:00.012+03:00

I've experimented with using pgRouting's shortest_path function to compute routes using the hiking trails defined in the OpenStreetMap data set.

Building the topology

In the OSM model as created by osm2pgsql, the relations table models a hiking trail based on a set of members (each member a LINESTRING geometry from the planet_osm_line table).

The problem is you cannot build a topology based solely on the start and end points of each trail segment, because more often then not, trail segment intersect mid-way:

So I used the relations table to build the topology where each trail segment is split based on the POINTS that are part of its LINESTRING geometry. This means that every point from the trail segment becomes a node in our topology (this is very inefficient, but I could not find a faster way of not missing mid-trail intersections).

I've built the topology using the following SQL code:


drop table if exists trail_network;

create table trail_network(gid serial, osm_id integer, name varchar, symbol varchar, the_geom geometry, source integer, target integer, length float);



CREATE OR REPLACE FUNCTION compute_trail_network() RETURNS text as $$

DECLARE

relationRecord record;

partIndex integer;

lineRecord record;

wayRecord record;

geomFragment record;

BEGIN

	FOR relationRecord in select id, parts, tags from planet_osm_rels where tags @> '{osmc:symbol}' and tags @> '{hiking}' LOOP

		FOR partIndex in array_lower(relationRecord.parts, 1)..array_upper(relationRecord.parts, 1) LOOP

			SELECT name, highway, way from planet_osm_line where osm_id = relationRecord.parts[partIndex] into lineRecord;

			SELECT * from planet_osm_ways where id = relationRecord.parts[partIndex] into wayRecord;

			FOR pointIndex in 1..ST_NPoints(lineRecord.way)-1 LOOP

				select st_makeline(st_pointn(lineRecord.way, pointIndex), st_pointn(lineRecord.way, pointIndex+1)) as way into geomFragment;

				INSERT into trail_network(osm_id, name, symbol, the_geom, length) values(relationRecord.parts[partIndex], substring(array_to_string(relationRecord.tags, '#'), '#name#([^#]*)#?$'), substring(array_to_string(relationRecord.tags, '#'), '(red_cross|blue_cross|red_triangle|blue_triangle|red_dot|blue_dot|red_stripe|blue_stripe|yellow_stripe|yellow_triangle|yellow_cross)'), geomFragment.way, st_length(geomFragment.way));

			END LOOP;

		END LOOP;

	END LOOP;

return 'Done';

END;

$$ LANGUAGE 'plpgsql';



select * from compute_trail_network();

Once the topology has been created, I called assign_vertex_id() to compute source and target values (for some reason, probe_geometry_columns does not work on my machine so I had to fill in my topology table's geometry column data by hand):


insert into geometry_columns(f_table_catalog, f_table_schema, f_table_name, f_geometry_column, coord_dimension, srid, type) values('', 'public', 'trail_network', 'the_geom', 2, 900913, 'LINESTRING');



SELECT assign_vertex_id('trail_network', 2, 'the_geom', 'gid');

Computing a route

The first thing required is finding topology nodes close to the POI's that represent departure and arrival points.
In this example, we will compute a route between Cabana Piscul Negru and Negoiu:


osm=# select t.gid, t.source, t.target, t.osm_id, t.name, st_distance(p.way, t.the_geom) from trail_network t, planet_osm_point p where st_buffer(p.way, 100) && t.the_geom and p.name='Cabana Piscul Negru' order by st_distance(p.way, t.the_geom) asc limit 5;

  gid  | source | target |  osm_id  |              name              |   st_distance    

-------+--------+--------+----------+--------------------------------+------------------

 41354 |  17684 |  17685 | 73964460 | Piscu Negru - Refugiul Călțun  | 94.9336094118476

 41532 |  17684 |  17685 | 73964460 | Piscul Negru - Șaua Paltinului | 94.9336094118476

(2 rows)

there are two edges in close proximity to Piscul Negru (94 meters away). They share source and target nodes because they are in fact over a segment shared between the two different trails.

We repeat the query to find out the node for Negoiu:


osm=# select t.gid, t.source, t.target, t.osm_id, t.name, st_distance(p.way, t.the_geom) from trail_network t, planet_osm_point p where st_buffer(p.way, 100) && t.the_geom and p.name='Negoiu' order by st_distance(p.way, t.the_geom) asc limit 5;

  gid  | source | target |  osm_id  |      name      |   st_distance    

-------+--------+--------+----------+----------------+------------------

 42688 |  18832 |  18833 | 27182093 | Traseu creasta | 2.03276684697733

 42687 |  18831 |  18832 | 27182093 | Traseu creasta | 2.18879960182101

 42686 |  18830 |  18831 | 27182093 | Traseu creasta | 13.9052808592012

 42685 |  18829 |  18830 | 27182093 | Traseu creasta | 35.2142772300472

 42684 |  18828 |  18829 | 27182093 | Traseu creasta |  63.933925517158

(5 rows)

Now that we've got the start (17684) and end (18832) nodes, we can ask pgRouting to compute the route:


osm=# select tn.name, p.edge_id, tn.symbol from trail_network tn, (select edge_id from shortest_path('

osm'# SELECT gid as id,

osm'# source,

osm'# target,

osm'# length as cost

osm'# FROM trail_network',

osm(# 17684, 18832,false,false)) p

osm-# where p.edge_id = tn.gid;

                name                 | edge_id |    symbol     

-------------------------------------+---------+---------------

 Piscu Negru - Refugiul Călțun       |   41343 | blue_triangle

 Piscu Negru - Refugiul Călțun       |   41341 | blue_triangle

 Piscu Negru - Refugiul Călțun       |   41338 | blue_triangle

 Piscu Negru - Refugiul Călțun       |   41335 | blue_triangle

[..]

If we retrieve a summary of the involved trails, we get:


osm=# select max(tn.name), max(tn.symbol) from trail_network tn, (select edge_id from shortest_path('

osm'# SELECT gid as id,

osm'# source,

osm'# target,

osm'# length as cost

osm'# FROM trail_network',

osm(# 17684, 18832,false,false)) p

osm-# where p.edge_id = tn.gid group by tn.name;

                 max                 |      max      

-------------------------------------+---------------

 Piscu Negru - Refugiul Călțun       | blue_triangle

 Lacul Călțun - Tunel Transfăgărășan | blue_cross

 Traseu creasta                      | red_stripe

(3 rows)

While the choice of the blue cross trail might look weird, it happens because in this particular section, the blue cross trail shares the same path as the blue triangle trail. From the routing engine's point of view, they are perfectly equivalent but a human might argue that the correct trail sequencing would be "Piscu Negru - Refugiul Caltun" following the blue triangle and then the ridge trail following the red stripe up to Negoiu.

You can also use the driving distance functions from pgRouting.
This is a colormap showing a classification of reachable places starting from Cabana Piscul Negru:

Piatra Craiului mashup

2011-08-19T19:08:00.006+03:00

A mashup of the OpenStreetMap hiking trails displayed in Google Earth:

I've initially experimented with displaying the actual OSM tiles over Google Earth (that automatically maps them using the terrain's altitude profile) but I obtained mixed results - for one, the tiles overlapped the satellite imagery displayed by GE and, most annoyingly, I didn't manage to find a way to correctly match GE's bounding box request to OSM's discrete zoom levels.
If you're interested in displaying the OSM tiles over GE, check out Markus Bader's OpenStreetMapLayer.kmz

I've instead displayed the vector version of the trails (linestrings and icons) using a Python script that extracts data from my local OSM Postgresql database. The script runs as a CGI and exports a bounding box worth of data to Google Earth using the KML NetworkLink feature.

Hacky code sample follows (builds the KML Placemarks for the OSM tourism points of interest):


def poisFromBBOX(east,west,south,north):

        result = ""

        conn = psycopg2.connect("dbname=...")

        cur = conn.cursor()

        stmt = cur.mogrify('select name,tourism, st_astext(transform(way, 4326)) from planet_osm_point where tourism is not null and st_contains(st_transform(ST_SetSRID(ST_MakeBox2D(ST_Point(%s, %s), ST_Point(%s, %s)), 4326), 900913), way)', (east, south, west, north))

        cur.execute(stmt)

        tl = cur.fetchall()

        for item in tl:

                name = item[0]               

                if name == None:                        

                    continue

                tourism = item[1]                

                point = item[2]                

                m = re.search('(?<=POINT\()([0-9.]+) ([0-9.]+)', str(point))

                result = result + '<Placemark><styleUrl>#myStyleMap</styleUrl><name>' + escape(str(name))  + '<description>OSM tourism: ' + escape(str(tourism)) + '</description><Point><coordinates>' + str(m.group(1)) + ',' + str(m.group(2)) + ',0</coordinates></Point></Placemark>'

        cur.close()

        conn.close()

        return result

Symbol reference in Piatra Craiului

2011-08-16T15:52:00.007+03:00

Around Refugiul Diana

Valea Urșilor - Refugiul Diana (2h from the road, 2.5 h from Plaiul Foii)

Padina Popii (1.5-2h)

Brâna Caprelor (2h to Șaua Padinei Închise)

La Zaplaz, on the trail leading to La Om

Plaiul Foii - Refugiul Șpirla - La Zaplaz - La Lanțuri - Vârful La Om (5-6 h)

La Om peak

Around Plaiul Foii

Plaiul Foii - Vârful Tămașul Mare Note: this trail is not marked all the way to Tămașul Mare - in my survey August 2011 the yellow stripe marking disappears in a big heath.

Trail leading to Făgăraș (leaves Plaiul Foii by following the road on the bank of Bârsa Groșetului)

Java Topology Suite

2011-07-28T20:21:00.004+03:00

If you're looking for a way to visually check geometries, their validity, intersections and so on, the Java Topology Suite might be a very good idea.

It runs out of the box on Mac OS X (checked on 10.6.8) but you do have to run testbuilder.sh yourself (from the bin directory of the JTS distribution).

You can paste geometries out of postgis directly (by using st_asekwt and stripping out the SRID definition).
Just as postgis, it conforms to the Simple Features Specification for SQL published by the Open GIS Consortium so overlapping, crossing, and so on have the same meanings.

Cling, the interactive C++ interpreter

2011-07-28T10:37:00.004+03:00

Some folks from CERN and Fermilab are working on cling, a C++ interpreter based on llvm and clang.

It works as advertised (you have to build it as part of the llvm tree, near clang) and it's amazingly cool:



llvm diciu$ ./Release+Asserts/bin/cling 

**** Welcome to the cling prototype! ****
* Type C code and press enter to run it *
* Type .q, exit or ctrl+D to quit       *
*****************************************
[cling]$ FILE * f = fopen("/dev/random", "r");
[cling]$ unsigned char rnd;
[cling]$ int i=0;
[cling]$ while(++i<10) { fread(&rnd, sizeof(rnd), 1, f); printf("%03d\n", rnd); }
201
161
186
086
056
092
125
063
203
[cling]$ .q

Red Bull X-Alps 2011

2011-07-22T21:07:00.002+03:00

Unbelievable effort by everyone involved in the X-Alps competition.
In a single day today Christian Maurer flew some 350 Km leaving Romanian Toma Coconea 150 Km behind.

OSMC symbol reference for Piatra Craiului around Șaua Padinei Închise (surveyed July 2011)

2011-07-16T22:50:00.009+03:00

Updates on final section of the southern blue stripe trail:

Curmătura-Șaua Padinei Închise

I've added relations for a couple of trails (both linking Refugiul Diana to Vârful Turnu) on the northern side based on existing OSM ways:

Brâna Caprelor
Padina Popii

Șaua Padinei Închise:

Vârful Turnu:

Descent through Padina Șindrilăriei:

OSMC symbol reference for Piatra Craiului around Cabana Curmătura (surveyed June 2011)

2011-06-18T21:48:00.004+03:00

Signs towards the east:

Signs towards the west:

Intersection where the red cross trail leaves the track going through Prăpăstiile Zărneștilor:

Part of the blue dot trail leading from Șaua Crăpăturii towards Piatra Craiului Mică peak:

Hiking trails in Iezer

2011-05-30T07:48:00.002+03:00

Signpost where the blue dot trail leaves the blue cross trail:

Signpost at convergence of blue dot and blue cross trails, very close to Iezer shelter:

Sign at the Iezer shelter, indicating distance to Crucea Ateneului (located by the ridge trail):

Sign on the main ridge trail (marked with a red stripe -> left(south east) towards Cabana Voina by Văcarea, right (north west) towards Iezer and Roșu peaks):

View of Iezer Lake and shelter from Crucea Ateneului:

Views from the shelter:

Updated trails:

Cabana Voina - Lacul și Refugiul Iezer (prin jepi)

Voina - Cabana Cuca - Vârful Păpușa

Voina - Culmea Văcarea - Crucea Ateneului - Vârful Iezer

Cabana Voina - Lacul și Refugiul Iezer prin stâna din Iezer

Updates on hiking trails in Cozia

2011-05-15T07:50:00.006+03:00

Căciulata - Valea Păușa - Mănăstirea Stânișoara

Căciulata - Satul Păușa - Șaua "La Troiță" - Mănăstirea Stânișoara - Muchia Vlădesei - Muntele Durduc - Cabana Cozia (incomplete)

Poiana Stânișoara - Cascada Gardului

Trail signs at Mănăstirea Stânișoara:

Cascada Gardului

Mălaia - Vârful Zmeureț

2011-05-10T07:59:00.009+03:00

A 4 hour trail in Munții Căpătânii, demanding because of the altitude gain (you climb from 500 m to 1900 m).

The trail is fairly well marked, but has not been maintained recently (surveyed May 2011) so there are some areas that have been covered by fallen trees where the trail is easily lost.

I've added the trail relation in openstreetmap, see here: Mălaia - Vârful Zmeureț

Using TesseractGUI with other languages (non English)

2011-05-05T15:53:00.004+03:00

It is possible to use TesseractGUI 0.3 to do OCR on other languages but you have to download the language pack for the required language and install it as part of the TesseractGUI application bundle.

E.g. for German:

1/ Download the German training data (e.g. http://code.google.com/p/tesseract-ocr/downloads/detail?name=deu.traineddata.gz&can=2&q=)

2/ The archive will expand into a file called deu.traineddata

3/ Copy the deu.traineddata inside the TesseractGUI application bundle, under Contents/Resources/tessdata/

4/ Configure TesseractGUI to use the language pack (Set Preferences->Custom language spec to read deu)

5/ Load a TIFF file containing a scan of something written in German.

That should be it.

Updates on hiking trails in Piatra Craiului (surveyed Apr 2011)

2011-04-26T07:29:00.011+03:00

Hiking relations for:

Zărnești - Prăpăstiile Zărneștiului - Curmătura (incomplete)

Zărnești - Prăpăstiile Zărneștiului - Cheile Pisicii - La Table - Moeciu (incomplete)

Zărnești - Botorog - Zănoaga - Curmătura

Intersection near Fântâna lui Botorog where Curmătura trail leaves the road:

From the trail towards cabana Curmătura a trail marked with a blue dot opens up towards Piatra Mică peak:

Poiana Zănoaga - Vârful Piatra Mică - Șaua Crăpăturii

On the trail to Piatra Mică there's a beautiful vantage point (called "Crucea din Carpați and marked by a couple of crosses) offering spectacular views of the surroundings:

Trail reference:

Update on hiking trails in Predeal area

2011-03-28T11:53:00.005+03:00

Updates on:

Cabana Cioplea - Cabana Susai

Cabana Cioplea - Stâna Pietricica (prin valea Timișului Sec)

Cabana Cioplea - Cabana Susai - Cabana Piatra Mare

Hiking relation for Clăbucet Plecare - Cabana Susai:

Updates on: Clăbucet Sosire - Cabana Gârbova - Azuga

Work in progress

2011-03-10T12:04:00.004+02:00

Proiect in lucru: web site cu OpenLayers/ExtJS pentru afisare trasee montane din OpenStreetMap.

OSMC symbol reference for Diham/Bușteni

2011-02-12T08:00:00.008+02:00

This is a set of photographs of hiking symbols in Busteni - surveyed in February 2011.

Description of red stripe trail on a tree on the Gura Diham - Cabana Diham trail:

Trail intersection in Șaua Baiului:

Trail intersection at Cabana Diham:

Trail intersection at Cabana Poiana Izvoarelor:

Mountain Hiking tracks in Piatra Mare. Timișu de Sus - Piatra Mare

2011-01-12T11:43:00.013+02:00

The fastest way to reach Piatra Mare Peak (1844 m) is to start from Timișu de Sus and continue upwards by Cascada Tamina (Tamina waterfall). It's a 3.5 - 4 hour hike up to the peak and 2.5 - 3 hours back.

If you're driving on DN1 from Predeal to Brașov, you can leave your car in the parking lot on the right of the road, just after you've exited Timișu de Sus. The trail opens on the right hand side with a small bridge leading to a forestry road marked with a blue stripe.

The blue stripe trail leads to Cascada Tamina. The waterfall is one hour away, and you can hike through it (there are steel stairs installed inside the canyon formed by the river).

To hike through the waterfall leave the trail when you see the waterfall sign, descend on the right side until you reach the bottom of the waterfall and then climb your way up using the steel stairs. Once you've exited the waterfall, climb back to where you've started from and re-enter the trail towards Piatra Mare.

The trail continues through the forest, intersecting the forestry road you've started from, until you hit the alpine region at Stâna Pietricica.

The trail then follows the red cross mark () and then the red stripe () towards the peak.

Google Maps version of the trail above: Timișu de Sus - Piatra Mare Map
The KML is loadable in Google Earth and the visualization is pretty spectacular:

sprintf_chk in Mac OS X

2010-12-16T14:48:00.004+02:00

Under Snow Leopard, sprintf calls are automatically replaced by calls to sprintf_chk, a functionally equivalent function that performs bounds checking on the sprintf argument.

This means the following program (that deliberately overwrites the end of t):


#include <stdio.h>
int main()
{
 char t[1] = {0x41};
 char v[1] = {0x42};

 sprintf(t, "%d%d", 0x43, 0x44);
}

produces assembler code where the sprintf call is replaced to a sprintf_chk call:

(gdb) disass
Dump of assembler code for function main:
0x0000000100000eec <main+0>: push   %rbp
0x0000000100000eed <main+1>: mov    %rsp,%rbp
0x0000000100000ef0 <main+4>: sub    $0x10,%rsp
0x0000000100000ef4 <main+8>: movb   $0x41,-0x1(%rbp)
0x0000000100000ef8 <main+12>: movb   $0x42,-0x2(%rbp)
0x0000000100000efc <main+16>: lea    -0x1(%rbp),%rdi        # buffer argument
0x0000000100000f00 <main+20>: mov    $0x44,%r9d             # second va_arg argument
0x0000000100000f06 <main+26>: mov    $0x43,%r8d             # first va_arg argument 
0x0000000100000f0c <main+32>: lea    0x47(%rip),%rcx        # format string ("%d%d")
0x0000000100000f13 <main+39>: mov    $0x1,%edx              # storage size as detected by the compiler       
0x0000000100000f18 <main+44>: mov    $0x0,%esi              # flags argument (0)
0x0000000100000f1d <main+49>: mov    $0x0,%eax              # MMS registers are not used to pass va_args
0x0000000100000f22 <main+54>: callq  0x100000f2a <dyld_stub___sprintf_chk>
0x0000000100000f27 <main+59>: leaveq 
0x0000000100000f28 <main+60>: retq   
End of assembler dump.

sprintf_chk has the following prototype:

int __sprintf_chk (char *s, int flags, size_t len, const char *format, ...)

that implies that the compiler is aware of the storage size of the destination buffer used by sprintf.

If the compiler knows the storage size of the destination buffer it uses it in the sprintf_chk call, which in turn calls vsnprintf and checks that the number of characters written in the buffer is less or equal to the size of the buffer (if this is not the case, it aborts).

sprintf_chk will not work with buffers that have been allocated on the heap (it will be called with length set to -1).

Compressing Mac OS X executables using UPX

2010-12-03T09:41:00.004+02:00

UPX stands for Ultimate Packer for eXecutables and you can use it to compress many executable formats including Mac OS X fat binaries. UPX will create a new version of your executable that will self extract itself at runtime.

Things to remember when considering production code

UPX changes your executable. otool no longer produces the output you would expect and running the executable under GDB is completely un-symbolized (although you can get around this by attaching to the binary after it's been uncompressed in memory).

The crash reports created by Mac OS X on executables produced by UPX are almost empty - they only contain the registry dump because Mac OS X is unable to resolve the backtraces on the application threads.

Compiling UCL

UPX depends on UCL (portable lossless data compression library). Download the latest UCL version (ucl-1.03.tar.gz) from http://www.oberhumer.com/opensource/ucl/download/ and untar it.

Compile UCL using the usual ./configure, make sequence.

Compiling UPX

Download the source package from upx.sourceforge.net.

To compile upx, you need to tell it where the UCL library is stored (unless you're installed it).

cristi:upx-3.07-src diciu$ cd src/
export UPX_UCLDIR=/Users/diciu/Downloads/ucl-1.03
make

Once its compiled you can use it to compress any executable:


cristi:src diciu$ ./upx.out  /tmp/LoudHush 
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2010
UPX 3.07        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 08th 2010

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    584560 ->    216576   37.05%    Mach/fat     LoudHush                      

Packed 1 file.

PS. If you get a NotCompressibleException it may be that your executable is too small and cannot be compressed (the size of the compressed binary exceeds the size of the original binary).

Looking at memset's implementation

2010-11-28T15:17:00.005+02:00

Not all LibC functions are written in C. Some are written in assembler because they are optimized to take in consideration the data size and alignment of LibC call arguments. This post describes memset based on GDB's disassembly.

void * memset(void *b, int c, size_t len);

Following the 64 bit ABI, the parameters to the function are set as follows:

b is the destination ($rdi)
c is the character to fill the destination with ($rsi) (my example uses 0x38 - character '8')
len is the length ($rdx)

The first thing memset does is to check if the character argument is 0 (logical AND with 0xFF)
If it is, the memset call is equivalent to a bzero call and the code jumps to the bzero implementation (0x7fffffe00600).

0x00007fff8749f414 <memset+0>: and    $0xff,%esi
0x00007fff8749f41a <memset+6>: jne    0x7fff8749f42b <memset+23>

RSI is zero - this memset call is equivalent to a bzero call (0x7fffffe00600).

0x00007fff8749f41c <memset+8>: mov    $0x7fffffe00600,%rax
0x00007fff8749f426 <memset+18>: mov    %rdx,%rsi
0x00007fff8749f429 <memset+21>: jmpq   *%rax

RSI is non-zero.
Save the destination address to R8 (it will be used later on as return value of memset.

0x00007fff8749f42b <memset+23>: mov    %rdi,%r8

The following section fills each byte of EAX with the character argument (0x38383838)

0x00007fff8749f42e <memset+26>: mov    %esi,%eax
0x00007fff8749f430 <memset+28>: shl    $0x8,%esi
0x00007fff8749f433 <memset+31>: or     %esi,%eax
0x00007fff8749f435 <memset+33>: mov    %eax,%esi
0x00007fff8749f437 <memset+35>: shl    $0x10,%esi
0x00007fff8749f43a <memset+38>: or     %esi,%eax

Check if the size of the fill is bigger then 256 bytes.

0x00007fff8749f43c <memset+40>: cmp    $0xff,%rdx
0x00007fff8749f443 <memset+47>: ja     0x7fff8749f490 <memset+124>

Check if the size of the fill is bigger then 16 bytes.

0x00007fff8749f445 <memset+49>: cmp    $0x10,%edx
0x00007fff8749f448 <memset+52>: jge    0x7fff8749f464 <memset+80>

Check if the size of the fill is zero.

0x00007fff8749f44a <memset+54>: test   %edx,%edx
0x00007fff8749f44c <memset+56>: je     0x7fff8749f48c <memset+120>

Size of fill is less then 16 bytes but not zero.
Fill is the destination buffer (RDI is pointing to it) one byte at a time, decrementing RDX at each step.
Continue until RDX is zero, and jump at function end.

0x00007fff8749f44e <memset+58>: mov    %al,(%rdi)
0x00007fff8749f450 <memset+60>: add    $0x1,%rdi
0x00007fff8749f454 <memset+64>: sub    $0x1,%edx
0x00007fff8749f457 <memset+67>: jne    0x7fff8749f44e <memset+58>
0x00007fff8749f459 <memset+69>: jmp    0x7fff8749f48c <memset+120>

0x00007fff8749f45b <memset+71>: mov    %al,(%rdi)
0x00007fff8749f45d <memset+73>: add    $0x1,%rdi
0x00007fff8749f461 <memset+77>: sub    $0x1,%edx

Size of fill is more then 16 bytes.
If the destination is not 4 byte aligned, fill in the first bytes until the destination is aligned.

0x00007fff8749f464 <memset+80>: test   $0x3,%edi
0x00007fff8749f46a <memset+86>: jne    0x7fff8749f45b <memset+71>

The destination is now aligned, so it can be filled in 4 bytes at a time.

0x00007fff8749f46c <memset+88>: mov    %edx,%ecx
0x00007fff8749f46e <memset+90>: shr    $0x2,%edx
0x00007fff8749f471 <memset+93>: mov    %eax,(%rdi)
0x00007fff8749f473 <memset+95>: add    $0x4,%rdi
0x00007fff8749f477 <memset+99>: sub    $0x1,%edx
0x00007fff8749f47a <memset+102>: jne    0x7fff8749f471 <memset+93>

If the size was also a multiple of 4, we're done.

0x00007fff8749f47c <memset+104>: and    $0x3,%ecx
0x00007fff8749f47f <memset+107>: je     0x7fff8749f48c <memset+120>

If the size was not a multiple of 4, there are some leftovers to copy.

0x00007fff8749f481 <memset+109>: mov    %al,(%rdi)
0x00007fff8749f483 <memset+111>: add    $0x1,%rdi
0x00007fff8749f487 <memset+115>: sub    $0x1,%ecx
0x00007fff8749f48a <memset+118>: jne    0x7fff8749f481 <memset+109>

Load RAX with the value of R8 (pointer to fill buffer saved at beginning) and return.

0x00007fff8749f48c <memset+120>: mov    %r8,%rax
0x00007fff8749f48f <memset+123>: retq

A fill operation with more then 256 bytes.
This is done using MMS instructions. xmm0 is set to 16 bytes of 0x38 (our character)

0x00007fff8749f490 <memset+124>: movd   %eax,%xmm0
0x00007fff8749f494 <memset+128>: pshufd $0x0,%xmm0,%xmm0

Check for alignment of destination.

0x00007fff8749f499 <memset+133>: mov    %rdi,%rcx
0x00007fff8749f49c <memset+136>: neg    %ecx
0x00007fff8749f49e <memset+138>: and    $0xf,%ecx
0x00007fff8749f4a1 <memset+141>: je     0x7fff8749f4b1 <memset+157>

The destination is not 16 byte aligned, copy some bytes until rest of destination will be aligned.

0x00007fff8749f4a3 <memset+143>: sub    %rcx,%rdx
0x00007fff8749f4a6 <memset+146>: mov    %al,(%rdi)
0x00007fff8749f4a8 <memset+148>: add    $0x1,%rdi
0x00007fff8749f4ac <memset+152>: sub    $0x1,%ecx
0x00007fff8749f4af <memset+155>: jne    0x7fff8749f4a6 <memset+146>

Call _memset_pattern.

0x00007fff8749f4b1 <memset+157>: mov    $0x7fffffe01000,%rax
0x00007fff8749f4bb <memset+167>: callq  *%rax

Load RAX with the value of R8 (pointer to fill buffer saved at beginning).

0x00007fff8749f4bd <memset+169>: mov    %r8,%rax
0x00007fff8749f4c0 <memset+172>: retq

Calling _printf from 64bit code under Mac OS X

2010-11-23T11:16:00.003+02:00

If you're trying to figure out why your printf call sometimes crashes, check that you're zeroing out %rax before the printf call (unless you're actually printf-ing floating point numbers).

Under the 64bit ABI %rax is described as: temporary register; with variable arguments passes information about the number of vector registers used; 1st return register. The vector registers are xmm0-xmm7 and they are used when passing floating point arguments.


#============== asm.s  ==============#
.data
msg: .ascii "Hello World\0"
fmt1: .ascii "%s\n\0"
fmt2: .ascii "This is pi:%.10f\n\0"
pi: .quad 0x400921FB53C8D4F1

.text
.globl _dis
_dis:
 sub $8, %rsp

 movq $0, %rax
 movq fmt1@GOTPCREL(%rip), %rdi
 movq msg@GOTPCREL(%rip), %rsi
 call _printf

 movsd pi(%rip), %xmm0
 movq $1, %rax
 movq fmt2@GOTPCREL(%rip), %rdi
 call _printf
 

 add $8, %rsp
 ret


#============ mainasm.c  ============#
extern void dis();
int main()
{
 dis();
}


cristi:tmp diciu$ as -arch x86_64 asm.s -o asm.o
cristi:tmp diciu$ gcc mainasm.c asm.o
cristi:tmp diciu$ ./a.out
Hello World
This is pi:3.1415926500

If you're looking for a way to get the hex representation of PI (0x400921FB53C8D4F1), this is what I used: IEEE-754 Floating-Point Conversion.

Debugging session summary

2010-11-22T12:31:00.003+02:00

Hello world in 64bit, Mac OS X

2010-11-18T10:10:00.005+02:00

System call numbers in 64 bit are different when compared to the 32 bit versions. They are obtained by adding the 32 bit system call number to 0x2000000 (see SYSCALL_CONSTRUCT_UNIX in syscall_hw.h).

Other 64 bit particularities are instruction pointer (%rip) relative data references.

If you try to assemble:


.data    
msg:
 .ascii "Hello, world!\n"
len:
 .long . - msg

.text  # text section
.globl _start # declare _start as global symbol for the linker to find

_start:
 movq len, %rdx  # length of string

you get:


cristi:64bit diciu$ as -arch x86_64 rdf.s -o rdf.o
rdf.s:14:32-bit absolute addressing is not supported for x86-64
rdf.s:14:cannot do signed 4 byte relocation

This is explained in the x86-64 Code Model:

All local and small data is accessed directly using addressing that’s relative to the instruction pointer (RIP-relative addressing). All large or possibly nonlocal data is accessed indirectly through a global offset table (GOT) entry. The GOT entry is accessed directly using RIP-relative addressing.

Our hello world assembler program becomes:


============= hello64bit.s ===================
#as -arch x86_64 hello64bit.s -o hello64bit.o
#ld -e _start -arch x86_64 hello64bit.o -o hello64bit

.data  # data section
msg:
 .ascii "Hello, world!\n" # ascii string to be printed
len:
 .long . - msg  # string length

.text  # text section
.globl _start # declare _start as global symbol for the linker to find

_start:
 movq $0x2000004, %rax   # write call (see SYSCALL_CONSTRUCT_UNIX)
 movq $1, %rdi   # file descriptior (stdout)
 movq msg@GOTPCREL(%rip), %rsi # string to print
 movq len(%rip), %rdx  # length of string
 syscall    # call write

 movq $0x2000001, %rax  # exit call
 movq $0, %rdi   # return code
 syscall    # call exit

Panorama Stâna Pietricica (Piatra Mare)

2010-11-15T08:58:00.004+02:00

Charpy - a helper application for diacritical marks

2010-10-29T10:55:00.005+03:00

I've built a small (around 70 lines of code thanks to PyObjC) helper application that shows UTF-8 encodings and the Unicode description for a character.

You can download it here: charpy.zip
MD5 sum: a4f7d56ac7342b4ab0ec403c0b4c4b43.

psql with diacritical marks on Mac OS X 10.6.4

2010-10-28T10:05:00.002+03:00

If you're typing diacritical marks (ăâșt) into the psql command line on Mac OS X 10.6.4 and getting question marks in place of the expected marks, you're facing a problem in the libedit library used by psql for enhanced command line editing.

You can tell psql to stop using the libedit library, using a command line option:


Input and output options:
[..]
  -n, --no-readline        disable enhanced command line editing (readline)

The disadvantage is that psql will loose all enhanced editing features (such as going to the begging of the line by typing Ctrl+A). To get around this problem, I usually run two psql sessions, one without read line capabilities (the "Romanian session") and one with enhanced editing features (the "normal session").

The 10.6.4 libedit library also causes crashes when using TAB completion on some object names. Reference here: archives.postgresql.org.

Point that is part of way does not intersect way

2010-09-28T09:03:00.007+03:00

If you're wondering why a point that is obviously part of a way does not intersect the way, it's because coordinates are stored differently in planet_osm_point vs. planet_osm_line.

Case in point, a subway station drawn as a node on a subway railway is reported by postgis as not intersecting the railway.

The apparent cause is that, when saving coordinates in planet_osm_line, osm2pgsql rounds them to two decimal places (I've only checked for SRID 900913):


openmapdb=# select st_asewkt(way) from planet_osm_point where osm_id = 293317918;
 SRID=900913;POINT(2905477.18172046 5539892.28016006)
(1 row)


openmapdb=# select st_asewkt(way) from planet_osm_line where osm_id = 45736605;
SRID=900913;LINESTRING([..]2905477.18 5539892.28

The side effect of this rounding is that st_intersects applied to the point and the line will return false because the distance difference caused by the rounding is larger then the margin used by the intersection function.

Mountain hiking tracks in Făgăraș. Cabana Bâlea - Vânătoarea lui Buteanu (part I)

2010-09-15T12:14:00.016+03:00

With few exceptions (Ceahlău's excellent national park office comes to mind) I am always frustrated with the quality of signage in Romania's mountain trekking areas.

I usually document my trips by searching on the Internet but sometimes it's hard to find trail descriptions at the level of detail I require. It may be that the writer assumes I carry a tent and camp out for the night, or he proposes start points that are un-drivable.

This post, and future follow-ups will describe trails that can be walked within a summer's daylight span (9 hours at most).

Disclaimer
While I've worked on collecting some of the data used in these maps and I've performed the rendering myself, I cannot guarantee for their correctness or accuracy. When going up for a hike, make sure to take decent maps and appropriate gear. Around Bâlea, most of the alpine huts sell good paper maps.

Cabana Bâlea - Vânătoarea lui Buteanu

Cabana Bâlea - Vânătoarea lui Buteanu is probably the easiest, but still very rewarding trail when starting from Bâlea. The trail is not very long but it is somewhat steep (500m altitude gain).

Map of the trails to the west of Bâlea Lake (click here for the standard OSM map):

Symbol	Trail name	Duration estimate
	Cabana Bâlea - Șaua Caprei	45 min - 1 h
	Șaua Caprei - Vânătoarea lui Buteanu (2507m)	45 min

Take the blue triangle trail to Șaua Caprei. Once you reach Șaua Caprei, you can walk around Capra Lake (~2200m), a beautiful glacial lake:

From Șaua Caprei, you can continue on the blue cross trail to Vânătoarea lui Buteanu (2507 m), the easiest big peak to reach in Făgăraș when starting from Bâlea, offering spectacular views of the surrounding mountains (photo shows Negoiu peak as seen from Vânătoarea lui Buteanu):

View towards Capra Lake (colleague's photo)

In the background (trapezoid shape) Viștea Mare on the left, Moldoveanu on the right (roughly 23 Km away in a straight line).
To the left of Viștea Mare and closer to the viewer, Podragu peak.

This map schematic shows the visible peaks present in the photograph above:

In 64bit sizeof(int) != sizeof(char *) and an unexpected side effect

2010-08-16T15:23:00.006+03:00

While trying to figure out how the PATH environment variable can be set under Mac OS X, I wrote a little C program to test what getenv returns.

This is how I usually write them, not really taking care to include headers, as I leave it to gcc to do the right thing:

int main()
{
        char * t = getenv("PATH");
        printf(">%s<\n", t);
}

In this case however, the binary crashed with a segmentation fault in the printf call.
Compiling on 32 bit makes the problem go away, so what is the problem?

Well, if we look at the assembler code, gcc has added an extra call (cltq) to convert the 4 byte integer result of getenv to 8 bytes (the size of char * on 64 bit):

0x0000000100000ee4 <main+8>: lea    0x67(%rip),%rdi        # 0x100000f52
0x0000000100000eeb <main+15>: mov    $0x0,%eax
0x0000000100000ef0 <main+20>: callq  0x100000f18 <dyld_stub_getenv>
0x0000000100000ef5 <main+25>: cltq   
0x0000000100000ef7 <main+27>: mov    %rax,-0x8(%rbp)
0x0000000100000efb <main+31>: mov    -0x8(%rbp),%rsi
0x0000000100000eff <main+35>: lea    0x51(%rip),%rdi        # 0x100000f57
0x0000000100000f06 <main+42>: mov    $0x0,%eax
0x0000000100000f0b <main+47>: callq  0x100000f1e <dyld_stub_printf>

This conversion clobbers the rax register, clearing the top 4 bytes - this makes the t pointer to point outside the segment, causing the segmentation fault at printf time.

Under 32bit it works as the pointer size is the same as the integer's size.

Vaguely related, I like clang's output better:

clang output

cristi:tmp diciu$ clang test.c
test.c:3:13: warning: implicit declaration of function 'getenv' is invalid in C99 [-Wimplicit-function-declaration]
        char * t = getenv("PATH");
                   ^
test.c:3:9: warning: incompatible integer to pointer conversion initializing 'char *' with an expression of type 'int'
        char * t = getenv("PATH");
               ^   ~~~~~~~~~~~~~~
test.c:4:2: warning: implicitly declaring C library function 'printf' with type 'int (char const *, ...)'
        printf(">%s<\n", t);
        ^
test.c:4:2: note: please include the header  or explicitly provide a declaration for 'printf'
3 warnings generated.

gcc output

cristi:tmp diciu$ gcc test.c
test.c: In function ‘main’:
test.c:3: warning: initialization makes pointer from integer without a cast
test.c:4: warning: incompatible implicit declaration of built-in function ‘printf’

Parks in Bucharest

2010-08-04T21:49:00.003+03:00

Parks in Bucharest, based on data from OpenStreetMap:

In numbers: 9.3 sq km out of Bucharest's 236.5 sq km.

usleep in plpgsql

2010-07-28T07:42:00.003+03:00

There is no sleep function in plpgsql under postgres 8.1, but you can create your own usleep function by using the one provided by glibc:

openmapdb=# CREATE OR REPLACE FUNCTION usleep(int) RETURNS int AS
openmapdb-# '/lib/libc.so.6', 'usleep' LANGUAGE 'C' STRICT;

In your plpgsql program, you can then call:

PERFORM usleep(100000);

Running LLDB with a self-signed certificate

2010-06-10T14:40:00.004+03:00

In a previous post I was describing running lldb using sudo to get around the need for elevated privileges as required by task_for_pid.

The obvious solution is to codesign the debugserver binary. You can use a self-signed certificate, providing that the self signed certificate is trusted for code signing in your keychain.

Using Keychain Access, run Certificate Assistant -> Create a Certificate Authority...
From the Keys category, select the newly created key and the contained certificate.
Right click the certificate and choose Get Info.
Disclose the Trust triangle and configure the Code Signing option to Always trust.

Codesign the debugserver binary:

cd ./build/BuildAndIntegration/LLDB.framework/Versions/A/Resources/
codesign -s 'Cristian Draghici' ./debugserver

Once you codesign the debug server, you should be able to run lldb as your current user.

LLDB, a new debugger on Mac OS X

2010-06-09T10:34:00.004+03:00

The LLVM team is working on a new debugger called lldb.

It compiles cleanly so you can give it a try if you have the devtools (including SVN client) installed on your machine.

Running it

You need to run the lldb-tool with the firewall disabled and as the superuser because it uses task_for_pid and in 10.6 this requires superuser privileges or code signing on the binary. Since the binary is not signed, you need to run it with superuser privileges, otherwise you will get an error like: "failed to send the qLaunchSuccess packet".

I run it this way:

sudo DYLD_FRAMEWORK_PATH=/Users/diciu/Programming/external/lldb/build/BuildAndIntegration/ ./build/BuildAndIntegration/lldb /tmp/a.out

LLDB vs GDB

LLDB	GDB
(lldb) breakpoint set -n main Breakpoint created: 1 Breakpoint by name: 'main' with 1 location; (lldb) r Launching '/private/tmp/a.out' (x86_64) (lldb) Process 50271 Stopped * thread #1: tid = 0x2c03, pc = 0x0000000100000edb, where = a.out`main + 11 at /private/tmp/test.c:14, stop reason = breakpoint 1.1, queue = com.apple.main-thread 11 12 int main() 13 { 14 -> struct i10 v = fun(); 15 }	(gdb) b main Breakpoint 1 at 0x100000edb: file test.c, line 14. (gdb) r Starting program: /private/tmp/a.out Reading symbols for shared libraries +. done Breakpoint 1, main () at test.c:14 14 struct i10 v = fun();
(lldb) disass 0x100000ed0: pushq %rbp 0x100000ed1: movq %rsp, %rbp 0x100000ed4: subq $131072, %rsp 0x100000edb: leaq -131072(%rbp), %rdi 0x100000ee2: movl $0, %eax 0x100000ee7: callq 0x100000ea0 ; fun at /private/tmp/test.c:7 0x100000eec: leave 0x100000eed: ret	(gdb) disass main Dump of assembler code for function main: 0x0000000100000ed0 : push %rbp 0x0000000100000ed1 : mov %rsp,%rbp 0x0000000100000ed4 : sub $0x20000,%rsp 0x0000000100000edb : lea -0x20000(%rbp),%rdi 0x0000000100000ee2 : mov $0x0,%eax 0x0000000100000ee7 : callq 0x100000ea0 0x0000000100000eec : leaveq 0x0000000100000eed : retq End of assembler dump.

The Missing Sync for Android 2.1 only half compatible with HTC Legend

2010-05-19T13:03:00.005+03:00

Update June 11th

I've upgraded both the Missing Sync Android client and the Mac OS X version to 1.4+ and the problem has been fixed - contacts appear correctly on the phone.

If you're looking into using The Missing Sync for Android as a sync solution for your HTC Legend, you may want to know that it only works half way:

Contacts from the Mac Address Book cannot be edited on the phone

Contacts from the Mac Address Book do not appear in the People app, under Applications

Details here: The Missing Sync for Android 2.1: HTC Sense UI (HTC Desire, Legend, Droid Eris, and Droid Incredible) Contact Sync Issues.

I have to say I am disappointed considering the price point they chose for their software (50 USD).

Processing tiff images for TessertactGUI

2010-03-31T14:06:00.006+03:00

When processing images, tesseract is very restrictive about the images it processes.
It only touches TIFF files, with the extension "tif" and uncompressed.

Unfortunately, TesseractGUI is not very straightforward about the reason it rejects files from processing (it would just say "Error reading tesseract output").

Here's how you can identify the error and fix it:

Run tesseract from the command line to find out more about the rejection cause


cristi:~ diciu$ export TESSDATA_PREFIX=/Applications/TesseractGUI.app/Contents/Resources/
cristi:~ diciu$ /Applications/TesseractGUI.app/Contents/Resources/tesseract ~/Desktop/tiffs/page1.tif /tmp/ocrtest.txt
Tesseract Open Source OCR Engine
read_tif_image:Error:Illegal image format:Compression
/Applications/TesseractGUI.app/Contents/Resources/tesseract:Error:Read of file failed:/Users/diciu/Desktop/tiffs/page1.tif
Signal_exit 31 ABORT. LocCode: 3  AbortCode: 3

the problem with this particular TIFF file is the compression.

Fixing the problem

Step 1/ Download ImageMagick

Step 2/ Identify the TIFF file we want to use:


cd /Users/diciu/Downloads/ImageMagick-6.5.8/bin
export DYLD_FALLBACK_LIBRARY_PATH=/Users/diciu/Downloads/ImageMagick-6.5.8/lib
export MAGICK_HOME=/Users/diciu/Downloads/ImageMagick-6.5.8/
cristi:bin diciu$ tiffutil -info ~/Desktop/tiffs/page1.tif 
Directory at 0x837f8
  Subfile Type: (0 = 0x0)
  Image Width: 1200 Image Length: 2088
  Resolution: 200, 200
  Resolution Unit: pixels/inch
  Bits/Sample: 8
  Compression Scheme: Lempel-Ziv & Welch encoding
  Photometric Interpretation: palette color (RGB from colormap)
  Predictor: none
  Samples/Pixel: 1
  Rows/Strip: 10
  Number of Strips: 209
  Planar Configuration: Not planar
  Color Map: (present)

Note the compression scheme (LZW).

Step 3/ Uncompress the TIFF file


tiffutil -none ~/Desktop/tiffs/page1.tif -out ~/Desktop/tiffs/page1_uncompressed.tif

Now use page1_uncompressed.tif with tesseract.

Related:

TesseractGUI

tesseract

Send HTML email from Cocoa. Take 2.

2010-03-02T15:50:00.004+02:00

This is an alternate solution to sending HTML email from Cocoa, that takes advantage of python's email capabilities.

The python email package allows a much cleaner source when writing the code for sending the HTML email:

==================== htmlemail.py ==================


from email.mime import Multipart, Text, Image
import smtplib

email_from = "sender@domain.com"
email_to = "recipient@domain.com"
msg = Multipart.MIMEMultipart('alternative')
msg['Subject'] = "HTML email"
msg['From'] = email_from
msg['To'] = email_to

text = "Plain text version"
html = "<html><p>A little bit in <b>bold</b> and some <i>italic</i></p><p><img src="cid:55665566" /></p><p>The end</p></html>"

part1 = Text.MIMEText(text, 'plain')

# the embedded image
fp = open("/Users/diciu/Desktop/test.png")

# assemble the message based on the diagram
part2 = Multipart.MIMEMultipart('related')
part21 = Text.MIMEText(html, 'html')
part22 = Image.MIMEImage(fp.read(), _subtype='PNG')
part22.add_header('Content-ID', '55665566')

msg.attach(part1)
part2.attach(part21)
part2.attach(part22)
msg.attach(part2)

s = smtplib.SMTP('some.smtp.server')
s.sendmail(email_from, email_to, msg.as_string())

s.quit()

To use the Python code from Cocoa, we use the Python framework:

==================== htmlemail.m ==================


// build with:
//      gcc htmlemail.m -framework Cocoa -framework Python
//
#import <Cocoa/Cocoa.h>
#import <Python/Python.h>

int main()
{
 NSAutoreleasePool * m = [NSAutoreleasePool new];
 NSError * err = nil;
 NSString * pycode = [NSString stringWithContentsOfFile:@"./htmlemail.py" 
    encoding:NSASCIIStringEncoding error:&err];

 Py_Initialize(); 
 PyRun_SimpleString([pycode cStringUsingEncoding:NSASCIIStringEncoding]);
 Py_Finalize();
 
 [m release];

 return 0;
}

BizBazar canceled

2010-02-28T10:26:00.004+02:00

I've never written an opinion piece on this here blog, partly because this blog is more of a technical knowledge base I keep for myself and also because I don't like blurting out my thoughts in the forever archived searchable medium that is the internet.

I will make an exception because one of the things I care deeply about is gone. I find it surprising that I'm writing I care about a TV show, but this show I'm talking about was a special show, an example of a great team working hard for what I can only describe as true journalism.

Television in Romania

I don't really know how TV is elsewhere these days, but in Romania it's in a state of suffering. News shows are based on violence, be it physical or psychological violence. The viewer is fed 30 minutes of car accidents, rapes, killings, stories on raising taxes, raising prices, corruption in politics, repeated ad nauseam.

News are written to stir up feelings, with passion rather then detachment, thus it's a common occurrence for the news anchor to pass judgment when presenting the news. A suspect is more often than not shown as guilty, stories are written in such a manner that the viewer is gratified with a speedy one minute trial of the scene unraveled before him.

If it's a car accident, the reporter will interview a couple of eye witnesses that will describe how the driver involved in the accident was doing at least 100 KPH just before the accident. It's common knowledge that a human can estimate car speeds by just looking, right? The reporter will never fail to mention that the driver's license is less then 5 years old, not-so-subtly implying that lack of experience is a major contributing factor to the crash.

BizBazar

The show that's gone is called BizBazar and what made it special was that it always tried to present both sides of the story. It was refreshing to see a conversation between a host and his guest where the host is presenting the guest with an opportunity to speak his mind, although it's obvious their views on the matter diverge greatly.

BizBazar was one of the rare shows whose team did not claim expertise in every conceivable subject, and this was a very welcome change compared to the rest of the shows out there, whose hosts will prefer talking nonsense to admitting the subject at hand is a big unknown.

BizBazar has been a fresh take on the news, light years away from the paranoia that characterizes so much of our media these days.

My hat's off to the team, good luck in whatever you will try next, and thanks for all the fish.

vim - replace command conditioned by preceding and following atoms

2010-02-25T16:38:00.006+02:00

Going from:

John has one hundred apples
Jane has twelve hundred and fifty seven apples
Mary has thirty seven apples

John has one_hundred apples
Jane has twelve_hundred_and_fifty_seven apples
Mary has thirty_seven apples

in vim:

.,$s/\(has .*\)\@<= \(.* apples\)\@=/_/g

Compiling Valgrind on Snow Leopard

2010-02-24T10:14:00.004+02:00

Checkout the valgrind sources:

svn co svn://svn.valgrind.org/valgrind/tags/VALGRIND_3_5_0 valgrind

Download Greg Parker's 10.6 patch

curl http://bugsfiles.kde.org/attachment.cgi?id=36999 > 10.6.patch
mv 10.6.patch ./valgrind

Apply the patch

cd valgrind
patch -p0 < 10.6.patch

Compile valgrind (if you want to run it on 64 bit binaries, use "./configure --build=amd64-darwin", otherwise you'll build it for 32 bits)

./autogen.sh
./configure
make

Install it

sudo make install

Run it

cristi:tmp diciu$ cat /tmp/test.c
int main()
{
 char * t = malloc(1024);
 char * w = "Ana";
 
 t = w;
}
cristi:tmp diciu$ gcc -arch i386 /tmp/test.c
cristi:tmp diciu$ valgrind --leak-check=full /tmp/a.out
[..]
==938== HEAP SUMMARY:
==938==     in use at exit: 1,344 bytes in 8 blocks
==938==   total heap usage: 8 allocs, 0 frees, 1,344 bytes allocated
==938== 
==938== 1,024 bytes in 1 blocks are definitely lost in loss record 4 of 4
==938==    at 0xF666: malloc (vg_replace_malloc.c:195)
==938==    by 0x1F66: main (in /tmp/a.out)

Using the Snow Leopard sandbox

2010-01-06T14:08:00.004+02:00

First and foremost, a warning that's prominently displayed in all sandbox definition files:

WARNING: The sandbox rules [..] currently constitute Apple System Private Interface and are subject to change at any time and
without notice. The contents of this file are also auto-generated and not user editable; it may be overwritten at any time.

The sandbox being a private interface is probably the reason why the sandbox policy files are un-documented. However, in the standard Snow Leopard install there are a bunch of files that define sandboxing policies ( under /System/Library/Sandbox/Profiles/ and /usr/share/sandbox/) that can be used to derive new policy files.

When creating a policy file for HTTP, the most verbose section out of our sandbox definition turns out to be the section for allowing DNS queries.

http.sb


(version 1)
(debug deny)

;; DNS
(allow file-read-metadata
        (literal "/var")
        (literal "/etc"))

(allow ipc-posix-shm)

(allow mach-lookup
  (global-name "com.apple.SystemConfiguration.configd")
  (global-name "com.apple.system.notification_center"))


(allow network-outbound
       (remote unix-socket (path-literal "/private/var/run/mDNSResponder")))

;; HTTP traffic
(allow network-outbound (remote ip4 "*:80"))

(deny default)

Our test program runs a couple of functions - a curl based HTTP client and a file read operation.

test.c


#include <sandbox.h>
#include <stdio.h>
#include <curl/curl.h>

size_t write_data(void *buffer, size_t size, size_t nmemb, void *userp)
{
 int i=0;
 //exit after 80 chars
 for(;i<size * nmemb && i < 80;i++)
  fprintf(stderr, "%c", (unsigned char)((char *)buffer)[i]);
 fprintf(stderr, "[..]\n");
}
int httpRequest()
{
 CURL * easyhandle = curl_easy_init();
 curl_easy_setopt(easyhandle, CURLOPT_URL, "http://loudhush.ro/changelog.txt");
 curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, write_data);
 curl_easy_perform(easyhandle);
}
int readFile()
{
 FILE * f = fopen("/etc/passwd", "r");
 if(f == NULL)
 {
  perror("Error opening file");
  return 1;
 }

 char line[1024];
 while(fgets(line, 1023, f) != NULL)
  fprintf(stdout, "Line: %s", line);

 fclose(f);
 return 0;
}
int main()
{
 char * errbuf;
 int retval = sandbox_init("/Users/diciu/Programming/sandbox/http.sb", SANDBOX_NAMED, &errbuf);
 if(retval)
 {
  fprintf(stderr, "Sandbox init error: %s\n", errbuf);
  return 1;
 }
 fprintf(stderr, "--- HTTP request ---\n");
 httpRequest();
 fprintf(stderr, "\n--- File read request ---\n");
 readFile();
}

As expected, once the sandbox is initialized, HTTP requests go through but the file access operation is denied:


cristi:sandbox diciu$ ./a.out 
--- HTTP request ---
LoudHush 1.3.21

- Fixed a couple of leaks in AccountController/Management
- Fix[..]

--- File read request ---
Error opening file: Operation not permitted

Attempts to go out of the sandbox are logged in syslog:


Jan  6 14:13:37 cristi sandboxd[59224]: a.out(59222) deny file-read-data /private/etc/passwd

LoudHush compatibility with Asterisk 1.4.26.2+ requires configuration on server

2010-01-04T10:51:00.005+02:00

With recent versions of Asterisk, a security triggered upgrade broke the IAX protocol as LoudHush knows it, by adding a new IAX IE called CALLTOKEN (see IAX2 Security).

When trying to register with an IAX account with CALLTOKEN enabled, LoudHush will display "Registration rejected" and the Asterisk server will log:


chan_iax2.c: Call rejected, CallToken Support required.

To disable the CALLTOKEN functionality from the server, allowing LoudHush to register, the following lines need to be added in the iax configuration:


calltokenoptional = 0.0.0.0/0.0.0.0
maxcallnumbers = 16382

Sending HTML email from Cocoa

2009-12-25T21:08:00.009+02:00

In a previous post I described sending email using c-client. This post takes things a little bit further and describes sending an email containing HTML content using the same c-client API.

First things first - people usually call HTML email an HTML fragment that refers inline images. These images are part of the email too and they are referred in the HTML source by their content id (using a special construct on the source reference, i.e. img src="cid:image_content_id").

Mail clients (at least Snow Leopard's Mail.app) expect the HTML fragment and the images they refer to be part of the same container. This container is a MIME multipart/related container.

The simplest HTML message that we can send is based on the following MIME multipart structure:


#include <c-client.h>

#include "c-client-interface.c"
int main()
{
        #include "linkage.c"

        char * hostlist[] = { "smtp.gmail.com/ssl/user=user@gmail.com/smtp", NULL };

        SENDSTREAM * sndstream = smtp_open(hostlist, 0);
        if(sndstream == NULL)
        {
                fprintf(stderr, "Can't open SMTP connection\n");
                return 1;
        }

        BODY * body = mail_newbody();

        ENVELOPE * env = mail_newenvelope();
        ADDRESS * to = mail_newaddr();

        env->from = mail_newaddr();
        env->from->personal = strdup("User Name");
        env->from->mailbox = strdup("username");
        env->from->host = strdup("gmail.com");
        env->subject = strdup("subject");

        rfc822_parse_adrlist(&env->to, strdup("username"), strdup("destination.dom"));

        char *text = (char *) fs_get (8*MAILTMPLEN);
        strcpy(text, "test message \015\012\0");

        #define HTMLfragment "<HTML><p>A little bit in <b>bold</b> and some <i>italic</i><br/><br/> </p><p><img src=\"cid:55665566\"></p><p>The end</p></HTML>"

/* We need three parts, a part for HTML, a part for the image and 
a part that groups them (called related). */



        PART * htmlpart, *imagepart, *related;
        related = mail_newbody_part();
        related->body.type = TYPEMULTIPART;
        related->body.subtype =  cpystr("related");

        htmlpart = mail_newbody_part();
        htmlpart->body.encoding = ENCBASE64;
        htmlpart->body.contents.text.data = (char *) rfc822_binary (HTMLfragment, strlen(HTMLfragment), &htmlpart->body.contents.text.size);

        htmlpart->body.parameter = mail_newbody_parameter ();
        htmlpart->body.parameter->attribute = cpystr ("CHARSET");        
        htmlpart->body.parameter->value = cpystr ("US-ASCII");        
        htmlpart->body.type = TYPETEXT;        
        htmlpart->body.subtype = cpystr ("HTML");
        htmlpart->body.description = cpystr ("html version");

        //for brevity, we define a GIF file inline. in real code, you would probably load this from a file
        unsigned char buf[] = {0x47, 0x49, 0x46,0x38,0x37,0x61,0x0c,0x00,0x0c,0x00,0xc2,0x04,0x00,0x00,0x24,0xff,0xff,0x00,0x00,0x00,0xa0,0x0b,0xf6,0x7a
,0x00,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0x2c,0x00,0x00,0x00,0x00,0x0c,0x00,0x0c,0x00,0x00,0x03,0x23,0x18,0xba,0xd3,0xbe,0xcc,0
x35,0xa8,0xe4,0xa4,0x76,0x50,0x42,0x2c,0xe4,0x9c,0x24,0x8c,0x02,0x48,0x00,0x28,0x40,0x8e,0x5c,0x8a,0xae,0xa3,0xfb,0xc2,0xb2,0x4a,0xcb,0xb0,0x50,0x27,0x0
0,0x3b};

        /* allocate a MIME part for the image content at the end of the HTML part */
        imagepart = (htmlpart->next = mail_newbody_part());
        /* the image is binary so we encode in base64 */
        imagepart->body.encoding = ENCBASE64;
        imagepart->body.contents.text.data = (char *) rfc822_binary(buf, 86, &imagepart->body.contents.text.size);
        imagepart->body.type = TYPEIMAGE;
        imagepart->body.subtype = cpystr ("GIF");
        imagepart->body.id = cpystr("<55665566>");

/* Our body's content is of type multipart and it contains a nested part 
that is in fact the related part that we have created above */

        body->type = TYPEMULTIPART;
        body->subtype = cpystr("alternative");
        body->nested.part = mail_newbody_part ();
        body->nested.part->body.contents.text.data = text;
        body->nested.part->body.contents.text.size = strlen (text);


        related->body.nested.part = htmlpart;
        body->nested.part->next = related;

        long result = smtp_mail(sndstream, "MAIL", env, body);

        if(sndstream)
                smtp_close(sndstream);

        mail_free_envelope(&env);
        mail_free_body(&body);
}

Our message in Mail.app:

Creating QR codes on Mac OS X Snow Leopard

2009-12-11T09:13:00.009+02:00

This post describes a step by step procedure to compile and run qrencode, a command line tool for encoding text into png files of QRcodes written by FUKUCHI Kentaro as part of libqrencode (licensed under LGPL).

1/ Create a directory for our project

cristi:~ diciu$ mkdir qr
cristi:~ diciu$ cd qr/

2/ Download the libqrencode source, and untar it.

cristi:qr diciu$ tar zxvf qrencode-3.1.0.tar.gz
cristi:qr diciu$ cd qrencode-3.1.0

3/ We need to set a couple of environment variables to tell configure where the libpng libraries and headers are:

cristi:qrencode-3.1.0 diciu$export png_CFLAGS="-I /usr/X11/include/"
cristi:qrencode-3.1.0 diciu$export png_LIBS="-lpng -L/usr/X11/lib/"

4/ Even though we don't need pkg-config (that's why we exported vars in step 3), configure requires a pkg-config script, so we'll create an empty one and add it to the path.

cristi:qrencode-3.1.0 diciu$ touch pkg-config
cristi:qrencode-3.1.0 diciu$ chmod +x pkg-config 
cristi:qrencode-3.1.0 diciu$ export PATH=$PATH:.

5/ Now we have everything we need, so we can run ./configure

./configure

6/ Once configure runs, we can run make to build it

make

7/ And finally we get to encode a qr code:

./qrencode -o ~/Desktop/modulo.png "http://www.modulo.ro"

The end result:

Query strings received from Google

2009-12-04T20:08:00.004+02:00

Some of the unexpected queries that get here and some hopefully useful answers:

1/ Some netstat based commands, e.g.:


netstat -anp | grep "tcp\|udp" | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

As far as I can tell, it prints a summary of network connections. This includes established connections and connections that are about to close, and also connections that are locally awaiting for a client.

In the Mac OS X version of netstat, the "-p" option is not supported; sort is also capable of uniquing its results so you can rewrite the line as:


netstat -an | grep "tcp\|udp" | awk '{print $5}' | cut -d: -f1 | sort -nu

2/ check a word how many times in nsstring

I'd do something like:


int count = 0;
NSString * s = [NSString stringWithString:@"foobar contains foobar and some more foobar"];
while([s rangeOfString:@"foobar"].location != NSNotFound)
{
    count++;
    s = [s substringFromIndex:([s rangeOfString:@"foobar"].location + [s rangeOfString:@"foobar"].length)];
}

3/ /tmp/msgsends

It's the prefix of files created by setting the variable NSObjCMessageLoggingEnabled to YES in the environment of a Cocoa application. The ObjC runtime will write an entry for each ObjC message received in the application.

4/ average life expectancy of macbook

I don't really know - we've had two in the office, one ran for about 18 months before dying (mine), the other is still running (after more then 2 years).

5/ core image histogram
Look for vImageHistogramCalculation_ARGB8888, part of Accelarate.framework.

6/ raising x to power of y "objective c"

NSLog(@"2 to the power of 3: %f", pow(2.0, 3.0));

Diacritical folding in SQLite

2009-11-08T18:17:00.003+02:00

Diacritical folding can be directly achieved in SQLite by implementing a custom function call. The following code demonstrates a possible implementation of such a function, taking in a string and returning a copy of the string with its diacritical marks folded.


#include <stdlib.h>
#include <sqlite3.h>
#include <stdio.h>
#include <CoreFoundation/CoreFoundation.h>

// this will be called by sqlite to dispose of the memory we malloced in diactrics_folding
void destruct_result(void * ptr)
{
    free(ptr);
}
void diactrics_folding(sqlite3_context* ctx,int cnt,sqlite3_value** val)
{
    char * in = (char *)sqlite3_value_text(*val);
    if(in == NULL)
        return;

    CFStringRef cstr = CFStringCreateWithCString(NULL, in, kCFStringEncodingUTF8);
    CFMutableStringRef str = CFStringCreateMutableCopy(NULL, 0, cstr);
    CFStringFold(str, kCFCompareDiacriticInsensitive, NULL);

    char * resultAsCString = malloc(strlen(in));
    if(!CFStringGetCString(str, resultAsCString, strlen(in), kCFStringEncodingUTF8))
    {
        free(resultAsCString);
        sqlite3_result_text(ctx, in, strlen(in), NULL);
        return;
    }
    sqlite3_result_text(ctx, resultAsCString, strlen(resultAsCString), &destruct_result);

    CFRelease(cstr);
    CFRelease(str);
}
int cback (void* udata,int ncol,char** value,char** colname)
{
    int i=0;
    for(;i<ncol;i++)
        printf("Result column: %s value: %s   \n", colname[i], value[i]);
    return 0;
}
int main()
{
    sqlite3 * handle;
    int res = sqlite3_open("./test.sql", &handle);

    res = sqlite3_create_function(handle, "diactrics_folding", 1, SQLITE_UTF8, NULL, &diactrics_folding, NULL, NULL);
 
    char * errmsg = NULL;   
    res = sqlite3_exec(handle, "select w, diactrics_folding(w) from t", &cback, NULL, &errmsg);
    printf("sqlite3_exec result: %d %s\n", res, errmsg != NULL ? errmsg : "No error");

    sqlite3_close(handle);
}


diciu$ gcc sqlite_diacritics.c -framework CoreFoundation -lsqlite3
diciu$ ./a.out
Result column: w value: mămăliguță cu brânză și smântână   
Result column: diactrics_folding(w) value: mamaliguta cu branza si smantana   
sqlite3_exec result: 0 No error

Web Services from Cocoa/Objective-C (Take 2)

2009-10-29T12:57:00.005+02:00

One solution to the web services from Cocoa problem is using gSoap. It's not 100% to my liking, in that solves web services by reducing them to SOAP and I find that mildly annoying - in my opinion web services tools should do their best to abstract away the transport.

However, gSoap does a nice job of creating pure C code that can be used as a web services client so I'll ignore my reluctance to think of SOAP when talking about web services. I am using complete paths (/Users/diciu/Downloads) because I've not installed gSoap - it makes compiling a bit more verbose but I hate writing in my /usr/ folder. YMMV.

Here's how you build a C web services client to the EchoHeaders sample web service that's part of Apache Axis.

Step 1. WSDL to SOAP


/Users/diciu/Downloads/gsoap-2.7/gsoap/wsdl/wsdl2h  -c -o test.h EchoHeaders.wsdl

Step 2. SOAP stub and skeleton creation


/Users/diciu/Downloads/gsoap-2.7/gsoap/bin/macosx/soapcpp2 test.h

This step will create a flurry of XML files and a namespace map file (EchoHeadersSoapBinding.nsmap).

Step 3. Creating a client (test.c)


#include <stdlib.h>
#include "soapH.h"
#include "EchoHeadersSoapBinding.nsmap"

int main()
{
    struct soap *soap = soap_new(); 
    char * t;
 
    if(soap_call_ns2__echo(soap, NULL, NULL, "test echo", &t) == SOAP_OK)
    {
        printf("echo result: >%s<\n", t);
    }
    else
    {
        soap_print_fault(soap, stderr); 
    }
}

Step 4. Compiling the client


gcc test.c  soapC.c soapClient.c  -I/Users/diciu/Downloads/gsoap-2.7/gsoap/ /Users/diciu/Downloads/gsoap-2.7/gsoap/libgsoap.a

Step 5. Running the client


cristi:webservices diciu$ ./a.out 
echo result: >test echo<

Of course, this last step assumes that Apache Axis is running on localhost port 8080, otherwise you will get a SOAP fault:


cristi:webservices diciu$ ./a.out 
SOAP 1.1 fault: SOAP-ENV:Client [no subcode]
"Connection refused"
Detail: connect failed in tcp_connect()

Evaluating arithmetic expressions

2009-10-19T08:50:00.008+03:00

Or computing the result of 1+2*3-(7*7*9)+1-(2*(2+5-1)) to -445, without using "eval".

In languages missing evaluation capabilities, evaluating arbitrary expressions is not necessarily straight forward, because expressions written in infix form can't be intuitively parsed or evaluated. The quickest solution is probably embedding a scripting engine at runtime and using the scripting engine to evaluate the expression but this is not always feasible.

This post describes an alternate solution, that is parsing the expression into Reverse Polish Notation and evaluating the RPN form, one operator at a time.

Our infix expression parser requires three steps:
- parsing the expression into tokens.
- infix to RPN (reverse polish notation) (we'll do this using the Shunting-yard algorithm).
- evaluate the RPN expression.

We first define the token types our expression will be parsed into.
There's a limited set of operators used in our example: addition, subtraction, multiplication and division. We'll also support parenthesis for defining groups.


------------------------- defs.h ------------------------
typedef enum {
    ENDFILE, ERROR,
    LPAREN, RPAREN,
    OPPLUS, OPMINUS,
    OPMUL, OPDIV,
    CONSTNUMBER} TokenType;

We'll parse the expression using (f)lex so we need to define our tokens for lex to parse:


-------------------------- infix.lex ----------------------
%{
#import <Cocoa/Cocoa.h>
#import "defs.h"
#import "Token.h"

%}

digit [0-9]
number {digit}+

whitespace [ \t]+
newline \n
%%
{number}        {return CONSTNUMBER;}
"+"             {return OPPLUS;}
"-"             {return OPMINUS;}
"*"  {return OPMUL;}
"/"  {return OPDIV;}
"("             {return LPAREN;}
")"             {return RPAREN;}
{whitespace}    {/* skip whitespace */}
{newline}       {return ENDFILE;}
.               {return ERROR;}
%%
Token * getToken()
{
    Token * t = [[[Token alloc] init] autorelease];
    [t setType:yylex()];
    [t setValue:[NSString stringWithCString:yytext encoding:NSASCIIStringEncoding]];

    return t;
}

In the lex definition file, we define the token types based on the regular expressions they match. The getToken function will be repeatedly called to create and consume tokens. The token itself has two attributes, the token type (as returned by yylex()) and the corresponding token value (yytext).

We'll use a very simple ObjC class to store and manipulate our tokens.


------------------------- Token.h ------------------------
#import <Cocoa/Cocoa.h>
#import "defs.h"

@interface Token: NSObject
{
    TokenType type;
    NSString * value;
}
@property TokenType type;
@property (retain) NSString * value;
- (BOOL) isOperator;
- (int) precendence;

+ (Token *) eval:(Token *)larg :(Token *)op :(Token *)rarg;


------------------------- Token.m ------------------------
#import "Token.h"

@implementation Token

@synthesize type;
@synthesize value;

- (BOOL) isOperator
{
    if(type == OPPLUS || type == OPMUL || type == OPDIV || type == OPMINUS)
        return YES;
    return NO;
}
- (int) precendence
{
    if(type == OPPLUS || type == OPMINUS)
        return 10;
    if(type == OPMUL || type == OPDIV)
        return 100;
    return 0;
}

- (NSString *) description
{
    return value;
}

+ (Token *) eval:(Token *)larg :(Token *)op :(Token *)rarg
{
        NSInteger lval = [[larg value] integerValue];
        NSInteger rval = [[rarg value] integerValue];

        Token * r = [[[Token alloc] init] autorelease];
        [r setType:CONSTNUMBER];

        int res;

        if([op type] == OPMUL)
                res = lval * rval;
        else if([op type] == OPDIV)
                res = lval / rval;
        else if([op type] == OPPLUS)
                res = lval + rval;
        else if([op type] == OPMINUS)
                res = lval - rval;

    NSString * s = [NSString stringWithFormat:@"%d", res];
    [r setValue:s];
    return r;
}

The Evaluate class defines most of the logic. It consumes tokens following the Shunting-yard algorithm.


------------------------- Evaluate.m ------------------------
#import "Evaluate.h"
#import "defs.h"
#import "Token.h"
@implementation Evaluate

extern Token * getToken();

-(void) run
{

    NSMutableArray * outputQueue = [NSMutableArray arrayWithCapacity:10];
    NSMutableArray * operatorStack = [NSMutableArray arrayWithCapacity:10];
 
    Token * t = getToken();
    while([t type] != ENDFILE)
    {
        if([t type] == ERROR)
        {
            NSLog(@"Error parsing token: %@, halting", [t value]);
            break;
        }
        else if([t type] == CONSTNUMBER)
        {
            NSLog(@"Adding constant: %@", [t value]);
            [outputQueue addObject:t];
        }
        else if([t isOperator])
        {
            NSLog(@"Adding operator: %@", [t value]);
            while([operatorStack count]  && [[operatorStack objectAtIndex:0] isOperator])
            {
                Token * topOfStack = [operatorStack objectAtIndex:0];
                if([t precendence] <= [topOfStack precendence])
                {
                    NSLog(@"Popping operator %@ off the stack", topOfStack);
                    [outputQueue addObject:topOfStack];
                    [operatorStack removeObjectAtIndex:0];
                }
                else
                {
                    NSLog(@"Not popping off the stack, top is %@, current op is %@", topOfStack, t);
                    break;
                }
            }
            [operatorStack insertObject:t  atIndex:0];
        }
        else if([t type] == LPAREN)
        {
            NSLog(@"Adding operator: %@", t);
            [operatorStack insertObject:t atIndex:0];
        }
        else if([t type] == RPAREN)
        {
            NSLog(@"Adding operator: %@", [t value]);
            while([operatorStack count]  && [(Token *)[operatorStack objectAtIndex:0] type] != LPAREN)
            {
                NSLog(@"Popping LPAREN off the stack");
                Token * topOfStack = [operatorStack objectAtIndex:0];
                [outputQueue addObject:topOfStack];
                [operatorStack removeObjectAtIndex:0];
            }
            if([operatorStack count]  && [(Token *)[operatorStack objectAtIndex:0] type] == LPAREN)
                [operatorStack removeObjectAtIndex:0];
            else
                NSLog(@"Missing LPAREN");

        }

        t = getToken();
    }

    if([operatorStack count]  && ([(Token *)[operatorStack objectAtIndex:0] type] == LPAREN  || 
        [(Token *)[operatorStack objectAtIndex:0] type] == RPAREN))
            NSLog(@"Mismatched paren");

    int i=0;
    for(i=0;i<[operatorStack count];i++)
    {
        if([(Token *)[operatorStack objectAtIndex:i] type] == LPAREN || 
            [(Token *)[operatorStack objectAtIndex:i] type] == RPAREN)
        {
            NSLog(@"ERROR: Non operator on stack");
            return;
        }
        [outputQueue addObject:[operatorStack objectAtIndex:i]];
    }

    NSLog(@"Output queue: %@", outputQueue);

    NSLog(@"Trying to eval expression from queue");
    NSMutableArray * rpStack = [NSMutableArray arrayWithCapacity:10];
    for(i=0;i<[outputQueue count];i++)
    {
        Token * ct = (Token *)[outputQueue objectAtIndex:i];
        if([ct type] == CONSTNUMBER)
        {
            NSLog(@"Consuming numeric token: %@", ct);
            [rpStack addObject:ct];
        }
        if([ct isOperator])
        {
                NSLog(@"Consuming operator token: %@", ct);
                /* pop two args off of the stack*/
                if([rpStack count] < 2)
                {
                    NSLog(@"Error: Operator %@ is missing arguments", t);
                    return;
                }
                Token * t1 = [rpStack objectAtIndex:[rpStack count]-1];
                [rpStack removeLastObject];
                Token * t2 = [rpStack objectAtIndex:[rpStack count]-1];
                [rpStack removeLastObject];

                Token * eval = [Token eval:t2 :ct :t1];
                NSLog(@"Eval: %@ %@ %@ result: %@", t2, ct, t1, eval);
                [rpStack addObject:eval];
        }
    }

    if([rpStack count] != 1)
    {
        NSLog(@"ERROR: too many values");
        return;
    }

    NSLog(@"RP stack: %@", rpStack);

}
@end

Sending mail on Snow Leopard using the c-client API

2009-10-16T15:18:00.007+03:00

c-client is an API that you can use to build email clients and servers.
It's been around for more then 10 years and it implements virtually anything you would want to do with SMTP, IMAP or POP, albeit in ANSI C which makes things a bit more challenging, especially when you go through the sources and try to follow the pointer arithmetic being used.

I haven't touched c-client since 2001 (that's 8 years ago) and I've run into a pleasant surprise finding c-client alive and well in 2009, and compiling "out of the box" on Snow Leopard on 64 bit. Many thanks, Mark Crispin.

Here's how you write a small application that acts as a SMTP client.

1/ Download the distribution (ftp://ftp.cac.washington.edu/imap/imap.tar.Z)

2/ Un-archive the distribution

3/ Build the distribution

cd imap-2007e/
make osx

4/ Build a small application that sends an SMTP message

c-client requires an application linking against it to implement a set of functions.
These are grouped in our example under the file called c-client-interface.c:


// --------------------- c-client-interface.c --------------------------------
#include <stdio.h>

void mm_searched (MAILSTREAM *stream,unsigned long number) {}
void mm_exists (MAILSTREAM *stream,unsigned long number) {}
void mm_expunged (MAILSTREAM *stream,unsigned long number){}
void mm_flags (MAILSTREAM *stream,unsigned long number) {}
void mm_notify (MAILSTREAM *stream,char *string,long errflg) {}
void mm_list (MAILSTREAM *stream,int delimiter,char *mailbox,long attributes) {}
void mm_lsub (MAILSTREAM *stream,int delimiter,char *mailbox,long attributes) {}
void mm_status (MAILSTREAM *stream,char *mailbox,MAILSTATUS *status) {}

void mm_log (char *string,long errflg)
{
    fprintf(stdout, "mm_log: %s\n", string);
}

void mm_dlog (char *string)
{
    fprintf(stdout, "mm_dlog: %s\n", string);
}

void mm_login (NETMBX *mb,char *user,char *pwd,long trial)
{
    fprintf(stdout, "mm_login for mb->user: %s\n", mb->user);
    strcpy(user, mb->user);
    strcpy(pwd, "passwordGoesHere");
}

void mm_critical (MAILSTREAM *stream) {}
void mm_nocritical (MAILSTREAM *stream) {}
long mm_diskerror (MAILSTREAM *stream,long errcode,long serious) {}
void mm_fatal (char *string) {}

We implement only three functions from the interface, namely mm_log and mm_dlog for logging purposes and mm_login, required to provide the password for our SMTP authentication when the server requires it.

The application itself is as basic as it gets. Note the host specification (it contains "novalidate-cert" because the certificate on our mail server is invalid).
The host specification is used by the smtp_open call that takes care of all the hard work that goes behind the scenes, that is the negotiation in between c-client and the SMTP server.


// --------------------- test.c ------------------------------------
#include <c-client.h>

#include "c-client-interface.c"
int main()
{
    #include "linkage.c"

    char * hostlist[] = { "mail.somedomain.com/ssl/novalidate-cert/user=senderUserName", NULL };

    SENDSTREAM * sndstream = smtp_open(hostlist, 0);
    if(sndstream == NULL)
    {
        fprintf(stderr, "Can't open SMTP connection\n");
        return 1;
    }

    BODY * body = mail_newbody();

    ENVELOPE * env = mail_newenvelope();
    ADDRESS * to = mail_newaddr();

        // data about the sender
    env->from = mail_newaddr();
    env->from->personal = strdup("Sender Full Name");
    env->from->mailbox = strdup("senderUserName");
    env->from->host = strdup("somedomain.com");

        // subject for our message
    env->subject = strdup("subject");

        // destination for our message (the recipient)
    rfc822_parse_adrlist(&env->to, strdup("recipientUserName"), strdup("destinationdomain.com"));

        // the content of our message
    body->type = TYPETEXT;
    char *text = (char *) fs_get (8*MAILTMPLEN);
    strcpy(text, "test message \015\012\0");
    body->contents.text.data = text;
    body->contents.text.size = strlen(text);

 

    long result = smtp_mail(sndstream, "MAIL", env, body);

    if(sndstream)
        smtp_close(sndstream);

    mail_free_envelope(&env);
    mail_free_body(&body);
}

To build the application, once you saved the c-client-interface.c and the test.c files, you need to run (assuming you've unarchived the imap archive on your Desktop):


gcc test.c -L ~/Desktop/imap-2007e/c-client/c-client.a -I ~/Desktop/imap-2007e/c-client -lcrypto -lssl -lkrb4

Oh, and if you compile on Snow Leopard you will get a 64 bit application:


cristi:c-client diciu$ file ./a.out 
./a.out: Mach-O 64-bit executable x86_64

About LoudHush 1.3.21

2009-09-01T11:28:00.004+03:00

LoudHush 1.3.21 is out, the change log follows:

Fixed a couple of leaks in AccountController/Management

Fixed bug that caused LoudHush to hang in Snow Leopard (due to updating UI from two different threads)

Address book browser ordering

When hovering over missed calls, click opens the call history window

When call history is brought up, reset missed calls counter

Please note 1.3.21 has received limited testing; it is being released early because 1.3.20 is hanging on Snow Leopard/10.6.
If you encounter problems using 1.3.21, write us (email address available over at loudhush.ro site) and we'll try to help.

On Romanian GIS data, Stereo 1970 and WGS84

2009-07-10T09:02:00.012+03:00

GIS data for Romania is sparse. The government have it, but they are slow to share it and even when they do, they fail to release under clear licensing terms.

Once GIS data is obtained, another problem arises - the data is most likely in the Stereo 1970 projection (EPSG 31700). Modern GIS systems, including openstreetmap, have no capability of interpreting data in anything other then WGS 84 (EPSG 4326).

A translation is needed from Stereo 1970 to WGS84.

One can attempt to translate coordinates using the excellent libproj.
Given the coordinates for Cabana Fantanele, a hut in the Ceahlau mountains, in Stereo 1970, we can use the tools provided by libproj to translate to WGS84:


cristi:proj diciu$ ./src/cs2cs  +proj=stere +lat_0=46 +lon_0=25 +k=0.999750 \
> +x_0=500000 +y_0=500000 +ellps=krass +units=m +no_defs  +to  \
>+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs -f "%.6f"
572176.849  611484.777
25.949088 46.999202 0.000000

However, if you plot this WGS84 coordinate set on Google Earth (pin name is "Fantanele proj"), you will see the coordinates are approximately 100 meters away from the hut.

Thanks to the work of a government agency called ANCPI (Agentia Nationala de Cadastru si Publicitate Imobiliara) we have a tool that allows accurate translations between Stereo 1970 and WGS84: TransDatRO.

Unfortunately accurate translation between Stereo 1970 and ETRS89 requires a set of common points and the tool only includes enough points for the following counties: Caras Severin, Gorj, Valcea, Dambovita, Teleorman, Braila, Botosani, Suceava.

Degrees, minutes, seconds can be converted to decimal degrees (as required by openstreetmap) using libproj's cs2cs:


cristi:proj diciu$ ./src/cs2cs -f “%.6f” +proj=latlong +datum=WGS84 +to +proj=latlong +datum=WGS84
46d59'56.28495N 25d56'51.25243E
“46.998968” “25.947570” “0.000000”

When we plot the coordinates as returned by TransDat, even though Neamt county is not part of the accurate county list, we observe that coordinates returned by TransDat are much closer to what's revealed by the satellite imagery (see how "Fantanele Topo" is in close proximity to the hut's roof):

Note: I am not a geographer. All of this explained above is derived from personal, rather empiric research.
I have lots of questions that I couldn't find an answer to such as why there is no apparent difference when converting from ETRS89 to WGS84 - you might notice that I use ETRS89 and WGS84 interchangeably.

postgresql drop Romanian diacritics

2009-06-18T16:36:00.001+03:00


create or replace function strip_diacritics(varchar) returns varchar as $$
DECLARE 
  src ALIAS FOR $1;  
  result VARCHAR;

BEGIN  result := TRANSLATE(src, E'\xc3\x82\xc3\x83\xc3\xa2\xc3\xa3\xc4\x82\xc3\x8e\xc3\xae\xc5\x9e\xc5\x9f\xc5\xa2\xc3\xa3\xc7\x8d\xc7\x8e\xc8\x98\xc8\x99\xc8\x9a\xc8\x9B\xc3\x83', 'AaaaAIiSsTtAaSsTtA');
return result;
END;
$$ LANGUAGE 'plpgsql' IMMUTABLE;

Note to self

2009-06-10T16:27:00.005+03:00

OpenStreetMap places less then 2000 meters apart with names of the same length:


select p1.name, p2.name, p1.osm_id, p2.osm_id, p1.place, p2.place, distance(p1.way, p2.way) from planet_osm_point p1, planet_osm_point p2  where p1.place !='' and p2.place!='' and p1.osm_id != p2.osm_id and distance(p1.way, p2.way) < 2000 and octet_length(p1.name) = octet_length(p2.name) order by p1.name;

Spell checking in Objective-C

2009-04-28T15:58:00.007+03:00

I've finally beat python at running the very simple spell checker described in an earlier post. The code is not "pure" Objective-C, some parts are just too slow so I've had to mix in some C.

The major difference compared to the versions described in the earlier post is this addition in the loop of - (NSSet *) knownEdits2:(NSString *) word of an additional condition:

if([ts intersectsSet:knownWords])
   [edits2 unionSet:[self edits1:w]];

The intersectsSet condition avoids growing the set of edit distance 2 very effectively - in fact I suspect the python reference code does exactly that - I'm not sure how to read the python construct into distinct loops: return set(e2 for e1 in edits1(word) for e2 in edits1(e1) if e2 in NWORDS) so I can't tell for sure.

CSSpell .h


#import <Cocoa/Cocoa.h>

@interface CSSpell : NSObject 
{
    NSMutableDictionary * model;
    NSDictionary * NWORDS;
    NSMutableSet * knownWords; 
}
@end

CSSpell.m


#import "CSSpell.h"

@interface CSMutableNumber:NSObject
{
    NSUInteger mNumber;
}
@end

@implementation CSMutableNumber
- (id)initWithInt:(int)value 
{
    self = [super init];
    if(!self) return nil;
    mNumber = value;
         return self;
}
+ (CSMutableNumber *)numberWithInt:(int)value 
{
     return [[[CSMutableNumber alloc] initWithInt:value] autorelease];
}
- (void)setIntValue:(int)value 
{
     mNumber = value;
}
- (void) increment
{
     if(mNumber < NSUIntegerMax) mNumber++;
}
- (int) intValue
{
     return mNumber;
}
@end

@implementation CSSpell

- (NSDictionary *) trainv2:(NSString *)text
{
     if(model == nil)
     {
          model = [NSMutableDictionary dictionaryWithCapacity:40000];
          [model retain];
     }

     NSString * t = [text lowercaseString];
     char * buf = (char *)[t cStringUsingEncoding:NSASCIIStringEncoding];
 
     char **bp = &buf;
     char *tok;
     while (tok = strsep(bp, " \r\t,.*()[]#\n\"-!?&/'~;:"))
     if(strlen(tok)>2)
     {
           NSString * w = [NSString stringWithCString:tok];
   
           CSMutableNumber * wordVal = [model objectForKey:w];
           if(wordVal)
                [wordVal increment];
           else
                 [model setObject:[CSMutableNumber numberWithInt:1] forKey:w];
   
      }

     NSLog(@"Caching set of known words");
 
     knownWords = [NSSet setWithArray:[model allKeys]];
     return model;
}



/* returns a set of words at edit distance 1 away */
- (NSSet *) edits1:(NSString *) word
{
     NSMutableSet * s = [NSMutableSet setWithCapacity:2000];
 
     int i=0, len;
     char c;
     #define MAX_LEN_C 500
     char str[MAX_LEN_C];
     char temp[MAX_LEN_C];
 
 
     strcpy(str, [word cStringUsingEncoding:NSASCIIStringEncoding]);
     for(i=0;i<[word length];i++)
     {
          strcpy(temp, str); temp[i] = str[i+1]; temp[i+1] = str[i];
          len = strlen(temp);
  
          if(i<[word length]-1)
          {
               NSString * transposed = [NSString stringWithCString:temp length:len];
               [s addObject:transposed];
          }
  
          strcpy(temp + i, str+i+1);
          NSString * deletion = [NSString stringWithCString:temp length:strlen(temp)];
          [s addObject:deletion];
  
          for(c='a';c<='z';c++)
          {
               temp[i] = c; strcpy(temp + i + 1, str+i+1);
               NSString * replaced = [NSString stringWithCString:temp length:len];
               [s addObject:replaced];
   
               strcpy(temp + i + 1, str+i);
               NSString * inserted = [NSString stringWithCString:temp length:len+1];
               [s addObject:inserted];
          }
     }
 
     return s;
}



/* returns a set of known words based at edit distance 2 away */
- (NSSet *) knownEdits2:(NSString *) word
{
     NSMutableSet * edits2 = [NSMutableSet setWithCapacity:5000];
     NSSet * edits1 = [self edits1:word];
 
     NSEnumerator * en1 = [edits1 objectEnumerator];
     NSString * w;
     while(w=[en1 nextObject])
     {
          NSSet * ts = [self edits1:w];
  
  
          if([ts intersectsSet:knownWords])
          [edits2 unionSet:[self edits1:w]];
      }
 
      //NSLog(@"Edits2 has %d", [edits2 count]);
 
     [edits2 intersectSet:knownWords];
     NSLog(@"KW for %@: %@", word, edits2);
 
     return edits2;
}

/* returns a set of known words from a given set of words */
- (NSSet *) known:(NSSet *) words
{
     NSMutableSet * s = [NSMutableSet setWithCapacity:100];
     [s unionSet:words];
     [s intersectSet:knownWords]; 
     return s;
}

- (NSString *) correct:(NSString *) word
{
     if([word length] > 300)
          return nil;
 
 
     NSMutableSet * candidates = [NSMutableSet setWithCapacity:10];
  
     if([candidates count] < 1)
     {
          [candidates unionSet:[self known:[NSSet setWithObject:word]]];
          //NSLog(@"Adding known words set: %@", candidates);
     }
 
     if([candidates count] < 1)
     {
          [candidates unionSet:[self known:[self edits1:word]]];
          //NSLog(@"Adding edit distance 1 set: %@", candidates);
      }
 
     if([candidates count] < 1)
     {
           [candidates unionSet:[self knownEdits2:word]];
     }
     if([candidates count] < 1)
     {
          //NSLog(@"Adding the word itself");
          [candidates addObject:word]; 
     }
 
     int max=0;
     NSString * chosen = @"Unknown#";
     NSEnumerator * e = [candidates objectEnumerator];
     NSString * w;
     while(w=[e nextObject])
     {
          CSMutableNumber * wordVal = [NWORDS objectForKey:w];
          if(wordVal)
          {
               if([wordVal intValue] > max)
               {
                    chosen = w;
                    max = [wordVal intValue];
               }
          }
     }
 
 
     return chosen;
}

- (void) awakeFromNib
{
 
     NSString * cnt = [NSString stringWithContentsOfFile:@"/Users/diciu/Desktop/big.txt" encoding:NSASCIIStringEncoding error:nil];
     NSLog(@"Dictionary has been loaded");
 
     NWORDS = [self trainv2:cnt];
     NSLog(@"Training done, %d words", [NWORDS count]);
 
     NSDate * startDate = [NSDate date];
 
     NSArray * tests = [NSArray arrayWithObjects:@"generataed", @"acount", @"guidlines", 
        @"wheere", @"myne", @"graet", @"silenc", @"aggresive", 
                                           @"appreceiated", @"aquantance", @"beggining",  nil];
     NSEnumerator * en = [tests objectEnumerator];
     NSString * c;
     while(c=[en nextObject])
     {
          NSDate * sd = [NSDate date];
          NSString * correct = [self correct:c];
          NSLog(@"%@ / %@ \t\t\t in %f seconds", c, correct, [sd timeIntervalSinceNow]);
     }
 
 
     NSLog(@"Total runtime: %f", [startDate timeIntervalSinceNow]);
 
}
@end

On (my) Objective-C code that's slower then expected

2009-04-19T21:28:00.014+03:00

I was looking into building a spell checker for the geographical names in BucharestApp based on a Peter Norvig's article on building a spell checker so I've started writing a translation of Peter's python script in Objective-C.

The unpleasant surprise came when I timed my Objective-C code against the python script.
Running the spell checker on one of my word lists in python took 1.11 seconds; the Objective-C code was 4.69 times slower.

After factoring out training time because this can be very easily optimized by something as simple as serializing the training dictionary instead of computing it at runtime, I watched how my Objective-C code takes a whopping 5.3 seconds for spellchecking 11 words.

generataed	generate	-1.459070 s
acount	count	-0.003310 s
guidlines	guideline	-1.173020 s
wheere	where	-0.002277 s
myne	mine	-0.001197 s
graet	great	-0.002184 s
silenc	silent	-0.001916 s
aggresive	aggressive	-0.003051 s
appreceiated	appreciated	-0.004764 s
aquantance	acquaintance	-1.483899 s
beggining	beginning	-1.185744 s
Total runtime		-5.329394

The data set shows a cluster of words performing at .003 seconds and another cluster performing at 1.4 seconds.
What the miss-spellings "generataed", "guidlines", "aquantance", "beggining" have in common is edit distance. Unlike the other miss-spellings that are all at edit distance 1, they are all at edit distance 2. The problem with edit distance 2 is that it produces many variations of the miss-spelled word (around 200000) - Peter explains all this in his article.

My first instinct was to blame NSSet operations. After all, I was intersecting a 200000 set (the words at edit distance 2) with a 40000 set (the dictionary).
This must be slow, right? Wrong. After messing with the initial capacity of my NSSets and poking around Google for some explanation of what I was seeing, I started looking at the data Shark shows.

As expected, Shark shows the bottleneck in knownEdits2, the method which computes variations at edit distance 2.
But it's not NSSet operations that take the most CPU, but rather unexpectedly it's [NSString stringWithFormat].

-[CSSpell knownEdits2] does most of its work in -[CSSpell edits1StringsInObjCUnoptimized]:


- (NSSet *) edits1StringsInObjCUnoptimized:(NSString *) word
{
 NSMutableSet * s = [NSMutableSet setWithCapacity:2000];
 
 int i=0;
 char c;
 //deletions
 for(i=0;i<[word length];i++)
  [s addObject:[NSString stringWithFormat:@"%@%@",
       [word substringToIndex:i],
       [word substringFromIndex:i+1]]];
  
  
  //replaces
  for(i=0;i<[word length];i++)
   for(c='a';c<='z';c++)
    [s addObject:[NSString stringWithFormat:@"%@%c%@",
         [word substringToIndex:i],
         c,
         [word substringFromIndex:i+1]]];
  
  // inserts
  for(i=0;i<[word length];i++)
   for(c='a';c<='z';c++)
    [s addObject:[NSString stringWithFormat:@"%@%c%@", 
         [word substringToIndex:i],
         c,
         [word substringFromIndex:i]]];
  
  //transposes
  for(i=0;i<[word length]-1;i++)
   [s addObject:[NSString stringWithFormat:@"%@%@%@%@",
        [word substringToIndex:i],
        [word substringWithRange:NSMakeRange(i+1, 1)],
        [word substringWithRange:NSMakeRange(i, 1)],
        [word substringFromIndex:i+2]]];
 
 return s;
}

-[CSSpell edits1StringsInObjCUnoptimized] is shown by Shark to call -[NSString stringWithFormat] lots of times.
Which makes sense, since edits are formed using loops. But when I looked at how otx shows the assembler code generated from this method, I had another surprise (many parts cut for brevity):


-(id)[CSSpell edits1StringsInObjCUnoptimized:]
[..]
   +36 0000281b  e8ef270000     calll  0x0000500f   +[NSMutableSet setWithCapacity:]
[..]
  +101 0000285c  e8ae270000     calll   0x0000500f  -[(%esp,1) substringToIndex:] ///--
[..]
  +134 0000287d  e88d270000     calll   0x0000500f  +[NSString stringWithFormat:]
[..]
  +158 00002895  e875270000     calll   0x0000500f  -[(%esp,1) addObject:]
[..]
  +268 00002903  e807270000     calll   0x0000500f  -[(%esp,1) substringToIndex:] ///--
[..]
  +306 00002929  e8e1260000     calll 0x0000500f    +[NSString stringWithFormat:]
[..]

The surprise is that substringToIndex is called twice.
Since it's called on an immutable string and thus it yields the same result within the method's scope, I would have expected GCC to optimize the call.

So I rewrote the method, optimizing calls to substring methods:


- (NSSet *) edits1StringsInObjC:(NSString *) word
{
 NSMutableSet * s = [NSMutableSet setWithCapacity:2000];

 int i=0;
 char c;
 
 for(i=0;i<[word length];i++)
 {
  NSString * toIndex = [word substringToIndex:i];
  NSString * fromIndex = [word substringFromIndex:i+1];
  NSString * fromIndex2 = [word substringFromIndex:i];
  
  
  if(i<[word length]-1)
   [s addObject:[NSString stringWithFormat:@"%@%c%c%@",
        toIndex,
        [word characterAtIndex:i+1],
        [word characterAtIndex:i],
        [word substringFromIndex:i+2]]]; // transposes
  
  
  [s addObject:[NSString stringWithFormat:@"%@%@", toIndex, fromIndex]]; //deletion
  
  for(c='a';c<='z';c++)
  {
   [s addObject:[NSString stringWithFormat:@"%@%c%@",
        toIndex,
        c,
        fromIndex]]; //replace
   [s addObject:[NSString stringWithFormat:@"%@%c%@", 
        toIndex,
        c,
        fromIndex2]]; //insert   
  }  
 }
 
 return s;
}

Timing the code after the optimization shows:

generataed	generate	-1.142463 s
acount	count	-0.001646 s
guidlines	guideline	-0.931142 s
wheere	where	-0.002310 s
myne	mine	-0.001241 s
graet	great	-0.001417 s
silenc	silent	-0.001597 s
aggresive	aggressive	-0.002322 s
appreceiated	appreciated	-0.003206 s
aquantance	acquaintance	-1.157321 s
beggining	beginning	-0.938545 s
Total runtime		-4.192067

This optimization has shaved off ~0.35 seconds when checking miss-spells at edit distance 2.
In the total runtime, we've decreased to 78% out of the initial runtime.

If we check this new version using Shark, the bottleneck is still shown in -[NSString stringWithFormat], in fact the stringWithFormat calls have increased their presence in CPU time to 34% out of the total run-time. Considering the Shark data takes into account training time too, stringWithFormat is in fact burning probably around 50% of our CPU time.

Googling around yet again, I've arrived to the conclusion that NSString might be indeed slow because of the complete UTF-8 support it offers. For the purpose of the spell checker, UTF-8 is not needed, but unfortunately you just can't tell NSString to not care about UTF-8.
How about replacing stringWithFormat and forming a NSString out of pure C strings?


- (NSSet *) edits1StringsInC:(NSString *) word
{
 NSMutableSet * s = [NSMutableSet setWithCapacity:2000];
 
 int i=0;
 char c;
 
 #define MAX_LEN_C 500
 char toIndex[MAX_LEN_C];
 char fromIndex[MAX_LEN_C];
 char fromIndex2[MAX_LEN_C];
 char str[MAX_LEN_C];
 char temp[MAX_LEN_C];
 
 if([word length] > MAX_LEN_C)
  return [self edits1StringsInObjC:word];
 
 strcpy(str, [word cStringUsingEncoding:NSASCIIStringEncoding]);
 
 for(i=0;i<[word length];i++)
 {
  strncpy(toIndex, str, i);
  toIndex[i] = '\0';
  strcpy(fromIndex, str+i+1);
  strcpy(fromIndex2, str+i);
  
  sprintf(temp, "%s%c%c%s", toIndex, str[i+1], str[i], str+i+2);
  NSString * transposed = [NSString stringWithCString:temp length:strlen(temp)];
  if(i<[word length]-1)
   [s addObject:transposed];
  
  sprintf(temp, "%s%s", toIndex, fromIndex);
  NSString * deletion = [NSString stringWithCString:temp length:strlen(temp)];
  [s addObject:deletion];
  
  for(c='a';c<='z';c++)
  {
   sprintf(temp, "%s%c%s", toIndex, c, fromIndex);
   NSString * replaced = [NSString stringWithCString:temp length:strlen(temp)];
   [s addObject:replaced];
   sprintf(temp, "%s%c%s", toIndex, c, fromIndex2);
   NSString * inserted = [NSString stringWithCString:temp length:strlen(temp)];
   [s addObject:inserted];
  }  
 }
 
 return s;
}

Timing the code after changing string forming to C shows a total runtime of 2.04 seconds.
Our optimization has shaved off 1 second when checking miss-spells at edit distance 2 compared to the original implementation.
In the total runtime, we're down to 38% from the initial implementation runtime. Mind you, it's still twice as slow when compared to the python equivalent.

In this latest version, the CPU time is split between edits1StringsInC (3X), intersectSet (2X) and unionSet (1X).
edits1StringsInC still takes most of the CPU time, with the most intensive operations being sprintf at 41% and stringWithCString at 30%.

The next thing I tried is replacing sprintf with strcpy.
Yet another half a second in speed gain, but the code becomes less and less readable:


- (NSSet *) edits1StringsInCv2:(NSString *) word
{
 NSMutableSet * s = [NSMutableSet setWithCapacity:2000];
 
 int i=0, len;
 char c;
 
 #define MAX_LEN_C 500
 char str[MAX_LEN_C];
 char temp[MAX_LEN_C];
 
 if([word length] > MAX_LEN_C-1)
  return [self edits1StringsInObjC:word];
 
 strcpy(str, [word cStringUsingEncoding:NSASCIIStringEncoding]);
 for(i=0;i<[word length];i++)
 {
  strcpy(temp, str); temp[i] = str[i+1]; temp[i+1] = str[i];
  len = strlen(temp);
  
  if(i<[word length]-1)
  {
   NSString * transposed = [NSString stringWithCString:temp length:len];
   [s addObject:transposed];
  }
  
  strcpy(temp + i, str+i+1);
  NSString * deletion = [NSString stringWithCString:temp length:strlen(temp)];
  [s addObject:deletion];
  
  for(c='a';c<='z';c++)
  {
   temp[i] = c; strcpy(temp + i + 1, str+i+1);
   NSString * replaced = [NSString stringWithCString:temp length:len];
   [s addObject:replaced];

   strcpy(temp + i + 1, str+i);
   NSString * inserted = [NSString stringWithCString:temp length:len+1];
   [s addObject:inserted];
  }
 }

 return s;
}

Using strcpy and hard to follow pointer arithmetic instead of sprintf, we get at 1.5 seconds:

generataed	generate	-0.402929 s
acount	count	-0.001699 s
guidlines	guideline	-0.329685 s
wheere	where	-0.000743 s
myne	mine	-0.000473 s
graet	great	-0.001423 s
silenc	silent	-0.000769 s
aggresive	aggressive	-0.001243 s
appreceiated	appreciated	-0.001449 s
aquantance	acquaintance	-0.421072 s
beggining	beginning	-0.340965 s
Total runtime		-1.509337

In knownEdits2 Shark shows a tie between edit1StringsInCv2 and -[NSMutableSet intersectSet].

At this point, further optimizations in knownEdits probably don't make sense unless NSMutableSet is replaced too. And working with sets in plain C is not fun, so for now I'll settle with a 1.50 seconds, 35% slower run-time when compared to the python equivalent.

Conclusions:

Python string handling is fast

Python set operations are fast

Objective-C string operations are slow

Network connections by geographic location

2009-04-17T14:28:00.004+03:00

A couple of days ago, a colleague made me aware that iplocationtools.com exposes a cool web service that allows you to get the geographical location based on an IP address.

Here's a script that parses the IP addresses the workstation is connected to, and then resolves each one to its geographical location:

clear; netstat -an | grep ESTABLISHED | awk {'print $5'} | sort -u | sed "s/$[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$.*/\1/g" | while read sw; do url="http://iplocationtools.com/ip_query.php?ip=$sw"; curl $url -s | ~/tools/xmlstarlet sel -t -m "/Response/CountryName" -v . -o "," -m "/Response/City" -v . -; done

Note the dependancy on xmlstarlet that allows easy extraction of values from XML content. Unlike xsltproc which demands a XSL file, xmlstarlet allows you to run an Xpath query, building the needed XSL file by itself.

And here's a rundown of the script:

clear just clears the shell screen.

clear; netstat -an -> get a list of connections, without attempting to solve names

clear; netstat -an | grep ESTABLISHED -> keep only lines containing ESTABLISHED, that is active connections

clear; netstat -an | grep ESTABLISHED | awk {'print $5'} -> keep only the 5th field from each line, containing the IP address we are connected to

clear; netstat -an | grep ESTABLISHED | awk {'print $5'} | sed "s/$[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$.*/\1/g" -> strip out the port number from the IP address.

Without sed, netstat outputs stuff like 74.125.43.83.443 where 443 is the TCP port we are connected to.
To strip out the port, we can use sed, matching IP addresses to 4 groups of digits separated by a dot, i.e. $[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$. This is later matched by the "\1" reference.

clear; netstat -an | grep ESTABLISHED | awk {'print $5'} | sort -u | sed "s/$[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$.*/\1/g" | while read sw; do command done

What follows is a while loop, where we read each ip address, one at a time and execute the localization command.

command is:

url="http://iplocationtools.com/ip_query.php?ip=$sw"; curl $url -s | ~/tools/xmlstarlet sel -t -m "/Response/CountryName" -v . -o "," -m "/Response/City" -v . -

We use curl to get an XML document that lists the geolocation of the IP we and then process the XML document using xmlstarlet, by extracting the country and city name.

Network statistics on OS X using the sysctl interface

2009-03-10T14:04:00.006+02:00

sysctl offers read access to a couple of kernel structures that contain network statistics.

I am yet to figure out the difference in between the IP statistics as read from the "net.inet.ip.stats" sysctl value when compared to the values as read by netstat (as seen below, IP stats prints 114243 packets and netstat reports 116840).


cristi:tmp diciu$ ./a.out
IP packets received:        114243
IP packets generated here:  85251
TCP connection attempts:    3337
TCP total packets sent:     81779
TCP total packets received: 105700


cristi:tmp diciu$ netstat -bi
Name  Mtu   Network       Address            Ipkts Ierrs     Ibytes    Opkts Oerrs     Obytes  Coll
[..]
en0   1500  Link#4    xx:xx:xx:xx:xx:xx   116840     0  113745333    85202     0   13307330     0
[..]

Here's the C source for the sysctl reader:


#include <sys/types.h>
#include <sys/sysctl.h>

#include <stdlib.h>
#include <stdio.h>

#include <sys/socketvar.h>
#include <netinet/ip.h>
#include <netinet/ip_var.h>
#include <netinet/tcp.h>
#include <netinet/tcp_var.h>

int main()
{
 void * oldp = malloc(1024);
 size_t oldlen = sizeof(struct ipstat), newlen;
 void * newp = NULL;

 int retval = sysctlbyname("net.inet.ip.stats", oldp, &oldlen, newp, newlen);


 struct ipstat * g = (struct ipstat *) oldp;
 printf("IP packets received:        %d\n", g->ips_total);
 printf("IP packets generated here:  %d\n", g->ips_localout);


 struct tcpstat * t = (struct tcpstat *) oldp;
 oldlen = sizeof(struct  tcpstat);
 retval = sysctlbyname("net.inet.tcp.stats", oldp, &oldlen, newp, newlen);
 printf("TCP connection attempts:    %d\n", t->tcps_connattempt);
 printf("TCP total packets sent:     %d\n", t->tcps_sndtotal);
 printf("TCP total packets received: %d\n", t->tcps_rcvtotal);
}

Tesseract GUI for Mac OS X

2009-02-02T14:58:00.009+02:00

Update 3 May 2011:

I've released new version for TesseractGUI (0.3) - the new version is using tesseract version 3 and it is linked against libtiff so it is able to open compressed TIFF files (packbits, lwz).

I've created a very simple user interface on top of tesseract.

It's available, along with sources under the MIT license, at: http://download.dv8.ro/files/TesseractGUI/

To use Tesseract GUI, open a TIF image.

After a couple of seconds, Tessertact GUI will switch to the text tab where it displays whatever it was able to recognize:

Nothing is saved, so if the recognition was successful you probably want to copy and paste it in some document.

Bătăi de cîmpi: De ce scriu cu î și nu cu â?

2009-01-19T21:42:00.007+02:00

A new take on the diacritical marks in Romanian (written in Romanian) : Bătăi de cîmpi: De ce scriu cu î și nu cu â?

Linguistics, history and common sense aside, the programming solution of string diacritic insensitive folding when performing searches no longer works as "â" is folded to "a" and "î" is folded to "i"; "a" is not equal to "i".

I for one do not agree with not using â simply because it's already been (widely) adopted - as Google puts it:

Results 1 - 10 of about 9,910,000 for brânză
Results 1 - 10 of about 163,000 for brînză

The damage is done, the two versions are hard to support side by side (see diacritical folding reason), politics sucks, be a man and admit it - â has won.

PS Diacritical folding failure test case below:


#include <Cocoa/Cocoa.h>

int main()
{
 NSAutoreleasePool * pool = [NSAutoreleasePool new];
 NSString * str1 = [NSString stringWithString:@"brinza"];
 NSString * str2 = [NSString stringWithString:@"brînză"];

 int res = [str1 compare:str2 options:NSDiacriticInsensitiveSearch];
 NSLog(@"The strings %@, %@ are %@equal", str1, str2, res == NSOrderedSame ? @"" : @"not ");


 NSString * str3 = [NSString stringWithString:@"brânză"];
 res = [str1 compare:str3 options:NSDiacriticInsensitiveSearch];
 NSLog(@"The strings %@, %@ are %@equal", str1, str3, res == NSOrderedSame ? @"" : @"not ");

 [pool release];
}


cristi:tmp diciu$ ./a.out 
2009-01-19 21:40:45.935 a.out[6734:10b] The strings brinza, brînză are equal
2009-01-19 21:40:45.965 a.out[6734:10b] The strings brinza, brânză are not equal

BucharestApp 1.8 is out

2009-01-09T13:06:00.008+02:00

BucharestApp 1.8 is out with some work in the place finding area (Point Of Interest finding and finding places around a given geographical location picked by clicking on the map).

There's a new menu item in the Control-Click contextual menu titled "Coordinates" that allows you to start the POI finder panel centered around the click location or to copy the click coordinates to the clipboard:

The POI finder offers a number of built-in POI queries (for ATMs, banks, bus stops, malls, etc) but it does not expose all the capabilities of the namefinder (you are not able to search for pubs, hospitals, airports, etc using the POI finder).

However, you can copy the coordinates to clipboard and locate pubs through a custom search in the place finder panel:

The results are displayed based on distance from the coordinates used in the query, from the nearest to the furthest:

As with everything else, the chance of actually finding something you're looking for depends on the work of the folks contributing on OpenStreetMap so if you can't find you favorite pub or cafe you might want to log in the OpenStreetMap site and add it yourself.

Open source OCR on Mac OS X: tesseract

2008-12-22T18:53:00.006+02:00

I am looking for a optical character recognition solution and I've checked out OCRopus, but the early alpha stages it's in make it very hard to compile. OCRopus lists tesseract as a dependancy so I've compiled and ran tesseract on a couple of scanned pages.

The results are impressive (see below the results of running it on a page from a Cisco manual).

Chapter 24 • Mixed-Media Bridging

ending delimiter, which follows the data field) are treated differently depende
ing on the bridge manufacturer Some bridge manufacturers simply ignore the
bits. Others have the bridge set the C bit (to indicate that the frame has been
copied) but not the A bit (which indicates that the destination station recog-
nizes die address). Ln the former case, a Token Ring source node determines
whether the frame it sent has become lost. Proponents of this approach sug~
gest that reliability mechanisms, such as the tracking of lost frames [..]

Web Services from Cocoa/Objective-C

2008-12-18T20:02:00.004+02:00

If you're looking into using web services from Cocoa don't bother with WSMakeStubs.

It's so bad it's not even funny.

Trying out an OS X framework without installing it

2008-12-17T20:05:00.006+02:00

One of the things I dread is installing stuff on my box. And by installing I refer to those things using the OS X installer that copy themselves deep within the system. I am aware of lsbom and /Library/Receipts but whenever I can avoid it I choose not to install.

I especially don't like installing frameworks because they make my system incompatible with "vanilla" systems. I may at some point ship an application that depends on a framework that works fine on my system but instantly fails on a "vanilla" system, the dynamic linker complaining "image not found" as it tries to load the missing framework.

The solution is using a framework without actually installing it in /System/Library.

This can be done in XCode but I prefer the console - I find using gcc and the linker from the command line a good habit as it acts as a constant reminder of the process XCode hides away behind the build button.
Here's how to use gcc to compile a simple binary against a framework, without the framework being installed on the system:


#import <Cocoa/Cocoa.h>
#import <RegexKit/NSString.h>

int main()
{
 NSAutoreleasePool * pool = [NSAutoreleasePool new];

 NSString * t = @"This is a string with 12 or more characters on line 1";
 NSLog(@"t is %@", t);

 NSString * nodigits = [t stringByMatching:@"(\\d)" 
                                                   replace:RKReplaceAll 
                                                   withReferenceString:@"5"];
 NSLog(@"\nInput: >>> %@\nOutput: >>>  %@", t, nodigits);

 [pool release];
 return 0;
}

Our example above replaces any digit with an NSString with the digit "5".
It uses the RegexKit framework which I've previously unarchived on the Desktop (click right on RegexKit_Framework.pkg -> choose "Show Package Contents" -> unzip Archive.pax)


diciu$ gcc regexp.m -framework Cocoa   -framework RegexKit  -F/Users/diciu/Desktop/

When running the binary you have to remember to set the DYLD_FRAMEWORK_PATH environment variable:


diciu$ DYLD_FRAMEWORK_PATH=/Users/diciu/Desktop/ ./a.out
2008-12-17 20:12:49.979 a.out[3925:10b] 
Input: >>> This is a string with 12 or more characters on line 1
Output: >>>  This is a string with 55 or more characters on line 5

Don't trust clang blindly

2008-12-12T11:03:00.007+02:00

Update: Thanks to a comment by Ted Kremenek I discovered that the official clang tool does not emit the warning discussed below. It turns out that the AnalysisTool.app frontend application that I've used ships with its own modified clang.

Reading his comment I've also realized that my post might be interpreted as a criticism on clang and this is not the case - I think clang is a great addition to the existing tools on Mac OS X. It's just that, unlike leaks, static checkers should be used before the testing stage.

clang and the static analyzer that's built on top of it work by analyzing your sources while being compiled and making suggestions. Sort of like the warnings in GCC but on a deeper level, e.g.:


NSString *string;
string = [[NSString alloc] initWithData: data encoding: NSUTF8StringEncoding];

'autorelease' should be called right after alloc & init. Do not leave the responsibility of calling 'release' until later stages.

But clang goes wrong as the code that follows the lines above is:


if(string != nil)
  [textBuffer appendString:string];
  
[string autorelease];

Trusting clang's warning without realizing that two lines later the string variable was already being auto-released results in a crasher due to over-releasing.

Sure, clang is right and it doesn't really make sense to not auto-release the string immediately on but from a functional point of view not acting on clang's warning produces leak-free workable code, while following it blindly (i.e. without re-reading the entire context) makes clang happy but yields a crasher.

Which is exactly what happened with releases 1.5 and 1.6 of BucharestApp.
The moral of the story? never ship untested code.

BucharestApp 1.5 is out

2008-12-11T09:30:00.008+02:00

Download link: BucharestApp.zip [50Mb]

Yet another later edit: make that 1.7

Changes in version 1.7

Fixed a crasher when computing a route caused by an over-released NSString.

Later edit: make that 1.6.

Changes in version 1.6

Fixed a crasher when installing the map caused by an over-released NSString.

Changes in version 1.5

Place finder filters search results by geographical coordinates

Rendering based on data from December 2008

A bit more sane window and panel placement when Desktop spans multiple monitors

A map versioner so that new application versions can override cached data if necessary

Control-click for contextual menu for setting route from,to

Route and segment length computed in Km

Highlight segments table

New stuff includes entire Romania rendering at zoom level 10 (thus the small increase in app size), a bit of work on route finding and displaying the segments that are part of a route. The place finder is now based on interaction with a server side component that is hosted in Romania (we're hosting our own gazetteer).

Cisco VPN client used over 3G modem incompatible with Parallels

2008-12-04T14:12:00.005+02:00

If your Cisco VPN Client is disconnecting your VPN connection a couple of seconds after it's established when you're using a 3G USB modem, the culprit might be Parallels.

In my case, disabling IPv6 and IPv4 on both Parallels interfaces made the problem disappear.

gcc poison pragma

2008-11-27T13:43:00.004+02:00

Useful when you want to get rid of C calls that should not be used because they are unsafe, like strcpy or sprintf: GCC's pre-processor allows you to mark a function as poisonous, causing calls to it to become a compile time error:


#include <string.h>
#include <stdio.h>

#pragma GCC poison strcpy

int main()
{
    char t[5];
    strcpy(t, "A");
    printf("Hello World\n");
}


cristi:~ diciu$ gcc strcpy.c 
strcpy.c:10:2: error: attempt to use poisoned "strcpy"
cristi:~ diciu$

BucharestApp 1.4

2008-11-17T11:05:00.003+02:00

I've released Version 1.4 of BucharestApp

Changelog:

Help book

Rendering based on data from November 2008

OpenLayer library off-line copy (complete off-line map)

Route from, to picking from map (shift-click sets from, alt-click sets to)

Proper resize on window resize (old version was reloading the entire document)

Keyboard navigation (arrow keys for navigation, plus key to zoom in, minus key to zoom out)

Diacritical insensitive comparison, string folding

2008-11-07T15:31:00.004+02:00

One of the common problems when dealing with language containing diacritical marks is matching an improper writing of a word missing the diacritical marks.

This is especially true for Romanian, as most Romanians (myself included) don't use diacritical marks when writing in Romanian.

I'm not here to pass judgment on writing with or without diacritics. There are others much better informed on the subject.

The reality is improper use of diacritics will not disappear in the near future and the good news is that Cocoa has built-in support for dealing with this:


NSString * stringDia = [NSString stringWithString:@"Mămăligă"];
NSString * string = [NSString stringWithString:@"Mamaliga"];

int res = [stringDia compare:string options:NSDiacriticInsensitiveSearch];
NSLog(@"The strings are %@equal", res == NSOrderedSame ? @"" : @"not ");

There's even an implementation for string folding which may be worthwhile if you're building an index of place names:

CFStringFold

Discussion
Character foldings are operations that convert any of a set of characters sharing similar semantics into a single representative from that set.

You can use this function to preprocess strings that are to be compared, searched, or indexed.[..]

PS In care you were wondering: Mămăligă.

When Vodafone Mobile Connect refuses to start

2008-10-27T07:50:00.005+02:00

The problem: at some point Vodafone Mobile Connect refuses to start. The application icon bounces in the dock for a couple of minutes and the application window never gets displayed. The only way to quit the application at this point is using "Force Quit".

Quick solution

1/ Force quit VMC

2/ Delete "/Library/Application Support/nova media/VMC/ppp_session.log"

3/ Start VMC

Please note that deleting the ppp_session.log file wipes your PPP session history so VMC will re-start the traffic statistics from zero.

Why it hangs

VMC maintains a log of all the PPP traffic that flows through the 3G PPP interface. This log is saved under the ppp_session.log file mentioned above.

In certain conditions (e.g. traffic exceeding 90 MB in the previous session), VMC enters a function that never finishes execution namely:

001a428e t _ClassConnectionAbstract.TrafficFormat%s%o<ClassConnectionAbstract>i4

Here's an Instruments screenshot demonstrating the endless loop:

This is the ppp_session line that causes the infinite loop on my Mac:


diciu$ diff ~/Desktop/ppp_session.log.crasher ~/Desktop/ppp_session.log.works
200d199
< [2008/10/25 13:7:14] out:8736305 in:99719384 time:14036

The problem with that line is the "in" value which is too big - I'm guessing it's causing "pow" (the C function for computing the raising of X to the power of Y) to return a range error and the calling function doesn't process this error correctly (i.e. it calls pow$fenv_access_off again instead of exiting).

On Leopard, the effect is you cannot download more then 90 MB in one session - if you do, Vodafone Mobile Connect will refuse to start the next time you run it.

To fix the problem you either edit the ppp_session.log file and remove the last line, or, you delete the file altogether.

OpenStreetMap map of Bucharest as .app

2008-10-13T16:33:00.004+03:00

BucharestApp is a map rendering of Bucharest based on OpenStreetMap data, with a place finder panel and routing based on Gosmore.

F-Script and Core Data

2008-10-01T18:04:00.009+03:00

F-Script has very powerful support for array and set operations. This makes it particularly interesting when analyzing core data as it allows very easy querying of the data sets.

The first part is setting up the core data entities array in F-Script - this is pretty lengthy as you have to load both the object model and the managed object context tied to the persistent store holding the actual data.

If you want to paste the data below in the F-Script interpreter window, press F7. F-Script will say "When pasting command, newline is now interpreted as subcommands separator"


mompath := NSString stringWithString:'/Users/diciu/FoodBrowser/build/Release/FoodBrowser.app/Contents/Resources/FoodHistory.mom'
momurl := NSURL URLWithString:mompath 
mom := NSManagedObjectModel alloc initWithContentsOfURL:momurl

pstore := NSPersistentStoreCoordinator alloc initWithManagedObjectModel:mom

sqfile := NSString stringWithString:'/Users/diciu/Library/Application Support/FoodBrowser/FoodHistory.sql'
sqfileurl := NSURL fileURLWithPath:sqfile
pstore addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:sqfileurl options:nil error:nil

mocontext:= NSManagedObjectContext alloc init
mocontext setPersistentStoreCoordinator:pstore

edes := NSEntityDescription entityForName:'FoodItem' inManagedObjectContext:mocontext
fetchreq := NSFetchRequest alloc init
fetchreq setEntity:edes  

dataitems := mocontext executeFetchRequest:fetchreq error:nil

Once you've fetched your data, you can use the power of F-Script to analyze it:


> dataitems
_PFArray { (entity: FoodItem; id: 0x4da020  ; data: {
    energy = 380;
    name = "Snacks, pretzels, hard, plain, salted";
    quantity = 100;
    time = 2008-06-27 09:30:00 +0300;
    value = 380;
}), 
 (entity: FoodItem; id: 0xda192c0  ; data: {
    energy = 0;
    name = "Water, bottled, generic";
    quantity = 100;
    time = 2008-06-27 10:02:00 +0300;
    value = 0;
}),

Running sum through reduction:


> (dataitems energy) \#+
19777.38003540039

Average:


> (dataitems energy) \#+ / (dataitems count)
229.9695352953534

Maximum:


> (dataitems energy) \#max:
1080

Minimum


> (dataitems energy) \#min:
0

Is minimum value sane? (retrieve the index)


> (dataitems energy) ! ((dataitems energy) \#min:)
1


Print the object at index one:

> dataitems at:1
 (entity: FoodItem; id: 0xda192c0  ; data: {
    energy = 0;
    name = "Water, bottled, generic";
    quantity = 100;
    time = 2008-06-27 10:02:00 +0300;
    value = 0;
})

To run the graphical browser on the set of managed items, use:


> sys browse:dataitems

Reference: ExploringCocoaWithFScript

FoodBrowser 1.3 beta released

2008-09-30T14:37:00.002+03:00

Change log:

Monthly graph - work in progress

Calorie estimation box based on Mifflin and St Jeor

Copying foods in the history (copy, paste, drag and drop)

Editing time in the history

Fixed bug that caused some missing dates in the Energy Calculator window

Fixed bug in drag and drop of some foods

A failing disk and the first time I see S.M.A.R.T. actually working

2008-09-25T17:08:00.004+03:00

One of the disks that are part of the software RAID powering loudhush.ro has developed a bad sector.
The bad news arrived via email, thanks to smartmontools:

The following warning/error was logged by the smartd daemon:
Device: /dev/hdc, 1 Currently unreadable (pending) sectors
For details see host's SYSLOG (default: /var/log/messages).

The good news is you can tell the disk to re-allocate the bad sector somewhere else.
I've followed the instructions here: BadBlockHowTo.txt, almost to the letter (I did some additional work removing the disk from RAID), and now the disk looks better.

I still have to replace it, since a bad block is likely to be followed by others but it's somewhat reassuring to know that smartmontools is smart enough to help.

Demo screencast: configuring an IAX account with LoudHush

2008-09-15T17:27:00.002+03:00

Text to speech in LoudHush

2008-09-15T16:25:00.004+03:00

I've included a text to speech translator with the current development version of LoudHush. The feature will requires Leopard, otherwise the menu item leading to the text to speech window will remain disabled.

It's using the default voice as selected in the system preferences - i.e. the Alex voice on Leopard.

At this point I'm not sure about real use cases behind this feature so if you're willing to try a beta version please leave a comment or contact us at info at loudhush dot ro.

FoodBrowser released as beta

2008-09-08T17:48:00.002+03:00

I didn't find the time to work on FoodBrowser lately so I've decided to go ahead and release it as beta.

I've started writing it out of sheer frustration over the lack of a native OS X application that counts calories in various foods. Found a bunch, hated them. I've hated them because they record data in proprietary formats, locking data analysis into their own GUIs. Core Data is nice because you can always get to the data using sqlite.

So did my attempt fare any better?
Well, no. FoodBrowser is the point where most of my Cocoa projects get - usable but screaming "the devil is in the details". The devil is in the details and details take time and patience, none of which are cheap.

OpenStreetMap

2008-08-29T13:29:00.003+03:00

www.openstreetmap.org provides one of the first decent maps of Bucharest. It's based on community work, just like Wikipedia and their progress is unbelievable. Two years ago Bucharest had just one street on the map and now it looks almost completely mapped.

If you're thinking of participating, Merkaartor is probably the best way to edit maps on Leopard. It's based on QT and a bit quirky but it's fast and very much usable.

View Larger Map

See also www.informationfreeway.org, an online result of openstreemap's effort.

libGMP

2008-08-29T13:25:00.003+03:00

GMP is a free library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating point numbers. There is no practical limit to the precision except the ones implied by the available memory in the machine GMP runs on. GMP has a rich set of functions, and the functions have a regular interface.

Another definition:


cristi:demos diciu$ date; ./primes 100000000 | tail ;date
Fri Aug 29 13:24:22 EEST 2008
99999787
99999821
99999827
99999839
99999847
99999931
99999941
99999959
99999971
99999989
Fri Aug 29 13:24:23 EEST 2008

Ratproxy

2008-07-04T14:46:00.001+03:00

Google's open source ratproxy

A semi-automated, largely passive web application security audit tool, optimized for an accurate and sensitive detection, and automatic annotation, of potential problems and security-relevant design patterns based on the observation of existing, user-initiated traffic in complex web 2.0 environments.

timestamp to human readable string in sqlite3

2008-07-01T12:46:00.004+03:00

sqlite> SELECT datetime(ZTIME, 'unixepoch', '+31 years') from zfooditem;

Using divert sockets to log HTTP requests

2008-06-13T10:35:00.004+03:00

One of the cool things about divert sockets is they allow you to spy on local traffic with virtually zero overhead. You don't have to worry about re-forwarding the traffic to the intended recipient port, re-setting headers inside the TCP packet, etc.
Using divert tees you can just get a copy of the traffic matching a given rule, free for you to use in any way without interfering with the regular traffic.


cristi:~ diciu$ sudo ipfw add tee 8080 tcp from 193.231.199.80 to any 80 
00100 tee 8080 tcp from 193.231.199.80 to any dst-port 80
cristi:~ diciu$ sudo ipfw list
00100 tee 8080 tcp from 193.231.199.80 to any dst-port 80
65535 allow ip from any to any

The ipfw rule we are using is "tee" - the concept is similar to the Unix tee in that we get a copy of the traffic matched by the rule. Once we've set up a divert tee, we need a divert socket reader.
The protocol type for the divert socket is defined in netinet/in.h:


/usr/include/netinet/in.h:
#define IPPROTO_DIVERT  254  /* divert pseudo-protocol */

The code below binds a divert sockets and looks for HTTP GET and POST requests inside the content received from the socket:


import socket
import select
import re

IPPROTO_DIVERT = 254

sock = socket.socket(socket.AF_INET, socket.SOCK_RAW, IPPROTO_DIVERT)
fd = sock.bind(('127.0.0.1', 8080))
sock.setblocking(True)

MSGLEN = 32768

while(1):
 msg = ''
 while len(msg) < MSGLEN:
  chunk = sock.recv(MSGLEN-len(msg))
  if chunk == '':
   raise RuntimeError, "Socket gone"
  msg = msg + chunk

  t = re.compile(r"(?P(.*)(GET|POST)(.*)(HTTP/\d+\.\d+)(.*)(Host: )([a-zA-Z\.0-9-]*)(.*))", re.DOTALL)
  m = t.match(msg)
  if m:
   print m.group(3) + " " + m.group(4) + " " + m.group(5) + " " + m.group(8)

to run the code, you need to run python as root, because otherwise you'll get a permission denied when trying to create a RAW socket:


sudo python divert.py

Once you're done, you need to delete the divert tee:


sudo ipfw delete 100

Configuring Cisco VPN for local DNS connections on Leopard

2008-06-02T15:33:00.005+03:00

1/ Connect CISCO VPN client - it will overwrite DNS settings in /etc/resolv.conf with whatever is read from the network the VPN is connecting to.

If you're unlucky at this point normal DNS queries will no longer work.

2/ Use scutil to add a local DNS server:


diciu$ sudo scutil
> get State:/Network/Service/com.cisco.VPN/DNS

> d.show
 {
  ServerAddresses :  {
    0 : 10.10.10.1
    1 : 10.10.10.2
  }
  DomainName : somedomain.com
  SearchDomains :  {
    0 : somedomain.com
    1 : modulo.ro
  }
}
> d.add ServerAddresses * 193.231.199.1
> set State:/Network/Service/com.cisco.VPN/DNS
> d.show
 {
  ServerAddresses :  {
    0 : 193.231.199.1
  }
  DomainName : somedomain.com
  SearchDomains :  {
    0 : somedomain.com
    1 : modulo.ro
  }
}

Single line command:

printf "get State:/Network/Service/com.cisco.VPN/DNS\nd.add ServerAddresses * 193.231.199.1\nset State:/Network/Service/com.cisco.VPN/DNS" | sudo scutil

References: Using scutil to set DNS server

How to tell how many documents are being opened in a multi document application

2008-05-26T13:39:00.006+03:00

If you've subclassed NSDocumentController, when the user opens multiple documents, the method


- (id)openDocumentWithContentsOfURL:(NSURL *)absoluteURL display:(BOOL)displayDocument error:(NSError **)outError

will be called once per each document.

If you need to figure out how many documents are being opened (you might to do something after the last document that is part of a batch has been opened), you need to do a couple of things.

For documents opened via the open panel, things are easy:
In your NSDocumentController subclass, override the implementation for URLsFromRunningOpenPanel:


- (NSArray *)URLsFromRunningOpenPanel
{
    NSArray * a = [super URLsFromRunningOpenPanel];
    documentsPendingOpen = [a count];
 
    return a;
}

For documents being opened through Applescript events (such as documents dropped over the application icon), this is my not very elegant but efficient solution:


- (id)openDocumentWithContentsOfURL:(NSURL *)absoluteURL display:(BOOL)displayDocument error:(NSError **)outError
{
 NSAppleEventDescriptor * aed = [[[NSAppleEventManager sharedAppleEventManager]
        currentAppleEvent]
        paramDescriptorForKeyword:keyDirectObject];
 if(aed != nil && cachedAppleEventDescriptor != nil && ![[aed data] isEqualToData:[cachedAppleEventDescriptor data]])
 {
    cachedAppleEventDescriptor = aed;
    documentsPendingOpen = [aed numberOfItems];
 }
[..]

XML command line tools in Mac OS X (1)

2008-05-25T09:28:00.004+03:00

xmllint

xmllint is a parser for XML files that allows you to check for the well-formed state of the input.

xmllint can re-format the output, taking one long line of unreadable input and indenting it in a more readable form:


diciu$ cat test.xml 
<?xml version="1.0" encoding="utf-8"?>

<a> <m> <p>one, two, three</p> </m> <m> <p>three, four</p> </m> <m> <p>five</p> </m> </a>

cristi:~ diciu$ xmllint --format test.xml 
<?xml version="1.0" encoding="utf-8"?>
<a>
  <m>
    <p>one, two, three</p>
  </m>
  <m>
    <p>three, four</p>
  </m>
  <m>
    <p>five</p>
  </m>
</a>

In the Leopard versions, xmlllint and most of the other XML tools have a built in XPath browser implemented through a shell, allowing you to browse an XML's file hierarchy of nodes just like you would browse a filesystem:


diciu$ cat test.xml 
<?xml version="1.0" encoding="utf-8"?>

<a>
<m>
 <p>one, two, three</p>
</m>

<m>
 <p>three, four</p>
</m>

<m>
 <p>five</p>
</m>
</a>

The built in commands (viewable with the help command), include cat, cd, ls, pwd that work in similar ways to their Unix cousins:


diciu$ xmllint --shell test.xml 
/ > cat a
 -------
<a>
<m>
 <p>one, two, three</p>
</m>

<m>
 <p>three, four</p>
</m>

<m>
 <p>five</p>
</m>
</a>
/ > cd a
a > pwd
/a
a > cd m[position()=1]
m > pwd
/a/m[1]
m > ls
t--        2   
---        1 p
t--        1

Testing your product's resilience to network errors

2008-05-19T18:03:00.007+03:00

Sometimes it's useful to test your product in adverse network conditions. Networking is a very complex layer and it's fortunate that the popularity of TCP throughout the last decade shields us from the implementation that has been moved within the kernel code.
Many bugs in applications using networking are obscured by the developer's use of a fast network - he may use the product on a daily basis and still not hit certain conditions that are only met in conjunction with using the product on a slow network.

If your product has auto-update functionality, there's some code that will try to query the home server for a new version. This usually occurs over HTTP (that's how Sparkle does it).

Have you ever wondered how your product's auto-update code behaves when there's a 10 second delay on the network?

The IP firewall running on your Leopard box makes it very easy to find out.
Here's how you create a 10 second delay for your HTTP traffic towards a given host:

First we configure a pipe that delays packets by 10000 milliseconds:


sudo ipfw pipe 1 config noerror delay 10000

Then we redirect the traffic we want to delay through that given pipe:


sudo ipfw add prob 1 pipe 1 tcp from any to www.home.com http

Listing the rules from the IP firewalls shows the new rule we've just added:


cristi:~ diciu$ sudo ipfw list
Password:
00100 pipe 1 tcp from any to 192.168.1.1 dst-port 80
65535 allow ip from any to any

Testing with network delays

Once you've configured the firewall, you can test your product. As a side note, the auto-update from Sparkle behaves admirably - it will run the query in the background so the 10 second delay is not experienced within the application.

Resetting the delay rule

To clear the ipfw rules once you're done testing:


cristi:~ diciu$ sudo ipfw flush
Are you sure? [yn] y

Flushed all rules.

FrameAway 3.0

2008-04-30T09:20:00.005+03:00

A preview on how FrameAway version 3.0 looks at this point:

LoudHush weird message when auto updating?

2008-04-23T09:11:00.003+03:00

If you're seeing a message such as the one below when auto-updating LoudHush, the most likely cause is that you have renamed the LoudHush application.

The update archive didn't contain an application with the name I was expecting (/var/tmp/tmp.0.xz9xho/LoudHush 2.app). Remember, the updated app's file name must be identical to the running app's filename!

UDP traffic over 3G modem cut after 2 minutes?

2008-04-16T15:03:00.004+03:00

On Leopard 10.5.2, UDP traffic over the Vodafone 3G Modem is cut after 2-3 minutes.
I'm not sure what's to blame, symptoms are

ping showing a message I've never seen before


some-macbook-pro:~ u$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
ping: sendto: No buffer space available

and a message in the system.log


dlil_output: output error retval = 37

The solution that works in my case is to stop the Cisco VPN client:


sudo SystemStarter stop CiscoVPN

And this is the thread over at Apple Discussions that discusses the problem and possible solutions.

MacBook's life expectancy decreasing steadily

2008-04-10T10:22:00.002+03:00

Google shows 91 results when searching for "disk0s2: 0xe0030005 (UNDEFINED)"

Make this the 92nd:


Apr 10 09:05:15 cristi kernel[0]: disk0s2: 0xe0030005 (UNDEFINED).
Apr 10 09:05:15 cristi kernel[0]:
Apr 10 09:05:24: --- last message repeated 1 time ---

cgdisplayfade

2008-04-01T18:31:00.006+03:00

The code snippet below fades the display to a solid blue color and back


cd:~ diciu$ cat testfade.c 
#include <stdio.h>
#include <ApplicationServices/ApplicationServices.h>

int main()
{
 fprintf(stdout, "Fading to blue...\n");

 CGDisplayFadeReservationToken token;
 CGError err = CGAcquireDisplayFadeReservation(10, &token);
 CGDisplayFade(token, 2.0, kCGDisplayBlendNormal, kCGDisplayBlendSolidColor, .1, .3, 1.0, true);
 CGDisplayFade(token, 2.0, kCGDisplayBlendSolidColor,kCGDisplayBlendNormal, .1, .3, 1.0, true);
 err = CGReleaseDisplayFadeReservation(token);
}
cd:~ diciu$gcc  -framework ApplicationServices testfade.c
cd:~ diciu$ ./a.out 
Fading to blue...

Core Image filter for overexposed areas

2008-03-31T13:02:00.012+03:00

Because cameras are not as smart as they should be when it comes to dynamic range, neither from a hardware nor software standpoint, and mostly because I'm not very good at using my camera, I sometimes get overexposed areas in my photos. An overexposed area is an area where pixels are all-white (i.e. lit up all the way). If the overexposed area is large enough, the photo gets an unpleasant "burned" look.

In our sample photo on the left side, the overexposed area is caused by the camera facing the light source - a common reason for overexposures as the camera's software computed the exposure based on the tulip's shaded side.

You can detect overexposed areas in a number of ways - on my Nikon D70 there's a highlights mode that blinks overexposed areas and there's a histogram on the green channel - the histogram less reliable because it does not show the red and blue channels as well so a relatively big value in the right hand side of the histogram might not necessarily mean an area has been overexposed.

Overexposed ares show up in the photo's histogram (see below for a RGB histogram - the white bar on the right side represents overexposed pixels - nearly 10% of all the photo's pixels have been overexposed).

Using Core Image filters, overexposed areas are very easy to identify.
With Core Image, using a histogram doesn't make sense - building a histogram is a computationally intensive task because it requires going through large amounts of data (a 6MB photo has, yes, 6MB of data) - it's better to write a filter that highlights the areas that have been overexposed.

Here's my (possibly naive) solution:


kernel vec4 replaceOverExposed(sampler image, __color color)
{
 vec4 p = sample(image, samplerCoord(image));
 float multiplier = 1;
 
 float level = (p.r + p.g + p.b) / 3.0;
 
 vec4 res;
 res.r = level > .99 ? 1.0 : p.r;
 res.g = level > .99 ? 0.0 : p.g;
 res.b = level > .99 ? 0.0 : p.b;

 
 return vec4(res.r, res.g, res.b, p.a);
}

Here's a rendering of the overexposed filter above applied on the tulip photo.
The filter has replaced all the overexposed pixels with a red pixel (p.r 1.0, p.g 0.0 and p.b 0.0).

About LoudHush 1.3.15

2008-03-28T08:31:00.004+02:00

We're run into an incompatibility issue with version 1.3.15 that caused it not to run on non-Leopard systems (10.4, 10.3).
We have fixed this issue and are right now working to test the fixed version.

In the meantime, we've pulled 1.3.15 from the site and from the auto-updater application cast.

We will publish 1.3.16 with a correction for the problem as soon as possible.

If you've end up with a non-functioning 1.3.15, please re-download the old version from the http://www.loudhush.ro site.

Sorry for the trouble.

New icon for LoudHush (1.3.15)

2008-03-27T17:29:00.003+02:00

The LoudHush icon has finally been updated to Leopard requirements (512x512).

Exif Orientation

2008-03-17T08:53:00.001+02:00

Orientation
The image orientation viewed in terms of rows and columns.
Tag = 274 (112.H)
Type = SHORT
Count = 1
Default = 1

1 = The 0th row is at the visual top of the image, and the 0th column is the visual left-hand side.
2 = The 0th row is at the visual top of the image, and the 0th column is the visual right-hand side.
3 = The 0th row is at the visual bottom of the image, and the 0th column is the visual right-hand side.
4 = The 0th row is at the visual bottom of the image, and the 0th column is the visual left-hand side.
5 = The 0th row is the visual left-hand side of the image, and the 0th column is the visual top.
6 = The 0th row is the visual right-hand side of the image, and the 0th column is the visual top.
7 = The 0th row is the visual right-hand side of the image, and the 0th column is the visual bottom.
8 = The 0th row is the visual left-hand side of the image, and the 0th column is the visual bottom.
Other = reserved

Exif2.2 Spec

Backtrace call in 10.5

2008-03-03T12:11:00.003+02:00

backtrace() writes the function return addresses of the current call stack to the array of pointers referenced by array.

man backtrace