A Comparative Evaluation of Parsers

Robustness to ungrammaticality

Test Sentences

• I is a girl.
• Ram reading.
• I is was a girl.
• I been a girl.
• I was being girl.
• I was be girl.

Ranking in case of multiple parses

Test Sentences

• I saw the boy with a telescope.
• Time flies like an arrow.
• I once shot an elephant in pajamas.
• I saw her duck.
• I said I would see you on Tuesday.
• They hit the man with a cane.
• The old men and women left the room.
• Bill sold the invisible man's hat.
• Shoppers hate annoying beggars.

RASP differs from Stanford in "I once shot an elephant in pajamas".

Time taken

Test Sentences

Stanford 2.35 min RASP 9.0 secs Collins 1.3 sec Charniak 1.0 sec

How efficiently is embedding handled

Test Sentences

• The cat that killed the rat that stole the milk that spilled on the floor that was slippery escaped.
• Alice was just beginning to think to herself, ‘Now, what am I to do with this creature when I get it home?’ when it grunted again, so violently, that she looked down into its face in some alarm.
• I said I would see you on Tuesday.
• She came to the party after he left.
• He told us where to go.
• She belongs to a family which is facing poverty which is a problem facing India.
• Sachin who plays cricket which is the best game of India which is a country with varied heritage, likes playing tennis.

How effectively is multiple POS handled

Test Sentences

• Seeing is believing.
• Two share a house devoid of furniture.
• Time flies like an arrow.

Only Stanford and charniak give the correct parse structure for the first sentence. In second sentence, Charniak parser tags share as a noun since it is trained on Wall Street Journal, where share appears more in context of trading shares.

Can it handle repeated words with changing POS

Test Sentences

• Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo.
• Black cows brown cows cow cow white cows.
• Above orders are from above

Stanford RASP Collins Charniak

Length of sentence it can handle

Test Sentences

• Knowing that millions of people around the world would be watching in person and on television and expecting great things from him ? at least one more gold medal for America, if not another world record ? during this, his fourth and surely his last appearance in the World Olympics, and realizing that his legs could no longer carry him down the runway with the same blazing speed and confidence in making a huge, eye-popping leap that they were capable of a few years ago when he set world records in the 100-meter dash and in the 400-meter relay and won a silver medal in the long jump, the renowned sprinter and track-and-field personality Carl Lewis, who had known pressure from fans and media before but never, even as a professional runner, this kind of pressure, made only a few appearances in races during the few months before the Summer Olympics in Atlanta, Georgia, partly because he was afraid of raising expectations even higher and he did not want to be distracted by interviews and adoring fans who would follow him into stores and restaurants demanding autographs and photo-opportunities, but mostly because he wanted to conserve his energies and concentrate, like a martial arts expert, on the job at hand: winning his favorite competition, the long jump, and bringing home another Gold Medal for the United States, the most fitting conclusion to his brilliant career in track and field.
Long Sentence 2: Good Girl