US Soccer History

Perseverance rating: A better way to statistically evaluate soccer goalkeepers

Featured image: Courtesy of nasljerseys.com

In 2008, I wrote the following article for a blog post on BigSoccer. With the recent publication of The Numbers Game: Why Everything You Know About Soccer Is Wrong (Anderson, Chris and David Sally), as well as last year’s Moving The Goalposts by Rob Jovanovic (referenced heavily in my series on top U.S. club sides last winter), I thought it would be a good time to revisit some of the concepts I discussed five years ago and see if there was any interest in further examining a better way to evaluate goalkeepers.

Unlike pretty much every other sport in America, soccer is not one that lends itself to a lot of statistical analysis. In other countries, player statistics are limited to games played and goals scored or, if you are a goalkeeper, games played and “clean sheets.” There are also “match ratings,” a completely subjective evaluation of a player’s performance in games. Otherwise, there is not much else to base a fantasy league on.

Of course, since we are Americans, we have done our best to foist statistics upon the Beautiful Game. This may be an attempt to make the game more palatable to the infamous “casual fan,” or has perhaps come about because, as soccer fans, we are jealous of fans of other sports’ ability to babble on incessantly about so-and-so’s on-base percentage or what’s-his-face’s quarterback rating. In any event, only in America will you find such nebulous statistics as “touches,” “times fouled,” “chances created,” and other data designed to more fully measure a player’s contribution to a team’s performance. As Billy Beane becomes more involved with the San Jose Earthquakes, we may see more of these “moneyball”-type concepts enter scouting reports and other player evaluations.

Even before these New Age stats came about, however, North American soccer has always included two statistics not seen in the rest of the world: assists, and goals against average. Both borrowed from ice hockey, neither has found great acceptance among the world football community. While this is somewhat surprising in the case of assists–as fans appreciate great vision and service from players, one would think that a stat rewarding such talents would be in regular use by now–it is completely understandable when it comes to GAA.

It is universally accepted among soccer fans that goals against average is an especially deceiving statistic. Unlike in hockey, where it is easier to create quality shots on goal and, thus, make a goalie earn his money, in soccer a low GAA can be attributable to a number of factors which have little to do with the goalkeeper’s value. For example, was his team so dominant in possession that the other team had few shots on goal? Did his team utilize an aggressive offside trap, limiting the other team’s offensive chances? Did he play during a particularly defensive era? Any one of these factors can make a goalkeeper statistically impressive. Conversely, a goalkeeper saddled with a particularly porous defense can suffer from a high GAA which would make him look bad, even though fans could see that he was a tremendously acrobatic player who kept his side in games with more tremendous saves than one had a right to expect

As a result, evaluation of goalkeepers in MLS (and, before it, the North American Soccer League) has been largely anecdotal. Among my memories of the NASL is the constant lauding of goalkeepers such as Mick Poole and Jan Van Beveren as among the best in the world, even though their GAAs were decidedly average, at best, while perennial league leaders in GAA such as Shep Messing or Phil Parkes were deemed overrated.

Even with the wider acceptance of GAA in hockey, however, similar debates raged. A few famous examples include Denis Herron and Gilles Meloche, two goalies who labored for the worst teams in NHL history (the Kansas City Scouts and California Golden Seals, respectively). Each often suffered with inordinately high GAAs but, claimed most pundits, either would have each been the second coming of Glenn Hall if they just had a defense in front of them.

Two individuals who questioned the true value of GAAs in hockey were Jeff Klein and Karl-Eric Reif. In their seminal work, The Hockey Compendium, the two created a new statistic designed to measure and reward a goalie’s “perseverance.” Although the NHL had adopted the “save percentage” statistic, this, too, was found wanting, as it overly-credited a goalkeeper who was fortunate enough to face fewer shots. For example, if you have two goaltenders with a .900 save percentage, who deserves the greater credit: one who saved 9 out of 10 shots, or one who saved 45 out of 50?

To try to address these issues, Klein and Reif devised a “perseverance rating” statistic. Basically taking total number of shots faced and shots faced per game, the duo constructed the formula as follows:

To find Efficiency:

(Saves divided by Total Shots Faced) X 100

To find Shots Faced per Game:

Total Shots Faced divided by (Minutes Played divided by 60)

To find Perseverance Rating:

(Efficiency X 6) + Shots Faced per Game = µ

µ divided by .6 = Perseverance Rating

Using this formula, Klein and Reif were able to determine which goaltenders were able to earn their money, and which were simply benefiting from strong, close-checking defenses.

****

I thought Klein and Reif’s approach was as close to perfect as one could get, and set about applying it to soccer goalkeepers (obviously adjusting the formula for minutes played and multipliers along the way). However, I immediately encountered a problem: the perseverance numbers tended to skew wildly from year to year, and, especially, when comparing the NASL with MLS. Whether the NASL was a more offensive league or the league’s statisticians were overly generous with saves, the result was that the worst NASL goalkeeper in a given year achieved a higher Perseverance Rating than the best MLS goalkeeper in a given year. This was especially problematic for me, as I like to write about and compare American soccer players from different leagues and eras.

As a result, I borrowed yet another concept from hockey: plus/minus. Specifically, I calculated a league’s seasonal Perseverance Rating, and then calculated a PR plus/minus by taking an individual goalkeeper’s PR and subtracting the overall season PR from that. In this way, one could more accurately judge a goalkeeper’s performance in a season, but could also use the +/- rating to judge his performance historically. Specifically, the +/- figure tended to equalize differences in shots faced which might be attributable to either generous stat keeping (as in the NASL) or different “fads” in styles of play (or, in MLS’ case, the gradual improvement in the quality of play) which might effect statistics from year to year.

(Note: after first writing this in 2008, I came across the revised edition of the Compendium, wherein Klein and Reif also recognized the problem and adopted the above adjustment—to give credit where it’s due, they did it first. I just didn’t know it in 2008)

Applying the modified Klein/Reif Perseverance Rating to soccer goalkeepers revealed some interesting results, and tended to indicate that it would more accurately a goalkeeper’s performance.

A good example is found in MLS’ 2004 season. That year, the league’s leading goalkeepers were as follows:

min. 1000 minutes

Name

Team

Min.

Svs

GA

ShO

GAA

Nick Rimando D.C.

1170

26

13

4

1.00

Tony Meola Kansas City

1890

78

22

7

1.05

Joe Cannon Colorado

2700

150

32

10

1.07

Jon Busch Columbus

2610

132

31

10

1.07

Pat Onstadt San Jose

2250

98

32

6

1.28

Kevin Hartman Los Angeles

2655

117

39

7

1.32

Matt Reis New England

2115

97

32

3

1.36

Henry Ring Chicago

2520

128

40

7

1.43

Jeff Cassar Dallas

1632

65

27

4

1.49

Scott Garlick Dallas

1068

52

18

4

1.52

Jonny Walker New York

2520

99

45

5

1.61

Troy Perkins D.C.

1440

52

26

3

1.63

If GAA were the only indicator, we would believe that Rimando was a key element in D.C. United’s title run, while Henry Ring was a below average performer for Chicago.

However, the Perseverance Ratings reveal the following leaders:

Name

Team

Min.

Svs

GA

ShO

PR

Joe Cannon Colorado

2700

150

32

10

834.29

Jon Busch Columbus

2610

132

31

10

819.18

Tony Meola Kansas City

1890

78

22

7

787.94

Henry Ring Chicago

2520

128

40

7

771.90

Pat Onstadt San Jose

2250

98

32

6

762.51

Matt Reis New England

2115

97

32

3

761.09

Kevin Hartman Los Angeles

2655

117

39

7

758.81

Scott Garlick Dallas

1068

52

18

4

752.69

Jeff Cassar Dallas

1632

65

27

4

714.98

Jonny Walker New York

2520

99

45

5

696.07

Troy Perkins D.C.

1440

52

26

3

674.79

Nick Rimando D.C.

1170

26

13

4

671.67

He’s an unquestioned great today, but in 2004 Rimando was kind of along for the ride in D.C.’s MLS Cup run

The Perseverance Ratings suggest that D.C.’s title was largely attributable to a high-possession offense which resulted in few quality opportunities on goal for the opposition, which, apparently, was a good thing, given the dismal PRs registered by Troy Perkins and Nick Rimando. Ring, on the other hand, reveals himself to be a goalkeeper who did his best in the face of a heavier workload.

But does this mean that Ring had a great year, or that Rimando had a really bad one? Not necessarily, when it is seen that Ring’s PR +/- was +7.05; in other words, he was just above average. Rimando, with a PR +/- of -93.18, was absolute rubbish–notwithstanding his league-leading 1.00 GAA. In fact, Rimando’s +/- may be the worst ever for an MLS goalkeeper.

Does this mean Rimando is an overrated stiff, unworthy of playing in MLS? Let’s take a look at 2007. The GAA leaders for that year were:

min. 1000 minutes

Name

Team

Min.

Svs

GA

ShO

GAA

Pat Onstadt Houston

2418

85

22

11

0.82

Brad Guzan Chivas USA

2430

87

25

13

0.93

Bouna Coundoul Colorado

2668

120

32

9

1.08

Jon Conway New York

1143

57

14

4

1.10

Troy Perkins D.C.

2610

117

32

8

1.10

Matt Pickens Chicago

2430

102

31

10

1.15

Nick Rimando SaltLake

2430

146

37

7

1.37

Matt Reis New England

2700

120

43

10

1.43

Will Hesmer Columbus

1800

71

29

5

1.45

Kevin Hartman Kansas City

2700

110

45

5

1.50

Joe Cannon Los Angeles

2610

119

46

5

1.59

Dario Sala Dallas

1620

64

29

5

1.61

Ken Stamatopoulos Toronto

1080

41

20

2

1.67

Ronald Waterreus New York

1557

59

31

5

1.79

As we can see, Mr. Rimando is smack dab in the middle of the pack, and nothing to write home about as far as GAA is concerned. His PR rating shows, however, that his performance in goal for the Royals was much better than his GAA would indicate:

Name

Team

Min.

Svs

GA

ShO

PR

Jon Conway New York

1143

57

14

4

812.13

Nick Rimando SaltLake

2430

146

37

7

809.11

Pat Onstadt Houston

2418

85

22

11

801.03

Bouna Coundoul Colorado

2668

120

32

9

798.02

Troy Perkins D.C.

2610

117

32

8

793.80

Brad Guzan Chivas USA

2430

87

25

13

783.70

Matt Pickens Chicago

2430

102

31

10

775.13

Matt Reis New England

2700

120

43

10

745.25

Joe Cannon Los Angeles

2610

119

46

5

730.69

Will Hesmer Columbus

1800

71

29

5

718.33

Kevin Hartman Kansas City

2700

110

45

5

718.29

Dario Sala Dallas

1620

64

29

5

696.78

Ken Stamatopoulos Toronto

1080

41

20

2

680.60

Ronald Waterreus New York

1557

59

31

5

664.23

Thus, according to PR, Rimando had a great year playing for the next-to-worse team in MLS. In addition, it appears the Red Bulls made a real mistake in giving Ronald Waterreus so much playing time, as Jon Conway’s PR was the best in the league. Also surprising is Brad Guzan’s rather mediocre PR, despite 13 shutouts; apparently, he got to sit back and enjoy the view in many of the Goats’ matches.

Furthermore, both Rimando and Conway significantly outperformed the league average, best represented by Matt Reis’ 745.25 PR (a -7.86).

The ever-acrobatic Bob Rigby

As anticipated, the PR +/- was most useful in comparing goalkeeper performances in different eras. For example, Bob Rigby’s rookie season in 1973 for the Philadelphia Atoms resulted in the current U.S. Division One professional record for GAA in a season, 0.62. As might be expected, Rigby also led the NASL in PR that year, with a 918.13 rating. As this puts him well ahead of the league-leading PRs posted by Cannon and Conway in 2004 and 2007, respectively, one would think it proves that Rigby did, in fact, have one of the greatest seasons for a goalkeeper ever. However, Rigby’s 1973 PR was only +56.62 above the 1973 PR average; thus, in comparison with their respective peers, Cannon’s +69.44 in 2004 and Conway’s +59.02 in 2007 were statistically equal to, if not better than, Rigby’s 1973 performance.

Do you want an example of a truly dominant season? Tino Lettieri’s breathtaking 1983 performance with the Vancouver Whitecaps appears to be the greatest season ever by a goalkeeper in a major professional league. That season, he not only led the NASL with a 0.87 GAA, but also led all qualifying goalkeepers with a 874.77 PR–a +112.68 for the season. This, by itself, is stunning; the fact that it was 82.28 points higher than the next best PR reveals that Lettieri was truly in a class by himself in the North American Soccer League’s penultimate season. By comparison, Rigby’s 1973 PR was only 20.38 points better than his closest competitor, Dallas’ Ken Cooper…or, as we now have to refer to him, Ken Cooper, Sr.

(By the way, I humbly submit that, if statistics for the era were available, Findlay Kerr’s 1924-25 season for the Fall River Marksmen—when he posted 20 shutouts and an 0.86 GAA in an extremely offensive-minded year which saw, among other things, Archie Stark score 67 goals in 44 games—would be the most statistically dominating single-season performance by a goalkeeper in a major American league. As it is impossible to reconstruct save figures from game reports of the era, however, this will forever remain conjecture.)

Speaking of the NASL, remember how I remarked how goalkeepers like Van Beveren and Poole were lauded by pundits of the era as true greats, while Shep Messing was considered by those same pundits to be little more than an overrated showboater and Phil Parkes was said to benefit from playing behind great defenses? The following are the PRs of all NASL goalkeepers with at least 10,000 minutes played, along with the PR +/- compared with the NASL’s career PR.

Name

Team(s)

Min.

Svs

GA

ShO

GAA

PR

+/-

Shep Messing Bos-NY-Oak-Roch

10953

905

183

25

1.50

846.70

46.06

Hubert Birkenmeier New York

13254

883

209

30

1.42

820.97

20.33

Zeljko Bilecki Tor-TB-LA-Tul

10234

644

153

37

1.35

819.71

19.07

Tino Lettieri Minn Kicks-Van-Minn Strk

14396

848

209

36

1.31

813.28

12.64

Jack Brand Tor-Roch-NY-Tul-Sea-TB

11652

763

191

31

1.48

812.07

11.43

Ken Cooper Dallas

15435

928

231

49

1.35

811.95

11.31

Alan Mayer Balt-LV-SD-Calif

14849

1176

304

31

1.84

809.55

8.91

Tony Chursky Seattle-Calif-Chic-Tor

13148

880

232

38

1.59

804.05

3.41

Mick Poole Den-Port

10910

773

207

16

1.71

802.25

1.61

Volmar Gross San Diego

12803

802

215

33

1.51

800.51

-0.13

Bob Rigby Phil Atoms-NY-LA-Phil Fury-Mont-GB

19550

1341

366

37

1.68

798.69

-1.95

Phil Parkes LA (USA)-Van-Chic-SJ-Tor

12355

660

178

33

1.30

797.76

-2.88

Paul Hammond TB-Hou-Sea-TA-Tor

16402

1032

282

40

1.55

797.40

-3.24

Arnie Mausser Hart-TB-Van-Col-Ft.L-NE-Jack-TA

20556

1495

414

35

1.81

797.06

-3.58

Mike Hewitt TB-SJ-Mont

15672

1023

313

38

1.80

778.51

-22.13

Jan Van Beveren Ft.Lauderdale

10148

658

206

17

1.83

774.35

-26.29

Winston DuBose Tampa Bay-Tulsa

14523

795

266

31

1.65

760.25

-40.39

Bill Irwin Wash-Dal-Port-GB

16449

921

311

26

1.70

758.80

-41.84

Shep Messing–Bronx-born U.S. soccer legend…and the best goalkeeper in NASL history

Surprisingly, using PR, we see that Shep Messing was, in fact, the best goalkeeper in NASL history, sporting not only the best lifetime PR, but also a +/- which puts him over 25 points higher than the next best keeper, the often (and, as it turns out, justly) lauded Hubert Birkenmeier. Meanwhile, it turns out the perception of Parkes was entirely accurate; although he has the best career GAA of the group, it appears he did little to earn it as he was slightly below average at stopping the shots he had to face.  Most surprisingly, Jan Van Beveren–who routinely finished near the top of most Soccer Digest “best goalkeeper” polls–is revealed to be incredibly overrated, more than 25 points behind the NASL lifetime average. Similarly, Arnie Mausser–long time U.S. National Team goalkeeper and the only Hall of Fame member on the list–is revealed to be a slightly-below average goalkeeper. This begs the question: if Mausser is in the Hall, aren’t Messing, Birkenheimer, Lettieri and the largely-forgotten Zeljko Bilecki also deserving of induction?

Is the Perseverance Rating and the PR Plus/Minus the final word on evaluating goalkeepers? Given the fact that saves remain a rather fluid statistic, and the fact that team play in soccer can effect the number of shots on goal much more than in hockey, both probably remain less than perfect. Also, a goalkeeper who enjoys little work because of his team’s aggressive offside trap may find himself facing much more dangerous opportunities (i.e., breakaways) once the opposition has broken the trap, resulting in a higher goal-to-shot ratio (which might explain, but not totally excuse, Rimando’s dismal 2004 PR numbers). The effect of penalty kicks is also hard to judge. All told, however, PR and PR Plus/Minus remain far superior to GAA as an evaluator of a goalkeeper’s true worth to his team, and it would be nice to see the PR and PR +/- adopted as a soccer statistic.

(So what’s the point of revisiting all this five years later? Join us next time, when we apply the metric to the goalkeepers the Union have employed in their short history.  The results will shock you…)

4 Comments

  1. Huge Hockey Fan says:

    The article is an interesting effort, but it overcomplicates the analysis. Save percentage is the key stat in evaluating GK performance in the NHL. I passionately follow the NHL / watch the NHL Network etc. and you virtually never hear the Perseverance Rating referenced or cited. Not saying I’m ignorant and never heard of the metric, I have, but it never gained traction in the evaluation of hockey goaltenders. That fact makes the piece a touch irrelevant.

  2. hdjcbujwbrosBGjebsf

Leave a Reply

Your email address will not be published. Required fields are marked *

*