Introducing Bridge ACES – The Advanced Convention Evaluation System
Ever since the early days of bridge, our mastery of card play and defense has far outpaced our understanding of optimal bidding methods. At the world championship level, far more results are decided in the auction rather than in the play. It is relatively easy to get advanced players to agree on the optimal play or defense on a given hand, especially in hindsight or post-mortem. As for bidding, however, a cursory read of articles like “Master Solvers Forum” in The Bridge World, or “It’s Your Call” in the ACBL Bulletin, shows how little agreement there is in the expert community on the right action on most issues. Bidding systems undergo constant innovation and tweaking, and there is no consensus even on some basic questions such as, say, whether 2/1 is a better system than Precision. Advances in play or defense, on the other hand, are rare, and typical problems found in the bridge literature start having an air of sameness to the avid reader.
The same disparity exists in the confluence of computers and bridge. Today’s computer programs, such as the BBO Bots, do quite an adequate job on play and defense, and their shortcomings, too, are well understood and analyzed. On bidding, however, the state of the art is fairly primitive, and the Bots often reach absurd contracts. Even the ability to specify and articulate a bidding system is not sufficiently developed. Current software codifies a bunch of commonly used conventions and allows some level of customization and options, but does not permit users to describe new conventions or make modifications beyond the limited vocabulary of the system.
There is vast, untapped potential for using computers to help our understanding of bidding and for making better bidding programs, and we can define enough projects to keep a small army of bridge software enthusiasts busy for quite a while. Rather than tackling the whole space, Bridge ACES focuses on a subset of the larger problem, while still producing something that is useful to the bridge community and that can be leveraged by other efforts in advancing bidding software.
Bridge ACES tackles the problem of specifying and evaluating bridge systems and conventions. As a by-product, it creates a software that will bid hands using a user-specified system with a combination of human and computer bidders, and provides an analysis of how good the final contract is.
The problem of evaluating a bidding system is very broad in scope. The first phase of the ACES project is restricted to non-competitive bidding. The hand generation, bidding, and the contract evaluation all make the assumption that East-West do not enter the bidding, which further implies that they don’t hold hands that would warrant a bid at any stage. Of course, the problems of competitive auctions are far more complex, interesting, and frequent, but the software to handle them is an order of magnitude harder to develop. Hence competitive auctions have been deferred to the second phase of the project.
The project also makes no effort to limit the information passed to the opponents by the auction, and simply tries to evaluate the final contracts reached. This, too, is a choice driven by implementation difficulties, and could be addressed in a subsequent phase.
Finally, the evaluation can be done either for an entire bidding system, or a part of the system, or for a specific convention or treatment. The limitation is in the effort required to specify the system or convention. Of course, specifying a complete system is far more work than a single convention. As an example of evaluating a single convention, a partner of mine mentioned that after a 2N opening, it may be useful to super-accept a transfer even with just three card support, rather than only with four. This was one of my first experiments with ACES – to compare auctions given the two treatments for super-acceptance. ACES showed, in this case, that the three-card super-acceptance loses about 1 IMP per deal across all 2N-xfer-to-major sequences.
The first key step is to agree on the definition of success, and that too is far from trivial. A theoretically ideal system will reach the optimal contract on each deal. What does this really mean? If, on a given hand, South can make 6N because three finesses happen to work and a key suit breaks three-three, is that really the optimal contract for the hand? Would we say that a system that reaches 6N on that hand is better than one that stops in game, even for that specific hand? Of course we would not, even though the hand record would tell us that 6N is the par result.
So we distinguish between the double-dummy par on a deal, which is the best result with all cards known, and the single-dummy par, which is the best contract with only North-South hands known. This single-dummy par is the result we should hope to achieve on each deal, and what ACES uses to evaluate systems and conventions. Note, however, that this is the correct measure ONLY for situations in which opponents are completely silent. In competitive scenarios, the problem becomes more complex, since the opponents have revealed more information, and there is the element of preventing them from reaching their par.
To evaluate the single-dummy par, the program keeps the North-South hands fixed and generates a large (configurable) number of East-West hands. It then analyzes the hands using Bo Haglund’s Double Dummy Solver to evaluate the score for each contract, and averages the contracts for all the hands. So for instance, a 4S contract that makes 9 or 10 tricks depending on a single finesse will score an average of (420 – 50)/2, or 185 points non-vulnerable. A 3S contract with the same cards will score an average of (140 + 170)/2, or 165 points on the same hand. This method gives us the average expected score, or eScore, for any contract.
With this approach, we can define an overall system score, or oScore, as the total eScore achieved by the system over a large collection of hands, divided by the total of the single-dummy par eScore for the same set of hands, expressed as a percentage. Hence a “perfect system” is one which will reach the single-dummy par on every deal and get an overall score of 100. Any system can be compared with the perfect score and with one another using this approach.
This approach is far from perfect. It is most accurate for aggregate scoring, and reasonably accurate for IMPs, where the conversion from raw to IMP scores will have some effect on the final evaluations. For matchpoints, the results are even less reliable. A system that will reach 40% vulnerable games will get a better overall score than one that only reaches 50% or better games, while clearly, the former is inferior for matchpoints. It is difficult to come up with accurate predictions for matchpoint scores, since they also depend on the fraction of the field that reaches each possible contract. We will defer this problem to a second phase, and welcome suggestions on how to award matchpoint scores.
So given these methods, what is the range of oScores that we can expect from bidding systems? What is the oScore, say, for SAYC or Precision? This last question is one of the objectives of this project, although we are far from reaching this answer yet. Let us first discuss the range. The maximum oScore possible is 100, which is when the system always reaches the single-dummy par. Practically, this only happens if North-South show their hands to each other, so the real goal is to come as close to it as possible. On the other extreme, what is the lower bound of this range? Theoretically, you could even get negative scores by bidding unmakable contracts all the time, or an oScore of 0 for a system that passes every hand, but these are not useful exercises. Another way of looking at a minimum is to think of a system that passes zero information to partner, in effect, having a player guess the final contract blindly, knowing only, in this instance, that opponents are silent and hence don’t have the values to bid. Using this approach, the software ran simulations with only one hand known, generating a large number of permutations for the other three hands. The resulting contracts generated an oScore of 35. Any reasonable system will generate some useful information for partner and hence improve on this. Hence systems should have an oScore somewhere between 35 and 100 (probably far from either extreme). We will get more specific about this as we make more progress in the project.
The most complex part of the project, apart from developing the software itself, is the framework for specifying the bidding system and bidding the hand. Again, we have used an incremental approach, starting with a simple framework and adding more functionality in phases.
The primary construct in the specification is a rule. A rule is a logical condition that can be applied to a single hand and evaluates to True or False. ACES defines a simple, intuitive language for specifying rules, comprising keywords such as Points, Spades, or shape, logical operators such as ‘>’ (greater than), ‘?=’ (is equal to), or ‘!=’ (does not equal), and parentheses and conjunctions (‘AND’ and ‘OR’). Each rule has a name that must start with a ‘$’ sign. It is set through the assignment operator ‘:=’, and is terminated by a semicolon and new line. This allows us to combine rules to create more complex rules. It is best to look at some examples:
$balanced := shape [4,3,3,3] OR [5,3,3,2] OR [4,4,3,2];
$StrongNT := (15 TO 17 Points) AND $balanced;
$WeakNT := (12 TO 14 Points) AND $balanced;
$1Sopening := (12 TO 20 Points) AND (Spades > 4) AND NOT $StrongNT;
The rules listed above are purely illustrative: you could refine the definition of balanced, for instance by allowing [6,3,2,2] shapes with a six card minor or [2-2-4-5] or [2-2-5-4] pattern, if such is your partnership styles.
While a rule does not necessarily describe a single bid, a specific naming convention associates rules with bids. Each bid is in the context of the auction, so it is necessary to specify the entire auction up to that bid. We do this by separating the bids by periods (‘.’), such as $.1S. for a 1S opening, and $.1N.2C.2H.4H. for the sequence 1N-2C-2H-4H. Keep in mind, this phase of the project assumes silent opponents; later, we will extend the syntax for contested auctions.
Given this syntax, we build a set of rules and use that to specify a system. When ACES bids a hand, it looks at the available rules at each step of the auction. If it finds a matching rule, it makes the corresponding bid, and goes to the next step. Of course, system specifications are never complete, since there are over sixteen billion unique sequences in uncontested auctions, and no one is going to sit and define each one of them. Hence at some point ACES will fail to find a rule, whereupon it will place the final contract.
ACES picks the final contract by looking the hand whose turn it is to bid and the information conveyed by partner in the auction so far, dealing a fixed (configurable) number of the other three hands consistent with the information, and evaluating results of all contracts. It picks, of course, the contract that fares best in this simulation. This is not too different from how an advanced player would try to judge the final contract, except that it is done through a more extensive analysis than by a human.
There are some key limitations of the above approach, which we hope to address in subsequent phases. The first is that it requires each bidding sequence to be specified individually, and has no syntax for general rules that can be applied to a number of bids. For instance, it would be useful to have a single rule for 1M-2M (M = major, m = minor), which ACES would translate to two separate rules, one for 1S-2S, another for 1H-2H. Or a rule that says that 4N is RKC after a fit is agreed.
Second, in our current approach, all bids provide information rather than ask for it. All bidding rules are of the form “If you have XXX, bid Y”. We don’t yet have a way of specifying asking bids, such as “To find out if your partner has XXX, bid Y”. The problem is around how to instruct ACES when you must make the asking bid. In some simple cases, we can frame it in the canonical form, such as $.1N.2C. showing, for instance, 8+ points, 4 card heart or spade, and no 5-card major.
Finally, once a final contract has been placed by the first hand that does not find a usable rule, the partner just accepts the result and passes. There often will be situations where the other hand has additional information and should know to correct to a different contract, but the current implementation does not allow that in the absence of specific rules.
With the current limitations, ACES is best suited to explore alternatives for specific bidding situations, rather than the more ambitious goal of comparing a full system. Two such comparisons were done for the first trial. The first, mentioned earlier, was to explore the cost/benefits of pre-acceptance with three-card support after 2N and a transfer to 3S. This vindicated the merits of the conventional approach – pre-acceptance with three cards cost an IMP per deal on average across the deals where the two treatments led to different results. The second comparison related to the bidding after 1S-2S, comparing long-suit game tries against short-suit game tries. Here, an extensive simulations found practically no statistically significant difference.
Over time, we will conduct other experiments and publish the results, while addressing the limitations of the software in parallel. At this time, we wanted to release our first ACES prototype and describe our effort to get some early feedback and suggestions in two particular areas: (1) other changes or improvements desired in the software, and (2) other problems worth posing to ACES that can be described relatively easily through the current rules framework.
You can download a copy of ACES at Downloadable zip file for ACES (Uresh's bidder program). To run it, extract all the files to a common directory and run ACES from a command prompt in that directory. The command syntax for ACES is
.\ACES –i input.txt
–o output.csv [–p deals.pbn] ndeals nrule $.P. srule $.P.
where input.txt contains all
the rules defined for ACES, output.csv
will contain the results of the analysis, and nrule
and srule are rules that define the restrictions on
the North and South hands respectively. Note that both nrule
and srule must be defined in input.txt.
The rule $.P. is used for the East and West hands, and translates to a hand
that does not have the values to open or overcall. The current input.txt has a very primitive
definition of $.P.; the user is welcome to refine
it. If the optional -p deals.pbn
parameter is used, deals.pbn contains a set of
input deals in PBN format, with North and South hands specified. These deals
will be used instead of randomly generated deals for the purposes of the
analysis. Note that you can change the filenames for input.txt, output.csv, and deals.pbn.
The download package also contains a set of example input files, plus batch files that illustrate the use of ACES to compare methods. These files are listed below:
|
1s2sLng.txt |
Long-suit game tries after 1S-2S |
|
1s2sSh.txt |
Short-suit game tries after 1S-2S |
|
2n3.txt |
3-card pre-acceptance after 2N-3H |
|
2n4.txt |
4-card pre-acceptance after 2N-3H |
|
Forcing1N.txt |
Bidding after 1S-1N-2C playing 1S-1N forcing |
|
NF1n.txt |
Bidding after 1S-1N-2C playing 1S-1N non-forcing |
|
input2Over1.txt |
Opening bids in 2/1 |
|
PrecOpen.txt |
Opening bids in Precision |
|
inputPrecision.txt |
Partial description of Precision |
|
1S1N2C.bat |
Comparing auctions after 1S-1N-2C, playing forcing or
non-forcing 1N. |
|
1s2s.bat |
Comparing long-suit and short-suit game tries after 1S-2S. |
|
2n3h.bat |
Comparing 3-card vs 4-card pre-acceptance after 2N-3H. |
|
Comparison Operators |
|
|
<= |
Less than or equal |
|
>= |
Greater than or equal |
|
< |
Less than |
|
> |
Greater than |
|
?= |
Comparison (is equal) |
|
!= |
Is not equal |
|
Boolean Operators |
|
|
“||” or “or” or “OR” |
OR |
|
“&&” or “and” or “AND” |
AND |
|
“NOT” or “not” |
NOT |
|
Other Operators and Keywords |
|
|
:= |
Rule definition |
|
“end” or “END” |
End of all rules (last line) |
|
“TO” or “to” |
Range specifier |
|
“shape” or “Shape” |
Shape specifier |
|
“pattern” or “Pattern” |
Pattern specifier |
|
Points |
Total HCP |
|
“spades” or “Spades” |
Spade length |
|
“hearts” or “Hearts” |
Heart length |
|
“diamonds” or “Diamonds” |
Diamond length |
|
“clubs” or “Clubs” |
Club length |
|
“(“ and “)” |
Parentheses for evaluation order |
General syntax:
Rule_name := rule;
Rule_name := rule;
…
end
Rule_name must start with a ‘$’ sign, followed by a string containing upper or lower case letters, numbers, or the characters ‘-‘, ‘_’, or ‘.’. To associate a rule with a bidding sequence, the rule must have the specific syntax
$.bid1.bid2.bid3....bidn.
e.g., $.P.1C.1H.1N.
Range Specification:
A range can be specified using the “TO” or “to” keyword as below:
12 TO 15 Points
2 to 4 Spades
You could also just give an upper or lower bound, e.g.,
Spades > 4
Hearts <= 3
Shape specification:
The keyword “Shape” or “shape” represents a generic hand shape. The syntax is as below:
(Shape [4,3,3,3])
(shape [4,4,3,2] or [5,4,2,2])
Shape [4,3,3,3], for instance, refers 4-3-3-3, 3-4-3-3, 3-3-4-3, or 3-3-3-4 pattern.
Pattern specification:
The keyword “Pattern” or “pattern” refers to a specific pattern. The syntax is as below:
(Pattern [1-4-3-5])
(Pattern [2-2-4-5] OR [2-2-5-4])
Pattern [1-4-3-5], for instance, refers to a hand with 1 spade, 4 hearts, 3 diamonds, and 5 clubs.