Determining Bidding Strategies in Sequential Auctions:Quasi-Linear Utility and Budget Constraints

Hiromitsu Hattori,1 Makoto Yokoo,2 Yuko Sakurai,2 and Toramatsu Shintani1

1Department of Intelligence and Computer Science, Nagoya Institute of Technology, Nagoya, 466-8555 Japan

2NTT Communication Science Laboratories, NTT Corporation, Kyoto, 619-0237 Japan

SUMMARY

A great deal of research is actively being conductedconcerning Internet auctions. A method has previously beenproposed for obtaining the optimal bidding strategy byusing dynamic programming in a sequential auction, whichis one form of auction. However, the conventional methodassumes that the utility of an agent who is bidding takes ageneral additive form, and the remaining endowment ofmoney during bidding has to be represented in each of thevarious states that are taken into consideration in the dy-namic programming procedure. As a result, when the initialendowment of money m is large, the number of states thathave to be taken into consideration becomes extremelylarge. In this paper, by assuming that the utility of an agenttakes a quasi-linear form, which is a type of additive form,the authors show that the optimal strategy using dynamicprogramming is obtained since the number of states isreduced to 1/m times the number obtained when the utilitytakes a general additive form. However, when the agentutility is assumed to take a quasi-linear form, it is impossi-ble to represent budget constraints. Therefore, in this paper,the authors propose a method of quickly obtaining a quasi-optimal strategy when budget constraints exist by modify-ing the strategy that is obtained when quasi-linear utility isassumed and show the effectiveness of the proposed methodthrough experimental evaluations. © 2007 Wiley Peri-odicals, Inc. Syst Comp Jpn, 38(8): 72–83, 2007; Published

online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10465

Key words: sequential auction; quasi-linear utility;budget constraints; bidding strategy.

1. Introduction

Electronic commerce has undergone rapid expansionin recent years, and among the various forms it takes, manycommercial sites for Internet auctions currently exist [4,13]. There exist a great variety of research projects concern-ing Internet auctions, ranging from the theoretical to thepractical [6, 7, 17, 18, 22–25].

The development of Internet auctions has made itpossible for a bidder to participate in many auctionsthroughout the world, and in such cases, bidders may havecomplementary/substitutional preferences for multipleitems. For example, in the U.S. Federal CommunicationsCommission’s (FCC) radio spectrum auctions [11], a bid-der may wish to obtain the spectrum rights for neighboringregions simultaneously (the rights for neighboring regionsare complementary). However, on the other hand, as longas the rights of a certain region are obtained, the bidder isindifferent to the specific channel that is assigned (thechannels within the same region are substitutional).

The combinatorial auction has been proposed as anauction method that can represent these kinds of comple-

© 2007 Wiley Periodicals, Inc.

Systems and Computers in Japan, Vol. 38, No. 8, 2007Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J85-D-I, No. 10, October 2002, pp. 974–984

72

mentary/substitutional preferences for multiple items. Agreat deal of research concerning this auction protocol hasbeen conducted [8, 9, 11, 20, 21]. In a combinatorial auc-tion, multiple items with interdependent values are soldsimultaneously, and bidders can bid on any combination ofitems. The method of allotting items is determined so thatthe total bid is maximized. Although using a combinatorialauction can increase both the utility of the participants andthe revenue of the seller, various problems arise whenactually conducting them. The combinatorial auction pro-tocol is complex and differs from the protocol that is usedat existing commercial auction sites. Therefore, to actuallyconduct a combinatorial auction, it is necessary for thesellers to change their systems and the participants to un-derstand the protocol. In addition, determining the optimalallotment is known to be NP complete [5, 16, 19].

On the other hand, a sequential auction is an auctionmethod for selling multiple items by consecutively con-ducting single-item auctions [8]. Since items are often soldindividually in actual Internet auctions, a sequential auctionis considered to be suited for modeling an Internet auction.Reference 3 proposes a method for using dynamic program-ming to obtain the optimal bidding strategy so that thebidder maximizes his/her expected utility in a sequentialauction based on several assumptions. However, the formu-lation described in Ref. 3 assumes that the utility of a biddertakes an additive form, and the remaining endowment ofmoney during the auction must be represented for each statethat is taken into consideration in the dynamic program-ming procedure. As a result, when the initial endowment ofmoney m is large, the number of states that must be takeninto consideration becomes extremely large.

In this paper, we assume that the utility of the biddertakes a quasi-linear form, which is a type of additive form.According to this assumption, by representing the paymentfor an item as a cost accompanying a state transition andnot representing the remaining endowment of money ineach state, we reduce the number of states and show thatthe optimal strategy is obtained by using dynamic program-ming. We also show through experimental evaluations thatthe processing time is speeded up more than m times.

An important practical case that can be representedin additive form but cannot be represented in quasi-linearform is the case when budget constraints exist [12]. In thispaper, we propose a method of quickly obtaining a quasi-optimal strategy when budget constraints exist by modify-ing the strategy that is obtained when quasi-linear utility isassumed and show the effectiveness of the proposedmethod through experimental evaluations.

The rest of this paper is organized as follows. InSection 2, as preparation, we define various terms andnotation related to a sequential auction. In Section 3, wedescribe the method of representing the problem that isshown in Ref. 3 and its solution method using dynamic

programming and point out related problems. In Section 4,we describe a new method of representing the problemwhen quasi-linear utility is assumed, and show that thenumber of states that must be taken into consideration indynamic programming can be reduced. In addition, weshow through experimental evaluations that the processingtime is speeded up more than m times. In Section 5, wedescribe the method for obtaining the quasi-optimal strat-egy that satisfies budget constraints and experimentallyevaluate the effectiveness of the proposed method. In Sec-tion 6, we further discuss the increase in speed that isobtained by the method proposed in this paper.

2. Preparations

In this section, we define the basic terms and notationused in this paper. We assume that there are n items to beauctioned denoted by r1, r2, . . . , rn and that the items areauctioned in the indicated order. The auctions for theseitems are denoted by A1, A2, . . . , An. For simplicity, weassume that the first-price sealed-bid auction protocol [15],which is frequently used in the real world, is the auctionprotocol used for each item. In a first-price sealed-bidauction, each bidder submits a bid without knowing anyoneelse’s bid, and the highest bidder wins and pays the amountof that bid. In this paper, we focus on a specific agent whobids for items and consider methods of obtaining the opti-mal bidding strategy of this agent. For auction Ai of item ri,we assume that the agent has a distribution function Fi(h)for predicting the highest bid h of other agents, and we alsoassume that the distributions of highest bids in each auctionare mutually independent. For simplicity, we assume thatthis agent wins the item when multiple agents submitted thehighest bid. According to this definition, when the agentbids z in auction Ai for item ri, the agent can use thedistribution function Fi(h) to obtain the probability of win-ning the item as Fi(z). Also, the probability Fi(z) is inde-pendent of any accepted bids for other items.

As indicated in Ref. 3, this assumption is extremelystrong and it means that agents who are participating in theauction determine their bids independently of each other’sactions. In other words, an agent does not determine his orher bidding strategy by taking into account auctions that arealready finished or that are to be held in the future, butdetermines the strategy myopically based only on the dis-tribution function of highest bids related to the item that iscurrently being auctioned. However, since it is difficult toaccurately estimate in advance the other agents’ evaluationvalues for an item, the distribution function that an agenthas is actually a function for obtaining estimates related tothe probability that the item is obtained. The agent obtainsthe optimal bidding strategy when the estimates that areobtained from this distribution function are assumed to be

73

correct. Also, an agent can modify the distribution of otheragents’ highest bids through experience as described in Ref.3.

For a subset Rs ⊂ R, which is a subset of the set of allitems R = {r1, r2, . . . , rn}, we denote the agent’s valuationfor Rs as v(Rs). The valuation of this set of items is deter-mined independently of other agents’ valuations. Thesekinds of items are called private-value items [10, 15]. Theassumption of private-value items is generally often usedbecause it simplifies theoretical analyses [25].

3. Dynamic Programming Based onAdditive Utility

In a sequential auction, an agent must determine bidsfor individual items. When there exist complementary/sub-stitutional dependency relationships among the values ofmultiple items, determining how to bid for an individualitem is a problem. Therefore, a method is proposed in Ref.3 for using dynamic programming [2] to find the optimalbidding strategy. Dynamic programming is a method forobtaining the optimal strategy in a Markov decision prob-lem [14]. A Markov decision problem is the problem ofdetermining a sequence of actions that optimize the sum ofrewards/costs, given a set of states, set of actions that canbe executed in each state, transition probabilities whenarbitrary actions are executed in each state, and re-wards/costs accompanying transitions.

For the method in Ref. 3, the utility of an agent isassumed to take an additive form. The utility of an agent isadditive means that when the set of items for which theagent’s bid was accepted is denoted by Rs and the remainingendowment of money is denoted by d at the time that allitem auctions are finished, the utility of the agent is givenby

where f is an arbitrary (additive) function that assigns someutility to the remaining endowment of money.

To obtain the optimal strategy, the decision-makingprocess or, in other words, the process of participating inmultiple auctions is divided into n + 1 stages, and an integervalue time index t (0 ≤ t ≤ n) for referencing each stage isassigned. Bids are submitted for items in the first n stages.The last (n + 1)-th stage is the state after all auctions arefinished. In other words, in the range 0 ≤ t ≤ n – 1, the agentis participating in auction At at stage t. Also, at a certainstage t, the state in which the set of items owned by thebidder is Rs and the remaining endowment of money is d isdenoted by the combination of these values ⟨Rs, d⟩t. Abidding strategy π is a mapping from a state to a bid, andπ(⟨Rs, d⟩t) = z means that when the set of items that are

owned at stage t is Rs and the remaining endowment ofmoney is d, then z is bid for item rt+1. Also, when the currentstate is ⟨Rs, d⟩t, the expected utility that is obtained byexecuting strategy π is denoted by Vπ(⟨Rs, d⟩t). If m denotesthe remaining endowment of money of the agent at theinitial state, then the expected utility when strategy π isexecuted is given by Vπ(⟨∅, m⟩0).

The optimal strategy π* is defined as follows. How-ever, Vπ(⟨Rs, d⟩t) = v(Rs) + f(d) is assumed for state ⟨Rs, d⟩t

in stage n.

where Q(⟨Rs, d⟩t, z) denotes the expected utility when z isbid in a certain state ⟨Rs, d⟩t at stage t. Also, V(⟨Rs, d⟩t)denotes the maximum expected utility when z is bid in state⟨Rs, d⟩t or, in other words, the expected utility that isobtained from the optimal strategy. With this formulation,the reward/cost accompanying a state transition is assumedto be 0. The optimal strategy π* can be obtained by execut-ing value iteration [14]. In other words, the optimal strategyof the state at stage t is calculated based on the expectedutility of the state at stage t + 1, and by repeating this, theexpected utilities of all states as well as the optimal strategyare finally determined.

Figure 1(a) shows the optimal strategy in the follow-ing simple example. Assume that there are two items r1 andr2 and that the initial endowment of money is 4. Let thevaluation for the set of items be 4 only when both items areowned and 0 otherwise. Also, let the highest bid of otheragents for each item be 1 with probability 1/2 or 2 withprobability 1/2. In other words, if 0 is bid, the probabilityof winning is 0; if 1 is bid, the probability of winning is 1/2;and if 2 or more is bid, the probability of winning is 1. Also,assume that the valuation of the remaining endowment ofmoney d at the final state is d itself.

Figure 1(a) shows the states to which a transition ispossible when the optimal strategy is followed and thevaluation V and strategy π in those states. The state transi-tions are indicated by arrows, and the transition prob-abilities are indicated by numerical values above thearrows. With the optimal strategy in this example, 1 is bidfor item r1. The result is that item r1 is obtained withprobability 1/2, and the item cannot be obtained with prob-ability 1/2. If item r1 is obtained, 2 is bid for item r2 to makecertain that both items are obtained. The utility in this caseis 4 + 1 = 5. On the other hand, if item r1 was not obtained,

74

0 is bid for item r2 since obtaining only item r2 is useless.In other words, the agent does not participate in the auction.The utility that is obtained in this case is equal to the utilityof the remaining endowment of money, which is 4. There-fore, the expected utility of the optimal strategy in thisexample is 0.5 × 5 + 0.5 × 4 = 4.5.

Consider the number of states at stage t when theproblem representation based on additive utility, which wasdescribed in this section, is used. First, if we assume thataccording to the items for which bids were accepted up tostage t, the number of possible combinations of items is 2t

and the initial endowment of money is the positive integerm, then excluding the case when no items are obtained atall, the number of possible variations of remaining endow-ments of money is m + 1. Therefore, the number of statesat stage t is (2t – 1) × (m + 1) + 1, and the total number ofall states is O(m × 2n). As a result, when the initial remainingendowment of money is large, the number of states becomesextremely large. One method that can be considered forreducing the number of states is to determine bids for eachitem in terms of a coarser unit (such as bids in units of $100).However, unless all agents are guaranteed to bid using thesame coarse units, determining bids in terms of coarserunits may reduce the expected utility.

4. Dynamic Programming Based onQuasi-Linear Utility

In this section, we show that when the utility of anagent is defined with a quasi-linear form, which is a type ofadditive form, the remaining endowment of money need notbe represented in each state, and the number of states canbe reduced to 1/m times the number obtained when theutility is defined with an additive form.

4.1. Basic ideas

When the agent has obtained the set of items Rs at acertain time in a sequential auction, if ZRs

denotes the sum

of the payments for the items, then quasi-linear utility isdefined by

In other words, the utility when the agent participates in theauction is given by the difference between the valuation ofthe set of items that were obtained and the sum of thepayments. This assumption of quasi-linear utility is fre-quently used in microeconomics research regarding suchtopics of auctions and mechanism design [10].

When utility has been assigned in additive form, ifthe utility of the initial endowment of money f(m) is rede-fined as the utility reference value, then not participating inthe auction has a utility of zero. At this time, the agent ownsthe set of items Rs, and the utility when the sum of thepayments for those items is ZRs

becomes v(Rs) + f(m – ZRs)

– f(m). Therefore, quasi-linear utility is equivalent to thecase when f(x) = x for additive utility. The assumption ofquasi-linear utility is considered to be appropriate when theamount that the agent pays in the auction is relatively smalland has little effect on items that are obtained outside of theauction (more accurately, when there is no income effect[10]).

4.2. Details of bidding strategy decisionsbased on quasi-linear utility

4.2.1. Problem representation based onquasi-linear utility

When quasi-linear utility is assumed, the problemrepresentation can be simplified as shown below.

When the utility of the agent takes an additive form,the state of stage t had been represented by the combination⟨Rs, d⟩t of the set of items that are owned and the remainingendowment of money. When quasi-linear utility is assumedand no upper limit is established for the amount of moneythat is to be bid by the agent (for example, when the agentcan pay by obtaining an interest-free loan), the subsequentoptimal strategy is exactly the same for two states ⟨Rs, d⟩t

Fig. 1. Example of the optimal bidding strategy.

75

and ⟨Rs, d′⟩t. In other words, the optimal strategy in sub-sequent auctions is exactly the same regardless of whetherthe agent obtained Rs for free or by paying one milliondollars. Also, the difference between the utilities of the twostates ⟨Rs, d⟩t and ⟨Rs, d′⟩t in stage n is equal to the differencebetween the remaining endowments of money d and d′.From the above property, we can formulate the problemwithout including the remaining endowment of money inthe state and by letting the payment be a cost accompanyinga state transition.

Therefore, the state at stage t can be denoted as ⟨Rs⟩t

by using only the set of items that are owned Rs. A biddingstrategy π is a mapping from a state to a bid, and π(⟨Rs⟩t) =z in relation to each state means that when the set of itemsRs has been obtained, then z is bid for item rt+1. In this paper,we will denote the expected utility that is obtained whenstrategy π is executed in state ⟨Rs⟩t as Vπ(⟨Rs⟩t). Also, theexpected utility when strategy π is executed in the initialstate is given by Vπ(⟨∅⟩0).

4.2.2. Optimal bidding strategy based onquasi-linear utility

Based on the problem representation described inSection 4.2.1, the optimal bidding strategy π* when quasi-linear utility is assumed can be defined as follows. How-ever, Vπ(⟨Rs⟩n) = v(Rs) is assumed for state ⟨Rs⟩n in stage n.

where Q(⟨Rs⟩t, z) denotes the expected utility when z is bidin a certain state ⟨Rs⟩t at stage t. Also, V(⟨Rs⟩t) denotes themaximum expected utility when z is bid in state ⟨Rs⟩t or, inother words, the expected utility that is obtained when theoptimal strategy is used. The optimal strategy π* can beobtained by executing value iteration in a similar manneras when utility is additive.

With the process for obtaining the optimal biddingstrategy based on the above definition, the upper bound ofthe bid z that must be taken into consideration in each stateis limited by the difference between V(⟨Rs ∪ {rt+1}⟩t+1 andV(⟨Rs⟩t+1). Clearly, bidding 0 provides a greater expectedutility than bidding an amount greater than or equal to thisdifference. On the other hand, when the utility takes anadditive form as has been defined in Ref. 3, no suitablemethod exists for establishing an upper bound for the bidthat must be taken into consideration in each state. When

the utility is represented in a general additive form withoutassuming quasi-linear utility, the utility of obtaining allitems can be set as the upper bound of the bid if f(x) = x isassumed. However, since this upper bound is greater thanthe upper bound that is established for quasi-linear utility,clearly more bids must be taken into consideration.

Figure 1(b) shows the states to which a transition ispossible when the optimal strategy is followed and thevaluation V and strategy π in those states for the sameproblem as shown in Fig. 1(a). The state transitions areindicated by arrows, and the transition probabilities areindicated by numerical values above the arrows. The opti-mal strategy is identical for both (a) and (b) as shown in thefigure. Also, if the utility of the remaining endowment ofmoney in each state in the representation in (a) is set as thereference, the value of V in each state will be identical forboth (a) and (b). For example, since the remaining endow-ment of money is 4 and V is 4.5 in the initial state in (a), ifthe utility of a remaining endowment of 4 is set as thereference, then following the optimal strategy increases theutility of the agent by 0.5. This value is the same as the valueof V in the representation of (b).

When the utility takes an additive form, if the initialendowment of money is the positive integer m, the totalnumber of states is O(m × 2n) since the number of possiblevariations of remaining endowments of money is m + 1 inrelation to a state in which a certain combination of itemsis obtained. On the other hand, when quasi-linear utility isassumed, the states need not be distinguished according tothe remaining endowment of money. As a result, the numberof states at stage t is 2t from the fact that the number ofpossible combinations of items is 2t according to the itemsfor which bids were accepted up to stage t, and the totalnumber of all states is O(2n). Therefore, when quasi-linearutility is assumed, the number of states is reduced to 1/mtimes the number obtained when the utility takes an additiveform.

4.3. Evaluation of the method for introducingquasi-linear utility

In this section, we experimentally evaluate the effec-tiveness of the new problem representation based on theintroduction of quasi-linear utility, which we proposed, andwe discuss the results that were obtained.

In this experiment, we assume that the number ofitems n is an even number and denote the initial endowmentof money by m. We also assume that the agent wants toobtain either {r1, r3, . . . , rn−1}, which is the set of itemshaving odd numbers, or {r2, r4, . . . , rn}, which is the set ofitems having even numbers, and that a valuation of 100 ×n/2 is assigned to each set. These two sets are mutuallysubstitutional, and even if extra items are obtained in addi-

76

tion to one of these sets of items, the utility is assumed notto increase. Also, the items within each set are mutuallycomplementary, and if any item within the set is not ob-tained, the utility is assumed to be 0. In addition, themaximum bids by other agents for each item in this experi-ment are assumed to be uniformly distributed in [0, 100].

Figure 2 shows processing times until the biddingstrategy is determined when the number of items n is variedfor both the problem representation when utility takes anadditive form and the problem representation proposed inthis paper. The horizontal axis in the figure represents thenumber of items, and the vertical axis represents the proc-essing time. Figure 2 shows three graphs for cases when theutility takes an additive form. These correspond to when theinitial endowment of money is 500, 1000, and 1500. Weperformed this experiment on a 170-MHz Sun TurboSparcworkstation and used Java for the programming language.

The method proposed in this paper was expected tospeed up processing time by approximately m times fromthe fact that the total number of states that are used indynamic processing is reduced to 1/m times the numberobtained when the utility takes an additive form. However,the experimental results show that the processing time isspeeded up by more than m times. The proposed methodnot only reduces the number of states, but also limits theupper bound of the bids in each state by the differencebetween V(⟨Rs ∪ {rt+1}⟩t+1) and V(⟨Rs⟩t+1). As a result, it isapparent from the experimental results that the number ofbids that must be taken into consideration in each state isreduced, and the processing time is speeded up more thanexpected.

5. Introduction of Budget Constraints

5.1. Basic ideas

In the previous sections, we showed that when theutility of the agent is assumed to take a quasi-linear form,since the payment for an item can be represented as a costaccompanying a state transition, the number of states isreduced to 1/m times the number obtained when the utilitytakes an additive form and the optimal strategy is obtainedby using dynamic programming. The assumption of quasi-linear utility is sufficiently general when dealing with se-quential auctions. However, an important practical case thatcan be represented in additive form but cannot be repre-sented in quasi-linear form is the case when budget con-straints exist. The existence of budget constraints meansthat an upper bound is established for the amount that theagent can pay in an auction. When utility takes an additiveform, by representing the remaining endowment of moneyin each state and setting the payment so that it cannot exceedthe remaining endowment of money, the optimal strategycan be obtained within the range of the budget constraint.On the other hand, with a quasi-linear form, although noproblem arises if the budget constraint is sufficiently large,when the budget constraint is severe, the strategy that isobtained may not be able to be executed based on the budgetconstraint. In other words, since the method that was pro-posed in Section 4 does not take the remaining endowmentof money into consideration in the process for determiningthe bidding strategy, the bidding strategy that is obtainedmay include a case in which the total payment exceeds theinitial endowment of money. For example, when the initialendowment of money is assumed to be 2 in the example inFig. 1(b), when the bid was accepted for only item r1, thetotal payment is 1 and no problem occurs. However, whenthe combination of items {r1, r2} is obtained, since the totalbid is 3, the initial endowment of money is exceeded.Therefore, for this case, the optimal strategy in Fig. 1(b)clearly cannot be executed.

Therefore, in this paper, we show a method of quicklyobtaining a quasi-optimal strategy π′, which can be exe-cuted based on budget constraints, by modifying the strat-egy π* that is obtained when quasi-linear utility is assumed(optimal strategy when there are no budget constraints).Specifically, we let the bid that is determined by π* be areference and set the upper bound of possible bids so thatthe budget constraint is satisfied in each state. Dynamicprogramming is used to determine the optimal biddingstrategy within this upper bound.†

†If this upper bound can be set appropriately, the strategy that is obtainedis the optimal strategy based on the budget constraints. However, since themethod proposed in this paper determines the upper bound of the bidheuristically with π* as a reference, the optimality of the strategy that isobtained cannot be guaranteed.Fig. 2. Comparison of processing times.

77

5.2. Details of the method for introducingbudget constraints

5.2.1. Procedure for determining the biddingstrategy

The method for obtaining the quasi-optimal biddingstrategy that satisfies budget constraints consists of thefollowing two steps.

(Step 1) Determine the optimal bidding strategy π*

when budget constraints are not taken into considerationaccording to processing that is based on dynamic program-ming when quasi-linear utility is assumed, which was de-scribed in Section 4.

(Step 2) For each state of the bidding strategy π* thatwas obtained in Step 1, sequentially perform processingbased on dynamic programming from stage n – 1. However,to satisfy the budget constraints in the dynamic program-ming process, calculate the upper bound of the bids in eachstate and determine bids within the range of the upperbound that was obtained.

To obtain the quasi-optimal bidding strategy accord-ing to the proposed method, dynamic programming is ap-plied twice. As shown in Section 4.3, the proposed methodwhen budget constraints were not taken into considerationwas able to speed up the processing time by more than mtimes compared with when utility takes an additive form.As a result, the processing time when this method is usedcan be expected to be at least m/2 times faster than whenutility takes an additive form.

5.2.2. Calculation of the upper bound

For state ⟨Rs⟩t at stage t, let Zformer denote the sum ofthe payments according to strategy π* from the initial state⟨∅⟩0 until this state is reached, and let zopt denote the optimalbid according to strategy π*. Also, let Zlatter denote thehighest value of the sum of the payments beginning withand following state ⟨Rs ∪ {rt+1}⟩t+1, which is the result whenthe agent’s bid is successfully accepted in state ⟨Rs⟩t. Notethat Zlatter is calculated by using bids that have already beenmodified. In addition, let Zbud denote the upper bound of thebudget.

The upper bound zmax of the bids in state ⟨Rs⟩t can bedetermined according to the formula

Since the bids have already been adjusted for a state at staget + 1 and following, to satisfy the budget constraints in everycase, the sum of the payments assumes the highest case.Therefore, the sum of the payments from the initial state⟨∅⟩0 until state ⟨Rs ∪ {rt+1}⟩t+1 is assumed to be less thanor equal to Zbud − Zlatter. The problem is deciding how todistribute this amount of money among the t + 1 states from

the initial state ⟨∅⟩0 to state ⟨Rs⟩t. The upper bound is set hereby proportionally distributing (prorating) it to each stateusing the bids determined by strategy π* as reference values.The method of determining the quasi-optimal strategy pro-posed in this paper calculates Q(⟨Rs⟩t, z) in relation to a bidthat does not exceed this zmax to determine the optimal bidand updates V(⟨Rs⟩t).

5.3. Evaluation of the method for introducingbudget constraints

In this section, we experimentally evaluate the effec-tiveness of the method of obtaining the quasi-optimal strat-egy based on budget constraints, which was proposed in theprevious section.

This experiment compares the method proposed inthe previous section (Prorated), the method in which utilityis additive (Additive), a trivial method that uses the optimalbidding strategy π* (Trivial), and a method that assigns theupper bound of the bids by equally dividing the remainingendowment of money (Uniform). The trivial method usedhere submits bids according to the designated amount ofmoney as long as the bid that is determined by the biddingstrategy π* can be paid. If the remaining amount of moneyis less than the amount that was determined by the biddingstrategy π*, the agent simply submits the entire remainingamount of money as the bid. When the remaining amountof money becomes zero, the agent stops participating insubsequent auctions. Also, although the method thatequally divides the remaining endowment of money obtainsthe strategy π″ by modifying the optimal strategy π*, in thatprocess, the upper bound of bids for each item is assignedaccording to the amount of money obtained by simplyequally dividing the remaining endowment of money in acertain state by the remaining number of items.

We will explain the bids based on each method belowfor an example in which there are three items r1, r2, and r3,the agent’s budget constraint is 150, the highest bids byother agents for each item are uniformly distributed in [0,100], and utility is obtained only when the bids are acceptedfor all items. First, with the method in which utility isadditive, the optimal strategy within the budget constraintrange can be determined after all states consisting of com-binations of items that can occur in each stage and remain-ing endowments of money are listed. On the other hand,with the optimal strategy π* that is obtained from themethod proposed in Section 4, the total bid may end upexceeding 150 depending on the valuations of items. Forexample, when v({r1, r2, r3}) = 300 and the valuation forother combinations of items is 0, then π*(⟨∅⟩0) = 50,π*(⟨{r1}⟩1) = 100, and π*(⟨{r1, r2}⟩2) = 100, and the totalbid is 250. With the trivial method, since bids are simplysubmitted according to the bidding strategy π*, the remain-ing endowment of money is zero when the auction for item

78

r2 is finished. As a result, the bid for item r3 becomes 0, andthe expected utility clearly decreases. Therefore, while thetrivial method does not require processing for modifyingthe strategy, it can only obtain a low utility in many cases.Next, with the method that was proposed in the previoussection, in all cases related to situations in which the bid isaccepted for an item based on the optimal strategy π*, thestrategy is modified so that the total bid is within the budgetconstraint range. In this example, first π′(⟨{r1, r2}⟩2) = 60,and also π′(⟨{r1}⟩1) = 60. Although the upper bound of bidsfor item r1 to keep the total bid from exceeding the budgetconstraint is 30 here, π′(⟨∅⟩) becomes 25 since the expectedutility of state ⟨{r1}⟩1 decreases due to the modification ofthe strategy up to this point. With the proposed method,since the strategy is modified according to the bid in theoptimal bidding strategy π*, the decrease in the expectedutility can be kept low. Also, with the method in which theremaining endowment of money is equally divided, al-though the upper bound of the bids in the auction for itemr1 is first 150/3 = 50, since π*(⟨∅⟩0) = 50, the strategy doesnot change, and π″(⟨∅⟩0) = 50. However, in the auction foritem r2, although π*(⟨{r1}⟩1) = 100, since the upper boundis 100/2 = 50, π′′(⟨{r1}⟩1) becomes 50, and similarlyπ″(⟨{r1, r2}⟩2) becomes 50. Consequently, the bids for r2,which had been bid with the highest amount of money instrategy π*, and r3 end up being significantly lowered. Withthe method in which the remaining endowment of moneyis equally divided, since no upper bound of bids is set basedon the bidding strategy π* as in the proposed method andthe valuations for items are not reflected in a strategymodification, it is difficult to obtain similar expected utili-ties to those of the optimal strategy.

We performed experiments at this time based on twoproblem setups.

[Setup 1] The number of items is set to n = 9. Theagent is assumed to want to obtain any one of the three setsof items {r1, r2, r3}, {r4, r5, r6}, and {r7, r8, r9}, and avaluation of 300 is assumed to be assigned for each set.Also, these sets are mutually substitutional, and even ifextra items are obtained in addition to one of these sets ofitems, the utility is assumed not to increase. Also, the itemswithin each set are mutually complementary, and if anyitem within the set is not obtained, the utility is assumed tobe 0. In addition, the highest bids by other agents for eachitem are assumed to be uniformly distributed in [0, 100].

Figure 3 shows the expected utilities that are obtainedfrom the four methods described above by varying thebudget from 10 to 260 when the highest payment in theoptimal strategy with no budget constraints is 251. Thehorizontal axis in the figure represents the budget, and thevertical axis represents the expected utility. From the ex-perimental results, it is apparent that the expected utilitythat is obtained by the proposed method is extremely closeto the expected utility that is obtained from the optimal

strategy. On the other hand, the expected utility that isobtained from the trivial method mostly consists of negativevalues. This is because the agent often can obtain only partof a set of complementary items. Although the method inwhich the remaining endowment of money is equally di-vided does not show an extreme decrease in the expectedutility like the trivial method does, it only obtains an ex-tremely low expected utility compared with the expectedutility obtained from the optimal strategy. In addition, asthe budget gets larger, the expected utility that is obtainedbecomes lower than that of the trivial method, and evenwhen there is a sufficient budget, the highest expectedutility is not obtained. This is because the upper limit of thebids cannot be set appropriately since the valuations ofitems are not reflected in the modification of the biddingstrategy. In particular, since the upper limit of bids ends upbeing held down in auctions that are conducted at earliertimes, it is difficult to take the optimal strategy.

[Setup 2] The number of items is set to n = 9. Theagent is assumed to assign valuations of 200, 300, and 400to the sets of items {r1, r2}, {r3, r4, r5}, and {r6, r7, r8, r9},respectively. In a similar manner as in Setup 1, these setsare mutually substitutional, and when multiple sets of itemsare obtained simultaneously, the highest valuation is as-signed as the valuation of that set. In other words, thevaluation is 300 when the set of items {r1, r2, r3, r4, r5} isobtained. Also, the items within each set are assumed to bemutually complementary. The highest bids by other agentsfor each item are assumed to be uniformly distributed in [0,100].

Figure 4 shows the expected utilities that are obtainedby varying the budget from 10 to 360 when the highest bid

Fig. 3. Comparison of expected utilities (setup 1).

79

in the optimal strategy with no budget constraints is 351. Ina similar manner as in Setup 1, the expected utility that isobtained from the trivial method mostly consists of negativevalues. In addition, until the budget exceeds 250, somecases are seen in which only part of a set of complementaryitems are obtained and conditions end up getting worse asa result of attempting to obtain items with a higher valu-ation. In a similar manner as in Fig. 3, although the methodin which the remaining endowment of money is equallydivided does not show an extreme decrease in the expectedutility like the trivial method does, it can only obtain a lowexpected utility. On the other hand, it is apparent that theexpected utility that is obtained by the proposed method isstill extremely close to the expected utility that is obtainedfrom the optimal strategy.

Figure 5 shows a comparison of processing timesbased on Setup 1. The horizontal axis in the figure repre-sents the budget, and the vertical axis represents the proc-essing time. With the trivial method, the requiredprocessing time is fixed. For the proposed method, theprocessing time required for processing based on dynamicprogramming in Step 1 is equal to the processing time ofthe trivial method. The processing time required for apply-ing dynamic programming in Step 2, which varies accord-ing to the budget, increases the total processing timeslightly. Also, although the method in which the remainingendowment of money is equally divided requires process-ing for dividing the remaining endowment of money, theprocessing time is almost equal to that of the trivial methodsince the extra required time is slight. On the other hand, itis apparent that the processing time for the method in whichthe utility is additive increases almost linearly with respectto the square of the budget.

From the above experiment, we showed that a quasi-optimal bidding strategy, which obtains an expected utility

that is extremely close to the expected utility obtained fromthe optimal strategy, can be calculated rapidly by using themethod that was proposed in this paper. However, thisexperimental evaluation alone is still not sufficient to provethe effectiveness of the proposed method, and further inves-tigations based on various problem setups must be per-formed. In particular, in this experiment, since the highestbids of other agents were uniformly distributed, the prob-ability of a bid being accepted changed very little forincreases or decreases in the agent’s bid. As a result, rela-tively good results could be obtained by the proposedmethod in which the budget was simply distributed propor-tionally (prorated). When the setup is changed so that theprobability of a bid being accepted changes radically rela-tive to changes in the bid, the effects that occur as a resultof bidding changes must be taken into consideration. In thiscase, a more complex method for determining the upperbound of the bids may be required. We are currently devel-oping and evaluating such as a method.

6. Discussion

The main contribution in this paper is that the proc-essing time for determining the optimal bid based on dy-namic programming is speeded up. As described in Section4.2.2, the method that was proposed in this paper indicateda speed-up in processing time based on the initial endow-ment of money m. As a result, in Section 4.3, we performedexperiments based on different endowments of money toprove the effectiveness of the proposed method. Dependingon the experimental setup here, when the utility is additive,there may exist cases in which the total of the payments inall auctions is greater than or equal to the valuation whenFig. 4. Comparison of expected utilities (setup 2).

Fig. 5. Comparison of processing time.

80

all items are obtained. At first glance, a payment that isgreater than or equal to the highest valuation may appear tobe meaningless. However, a situation that requires thecalculation of a total payment that is greater than or equalto the highest valuation can be considered as follows.

Assume that there are three items denoted by r1, r2,and r3 and that the items are auctioned in the indicated order.Let the valuation of each of the combinations of items {r1,r2} and {r1, r3} be 100, and assume that these sets of itemsare substitutional. In other words, assume that the valuationis 100 even if all of the items are obtained. Also, assumethat the items within an individual set are complementaryand that the valuation when an individual item is obtainedby itself is 0. In addition, assume the following regardingthe acceptance of bids for items.

• When the bid for item r1 is 40, the bid is certainlyaccepted, but a bid that is less than 40 is notaccepted.†

• When the bid for item r2 is 40, the bid is acceptedwith a probability of 99%, but a bid that is lessthan 40 is not accepted. Also, the probability ofthe bid being accepted does not increase even ifthe bid is greater than 40. In other words, there isno guarantee that the bid will certainly be acceptedeven if the bid is increased.

• When the bid for item r3 is 90, the bid is certainlyaccepted, but a bid that is less than 90 is notaccepted.

Let us consider the agent’s optimal bidding strategybased on the above setup. First, it is clear that π(⟨∅⟩0) = 40.This is because the expected utility cannot increase unlessthe agent at least obtains item r1. Next, from the assumptionrelated to the acceptance of the bid for the item, we canconclude that π(⟨r1⟩1) = 40. However, once in a great while,the agent’s bid for item r2 will fail to be accepted. If the bidfor item r2 fails to be accepted and the agent does not bid agreater amount, the agent’s utility will decrease by the bidof 40, which was paid for item r1. Therefore, if we nextconsider the bid for item r3, from the assumption related tothe acceptance of the bid for the item, it is clear that π(⟨r1⟩2)= 90. At this time, although the total of the payments is 130,which is greater than the highest valuation that is obtainedby the agent, which is 100, the reduction in utility can beheld down to 30. In other words, paying an amount greaterthan or equal to the highest valuation becomes the optimal

strategy for the agent. Therefore, it became clear throughthis investigation that it is necessary to take cases in whichthe total of the payments is greater than or equal to thehighest valuation into consideration in the process of ob-taining the optimal bidding strategy.

7. Conclusions

In this paper, we showed a method of using dynamicprogramming to determine the optimal bidding strategy ofan agent in a sequential auction. The conventional method,which assumes that the utility of an agent takes a generaladditive form, requires that the remaining endowment ofmoney during the auction has to be represented in each ofthe various states that are taken into consideration in thedynamic programming procedure. As a result, for a large-scale problem, in particular, when the initial endowment ofmoney is large, the number of states that have to be takeninto consideration becomes extremely large. Specifically,when the number of items is n and the endowment of moneyin the initial state is m, the total number of states that mustbe taken into consideration is O(m × 2n).

In this paper, we assumed that the agent’s utility takesa quasi-linear form, which is a type of additive form, andshowed that according to this assumption, by representingthe payment for an item as a cost accompanying a statetransition and not representing the remaining endowmentof money in each state, the number of states was reducedand the optimal strategy was obtained by using dynamicprogramming. We also showed through experimentalevaluations that the processing time was speeded up morethan m times and that the proposed method not only reducedthe number of states, but was also clearly effective inreducing the number of bids that had to be taken intoconsideration. On the other hand, an important practicalcase that can be represented in additive form but cannot berepresented in quasi-linear form is the case when budgetconstraints exist. In this paper, we proposed a method ofquickly obtaining a quasi-optimal strategy when budgetconstraints exist by modifying the strategy that is obtainedwhen quasi-linear utility is assumed or, in other words,when no budget constraints exist, and showed the effective-ness of the proposed method through experimental evalu-ations.

Although we assumed that the auction protocol usedfor each item was the first-price sealed-bid auction protocolto propose a novel method for determining the biddingstrategy in this paper, this method can also be applied toother auction protocols. For example, by simply consider-ing the upper bound of the amounts that can be bid toultimately be the same as the bid in a first-price sealed-bidauction protocol, the proposed method can be applied to an

†Although it is difficult to guarantee the certain acceptance of a bid in areal-world auction, the certain acceptance of a bid in an Internet auctioncan be guaranteed by bidding the price that was set in advance by the seller.For example, in Yahoo!Auctions, the auction ends immediately and thebidder is certainly guaranteed to obtain the item if a bid is submitted thatis greater than or equal to the desired winning amount that was set by theseller.

81

English auction protocol in which each bidder graduallyincreases the price by any amount.

As a future topic of research, we plan to investigatethe use of a method such as reinforcement learning [1] tolearn the optimal strategy through repetitive experiencewithout assuming that the agent has knowledge related tothe probability distribution of the highest bids of otheragents in advance.

REFERENCES

1. Barto A, Bradtke SJ, Singh S. Learning to act usingreal-time dynamic programming. Artif Intell1995;72:81–138.

2. Bellman R. Dynamic programming. Princeton Uni-versity Press; 1957.

3. Boutilier C, Goldszmidt M, Sabata B. Sequentialauctions for the allocation of resources with comple-mentarities. Proc 16th Int Joint Conf on Artif Intell(IJCAI-99), p 527–534.

4. eBay URL, http://www.ebay.com/.5. Fujishima Y, Leyton-Brown K, Shoham Y. Taming

the computation complexity of combinatorial auc-tions: optimal and approximate approaches. Proc16th Int Joint Conf on Artif Intell (IJCAI99), p 548–553.

6. Guttman RH, Moukas AG, Maes P. Agent-mediatedelectronic commerce: a survey. Knowl Eng Rev1998;13:147–159.

7. Harkavy M, Kikuchi H, Tygar JD. Electronic auc-tions with private bids. Proc 3rd USENIX Workshopon Electronic Commerce, 1998.

8. Klemperer P. Auction theory: a guide to the literature.J Econ Surv 1999;13:227–286.

9. MacKie-Mason JK, Varian HR. Generalized Vickreyauctions. http://www-personal.umich.edu/jmm/research.html, 1994.

10. Mas-Colell A, Whinston MD, Green JR. Microe-conomic theory. Oxford University Press; 1995.

11. Milgrom PR. Putting auction theory to work: thesimultaneous ascending auction. http://www.cramton.umd.edu/conference/Auction-Conference.html, 1998.

12. Okuno M, Suzumura K. Microeconomics I. IwanamiShoten; 1985.

13. Yahoo!Auctions URL, http://auctions.yahoo.com/.14. Puterman ML. Markov decision processes. Wiley;

1994.15. Rasmusen E. An economic analysis of games and

information II. (Hosoe M, Murata S, Arisada Y, trans-lators). Kyushu University Press; 1999.

16. Rothkopf MH, Pekec A, Harstad RM. Computation-ally manageable combinatorial auctions. Manage Sci1998;44:1131–1147.

17. Sakurai Y, Yokoo M, Matsubara S. A limitation of thegeneralized Vickrey auction in electronic commerce:robustness against false-name bids. Proc 16th Na-tional Conf on Artif Intell (AAAI-99), p 86–92.

18. Sandholm T. Limitations of the Vickrey auction incomputational multiagent systems. Proc 2nd Int Confon Multiagent Systems (ICMAS-96), p 299–306.

19. Sandholm T. An algorithm for optimal winner deter-mination in combinatorial auction. Proc 16th Int JointConf on Artif Intell (IJCAI-99), p 542–547.

20. Sandholm T, Suri S, Gilpin A, Levine D. CABOB: Afast optimal algorithm for combinatorial auctions.Proc 17th Int Joint Conf on Artif Intell (IJCAI-01), p1102–1108, 2001.

21. Varian HR. Economic mechanism design for com-puterized agents. Proc 1st Usenix Workshop on Elec-tronic Commerce, 1995.

22. Wurman PR, Walsh WE, Wellman MP. Flexible dou-ble auctions for electronic commerce: theory andimplementation. Dec Support Syst 1998;24:17–27.

23. Yokoo M, Sakurai Y, Matsubara S. The effect offalse-name declarations in mechanism design: to-wards collective decision making on the Internet.Proc 20th Int Conf on Distributed Computing Sys-tems (ICDCS-2000), p 146–153.

24. Yokoo M, Sakurai Y, Matsubara S. Robust multi-unitauction protocol against false-name bids. Proc 17thInt Joint Conf on Artif Intell (IJCAI-01), p 1089–1094, 2001.

25. Yokoo M. Theory and applications of Internet auc-tions. J Japan Soc Artif Intell 2000;15:404–411.

82

AUTHORS (from left to right)

Hiromitsu Hattori (student member) completed the first half of an engineering postgraduate doctoral course in electricaland computer engineering at Nagoya Institute of Technology in 2001 and is enrolled in the second half of that course. Hisresearch interests include multiagent systems, e-commerce support, and group decision support systems. He is a member of theAmerican Association for Artificial Intelligence, the Japanese Society for Artificial Intelligence, and the Japan Society forSoftware Science and Technology.

Makoto Yokoo (member) graduated in 1984 with a specialty in electronics from the Department of Engineering at theUniversity of Tokyo, completed his master’s course in 1986, and joined NTT Corporation. During 1990–1991, he was a visitingresearcher at the University of Michigan. Currently, he is employed in the NTT Communication Science Laboratories. He isengaged in research on constraint satisfaction problems, multiagent systems, and distributed artificial intelligence. He isinterested in constraint satisfaction/distributed constraint satisfaction, and agent consensus building mechanisms. He holds aPh.D. degree in engineering. He received the Japanese Society for Artificial Intelligence Paper Prize in 1992, the SakaiCommemorative Special Prize of the Information Processing Society of Japan in 1995, and the Excellent Paper Prize awardedat the National Conference of the Japanese Society for Artificial Intelligence in 1999. He is a member of the Japanese Societyfor Artificial Intelligence, the Information Processing Society of Japan, the Japan Society for Software Science and Technology,and the American Association for Artificial Intelligence.

Yuko Sakurai left Nara Women’s University in 1995 with a specialty in science and mathematics. In 1997, she completedher master’s course in the Department of Mathematical and Physical Science Mathematics, Nagoya University, and joined NTTCorporation. Currently, she is employed in the NTT Communication Science Laboratories. She is engaged in research onmultiagent systems. Her research interests include the rational decision-making mechanisms of agents. In 1999, she receivedthe Excellent Paper Prize at the Annual Japanese Society for Artificial Intelligence International Meeting. She is a member ofthe Japan Society for Software Science and Technology.

Toramatsu Shintani (member) completed a master’s course in 1982 at the Science University of Tokyo and joined FujitsuCorporation International Information and Social Science Research Institute where he engaged in research on intelligentinformation processing and logic programming. In 1993, he was appointed an assistant professor in the Department ofIntelligence and Computer Science at Nagoya Institute of Technology, and a professor there in 1999. During 1999–2000, hewas a visiting research scientist at the Robotics Institute of Carnegie Mellon University. He holds a Ph.D. degree in engineering.He is engaged in research on distributed artificial intelligence, decision support systems, and multiagent systems. He is a memberof the American Association for Artificial Intelligence, the Information Processing Society of Japan, and the Japan Society forSoftware Science and Technology.

83