In a coin game, you repeatedly toss a biased coin (0.4 for head, 0.6 for tail). Each head represents 3 points and tail represents 1 point. You can either Toss or Stop if the total number of points you have tossed is no more than 7. Otherwise, you must Stop. When you Stop, your utility is equal to your total points (up to 7), or 0 if you get a total of 8 points or higher. When you Toss, you receive no utility. There is no discounting (= 1).

(a) What are the states and the actions for this MDP? Which states are terminal?
(b) What is the transition function and the reward function for this MDP? Hint: The problem may be simpler to formulate using the general version of rewards: R(s, a, s')
(c) Run value iteration to find the optimal value function V* for the MDP. Show each Vk step (starting from Vo(s) = 0 for all states s). For a reasonable MDP formulation, this should converge in fewer than 10 steps. If you find it too tedious to do by hand, you may write a program to do this for you; however, there may be some benefit in seeing the calculation unfolding in front of you.
(d) Using the V* you found, determine the optimal policy for this MDP.

See answer in explanation

Step-by-step explanation:

State: current points if stop plus a terminal state, that is, 0,1,2,3,4,5,6,7,DONE

Action: Toss, Stop

2. What is the transition function and the reward function for this MDP?

Transition function:

T(Si , TOSS, Si+3) = 0.4 if i < 3

T(Si , TOSS, DONE) = 0.4 if i ≥ 3

T(Si , TOSS, Si+1) = 0.6 if i < 7

T(Si , TOSS, DONE) = 0.4 if i = 7

T(Si , STOP, DONE) = 1

Reward function:

R(Si , TOSS, ANY ) = 0

R(Si , STOP, DONE) = i

R(DONE, STOP, DONE) = 0

3. What is the optimal policy for this MDP? Please write down the steps to show how

you get the optimal policy.

Optimal policy: Toss for 0,1,2 ; STOP for others.

You should include the steps of value iteration. The value iteration will converge at

iteration 3. Result of iteration 3 is as follow,

V3: 0: 4.5 from Toss; 1: 5.4 from Toss; 2: 5.9 from Toss; 3: 3 from

Stop; 4: 4 from Stop; 5: 5 from Stop; 6: 6

step-by-step explanation:

a.   since the profit made in 2012 is equal to the profit made in 2014, the profit is           symmetric about the line y = 120,000.

step-by-step explanation:

given that megan owns a clothing boutique.

the profits of 2010, 11, are recorded and shown as follows:

years since     2010 0     1     2     3       4       5

profits made in 1000's   39       84     111     120   111     84

we find that profit gradually increases form 2010 to 2013 upto 120,   then starts decreasing.

the profits for 2012 and 2014 are the same while that of 2011 and 2015 are also the same.

this shows that there is symmetry about the line profit = 120000

hence answer is:

a.   since the profit made in 2012 is equal to the profit made in 2014, the profit is           symmetric about the line y = 120,000.

read more on -

27 + 13x + 15 = (x + 5)( =

step-by-step explanation:

27 + 13x + 15 = (x + 5)( = 27 + 13x + 15 = (x + 5)( = 27 + 13x + 15 = (x + 5)( =