## Assignments, Announcements, Hand-Outs, etc.

Welcome to Math 6380 (Stochastic Processes)! Homework will (usually) be posted here on Mondays or Tuesdays, and due in class the following Monday (unless stated otherwise). We will usually go over the homeworks when they are due, sometimes followed by a quick quiz on the material.

• Homework #1, due Monday, August 29 (Some tough & interesting problems. Enjoy!)

Chapter 1, problems #1,2,3,5,6,14,19,20,25

• Homework #2, due Wednesday, September 6.

Chapter 2, problems # 3,4,5,8,9

Continue problem 2.4 by calculating Cov(N(t),N(t+s)).

• Homework #3, due Monday, September 18.

(1) Let N(t) be a nonstationary Poisson process with rate function lambda(t) = t. Find the distribution of the time of the first event, given that there is exactly one event in [0,t].

(2) Let U1, U2, ... be independent U[0,1] r.v.'s. Fix t in [0,1], and let N be the smallest integer so that U1+U2+...UN > t. Find E(N). In particular, what is the answer when t=1? Hint: To get a feel for the answer, this is a perfect opportunity to practice writing simple simulations.

Chapter 2, problems #11,33,39

(September 18) I forgot to mention in class today that according to the syllabus, Exam #1 will be on Wednesday, September 27, i.e., next week. My exams are open-book, open-notes. Next Monday 9/25 will be a review day.

Also, in class today I didn't give a nice solution for one of the homework problems (chapter 2 #11). Here is a much better solution.

• Homework #4, due Monday, September 25.

(1) Consider N(t) from problem (2) in HW #3, i.e., N(t) is the least n so that U1+U2+...Un > t. Note that the renewal function m(t) is defined slightly differently. Make sure you understand that m(t) = N(t)-1. So, we showed that in this case m(t) = exp(t)-1 when t in [0,1]. Now, use the same "tricks" to find m(t) for t in [1,2].

(2) Problems 3.4 and 3.9 in the text.

• Here are solutions for the exam.

• Homework #5, due Monday, Octover 9.

Continuing Problem 1f on the exam:

(a) Find E(S_2 | N(t)=3) and E(S_3 | N(t) = 2)

(b) Can you find a general formula (expressed as a sum) for E(S_k | N(t) = n) ?

Here's an interesting problem about stopping times and Wald's Lemma: Let X_1,X_2,... be iid nonnegative rv's, and let N(t) be the associated counting process, i.e., N(t) = max(n: X_1+...+X_n < t).

(a) Fix t. Then N(t) is an integer valued rv. Is N(t) a stopping time?

(b) Suppose X ~ expon(lambda), i.e., N(t) is a Poisson process. Let Y = sum {i=1 to N(t)} X_i. Explain why it has to be true that E(Y) < t.

Is E(Y) = E(X)E(N(t))?

Ross, Chapter 4, problem #1.

October 4. In class today I gave examples of random sequences that are (provably) Markov chains. The first two were basically sums of iid rv's. The third was a simple urn game, and the last was the M/M/1 queue. However I didn't do a good job of describing a random sequence that was not a Markov chain. So here are two.

(1) Let X_0, X_1,... be rolls of a fair die, and let Y_n = X_n + X_{n-1}. Is {Y_1,Y_2,...} a Markov chain? Hint: Show that P(Y_3 = 8 | Y_2 = 7, Y_1 = 12) = 0, then show P(Y_3 = 8 | Y_2 = 7) > 0.

(2) There are two coins with P(H) = p_i for coin i. You choose one of the coins at random and flip it repeatedly. Let X_n = 0 or 1 depending on whether the n-th flip is H or T. Is {X_0,X_1,...} a Markov chain? Hint: Calculate P(X_2 = 0 | X_1 = 0, X_0 = 0) and P(X_2 = 0 | X_1 = 0).

Here is the derivation of the mean of the k-th order statistic for n U[0,1] rv's.

• Homework #6, due October 16

(1) Let {X_0,X_1,X_2,...} be a Markov chain on the states {1,2,...}, with transition matrix P(i,i+1) = i/(i+1), and P(i,1) = 1/(i+1) for each i=1,2,... . Show that state 1 is recurrent. Are all the states recurrent?

(2) Now suppose P(i,i+1) = i^2/(i^2 + 1), and P(i,1) = 1/(i^2 + 1). Show that state 1 is transient. Are all the states transient?

(3) Continuing problem (2), let N be the number of times {X_0,X_1,...} visits state 1. Find E(N). (Actually, the best you can probably do is estimate E(N). Feel free to use a computer.)

• Homework #7, due Monday October 23

Recall the Urn game where on each turn you first randomly choose a ball from Urn 1 and place it in Urn 2, then randomly choose a ball from Urn 2 and place it in Urn 1. This time suppose there are 4 Red balls and 4 Blue balls, with 4 in each urn. Find the transition matrix, and steady state equations. Solve the equations. (You can use a computer, but shouldn't need to.) Verify that \pi P = \pi.

Ross, Chapter 4 #12, 18

• Final Exam schedule is now available. Our exam will be on Monday, December 11, from 2:00-4:00.
• As mentioned in class, Midterm Exam #2 will be on Wednesday, November 1. The exam will cover chapters 3 and 4. The Monday class before the exam will be for review, so bring your questions!
• Homework #8, due Monday October 30

(1) For the urn game from HW#7 (with 4 balls in each urn) find

(a) Expected number of transitions to get from state 0 to state 4.

(b) Variance of the number of transitions to get from state 0 to state 4.

(c) Probability to reach state 0 before state 4, starting from states 1,2,3.

(2) You play Roulette at the casino where you win each game with probability 18/38 and lose with probability 20/38. You start with $10 and stop when you have$0 or $20. (Gambler's ruin problem.) (a) If you wager$1 each time, find the expected numberof games you will play, and the probability you end up with $0. (b) Do the same thing assuming you wager$2 each game.

Feel free to use a computer to solve these problems.

• Here are solutions for the second midterm exam.
• Homework #9, due Monday, November 27

1) Consider a "car wash" where potential customers arrive as a Poisson process with rate lambda = 10 per hour, and the time to wash a car is exponentially distributed with mean 10 minutes (i.e., mu = 6). If a customer arrives when there are n customers in the system, he enters with probability 1/(n+1), and otherwise "disappears".

a) Describe the model by finding the transition matrix for the embedded Markov chain and rate vector, and then write the infinitesimal generator, Q, for the model.

b) Write the equations to find the steady state vector, pi. Use Matlab (or R, Python, etc.) to solve. (You will have to truncate Q somewhere.)

c) Find the mean and variance of the time to get from state 0 to state 5.

d) How would you find the probability you reach state 0 before you reach state 5, starting from state 3?

e) If customers pay \$1 for the car wash, how much money does the place make per hour (on average)?

f) What fraction of potential customers enter the car wash?

• Homework #10, due Wednesday, December 6

Here are some problems to get started on. I will probably add a few more.

1) Let X(t) be standard Brownian motion. for s < t, find E(X(s)+X(t)) and Var(X(s)+X(t)).

2) For t1 < t2 < t3, find E(X(t1)X(t2)X(t3)).

• Here is Matlab code that solves the problems in HW#9.
• Here are solutions for the Final Exam. I should have them graded by the end of the week. I hope everybody enjoyed the class!