In our credit card example, the bank starts with some vague idea of what constitutes a good credit risk. So, as customers x1, x2, ... , XN arrive, the bank applies its vague idea to approve credit cards for some of these customers. Then, only those who got credit cards are monitored to see if they default or not. For simplicity, suppose that the first N customers were given cred it cards. Now that the bank knows the behavior of these customers, it comes to you to improve their algorithm for approving credit. The bank gives you the data (x1, y1), ... ' (xN , YN). Before you look at the data, you do mathematical derivations and come up with a credit approval function. You now test it on the data and, to your delight, obtain perfect prediction.
(a) What is M, the size of your hypothesis set?
(b) With such an M, what does the Hoeffding bound say about the probability that the true performance is worse than 2% error for N = 10000?
(c) You give your g to the bank and assure them that the performance will be better than 2% error and your confidence is given by your answer to part (b). The bank is thrilled and uses your g to approve credit for new clients. To their dismay, more than half their credit cards are being defaulted on. Explain the possible reason(s) behind this outcome.
(d) Is there a way in which the bank could use your credit approval function to have your probabilistic guarantee? How?