> "From 2001 through 2013, the fund’s worst year was a 21 percent gain, after subtracting fees. Medallion reaped a 98.2 percent gain in 2008, the year the Standard & Poor’s 500 Index lost 38.5 percent."
Holy cow, that thing is kinda crazy. There has to be a catch, right? (Or everybody would be doing it.) Or is their algorithm genuinely so clever that nobody else has ever figured it out?
They were (one of?) the first serious quant fund, started by James Simon who before that was a math professor at Stony Brook. Along with the NSA they are one of the most aggressive non-academic would be employers of top notch math Phds. They also hire Phds in physics and statistics. The big draw isn't so much the salary as the ability to invest in Medallion.
If you take some of the smartest people in the world and have them intensely focus on making money via trading you get Renaissance. Everyone can't do it because there are only so many some of the smartest people in the world intensely interested in making money via trading to go around.
A side-note on the smart people: Nick Patterson, a cold-war cryptologist who had worked at Renaissance for a few years, talked a little about how they had employed some very smart people doing some very basic things like simple regression, and that the people needed to be smart to know how to apply these tools properly, and ensure the data they were applying it on was good.
Everyone can't do it because there are only so many some of the smartest people in the world intensely interested in making money via trading to go around.
No, everyone can't do it because they are in an 0-sum game. If everyone was as good as Renaissance, then Renaissance wouldn't make money. But if all the smart people are in Renaissance, then Renaissance makes money.
I certainly think they are smart guys, but I doubt whether what they are doing is highly complex in the sense of being inexplicable to ordinary math graduates.
Like with a lot of math, you learn more about the basics once you are on to the advanced stuff.
And the more advanced stuff you learn, the more you see limitations. Very advanced can very well mean brittle.
I reckon what they have is a well distilled common sense setup where they are able to reject models that appear to be profitable but aren't. I'm guessing they are also able to procedurally generate models that capture the same few common sense notions in new ways as the market evolves. In contrast I reckon most quant shops are still at a stage where the models are hand coded, causing some severe issues that I'll touch upon.
There's also an organisational aspect about them. They're all very very good at math. Any BS is likely to get rumbled. Intellectual carelessness will probably be detected and the boss is a proper math guy too. Here's a subtle thing that many people I've worked with would not detect: to calculate my fund's information ratio, I take the monthly returns, average them, and divide by the standard deviation. I have to multiply by root 12 because there's 12 months in a year. Standard procedure.
Having worked with some mere mortals I can tell you a lot of time is spent convincing oneself of the intellectual merits of something one's found despite evidence to the contrary. I think the fact that quants tend to be low quality coders with high internal prestige causes a lot of funds to do less well than they should. A lot of effort goes into hand tuning models rather than trying a bunch of different ones. You get things like unconscious "optimizations" that are very hard to spot, and hard to argue against because the person doing the talking is of high rank.
Traditional, long-only asset managers are still hired mostly out of some combination of nepotism, status signalling, cover-your-ass advisory board issues.
When I worked in a long-only equities asset manager, there even were frank discussions with the client services team about how to organize materials for when a client was preparing to fire us. We actually assisted them with the process of firing us, told them what to say, etc., so they would save face with the rest of the board or whomever they answered to.
Basically, it's a lot like hiring a manager in international soccer. Very few of them actually bring results, but big, wealthy clubs will still go through a huge media circus, pay a huge wage, and do all sorts of things, to hire status-signalling managers. Then, later, when it's not working out, it is that manager's job to be sacked gracefully, once again creating media buzz that makes the club look good in front of the fan base for "doing the right thing" and firing the manager.
In asset management it's the same. "Hire-us-now-to-look-fancy" and "Fire-us-later-to-look-like-you're-dedicated-to-ideals-and-righting-the-wrongs" are both very much the major parts of the services they provide, and both are things that these companies go on at length about when courting a new prospective client.
Heck, some client service employees even get part of their bonuses based on how happy the clients are with the assistance they receive in drafting press releases, putting together slide decks, or writing white papers about firing that manager.
Yeah, I went off on a tangent. The stuff ordinary guys do is very easily systematised by a machine. Problem is the "system" often doesn't do that well (looking at long period numbers, so you get long periods of non performance), so you're back to "trust me, I've done this" and then the guy with the nice marketing docs wins again. Or we'll see. It's a common idea that you might as well follow the market in some way, so maybe cheap will win this time.
you can get very high stats (good shapre , low draw-downs, lots of of winning months) just by going 'long' short-duration bonds and or selling far out of money index options. The problem is you may not make much money. You can sell a S&P 500 1000 put but you'll only make $500 over 6-12 months with $100k at risk. However, this is path dependent, meaning that large losses may be sustained on paper until the storm subsides. Even long 3-month t bills gives an infinitely high sharpe but the returns are crap
You could only have infinitely high Sharpe if the standard deviation of your returns, minus your financing cost, is zero. That means that Return = Financing + Constant. Now, some people are able to achieve that because they have abnormally low financing (e.g. they are an insurance firm with a large float, or a bank which is able to loan money at a higher rate than it borrows) but for most investors, the only way to make that equation hold is if the constant is zero or negative.
Going long 3 month T-bills does not give an infinitely high Sharpe Ratio (if it did, everyone who could would lever it up and do it in large size).
As an experiment, I simulated holding a long position in the nearest to expire Eurodollar futures contract (which give a return similar to the 3 month T-bill return) and rolling every 3 months, which gives a Sharpe of 0.92, an annualized return of 0.65% and annualized standard deviation of 0.7%.
Similarly, a long position in nearest-month two year treasury futures contracts gives a Sharpe of 0.94 with 1.59% annual return and 1.67% volatility.
These are attractive Sharpes (better than equities!) but they are certainly not infinite, and to juice up the returns to anything approaching an equity investment you need to be looking at 5-10x leverage.
Your point about selling puts, with skewness/kurtosis risk which is not priced by Sharpe, is a fair one, and probably the most common method of gaming returns, but it is a side-issue.
> Even long 3-month t bills gives an infinitely high sharpe but the returns are crap
If your investment has a high Sharpe ratio (especially in respect to other investments), by definition you have the highest returns at the lowest risk (variance) possible, so you can't have a "infinitely high" Sharpe ratio with "crap" returns.
Investments with low returns and low risks are penalized by Sharpe just as much as investments with high returns and high risks for the same reason: the expected value is simply less in both cases than for the ones with highest returns at lowest risk.
If you're a hedge/venture fund you will want to optimize for a high Sharpe portfolio eventually, you simply can't survive if you keep losing money for investors in the long run (low expected value).
Selling out of the money puts will almost never generate a downdown. Hence an infinite sharpe because the denominator is zero. Technically you are still beating the risk-free rate but there is a lot of downside risk not accounted for in the shapre ratio. Often a huge amount of leverage is applied to boost returns.
except for the occasional once-in-a-decade event that would bankrupt you. On the time scale where selling out-of-money puts makes sense, the market is efficient and there is no free lunch.
spending a decade high rolling at yacht week is totally worth it though. don't pretend like unsustainable luxury is a bad thing, much better than just dreaming about it
also, you can mitigate black swan events by not being too leveraged up, and going further out on expirations. and finally, just keep rolling.
No, you can't mitigate black swans that way. To get any return on selling out of the money put options (with even the remote semblance of sustainability) you have to have a crazy 100x or 1000x leverage.
And you can't just going further out on expirations without posting proper margins. That is, you can, but not legally.
The difficult part is getting regulated, and raising money.
Robo-advisers are subject to the same regulation as traditional asset managers - a regulatory burden which is only likely to increase over time. The legal fees associated with regulation are not cheap.
Compounding the problem, a competitive robo-adviser needs to offer lower fees than a traditional asset manager. Probably they need to be in the 0.25-0.5% range, which means that for each $1m under management they are picking up $5,000 of revenue. Out of that they need to pay salaries, infrastructure costs, legal costs, rent, taxes etc. Let's be conservative and say that these costs are $2.5m per year. That means that a robo-advisor needs about $0.5bn under management before it begins to turn a profit.
I don't think it's really suitable as a side project.
Here is my attempt to work through the math and figure out how "surprising" this result is.
Clearly, we should expect that for small primes (< 100e6) it is less likely that a prime ending in K (in base B) will be followed by another prime ending in K - because for that to happen, none of the B-1 numbers in between can be prime.
A (very naive) model of the distribution of primes says that every number n has probability p(n) = 1/log(n) of being prime. Assume that a number n ends with a k in base b. Define p = 1/log(n). Then the probability that the next prime ends in k+j is, roughly,
q(j) = p * (1-p)^(j-1) * sum_{i=0}^{infinity} (1-p)^(i*b)
= p * (1-p)^(j-1) / (1 - (1-p)^b)
In this formula, j takes values 1 to b (where j = b represents another prime ending in k).
For n ~ 1,000,000 and working in base b, under this model we would expect to see around 6.97% of primes ending in k followed by another prime ending in k, whereas we expect to see 13.7% of primes ending in k+1 (it is apparent how naive the model is, since in fact we never see a prime ending in k followed by a prime ending in k+1, except for 2,3). It would not be hard to extend the model to rule out even primes, or multiples of 3 and 5, but I have not done this.
Around n ~ 10^60 the distribution starts to look more equal, as the primes are "spread out" enough that you expect to have long sequences of non-primes between the primes, which blurs out the distribution to be roughly constant.
I think this is what the article is getting at when it quotes James Maynard as saying "“It’s the rate at which they even out which is surprising to me". With a naive model of 'randomness' in the primes, you expect to see this phenomenon at low numbers (less then 10^60) and for it to slowly disappear at higher numbers. And indeed, you do see that, but the rate at which the phenomenon disappears is much slower than the random model predicts.
The results are particularly striking in base 11 - looking at primes below 100 million, only 4.3% of primes ending in 2 are followed by another prime ending in 2 (compared to the 9.1% you would naively expect) with similar numbers for other pairs.
A prime ending in 2 (in base 11) is also unlikely to be following by a prime ending in 5, 7 or 9, whereas it is particularly likely to be following by a prime ending in 4 or 8.
It would be interesting to know what structure there is (if any) in this NxN "transition matrix" for various bases.
Wow, that's really interesting. The same seems to be true for base 7; I haven't tried any other prime bases yet (and I don't know how it might extend to non-prime bases). Anyone have an idea why this seems to hold?
Edit: This basically works for base 10, too. I feel like the reason must either be very obvious or very deep.
Well, whether it's obvious or not, I think it's in the paper. Immediately under equation (1.1) of http://arxiv.org/abs/1603.03720, the authors are discussing the second correction term to the distribution of primes mod q, and they state:
"We can also show that c2(q; (a, b)) = c2(q; (−b,−a)) for any two reduced residue classes a and b (mod q)."
I'm not 100% certain this is responsible for the phenomenon that we're seeing, but it seems exceedingly likely. I think I'd need to stare at their formula for c2 for a long time to understand where this relation comes from, though.
I'm pretty solidly convinced of the pattern at this point: I've checked the "reflect across the anti-diagonal" pattern for the first 10 million primes, expressed in every base from 3 to 20, and it seems to hold up. (I haven't tried to establish any sort of bounds.)
As I've expressed in another comment, my preferred way of thinking about this symmetric pattern goes something like this:
The probability that (a prime congruent to x mod b is followed by a prime congruent to y mod b) seems to be equal to the probability that (a prime congruent to -y mod b is followed by a prime congruent to -x mod b).
I still haven't figured out whether it ought to be obvious, though if it is then I expect the language I've used above to be relevant. It's definitely not trivially obvious, because it's not an exact equality: if the pattern only shows up clearly after you've accumulated thousands or millions of primes, then it doesn't seem that it could be enforced by any sort of exact transformation. (For example, if the symmetric entries were somehow just counting the same pairs in two different ways, the numbers ought to be precisely equal rather than just increasingly close.)
There are a rather a lot of other patterns in the data; I expect that at least some of them must be accounted for in the original paper, but I haven't more than glanced through it yet.
So a number ending in 'x' is as likely to be followed by 'y' as a number ending in 'n-x' is to be preceded by a number ending in 'n-y' (where n is the base).
Can't think of a trivial reason why that would be the case, something weird is happening.
Framing this differently, a prime congruent to x mod b is as likely to be followed by a prime congruent to y mod b as a prime congruent to -y mod b is to be followed by one congruent to -x mod b.
Why would you expect 9.1%? At low numbers like these, primes are likely to be closely packed, meaning you are not as likely to have to search forward 11 numbers as just dividing by 11 would imply.
Miscalculation - obviously no primes end in 0 (base 11) so I should have said "10% as you would naively expect".
I take your point about primes being closely packed at low numbers, but I think this is a small correction (i.e. you might expect 8-9% of primes ending in 2 to be followed by another prime ending in 2, but certainly not <4%)
I did my own investigation using base 3 and noticed something peculiar.
In the first 100k primes, we go from 1 to 2 29028 times and from 2 to 1 29029 times.
Then I filtered out the twin primes since those are the ones that exploit the fact that the next "possible" prime is one that flips the last digit.
This filtered out 10249 primes going from 2 to 1 (bringing the total below both the number of primes that stay at 2 (21008) and the number of primes that stay at 1 (20932)).
It didn't factor out any primes that go from 1 to 2. Are there no twin primes (p,p') where p%3 == 1 and p'%3 == 2?
edit: Oh hey, this is obvious, if p%3 is 1 then p+2 is divisible by 3. It does mean we don't need to take measurements to know that the result we are investigating cannot possibly account for everything since it isn't a factor at all when going from 1 to 2.
> Are there no twin primes (p,p') where p%3 == 1 and p'%3 == 2?
By definition, no there aren't. Twin primes are a distance of 2 apart, so your mod 3 options are 0 -> 2, 1 -> 0, and 2 -> 1. Anything involving 0 means it's divisible by 3, so your only mod 3 possibility for twin primes is 2 -> 1.
EDIT: Excluding the possibility where 3 mod 3 = 0, of course. This allows a 0 -> 2 transition with (3,5).
Yeah, I managed to get an edit in just before you replied -- I thought I was missing something obvious but I kept looking for a bug in my code instead of realizing that it's mathematically obvious :)
> Lemke Oliver and Soundararajan’s first guess for why this bias occurs was a simple one: Maybe a prime ending in 3, say, is more likely to be followed by a prime ending in 7, 9 or 1 merely because it encounters numbers with those endings before it reaches another number ending in 3. For example, 43 is followed by 47, 49 and 51 before it hits 53, and one of those numbers, 47, is prime.
> But the pair of mathematicians soon realized that this potential explanation couldn’t account for the magnitude of the biases they found. Nor could it explain why, as the pair found, primes ending in 3 seem to like being followed by primes ending in 9 more than 1 or 7.
Ok so the random model "1,3,7,9 mod 10" doesn't fully work, but let's look at what happens mod 30. Large primes have the following possible remainders mod 30: 1, 7, 11, 13, 17, 19, 23, 29. We see that when a prime ends with a 3 then p + 6 (ending in 9) is always an option, but p + 4 (ending in 7) is an option only half of the time. I think that this fully explains why a prime ending with 3 is more likely to be followed by a prime ending in 9. So basically the OP is on the right track, and his random model just needed to be refined a bit.
> The primes' preferences about the final digits of the primes that follow them can be explained, Soundararajan and Lemke Oliver found, using a much more refined model of randomness in primes, something called the prime k-tuples conjecture.
So I guess that my observation is just a special case of this "prime k-tuples conjecture".
Are you contesting it, or just curious? You already know that my observation explains the "3 followed by 9" bias. You already know that the mathematicians call the conjecture "a much more refined model of randomness in primes" which is similar to how I described what I was doing. In addition, MathWorld's article on the k-Tuple Conjecture talks about residues mod q, which is similar to what I'm doing when I look at primes mod 10*3. All these elements point at some connection between the k-tuple conjecture and my observation.
Well 'points to a connection' is not the same as 'is a special case of', I'm not an expert on this subject, but looking at the conjecture is about the asymptotic distribution of certain patterns in prime numbers. I don't think that an example like the one you are giving is 'related' except in a hand-wavy vague way that anything dealing with prime numbers and patterns is related to everything else dealing with prime numbers and patterns of primes.
All I meant by "special case" was that "mod 30" isn't the whole story -- more like the most significant correction on top of what the OP said, with other smaller corrections possible, and the entire set of corrections being described by the k-tuple conjecture.
It's amazing how people can be picky and negative on HN. Someone positive would instead congratulate me for making the gist of what the prime k-tuple conjecture says about the biases easily understandable. Oh well.
So you are making comments about pure mathematics. If you want to use imprecise language and not be corrected, you should probably go write a book review or something. In math, precise language and correcting someone or forcing someone to give justification for something is expected and completely usual. It would be bizarre when talking to a mathematician about mathematics if they didn't immediately correct or demand clarification and justification when you say something vague or incorrect or unjustified.
The first three comments were all variations on "this isn't new" so let's take that point for granted, and see what else this article gives us.
It's not new that Wall Street is hiring Ph.D scientists and mathematicians (that's been happening since the 80s) but what's changing is the kind of role they are getting hired for.
From the 1980s up until 2005 or perhaps even later, most Ph.Ds were hired in "quant" roles, that is, to build mathematical models that could price and manage the risk of derivatives. They were generally not in trading roles.
More recently (i.e. in the last decade) it's common to hire Ph.Ds as traders - that is, to write algorithms that are used to make trading decisions. This requires a rather different set of skills - instead of being skilled in stochastic calculus, partial differential equations and numerical methods, the quant trader is skilled in statistics, data analysis, optimisation and machine learning.
It's true that Wall Street has Ph.Ds hired to build trading models for many years (e.g. Renaissance) but the change is that this is no longer an esoteric fringe pursuit. It is seen as standard that trading desks at investment banks will have a large number of quant traders. Large European investment banks have heads of trading who are quants. Quantitative hedge funds are no longer mysterious and exciting - many of them use well understood models that have been widely replicated across the industry.
To be honest, I am surprised that it has taken this long. Finance is so obviously suited to these kinds of quantitative methods that only an aggressive rearguard action by voice traders has been able to keep them at bay. The question is no longer whether quantitative traders and their algorithms will largely replace voice traders, but how long the voice traders will manage to hold out.
Jane Street is an interesting special case. From my (somewhat limited) interactions with them, they are neither wholly voice traders nor wholly algorithmic. Instead, their researchers and traders build algorithmic trading systems which can be "driven "by humans. For example, the system will continually calculate a set of useful metrics that a human trader uses to make a final buy/sell decision, and trade is then immediately executed by high frequency execution algorithms. Or the human trader decides to make a complicated spread bet (e.g. long an ETF vs. a cost-optimized basket of the underlying stocks) and the algorithm goes out to execute the basket as effectively as possible.
I think there's an interesting parallel to "Centaur chess" [0] where the combination of a computer and a human is much more powerful than either of them acting alone.
> This requires a rather different set of skills - instead of being skilled in stochastic calculus, partial differential equations and numerical methods, the quant trader is skilled in statistics, data analysis, optimisation and machine learning.
Although I'm not a quant and do not even directly work in the finance industry, my experience working as part of a machine learning outfit definitely corroborates this. Data Science may be a hard term to truly define (it's quite broad, particularly when you start applying it in the real world) -- but the general skill set is definitely seeping into computer software/engineering roles. I'm not sure how it will go in the future (there are many viewpoints on this) but I wouldn't be surprised for this trend to continue.
That's the essence of complementarity. Cowen's "Average is Over" talks about it a lot, and uses many chess examples specifically to back it up. Recommended.
I think you misunderstood much of what tomp said, and willfully misinterpreted some of the rest.
The number of trades does not determine alpha, but it means that you can be much more certain about whether someone has alpha or not. For example, John Paulson made billions on (essentially) a single trade in 2007 and early 2008. Does he have alpha? It's hard to say, because all of those profits were from one trade, and he could have been lucky. Virtu Financial generates millions of dollars each year, by making tens of millions of trades. Do they have alpha? Absolutely - you can be certain of it, because it would be statistically impossible to get lucky tens of millions of times.
The idea "alpha becoming beta" is an extremely relevant one for many hedge funds today. As strategies become well known, they become commodified, and are often offered at a lower fee, both by hedge funds, ETFs and investment bank products. Frequently, they are offered for little or no performance fee, so they cannot be called "alpha" and are often referred to as "smart beta". For example, AQR Capital Management offers many low-fee funds giving exposure to value investing, momentum investing, managed futures, the FX carry trade and others. It sounds like you are using a very narrow definition of beta (exposure to the stock market) whereas the usage in the industry is much broader.
Pointing out that quant funds use "algorithms" to trade rather than "computers" is pointlessly picking holes. It's clear what he means.
That's not quite true. Scion was up in 2001 and 2002 (when the market was down) and they beat the market in 2003 and 2004 - in each of those years, they were primarily a long-only equity fund.
I'm not sure what the performance was in 2005. In 2006 they were down 16% (because of premiums on their CDS positions) which they then recouped in 2007 and early 2008.
I use MATLAB for 90% of my day to day work, for a combination of reasons -
1. Historically it is what has been used at my firm. We have a lot of code already written in MATLAB, interfaces to internal apis, external data providers etc. Everyone at the firm understands MATLAB code.
2. It really is very good for numerical work - both in terms of speed, and clarity of the code (much better than Python and R for clarity - probably on a par with Julia).
I also used KDB+/Q very heavily in a previous job. It is blazing fast in its domain (financial time series) and enables extremely rapid prototyping. The fact that it is a combined query language / programming language is very appealing for data-focused research. I wouldn't want to write a production system in it though (though I know people who have done!)
When I was studying for my PhD I wrote a lot of Fortran 77 and IDL. Essentially because my supervisor used IDL and had a lot of code written in it, and because we were using an external tool which consumed Fortran 77 files as input.
I also dislike R. Strongly disagree with the argument that MATLAB is more clear than python, however. Have you tried numpy/scipy? Provides excellent performance and very powerful APIs.
I've always found that a great deal of the MATLAB syntax does not gel with my expectations. "./" vs. "./" for instance, or using parenthesis to address array elements.
I would much prefer a truely OO capable language, as well.
It's one-based indexing combined with length being defined as the number of entries, IIRC. There was a blog post on here a few months ago that explained how you need one or the other to minimize the need for off-by-one corrections, though I can't find it now.
I'd consider switching to numpy (ignoring organizational issues) if it didn't take 10 times as many characters to describe all the operations. If you're spending you're day doing algorithm development having least squares and every linear algebra operation you can think of as first class citizens is a huge plus for readability and code cleanliness.
a) Not scalable (e.g. most intraday market-making or scalping strategies)
b) Only available at a very high fee (think north of 3/30)
c) Not available at any price (e.g. Renaissance Medallion, which has been closed to outside investors for 20+ years)
So I'm not sure that it's particularly relevant to this discussion, which is about fee pressure on traditional, long-only asset managers.