As part of a bigger project, I needed to simulate a text string based on a source document, but at the character level. Just in case people find the code useful, I’ve uploaded it to MCMCtext.r.
In my simulated text, each character is chosen based on the transition probabilities in the source text from one character to another. The result is (nearly complete) gibberish without much interest to anyone, except perhaps those looking for a replacement for the standard Lorum Ibsum dummy text. More interesting fake text could be generated by using two character (or more) transition probabilities, or by working at the level of words.
Before moving on, I thought it might be interesting to see if anyone can “reverse engineer” my fake text output to figure out which original text was used as a source to generate it. Got that? The source text comes from Project Gutenberg. Hint: some features of the (fake) text could help you narrow the field of candidates.
First person to post a correct guess in the comments gets a copy of my comic and an unlimited supply of Hotpockets*. Limit one guess per person please.
* Hotpockets offer only valid if you are currently saving the planet from destruction.
The embarrassing failure of experts to predict the future is well known and has been exhaustively cataloged by writers such as Nassim Taleb and Nate Silver. It often seems like the more someone knows about a subject, the worse they are at predicting what will happen in that area. In this spirit, I bring you seven predictions for the coming year, based on an absolute minimal amount of knowledge. Combined, I’ve spent less than one hour studying these topics, with the exception of bubbles in general. Here they are, we’ll see how well this exercise in ignorance holds up.
Prediction: The EU crisis will not be solved, more good money will be thrown after bad. Source of prediction: The last three years. Also, bankers (for now) still rule the world.
Prediction: Bubbles will begin to burst. It could be the Higher Education Bubble, the Regulatory Complexity Bubble, the Government Money Printing (aka Fiat Currency) Bubble, or the closely related Global Debt Bubble. Source of prediction: In Western, developed nations, all of these these have gone exponential. That never lasts because it can’t.
Prediction: Obamacare (officially know as PPACA, had to look that up), will begin to look like the Democrat’s Vietnam. Those who supported the legislation will need to make a quick u-turn or double down on their support, riding the increasingly unpopular, cost-overrunning quagmire into the depths of bureaucratic hell. Source of prediction: I’m not a zombie.
Prediction: Speaking of soon-to-fail-in-its-intended-goals legislation, Dodd-Frank will have many nasty unintended consequences. Source of prediction: It’s 8,800 pages long and still leaves lots of things to be worked out later by regulators.
Prediction: The meltdown at Fukushima will still be a problem at the end of the year. Source of prediction: The most radioactive fish of all time (by a factor of 10) was just caught. The lovely, and glowing, Murasoi is shown at top.
Prediction: The Miami Heat will win the NBA. Oklahoma will peter out. Source of prediction: I stopped watching the NBA a few years ago, but as I recall Miami is a good team, and no one plays pro basketball in Oklahoma.
Prediction: At this year’s Oscars, lots of pretentious, overwrought crap will get awards. Source of prediction: Recent history, reading the nominee list, watching a trailer for Les Miz.
TRANSCRIPT OF VIDEO:
Hello, I’m Matt Asher with StatisticsBlog.com. This video is about my attempt to simulate a landmine clearing device built by Massoud Hassani called the Mine Kafon. I’ve put a link to his webpage at StatisticsBlog.com, I highly recommend checking out the video. Hassani’s device looks like this:
It’s a cheap, easy to build mine clearer that travels under the power of the wind. When I first saw the video, I was awestruck by what Hassani had done. It seemed like an awesome achievement, cleaver, creative, a fantastic idea for making the world better. A device that used the power of nature to clean up after man.
The more I thought about it, though, the more I wondered what might happen if hundreds of Kafons were sent out onto a mine field. So to examine that question, I built a simulation. I’ll run it now.
I’ve slowed it down so you can see what’s happening. Each blue line represents the path a Kafon might take across the minefield. The red circles represent exploded mines, and the gray parts are places where paths have overlapped.
Based on Hassani’s video, I’ve assumed there’s a prevailing wind that sweeps the devices right out into the field, and that the Kafons are released at equal intervals at the edge of the minefield. The movements up and down represent turbulence, uneven ground, or the natural tendency of the Kafons themselves to move with a wobble.
I’ve posted the code to this simulation on my website, as I do for all my blog posts. It’s written in R, a free and open source programming language. You can go into my code and easily change the wind speed and other parameters, then re-run the simulation. For example, I’ve set the number of mines that each Kafon can absorb before it stops working to 4, that’s on based on what Hassani estimates, but you can change that up or down.
Paths that stop at a red circle before they traverse the whole minefield are Kafons that have “plated out” after hitting 4 mines.
Just looking at the simulation in this way, it seems to be working very well. Most of the Kafons are finding land mines, and lot of land mines are getting cleared.
The biggest problem I see with Hassani’s approach has to do with efficiency, especially as you try to get more and more of the mines detected. The more Kafons you send into the field, the more overlapping you get, and the lower your efficiency becomes, and the harder it gets to detect remaining mines. I ran the simulation with different numbers of Kafons, always spaced at equal intervals, which gives them the best chance to clear as many mines as possible.
Here’s a graph showing the percentage of unexploded mines still left versus the number of Kafons that were released into the minefield. Each point on the graph represents a new simulation with that many Kafons. As you can see, at first adding more Kafons gives an almost linear decrease in the percentage of mines left, but the closer you get to clearing all the mines, the more elusive that goal becomes. Even after 2000 Kafons have been released, which if moving perfectly straight could have covered the area of our simulation four times over, there are still some mines left unexploded.
If you look on my blog you’ll see a post I did about something I call The Unicorn Problem, related to finding all of the new species in an environment. The problem there, as with this approach, is that the marginal rate of detection goes down as the number of attempts goes up. What’s happening, is that the more Kafons you use, the more they overlap territory.
Here you can see the amount of territory that’s be traversed more than once by a Kafon. The overall result is that the cost per mine destroyed gets higher and higher as you get closer to and closer to eliminating all of them. Here’s a plot of the cost per destroyed mine versus the number of Kafons used.
End result is cost per mine detected keeps increasing. Hassini estimates a cost of 40 euros to build each device, or about $50.
I wish Hassani the best of luck with his project, hopefully these issues can be addressed. Whatever the faults with this approach, this is a very important thing he’s doing. I noticed that in a more recent video he mentioned tracking the Kafon’s motion with GPS. I don’t know whether his initial price estimate included the cost of GPS. At any rate this would help to keep track of which areas have been covered and which haven’t, but unless he’s using a GPS accurate to within a foot, I wouldn’t want to try and walk in the exact path cleared by the device. In my simulation I’m assuming that every single time the Kafon is in the area of a mine it explodes it. It’s not clear that would be the case. It would be easy enough to add a probability of failure to the simulation.
There are adjustments you can make to the simulation or its parameters which would result in more of the Kafons being effective, though any design that relies on wind patterns is going to suffer from the same diminishing returns and unicorn problem, even if the wind is widely turbulent and increases the probability that all of the plates will get used, there’s still the problem of overlap and perhaps even worse performance if the Kafons get stuck in one area, or are quickly blown out of the mine field. Overall there are lots of reasons not to want to walk out into a mine field, no matter how many kafons have been through it.
Even if the Mine Kafon isn’t the best option for clearing an entire region of mines, they might still be an effective way to test for the presence of mines in an area, to do a sample of the area and see how likely it is to contain land mines, and if so how many and what regions might have higher concentrations of mines.
